surveillance planning team presentation to the engineering management council february 21, 20001...
TRANSCRIPT
Surveillance Planning Team
Presentation to the
Engineering Management Council
February 21, 20001
Greg Robinson
Revision 4
Purpose of team
• To improve probability of mission success for NASA projects by improving Agency guidance for risk-based, functionally integrated surveillance.• Surveillance suitable to the project• Surveillance proportional to the risk• Surveillance that meets project management needs
Team Products: Policy Direction Procedural Guidance Streamlined Simplified Tools to Do the Job
Team Products: Policy Direction Procedural Guidance Streamlined Simplified Tools to Do the Job
Surveillance as a method to Surveillance as a method to improve likelihood of mission improve likelihood of mission successsuccess
Key objectives:Key objectives: Integrate surveillance approach:Integrate surveillance approach:
Engineering, S&MA, Finance, ProcurementEngineering, S&MA, Finance, Procurement Improve effective utilization of surveillance resourcesImprove effective utilization of surveillance resources Enhance communication of surveillance resultsEnhance communication of surveillance results Surveillance to proactively mitigate risk (part of RBAM)Surveillance to proactively mitigate risk (part of RBAM) Make surveillance more response to project managementMake surveillance more response to project management
Background (PBC Report)
• July 1999, Administrator tasked the AA for Procurement to assess how the Agency could improve implementation of Performance Based Contracting (PBC)
• June 2000, the PBC Team released their report.
A Significant Finding: Many NASA Centers have not developed or implemented contract surveillance plans. This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.
A Significant Finding: Many NASA Centers have not developed or implemented contract surveillance plans. This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.
Background (NIAT)
• October 20, NASA Integrated Action Team (NIAT) Report, Enhancing Mission Success - A Framework for the Future
NIAT-13 Action Summary: Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.
NIAT-13 Action Summary: Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.
Surveillance Planning Team: Chartered by AA for Procurement and
AA for OSMA, November 2000 Also sponsored by the Office of the
Chief Engineer
Representation:
NASA Centers:• JSC• KSC• MSFC• SSC• GSFC
NASA Headquarters:• Code Q• Code H• Code M• Code AE• Code S• Code Y
Team Sponsorship
Current Members:• JSC - Jim Nise (SOMO Engineering Management) • JSC - Lee Norbraten (Management Systems Office)• KSC - Larry Tucci (S&MA)• KSC - David Culp (Procurement)• KSC- Terry Smith (Quality)• MSFC - Barry Musick (Engineering) (for Jim Kennedy) • MSFC - Charlie Chesser (S&MA)• SSC - Mike Smiles (S&MA)• GSFC - Mary Clark (EVM)• S - Nancy Porter (Program Office - Space Science)• Y - Tom Magner (Program Office - Earth Science)• M - John Castellano (Program Office - HEDS)• Q - Steve Newman (PBMA)• Q - Tom Whitmeyer (Quality)• H - Jeff Cullen (Procurement)
Team Membership
Tools To Be UsedTools To Be Used
Root Cause AnalysisRoot Cause Analysis
Customer CoordinationCustomer Coordination
Functional AnalysisFunctional Analysis
Benchmarking of Best PracticesBenchmarking of Best Practices
• NIAT and PBC findings analyzed to determine underlying contributing factors.
• EMC and PMC review.
• Team determines Engineering, Procurement, S&MA objectives for surveillance, and identify methods for improved implementation and integration.
• Team validates approach against best-of the best in surveillance activity.
Lack of meaningful guidance for conducting contract surveillance
Existing NASA Policy, Procedural Guidance, and FAR Supplementation inadequate
Need for continuous risk evaluation of contractor performance(not a one-time plan)
“Stove pipe” approach to performing surveillance
Contributing Factors• Lack of common definition• Range of contract mechanisms• Involvement of multiple functional organizations• Differences in approaches across programs and centers• Lack of agency wide training
Root Causes Currently Identified
Team Strategy
CHALLENGES:Large/Diverse Project Base Large/Diverse Functional Support Organizations
CHALLENGES:Large/Diverse Project Base Large/Diverse Functional Support Organizations
Define surveillance needs
Provide “risk based” options
Utilize Existing Guidance Where Possible: NPG 7120.5 NPG 8735.2 NASA FAR and Supplement Other (EVM etc)
Schedule and ProductsSchedule and Products
Initial Meeting
Agency Objectives
Outline for NPD
Draft NPD
One Week
Benchmark Study
Benchmark Report
Draft NPG ForSurveillance
Publish Summary of Guidance Examples
Jan-00 Feb-00 April-00Telecon
1/30/00
July-00
Final Report for
Chief Engineer
Recommended Follow On Actions
(Training, Etc.)
MeetingMeeting
1/9/00
EMC Meeting
2/22/00MeetingTelecon
Current Status - Next StepsCurrent Status - Next Steps Draft NPD Draft NPD Survey Survey NPG Initial Approach (see attached)NPG Initial Approach (see attached) Benchmarking Plans Benchmarking Plans
IndustryIndustry AgenciesAgencies
Status meeting with PMCStatus meeting with PMC
• AE/Office of the Chief Engineer is responsible for the NPD, project managers are responsible for initiating and maintaining the surveillance plans.
• Plans are required for contracts, grants, cooperative agreements and MOUs unless otherwise specified by the Center Director
• The surveillance plan will:• Establish and maintain direct linkage with risk of the contract• Have a level of surveillance penetration that is proportional to risk areas and to the
scope and risk of work to be performed.• Be a collaborative integrated effort with procurement, engineering, S&MA, and finance.• Be initiated during development of the SOW • Provide definition of government surveillance by type, responsibility, and frequency. • Specify if or how sub contract work is addressed.• Provide for the communication of surveillance plans and results in a defined manner to
appropriate levels of management.
• NASA engineering, procurement, safety and mission assurance, and other technical organizations shall provide surveillance support to program/project management through collaborative involvement in the development and maintenance of surveillance plans and performance of associated levels of surveillance penetration.
Surveillance NPD - Key Elements
EMC BACKUP CHARTSEMC BACKUP CHARTS
“Prime” (PBC) Contractor
NASA as Integrator
In-House
Increasing level of integration responsibility andaccountabilityassigned tocontractors
80% PBC Goal
Changing Environment
Project Management/Risk Management
ProjectManagement
Cost Schedule
TechnicalPerformance
ContractManagement
MissionAssurance
Safety/Security
Risk Management is a Project Management
Tool to Balance Requirements/Resources
S&MA
ProcurementEngineering
Functional Organizations Provide Expertise/Information to
Support this process
Finance
Elements of a PBC Contract
Performance Requirements and Standards
Completion Form Contract Type(FFP, FPIF, CPIF, CPAF -- but not LOE)
Well Defined Metrics
Contractor Quality AssurancePlan
Comprehensive Surveillance Plan
Fee for Results Not Best Efforts
Surveillance Methods and Tools
Control • Cause and Effect Analysis• Closing a Risk• Cost-Benefit Analysis • List Reduction• Mitigation Status Report• Multivoting• PERT Chart• Problem-Solving Planning• Risk Information Sheet• Spreadsheet Risk Tracking • Stoplight Chart
AnalyzePlan
Track
• Baseline Planning• Planning Decision Flowchart• Planning Worksheet
- Brainstorming- Cause and Effect Analysis- Cost-Benefit Analysis- Gantt Charts- Goal-Question-Measure- Interrelationship Digraph- List Reduction- Multivoting- PERT Chart- Work Breakdown Structure
• Risk Information Sheet
• Bar Graph• Mitigation Status Report• Risk Information Sheet• Spreadsheet Risk Tracking• Stoplight Chart• Time Correlation Chart• Time Graph
Surveillance PlanA Surveillance Plan documentshow risks will be monitored on aproject: the process, activities,milestones, and responsibilities
associated with surveillance. It is a subset of theproject plan and is written before the project begins.
Identify
• Baseline Identification and Analysis• Contractor provided metrics• Contractor Monthly Reports• Technical Interface Meetings• Milestone Reviews• Inspections• Assessments
• Test Reports• Financial reports• EVM• Failure reports
• Project Metrics
• Project Metrics
Identify
Analy
ze
Plan
Track
Control
Communicate
Risk Drives Areas of Surveillance
W e a k Im p le m e n ta t i o n
R i s k M a n a g e m e n t P r o c e s s
C o m m u n i c a t i o n
R e q u i r e m e n tD e f i n i t i o n
In fo r m a l o r R e s t r ic t e d
C o n t r o l l e d P ro c e s s
P l a n n in g
T e c h n o l o g yR e a d i n e s s
E x p e r ie n c e o fM a n a g e m e n t T e a m
C o s t
R e v i e w s
E x te n s i v e , P e e r &In d e p e n d e n t
L im i te d P ro je c t In te r n a l
P ro v e nT e a m
O J T
T R L 5 - 6
T R L 1 - 3
E x is t i n g
E x te n s i v e , U p - F r o n tR e a c t i v e
C le a r , F i x e d ,P a re n t - C h i ld
D e v e lo p e d a s N e e d e d ,F re e F lo a t
D y n a m ic ,In te r a c t i v e
T e c h n i c a l T e a mC o h e s i v e , A u th o r i t y
W e a k l y C o u p le d
T ra d a b le R e s o u rc e
P e r fo r m a n c e th eD o m in a n t V a r ia b le
L o w e s tR i s k
S a f e t y a n d M i s s i o nA s s u r a n c e
L im i te dIn v o l v e m e n t
E x te n s i v e In v o l v e m e n t
Surveillance Proportional To Risk
Surveillance Approach
Evaluated Risk
Based on “Risk Management”
Tailored toselected Contractor
Tailored toselected Contractor
Developed prior to awardDeveloped
prior to award
Not incorporated into the contract
Not incorporated into the contract
Insight and Oversight, commensurate with riskInsight and Oversight, commensurate with risk
DynamicDynamic
Surveillance Suitable to the Project
Based on “Risk Management”
DynamicDynamic
Low Dollar Amount
Non Flight
No Hazardous Operations
High Dollar Amount
Flight
Hazardous Operations
Surveillance Dynamics (RBAM)
Final Surveillance Plan
(High/MediumRisk) based on
selected contractor)
ACQUISITION STRATEGY PROCESS
EngineeringProcurement
S&MA, Other
REQUIREMENTS POST AWARDSOURCE
SELECTIONSOLICITATION DEVELOPMENT
ACQUISITION STRATEGY
DEVELOPMENT
Contract Performance
RISK RANKING
HIGH RISKMEDIUM RISK
LOW RISK
22
Project ManagerRisk Management
Surveillance Results
PreliminarySurveillance Plan
Level of Surveillance Penetration
Consequence of Failure
5
4
3
2
1
Likelihood
1 2 3 4 5
Did they do th
e
right th
ings?
Did they do th
ings
right an
d did they do
them rig
ht?
Increasi
ng technica
l penetr
ation
Review of
Processes
Review of P
rocess
and Implem
entation
Mapped on Risk Chart
Surveillance Continuum
Did they do the right things?
Did they do the right things and did they do them
right?
Increasing technical penetration
Review of Processes
Review of Process and
Implementation
Level 0 Level 1 Level 2 Level 3 Level 4
No Penetration
Level of surveillance contingent on defining an acceptable risk:Level of surveillance contingent on defining an acceptable risk:
• Technical risk levels
• Amount of trust in performing organization’s abilities
• How well processes are defined
• Level at which NASA is performing Task Agreements for the program
• Human rating of vehicle
• Program visibility and impact of failure
• Design complexity, manufacturing complexity, producibility
• Value of asset
Surveillance Evolves as Risk is Better Defined
WB
S
Formulation ImplementationSRR PDR CDR FRR OperationsProduction
WB
S
WB
S
Top-Level
Rqmts.
Trades
Concepts.
SystemSpec.
EndItem
Specs.
Analysis
Prelim.Drawings
DetailedDrawings
RiskPlan
Production/Manufacturing
Package
H/W & S/WVerification
Plan
Acceptance/COFR
Risk PenetrationSurveillance Matrix
Identified RiskIdentified Risk
No Penetration• Accept performing organization’s tasks at face value (based on assessment that no penetration is required)• Contractor develops and implements verification plan
Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve issues• Review verification plan, its implementation, and selected verification closure data
In-depth Penetration • Includes intermediate penetration with addition of:
• Methodical review of details• Independent models to check and compare vendor data, as required
• Review verification plan, implementation, and concur in all verification closure data
Total Penetration • Perform a complete and independent evaluation of each task• Independent review of all verification documentation (including closure data) and witness verification testing
Technical Penetration Levels (by discipline areas)
Low Penetration • Participate in reviews and Technical Interchange Meetings and assess only the data presented• Perform periodic audits on pre-defined process(es)• Chair board or serve as board member, or RID writer, at a formal review• Participate in resolution and closure of issues• Review verification plan and its implementation
Level 0
Level 1
Level 2
Level 3
Level 4
Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve evolving issues• Institute specific, regularly planned status and open action/issues reviews
In-depth Penetration • Includes intermediate penetration with addition of methodical review of details• Government Senior Business Managers may participate in reviews
Business Systems
Penetration Levels
Low Penetration • Participate in or perform periodic audits on pre-defined process(es) and/or systems• Participate in resolution and closure of issues• Hold periodic status and open actions/issues reviews
Level 0
Level 1
Level 2
Level 3
Level 4
Total Penetration • Perform a complete and independent evaluation of each task• Independent review of all Contractor final contract documentation • Regularly scheduled meetings of Senior Business Managers (Contractor and Government) to review current/anticipated issues and overall status
No Penetration• Accept performance at face value (assess that no penetration is required)
Technical Surveillance Plan
AREA OF PENETRATION EN
GIN
EE
ER
ING
SE
RV
ICE
Ele
ctr
ical P
ow
er
Instr
um
enta
tio
n a
nd
Co
ntr
ol
Co
mp
ute
rs a
nd
Data
Syste
ms
Flig
ht S
oft
ware
Avio
nic
Syste
ms
EE
E P
art
s a
nd
Packag
ing
Co
ntr
ol E
lectr
onic
s
Rad
io F
req
ue
ncy
Avio
nic
s S
imula
tio
n
Str
uctu
ral and
Dynam
ic L
oad
s
Str
eng
th A
naly
sis
Str
uctu
ral D
esig
n
GS
E a
nd
Me
chanis
ms D
esig
n
The
rmo
dynam
ics a
nd
He
at T
ransfe
r
The
rmal and
Flu
id S
yste
ms
Str
uctu
ral and
Dynam
ics T
esting
Enviro
nm
enta
l E
ffe
cts
No
n-D
estr
uctive
Eval &
Trib
olo
gy
Me
tallic
Mate
rials
and
Pro
ce
sse
s
No
n-M
eta
llic
Mtls &
Pro
ce
sse
s
Pro
ject E
ng
ine
ering
Che
mis
try
Manufa
ctu
ring
Se
rvic
es
Sp
ecia
l T
est E
quip
me
nt D
esig
n )
Sys E
ng
r S
up
po
rt
Co
nfig
ura
tio
n a
nd
Data
Mg
mt
Enviro
nm
ents
Structural Test Rqmts (Proof Loads) 1 1 1 2 1Peroxide Issues (Comp,Clean,Venting) 2 2MPS/RCS Propellant TransferC/SiC Performance 2 2 2 1SIGI 2 2Retractable Solar Array 2 2 2WLE Surface Temperatures 2Li Ion Batteries 3CADS 2Software V & V 3Communication System 2High Temperature Bearings 2 1Laser Initiated Pyrotechnic FTS 1Composite Materials 2Systems Engineering/Integration Issues 1 1 1 1AL 5254 Tank 2 2Component Qualification Rqmts 2 1 1 1 1Structural Verificaton & Testing 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1Environmental Assessment/Effects 1 1 2Star Tracker 1Actuator Control Electronics 2Computers/Data Systems 2Level 0 - No Penetration Level 1 - Low Level Penetration - Participate in reviews and TIMs, assess only the data presented, perform periodic audits Level 2 - Intermediate Level of Penetration - Level 1 plus daily or weekly involvement to identify and resolve issues Level 3 - In-depth Level of Penetration - perform independent assess. and run independent models to check and compare vendor dataLevel 4 - Total Penetration - perform a complete and independent evaluation of each task
X-37 Engineering Directorate
Insight Matrix(ED Interpretation Plus Additional
Recommended Areas)
Business Surveillance Plan
Estimating System
Accounting SystemProperty Mgmt. SystemMajor Sub-K Approval
Periodic Floor Audits
Proposal development and timely negotiation
Procurement System
Socio-Economic Performance
Labor Relations
Special Contract Schedules (LOE or T&M)
Performance Measurement Reviews or Audits
Corporate Financial Health
Performance of audits delegated to DCMA
Performance of audits delegated to DCMA
Performance of audits delegated to DCMA
Performance of audits delegated to DCMA
Performance of audits delegated to DCMA
NASA review & consent on all Sub-K action over $1M
Union contract negotiations scheduled for this summer.NASA Labor Relations Specialist involved.
Fortune 500 Company. No formal surveillance required.
Excellent performance to date. Reviewed every 6 Months
Monthly review with Senior Managers on status of all Proposal Development activity and negotiation status
Performance of audits delegated to DCMA
Not applicable
Surveillance ApproachSystem/Area