Lessons Learned Implementing EVM in the EVA Office

Les Sorge/SGT, Inc.

Kristen C. Kehrer/NASA KSC

Project Management Challenge 2011

Long Beach, CA

February 9-10, 2011

Used with permission

2

Presentation Agenda

Background Lessons Learned

Management Reserve Budget versus Funding Risk Management Impact to EAC & MR Work Authorization Traceability & Scope Identical Structures and Titles for WBS and IMS WBS/Charge Numbers and EVM Control Accounts/Work Packages Institutional or Project OBS Data Integration with Multiple Tools Integration of Prime Contractor Data (1 control account or multiple) Setting and Validating the PMB

Questions & Answers Acronyms and Reference Documents

3

Background

Constellation (CxP) EVA Systems Project Office (ESPO) was selected as the first of two pilot projects for the NASA Earned Value Management (EVM) Capability Project The goal of the pilot project was to capitalize off of the EVM implementation efforts in

EVA to develop an Agency EVM capability, test the Agency EVM capability processes and tools for integrated in-house and contractor EVM, and to incorporate identified changes and lessons learned into the documentation for Agency wide streamlined EVM processes and training

The EVMS Capability team identified 33 gaps and 6 other areas of concern during the initial gap analysis of the ESPO pilot project in March, 2010

Closure plans were developed and documented with a plan to track and coordinate them to ensure proper vetting across affected organizations

The closure plans were to be implemented and tested through monthly reporting cycles – did the fix work?

The pilot project was prematurely terminated in June, 2010 because of changes/uncertainty within the Constellation Program and the ESPO

Lessons learned from the pilot project were documented as part of the EVM Capability Project Two issue papers were written and presented to the EVM Capability Project Steering Committee for

consideration Inconsistency Between Plan and Performance Business Rules and Actual Costs

4

Lessons Learned

5

Management Reserve

Driving Event CxP provided direction for no Management Reserve (MR) for the projects Misunderstanding of financial reserves (Unallocated Future Expense – UFE) vs budget reserves

(Management Reserve - MR) Lesson Learned – Allow projects to budget for and report MR to manage project

risks and in-scope changes and use the ANSI/EIA-748 standard term of MR

Graphic modified from Chambers/Gibby 07/2010

UFE

AgencyBaseline

Commitment

Management Baseline -

PROGRAM

CONGRESS & OMB

AUTHORIZED

UFE

AUTHORIZEDDistributed

Budget

UndistributedBudget

ManagementReserve

AUW

PMB

NASA EVM ProjectProgram Office

ProjectBudgetBase

UFE

Key: Held Back Funds/Funding Commitment Agreement Held Back Funds/Funding Management Agreement Not Strictly “Budget”, but should be inviolate Budget

Funding Process Budget Process

6

0

20

40

60

80

100

120

140

1Qtr

2QTR

3QTR

4QTR

1Qtr

2QTR

3QTR

4QTR

1Qtr

2QTR

3QTR

4QTR

1Qtr

2QTR

3QTR

4QTR

1Qtr

2QTR

3QTR

4QTR

GFY 2011 GFY 2012 GFY 2013 GFY 2014 GFY 2015

Budget versus Funding

Driving Event Misunderstanding the difference between budget and funding for EVM by all levels of

management No distinction between EVM budget and funding driving replans of the EVM Performance

Measurement Baseline (PMB) when funding changes occur EVM not fully integrated with current NASA financial processes and software tools

Lesson Learned – To support EVM, NASA culture must differentiate between budget and funding and not replan the PMB (eliminate variances and trends) for every funding change. The validity and accuracy of the performance measurement and forecast data rely on a stable PMB

6

Management Baseline - PROGRAM

UFE

AUTHORIZED

NASA EVM Project

Program Office

Annual PPBE (Funding Process)

EAC

The BCWS & BAC represent the work

The ACWP, ETC, & EAC represent the funds (i.e. money) required for that

work

Time Now

The BCWP represents the work accomplished

BAC

Initial Project Budget Base(BAC with MR = EAC)

Funding Plan Actuals/ETC/EAC

Funding LimitsObligations

VAC

DistributedBudget

UndistributedBudget

ManagementReserve

AUW

The funding limit (obligation) represents the maximum funds

available to accomplish the work

VAC > 15% Notification to Program / Mission Directorate

Define Project Budget (Budget Process)

7

Risk Management Impact to EAC & MR

Driving Event Project risks captured in Integrated Risk Management Assessment (IRMA) tool but no

process defined to utilize risks to develop initial project MR No process defined to include risks in an EAC update or how it relates to uncertainty

Lesson Learned – Projects should develop processes to tie risk assessments to the development and justification of MR and EAC updates

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2400 2450 2500 2550 2600

Pro

ba

bil

ity

70% Probability

70% Estimate

0.0000000

0.0050000

0.0100000

0.0150000

0.0200000

0.0250000

0.0300000

0.0350000

2250 2300 2350 2400 2450 2500 2550 2600

Probability Density Function Cumulative Distribution Function (S-Curve)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2400 2450 2500 2550 2600

Pro

ba

bil

ity

70% Probability

70% Estimate

0.0000000

0.0050000

0.0100000

0.0150000

0.0200000

0.0250000

0.0300000

0.0350000

2250 2300 2350 2400 2450 2500 2550 2600

Probability Density Function Cumulative Distribution Function (S-Curve)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2400 2450 2500 2550 2600P

rob

ab

ilit

y

70% Probability

70% Estimate

0.0000000

0.0050000

0.0100000

0.0150000

0.0200000

0.0250000

0.0300000

0.0350000

2250 2300 2350 2400 2450 2500 2550 2600

Probability Density Function Cumulative Distribution Function (S-Curve)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2400 2450 2500 2550 2600P

rob

ab

ilit

y

70% Probability

70% Estimate

0.0000000

0.0050000

0.0100000

0.0150000

0.0200000

0.0250000

0.0300000

0.0350000

2250 2300 2350 2400 2450 2500 2550 2600

Probability Density Function Cumulative Distribution Function (S-Curve)

Cost S-curve

Schedule S-curve

Joint Confidence Level (JCL)10 11 12

Risk No L X C Title Mar Apr MayJun Jul AugSep Oct NovDec Jan Feb Mar Apr MayJun Jul AugSep Oct NovDec Jan Feb

10147 3 x 3 CR 383 - Orion/EVA IRD rev. C

9966 4 x 4 [CSAFE] [EVA Suit] Occupant Protection Risk

= = = = = = >

1968 3 x 2 [EVA SE&I] System mass under-allocation

= = = = = = = = = = >

1575 3 x 3 [EVA Project] Software funding not adequate to meet core block-1 EVA and C3I lunar software requirements

>

1086 2 x 3 [EVA] Suit System IOC procurement reconciliation

= = = = >

5 3

43 8,9

2 5

1 6

1 2 3 4 5

Lik

elih

oo

d

Consequence

Risk Register

Uncertainty (Funding Process)Risk (Budget Process)

MR $ = [Risk Impact $ x Risk Factor] + rule of thumb for unknown risks

Sample Process1. Identify Initial Risks &

Risk Mitigation Plans2. Generate a risk factor

multiplier (0 to 1)3. Develop rule of thumb

for unknown risks4. Define MR $5. Calculate Uncertainty6. Update Risks & Risk

Mitigation Plans7. Update EAC with Risk

Updates8. Update JCL for KDP

gate reviews

Known Risks (known unknowns)

Unknown Risks (unknown unknowns)

???70% JCL = Prob [Cost] × Prob [Schedule | Cost]

Rules of thumb based on historical projects

8

Work Authorization Traceability & Scope

Driving Event The authorization document (PAD) did not authorize all of the ESPO work scope, i.e.

lunar content, even though work was being performed/actuals collected on that work ESPO had to maintain a separate Deltek Cobra project with this scope

Authorizing documents are not kept current with updates Limited scope description to reference for in-scope vs. out-of-scope discussions

Lesson Learned Create authorization document(s) that cover the entire scope of work and use

incremental authorization so all actual costs trace to an authorizing document Maintain all levels of authorizing documents to provide traceability Develop more detailed Statements of Work (SOW) between NASA Programs and

Projects

FAD - Formulation Authorization DocumentPCA - Program Commitment AgreementPAD - Project Authorization DocumentWAD - Work Authorization DocumentITA - Internal Task Agreement

* - Not currently considered an authorization document

NASA AA

MDAA

Program Mgr

Project Mgr

CAM

Center Org

Program FAD

Program PCA

Project FAD

PAD

WAD

JSCITA*

Funding Process Budget/Funding Process

9

Identical Structures and Titles for WBS and IMS

Driving Event The WBS element nomenclature and hierarchy is not consistently used within the

project software toolset Lesson Learned – Use only the project approved WBS numbering scheme

and nomenclature for integration and easy traceability of all project cost, schedule, and technical content If CAMs prefer the schedule outline to match the technical WBS (e.g. without the 6

digit project code and numbering hierarchy), then create a separate data field for the financial WBS

Structure Element Long TitleStructure Element Short Title

Project/WBSWBS Level

Performing Center

Exploration Systems Exploration Systems . .

Constellation Systems Constellation System . .

Constellation Systems Constellation System . .

Extravehicular Activity Extravehicular Activ 731384 .

Project Management Project Management 731384.01 WBS2 01

Systems Engineering Systems Engineering 731384.02 WBS2 01

NASA SE&I NASA SE&I 731384.02.31 WBS3 01

EVA SE&I Mgmt & Admin EVA SE&I Mgmt&Admin 731384.02.31.01 WBS4 01

EVA SEI System Engineering EVA SEI Sys Eng 731384.02.31.02 WBS4 01

EVA SEI Analyses EVA SEI Analyses 731384.02.31.03 WBS4 01

JSC-EVA SE&I EDAC1 Analysis JSC-EVA SEI EDAC1 731384.02.31.03.01 WBS5 72

JSC-EVA SE&I EDAC2 Analysis JSC-EVA SEI EDAC2 731384.02.31.03.02 WBS5 72

JSC-EVA Arch & Analysis IC PDR Tasks JSC-EVA Arch&Analys 731384.02.31.03.03 WBS5 72

EVA V&V IC PDR Tasks EVA V&V IC PDR Tasks 731384.02.31.04 WBS4 01

GRC-EVA PA&S GRC-EVA PA&S 731384.02.31.05 WBS4 22

JSC-CSSS SE&I JSC-CSSS SE&I 731384.02.32 WBS3 72

Deltek Cobra

Primavera

NASA MdM / SAP

10

WBS/Charge Numbers and EVM Control Accounts/Work Packages

Driving Event NASA accounting processes equate a charge number to a WBS number, therefore

new charge numbers essentially “change” the WBS NASA SAP accounting system rule that funding obligations must occur at the same

level the costs are collected, i.e. the charge number De-obligating funds from a charge number is a labor intensive process

Lesson Learned – Project EVM implementations plans need to define how best to work with the existing NASA processes and also meet the intent of general guidelines for WP

WBS Title Project/WBS

Extravehicular Activity 731384Project Management 731384.01

Project Management 731384.01.01

NASA Project Management 731384.01.01.31

JSC-EVA Project Mgmt thru KDP C 731384.01.01.31.01

JSC-External Review Supt thru KDP C 731384.01.01.31.02

GRC-EVA Project Mgmt thru KPD C 731384.01.01.31.03

LaRC-External Review Supt thru KDP C 731384.01.01.31.04

PP - JSC PM PDR to Orion2 731384.01.01.31.a

PP - GRC PM PDR to Orion2 731384.01.01.31.b

PP - JSC PM Orion2 to Orion10 731384.01.01.31.c

PP - GRC PM Orion2 to Orion10 731384.01.01.31.d

PP - JSC PM Orion12 to Orion18 731384.01.01.31.e

PP - GRC PM Orion12 to Orion18 731384.01.01.31.f

JSC-CSSS Project Managment 731384.01.01.32

Project Planning & Control 731384.01.02

NASA PP&C 731384.01.02.31

PP&C Mgmt 731384.01.02.31.01

JSC-PP&C Mgmt thru KDP C 731384.01.02.31.01.01

PP - JSC PPC Mgmt PDR thru CDR 731384.01.02.31.01.a

PP - JSC PPC Mgmt CDR thru Orion2 731384.01.02.31.01.b

PP - JSC PPC Mgmt Orion2 thru Orion18 731384.01.02.31.01.c

PP&C CM/DM 731384.01.02.31.02

Work Packages / Charge Codes

Planning Packages

Control AccountCxP Reporting Level

Control Account

WP Guidelines• WP should be segregated by

elements of cost• WP should have durations of four

months or less• During a replan, a WP should be

closed with s=p to date, with the remaining work planned in a new WP

11

Institutional or Project OBS

Driving Event The Organizational Breakdown Structure (OBS) for the project Responsibility Assignment

Matrix (RAM) was changed from the NASA center institutional organizations to the project organizational structure (i.e. integrated product teams)

NASA operates with two organizational breakdown structures as a system of checks and balances

Lesson Learned – The project needs to define the RAM OBS that best reflects the organizational authority and how work scope is subdivided between the orgs

Questions to ask: Who is ultimately responsible for the work

and has the budget authority? Who does the planning of the

schedule and budget for that scope of work?

To what level is control required or appropriate?

Do the institutional organizations just provide extra bodies or do they provide a “finished product”?

Should other NASA centers be listed as separate organizations and therefore, separate control accounts within the RAM?

Center Director

Directorate

DivisionBranch ManagerStaff

Staff

Staff

Branch ManagerStaff

Staff

Staff

Branch ManagerStaff

Staff

Staff

Institutional (functional) Org Structure

Staff members are grouped by functions or specialties such as Engineering, Safety & Mission Assurance, Public Affairs, etc.

Mission Directorate

Program ManagerProject

ManagerStaf

f

Staff

Staff

Project

ManagerStaf

f

Staff

Staff

Project

ManagerStaf

f

Staff

Staff

Project Coordination

Project Org Structure

Staff members are grouped into departments or product teams

Project Coordination

12

Data Integration with Multiple Tools

Driving Event The prime contractor uses different software tools (MS Project, Deltek MpM) then the

ESPO (Primavera, Deltek Cobra) Lesson Learned – Ideally, all project partners should use the same

software toolset, however, this may not be practical, so request for proposals (RFP) should contain specific language in the Data Requirements Document(s) (DRD) for EVM software tool interfaces The project should develop an integration methodology prior to the RFP to help

define the interface requirements

MS Project

Deltek MpM

DeltekwInsight

DeltekCobra

DeltekwInsight

Primavera

• Standard Code Fields• Unique Task IDs• IMP Codes• WBS• EVM Percent Complete

• File Formats• File Types (.xls, .xml, etc)• Etc.

Interface Requirements

Prime Contractor EVA Systems Project Office

13

Integration of Prime Contractor Data(1 control account or multiple)

Driving Event ESPO defined multiple control accounts (and therefore multiple WBS/charge

numbers) for the prime contractor Each charge number required obligations which sometimes created “cost over obligations”

on a charge number even though the total cost was within the total obligations

Lesson Learned – Projects should develop contractor data integration plans and weigh the pros/cons of the different options

Single Control Account Multiple Control AccountsEasier to Obligate costs for a single WBS/CA in SAP

Easier to integrate and analyze EV data at an IPT level

Easier to integrate Contractor EVM data into the Project data with only one CA

NASA CAMs own their associated Contractor CA and work scope vs a single CAM for the entire contractor

Less maintenance/booking of contractor budgets in project PMB (contractor UB, MR, etc.)

All (NASA and contractor) functional work captured in the same WBS, i.e. all Mgmt in 1.0, etc.

Less impact if the project WBS or contractor WBS changes since no need to keep the them in-sync

Reinforces the “one project team” concept (NASA and prime)

731384.06.04.01Suit Element

.31 Govt CA

.32 Prime CA

WP / PP

WP / PP

731384.06.04.03Vehicle Interface Element

731384.06.04EVA HW Development

.31 Govt CA

.32 Prime CA

WP / PP

WP / PP

Multiple Control Accounts

731384.06.04.01Suit Element

Govt CA

WP / PP

731384.06.04.03Vehicle Interface Element

731384.06.04EVA HW Development

Govt CA

WP / PP

Example of Single Control Account

731384.06.04.05CSSS Contract

(Prime CA)

Pros / Cons

14

Setting and Validating the PMB

Driving Event Prime contractor operated for a year under letter contract without a Performance

Measurement Baseline (PMB) Constant, significant changes to the Constellation Program/Projects funding did not

allow for a stable baseline so as to set a PMB A PMB was developed for an Integrated Baseline Review (IBR), but the announcement to

cancel the Constellation Program and associated reduction in funding changed the focus of the IBR to an FY10 review

Prolonged proposal submittal and authorization process encumbered the change process

Lesson Learned Clearly define expectations and requirements to initialize the PMB and begin EVM

reporting to include letter contracts Utilize existing NASA resources such as the IBR Handbook, the EVM Handbook, and

the Schedule Handbook to define appropriate EVM requirements Projects and contractors should strive for timely contract authorization and

definitization of the base contract and subsequent contract modifications Given the fluid nature of funding, develop “right-sized” products

Schedules and budgets that are not overwhelming to update Use of planning packages and summary level planning packages Utilization of rolling wave planning

15

Questions & Answers

16

Acronyms and Reference Documents Acronyms

AUW – Authorized Unpriced Work CA – Control Account CAM – Control Account Manager CPR – Cost Performance Report CSSS – Constellation Space Suit System CxP – Constellation Program DRD – Data Requirements Document EAC – Estimate at Completion ESPO – EVA Systems Project Office ETC – Estimate to Completion EVA – Extravehicular Activity EVM – Earned Value Management EVMS – Earned Value Management System IBR – Integrated Baseline Review IMS – Integrated Master Schedule IPT – Integrated Product Team IRMA – Integrated Risk Management Application ITA - Internal Task Agreement JCL - Joint Confidence Level JSC – Johnson Space Center MR – Management Reserve OBS – Organizational Breakdown Structure PAD – Project Authorization Document PMB – Performance Measurement Baseline PP – Planning Package PPBE - Planning, Programming, Budgeting, and Execution RAM – Responsibility Assignment Matrix SAP – Systems Application and Products UFE - Unallocated Future Expense WBS – Work Breakdown Structure WP – Work Package

Reference Documents NPR 7120.5D – NASA Program and Project

Management Processes and Requirements CxP 72179 Anx01 – CxP ESPO EVM

Implementation Plan ANSI/EIA-748 – Earned Value Management

Systems