quantitative prediction and improvement of program … · 2017. 5. 18. · predictive...

Quantitative Prediction and Improvement of Program

Execution – A New Paradigm

Dr. Shawn Rahmani Boeing Defense, Space/Security

Huntington Beach, California

NDIA 16th Annual SE Conference October, 2013

See attached Boeing’s Assignment of Copyright. The work of authorship referred to herein is a work-made-for hire by one or more employees of The Boeing Company

Agenda

• Problem

• Approach

• Process

• Examples Products

• Recommended Practices and Lessons Learned

• Summary

2 Copyright © 2013 Boeing. All rights reserved.

3

Problem

Col Dave Madden: “. . SW metrics are too rear view mirror - want projections into the future that we can measure ourselves against . . .”


4

$1 M 24 months Based on X ESLOC

Problem (Software Example)

Proposal Estimation Activities

Program Planning

Award ATP

$0.9 M (allocated) 22 months (Based on Y ESLOC??)

Value of the Proposal BOE is ignored/discarded after the Program is awarded

Negotiation/ Budget/Schedule Updates

ESLOC Prod

Cost Cost ESLOC Prod

Copyright © 2013 Boeing. All rights reserved.

Approach: Integrated Model-based PM

Project Characteristics,

WBS, Size (I)

Total Cost and Schedule

(O)

Execution Plan, Cost, Staffing & Work Product

Profiles

(I/O)

Risk Reduction Defects,

Containment (I/O)

Technical Performance-

Cost, Schedule, EVM Data

(I/O) Model

Traditional: Estimating

Integrated Model-based PM Approach


Control

Measure

Predict

Proposal Execution

Estimate /Plan

0

100

200

300

400

500

600

700

800

900

1000

1100

1-M

ar

1-A

pr

1-M

ay

1-J

un

1-J

ul

1-A

ug

1-S

ep

1-O

ct

1-N

ov

1-D

ec

1-J

an

1-F

eb

1-M

ar

1-A

pr

1-M

ay

1-J

un

1-J

ul

1-A

ug

1-S

ep

1-O

ct

1-N

ov

1-D

ec

1-J

an

1-F

eb

1-M

ar

1-A

pr

1-M

ay

1-J

un

1-J

ul

1-A

ug

1-S

ep

1-O

ct

1-N

ov

1-D

ec

1-J

an

1-F

eb

1-M

ar

1-A

pr

1-M

ay

1-J

un

1-J

ul

1-A

ug

BTU 26c SW Tester

BTU 26b SW Coder

BTU 26a SW Designer

BTU 25a SW Requirement Analyst

BTU 25b SW Planner/Management

BTU 24 SW Configuration Management

Estimate /Plan

Detailed Staffing by BTU*

Technical/EAC Predictions

WP** Execution Profiles

What If Analyses

-50% 0% 50% 100%

10. Memory Constraints

9. Tar Sys Exper/Compl

8. Development System Volatility

7. Real Time Code

6. Special Display Requirements

5. Quality Assurance Level

4. Test Level

3. Target System Volatility

2. Requirements Volatility (Change)

1. Specification Level - Reliability

Rank Aerial Refueling Control Computer (ARCC) CSCI build 1

Identified Risks

Model

Metrics collection Periodic model-based performance prediction, risk assessment, and recommended corrections

Model-based estimate & plan and project validation Integrated SW risk mitigation plan, measure and control

A close-loop control, full lifecycle approach, for model-based, software project management

Repeat or

Re-plan

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2/1

2/2

01

2

3/1

2/2

01

2

4/1

2/2

01

2

5/1

2/2

01

2

6/1

2/2

01

2

7/1

2/2

01

2

8/1

2/2

01

2

9/1

2/2

01

2

10

/12

/20

12

11

/12

/20

12

12

/12

/20

12

1/1

2/2

01

3

2/1

2/2

01

3

3/1

2/2

01

3

4/1

2/2

01

3

5/1

2/2

01

3

6/1

2/2

01

3

7/1

2/2

01

3

8/1

2/2

01

3

9/1

2/2

01

3

10

/12

/20

13

11

/12

/20

13

12

/12

/20

13

1/1

2/2

01

4

2/1

2/2

01

4

3/1

2/2

01

4

4/1

2/2

01

4

5/1

2/2

01

4

6/1

2/2

01

4

7/1

2/2

01

4

8/1

2/2

01

4

SWRR SRR PDR CDR C&UT CI&TFQT SI&T

SW Reqts (SRS)

SW Test (STP, STD, Scripts, STR)

SW Planning

SW CM

SW Code & UT (EKSLOC)

SW Design (SDD)

Track Progress by WP** Complete and

BTU* Charging

Cost Volume/Basis of Estimate (BOE)

6

Process

*BTU - Basic Task Unit, standard task based cost accounting structure **WP – Work Product Copyright © 2013 Boeing. All rights reserved.

Advanced SW Estimating Toolset

(ASET)

• Master Input/Parse • Calibration Tool • CER Catalog • MS Project Templates

Traditional Inputs • Size • Environmental

Characteristics* • Historical Data**

Advanced Inputs

• Feature Roadmap (agile)

• Team structure

Inputs

Traditional Outputs • Effort (hours) • Schedule (calendar months)

Advanced Outputs

• Staffing Profiles • Work Product (Feature)

Plans • MS Project Plans • EAC Projections • Defect Predictions

Outputs

* Environmental Characteristics include personnel, development / target / integration environments, and program constraints ** Historical Data includes size, effort, schedule, and productivity and Cost Estimating Relationships (CERs)

Improvements

7

Before Current


INPUT Iterate

INPUT As needed

INPUT Periodic

2

PSRx Master SLOC Input Sheet

SEER-SEM & PPMC

PSRx

Define Project Model - Type - Parameters - WBS (CSCIs, CSCs …) - Sizes (builds)

Project Estimate(s) - Cost - Schedule - Defects

Proposal Execution

Plan & Measure Project - Metrics / Measurements - Planned & Actuals

Project Health Assessment

Reports

Analyze “What-if’s” to assess any

Corrective Actions

Adjust Model Parameters … Rebaseline(s)

Project Estimate - Cost - Schedule ( - Defects )

Tools

Main Activities

Reports etc

Add’l Tools

SEER-SEM & PPMC

Analyze “What-if’s”

to assess Estimate

Adjust Model Parameters

Tool Master SLOC Input Sheet

Model

Tool

Define Project Model - Type - Parameters - WBS (CSCIs, CSCs …) - Sizes (builds)

Project Estimate(s) - Cost - Schedule - Defects

Proposal Execution

Plan & Measure Project - Metrics / Measurements - Planned & Actuals

Project Health Assessment

Reports

Analyze “What-if’s” to assess any

Corrective Actions

Adjust Model Parameters … Rebaseline(s)

Project Estimate - Cost - Schedule ( - Defects )

Tools

Main Activities

Reports etc

Add’l Tools

Model

Analyze “What-if’s”

to assess Estimate

Adjust Model Parameters

INPUT Iterate

INPUT As needed

INPUT Periodic

1

Process Cycle


Capabilities and Products Actual & Predicted Cost/Schedule

versus Baseline Estimate

SPI & CPI (recent & cum)

Summary Stoplight

Staffing Profiles - Default & Constrained

Technical Metrics/Analysis Including Productivity by Work-Product and Phase

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

3/1

1/2

01

1

4/1

1/2

01

1

5/1

1/2

01

1

6/1

1/2

01

1

7/1

1/2

01

1

8/1

1/2

01

1

9/1

1/2

01

1

10

/11

/20

11

11

/11

/20

11

12

/11

/20

11

1/1

1/2

01

2

2/1

1/2

01

2

3/1

1/2

01

2

4/1

1/2

01

2

5/1

1/2

01

2

6/1

1/2

01

2

7/1

1/2

01

2

8/1

1/2

01

2

9/1

1/2

01

2

10

/11

/20

12

11

/11

/20

12

12

/11

/20

12

1/1

1/2

01

3

2/1

1/2

01

3

3/1

1/2

01

3

4/1

1/2

01

3

5/1

1/2

01

3

6/1

1/2

01

3

7/1

1/2

01

3

8/1

1/2

01

3

9/1

1/2

01

3

10

/11

/20

13

11

/11

/20

13

12

/11

/20

13

1/1

1/2

01

4

2/1

1/2

01

4

3/1

1/2

01

4

4/1

1/2

01

4

5/1

1/2

01

4

6/1

1/2

01

4

7/1

1/2

01

4

8/1

1/2

01

4

9/1

1/2

01

4

SRR SWRR PDR CDR C&UT CI&T FQT S IT&E

Work- Product Execution Profiles

-50% 0% 50% 100%

10. Memory Constraints

9. Tar Sys Exper/Compl

8. Development System Volatility

7. Real Time Code

6. Special Display Requirements

5. Quality Assurance Level

4. Test Level

3. Target System Volatility

2. Requirements Volatility (Change)

1. Specification Level - Reliability

Rank Aerial Refueling Control Computer (ARCC) CSCI build 1

A constrained model to track actuals against plan to assess impact of not meeting plan. An unconstrained model for independent analysis and what if trades

Independent Assessments

Program Driven Assessment

Periodic Actuals %complete hours etc

Program Model

Baseline Plan

Program Model Adjusted Prediction Assessment

Recalibrated Model

Relaxed Constraints

Model

Revised Model

Variations between Shadow and Nominal

Models

etc etc Models

Adjusted Parameters

Model

Replica Recommendations etc

Program Model Setup Reflects Program Characteristics

Off-Plan Trigger

Shadow Model

Nominal Model

Model Settings parameters calibration etc

Imposed Constraints milestones staffing-profile etc

Model

Risk Identification and Analysis

Proposal Support, What-if Scenario &

Independent Assessment

0

5

10

15

20

25

Mar

ch 1

, 20

11

Ap

ril 1

, 20

11

May

1, 2

01

1

Jun

e 1

, 20

11

July

1, 2

01

1

Au

gust

1, 2

01

1

Sep

tem

be

r 1

, 20

11

Oct

ob

er

1, 2

01

1

No

vem

be

r 1

, 20

11

De

cem

be

r 1

, 20

11

Jan

uar

y 1

, 20

12

Feb

ruar

y 1

, 20

12

Mar

ch 1

, 20

12

Ap

ril 1

, 20

12

May

1, 2

01

2

Jun

e 1

, 20

12

July

1, 2

01

2

Au

gust

1, 2

01

2

Sep

tem

be

r 1

, 20

12

Oct

ob

er

1, 2

01

2

No

vem

be

r 1

, 20

12

De

cem

be

r 1

, 20

12

Jan

uar

y 1

, 20

13

Feb

ruar

y 1

, 20

13

Mar

ch 1

, 20

13

Ap

ril 1

, 20

13

May

1, 2

01

3

Jun

e 1

, 20

13

July

1, 2

01

3

Au

gust

1, 2

01

3

Sep

tem

be

r 1

, 20

13

Oct

ob

er

1, 2

01

3

No

vem

be

r 1

, 20

13

De

cem

be

r 1

, 20

13

Jan

uar

y 1

, 20

14

Feb

ruar

y 1

, 20

14

Mar

ch 1

, 20

14

Ap

ril 1

, 20

14

May

1, 2

01

4

Jun

e 1

, 20

14

July

1, 2

01

4

Au

gust

1, 2

01

4

Sep

tem

be

r 1

, 20

14

BTU 26c SW Tester

BTU 26b SW Coder

BTU 26a SW Designer

BTU 25a SW Req

BTU 25b SW Planner/ManagementBTU 24 SW CM

Staffing ProfileDetailed Plans

Actual Defects versus Baseline Estimate

1 2 4

8

6

5 7

3

9


Sample Product

10 10

0

5

10

15

20

25

Mar

ch 1

, 20

11

Ap

ril 1

, 20

11

May

1, 2

01

1

Jun

e 1

, 20

11

July

1, 2

01

1

Au

gust

1, 2

01

1

Sep

tem

be

r 1

, 20

11

Oct

ob

er

1, 2

01

1

No

vem

be

r 1

, 20

11

De

cem

be

r 1

, 20

11

Jan

uar

y 1

, 20

12

Feb

ruar

y 1

, 20

12

Mar

ch 1

, 20

12

Ap

ril 1

, 20

12

May

1, 2

01

2

Jun

e 1

, 20

12

July

1, 2

01

2

Au

gust

1, 2

01

2

Sep

tem

be

r 1

, 20

12

Oct

ob

er

1, 2

01

2

No

vem

be

r 1

, 20

12

De

cem

be

r 1

, 20

12

Jan

uar

y 1

, 20

13

Feb

ruar

y 1

, 20

13

Mar

ch 1

, 20

13

Ap

ril 1

, 20

13

May

1, 2

01

3

Jun

e 1

, 20

13

July

1, 2

01

3

Au

gust

1, 2

01

3

Sep

tem

be

r 1

, 20

13

Oct

ob

er

1, 2

01

3

No

vem

be

r 1

, 20

13

De

cem

be

r 1

, 20

13

Jan

uar

y 1

, 20

14

Feb

ruar

y 1

, 20

14

Mar

ch 1

, 20

14

Ap

ril 1

, 20

14

May

1, 2

01

4

Jun

e 1

, 20

14

July

1, 2

01

4

Au

gust

1, 2

01

4

Sep

tem

be

r 1

, 20

14

BTU 26c SW Tester

BTU 26b SW Coder

BTU 26a SW Designer

BTU 25a SW Req

BTU 25b SW Planner/ManagementBTU 24 SW CM

Staffing Profile


BOEING PROPRIETARY

0

19

38

57

76

95

114

9-12 1-13 5-13 9-13 1-14 5-14 9-14 1-15 5-15 9-15

Months

% of Baseline Plan

Baseline EAC: 61155 Hours; 39.65 Months

Earned ValueActual SpentBaseline Plan

Forecast End Date: 1/12/2016Forecasting Method: Cum Performance

Time Now: 3/01/2014

Baseline Plan

Large Program - ADHRAs of 3/01/2014 -

Time Now

11

Execution Phase – Prediction Example

Case: Slow progress • Observations

• Reduced performance (green solid) • $ burn rate continues per plan (red

solid) • Projections (EAC)

• Late to 100% done (green dotted) • Over budget (red dotted)

Projected

EAC

Based on actual performance data the model predicts cost overrun

and late delivery

‘What if’ analysis can help determine path back to within budget Staff level

Personnel Characteristics

Development Environment

Target Environment


Recommended Practice/ Lessons Learned -1

12

• Process used in proposal phase creating

representative model of project and plans

applicable throughout lifecycle

• Lesson: retrofitting during execution is

cumbersome and costly

• Metrics Plans – Ensure proper information is

available

• Lesson: metric plans may not support

desired performance management



13

• Process should analyze metrics and cost data

ensuring:

• Cost charging alignment to WBS and use of

BTUs are correct and support EVM analysis

• Metrics are as planned and provide accurate

assessment of program progress

• Accurate and efficient historical data

collection

• Lesson: actual metric plan and cost

charging vary from planned approach and

may not provide needed information Copyright © 2013 Boeing. All rights reserved.


14

Process automation improves time and effort

to produce plans, collect/analyze EVM, and

prepare proposals

Core SMEs support estimating and proposal

development and mentoring of project

personnel

Lesson: planning and monitoring is a time

consuming activity for management

resources, especially when doing re-plans

during execution


15

Summary

Demonstrated Program Executability Improvements

• Integrated model-based project estimating, planning,

predicting, tracking and management

• Carries value of proposal BOE through program

execution

• Capability applied to software projects, expandable to

SE

• Best if used in proposal phase and continued during

contract execution

• Value proposition with metrics collection and process

automation


Backup Information

Abstract

Quantitative Prediction and Improvement of Program Execution – A New Paradigm By: Dr. Shawn Rahmani Boeing Defense, Space/Security Huntington Beach, California Flawless execution of programs is crucial for the US Government and its contractors. This presentation focuses on a “predictive methodology” that uses program data to provide leading indicators about the health of the program execution throughout the development cycle. While the methodology can be applied to an entire program, the initial focus has been on the “software” portion of a program (developed internally or by a supplier). Software implements complex system functionalities such as network operations and communications, system automation, control and autonomy, information assurance, data manipulation and management, and user interface capabilities. A number of DOD programs have experienced software related problems, including inaccurate and incorrect estimation of the software effort, lack of adequate and quantifiable definition of the planned work, and insufficient visibility and inaccurate measurement of the work performed. While methods based on Earned Value Measurement (EVM) have been in place for some time, they have proved to be insufficient for addressing the above problem. Specifically, the following contribute to the problem: Lack of integrated leading indicators (based on the product size, cost, schedule, quality, and staffing requirements) to predict technical success of the software activities within a program. Lack of value-added measures that provide predictive program insight while supporting lean improvements. Lack of ability to effectively package and communicate the predicted program impact and recommended solutions to decision makers across the life cycle. This presentation is based on the work done at Boeing to address the above issues. It covers: A set of predictive metrics for software engineering, consistent with Government’s predictive metrics for Probability of Program Success (PoPS) and Predictive System/Software Measurements (PSM). A set of raw software measurements that can be used for estimation and measurement of detailed work planned and performed on government contracts. A technical process that defines how software estimation, measurement, prediction, and control should be handled. Extensibility of current metrics (e.g., EVM, quality, risk). Information on how the process contributes to First Time Quality and Lean Program Management/Execution. Information about the tool that automates the collection of the data and implementation of the process. Use of the process and tool on trial projects and pilot programs Lessons learned and future plan.


ASSIGNMENT OF COPYRIGHT Date of Presentation(s): October 14, 2013 Copyright Form To: National Defense Industrial Association (NDIA) 16th Annual Systems Engineering Conference Publication Title: Quantitative Prediction and Improvement of Program Execution – A New Paradigm Author(s): Shawn Rahmani At least one, but possibly not all, of the authors of the Work is an employee of The Boeing Company or one of its wholly-owned subsidiaries (“BOEING”) preparing the Work within the scope of his or her employment. The Boeing employees in question are not authorized to assign or license Boeing’s rights therein, or otherwise bind Boeing. However, subject to the limitations set forth below, Boeing is willing to assign its copyright in the Work to Publisher. Notwithstanding any assignment or transfer to the Publisher, or any other terms of this agreement, the rights granted by Boeing to Publisher are limited as follows: (i) any rights granted by Boeing to the Publisher are limited to the work-made-for-hire rights Boeing enjoys in the Work; (ii) Boeing makes no representation or warranty of any kind to the Publisher or any other person or entity regarding the Work, the information contained therein, or any related copyright; and (iii) Boeing retains a non-exclusive, perpetual, worldwide, royalty-free right, without restriction or limitation, to use, reproduce, publicly distribute, display, and perform and make derivative works from the Work, and to permit others to do so. Candice Jordan Contracts Specialist Intellectual Property Management

quantitative prediction and improvement of program … · 2017. 5. 18. · predictive...

Documents