graham gilmer & eric druker - presentation
TRANSCRIPT
Integrated Cost & Schedule Risk Analysis Dynamic Integrated Cost Estimator (DICE) Model
This document contains Booz Allen Hamilton proprietary and confidential information and is intended solely for the use and information of the client to
whom it is addressed. This data shall not be released to other contractors without written consent from Booz Allen Hamilton.
Adelaide, Australia28 June, 2011
Booz Allen Proprietary/Not for Distribution
2
Outline
Introduction
RealTime Analytics™
Overview of Joint Confidence Level Analysis
NASA’s Joint Confidence Level Policy
Dynamic Integrated Cost Estimator (DICE) JCL Analysis Tool
3
Attempt at Humor
4
Whether assessing/analyzing NASA, DoD or Intelligence Community owned projects, the story is the same each time: Programs are increasingly experiencing growth above and beyond their initial cost and schedule estimates
This is not just a cosmetic problem: Cost and schedule growth delays capabilities and constraints the budgets of other programs causing a waterfall of instability
Studies have examined the reasons behind this growth reaching similar conclusions
1. Early program optimism leading to optimistic estimates
2. Insufficient cost and schedule reserves available to cover risk
3. Weak independent validation of cost and schedule
Recognizing this, many guides, including the GAO’s Cost Estimating Handbook, have included Risk Analysis as a required step in best practice cost estimating processes
Introduction
5
RealTime Analytics™
One of the challenges in performing cost and schedule analysis is the time required to run simulations
– Risk analysis, required for all cost estimates by 2009 Weapons System Acquisition Reform Act of 2009 and GAO, requires the use of simulations
– Run-times of minutes or hours prohibit most risk analyses models from being decision making tools as they can not be re-run during meetings
RealTime Analytics™ (RTA) is a collection of technologies, tools and methodologies allowing complex analytics to be performed far faster than using currently available methods
This presentation will focus on the Dynamic Integrated Cost Estimator (DICE) and the methodologies it addresses:
– Joint Confidence Level Analysis (Integrated Cost & Schedule Risk Analysis)
RTA tools allow simulations to run up to 99.99% faster than comparable industry tools
Simulations formerly taking minutes or hours to run now finish in under 1 second
– Allows decision makers to run an unlimited number of excursion scenarios without ever leaving the meeting room
6
Decision makers are starting to recognize that there is rarely a relation between cost risk analysis results and the program’s schedule
– This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth
From the other side, traditional schedule risk analysis typically leads to risk adjusted schedules that, if come to pass, will result in cost growth
Even when both of these analyses are performed on a program, they are typically done by disjoint groups under different sets of assumptions
Joint Cost & Schedule Risk Analysis is an attempt to integrate cost and schedule risk analysis in a way that produces meaningful, compatible results
This presentation will cover:
– NASA’s JCL Policy
– How JCLs are performed
Introduction to Joint Confidence Level Analysis
7
Historically, cost and schedule risk analyses are developed separately and their results are not compatible
– This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth and vice versa
– Joint Confidence Level Analysis combines cost and schedule risk analysis into a single, coherent output
Joint Confidence Level (JCL) analysis1 results in a bivariate distribution of final projected cost and schedule pairs
– Joint Confidence Levels represent the probability of finishing at or under both cost and schedule
JCL Analysis allows a program to:
– Defend budgetary and scheduling decisions
– Prioritize risks and other threats based on their overall impact to the program and not simply their anticipated local impact
– Develop more precise risk mitigation plans
What is Joint Confidence Level Analysis?
1Also known as Integrated Cost & Schedule Risk Analysis or Joint Cost & Schedule Risk Analysis
Cost
Pro
bab
ility
70% Joint Confidence Level Curve
Sch
edu
le
8
What is Joint Confidence Level Analysis?
The primary output from JCL analysis is the JCL scatter-plot
– Each point on the scatter plot represents 1 iteration of a Monte Carlo simulation performed on the JCL model
JCL scatter-plot provides:
– Joint Confidence Levels (e.g. For NASA Budgeting)
– Relationship between cost and schedule (e.g. $ cost growth/days schedule growth)
9/23/10
4/11/11
10/28/11
5/15/12
12/1/12
6/19/13
1/5/14
7/24/14
$603 $803 $1,003 $1,203
Laun
ch D
ate
Total Cost ($M)
Joint Confidence LevelFinal Results
Knee in the Curve Values50% 70%
Cost $787.7M $831.5MFOC 10/8/12 1/29/13
JCL DistrubutionProject Baseline = 61%70% JCL50% JCL
9
JCL Analysis: NASA Policy & Air Force Research
JCL Analysis becoming more common in industry
– In use by oil industry for years
– Already a requirement for NASA projects per NPD 1000.5
– Air Force completing research on JCL Analysis1
Several methods available for integrating cost & schedule; preferred method depends on program phase and available data
Parametric approaches:
– Typically used phase A and before or by oversight groups
– Divided into multiple regression and multivariate regression approaches
Build-up approaches:
– Typically used following phase B by PMOs
– Cost risk analysis and schedule risk analysis performed separately
– Quality checks performed to make sure analyses are compatible
1Joint Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011
10
JCL Analysis: Overview of Methods
1Joint Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011
Cost Risk Analysis:
•Inputs-Based Simulation•Outputs-Based Simulation•Scenario Based•Parametric Analysis
Schedule Risk Analysis:
•Parametric Analysis•Schedule-Based Simulation
Monte Carlo
Simulation
9/23/10
4/11/11
10/28/11
5/15/12
12/1/12
6/19/13
1/5/14
7/24/14
$603 $803 $1,003 $1,203
Laun
ch D
ate
Total Cost ($M)
Joint Confidence LevelFinal Results
Knee in the Curve Values50% 70%
Cost $787.7M $831.5MFOC 10/8/12 1/29/13
JCL DistrubutionProject Baseline = 61%70% JCL50% JCL
Joint Risk Analysis:
•Cost Loaded Schedule-Based Simulation (NASA Policy)•Multivariate Regression
9/23/10
4/11/11
10/28/11
5/15/12
12/1/12
6/19/13
1/5/14
7/24/14
$603 $803 $1,003 $1,203
Laun
ch D
ate
Total Cost ($M)
Joint Confidence LevelFinal Results
Knee in the Curve Values50% 70%
Cost $787.7M $831.5MFOC 10/8/12 1/29/13
JCL DistrubutionProject Baseline = 61%70% JCL50% JCL
Joint Risk Analysis methods estimate cost and schedule simultaneously; JCL scatter plot produced directly from analysis
When Joint Risk Analysis methods are not used, cost and schedule risk analyses must be combined using a Monte Carlo simulation
When risk analyses are combined, care must be taken to ensure results are compatible
11
JCL analysis can be performed using existing resources; likely risk management, cost estimating, and scheduling personnel
The artifacts needed to perform a JCL analysis are:
– A program schedule (IMS or analysis schedule) with uncertainty bounds on task durations
– A quantified risk register (probabilities, cost and schedule impacts) where each risk is mapped to a task in the IMS
– A cost estimate with uncertainty bounds that maps to the schedule
Creation of these artifacts requires communication between program’s cost estimating, scheduling and risk management staff
Joint Confidence Level Analysis Process at NASA
Integrated Risk Assessment
Program Risk
Register
Integrated Master
Schedule
Cost Estimate
12
Joint Confidence Level Analysis has gained significant momentum recently
– NASA is leading the way in the development of this methodology
– NASA Policy Directive 1000.5 mandates that programs will be baselined at the “70 percent confidence level” using a “joint cost and schedule probability distribution”2
– NPD 1000.5 also stipulates that projects are funded at no less than 50% of the JCL or as approved by the decision authority, maintaining JCLs through the program lifecycle
– The goal is to provide stronger assurance that NASA can meet cost and schedule targets3
– A recent GAO report cites NASA’s JCL policy as an effort “to provide transparency on the effects of funding changes on the probability of meeting cost and schedule commitments” 4
– NASA Cost Analysis Division (CAD) has developed a handbook to provide more information and guidance on this topic
– Programs conducting JCL Analysis include James Webb Space Telescope and SOFIA
While the methodology has made substantial strides, the cost and schedule communities must overcome political and technical obstacles before full adoption
NASA Joint Confidence Level Analysis Policy
2 – NPD 1000.5 - http://www.hq.nasa.gov/office/codeq/doctree/10005.htm - January 15, 20093 – JCL Status Report - http://www.nasa.gov/pdf/421542main_JCL%20Status%20Report-2010%20Feb.pdf – February 20104 – GAO Report – “NASA – Assessments of Selected Large-Scale Projects” - http://www.gao.gov/new.items/d11239sp.pdf - March 2011
13
NASA JCL Model Prototype: DICE
In Fall 2010, NASA CAD commissioned the development of a JCL model prototype
The intention of this effort was to explore the value of producing a standard toolset for NASA programs conducting JCL analysis
Booz Allen created the Dynamic Integrated Cost Estimator (DICE) with a focus on streamlining the JCL process and decreasing simulation runtimes
Other key features of the DICE prototype development included:
– Rapid schedule import from MS Project
– Cost-Loading
– Discrete Risk Analysis
– JCL Scatter Plots and Iso-Curves
– Benchmarking effort with other JCL tools
It is important to note that there are many tools that projects can use to develop JCLs, but DICE is optimized for this analysis
14
DICE facilitates Joint Confidence Level Analysis
DICE is an Adobe Flex-based tool for cutting-edge cost and schedule risk analysis
– Includes modeling capability for producing build-up Joint Confidence Levels (JCLs)
– Achieves industry-leading runtimes using Booz Allen’s RealTime Analytics
15
DICE – Benchmarking against Primavera Risk Analysis
Runtimes (3200-line schedule) S-Curves (at 0.4 correlation)
Correlation Key Points
– Identical input parameters to ensure consistency in benchmarking
– Importing risks from 3rd party template
– Outputs <1% variation from Primavera
Task Run Time DICE Primavera Opening Tool 0:02 0:05 Import Schedule 0:03 2:05:27 Load Risk File 0:03 0:01 Initialize Correlation 0:02 0:05 Run Simulation 0:09 2:27 Export Data 0:08 0:00
Schedule Cost
Total Project Cost
Project End Date
Project End Date
16
DICE Demo
17
Conclusion
Booz Allen’s recent innovations in simulation technology enable analysts to support decision making in near real-time
– Decision makers can see the impact of changes to their program, real-time, without ever leaving the meeting room
– There is no longer a limit on the number of excursions that can be run on an analysis
New Joint Confidence Level methods changes the way programs look at cost and schedule risk
– Analyses are no longer performed and viewed separately, but rather are integrated and optimized using a standard tool
– Decision makers have more insight into their program than ever before
– Opening of communication lines between cost, schedule and risk management staff results in better program management
18
Points of Contact
Booz | Allen | Hamilton
Colin SmithAssociate
Booz Allen Hamilton Inc.Suite 2100
230 Peachtree Street NWAtlanta, GA 30303
Tel (404) [email protected]
Booz | Allen | Hamilton
Booz | Allen | Hamilton
Brandon HerzogConsultant
Booz Allen Hamilton Inc.1530 Wilson Blvd., 10th Floor
Arlington, VA 22209Tel (703) 526-6040
Graham GilmerAssociate
Booz Allen Hamilton Inc.1530 Wilson Blvd., 10th Floor
Arlington, VA 22209Tel (703) 526-2413
Booz | Allen | Hamilton
Eric DrukerAssociate
Booz Allen Hamilton Inc.St. Louis, MO
Tel (314) [email protected]
19
DICE Functionality – Gantt Chart and Cost/Schedule Uncertainty
Organizes project tasks, costs, constraints, schedule interrelationships, and adds uncertainty to individual cost and schedule items
20
DICE Functionality – Cost-Loading
DICE can load schedules with time-dependent and time-independent costs, generating standard outputs such as JCL scatter plots and iso-curves
21
DICE Functionality – Fiscal Year Segmenting
DICE accounts for costs (and uncertainty) by fiscal year – aids budget planning
22
DICE Functionality – Run-time Features
DICE incorporates blanket correlation across the model, or specifies individual correlation between schedule tasks, time-independent costs, or risks
The model is optimized for Joint Confidence Level Analysis
– Greatly increases understanding of how schedule growth impacts cost
– Contains robust schedule logic functionality and discrete risk integration
DICE includes RealTime Analytics to enable industry-leading simulation runtimes
– Reduces the time required to evaluate decisions and provides decision makers with on-the-spot analysis
– Booz Allen’s RTA allows for quick initial runtimes and near-immediate re-runs
Interactive features provide intuitive user experience and rapid evaluation of alternatives
– Enables comparison of multiple different scenarios or confidence levels of the same project
– Builds off of existing tools (like MS Project, Excel) for seamless data integration, cost visualization, and navigation of risk analysis
23
RealTime Analytics: Excel Tool
24
RealTime Analytics Excel Tool™Introduction
RealTime Analytics™ Excel Tool
– Excel add-in enabling fast running of simulations
– Similar in capabilities to Crystal Ball and @Risk
Benefits:
– Runtime savings of >99% vs. COTS tools
– Analysis can be run without ever leaving the meeting room
– Decreased simulation runtimes allows running of unlimited number of excursion scenarios
– Not a numerical approximation approach such as Method of Moments, it simply runs simulations faster than existing tools
Uses:
– Cost estimating/risk analysis/risk management
– Insurance pricing/actuarial models
– Portfolio optimization/trade-off analysis
– Cash flow/profit analysis
25
RealTime Analytics Excel Tool:Benchmarking vs. Crystal Ball & @Risk
Comparison of Simulation Results
File Size265 MB
15 MB
Simulations include mix of triangular, normal, beta, lognormal and uniform distributions with same parameters in each model
444 correlated distributions with 9,620 forecasted values
Baseline scenario is the time to “prime” the models; includes adding in correlation
Each additional scenario is time to re-run model when a single distribution parameter is changed
Crystal Ball RTA
Simulation Run Times
Assumptions
0%10%20%30%40%50%60%70%80%90%
100%
$25,000 $35,000 $45,000 $55,000 $65,000 $75,000
Cum
ulati
ve P
roba
bilit
y
Cost
Results
CB RTA '@ Risk
<1 Second
4 Min
12 Min
12 Min
10 Min
6 Hrs 10 Min
7 Hrs 48 Min
13 Hrs 48 Min
Each "Excursion Scenario"
Baseline Scenario
Crystal Ball (Normal Mode) Crystal Ball (Extreme Mode) @Risk Real Time Analytics
~ ~
~