risk management using risk+ (v5)
DESCRIPTION
Employing Monte Carlo Simulation for the assessment of schedule and cost risk in the Integrated Master Schedule (IMS).TRANSCRIPT
INCREASING THE PROBABILITY OF PROGRAM SUCCESS USING RISK+
A workshop on the principles and practices of Risk+ and increasing the Probability of Program Success
1
Glen B. Alleman
Niwot Ridge LLC
A Warning We’re going to cover a lot of material in 3 hours
2
Risk involves uncertainty. Uncertainty involves probability.
3
Douglas Adams, Hitchhiker's Guide to the Galaxy 4
MOTIVATION?
Your motivation?
Your motivation is your pay packet on Friday.
Now get on with it.
– Noel Coward, English actor, dramatist, &
songwriter (1899 – 1973)
5
We have to know the underlying statistical behavior of the
processes driving the project
This means cost, schedule, and technical performance
measures with probabilistic models
We need to know how these three statistical drivers are
coupled
What drives what?
What are the multipliers between each random
variable? 6
Let’s start with the basics 7
Remember High School Statistics 8
The IMS is a collection of probabilistic processes all coupled together
9
What does this really mean?
In building a risk tolerant IMS,
we’re interested in the
probability of a successful
outcome…
“What is the probability of
making a desired completion
date?”
But the underlying statistics of
the tasks influence this
probability
The statistics of the tasks, their arrangement in a network of
tasks and correlation define how this probability based
estimated developed.
10
There are real problems with those pesky
Unknowns that get in the way of progress
Imprint of a bird on our west facing family room second story
window on a bright afternoon
The Bird survived 11
The “units of measure” of Risk
These classifications can be used to avoid asking the
“3 point” question for each task.
Anchoring and Adjustment† of all estimating
processes produces a bias.
Knowing this is necessary for credible estimates.
Classification Uncertainty Overrun
1 Routine, been done before Low 0% to 2%
2 Routine, but possible difficulties Medium to Low 2% to 5%
3 Development, with little technical difficulty Medium 5% to 10%
4 Development, but some technical difficulty Medium High 10% to 15%
5 Significant effort, technical challenge High 15% to 25%
6 No experience in this area Very High 25% to 50%
12
† Tversky and Khanemann Anchoring and Adjustment
We’re looking for knowledge of what is going to
happen in he future, with a known level of confidence
Harvard main library 13
14
With some principals behind us, let’s see how to use
Risk+ to address the problem of forecasting the future of
schedule and cost performance.
What is Monte Carlo Simulation? 15
A Quick Look At Monte Carlo
George Louis Leclerc,
Comte de Buffon, asked
what was the probability
that the needle would fall
across one of the lines,
marked here in green.
That outcome will occur
only if sinA l
16
Los Alamos Science, Special Issue 1987
17
Monte Carlo Simulation
Monte Carlo Simulation is named
after the city, in Monaco, of casinos
on the French Rivera.
Monte Carlo …
Examines all paths not just the critical
path.
Provides an accurate (true) estimate
of completion:
Overall duration distribution
Confidence interval (accuracy
range)
Sensitivity analysis of interacting tasks
Varied activity distribution types – not restricted to a single distribution
Schedule logic can include branching – both probabilistic and conditional
When resource loaded schedules are used – provides integrated cost and
schedule probabilistic model.
18
19
We need to answer the question …
What is the confidence we will complete “on or
before” at date and “at or below” at cost?
This is the question that should be asked and
answered on a periodic basis.
We need to have Schedule and Cost margin to
protect the deliverables and our Budget At
Completion.
What Are We Really After? 20
Here is some advice on how
to depict this margin and
where to place this margin.
No matter how we show
manage these two elements
in the IMS, if we don’t have
margin we are late and
over budget before we
start.
http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/CommitteeDocuments/
WhitePapers/NDIAScheduleMarginWhitePaperFinal-2010(2).pdf 21
Confidence levels for margin change as the program proceeds
As the program proceeds we want to have
Increased accuracy
Reduced schedule risk
Increasing visual confirmation that success can be reached
Current Estimate Confidence
22
Our REAL goal here is to Manage Margin using probabilistic models
Programmatic Margin is added between Development, Production and Integration & Test phases
Risk Margin is added to the IMS where risk alternatives are identified
Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives
This enables us to buy back schedule margin for activities further downstream
This enables us to control the ripple effect of schedule shifts on Margin activities
5 Days Margin
5 Days Margin
Plan B
Plan A
Plan B
Plan AFirst Identified Risk Alternative in IMS
Second Identified Risk
Alternative in IMS
3 Days Margin Used
Downstream
Activities shifted to
left 2 daysDuration of Plan B < Plan A + Margin
2 days will be added
to this margin task
to bring schedule
back on track
23
Sensitivity Analysis
The schedule sensitivity of a task measures the closeness with which change in the task duration matches change in the project duration over the simulation.
This closeness is the correlation between changes in individual activities and their impacts on other activities.
A task with high schedule sensitivity is more likely to be a major driver of the project duration than a lower ranked task.
: Models of the Schedule
24
Task Criticality Analysis
A measure of the frequency that an activity in the
project schedule is critical (Total Float = 0) in a
simulation
If a task is critical in 500 of the 1,000 iterations of
the simulation, it has a Criticality Index of 0.5
The higher the criticality index, the more certain it is
that the task will always be critical in the project
: Models of the Schedule
25
Cruciality shows each task’s tolerance to risk
Cruciality = Schedule Sensitivity x Criticality
Schedule Sensitivity can be statistically misleading:
A task with high sensitivity may not be on or near the critical path.
Thus a reduction in that task’s duration may have little effect on the project duration.
Cruciality sharpens the analytical focus:
It highlights critical or near–critical activities with high.
Schedule Sensitivity
These tasks are most likely to drive project duration.
: Models of the Schedule
26
Guiding the Risk Factor Process means weighting each level of risk
For tasks marked “Low” a reasonable
approach is to score the maximum
10% greater than the minimum.
The “Most Likely” is then scored as a
geometric progression for the
remaining categories with a common
ratio of 1.5
Tasks marked “Very High” are bound
at 200% of minimum.
No viable project manager would like
a task grow to three times the planned
duration without intervention
The geometric progress is somewhat
arbitrary but it should be used instead
of a linear progression
Min Most
Likely
Max
Low 1.0 1.04 1.10
Low+ 1.0 1.06 1.15
Moderate 1.0 1.09 1.24
Moderate+ 1.0 1.14 1.36
High 1.0 1.20 1.55
High+ 1.0 1.30 1.85
Very High 1.0 1.46 2.30
Very High+ 1.0 1.68 3.00
: Examples of Monte Carlo
27
Progressive Risk Factors
A geometric progression (1.534) of risk can be
used.
The phrases associated with increasing risk have
been shown at the Naval Research Laboratory to
correlate with an engineers “sense” of increasing
risk.
: Examples of Monte Carlo
28
Risk Factor Attributes
The “narrative” for each risk factor needs to be developed.
Each description is dependent on…
Discipline
Program stage
Complexity
Historical data
Current “risk state” of the program
This is currently missing from our efforts to quantify schedule and cost risk.
: Examples of Monte Carlo
29
Accuracy
Given a specified final cost or project duration, what is the probability of achieving this cost or duration?
Frequentist approach
Over many different projects, four out of five will cost less or be completed in less time than the specified cost or duration.
Bayesian approach
We would be willing to bet at 4 to 1 odds that the project will be under the 80% point in cost or duration.
Accuracy is needed to plan reserves.
Accuracy is needed when comparing competing proposals.
: What is the Purpose of Project Risk Analysis?
30
Structured Thinking
All estimates will be in error to some degree of variance.
Trying to quantify these errors will result in bounds too wide to be useful for decision making.
Risk analysis should be used to
Think about different aspects of the project
Try to put numbers against probabilities and impacts
Discuss with colleagues the different ideas and perceptions
Thinking things through carefully results in
Which elements of the programmatic and technical risk are represented in the IMS.
The process becomes more valuable than the numbers.
: What is the Purpose of Project Risk Analysis?
31
To properly use Schedule Margin†
Work must be represented in single units – either
task or work packages.
The overall schedule margin must be related to the
variation of individual units of work.
The importance of the units of work must be shared
among all participants (ordinal ranking of work and
its risk).
The schedule must be reasonable in some units of
measure shared by all the participants.
† “Protecting Earned Value Schedules with Schedule Margin,” Newbold, Budd, and Budd,
http://www.prochain.com/pm/articles/ProtectingEVSchedules.pdf
32
Let’s Apply a Monte Carlo Simulation Tool The Monte Carlo trolley, or FERMIAC,
was invented by Enrico Fermi and
constructed by Percy King. The drums
on the trolley were set according to
the material being traversed and a
random choice between fast and slow
neutrons.
Another random digit was used to
determine the direction of motion,
and a third was selected to give the
distance to the next collision. The
trolley was then operated by moving
it across a two dimensional scale
drawing of the nuclear device or
reactor assembly being studied.
The trolley drew a path as it rolled,
stopping for changes in drum settings
whenever a material boundary was
crossed. This infant computer was
used for about two years to
determine, among other things, the
change in neutron population with
time in numerous types of nuclear
systems. 33
34
Most Likely Isn’t Likely to be the Most Likely
When we say “most likely” what do we think this
actually means?
If you pick the wrong meaning, your Monte
Carlo model will be seriously flawed.
A Small Diversion 35
The problem with “Most Likelies”
For each activity the “best” estimate is …
The “most likely” duration – the mode of the distribution of
durations? (Mode is the number that appears most often)
It’s 50th percentile duration – the median of the distribution?
(Median is the number in the middle of all the numbers)
It’s expected duration – the mean of the distribution? (Mean
is the average of all the numbers)
These definitions lead to values that are almost always
different from each other.
Rolling up the “best” estimate of completion is almost
never one of these.
36
Durations are Probability Estimates not Single Point Values
We know this because…
“Best” estimate is not the only possible estimate, so other estimates must be considered “worse.”
Common use of the phrase “most likely duration” assumes that other possible durations are “less likely.”
“Mean,” “median,” and “mode” are statistical terms characteristic of probability distributions.
This implies activity distributions have probability distributions
They are random variables drawn from the probability distribution function (pdf).
“Actual” project duration is an uncertain quality that can be modeled as a sum of random variables
The pdf may be known or unknown.
37
Task Most Likely ≠ Project Most Likely
PERT assumes probability distribution of the project times is the same as the tasks on the critical path.
Because other paths can become critical paths, PERT consistently underestimates the project completion time. 1 + 1 = 3
: Managing Uncertainty in the IMS
3 38
Probability Distribution Function is the Lifeblood of good planning
Probability of occurrence as a function of the number of samples.
“The number of times a task duration appears in a Monte Carlo simulation.”
: Managing Uncertainty in the IMS
39
Remember the quote about statistics
Lies, Damn Lies, and Statistics
– Benjamin Disraeli
But we know better, we know that
any estimate without a variance is
not trustworthy.
We know that the variances have
to be calibrated from past
performance to be credible
40
41
One should expect that the expected
can be prevented, but the unexpected
should have been expected.
— Augustine Law XLV
A “Real World” Schedule Analysis 42
This is a must own book for everyone in our business. It defines
fundamental Laws of program and business management, which
are many times ignored – like the one above
Our Starting Point
Risk+ Installed
Let’s define the
needed fields
These are used
by Risk+ to hold
information and
run the
application.
43
If there are conflicts, you can make changes in Risk+ to work around your fields.
A Simple IMS 44
By simple it means serial cascaded work efforts.
Initial Field Usage
Minimum Remaining Duration
The duration that is least you’d expect this task to complete in
Most Likely Remaining Duration
The ML (Mode) of the duration
Maximum Remaining Duration
The duration that is the most you’d expect this task to complete in
Task Reporting ID
The tasks we want to watch
45
Define a View and Table for Risk+
Start with the Gantt View and Entry Table
Set up both to match the Risk+ field usage
Use the default if
there are no field
conflicts
46
Fields used for Risk+ example 47
Let’s actually “doing something”
Initialize the Most Likely.
This sets the Most Likely
duration to the same value
that is in the “Duration” field
of your IMS.
The “planned duration” now
becomes the ML duration.
If this “planned duration” is
bogus then your model will be
as well.
Choose wisely.
48
Now the ML = DURATION step
All the DURATION values have been moved to the ML field.
But remember our discussion of the ML’s
Choose them carefully
The next we’ll set the upper and lower limits of that ML value
Using risk factors.
OK, 3 point estimates if you have to.
49
Let’s do this the simple way
Let’s pick MEDIUM confidence.
MEDIUM means
–25%
+25%
And a NORMAL (Gaussian) curve
50
Let’s have Risk+ do something for us
Enter a “1” in the RPT field (Number 1)
This marks that ROW in the schedule as a work
activity we want to see the Monte Carlo output for
51
Now we’re ready to run
The RISK ANALYSIS command
starts the process going.
Let’s make 200 iteration and
look at the DURARTION
ANALYSIS for the activities we
are watching.
52
This is nice but what actually is Risk + doing?
Risk+ is picking a random number from under the normal distribution within the range of the
Least remaining and most remaining
This is not some ordinary random number it is chosen through an algorithm called the Latin Hypercube - more on that later.
Risk+ then plugs that number into the “real” DURATION field and does that for all the DURATIONS in the schedule
Then the F9 key is pressed and the date is recorded for the finish of UID 41.
This is done 200 times and a histogram of all the dates that appeared for those 200 time is recorded.
53
And We Get
Date: 11/29/2011 4:32:17 PM
Samples: 500
Unique ID: 19
Name: End Work Package 3
Completion Std Deviation: 2.06 days
95% Confidence Interval: 0.18 days
Each bar represents 1 day
Completion Date
Fre
qu
en
cy
Cu
mu
lative
Pro
ba
bili
ty
Mon 3/12/12Fri 3/2/12 Tue 3/20/12
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.18
0.20 Completion Probability Table
Prob ProbDate Date
0.05 Wed 3/7/12
0.10 Thu 3/8/12
0.15 Thu 3/8/12
0.20 Fri 3/9/12
0.25 Fri 3/9/12
0.30 Fri 3/9/12
0.35 Mon 3/12/12
0.40 Mon 3/12/12
0.45 Mon 3/12/12
0.50 Mon 3/12/12
0.55 Tue 3/13/12
0.60 Tue 3/13/12
0.65 Tue 3/13/12
0.70 Wed 3/14/12
0.75 Wed 3/14/12
0.80 Wed 3/14/12
0.85 Thu 3/15/12
0.90 Thu 3/15/12
0.95 Fri 3/16/12
1.00 Tue 3/20/12
54
Learning to Speak in Risk+
Risk +shows use the probability of finish “on or
before” a date
It does NOT show the probability of success.
But even the “on or before” term is loaded with
special meaning.
It means for the 500 iterations of Risk+ using the
upper and lower bounds of the duration, drawn
from the probability density function (pdf) with the
Normal (Gaussian) shape, 60% of the finish dates
were recorded to be on or before 3/12/12.
55
Medium confidence for a large project
Date: 11/30/2011 6:05:35 PM
Samples: 200
Unique ID: 17
Name: (SA) Systems Requirements Completed
Completion Std Deviation: 4.49 days
95% Confidence Interval: 0.62 days
Each bar represents 2 days
Completion Date
Fre
qu
en
cy
Cu
mu
lative
Pro
ba
bili
ty
Wed 5/16/12Wed 5/2/12 Mon 6/4/12
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.03
0.05
0.08
0.10
0.13
0.15
0.17
0.20
0.22 Completion Probability Table
Prob ProbDate Date
0.05 Fri 5/4/12
0.10 Wed 5/9/12
0.15 Thu 5/10/12
0.20 Fri 5/11/12
0.25 Mon 5/14/12
0.30 Mon 5/14/12
0.35 Tue 5/15/12
0.40 Tue 5/15/12
0.45 Wed 5/16/12
0.50 Wed 5/16/12
0.55 Thu 5/17/12
0.60 Thu 5/17/12
0.65 Fri 5/18/12
0.70 Mon 5/21/12
0.75 Mon 5/21/12
0.80 Tue 5/22/12
0.85 Wed 5/23/12
0.90 Thu 5/24/12
0.95 Mon 5/28/12
1.00 Mon 6/4/12
56
Low confidence for a large project
Date: 11/30/2011 10:30:05 PM
Samples: 200
Unique ID: 17
Name: (SA) Systems Requirements Completed
Completion Std Deviation: 9.14 days
95% Confidence Interval: 1.26 days
Each bar represents 3 days
Completion Date
Fre
qu
en
cy
Cu
mu
lative
Pro
ba
bili
ty
Thu 5/24/12Tue 4/24/12 Wed 6/27/12
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16 Completion Probability Table
Prob ProbDate Date
0.05 Thu 5/3/12
0.10 Tue 5/8/12
0.15 Wed 5/9/12
0.20 Mon 5/14/12
0.25 Tue 5/15/12
0.30 Thu 5/17/12
0.35 Fri 5/18/12
0.40 Mon 5/21/12
0.45 Wed 5/23/12
0.50 Wed 5/23/12
0.55 Fri 5/25/12
0.60 Mon 5/28/12
0.65 Wed 5/30/12
0.70 Wed 5/30/12
0.75 Fri 6/1/12
0.80 Mon 6/4/12
0.85 Wed 6/6/12
0.90 Fri 6/8/12
0.95 Thu 6/14/12
1.00 Wed 6/27/12
57
58
Let’s run some
simulations
59
Now that the schedule can be produced using probabilistic methods, it’s time to talk about the cost.
Cost does not have a linear relationship with schedule unfortunately.
Basic Principles of Probabilistic Cost
: Basic Principles of Probabilistic Cost
60
Basic Principles with Probabilistic Cost Estimating are coupled with scheduling
Cost estimates usually involve many CERs
Each of these CERs has uncertainty (standard error)
CER input variables have uncertainty (technical uncertainty)
Must combine CER uncertainty with technical uncertainty for many CERs in an estimate
Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples
Add and multiply probability distributions rather than numbers
Statistically combining many uncertain, or randomly varying, numbers
Monte Carlo simulation
Take random sample from each CER and input parameter, add and multiply as necessary, then record total system cost as a single sample
Repeat the procedure thousands of times to develop a frequency histogram of the total system cost samples
This becomes the probability distribution of total system cost
: Basic Principles of Probabilistic Cost
61
The Cost Probability Distributions as a function of the weighted cost drivers
$
Cost Driver (Weight)
Cost = a + bXc
Cost
Estimate
Historical data point
Cost estimating relationship
Standard percent error bounds Technical Uncertainty
Combined Cost Modeling
and Technical Uncertainty
Cost Modeling Uncertainty
: Basic Principles of Probabilistic Cost
62
The Risk Adjusted Cost Estimate Connected To The IMS
In the risk–adjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs
Since the input distributions tend to be right–skewed, the expected cost tends to be larger than the baseline estimate
In addition, the risk–adjusted cost distribution tends to be wider than the baseline estimate
The difference between the expected cost of the risk–adjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of RISK dollars included in the risk–adjusted estimate
: Basic Principles of Probabilistic Cost
63
Baseline versus Risk Adjusted Cost Estimates Usually Show a Cost Increase
Baseline vs. Risk-Adjusted Estimates
0 50 100 150 200 250 300 350
FY$M
Lik
eli
ho
od
Baseline:
Mean = $102.6M
Std Dev = $29.8M
Risk–Adjusted:
Mean = $122.6M
Std Dev = $42.8M
: Basic Principles of Probabilistic Cost
64
The S–Curve for Cost Modeling
Cumulative Distribution Function
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
$60 $80 $100 $120 $140 $160 $180 $200
FY00$M
Cu
mu
lati
ve
Pro
ba
bilit
y
Baseline Estimate
Mean $102.6M
50th percentile
$114.7M
Risk–adjusted
Estimate Mean
$122.6M
80th percentile
$153.5M
: Basic Principles of Probabilistic Cost
65
The Real Question Always Returns to… “But How Much Does It Cost? Really?”
This is impossible to answer precisely
Decision–makers and cost analysts should always think of a cost estimate as a probability distribution, NOT as a deterministic number
The best we can provide is the probability distribution – If we think we can be any more precise, we’re fooling ourselves
It is up to the decision–maker to decide where he/she wants to set the budget
The probability distribution provides a quantitative basis for making this determination
Low budget = high probability of overrun
High budget = low probability of overrun
: Basic Principles of Probabilistic Cost
66
67
Just having the pictures is necessary, but knowing
what they mean is required.
Making changes to the IMS to increase the
Probability of Program Success is the primary
outcome from Monte Carlo Simulation.
Some More Parts to using Risk+ 68
Without Integrating $, Time, and TPM you’re driving in the rearview mirror
Technical Performance (TPM)
69
Risk Management Demands a Well Defined Process
Statistics of a Triangle Distribution
Mode = 2000 hrs
Median = 3415 hrs
Mean = 3879 hrs
Minimum
1000 hrs
Maximum
6830 hrs
50% of all possible values are
under this area of the curve. This
is the definition of the median
Basic Statistics
71
TPM Trends & Responses directly impact risk and credibility of the IMS
Dr. Falk Chart – modified
25kg
23kg
28kg
26kg
PDR SRR SFR CA TRR CDR
ROM in Proposal
Design Model
Bench Scale Model Measurement
Detailed Design Model
Prototype Measurement
Flight 1st Article
Tech
nic
al P
erfo
rman
ce M
easu
re
Veh
icle
Wei
ght
72
Not A Mitigation Plan
Mitigation is too late, the risk has
turned into an issue. The money
has been spent, and the time has
passed.
73
Ordinal versus Cardinal
A variable is ordinally measurable if ranking is possible for values of the variable. For example, a gold medal reflects superior performance to a silver or bronze medal in the Olympics, or you may prefer French toast to waffles, and waffles to oat bran muffins. All variables that are cardinally measurable are also ordinally measurable, although the reverse may not be true.
A variable is cardinally measurable if a given interval between measures has a consistent meaning, i.e., if the measure corresponds to points along a straight line. For example, height, output, and income are cardinally measurable.
74
Ordinal Cardinal
Correcting Ordinal Risk Scales
Classify and calibrate risk ranking in units
meaningful to the decision makers
Risk rank 1, 2, 3, 4, is NOT sufficient
The Risk Rank must have a measurable value connected
to the actual behavior of the system being assessed
Calibration coefficients between ordinal probability
and consequences should also be used.
Ordinal analysis assumes ordering of the risks.
Cardinal analysis provides objective measures of
probability and consequential impact.
75
E
D
C
B
A
A B C D E
Level Likelihood Value
E Near Certainty E ≥ 90%
D Highly Likely 74% ≤ D ≤ 90%
C Likely 40% ≤ C ≤ 60%
B Low Likelihood 20% ≤ B ≤ 40%
A Not Likely A ≤ 20%
Level Technical Performance Schedule Cost
A Minimal or no consequence to technical performance.
Minimal or no impact Minimal or no impact
B Minor reduction in technical performance or supportability.
Able to meet key dates Budget increase or unit production cost increases. < (1% of Budget)
C
Moderate reduction in technical performance or supportability with limited impact on program objectives.
Minor schedule slip. Able to meet key milestones with no schedule float.
Budget increase or unit production cost increase < (5% of Budget)
D
Significant degradation in technical performance or major shortfall in supportability.
Program critical path affected
Budget increase or unit production cost increase < (10% of Budget)
E Severe degradation in technical performance.
Cannot meet key program milestones. Slip > X months
Exceeds budget increase or unit production cost threshold
Never multiply Likelihood by outcome. They are not “numbers,” they a probability distributions. Only convolution is possible
76
These are Cardinal measures of probability of occurrence and consequential impact
Example of Ordinal Probability Complexity Scale†
Definition of the Ordinal Scale Ranking Scale Level
Greater than 20% of the interface design has been
altered because of modifications to the ICD’s. E
Greater than 15% but less than 20% of the interface
design has been altered because of modifications of the
ICD’s.
D
Greater than 10% but less than 15% of the interface
design has been altered because of modifications of the
ICD’s.
C
At least 5% but less than 10% of the interface design has
been altered because of modifications of the ICD’s. B
At least 5% of the interface design has been altered
because of modifications of the ICD’s. A
† Effective Risk Management: Some Keys to Success, Edmund Conrow, AIAA Press, 2003
77
A “real” risk Ordinal Ranking Table
Risk
Rank Percent Variance Interpretation of Risk Ranking
A – 5% ≤ A ≤ 10% Normal business, technical & manufacturing
processes are applied
B – 5% ≤ B≤ 15%
Normal business & technical processes are
applied; new or innovative manufacturing
processes
C – 5% ≤ C ≤ 35% Flight software development & certification
processes
D – 10% ≤ D ≤ 25% Build & qualification of flight components,
subsystems & systems
E – 10% ≤ E ≤ 35% Flight software qualification
F – 5% ≤ F ≤ 175% ISS thermal vacuum acceptance testing
78
Untimely and unrealistic Latest Revised Estimates (LRE) Progress not monitored in a regular and
consistent manner Lack of vertical and horizontal traceability
cost and schedule data for corrective action Lack of internal surveillance and controls Managerial actions not demonstrated using
Earned Value
Inattention to budgetary responsibilities Work authorizations that are
not always followed Issues with Budget and data
reconciliation Lack of an integrated
management system Baseline fluctuations and
frequent replanning Current period and retroactive
changes Improper use of management
reserve EV techniques that do not
reflect actual performance Lack of predictive variance
analysis
Project Train Wrecks Occur When There is…
79
Our Final Check List
Set up the Risk+ fields, flags, views, and tables for the program standard IMS.
Build an IMS that passes the DCMA 14 Point Assessment with all GREEN.
Build the Ordinal Risk Ranking table for the various risk categories on the program.
Assign risk ranking to each activities in the IMS, with the variances defined in the Ordinal Table.
Run Risk+ to see the confidence in the deliverables.
Develop the needed schedule margin to protect the delivery to at least the 80% confidence level.
80
Advice from the school of hard knocks
Put margin in front of critical deliverables.
Build a margin burn down chart and allocate schedule margin just like you do MR for the PMB.
This real world advice is counter to the current DCMA guidance.
81
82
Putting This New Knowledge To Work 83
Managing margin is what Risk+ is all about
-40
-20
0
20
40
60
80
100
Critica
l P
ath
- T
ime
Re
se
rve
CP Total Float
Acceptable Rate of Float Erosion
Linear (CP Total Float )
Time Now October 31, 2005
Spacecraft Contract Delivery
December 10, 2007
Float Erosion: Critical Path Time Usage
84
How much margin do we need?
The Missing Link: Schedule Margin Management, Rick Price, PS–10, PMI–CPM EVM World 2008
85
Deterministic versus Probabilistic
Baseline
Plan
80%
Mean
Missed
Launch
Period
Launch
Period
Ready
Early
Sep 2011
Oct 2011
Nov 2001
Dec 2011
Jan 2012
Feb 2012
Mar 2012
Apr 2012
Margin
Risk
Margin
Current Plan
with risks is the
stochastic schedule
CD
R
PD
R
SR
R
FR
R
AT
LO
20%
Current Plan
with risks is the
deterministic schedule
Plan
The probability
distribution can
vary as a
function of time
86
87
References 88
References
“Protecting Earned Value with Schedule Margin,”
http://www.prochain.com/pm/articles/ProtectingEVSchedules.pdf
Depicting Schedule Margin in the Integrated Master Schedule,
http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/C
ommitteeDocuments/WhitePapers/NDIAScheduleMarginWhitePaperFinal-
2010(2).pdf
Effective Risk Management: Some Keys to Success, Second Edition, Edmund Conrow,
AIAA Press.
How to Lie with Statistics, Darrell Huff, Norton, 1954 (Available in paper back at
any good book store)
DID DI–MGMT–81650 “A management method for accommodating schedule
contingencies. It is a designated buffer and shall be identified separately and
considered part of the baseline.
89
References
Interfacing Risk and Earned Value Management, Association for Project Management,
150 West Wycombe Road, High Wycombe, Buckinghamshire, HP12 3AE, United
Kingdom.
Practice Standard for Earned Value Management, Second Edition, Project
Management Institute, 2011.
Effective Opportunity Management for Projects, David Hillson, Taylor and Francis,
2004.
Measuring Time: Improving Project Performance Using Earned Value, Mario
Vanhoucke, Springer, 2009.
Performance Based Earned Value, Paul Solomon and Ralph Young, Wiley, 2007.
Effective Risk Management: Some Keys to Success, Edmund Conrow, AIAA Press,
2003.
90
91
Niwot Ridge LLC
4347 Pebble Beach Drive
Niwot, Colorado 80503
(: 303.241.9633