unknowns, systemic risks and risk prioritization in ... · unknowns, systemic risks and risk...

Unknowns, Systemic Risks and Risk Prioritization

in Schedule Risk Analysis David T. Hulett, Ph.D., FAACE

Hulett & Associates, LLC

(c) 2016 Hulett & Associates, LLC 1

Secretary Rumsfeld’s use of “Unknown Unknowns”

In February 2002 DOD Secretary Donald Rumsfeld stated: “Reports that say that something hasn't happened are always interesting to me, because as we know:

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.

And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.” (source: http://en.wikipedia.org/wiki/There_are_known_knowns cited January 26, 2015)

2 (c) 2016 Hulett & Associates, LLC

http://en.wikipedia.org/wiki/There_are_known_knowns

Uncertainty including Inherent Variability, Estimating Error, Estimating Bias

• Inherent variability in project activities that arise because people and organizations cannot do things reliably on plan

• Estimating error – attaches to all types of estimates • Estimating bias – estimates may be slanted, usually

toward shorter durations, to make desired project results

“There are No Facts About the Future”


Lincoln Moses, Statistician and Administrator of Energy Information in the US DOE 1977 Annual Report to Congress)

Uncertainty

• Uncertainty in schedule duration is similar to “common cause” variation related to six sigma management, concepts developed by Walter Shewhart and championed by Edwards Demming

• “Common cause variability is a source of variation caused by unknown factors that result in a steady but random distribution of output around the average of the data …. Common cause variation is also called random variation, noise, non-controllable variation … ” (http://www.isixsigma.com/dictionary/common-cause-variation/ )

• Hence application of uncertainty alone establishes the earliest date that risk mitigation can achieve even if individual risks were fully mitigated

4

http://www.isixsigma.com/dictionary/common-cause-variation/

http://www.isixsigma.com/dictionary/common-cause-variation/

Example Schedule: Offshore Gas Production Platform Project


Case Study On Application of Uncertainty and Risk


Offshore Gas Production Platform Construction project, 3+ years and $1.7 billion. Using Polaris© from Booz Allen Hamilton

Correcting for Inherent Variability, Estimating Error and Bias

• Schedule uncertainty exhibiting inherent variation, estimating uncertainty and bias with ranges of .9, 1.05 and 1.3

• To replicate the estimated overall project ranges by using these parameters for individual activities we need to correlate the uncertainty


Correlated durations

Uncorrelated durations

Without Correlation P-80 duration is 113% of planned With Correlation at 1.0 the P-80 duration is 116% of planned

Variation Caused by Known Unknowns

• Project-specific risks are: – Characterized by probability of occurring that is

usually less than 100% – Represent “root causes” of variation in durations – May be reducible by risk mitigation actions

• These are specified during risk interviews and implemented with: – Estimated probabilities – Impact ranges (3-point estimates of multiplicative

factors called Risk Drivers”) if they occur – The activities or cost elements they influence


Project-Specific Risks, Pre-Mitigated

• Risk is similar to “special causes” in six sigma

• “… special cause variation is caused by known factors that result in a non-random distribution of output…Special cause variation is a shift in output caused by a specific factor such as environmental conditions or process input parameters. It can be accounted for directly and potentially removed...” (http://www.isixsigma.com/dictionary/variation-special-cause/)

• Hence, pre-mitigated risks are the subject of risk mitigation workshops. The improvement from risk mitigation is limited by the date determined by uncertainty alone

9

http://www.isixsigma.com/dictionary/variation-special-cause/

http://www.isixsigma.com/dictionary/variation-special-cause/

Applying Known Unknowns: Project-Specific Risk Drivers to the Offshore Platform


There are 8 project-specific Risk Drivers including one Organizational Risk Driver assigned to all tasks. Probability and impact multiplier ranges are specified, risks are assigned to activities. A risk can affect many activities, and activities can be affected by several risks.

Adding Project Specific Risk Drivers


Adding Project Specific Risk Drivers

Uncertainty correlated 1.0

Uncertainty not correlated

Adding Project-Specific Risk Drivers increases P-80 Duration to 125% of planned

Short Digression on Risk Drivers and Correlation


Risk Factors Model How Correlation Occurs Coefficients are Calculated (1)

Risk Probability = .5, Range .95, 1.05, 1.15

Activity 2 Activity 1

Correlation = 100%

We are very bad at estimating correlation coefficients directly

Risk Factors Model How Correlation Occurs Coefficients are Calculated (2)

• Correlation is modeled as it is caused in the project • Correlation coefficients are generated, not guessed • Correlation drives the results correctly • By modeling correlation we never get an inconsistent correlation

coefficient matrix

Risk Probability = .5, Range .95, 1.05, 1.15

Activity 2 Activity 1

Correlation = 37%

Risk Probability = .45, Range 1.0, 1.10, 1.20

Risk Probability = .25, Range .8, .95, 1.05

Unknown Unknowns May Not be Unknowable

• Interviews or workshops on risk often focus on risks and uncertainties that are close-in or actually happening now – This myopia leads to insufficient consideration of future

risks • It is arguable that some unknown unknowns are not

truly unknowable but just have not been thought of yet – This may be due to a lack of attention to risks that may

occur “down the road” or hard to think about • Since these risks are “unknown” right now:

– Extra effort to focus on down-stream risks during the risk interviews could improve our understanding of risks, including making some of these “Unknown Unknowns” known


Add a Fourth Category of “Unknown Knowns”

• Psychoanalytic philosopher Slavoj Zizek says that beyond these three categories there is a fourth, the unknown known, that which we intentionally refuse to acknowledge that we know

• German sociologists Daase and Kessler (2007) agree with a basic point of Rumsfeld in stating that the cognitive frame for political practice may be determined by the relationship between what we know, what we do not know, what we cannot know, but Rumsfeld having left out what we do not like to know


http://en.wikipedia.org/wiki/There_are_known_knowns, Cited 12/26/2015)

http://en.wikipedia.org/wiki/There_are_known_knowns cited January 24

Risk Interviews vs. Risk Workshops (1)

• Risk Interviews provide a safe environment to discuss both Unknown Unknowns and Unknown Knowns

• Specifically, Unknown Knowns – We find that the Risk Register is always incomplete – Many of the risk events found to be most important in

determining the schedule risk results are not in the Risk Register at all

– These can be safely discussed in one-on-one interviews where confidentiality is promised

• In these interviews new risks are discussed, whether they are hurtful to the project or embarrassing – Responding to open-ended questions, interviewees are

encouraged to talk about “the good, the bad and the ugly”


Risk Interviews vs. Risk Workshops (2) • We need to avoid the ostrich approach to risk management.

Identify, talk about and quantify the Unknown Knowns, the “elephant in the room,” and tell unbiased results

• Risk interviews provide a confidential environment where individuals can talk in depth without being afraid of being embarrassed, pressured or subjected to “shoot the messenger”

• Conversely, Risk Workshops often are places where social pressures or group dynamics limit debate – Groupthink – prefer unanimity, discourage dissent – “Moses factor” – adopting influential person’s ideas – Cultural Conformity – decisions match the group’s norms


David Hillson and Ruth Murray-Webster, Understanding and Managing Risk Attitude 2005

Risk Interviews Help Identify Unknown Unknowns and Unknown Knowns

• Think about risks later in the project that may have been ignored as “unknown” at first glance

• Suggest risks from experience to see if the interviewee has forgotten some that apply

• Ask open ended questions to elicit the interviewee’s own concerns about the project, with a pledge that nobody will be able to connect their name with any information they share


Considering Systemic Risks


Compare what Risk Analysis Typically Predicts vs. What Actually Happens


Source: John K. Hollmann, PE, “Reliable Risk Quantification for Project Cost and Schedule”, AACE International webinar December 15, 2015

Incorporate Systemic Risks into the Monte Carlo Simulation (MCS) • There are some who believe that Monte Carlo, by

building risk from the bottom up, does not deal with Systemic Risks that include: – Technical complexity, new technology challenging – Scope not fully known – Process definition not complete – Megaproject complexity, size / duration, participants – Project organization, e.g., joint venture, multiple EPCs – Project management, scheduling and estimating process,

bias • Some argue that these factors can be measured and

their impact on project success estimated using parametric techniques


Alternative Approach to Incorporate Systemic Risks

23

Source: John K. Hollmann, PE, “Reliable Risk Quantification for Project Cost and Schedule”, AACE International webinar December 15, 2015

Effect of Simply using Hollmann’s Correction for Actuals


P-80 for Uncertainty and Project Specific Risks = 125% of scheduled P-80 for Uncertainty set at .8, 1, 2.2 without Project Specific Risks = 162% of scheduled No information about which risks caused this difference beyond the wider ranges

Uncertainty and Project Specific Risks

Uncertainty set at .8, 1, 2.2 per Hollman Benchmarking, No Project Specific Risks

Inserting 3 Megaproject Systemic Risks with Original Uncertainty

• Identifying the systemic risks and inserting them with appropriately-large impacts allows us to: – Specify the probability of occurrence – Identify the risks for risk mitigation

• In this case study, these megaproject risks are: – May have interdependency issues between project

elements (20%, 1.4, 1.5, 1.7 assigned to all activities) – May have complex offshoring of supply chain and

even EPC contractors (10%, 1.3, 1.4, 1.6 all activities) – May have excessive schedule pressure “I want it

sooner” (15%, 1.2, 1.3, 1.5 all activities)


Individual Systemic Risks with Regular Uncertainty Approximates Hollmann


Regular Uncertainty, Project-specific Risks

Hollmann Benchmarking Uncertainty

Regular Uncertainty, Project-specific and Megaproject Systemic Risks

Parameters for specific Megaproject Systemic Risks are challenging, need to be selected based on historical data We need to be sure to include the systemic risks with the right probability and impact ranges based on data into the Monte Carlo simulation

Results for Total Project Duration from MCS


Some suggest that megaprojects that come in at 25% or less (cost) over the sanction budget are considered successful. This curve shows that about 50% do so. The P-80 duration is 1,913 days about 61% overrun of schedule

Results are Similar to those for Cost Provided by IPA


“Quantifying Estimate Accuracy and Precision for the Process Industries: A Review of Industry Data” Alexander Ogilvie, Robert A. Brown, Jr., Fredrick P. Biery and Paul Barshop from IPA. IPA’s Front End Loading (FEL) 2 is “Scoping Facility Planning,” comparable to AACE International Class 4 estimates which gets a high range of +30% to +50%. This is not high enough

+35 to +50%

Risk Prioritization and Risk Mitigation Illustrated


Problems with Typical Risk Prioritization Approaches

• Risk prioritization usually involves presenting: – Standard tornado diagrams based on correlation between

activities and finish date – Risk tornado diagrams, based on correlation between risks and

finish date – Correlation-based tornados have problems with risks that may

or may not occur – Risk criticality based on activities’ being on the critical path

• None of these methods give management what they need to assess mitigations – Risks prioritized at the P-80, not means as in correlation – Risks measured in days saved if mitigated – Risks have to be the arguments driving MCS to do this


Prioritizing Risks using MCS Selected by their Days Saved at P-80

31

Iterative Approach to Prioritizing Risks (Based on Days Saved at P-80) Risk # 1 2 3 4 5 6 7 8

Priority Level (Iteration #)

Abusive Bids

Offshore design firm

Suppliers Busy

Fab productivity

Geology unknown

Coordination during Installation

Problems at HUC

Resources may go to other projects

1 X X X X X X X 1 2 X X X 2 X X X 3 X 3 X X X X 4 X X X X 4 5 X 5 X X 6 X X 6 7 7 X 8 8

© 2015 Hulett & Associates

Risk Prioritization for Mitigation


Megaproject Systemic Risks

Risk Prioritization Results for Risk Mitigation Workshop


Risk Prioritization at P-80 with Days Saved Name Days Saved Megaproject may have interdependency problems 245 Megaproject may have excessive schedule pressure 128 The organization has other priority projects so personnel and funding may be unavailable 54 Megaproject may have coordination problems offshore sourcing 46 Fabrication yards may experience lower Productivity than planned 25 Engineering may be complicated by using offshore design firm 12 Suppliers of installed equipment may be busy 13 Fabrication and installation problems may be revealed during HUC 9 Bids may be Abusive leading to delayed approval 1 Installation may be delayed due to coordination problems 1 The subsea geological conditions may be different than expected 0 Days Saved from Project Specific and Systemic Risks 534 Days Contributed by Uncertainty alone 189 Total Days contingency at P-80 723

Risk Mitigation Workshop(s)

• This is a workshop with the project manager, deputy PM, team leads, controls personnel, SMEs with experience

• Use the prioritized risk list – Start at the top – Working on risks lower on the priority list will not be

effective. Those risks are not important until the top risk is dealt with as much as possible

– Determined by the structure of the schedule and which paths are risk critical – changes as risks are mitigated

34

Sample Risk Mitigation Entry

35

Risk: The organization has other priority projects so personnel and funding may be unavailable

Probability Low Most Likely High P-80 Date P-80 Cost

($ billions)

Pre-Mitigated parameters 65% 95% 105% 125% 1/22/2018 $2.13

Mitigation Action

Establish this project as top priority - needs top management action and commitment

Post -Mitigated parameters 15% 95% 100% 115% 10/20/2017 $1.99

Risk Owner: S. Smith

Days saved Cost Saved

Date of Action: Within 1 month Results 94 $0.14

Risk Action Owner: B. Blake Cost of

Mitigation $0.02 Risk is not completely mitigated. Cost saved is the reduction of cost contingency reserve held for schedule risk. For Net Cost Saved subtract the $20 million cost of mitigation

Creating the Post-Mitigated Scenario • A risk post-mitigated scenario can be constructed in the

software – Partially mitigate each risk, in this case just by reducing

probability by half – Estimate the cost of the risk, in this case each risk’s

mitigation = $50 million – Run the post mitigated scenario

• When schedule risks are mitigated the cost contingency reserve can be reduced since some was held for schedule growth

• However, the cost of the project now includes the assumed $50 million cost of each mitigation

Partially Mitigate all Risks – Finish Date

Mitigating all risks (Here, just reducing the probability by half) moves the P-80 date by total mitigation time of about 7.5 months.

Uncertainty Only

All Risks partially mitigated

Pre-mitigation results

Partially Mitigate all Risks – Total Cost Uncertainty Only

All Risks partially mitigated

Pre-mitigation results

Notice the effect of the mitigation costs – in the red circle – these are included and still there is some cost savings, largely from the schedule risk mitigation

Use the Joint Confidence Level (e.g., JCL-80) from the

Integrated Cost-Schedule Risk Analysis


Integrated Cost-Schedule Risk Analysis Using the Joint Confidence Level (JCL)

• When we have an integrated cost-schedule risk analysis we have the opportunity to determine cost and end date targets that can both be met

• These turn out to be somewhat higher and later than the targets when considered separately

• We prefer the JCL-80 to the P-80


Cost and Schedule Risk Analysis with Systemic Risks


P-80 for end date is March 20, 2021 P-80 for cost is $3.25 billion

Integrated Cost-Schedule Risk JCL Scatterplot Results with Systemic Risks


The JCL-80 date is 18 May 2021 and the JCL-80 cost is $3.3 billion for the same assumptions and target confidence level. The correlation between time and cost is 97%

The second “mode” of the distribution is represented by the higher / later grouping of results in the scatter plot

Summary (1)

• Unknown Unknowns may not be unknowable today, but some may become known during confidential interviews – Myopia: many risks people discuss even in

confidential interviews are happening now or in the near future

– Encourage a focus on the “far future” of the project may reveal these, Unknown may become Known with effort


Summary (2) • A new category of “Unknown Knowns,” or those risks which

are known to exist but are not talked about, is addressed – SMEs may be unwilling to discuss these risks in a risk workshop

with others (e.g., the “boss”) in the room • Confidential interviews allow interviewees to discuss

“unknown knowns” and they introduce risks not included in the current Risk Register – Open ended questions often reveal important, embarrassing or

potentially dangerous risks that are not talked about – Once introduced, other team members seem to be able to

contribute to quantifying those “unspoken risks” – These “interview risks” are often more impactful than the risks

in the Risk Register


Summary (3) • Monte Carlo simulation methods have been challenged

to address systemic risks – Some say that systemic risks are best captured by

parametric analysis – Monte Carlo can specify probability of occurring on this

particular project • We show how systemic risks, often identified with

megaprojects, can be handled in MCS, compared to simple expanding the uncertainty ranges

• Project results databases need to be created or mined if they exist to determine the impact multipliers for systemic risks.


Summary (4)

• Risk mitigation decisions are greatly helped with prioritized risks

• Risks need to be prioritized at the target level of certainty, e.g., P-80

• Risks prioritization needs to supply some metric that management can use in a benefit / cost analysis, e.g., “days saved” even if assuming perfect mitigation

• These tables have been useful in mitigation workshops


Summary (5)

• The P-80 level for the end date and the project cost are typically used for specifying confident finish and cost costs

• But if we have costs on the schedule and perform integrated cost-schedule risk analysis the Joint Confidence Level (JCL) can provide targets that are both 80% likely

• These JCL-80 targets are typically later and larger than the P-80 targets, depending on the level of correlation between time and cost


Unknowns, Systemic Risks and Risk Prioritization

in Schedule Risk Analysis David T. Hulett, Ph.D., FAACE

Hulett & Associates, LLC


unknowns, systemic risks and risk prioritization in ... · unknowns, systemic risks and risk...

Documents