s thomas sfield
DESCRIPTION
TRANSCRIPT
Practical Applications of Risk Informed Decision Making –
No Fear!
PM Challenge 2012Sharon L. Thomas
Scott L. Field
Presentation Themes
• Inherent riskiness of uncertain times
• Evolving risk management to stay relevant
• What exactly is Risk Informed Decision-Making (RIDM)?
• Why RIDM now, since we’re already doing Continuous Risk Management (CRM)?
• RIDM is scary!
• When to use RIDM and why
• Actual steps (really?!)
• Go forth & do good RIDM
2
not
Uncertainty = Reality – Perception?
• We live in uncertain times! Whether at NASA or in industry, it’s no longer business as usual – belts are tighter!
• Budgets are fluctuating, missions requirements are changing.
• Advocates and naysayers alike point to risk – known unknowns – and scarier still, unknown unknowns!
• However, simply acknowledging risk ≠ active risk management.
• Since uncertainty is rife with risk, NASA requires a more flexible or agile approach to risk management to help decision makers adapt.
3
Evolving Risk Management
• Risk is a triplet = a scenario with a chance or probability (L) of experiencing a negative consequence (C).
− Ref. NASA Procedural Requirements (NPR) 8000.4A
• In recent years, NASA Programs, Projects, and Directorates have employed Continuous Risk Management (CRM) to mitigate risk across key performance domains:
− Cost (to achieve or regain required performance, cost to mitigate risk, cost of inaction)
− Schedule (Critical Path)
− Technical
− Safety (and Mission Assurance)
− Core Capabilities (Critical Assets)
4
Evolving Risk Management (cont.)
• Why the need for more flexibility?
− CRM presupposes a linear mission or a mission that has clearly defined end-objectives.
– Today, missions can be less linear (especially for human spaceflight) and end-objectives can be dynamic as policies change.
– In this environment, decision-makers must be flexible and even innovative to readily adapt to changes in mission objectives, funding, and resources.
– To keep pace, we need to be innovative, systemic thinkers in the way we view risk!
5
Evolving Risk Management (cont.)
• To remain relevant, risk management practices must evolve as NASA adapts/evolves into a 21st century agency:
– For spaceflight, adaptation means a suite of capabilities independent of discrete mission objectives; i.e., build a launch capability whether the ultimate destination is a near-Earth-object, the Moon, or Mars.
– As NASA redefines itself, the nature of risk management must shift as well:
• From: Identifying and mitigating risk(s) that prevent a performing organization from meeting a known requirement.
• To: identifying and mitigating performance shortfalls while helping decision-makers find the best solution to meet tactical and strategic objectives.
• RIDM can help to evolve Risk Management practices to meet this challenge!
– NPR 8000.4A uses RIDM as a pre-quel to CRM.
– This can help frame the risk discussion from a systemic perspective.
6
Destination
Mars
LunarNEO
Policy
Direction
Mission
Objective
sC
on
verg
en
ce
7
Risk Management adds RIDM to CRM in an integrated framework to:
• Foster proactive management of threats to performance
• Better inform decision-making using relevant (timely) risk information
• More effectively manage risks by focusing CRM on existing baseline performance requirements and on those emerging via the RIDM process
• Through a graded approach, adapt to varying levels of risk over time.
Evolving Risk Management (cont.)
Risk Management – Avoiding Undesirable Consequences!
Interface is 50% right!
When should decision makers assess for risk outcomes?
Hey, we followed
the spec, I better get
paid
Not my problem! Too funny!
Huh?
RIDM is Scary
• RIDM terminology can seem daunting – not to worry, we will translate the key terms!
• A key component of RIDM per NPR 8000.4A is advocacy for quantitative risk analyses to add rigor to decision processes and trades for:
– Complex, multi-scenario projects– Projects with competing stakeholders– (fill-in your case).
• However desirable, quantitative analyses are not always feasible. Can be costly and may not be feasible for activities that are smaller, lower dollar, or in the operational phase.
– Graded approach allows you to match the rigor of analysis commensurate with the criticality and type of decision to be made.
• This presentation discusses RIDM in practical ways and outlines a RIDM case study to illustrate the how’s & why’s – let’s get started!
8
NOT
Consequence
To develop or revise:
RequirementsCapabilitiesOperations Processes
?
?
Generic Risk Model
9
Risk Event
Controls Controls/Response
Activity
Preventive Measures Reduce Severity
Activity?
RIDM can help to clarify objectives:
● Activities and Products? (“Shall requirements”)
● Support functions? (Insight/Oversight)
Consequence
Consequence
To meet:
Performance objectives, over time
(Based on Risk Scenarios, Requirements, Plans, Historical & Current Performance, Resources…)
Mitigate or Accept
Risk Model with RIDM (NPR 8000.4A)
10
Got Risk?
11
RIDM Process
• ID Scenario
• ID Stakeholders
• Performance Objectives
• Performance Measures
• Confirm Constraints
• Analyze
• Down Select
• Risk Inform Decision
Do you know where your alligators are today?
Key RIDM Terms
12
#Key Pt
Term "RIDM-Speak"RIDM
Handbook reference
Translation
1 √ Graded Approach
Determine depth of analysis needed to reduce uncertainty. Not all risk analysis can/should be quantitative. Use qualitative analysis, as appropriate.
Section 3.2.1.3,
p. 51
Rigor: Match level of analysis to significance of decision, project scale, likelihood, and/or level of impact.
2 √Performance Objective
Derived via an objectives hierarchy; objectives at each level of an organization.
General term
Mandated or delegated goals that organizations must meet at each level.
3 √Performance Measure
Metric to capture the extent to which a system, process, or activity meets a performance objective. Same measure can be used for all alternatives.
Generic term Figures of merit (FOM)
4 √Performance Commitment
Alternative-independent level of performance: Using accepted measures, compare across alternatives, and consider the risk tolerance of the decision-maker.
Sec 1.5, Part 3, pp.
19-22
Match performance objective to risk tolerance of decision-maker (to compare across alternatives at same level of confidence).
5 √Technical Basis for Deliberation (TBfD)
Evolving document of ground rules and assumptions used to structure a risk-informed decision. Specifies information needed to characterize uncertainty for each decision alternative.
p.17; App.D: Content
Guide, pp. 108-109
Basis of Estimate (BOE) that evolves with each iteration of risk analysis
Ref: NASA Risk Management Handbook NHBK_2011_3422
Iterative Risk Decision Process
13
Mission:● Technical Objectives● Tactical Operations● Schedule / Discrete Milestones● Logistics● Strategic Objectives
Systems Engineering Handbook
Mission Support:● Capability● Capacity / Infrastructure● Institutional Goals● Supply Chain● Strategic Objectives
RECAP
• Evolving Risk Management = RIDM + CRM
− Uncertainty and evolving Agency direction elevate the need for responsive Risk Management.
− CRM is still a successful tool in the Risk Management tool box.
− RIDM complements CRM by enhancing the risk analyses in a graded approach commensurate with decisions to be made.
− RIDM and CRM used together throughout your iterative decision process can ensure timely risk information.
• Risk-informed decisions = an adaptive process:
− Factoring timely risk information into key decisions adaptation to dynamic environments.
− CRM alone enables response to a potential risk.
− Adding RIDM iteratively can help with early insight into individual risks as well as overall performance risks over time = flexibility.
14
15
Perspective: RIDM-to-CRM
“Two Sides of the Same Coin”
• Start with a Performance Objective
• Frame the decision space (graded approach)
• Use CRM to analyze risk of each alternative
• Down-select alternatives
• Track/monitor via CRM
CRM-Centric
• Start with a Performance Shortfall (Risk)
• Use mini-RIDMs to evaluate/make handling decisions
• Assess qualitatively or quantitatively
• Develop/ implement mitigation
RIDM-Centric
Systems Engineering Handbook
16
Deciphering RIDM
1b. Derive Performance Measures
1a. Identify Performance Objectives
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
6b. Select an Alternative & Document
2. IdentifyFeasible Alternatives
4. Analyze Alternatives & Document
6a. Deliberate & Decide
Map of KSC Facilities and Launch Pads
Illustrated Example*
17*Loosely based on an actual risk scenario
CARGO HAZARDOUSSERVICING FACILITY
• Decision: 1. Select best location for Short Stack Hypergolic Servicing.
2. Base decision on performance objectives, performance measures, stakeholder considerations, and domain risks of each feasible alternative.
• Alternatives:1. Existing facilities (e.g., Pad)
2. New facility
3. Other?
1b. Derive Performance Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
Step 1a. Refine the Decision
• Concise statement of a performance shortfall.
• Establish common understanding of performance constraints.
• Document assumptions and scope of decision.
18
Scenario for Off-line Servicing & Assembly
etc.
1b. Criteria for Success
• Identify all Stakeholders and their expectations.
• Define Performance Objectives.
• Characterize Performance Measures.
19
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
Program Ground Ops
HQ KSC
EPA Public
?
“Show me the money!” is a cool phrase. For RM, it’s not just about the $$, we must consider all domains!
2. Feasible Alternatives
• Initially consider a range of solutions – even wild-eyed ideas okay at this point!
• Always include the “Do-Nothing” option.
• Use Graded Approach to determine appropriate level of rigor needed to refine your list of feasible alternatives.
20
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
MPPF (existing facility)
VAB (existing facility)
Pad (existing process)
Vehicle Integration Building (new facility)
Stop-n-Go Model (new alternative)
3. Set the Framework
• Select “Key” Decision Parameters.– Note: Parameters must apply to ALL alternatives.
• Determine reasonable Performance Measures and criteria needed for a decision-maker to commit to an alternative (minimum threshold).
• Establish weighted priority (basis) for making a decision.
21
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
Launch Availability
Life Cycle Cost
Contingency & Recovery
Safety
“Green Pad”
Flexibility Veh. Orientation
4. Analyze Alternatives
• Conduct analysis of each alternative’s key performance measures.
• Prune “unsuitable” alternatives expeditiously based on key parameters:– Reduces analysis cost / effort.
– Clarifies options for decision makers.
22
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
MPPF (existing facility)
VAB (existing facility)
Pad (existing process)
Vehicle Integration Building (new facility)
Stop-n-Go Model (new alternative)
Additional RIDM Terminology for Step 5
23
#Key Pt
Term "RIDM-Speak"RIDM
Handbook reference
Translation
6 √Risk Normalized Performance Commitment
Create means to compare “like” perspectives across alternatives by first establishing a “common break-point of uncertainty” for each analysis.
Section 3.3.1,
pp.73-78
Normalize performance to assess across a range of options
• For any measure, allowable uncertainty is based on a number of factors
For ex. Worst-case threshold
• Set standards used to compare alternatives at the same level of uncertainty; allocated resources represent some point value in an uncertainty range (.01 – .99) for each alternative.
Avoid co-mingling analysis uncertainty with the uncertainty of an alternative; each option produces its own PDF and CDF.
• A common “confidence” level would be used to arrive at a “normalized” value of uncertainty for each alternative Apply the same confidence level (i.e.,
risk tolerance) to each alternative for a criterion (criteria can have different risk tolerance levels)
• Additional uncertainty considerations depend on risk tolerance of decision-makers. If risk-neutral, an option with uncertain performance may be preferred if it performs at the “mean value” of a performance measure pdf.
Risk-neutral not likely today!
Ref: NASA Risk Management Handbook NHBK_2011_3422
• Consider risk tolerance of the decision-maker and all key stakeholders for each performance measure.
• Determine degree of certainty required for each performance measure for comparison.
• Each performance measure should yield a probability distribution function (pdf) for each alternative.
Step 5. Risk Normalize Commitments
24
1b. Derive Performance Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
Note:Pdf – Cdf applies for quantitative analyses; similarly for qualitative
assessments, one should apply the same level of confidence when evaluating alternatives (constructed scales).
Risk Normalized Commitments
• Each alternative yields its own PDF and corresponding cumulative distribution function (CDF).
25
NOTES:
1. A common “uncertainty” level would be used to arrive at a “normalized” value of uncertainty for each alternative.
2. Assumed 20% Risk Tolerance = 80% Certainty (use 80% value of CDF)
High- Level Objectives Performance Objectives Measures Pad VAB MPPF VIB S & G WeightMeets CARD Yes Yes Yes Yes Yes 0%Optimal: Exceeds L.A. reqmts No Yes Yes Yes No 7%Construction/ refurbishmt ($M) 2.5 5 4 20 6 8%Operations & Maintenance ($M) 4 2.5 4 1 1 10%Discovery 1 - Low Risk 3 1 1 1 2 6%Verification 2 - Mild Risk 2 1 1 1 1 8%Repair 3 - Moderate Risk 1 2 1 1 3 3%
4 - Potentially UnreliableOff-nominal (de-servicing) 5 - Unreliable 3 2 3 1 1 2%Minimize Pad Time***(Clean Pad) Quan (Hr) 106 1 1 1 1 11%De-couple critical path Quan (Hr) 96 12 24 12 12 2%Quantity Distance Impact Quan (hrs contact) 3 16 20 20 3 10%Hypergol Contamination Quan (H/L) 3.2 2.8 2.2 1.6 0.8 10%Adequate “clear” Y/N Yes Yes Yes Yes Yes 0%Minimize Wet Time Quan (Hr) 8 106 106 106 16 6%Scalability (flight rate) Flights/Yr 52 52 70 100 100 10%- Flexibility Re-Cycle Time (Hr) 48 48 12 12 0 7%
Tally 85 113 161 178 140 100%
Schedule
Launch Availability
Life Cycle Cost
Contingency Capabilities Flexibility and Recovery
Processing Effi ciency
Safety
Example of Analysis Documented on TBfD Matrix
26
Note: Conceptual only – data & values fabricated for illustrative purposes only.
Step 1 Step 1b Step 2 Step 3
Step 6a. Decide & Deliberate
• Apply analysis results to TBfD and make recommendation to Decision-Maker.
• Reconsider TBfD if no clear leader and report results to Decision-Maker.
• Reconsider requirements / constraints if no acceptable alternatives.
27
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Iden
tifi
cati
on
of
Alt
ern
ativ
es
5. Risk-Normalize Commitments
3. Set the Framework (TBfD)*
Ris
k A
na
lys
is o
f A
lter
nat
ives
6b. Select an Alternative &
Document
Ris
k-I
nfo
rmed
Alt
ern
ativ
e S
elec
tio
n
2. IdentifyFeasible Alternatives
4. Analyze Alternatives &
Document
6a. Deliberate & Decide
Analysis Findings (for example):− Recommend New Facility− Document “Green Pad” Requirement− Work to Minimize Wet Time
28
Recap: Risk Management = RIDM + CRM
1b. Derive Performance
Measures
1a. Identify Performance Objectives
Identification of Alternatives
5. Risk-Normalize
Commitments
3. Set the Framework
(TBfD)*
Risk Analysis of Alternatives
Build Performance Requirements
6b. Select an Alternative &
Document
Risk-Informed Alternative Selection
2. IdentifyFeasible
Alternatives
Iterative
RIDM: Inform decision-making through use of quantitative and qualitative risk info to establish baseline performance requirements for mission support organizations, programs and projects.
CRM: Manage risk associated with implementing baseline performance requirements using a systematic and iterative process to identify, analyze, plan, track, control, communicate and document risks.
4. Analyze Alternatives &
Document
6a. Deliberate &
Decide
Ref. RIDM Handbook Fig. 11. RIDM Process Steps (p.30)
TBfD:*Technical Basis for Deliberation; also known as (BOE) Basis of Estimate
Concluding Remarks
• Uncertainty, evolving objectives, scarce resources – require an evolution in the way we approach risk management.
• The key is coupling RIDM with CRM to evolve Risk Management practices to be more pro-active vs. simply reactive.
• Aim: Employ an integrative, systemic approach to yield relevant and timely information for missions and mission support.
• Doing so keeps NASA Risk Management practices fresh and relevant to enable risk-informed decisions.
• Go forth & do good RIDM (and CRM)!
29
Always consider mission support risks!
30
JSC\AC3 Sharon L. Thomas [email protected](281) 244-7668
JSC\NA Scott L. Field, [email protected](281) 335-2440