risk management process metrics - attwater consulting conf pres/risk management... · risk...

Risk Management Metrics

International Council on Systems Engineering

Risk Management Working Group Mark Powell, Chair

Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 2

Proper Attribution• A project in work by the INCOSE RMWG• Paper Authored by Barney B. Roberts and

Richard Kitterman• Originally Presented at INCOSE

International Symposium July 2005 in Rochester, NY

• Presenter: Mark Powell, RMWG Chair


The Problem with Risk Management

• A good risk management process results in nothing happening

• How does one measure process performance?• Multiple choice:

• If a project meets its performance goals, then …• A. It’s risk management process was successful• B. The project had a run of good luck• C. The project was under-constrained• D. All of the above

• If a project overruns its cost commitments, then …• A. Its risk management process failed• B. The project got a bad roll of the dice• C. The project was over constrained• D. All of the above


Risk Management Measurements?

• What do you measure?• How do you measure it?• How do we know what is a “good”

measurement, or a “bad” measurement?

• INCOSE Chartered RMWG to Investigate RM Metrics


Potential RM Metrics• Comparison Metrics

• Other standards• Using CMMs• Problem: Assumes that the others are “good”

• Return on investment Metrics• Cost of investment of risk management process

execution ratio-ed to the reduction of risk• Estimate the risk without a mitigation plan, then ratio to

estimated risk after planned mitigation, compare to • Problem: both are estimates, inadequate historical basis

• Efficiency and Effectiveness Metrics• Measure attributes of the process that indicate

efficiency and effectiveness• Problem: Has promise, but very limited experience


Potential Metrics Continued• Staleness Metrics

• Measure the lag in the flow of products through the risk management process – too long in one step of the process is “bad”

• Problem: Not specific to the quality of the process -- will also measure the lag in a bad process

• Trending Metrics• Measure the change in the number of risks in various

categories over time• Problem: Assumes that all risks are equal – one “very

bad” risk may overwhelm many other “bad” risks and give a false sense of security


Potential Metrics Concluded• Results of RMWG Investigation

• The best measure – actually compare estimates to outcomes

• Problem: Statistical in nature and requires sampling and analyses of many cases to develop models and relationships

So, what to do? A proposal follows.So, what to do? A proposal follows.


RMWG RM Metrics Proposal• RM Metrics Classified by Usage Frequency

• Infrequent Metrics• Usually before or after a project• When significant performance issues are noted• During the development of a Risk Management process

• Continuous Metrics• Measure the process during execution• Measure the quality of the products during execution• Attempt to make interim corrections if needed

• On-demand Metrics• When a measurable result is available, compare to

expectations• Ad Hoc or Periodic


Infrequent Metrics and Measurements

• Compliance to organization’s standards• Build a compliance matrix – extract “Shall” statements• Compare project’s process against the organization's

standardsProject

Compliant? “Shall” Statement Project RM Plan Paragraph

5 The identification of risk shall be actively encouraged at all levels in the Project Team Pg. 2, Section 1.6

4 Risk identification shall be accomplished by entering the information into the Kepler Risk Database through the Kepler Risk Tracking Tool.

Pg. 2, Section 1.6

All identified risks shall be reviewed by Project Management on a cyclical basis and accepted, assigned to a Risk Owner for action, rejected or retired.

Pg. 2, Section 1.6

Corrective Action?

Corrective Action?

A scale of 1 to 5 indicating level of compliance

A scale of 1 to 5 indicating level of compliance


Infrequent Metrics and Measurements

• Compliance to “Best Practices”• No consensus on what are “Best Practices”• Use CMMs as a substitute

FA 1.1 FA 1.2 FA 2.5Manage

RiskFA 3.3 FA 3.4

LEVEL 5

LEVEL 4

LEVEL 3

LEVEL 2

LEVEL 1

LEVEL 0

Quantitative effectiveness goals are establishedContinuous process improvement

Evidence that work is accomplished

Performance is planned and tracked

Standard process is definedPrograms tailor the standard process

Measurable goals are establishedPerformance is predicted

GENERICPRACTICES &ATTRIBUTES

FA- SPECIFICPRACTICES


Sample CMM Assessment Management Category

Leve

l 2 G

ener

ic P

ract

ices

are

per

form

ed

Res

ults

are

at l

east

of a

dequ

ate

utili

ty

Leve

l 3 G

ener

ic P

ract

ices

per

form

ed

Res

ults

are

of a

t lea

st s

igni

fican

t util

ity

Leve

l 4 G

ener

ic P

ract

ices

per

form

ed

Res

ults

are

at l

east

of m

easu

rabl

y si

gnifi

cant

util

ity

Leve

l 5 G

ener

ic P

ract

ices

per

form

ed

Res

ults

are

of o

ptim

um u

tility

Cap

abili

ty M

atur

ity L

evel

BY

Ass

essm

ent T

ool

Project 01A ### # ### 0Project 01B 2Project 02 2Project 03 2Project 04 2Project 05 1Project 06 2Project 07 2Project 08 2

Level 1 Level 2 Level 3 Level 4 Level 5

Leve

l 4 S

peci

fic P

ract

ices

are

per

form

ed

Leve

l 5 S

peci

fic P

ract

ices

are

per

form

ed

Res

ults

are

at l

east

of m

argi

nal u

tility

Leve

l 1 S

peci

fic P

ract

ices

are

per

form

ed

Leve

l 2 S

peci

fic P

ract

ices

are

per

form

ed

Leve

l 3 S

peci

fic P

ract

ices

are

per

form

ed

####

The CMM Questionnaire Assessment Tool showed projects interviewed as Level 2 or less

The CMM Questionnaire Assessment Tool showed projects interviewed as Level 2 or less


Continuous Metrics

• Compliance to Plan• Performance

• Effectiveness• Efficiency• Staleness

• Trending


Continuous Metrics: Compliance to Plan

• Compliance to RM Plan• Is the project actually doing what it said it

would do?• A simple compliance matrix• Shall statements from the RM Plan for the

project versus evidence that the activities are actually performed

• Determine corrective action


Continuous Metrics: Performance

• Performance – measure the performance of the process• Effectiveness

• Effective: No or very few unforeseen “problems” occur• Approach: How many “problems” occurred that were

never identified as risks• Efficiency

• Efficient: Catching risks early when it is more cost effective to mitigate them

• Approach: Measure the time between when a risk was identified and when it became a problem

• Staleness• How many risk products are “stuck” in a process step and

how long have they been there


The Effectiveness Metric• Performance – Effectiveness

• Premise: An effective risk management system will prevent unexpected problems

• PE, Process Effectiveness is the ratio of problems encountered, Np, that were not identified as risks, to the risks identified, Nr

PE = 1 – Np/(Np + Nr)• Measure of goodness, 90% good, 80% watch,

70% Action• Action, causal analysis and process

improvement


The Efficiency Metric• Performance – Efficiency

• Premise: An efficient risk management system is one in which the planning and mitigation of risks occurs well before they become problems

• For n realized risks, Pe, Process efficiency, is the average time lapse between all risks’ identification date, TID, and the time that it is realized, TR,

• Pe = Σ(TR,i – TID,i)/n, • Measure of goodness, 90% good, 80% watch, 70%

Action• Action, causal analysis and process improvement

i=1,n


The Staleness Metric• Performance – Staleness

• Residence time for risks in major steps• Short residence times: < ~30 days, are “Excellent”

long residence times: ~180 are “Very Poor”• Measure of goodness: 90% good, 80% watch, 70%

Action• Action: directed project management attention to

insure actions• Example measures:

• First Latency: Time identified to time first action by project management

• Second Latency: Time from assignment to a Risk Owner to time the project “Accepts” the risk mitigation plan

• Subsequent Latencies: Lateness tracked against dates on the steps in the risk mitigation plan


Sample Latency Measurement

IDRisk

Coordinator Evaluates

Project Mgr Decision

Risk Owner Develops Mitigation

Risk Coordinator Evaluates

Project Mgr Decision

Track Progress on mitigation

steps

Retire

Retire

Reject

Accept W/O Mitigation

Accept, Mitigation Required

Time, tt = 0

First Latency Calculation, t1, Residency in “Active” Status, Time from ID to first decision by project

t = t1

Second Latency Calculation, t2, Residency in Mitigation Planning

t = t2

Third through n Latency Calculations, tn, “Lateness” on mitigation steps

30

20

10

030 60 90 120 180

Latency, Days

Num

ber

t1 t2Mitigation Steps


Trending MetricsDevelopment Risks

12

5 4 3 2 1 0 0

14

16 16 14 1413 13

10 8

16 8 10 10

10 1014 16

40

5

10

15

20

25

30

PDR

09/2

4/98

10/2

0/98

11/2

5/98

01/1

4/99

03/0

9/99

05/2

1/99

06/0

3/99

08/1

7/99

Time

Num

ber o

f Ris

ks

# Low

# Medium

# High

Risk Mix by Level

44

19 14 14 11 80

52

5957

50 5252

54

42

4

2229

36 37 40

58

4

42

0

10

20

30

40

50

60

70

80

90

100

PDR

09/2

4/98

10/2

0/98

11/2

5/98

01/1

4/99

03/0

9/99

05/2

1/99

06/0

3/99

Time

Perc

ent

% Low

% Medium

% High

• Body Count versus time• Measure change• Goodness is more

vague on this one• No change is “bad”• Increasing risk numbers

may be “bad”• A decreasing trend in the

red and yellow is good• Action, directed project

management attention to insure actions


More Trending Metrics• Waterfall charts• Measure

• Latency (Calculated elsewhere)• Errors in prediction of impact or effectiveness of mitigation

• Goodness: Miss residual risk by “half a color”• Action: directed project management attention to correct

Ris

k Le

vel

Time

Mitigation Plan Mitigation Effectiveness

AMEs Planned

AMEs Actual


Example: Consumption of Reserve

• Comparison of Expected Value of risks to available reserve

Risk Item (Or WBS)

Risk Lien in $K

1 35$ 2 155$ 3 76$ 4 265$ 5 46$ 6 52$ 7 354$ 8 66$ 9 78$ 10 465$ 11 321$

SUM 1,913$ RSS 749$

Current Reporting Period

Consumption Of Reserve

$-

$500

$1,000

$1,500

$2,000

$2,500

1 2 3 4 5 6 7 8 9 10 11Reporting Period

Res

erve

s

Total Risk Soft LiensAvailable Reserve


On-Demand Metrics: Results

• Based on risks that have been accepted by the project either with or without mitigation

• Should those risks be realized, the impact is measured

• Compare the measured impact with the predicted impact

• Action, causal analysis and process improvement

Distribution for Program/Cost

Values in Billions

0.000

0.200

0.400

0.600

0.800

1.000

4.28 4.325 4.37 4.415 4.46

4.34.3

4.28 4.325 4.37 4.415 4.46

70% 30% 4.3651

Actual


Summary of Metrics• Infrequent

• Measure the quality of the process• Compliance to the organization’s standards• Comparison with Best Practices

• Continuous• Compliance to plan• Performance

• Effectiveness• Efficiency• Staleness

• Trending• On-demand

• Results - for “accepted” risks that are realized, compare actual risk outcomes with predictions.


INCOSE RMWG Recommendation

• A Proper Combination of Metrics should be Selected• Individual Metrics are Not Sufficient• Should Include Each Frequency Type

• RM Process Performance is Measurable and Hence Manageable

• RM Process Performance Can be Justified

risk management process metrics - attwater consulting conf pres/risk management... · risk...

Documents