using simulation to manage software delivery risk
Post on 10-May-2015
685 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using Simulation to Manage Software Delivery Risk
Effective Modeling and Simulating Kanban and Scrum Projects using
Monte Carlo Techniques
Troy Magennis Troy.magennis@focusedobjective.com
@AgileSimulation
Are we meeting our commitments?
Schedule Risk
What team skillset additions (or losses) have the biggest
impact?
Staff Risk
What are the top three risks jeopardizing delivery?
Risks
Arm my teams (and yours) with the tools and techniques
to answer these questions
My Mission
And manage risk more effectively
• Currently: Founder and CTO Focused Objective • Previously: Vice President of Technology (Arch)
• Travelocity and Lastminute.com • Director Architecture, Corbis • Various: Automotive, Banking
Contact: @AgileSimulation and @t_magennis FocusedObjective.com Troy.Magennis@focusedobjective.com
DEFINITIONS, HISTORY AND USE What, when, who, why
A model is a tool used to mimic a real world process
A tool for low-cost experimentation
Definition: Model
A technique of using a model to determine a result given a
set of input conditions
Definition: Simulation
Performing a simulation of a model multiple times using
random input conditions and recording the frequency of
each result occurrence
Monte Carlo Simulation
DEMO: VISUAL MODEL SIMULATION DEMO: MONTE-CARLO SIMULATION
Simple to more complex model and simulation of a software project
In case of demo disaster, press here…
When there is no correct single answer (knowable in
advance) or when the time/effort taken to compute an answer is beyond realistic
When to use Monte Carlo Simulation
When a range of input conditions can MASSIVELY
alter the final outcome
When to use Monte Carlo Simulation
High risk industries Natural resource exploration, insurance,
finance, banking, pharmaceutical…
Who Uses Monte Carlo Simulation
Software Development == High Risk! Just look at our reputation, and on-time, on-budget success rate…
APPLYING MONTE CARLO METHODS TO SOFTWARE DEV
Why? To Answer Tough Questions…
Date and cost forecasts Impact of staff hire/loss
Cost of defects Cost of blocking events
… And my three 1:1 questions each week!
But doesn’t it require estimates?
Yes, but very few…
MUST: Estimate major risks
SHOULD: Column cycle-times and story counts
We need to estimate risk events
We spend all our time estimating here
1 2 3
**Major risk events have the predominate role in deciding where deliver actually occurs **
Is it Accurate?
1. Gin still equals Gout
2. Doesn’t suffer from the “Flaw of Averages”
Flaw of Averages
50% Possible Outcomes
50% Possible Outcomes
Freq
ue
ncy
of
Res
ult
1
5
10
15
20
25
Major issue: Race condition, third party component failure…
The average release date!!!
Major Risk Event Shifts Delivery Shape Right
Developer Estimates
We need to estimate risk events
See model example…
We spend all our time estimating here
1 2 3
**Major risk events have the predominate role in deciding where deliver actually occurs **
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
DEMO: FORECASTING (DATES & COST) DEMO: SENSITIVITY (COST OF DEFECTS) DEMO: STAFF IMPACT (STAFF RISK) In case of demo disaster or no internet, press here…
BEST PRACTICES AND TIPS
Model (a little)
Visually Test
Monte-Carlo Test
Sensitivity Test
The Model Creation
Cycle
Baseline
Make Single
Change
Compare Results
Make Informed
Decision(s)
The Experiment
Cycle
Best Practice 1
Start simple and add ONE input condition at a time.
Visually / Monte-carlo test
each input to verify it works
Best Practice 2
Find the likelihood of major events and estimate delay E.g. vendor dependencies,
performance/memory issues, third party component
failures.
Best Practice 3
Only obtain and add detailed estimates and opinion to a
model if Sensitivity Analysis says that input is material
Best Practice 4
Use a uniform random input distribution UNTIL sensitivity
analysis says that input is influencing the output
Best Practice 5
Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or
95th confidence level (average is NEVER an option)
Q1. Are we meeting our commitments?
Q2. What are the top three risks jeopardizing on-time delivery?
Q3. What skillsets do your next three hires need to have?
Is the likelihood of the models forecast date increasing or decreasing?
Top three items in the Sensitivity report
Skills applicable to the top three WIP limit increases that cause the biggest reduction in forecast
Call to action
• Read these books
• Download the software FocusedObjective.com
• Follow @AgileSimulation
• Learn: http://strategicdecisions.stanford.edu/
Questions?
My Contact Details and to get these slides, the software or the book used in this session -
FocusedObjective.com
Me: Troy.magennis@FocusedObjective.com
Follow: @AgileSimulation and @t_magennis
BASICS OF MODELING AND SIMULATION
Return to main presentation…
Manual Kanban Model & Simulation
Backlog Design Develop Test
Deployed
2
1 – 2 days 1 – 2 days 1 – 2 days
1
2 3 4
PLUS: For this manual example, at least 1 defect, blocking event and scope-creep item.
5
Day 1
Backlog Design Develop Test
Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
1 Day picked at random for this columns cycle-
time range
Day 2
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Day 3
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Day 4
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 5
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 6
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 7
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 8
Backlog Design Develop Test
Deployed
2 1 day
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 9
Backlog Design Develop Test
Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 10
Backlog Design Develop Test
Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Day 11
Backlog Design Develop Test
Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
Added Scope
Result versus Frequency (50 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Result versus Frequency (250 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Result versus Frequency (1000+ runs)
More Often
Less Often Result Values – For example, Days
15 10 60
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
Central Limit Theorum
Return to main presentation…
Flaw of Averages
50% Possible Outcomes
50% Possible Outcomes
Return to main presentation…
Software Development Model
Model Columns & WIP
Work
Defects
Blocking Events Added
Work
Staff Vacations
…
1
2
3 4
5
6
7
Return to main presentation…
Return to main presentation…
SIMULATION EXAMPLES
Return to main presentation…
unlikely
certain
Forecasts Return to main presentation…
unlikely
certain
Forecasts
50% Possible
Outcomes
50% Possible Outcomes
Return to main presentation…
Actively Manage
Ignore for the moment
Sensitivity Report Return to main presentation…
Staff Skill Impact Report
Explore what staff changes have the greatest impact
Return to main presentation…
Return to main presentation…
MULTI-MODAL RESULT MODEL
Return to main presentation…
<setup> <backlog type="custom" > <deliverable name=“work"> <custom count="10">Build website</custom> </deliverable> <deliverable name="performance issues, add caching" skipPercentage="50"> <custom count="10" >Rework: Performance Issues</custom> </deliverable> </backlog> <columns> <column id="1" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Develop</column> <column id="2" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Test</column> </columns> </setup>
Return to main presentation…
Return to main presentation…
top related