using simulation to manage software delivery risk

Using Simulation to Manage Software Delivery Risk

Effective Modeling and Simulating Kanban and Scrum Projects using

Monte Carlo Techniques

Troy Magennis [email protected]

@AgileSimulation

mailto:[email protected]

Are we meeting our commitments?

Schedule Risk

What team skillset additions (or losses) have the biggest

impact?

Staff Risk

What are the top three risks jeopardizing delivery?

Risks

Arm my teams (and yours) with the tools and techniques

to answer these questions

My Mission

And manage risk more effectively

• Currently: Founder and CTO Focused Objective • Previously: Vice President of Technology (Arch)

• Travelocity and Lastminute.com • Director Architecture, Corbis • Various: Automotive, Banking

Contact: @AgileSimulation and @t_magennis FocusedObjective.com [email protected]

DEFINITIONS, HISTORY AND USE What, when, who, why

A model is a tool used to mimic a real world process

A tool for low-cost experimentation

Definition: Model

A technique of using a model to determine a result given a

set of input conditions

Definition: Simulation

Performing a simulation of a model multiple times using

random input conditions and recording the frequency of

each result occurrence

Monte Carlo Simulation

DEMO: VISUAL MODEL SIMULATION DEMO: MONTE-CARLO SIMULATION

Simple to more complex model and simulation of a software project

In case of demo disaster, press here…

History

Stan Ulam Holding the FERMIAC

Credits: Wikipedia

http://en.wikipedia.org/wiki/FERMIAC

When there is no correct single answer (knowable in

advance) or when the time/effort taken to compute an answer is beyond realistic

When to use Monte Carlo Simulation

When a range of input conditions can MASSIVELY

alter the final outcome

When to use Monte Carlo Simulation

High risk industries Natural resource exploration, insurance,

finance, banking, pharmaceutical…

Who Uses Monte Carlo Simulation

Software Development == High Risk! Just look at our reputation, and on-time, on-budget success rate…

APPLYING MONTE CARLO METHODS TO SOFTWARE DEV

Why? To Answer Tough Questions…

Date and cost forecasts Impact of staff hire/loss

Cost of defects Cost of blocking events

… And my three 1:1 questions each week!

But doesn’t it require estimates?

Yes, but very few…

MUST: Estimate major risks

SHOULD: Column cycle-times and story counts

We need to estimate risk events

We spend all our time estimating here

1 2 3

**Major risk events have the predominate role in deciding where deliver actually occurs **

Is it Accurate?

1. Gin still equals Gout

2. Doesn’t suffer from the “Flaw of Averages”

Flaw of Averages

50% Possible Outcomes


Freq

ue

ncy

of

Res

ult

1

5

10

15

20

25

Major issue: Race condition, third party component failure…

The average release date!!!

Major Risk Event Shifts Delivery Shape Right

Developer Estimates

We need to estimate risk events

See model example…

We spend all our time estimating here

1 2 3

**Major risk events have the predominate role in deciding where deliver actually occurs **

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval

DEMO: FORECASTING (DATES & COST) DEMO: SENSITIVITY (COST OF DEFECTS) DEMO: STAFF IMPACT (STAFF RISK) In case of demo disaster or no internet, press here…

BEST PRACTICES AND TIPS

Model (a little)

Visually Test

Monte-Carlo Test

Sensitivity Test

The Model Creation

Cycle

Baseline

Make Single

Change

Compare Results

Make Informed

Decision(s)

The Experiment

Cycle

Best Practice 1

Start simple and add ONE input condition at a time.

Visually / Monte-carlo test

each input to verify it works

Best Practice 2

Find the likelihood of major events and estimate delay E.g. vendor dependencies,

performance/memory issues, third party component

failures.

Best Practice 3

Only obtain and add detailed estimates and opinion to a

model if Sensitivity Analysis says that input is material

Best Practice 4

Use a uniform random input distribution UNTIL sensitivity

analysis says that input is influencing the output

Best Practice 5

Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or

95th confidence level (average is NEVER an option)

Q1. Are we meeting our commitments?

Q2. What are the top three risks jeopardizing on-time delivery?

Q3. What skillsets do your next three hires need to have?

Is the likelihood of the models forecast date increasing or decreasing?

Top three items in the Sensitivity report

Skills applicable to the top three WIP limit increases that cause the biggest reduction in forecast

Call to action

• Read these books

• Download the software FocusedObjective.com

• Follow @AgileSimulation

• Learn: http://strategicdecisions.stanford.edu/

http://www.focusedobjective.com/

http://strategicdecisions.stanford.edu/

http://strategicdecisions.stanford.edu/

Questions?

My Contact Details and to get these slides, the software or the book used in this session -

FocusedObjective.com

Me: [email protected]

Follow: @AgileSimulation and @t_magennis

http://www.focusedobjective.com/

mailto:[email protected]

BASICS OF MODELING AND SIMULATION

Return to main presentation…

Manual Kanban Model & Simulation

Backlog Design Develop Test

Deployed

2

1 – 2 days 1 – 2 days 1 – 2 days

1

2 3 4

PLUS: For this manual example, at least 1 defect, blocking event and scope-creep item.

5

Day 1


Deployed

2

1 – 2 days 1 – 5 days 1 – 2 days

1 Day picked at random for this columns cycle-

time range

Day 2


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Day 3


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Day 4


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 5


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 6


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 7


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 8


Deployed

2 1 day

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 9


Deployed

2

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 10


Deployed

2

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Day 11


Deployed

2

1 – 2 days 1 – 5 days 1 – 2 days

Added Scope

Result versus Frequency (50 runs)

More Often

Less Often Result Values – For example, Days

15 10 20

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Result versus Frequency (250 runs)

More Often


15 10 20

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Result versus Frequency (1000+ runs)

More Often


15 10 60

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Central Limit Theorum


Flaw of Averages




Software Development Model

Model Columns & WIP

Work

Defects

Blocking Events Added

Work

Staff Vacations

…

1

2

3 4

5

6

7


SIMULATION EXAMPLES


unlikely

certain

Forecasts Return to main presentation…

unlikely

certain

Forecasts

50% Possible

Outcomes



Actively Manage

Ignore for the moment

Sensitivity Report Return to main presentation…

Staff Skill Impact Report

Explore what staff changes have the greatest impact


MULTI-MODAL RESULT MODEL

<setup> <backlog type="custom" > <deliverable name=“work"> <custom count="10">Build website</custom> </deliverable> <deliverable name="performance issues, add caching" skipPercentage="50"> <custom count="10" >Rework: Performance Issues</custom> </deliverable> </backlog> <columns> <column id="1" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Develop</column> <column id="2" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Test</column> </columns> </setup>


using simulation to manage software delivery risk

Business