1

E S T I M A T E

Maximising Capital Investments:IS GUESSWORK ERODING YOUR BOTTOM LINE? E S T I M AT I O N B I A S A N D M I T I G AT I O N 2 0 1 7 E D I T I O N

Dan Galorath for Unico Enterprise Services

© 2016 Copyright Galorath Incorporated

Contact Details: Michael McKeon Unico Enterprise Services +61429054486 michael.mckeon@unico.com.au

E S T I M AT E • A N A LY Z E • P L A N • C O N T R O L

2

Key Points

Every forecast is subject to estimation bias… a major cause

of program failure and corporate misspending.

Experts Bias is probably costing - in cost,

schedule, and less than hoped for benefits.

When bias is managed better decisions

are made and more successes follow.

3

Your People Are Optimism Biased

HARVARD BUSINESS REVIEW EXPLAINS THIS NOBEL PRIZE WINNING PHENOMENON:

- Humans seem hardwired to be optimists - Routinely exaggerate benefits and discount costs

- Bias permeates opinions & decisions & causes waste & failure Delusions of Success: How Optimism Undermines Executives’ Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

Solution: Temper with “outside view”: past measurement Results, traditional forecasting, risk analysis and statistical parametrics can help. Don’t remove optimism, but balance optimism and realism.

4

IT Failure Can Impact Business Dramatically

CASE STUDY: LEVI STRAUSS

• $5M ERP deployment contracted• Risks seemed small• Difficulty interfacing with customer’s systems• Had to shut down production• Unable to fill orders for 3 weeks• $192.5M charge against earnings on a $5M IT project failure

“IT projects touch so many aspects of organization they pose a new singular risk”

5

Cognitive Bias: tendency to make systematic decisions based on PERCEPTIONS

rather than evidence.

“Perception has more to do with our desires—with how we want to view ourselves—

than with reality.” Behavioral Economist Dan Ariely

Researchers theorize in the past, biases helped survival.

Our brains using shortcuts (heuristics) that sometimes

provide irrational conclusions.

Bias affects everything:

- From deciding how to handle our money

- To relating to other people - How we form memories

Bias Guesses & Forecasts SOURCE: BEINGHUMAN.ORG

6

The Planning Fallacy (KAHNEMAN & TVERSKY, 1979)

• Manifesting bias rather than confusion

• Judgement errors made by experts and laypeople alike

• Errors continue when estimators are aware of their nature

JUDGEMENT ERRORS ARE SYSTEMATIC AND PREDICTABLE, NOT RANDOM

OPTIMISTIC DUE TO OVERCONFIDENCE IGNORING UNCERTAINTY

ROOT CAUSE: EACH NEW VENTURE VIEWED AS UNIQUE

• Underestimate costs, schedule, risks

• Overestimate benefits of the same actions

• “inside view” focusing on components rather than outcomes of similar completed actions

• FACT: Typically, a past more similar assumed

• Even ventures may appear entirely different

7

Bias Mitigation Reference Class Forecasting PREDICTS OUTCOME OF PLANNED ACTION BASED ON ACTUAL OUTCOMES IN A REFERENCE CLASS: SIMILAR ACTIONS TO THAT BEING FORECAST.

Attempt to force the outside view and

eliminate optimism and misrepresentation

Choose relevant “reference class”

completed analogous projects

Compute probability distribution

Compare range of new projects to completed

projects

Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize Economics 2002)

8

Bias Mitigation Reference Class Forecasting PREDICTS OUTCOME OF PLANNED ACTION BASED ON ACTUAL OUTCOMES IN A REFERENCE CLASS: SIMILAR ACTIONS TO THAT BEING FORECAST.

Attempt to force the outside view and

eliminate optimism and misrepresentation

Choose relevant “reference class”

completed analogous projects

Compute probability distribution

Compare range of new projects to completed

projects

Provide an “outside view” focus

on outcomes of analogous projects

Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize Economics 2002)

9

E X A M P L E :

Reference Class Forecasting With SEER Estimate

10

E X P E R I M E N T :

Anchoring Biases EstimatesSOURCE: MYWEB.LIU.EDU/~UROY/ECO23PSY23/PPT/04-ANCHORING.PPTX

Result:Those who got higher numbers on the wheel of fortune guessed bigger numbers in Step 3

123

Subject witnesses the number that comes up when a wheel of fortune is spun

Is asked whether the number of African countries in the U.N. is greater than or less than the number on the wheel of fortune

Is asked to guess the number of African countries in the U.N.

If given a number, that biases estimates

11

Flaw of Averages Case Studies

(SOURCE: HBR)

• U.S. Weather Service forecast that North Dakota’s rising Red River would crest at 49 feet.

• Made flood management plans based on this average figure

• In fact, the river crested above 50 feet, breaching the dikes, and unleashing a flood that forced 50,000 people from their homes.

EXAMPLE: $2 BILLION PROPERTY DAMAGE IN NORTH DAKOTA

12

M C K I N S E Y - O X F O R D

2013 Study on IT Program Performance

13

A G I L E :

Detailed Software Development Life Cycle Management

(SCRUM EXAMPLE)

- Focus is on what features can be delivered per iteration- Not fully defined what functionality will be delivered at the end?

- Iterations are often called a “Sprint”

14

Agile Is Not a Silver BulletSOURCE: HTTP://WWW.JAMASOFTWARE.COM/BLOG/RETHINK-AGILE-MANIFESTO-PROJECTS-STILL-FAIL/

- Projects still fail at roughly the same rate as 2001- Dr. Dobbs: Agile is not a productivity revolution

0%

20%

40%

60%

80%

Agile (72%) Traditional (64%)8% Improvementnot Revolution

Agile 8% More Success

15

Software Project Performance Still Disappointing

(McKinsey)

-80%

-60%

-40%

-20%

0%

20%

40%

60%

45% Over Budget 56% less Functionality thanplanned

McKinsey Study: Projects over $15m

So does skipping estimating solve this? Well: If commitments aren’t made there can be no disappointment

16

Estimation & Planning A COMPLETE ESTIMATE DEFINED

An estimate is the most knowledgeable statement you can make at a particular point in time regarding:• Effort / Cost• Schedule• Staffing• Risk• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A DISTRIBUTION

17

If A Promise or Hope Is Good Enough You Don’t Need Viable Estimates

NO NEED TO ESTIMATE IF:

• A promise of 5 sprints with 4 people is good enough• You are willing to spend whatever it costs in whatever time it takes• Estimates don’t impact planning or decision processes• You don’t need to know the probability that it will be complete or when • You aren’t concerned if it overruns substantially• You aren’t concerned with failure and contingency• You don’t need to consider total ownership costs• You are ok if this is an overestimate and resources are not optimized• There will be no system testing on top of development• If lack of documentation is ok for maintenance

18

Agile Trying To Kill Estimates Without Considering Business

Needs For Estimates #NOESTIMATES

19

#noestimates is Viable For Detailed DevelopmentBUT SHOULD NOT ABDICATE BUSINESS ISSUES FOR SUBSTANTIAL DEVELOPMENTS

Business Case Evaluation of alternatives

Agile or Hybrid Agile Software Development

System Test (when appropriate)

Maintenance and Support

FOR SUBSTANTIAL SYSTEMS

Hybrid Agile: Requirements & Design

How Much? How Long? Ownership Cost Go / No Go

• Agile development = root level software development management…• Story point estimating is short term productivity management• It is not a business decision making process

20

Mitigate Planning Fallacy & Bias

Additional Historical Data

Trend line

Trend line +1 sigma

Trend line -1 sigma

Matches Platform/Application

Recently added

Used For Calibration

Current Estimate

1

10

100

1,000

10,000

100,000

1,000,000

Dev

elop

men

t Effo

rt H

ours

(Log

)1 10 100 1,000 10,000100,0001,000,00010,000,000

Effective Size (Log)

Development Effort Hours vs Effective Size

Additional Historical Data

Trend line

Trend line +1 sigma

Trend line -1 sigma

Matches Platform/Application

Recently added

Used For Calibration

Current Estimate

1

10

100

1,000

10,000

100,000

1,000,000

Dev

elop

men

t Effo

rt H

ours

(Log

)1 10 100 1,000 10,000100,0001,000,00010,000,000

Effective Size (Log)

Development Effort Hours vs Effective SizeEstimate Agile Projects With SEERTO UNDERSTAND WHAT COST / SCHEDULE TO EXPECT

21

STEP

1

2

3

4

5

6

7

8

9

10

EXPECTED

30

50

80

50

90

25

35

45

70

25

500

Adding Reality to EstimatesEXAMPLE – 2 (SOURCE SEI)

22

STEP

1

2

3

4

5

6

7

8

9

10

BEST

27

45

72

45

81

23

32

41

63

23

EXPECTED

30

50

80

50

90

25

35

45

70

25

500

WORST

75

125

200

125

225

63

88

113

175

63

Adding Reality to EstimatesEXAMPLE – 2 (SOURCE SEI)

What would you forecast the schedule duration to be now?

23

Monte Carlo Analysis TO QUANTIFY AND MITIGATE BIAS

90% confidence, project under

817 days

50% confidence, project will be under 731 days

Almost guaranteed to miss the 500 days

24

SEER: Risk Driven Estimates THE ENGINE FOR PROJECT EVALUATION

• SEER predicts outcomes

• SEER uses inputs to develop probability distributions

• The result is a probabilistic estimate

• SEER will predict a likely range of outcomes

• Monte Carlo provides project-level assessments of risk

Least, likely, and most inputs provide a range of cost and schedule outcomes

Confidence (probability) can be

set and displayed for any estimated item

25

Key Points

Every forecast is subject to estimation bias…a

major cause of program failure and corporate

mis-spending

Experts Bias is probably costing in cost, schedule,

and less than hoped for benefits

When bias is managed better decisions

are made and more successes follow

26

• FLYVBJERG, BENT, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, European Planning Studies Vol 16 No 1, Jan 2008,

• Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of Decision under Risk.” Econometrica 47, no. 2, March 1979.

• Johnson, H. Thomas. Relevance Regained. Free Press.

• Mislick, Gregory K.; Nussbaum, Daniel A.. Cost Estimation: Methods and Tools (Wiley Series in Operations Research and Management Science) (p. 143). Wiley.

• Rose, Todd. The End of Average: How We Succeed in a World That Values Sameness, HarperCollins.

• Taleb, Nassim Nicholas. Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (Incerto). Random House Publishing Group.

Bibliography (Partial)• Glen, Paul; McManus, Maria. The Geek Leader’s Handbook:

Essential Leadership Insight for People with Technical Backgrounds. Leading Geeks Press.

• Kogon, Kory; Merrill, Adam; Rinne, Leena. The 5 Choices: The Path to Extraordinary Productivity. Simon & Schuster.

• Patterson. Crucial Conversations Tools for Talking When Stakes Are High, Second Edition. McGraw-Hill.

• Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux.

• Hubbard, Douglas W.. The Failure of Risk Management: Why It’s Broken and How to Fix It . Wiley.

• Hubbard, Douglas W.. How to Measure Anything: Finding the Value of Intangibles in Business .

• Galorath, Evans, Software Sizing, Estimation, and Risk Management: When Performance is Measured Performance Improves, Auerbach Publications, 2007