lkce16 - estimation made easy by pawel brodzinski and tomek rusilko
TRANSCRIPT
ESTIMATION MADE EASYSTATISTICAL FORECASTING
Paweł Brodziński
@pawelbrodzinski
Tomek Rusiłko
@rusilko
EXPERT GUESS
IF A PROJECT HAS NO RISKS, DON'T DO IT.
Tom DeMarco & Timothy Lister
DON’T DO IT
T. DEMARCO, T. LISTER: WALTZING WITH BEARS
ACCURACY IN ESTIMATING DID NOT IMPROVE AS INFORMATION ACCUMULATED, WHILE CONFIDENCE INCREASED CONSISTENTLY.
Claire Tsai, Joshua Klayman, Reid Hastie
ACCURACY OF ESTIMATION
SOURCE: TSAI, KLAYMAN, HASTIE: EFFECTS OF AMOUNT OF INFORMATION ON JUDGMENT ACCURACY AND CONFIDENCE
SCIENTISTS AND WRITERS ARE NOTORIOUSLY PRONE TO UNDERESTIMATE THE TIME REQUIRED TO COMPLETE A PROJECT, EVEN WHEN THEY HAVE CONSIDERABLE EXPERIENCE OF PAST FAILURES TO LIVE UP TO PLANNED SCHEDULES. A SIMILAR BIAS HAS BEEN DOCUMENTED IN ENGINEERS' ESTIMATES.
Daniel Kahneman, Amos Tversky
ESTIMATION BIAS
STORY POINTS
LEAD TIME
WORK TIME
LEAWAIT TIME
WORK TIMELEAD TIME
FLOW EFFICIENCY=
LEAD TIME
WORK TIME
LEAWAIT TIME
LEAWAIT TIME
LEAWAIT TIME
COUNTING THE NUMBER OF STORIES METRIC DOESN'T TAKE THE SIZE INTO ACCOUNT. IT TURNS OUT IT DOESN'T MATTER. THE SIZE OF STORIES IS GELLED TO A VERY COMMON SIZE. WE COULD USE THROUGHPUT VERY SUCCESSFULLY WITH THE RESEARCH.
Larry Maccherone
THROUGHPUT VS STORY POINTS
estimation.lunarlogic.io
EXPERT GUESS DOESN’T WORK
YOUR PLANS
UNIVERSE’S PLANS
YOUR PLANS
UNIVERSE’S PLANS
source: DOGHOUSEDIARIES
ALTERNATIVES?
STATISTICAL FORECASTINGBASED ON THE WORK OF TROY MAGENNIS, LARRY MACCHERONE AND DOUGLAS HUBBARD
PREDICTING THE FUTURE FROM (HISTORICAL) DATA
GETPROJECTR.COM
MONTE CARLO SIMULATION
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
AVERAGE PACE FORECAST - SIMPLE REGRESSION
Today
Burn-up
Scope
6 iterations
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE: THROUGHPUT SAMPLES
Today
Burn-up
Scope
2
2
5
8
4Throughput
Values
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE: THROUPUT DISTRIBUTION
Today
Burn-up
Scope
2
2
5
8
4Throughput
Values
8 tasks 20 %
5 tasks 20 %
4 tasks 20 %
2 tasks 40 %
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE: UNLUCKY BRIAN
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE: WORST CASE SCENARIO
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
12 iterations
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE: BEST CASE SCENARIO
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
12 iterations
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
12 iterations3 iterations
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
8 tasks
5 tasks
4 tasks
2 tasks
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
8 tasks
5 tasks
4 tasks
2 tasks
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
MONTE CARLO EXAMPLE
Today
Burn-up
Scope
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
8 tasks
5 tasks
4 tasks
2 tasks
HISTORICAL DATA: COMPLETED STORIES
start date end date2016-02-01 2016-02-042016-02-01 2016-02-032016-02-05 2016-02-112016-02-04 2016-02-102016-02-08 2016-02-082016-02-09 2016-02-122016-02-09 2016-02-112016-02-12 2016-02-162016-02-11 2016-02-162016-02-19 2016-02-222016-02-18 2016-02-22
start date end date2016-02-19 2016-02-242016-02-17 2016-02-242016-02-23 2016-02-242016-02-23 2016-02-252016-02-25 2016-02-252016-02-25 2016-02-262016-02-26 2016-03-032016-02-26 2016-03-032016-02-29 2016-03-022016-03-01 2016-03-03
HISTORICAL DATA: TIMELINE
1 2 3 4 5 8 9 10 11 12 15 16
17 18 19 22 23 24 25 26 29 1 2 3
LEAD TIMESTATISTICAL MODEL: INPUT #1
WHY LEAD TIMES?
Mon Tue Wed Thur Fri
WHY LEAD TIMES?
5 DAYS
WHY LEAD TIMES?
Mon Tue Wed Thur Fri
BUG FIX SICK
LEAD TIME
5 DAYSWT
HISTORICAL DATA: LEAD TIMES
1 2 3 4 5 8 9 10 11 12 15 164
3 5
5
4
3
1
3
4
17 18 19 22 23 24 25 26 29 1 2 3
3
2 2
4
6
1
2
3 5
5
3
3
LEAD TIME DISTRIBUTION
4
355
4
3
1
3
4
3
22
4
6
1
2
3
55
33
LEAD TIME DISTRIBUTION
6 days
5 days
4 days
3 days
2 days
1 day
WORK IN PROGRESSSTATISTICAL MODEL: INPUT #2
HISTORICAL DATA: WORK IN PROGRESS
1 2 3 4 5 8 9 10 11 12 15 16
17 18 19 22 23 24 25 26 29 1 2 3
2 2 2 2 2 3 4 4 4 3
1 2 4 4 4 4 3 3 3 4
2 2
4 3
WORK IN PROGRESS DISTRIBUTION
4 tasks
3 tasks
2 tasks
1 task
READY…
4 tasks
3 tasks
2 tasks
1 task6 days
5 days
4 days
3 days
2 days
1 day
0
10
20
30
40
50
Iteration1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
} 24 tasks to go
6 days
5 days
4 days
3 days
2 days
1 day 3
5
5
4
3
1
34
32
26
1
3
5
5
3
3
3
3 55
5
5
SAMPLING LEAD TIME VALUES
13 14 15 16 17 18 19 20 21 22 23 24
1 2 3 4 5 6 7 8 9 10 11 12
SAMPLING LEAD TIME VALUES
3
5
5
4
3
1
3
4
3
2
2
6
1
3
5
5
3
3
3
3
5
5
5
5
SUM(LEAD TIME VALUES) = UNITS OF WORK
3
55
4
3
1
3
4
322
6
1
35
5
33
3
355
55
87
1 2 3 4 5 6 7 8 9 10 11 12
SAMPLING WORK IN PROGRESS
4 tasks
3 tasks
2 tasks
1 task 6 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 66 6
66 6 6 6 6
87
1 2 3 4 5 6 7 8 9 10 11 12
SAMPLING WORK IN PROGRESS
4 tasks
3 tasks
2 tasks
1 task 6 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 66 6 6 6 6 6 6 6 6 6
6
6
6
6 6 6
6
6
6
6
6
6
6
6
6
6
6
66
6
6
6
66
6
6
6
6
6
6
6
66
6
6
6
6
87
1 2 3 4 5 6 7 8 9 10 11 12
13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36
66
66
666
6666
66
666
6
6666
6666
6
66
6666
66
66
6
66
66
6
66
6
66
66
6
6 6 6 6
66 6
6
66
6 6 6 6
6
6
66
66
6 6 6 6
6
6
6
6 6 6
6
6 6 6
66
6
DAYS29
28 29
27
30
29
29
27 31 32
31
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
50%
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
90%
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Valu
e A
xis
0
400
800
1200
1600
Workdays
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
99%
GETPROJECTR.COM
A NEW PROJECT VERSUS AN ONGOING PROJECT
90% CONFIDENCE LEVEL MEANS THAT OUR ESTIMATE IS CORRECT 9 TIMES OUT OF 10. KINDA.
Pawel Brodzinski
90% CONFIDENCE INTERVAL?
UNCERTAINTY
SOURCES OF UNCERTAINTY
Project Alfa
Project Alfa
Project Alfa
Project Alfa
Broad range estimates are not welcome
Project Bravo
Project Bravo
Project Bravo
Project Bravo
Project Bravo
Project Bravo
Project Bravo
Garbage in Garbage out
Project Bravo
ANY PROPOSED FORECASTING METHOD JUST HAS TO BE BETTER THAN WHAT YOU DO NOW, OR AT LEAST LESS EXPENSIVE WITH A SIMILAR RESULT.
Troy Magennis
BE BETTER
Paweł Brodziński
@pawelbrodzinski
Tomek Rusiłko
@rusilko
Thank You!
SOURCES & RESOURCES
▸ Troy Magennis original work: http://www.lkce13.com/videos/magennis/
▸ http://focusedobjective.com/wp-content/uploads/2013/05/Modeling-and-Simulating-Software-Projects-Troy-Magennis.pdf
▸ http://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-logic/
▸ https://www.chicagobooth.edu/research/workshops/marketing/archive/workshoppapers/s06/tsai.pdf
▸ Planning Fallacy: https://books.google.pl/books?id=R-syxO7M67AC&pg=PA9&q=&redir_esc=y#v=onepage&q&f=false
▸ https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555
▸ Flow efficiency: https://hakanforss.wordpress.com/2014/06/17/flow-thinking-aceconf/
▸ http://zsoltfabok.com/blog/2013/12/flow-efficiency/
▸ https://www.infoq.com/presentations/agile-quantify
▸ http://brodzinski.com/2015/02/story-points-velocity-the-good-bits.html
▸ https://estimation.lunarlogic.io/
▸ https://www.agilealliance.org/estimation-and-forecasting/
▸ Projectr: http://getprojectr.com/