From  Waterfall  to  Weekly  Releases:  A  Case  Study  in  using    

Evo  &  Kanban  (2004-­‐05)  

Tathagat  Varma  hHp://thoughtleadership.in    

Our Product •   Network Management domain •   Windows-based specialized hardware

(“Appliances”) •   Installed in data centers for traffic monitoring,

analysis and network troubleshooting –   but not generally on production network

•   Typical users are technical folks – CIO, Network Manager, Network Engineers

•   Selling cycles typically align with quarterly or annual budget cycles

•   Many sales require customer specials

Old Process, circa 2003 •   Customer Bugs prioritized based on multiple business parameters,

including (partial list) - –   Severity –   Impact on Revenue, Volume, Competitive, etc. –   Case age

•   PMO would prepare Maintenance Release Plan of Record (MR POR) and get buy-in for various types of MRs - –   Service Packs – bunch up ~50-60 bugs typically every quarter –   Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP –   Patches – workaround for customer-specific issues

•   SPs would have –   Above The Line (ATL) requirements – must fix –   Below The Line (BTL) requirements – fix if time permits

Problems with Old Process •   Dev team had no bandwidth to take on maintenance releases •   Huge pile of customer escalations without “home” •   Compounded by high incoming field rate •   Low closure rate (largely due to no dedicated resources) •   Large wait for customers to get bug fixes •   Tech Support often tasked team directly and broke the process •   Hot fixes not always available to all customers •   Sometimes, a new bug fix might break a hot fix •   If a hot fix failed in the field, rollbacks would be very difficult •   Difficult to estimate time to resolve a bug and give an ETA •   High-priority bugs could arrive at any time •   Customer specials could arrive anytime with top priority •   High internal rejection rate of bug fixes by Tech Support

New Process, 2004-05

Dedicated Customer Sustaining

Team  

New “Cumulative

Hot Fix” process

Improved collaboration

with all stakeholders

EvoluLonary  Project  Management  “Evo”  

•   E1:  Decompose  by  performance  results  and  stakeholders  •   E2:  Do  high-­‐risk  steps  early,  learn  how  ‘unknowns’  really  perform    •   E3:  Focus  on  improving  your  most  valuable  performance  objecLves  

first  •   E4:  Base  your  early  evoluLon  on  exisLng  frameworks  and  

stakeholders  •   E5:  Design  to  cost  dynamically  •   E6:  Design  to  performance  dynamically    •   E7:  Invest  in  an  open-­‐ended  architecture  early  on    •   E8:  MoLvate  your  team  by  rewarding  results    •   E9:  PrioriLze  changes  by  value,  not  place  in  queue    •   E10:  Learn  fast,  change  fast,  adapt  to  reality  fast.  

Process improvement…the beginnings…

Cumulative Hot Fix Process

Weekly Build Process

Product C

Product D

Protocols

Decide Drivers

Backend

Product B

GUI

Product A

Our Kanban Process in action…

PMO  

CST  Manager  

QA Team

Dev  Team  =  15  

WIP  =  15  

WIP  =  2  

WIP  =  1wk  

WIP  =  1wk  

Tech  Support  Que

ue  =  0  

Queue  =  0  

So, what is happening? •   Though not an originally stated vision or goal, the “Work in

Progress” (WIP) is being limited to # of team members •   At any time, one developer is assigned only one piece of work,

thereby achieving “One-Piece Flow” •   New work is only assigned when current work is

completed (or cancelled/stalled), and a team member is available

•   No wait state or switching costs at an individual level •   Smaller lead time for bugs (in contrast to lead time for SP) •   The process is allowing ‘weekly deployment’ of each of the

hot fixes •   Finally, the flexibility gained is not a zero-sum game – there is

no penalty on performance in rest of the process

Did this move the needle? •  Bugs addressed each quarter •  Quality of bug fixes •  “Homes” for bugs •  Total bugs open •  Open days open •  People motivation

Shift from SPs to Cumulative Hot Fixes while maintaining High Quality

Maintenance Releases (Service Packs, Patches, Hot Fixes) Q3 2003 Through Q4 2006 (Fiscal Year)

0 2 3 4 4 4 4 2 2 2 1 2 3 15 2 1

5 3 1 00 1 0 0 0

10

28

11

16 18 20

62

28 2726 26 24

28

21

100

92

87

96

8892

85

80

97 9693

100

9491

7

1215

25 25 25

30 3028 27 26

32

22

66

0

10

20

30

40

50

60

70

Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 060

10

20

30

40

50

60

70

80

90

100

Service Packs Patches Hot Fixes % Successful

Percentage of released Maintenance Releases (Service Packs, Patches, Hots Fixes) that addressed customer reported

Total Maintenance Releases (Service Packs, Patches, Hot Fixes) for this quarter.

ServicePacks

Patches

Hot Fixes

Increase in bugs with “homes”

Customer Direct and Indirect % w/Homes - HistoricalWeek Ending February 03, 2006

55

46 45 47 44 45 45 44

5552 54 56

35

4539 38 41

5146

31

40 40

5963 60 61 61

68 6974 72 69 69

55

73

61

53565548

54 5552 51 51 51 50 49

54 55

47 47

1115 18 16

4753 55

43 42 43

7582

76 76 7582 83 82 81

71 7265

8782

7580

55

4751 51 50 48 48 48 48 48

53 54 51 51

2127 27 25

44

52 51

43 41 42

6772

68 69 6875 76 78 76

70 71

60

80

7164

68

44 47

0

10

20

30

40

50

60

70

80

90

100

5/6/200

5

5/13/2

005

5/20/2

005

5/27/2

005

6/3/200

5

6/10/2

005

6/17/2

005

6/24/2

005

7/1/200

5

7/8/200

5

7/15/2

005

7/22/2

005

7/29/2

005

8/5/200

5

8/12/2

005

8/19/2

005

8/26/2

005

9/2/200

5

9/9/200

5

9/16/2

005

9/23/2

005

9/30/2

005

10/7/

2005

10/14

/2005

10/21

/2005

10/28

/2005

11/4/

2005

11/11

/2005

11/18

/2005

11/25

/2005

12/2/

2005

12/9/

2005

12/16

/2005

12/23

/2005

12/30

/2005

1/6/200

6

1/13/2

006

1/20/2

006

1/27/2

006

2/3/200

6

Direct Indirect Goal (80% w/Homes) Combined

Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)

Total Bugs Open Severity Across Products (Historical)

Week Ending March 31, 2006

222

191

243

213

224230

236

201199

210211

196197195

161

147146146146

132

122

131

121127

111

127125

148150

65605857

5357

626466676768

71747575

6866

4947465049495052

606369707069

747368

61605555

47495149

4435383233323435

36

0

10

20

30

40

50

60

70

80

90

100

110

120

130

31-Dec-03

01-Oct-04

29-Oct-04

19-Nov-04

10-Dec-04

31-Dec-04

21-Jan-05

11-Feb-05

04-Mar-05

25-Mar-05

15-Apr-05

06-May-05

27-May-05

17-Jun-05

08-Jul-05

29-Jul-05

19-Aug-05

09-Sep-05

30-Sep-05

21-Oct-05

11-Nov-05

02-Dec-05

23-Dec-05

13-Jan-06

03-Feb-06

24-Feb-06

17-Mar-06

0

20

40

60

80

100

120

140

160

180

200

220

240

260

Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

Average Days Open Severity: Average Days By Severity (Historical)

Week Ending March 31, 2006

203

155

136

244242246246244248

234235226223

232

244

260268275275

299

232

201210207

179

165172

156162

180

168163156152

147142132135

139143148

140144147146144

155

135145

152148

157156156149

133

144140

144147154155157

163166

177179183182

164162165

154146

141149

123117

134134

123

0

25

50

75

100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

475

500

31-Dec-03

01-Oct-04

29-Oct-04

19-Nov-04

10-Dec-04

31-Dec-04

21-Jan-05

11-Feb-05

04-Mar-05

25-Mar-05

15-Apr-05

06-May-05

27-May-05

17-Jun-05

08-Jul-05

29-Jul-05

19-Aug-05

09-Sep-05

30-Sep-05

21-Oct-05

11-Nov-05

02-Dec-05

23-Dec-05

13-Jan-06

03-Feb-06

24-Feb-06

17-Mar-06

0

25

50

75

100

125

150

175

200

225

250

275

300

Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

People motivation

12 12 12 12 12 10 10 10

4 4 4 4 4 6 6 6

16 16 16 16 16 16 16 16 16 16 16 16

0 2 4 6 8

10 12 14 16 18

Resource Staffing

Sustaining Mainline Approved Headcount

•   Started with 16 people dev team •   We had zero attrition in the team •   Once the backlog started coming down, engineers were ramped

off the team to do new features •   Eventually dismantled the team and rolled-up engineers into dev

teams when backlog came down to single digits

What is Kanban? •   Kanban (literally signboard or billboard) is a scheduling system

for lean and just-in-time (JIT) production. According to its creator, Taiichi Ohno, kanban is one means through which JIT is achieved.

•   Kanban is not an inventory control system; it is a scheduling system that helps determine what to produce, when to produce it, and how much to produce.

•   The need to maintain a high rate of improvement led Toyota to devise the kanban system. Kanban became an effective tool to support the running of the production system as a whole.

•   In addition, it proved to be an excellent way for promoting improvements because reducing the number of kanban in circulation highlighted problem areas.

hHps://en.wikipedia.org/wiki/Kanban    

A Kanban System at Toyota dealership

hHps://twitpic.com/het3u    

How does it work?

hHp://www.toyota-­‐global.com/company/vision_philosophy/toyota_producLon_system/just-­‐in-­‐Lme.html    

How Kanban helps achieve “Just-in-Time”

•   For  example,  to  efficiently  produce  a  large  number  of  automobiles,  which  can  consist  of  around  30,000  parts,  it  is  necessary  to  create  a  detailed  producLon  plan  that  includes  parts  procurement.  Supplying  "what  is  needed,  when  it  is  needed,  and  in  the  amount  needed"  according  to  this  producLon  plan  can  eliminate  waste,  inconsistencies,  and  unreasonable  requirements,  resulLng  in  improved  producLvity.  

 

hHp://www.toyota-­‐global.com/company/vision_philosophy/toyota_producLon_system/just-­‐in-­‐Lme.html    

Kanban for Software •   Visualize the Workflow: Represent the work items and

the workflow on a card wall or electronic board •   Limit Work-in-Progress (WIP): Set agreed upon limits

on how many work items are in progress at a time •   Measure and Manage Flow: Track work items to see if

they are proceeding at a steady, even pace •   Make Process Policies Explicit: Agree upon and post

policies about how work will be handled •   Use Models to Evaluate Improvement Opportunities:

Adapt the process using ideas from Systems Thinking, Deming, etc.

Kanban: Successful Evolutionary Change for your Technology Business – David Anderson

Why Kanban in Software Engineering?

Don’t  build  features  that  nobody  needs  right  now  

Don’t  write  more  specs  than  you  can  code  

Don’t  write  more  code  than  you  can  test  

Don’t  test  more  code  than  you  can  deploy  

hHps://leanandkanban.files.wordpress.com/2009/04/kanban-­‐for-­‐sohware-­‐engineering-­‐apr-­‐242.pdf    

What did We learn?

•   Process improvement should be driven by business needs – and NOT because some process looks sexy!

•   Don’t let a process limit your potential – think beyond gurus!

•   Don’t let absence of a process limit your potential – do whatever it takes to serve customer better!

Resources •   Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and

http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166 •   http://refcardz.dzone.com/refcardz/getting-started-kanban •   Ship early and ship twice as often,

https://www.facebook.com/notes/facebook-engineering/ship-early-and-ship-twice-as-often/10150985860363920

•   How we built Flickr, http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/

•   Continuous Deployment at IMVU: Doing the impossible fifty times a day, http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

•   A New Version of Google Chrome now due every six weeks, http://techcrunch.com/2010/07/22/google-chrome-versions/

•   In praise of continuous deployment: The WordPress.com story, http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/

•   CONWIP, https://en.wikipedia.org/wiki/CONWIP •   Kanban applied to Software Development: from Agile to Lean,

http://www.infoq.com/articles/hiranabe-lean-agile-kanban

Photo by orcmid - Creative Commons Attribution License https://www.flickr.com/photos/91555706@N00   Created with Haiku Deck  

Photo by Neil Melville-Kenney - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/27774670@N07   Created with Haiku Deck  

Photo by Brett Whaley - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/21920030@N02   Created with Haiku Deck  

Photo by raneko - Creative Commons Attribution License https://www.flickr.com/photos/24926669@N07   Created with Haiku Deck  

Photo by Photosightfaces - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/30595068@N06   Created with Haiku Deck  

Photo by geezaweezer - Creative Commons Attribution License https://www.flickr.com/photos/33909206@N04   Created with Haiku Deck  

Photo by familymwr - Creative Commons Attribution License https://www.flickr.com/photos/36196762@N04   Created with Haiku Deck  

Photo by Mónica, M - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/52329444@N00   Created with Haiku Deck  

Photo by joiseyshowaa - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/30201239@N00   Created with Haiku Deck  

Photo by `James Wheeler - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/24128704@N08   Created with Haiku Deck  

Photo by William Christiansen - Creative Commons Attribution License https://www.flickr.com/photos/56222080@N04   Created with Haiku Deck  

Photo by merrycrafts - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/10314331@N08   Created with Haiku Deck  

Photo by susanvg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/28362388@N00   Created with Haiku Deck  

Photo by monkeyc.net - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73584213@N00   Created with Haiku Deck  

Photo by Romain [ apictureourselves.org ] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/72898690@N00   Created with Haiku Deck