Photo by Bert van Dijk - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/75478114@N00   Created with Haiku Deck  

Photo by Timothy Wildey - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/36662615@N05   Created with Haiku Deck  

Photo by -Dreamflow- - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/46070773@N05   Created with Haiku Deck  

Photo by bixentro - Creative Commons Attribution License https://www.flickr.com/photos/82947612@N00   Created with Haiku Deck  

Photo by James Marvin Phelps - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/66727626@N00   Created with Haiku Deck  

Photo by dcmaster - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/76224602@N00   Created with Haiku Deck  

Photo by Great Beyond - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/26104563@N00   Created with Haiku Deck  

Photo by "The Wanderer's Eye Photography" - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/8997331@N04   Created with Haiku Deck  

Photo by -iamok- - Creative Commons Attribution License https://www.flickr.com/photos/8854069@N03   Created with Haiku Deck  

Photo by santiago nicolau - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/23552357@N05   Created with Haiku Deck  

Photo by Loving Earth - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/26235754@N05   Created with Haiku Deck  

Photo by dullhunk - Creative Commons Attribution License https://www.flickr.com/photos/14829735@N00   Created with Haiku Deck  

Photo by markhillary - Creative Commons Attribution License https://www.flickr.com/photos/56087830@N00   Created with Haiku Deck  

Photo by orcmid - Creative Commons Attribution License https://www.flickr.com/photos/91555706@N00   Created with Haiku Deck  

Photo by Neil Melville-Kenney - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/27774670@N07   Created with Haiku Deck  

Photo by Brett Whaley - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/21920030@N02   Created with Haiku Deck  

Photo by raneko - Creative Commons Attribution License https://www.flickr.com/photos/24926669@N07   Created with Haiku Deck  

Photo by Photosightfaces - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/30595068@N06   Created with Haiku Deck  

Photo by geezaweezer - Creative Commons Attribution License https://www.flickr.com/photos/33909206@N04   Created with Haiku Deck  

Photo by familymwr - Creative Commons Attribution License https://www.flickr.com/photos/36196762@N04   Created with Haiku Deck  

Photo by Mónica, M - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/52329444@N00   Created with Haiku Deck  

Photo by joiseyshowaa - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/30201239@N00   Created with Haiku Deck  

Photo by `James Wheeler - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/24128704@N08   Created with Haiku Deck  

Photo by William Christiansen - Creative Commons Attribution License https://www.flickr.com/photos/56222080@N04   Created with Haiku Deck  

Photo by merrycrafts - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/10314331@N08   Created with Haiku Deck  

Photo by susanvg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/28362388@N00   Created with Haiku Deck  

Photo by monkeyc.net - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73584213@N00   Created with Haiku Deck  

Photo by juhansonin - Creative Commons Attribution License https://www.flickr.com/photos/38869431@N00   Created with Haiku Deck  

Our Product •  Network Management domain •  Windows-based specialized hardware (“Appliances”) •  Installed in data centers for traffic monitoring, analysis

and network troubleshooting •  but not generally on production network

•  Typical users are technical folks – CIO, Network Manager, Network Engineers

•  Selling cycles typically align with quarterly or annual budget cycles

•  Many sales require implementing customer specials

Old Process, circa 2003 •  Customer Bugs prioritized based on multiple business parameters,

including (partial list) - •  Severity •  Impact on Revenue, Volume, Competitive, etc. •  Case age

•  PMO would prepare Maintenance Release Plan of Record (MR POR) and get buy-in for various types of MRs - •  Service Packs – bunch up ~50-60 bugs typically every quarter •  Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP •  Patches – workaround for customer-specific issues

•  SPs would have •  Above The Line (ATL) requirements – must fix •  Below The Line (BTL) requirements – fix if time permits

Problems with Old Process •  Dev team had no bandwidth to take on maintenance releases •  Huge pile of customer escalations without “home” •  Compounded by high incoming field rate •  Low closure rate (largely due to no dedicated resources) •  Large wait for customers to get bug fixes •  Tech Support often tasked team directly and broke the process •  Hot fixes not always available to all customers •  Sometimes, a new bug fix might break a hot fix •  If a hot fix failed in the field, rollbacks would be very difficult •  Difficult to estimate time to resolve a bug and give an ETA •  High-priority bugs could arrive at any time •  Customer specials could arrive anytime with top priority •  High internal rejection rate of bug fixes by Tech Support

New Process, 2004-05

Dedicated Customer Sustaining

Team  

New “Cumulative

Hot Fix” process

Improved collaboration

with all stakeholders

Product C

Product D

Protocols

Device Drivers

Backend

Product B

GUI

Product A

Our Kanban Process in action…

PMO  

CST  Manager  

QA Team

Dev  Team  =  15  

WIP  =  15  

WIP  =  2  

WIP  =  1wk  

WIP  =  1wk  

Tech  Support  Que

ue  =  0  

Queue  =  Infinite  

Process improvement…the beginnings…

Cumulative Hot Fix Process

Weekly Build Process

So, what is happening? •  Though not an originally stated vision or goal, the “Work in

Progress” (WIP) is being limited to # of team members •  At any time, one developer is assigned only one piece of

work, thereby achieving “One-Piece Flow” •  New work is only assigned when current work is

completed (or cancelled/stalled), and a team member is available

•  No wait state or switching costs at an individual level •  Smaller lead time for bugs (in contract to lead time for SP) •  The process is allowing ‘continuous deployment’ of each of

the hot fixes – even though it is only being practiced in a limited manner

•  Finally, the flexibility gained is not a zero-sum game – there is no penalty on performance in rest of the process

Did this move the needle?

•  Bugs addressed each quarter •  Quality of bug fixes •  “Homes” for bugs •  Total bugs open •  Open days open •  People motivation

From SPs to Cumulative Hot Fixes Maintenance Releases (Service Packs, Patches, Hot Fixes)

Q3 2003 Through Q4 2006 (Fiscal Year)

0 2 3 4 4 4 4 2 2 2 1 2 3 15 2 1

5 3 1 00 1 0 0 0

10

28

11

16 18 20

62

28 2726 26 24

28

21

100

92

87

96

8892

85

80

97 9693

100

9491

7

1215

25 25 25

30 3028 27 26

32

22

66

0

10

20

30

40

50

60

70

Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 060

10

20

30

40

50

60

70

80

90

100

Service Packs Patches Hot Fixes % Successful

Percentage of released Maintenance Releases (Service Packs, Patches, Hots Fixes) that addressed customer reported

Total Maintenance Releases (Service Packs, Patches, Hot Fixes) for this quarter.

ServicePacks

Patches

Hot Fixes

Increase in bugs with “homes”

Customer Direct and Indirect % w/Homes - HistoricalWeek Ending February 03, 2006

55

46 45 47 44 45 45 44

5552 54 56

35

4539 38 41

5146

31

40 40

5963 60 61 61

68 6974 72 69 69

55

73

61

53565548

54 5552 51 51 51 50 49

54 55

47 47

1115 18 16

4753 55

43 42 43

7582

76 76 7582 83 82 81

71 7265

8782

7580

55

4751 51 50 48 48 48 48 48

53 54 51 51

2127 27 25

44

52 51

43 41 42

6772

68 69 6875 76 78 76

70 71

60

80

7164

68

44 47

0

10

20

30

40

50

60

70

80

90

100

5/6/200

5

5/13/2

005

5/20/2

005

5/27/2

005

6/3/200

5

6/10/2

005

6/17/2

005

6/24/2

005

7/1/200

5

7/8/200

5

7/15/2

005

7/22/2

005

7/29/2

005

8/5/200

5

8/12/2

005

8/19/2

005

8/26/2

005

9/2/200

5

9/9/200

5

9/16/2

005

9/23/2

005

9/30/2

005

10/7/

2005

10/14

/2005

10/21

/2005

10/28

/2005

11/4/

2005

11/11

/2005

11/18

/2005

11/25

/2005

12/2/

2005

12/9/

2005

12/16

/2005

12/23

/2005

12/30

/2005

1/6/200

6

1/13/2

006

1/20/2

006

1/27/2

006

2/3/200

6

Direct Indirect Goal (80% w/Homes) Combined

Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)

Total Bugs Open Severity Across Products (Historical)

Week Ending March 31, 2006

222

191

243

213

224230

236

201199

210211

196197195

161

147146146146

132

122

131

121127

111

127125

148150

65605857

5357

626466676768

71747575

6866

4947465049495052

606369707069

747368

61605555

47495149

4435383233323435

36

0

10

20

30

40

50

60

70

80

90

100

110

120

130

31-Dec-03

01-Oct-04

29-Oct-04

19-Nov-04

10-Dec-04

31-Dec-04

21-Jan-05

11-Feb-05

04-Mar-05

25-Mar-05

15-Apr-05

06-May-05

27-May-05

17-Jun-05

08-Jul-05

29-Jul-05

19-Aug-05

09-Sep-05

30-Sep-05

21-Oct-05

11-Nov-05

02-Dec-05

23-Dec-05

13-Jan-06

03-Feb-06

24-Feb-06

17-Mar-06

0

20

40

60

80

100

120

140

160

180

200

220

240

260

Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

Average Days Open Severity: Average Days By Severity (Historical)

Week Ending March 31, 2006

203

155

136

244242246246244248

234235226223

232

244

260268275275

299

232

201210207

179

165172

156162

180

168163156152

147142132135

139143148

140144147146144

155

135145

152148

157156156149

133

144140

144147154155157

163166

177179183182

164162165

154146

141149

123117

134134

123

0

25

50

75

100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

475

500

31-Dec-03

01-Oct-04

29-Oct-04

19-Nov-04

10-Dec-04

31-Dec-04

21-Jan-05

11-Feb-05

04-Mar-05

25-Mar-05

15-Apr-05

06-May-05

27-May-05

17-Jun-05

08-Jul-05

29-Jul-05

19-Aug-05

09-Sep-05

30-Sep-05

21-Oct-05

11-Nov-05

02-Dec-05

23-Dec-05

13-Jan-06

03-Feb-06

24-Feb-06

17-Mar-06

0

25

50

75

100

125

150

175

200

225

250

275

300

Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

People motivation

12 12 12 12 12 10 10 10

4 4 4 4 4 6 6 6

16 16 16 16 16 16 16 16 16 16 16 16

0 2 4 6 8

10 12 14 16 18

Resource Staffing

Sustaining Mainline Approved Headcount

•  Started with 16 people dev team •  We had zero attrition in the team •  Once the backlog started coming down, engineers were ramped

off the team to do new features •  Eventually dismantled the team and rolled-up engineers into dev

teams when backlog came down to single digits

Photo by Romain [ apictureourselves.org ] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/72898690@N00   Created with Haiku Deck