implementing scrum on a large scale

Orlando Scrum Gathering

1

Implementing Scrum on a Large ScaleDan LeFebvreAgile Coach, CSP

3/16-18/2009


2

Biography

•Over 20 years in software product development as a developer, manager, director, and coach

•Last 5 years using agile development practices

•Last 2 years implementing Scrum in a 700 person engineering organization as an Agile Coach▫Sites in MA, OR, TX, GA, IL▫Also in Canada – BC, Que▫Also in Belgium and Noida, India

3/16-18/2009


3

Intent of this Session• To take you through the journey of that 700

person engineering organization toward agility through Scrum. ▫How did they decide to go agile? ▫How did they decide to go Scrum? ▫How do they coordinate a large suite release? ▫How do they handle organizationally

impediments? ▫What about progress reporting? ▫What were their biggest challenges? ▫What are their challenges yet to face?

3/16-18/2009


4

Something is not right

3/16-18/2009


5

Productivity was plummeting

Version

New Feature Development (months)

Stabilize (months)

Total (months)

5.0 actual 8 6 14

5.1 actual 6 7 13

5.2 planned 5 7 12

3/16-18/2009


6

Cutter Consortium Evaluation

•Michael Mah and Jim Highsmith•Evaluation – both quantitative and

qualitative•Recommended and sold Agile to CTO

3/16-18/2009


7

“Agile Lite” Goal 11-1

•Iterative/incremental development•Test Driven Development•Emergent/Evolutionary Design•Daily Standups•Retrospectives

•Phase gated governance model ▫Commit to contents, dates, cost for a 12

month release▫Account for entire organizational effort

3/16-18/2009


8

Weakest Link Theory

•Interdependent suite of products•All must ship simultaneously•If any of them not agile, suite is in trouble

•Roll out to entire organization at once•5 full time coaches from Cutter, 5 part

time internal coaches•After 1 year, only used 5 part time

coaches

3/16-18/2009


9

Results

•Delivered 7-5•Improved Quality•More automated tests•Agile in the culture

3/16-18/2009


10

Entropy“Inevitable and steady deterioration of a system or society.”

The American Heritage® Dictionary of the English Language, Fourth Edition

3/16-18/2009


11

Slip back to old ways• All part-time coaches back to their day jobs• Committed to 3 suite-wide projects within the

same release (each considered all or nothing)▫Globalization, New UI and new Reporting

platform• Results

▫Projects fell behind▫All UI-based automation broke▫2 projects stopped doing retrospectives▫Team morale suffered▫No or meaningless burncharts so no transparency

3/16-18/2009


12

The Agile Coach

•Without the coaches, teams had no help•Decided to get a full-time Agile Coach

▫Teach agile to new employees▫Observe and help teams that are struggling▫Roll out agile to newly acquired companies▫Become the agile conscience of the

organization

3/16-18/2009


13

What to do with management

3/16-18/2009


14

Management still too “Command and Control”•Culture is very hierarchical•People treated as “resources”

▫Regularly moved from team (workgroup?) to team to handle crises and delays

•Management makes most decisions▫Teams do not feel empowered▫Many teams just going through the motions

of agile development•“Blame” is typical reaction

3/16-18/2009


15

Collaboration Explained

•90 people trained in collaboration skills by Jean Tabaka and Ronica Roth from Rally▫Learned tools▫Facilitation techniques▫Lots of hands-on exercises

•Results▫Meetings ran better▫Better agendas and capturing of group

learning▫Better retrospectives

3/16-18/2009


16

Management not Agile

•Management not quite agile▫Very little training was given to managers

•Still division between Dev and QA

3/16-18/2009

Customer Community

Customer Community

Developer Community

Developer Community

Manager Community

Manager Community


17

Scrum to the rescue• Scrum is a management framework• Training focused on the managers and product

managers▫60 ScrumMaster receive CSM from Ken Schwaber▫30 Product Owners receive CSPO from Ken

Schwaber• Create true cross-functional teams

▫No more “communities”▫1 ScrumMaster, 1Product Owner, Team of

developers, testers, writer, etc.• Managers became ScrumMasters, Product

Managers are now the Product Owners

3/16-18/2009


18

Scaling issues

3/16-18/2009


19

Large Interdependent Suite

CTO/Executive VP

VP Applications

Director – Flagship

Director – Add-ons

VP Systems

Director – Developm

ent

Director – QA

VP Customer Engineering

Director – Developm

ent

Director – QA

3/16-18/2009

• About 30 teams need to coordinate effort to release the suite• Deemed impossible to synchronize sprints


20

Parallel Project Organization• Suite Management Team (SMT)

▫Steering committee▫1 Director and 1 Product Owner from each of the 3

organizations• Suite Integration Team (SIT)

▫ Installs suite on a set of integration servers each week▫Creates and runs series of suite-wide tests

• Suite Release Team (SRT)▫ScrumMaster and Product Owner from each team (60

people)▫Responsible for coordinating dependencies and resolving

suite-wide issues▫Primary communication vehicle for the suite

3/16-18/2009


21

How do they synchronize and review the suite?•Created the concept of “Release

Candidates”▫Every 6 to 8 weeks▫Entire system integrated and tested

together▫Suite build and battery of suite tests run

weekly•Teams hold their own sprints but

integrate into last integrated system each week▫Each team has daily builds and battery of

daily tests (some team build more often)

3/16-18/2009


22

Heartbeats

3/16-18/2009

•Beginning each RC•Each team presents release plan focusing on

dependencies and impacts to SRT•End of each RC

•SRT holds an RC Retrospective•Product Owners hold an RC Review for

organization

RC1 RC2 RC3

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Sprint 5

Sprint 6

Sprint 7

Sprint 8

Sprint 9

Suite heartbeat

Team 1 heartbeat

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Sprint 5

Sprint 6Team 2

heartbeat

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Sprint 5

Sprint 6

Sprint 7Team 3

heartbeat


23

Dealing with Impediments

3/16-18/2009


24

Dealing with suboptimization

•Each team is run fairly independently•Each identify and resolve impediments

▫Experiencing the same impediments▫Different solutions▫Some solutions impacted other teams

3/16-18/2009


25

First Attempt – Etc.

•Senior Execs doing the work of prioritizing and resolving organizational impediments ▫Identified by the development teams▫Resolve by creating small teams to remove

the impediment▫Use Scrum to run the process

• Issues▫Execs had no time for this work▫Reluctant to assign people to impediment

removal teams

3/16-18/2009


26

Second Attempt – Scrum Implementation Team•Small group of 8 people from across the

organization •Issues

▫Focused more on process definition instead of impediment removal

▫Not all the skills represented▫Limited time to work on this, still had day

jobs

3/16-18/2009


27

Third Attempt – ScrumMaster Meeting

•Hold a regular meeting of ScrumMasters to identify, prioritize, and volunteer to resolve impediments

•This group got some traction•Issues

▫Many impediments around Product Ownership

▫Also many architectural impediments

3/16-18/2009


28

Final Attempt – Agile Leaders Meeting

•Regular meeting of ScrumMasters, Product Owners, and Architects led by Agile Coach

•Agenda includes a brief coaching session on an agile topic from one of the team

•Review of resolved impediments, identifying and prioritizing new ones, and volunteering to resolve highest priority

•Q & A session to solicit help from team and coach on individual issues

•Resolved over 50 impediments in 1 year

3/16-18/2009


29

Transparency Issues

3/16-18/2009


30

Transparency is more translucent•Each team uses their own tools for

tracking data▫Flipcharts and post-its▫Excel – Various workbooks▫X-Planner

•Rollup data collected manually▫Sometimes late▫Sometimes not complete▫Apples to Oranges rollup

•“Blame” culture prevents full disclosure too soon

3/16-18/2009


31

Clearing it up

•SMT created a standard workbook that normalizes data to facilitate a suite-wide rollup

•Includes a projected organizational velocity to predict end date

“I have never been in an organization that knew more about where it was throughout the lifecycle of the project” – One Senior Staff Member

3/16-18/2009


32

Clearing it up more

•Implementing VersionOne ▫Centralize the project information ▫Give execs instant access to status of any

project▫Standardize how information is collected

and managed throughout the organization▫Simplify suite rollup▫Provide unprecedented transparency

•Create standard reports

3/16-18/2009


33

Dealing with “Lite”

3/16-18/2009


34

Suite planning still phase-gated•Need still exists to commit to an annual

plan •Company expects large features to justify

the 700 person engineering staff•Outside engineering still driven by

“waterfall” model▫Cannot or will not take advantage of

iterative delivery

3/16-18/2009


35

Creating a Balance

•They created a multi-tiered content strategy ▫Commit at the high level▫Establish budgets at the mid-level▫Stay flexible at the details

•Budgets give guidance to Product Owners on what senior management is willing to invest to get the benefit or return

3/16-18/2009


36

Requirements hierarchy• Initiatives – Broad areas of focus tied to corporate strategy

• Headlines – Major Feature Sets/Capabilities within an Initiative▫ Engineering commits to these within a budget

• Shippable Units – The smallest feature that is worth shipping to a customer▫ Analogous to Minimal Marketable Feature from Software by

Numbers

▫ Engineering delivers as many of these within the budget of a headline

▫ Business Value measured here

• Stories – User stories as we all know and love

3/16-18/2009


37

Planning Onion

Strategy – Sets Initiatives

Release - Stories to Sprints

Sprint – Tasks to Stories

Daily - Tasks

3/16-18/2009

Planning and Outputs

Release PlanSprint 1 Sprint 2 Sprint 3

As a User, I can jfh hf jahdsdf As a User, I can jfh hf jahdsdf

As a User, I can jfh hf jahdsdf










As a User, I can jfh hf jahdsdf As a User, I can jfh hf jahdsdf



Product Plan

SU

RC 1 RC 2 RC 3 RC 4

SU SU

SU SU

SU SU

SU SU

SU SU

SU SU

SU SU

SU SU SU

SU SU

Portfolio Plan

Rel 6.2

Rel 6.1.1

Rel 6.1.2

HeadlineHeadline

HeadlineHeadline

HeadlineHeadline

HeadlineHeadline

HeadlineHeadline

HeadlineHeadline

HeadlineHeadline

Headline Sprint PlanAs a User, I can jfh hf jahdsdf

Started Done

TaskTask

TaskTask

TaskTask


39

Is Done really Done?

3/16-18/2009


40

Unpredictable “tail”

•Planned 3 months for final stabilization ▫Always goes longer▫Many unpredictable issues found

•The Definition of Done (DoD) was not the same across teams▫Some due to the large amount of legacy

code▫Some based on management interpretation

– “Spirit of the Criteria”

3/16-18/2009


41

Executive Bias• Setting DoD too “weak” - guideline

▫Leads to too much work during the tail▫Makes it hard to know where you are▫Causes integration issues due to impedance

mismatch • Setting DoD too “Strong” - rule

▫No real progress can be made due to late issues▫Sets the bar too low on content

• Executives wants team to “stretch” by setting aggressive DoD but knows he’ll get less (and accepts it)▫This line is not apparent to the teams

3/16-18/2009


42

Created 2 lists of criteria

•Definition of Done▫This is the must be complete to declare

victory•Quality Goal

▫Strive to complete and must justify shortfall

•Example▫DoD – Performance test must be completely

run▫Quality Goal – Performance must not have

degraded by more than 5% with no S1 defects

3/16-18/2009


43

DoD is too much for each Sprint•Doing all the work needed for shipping

each sprint is considered wasteful•Some tests run almost as long as a sprint!•Lots of legacy code with manual tests

3/16-18/2009


44

Modify DoD and QG by timebox

•Defined unambiguously by SMT

•For Ship – include everything that must be done

•For RC – remove what they are unable or unwilling to do each RC

•For Sprint – remove what they are unable or unwilling to do each Sprint

3/16-18/2009


45

Results

3/16-18/2009


46

Increased Automation

3/16-18/2009

RC1 RC2 RC3 RC4 RC5 RC6 RC7 RC80

10,000

20,000

30,000

40,000

50,000

60,000

Automated Tests Run


47

What happened to quality

3/16-18/2009

0 10 20 30 40 50 60 70 80 90 1001101200

1000

2000

3000

4000

5000

V6.0V6.1

Weeks

Op

en

D

efe

cts

Pre-Scrum

Scrum


48

Next Challenge

3/16-18/2009


49

Enticing rest of Organization to be more Agile•There is still a “batch” approach to

preparing Service and Customer training •Marketing not strongly tied in with

Product Management and the Product Owners

3/16-18/2009


50

Performance and Compensation still very individualized•Has a negative effect on teamwork•Some teams view ScrumMaster/Manager

as an evaluator and aren’t as open with issues

•Individual goals trump team goals or organizational goals

3/16-18/2009


51

Still treat people as “resources”•Cannot break habit of moving people from

team to team•Cannot break habit of needing “full

utilization”•Cannot break habit of accounting for

hours worked

3/16-18/2009


52

Recent Restructuring

•Increased fear▫Potential to reduce transparency▫Potential to reduce teamwork to “CYA”

•Lowered morale ▫Lower energy▫Potential less commitment

3/16-18/2009


53

Loss of the Coach

•They eliminated the position of Agile Coach

•Entropy may set in again▫No focus for getting help to the teams▫Impediments mechanism may break down▫Training of new employees and

organizations will be affected

3/16-18/2009


54

Summary

3/16-18/2009


55

Where are they?

•Scrum is implemented throughout•Mechanism for organizational

impediments in place•Transparency is improving•Improvements to engineering practices

are on-going•Quality is improving•Planning is becoming more flexible

3/16-18/2009


56

Questions?Dan LeFebvreDirector, Software Engineering; Agile [email protected]://PracticalAgileByDan.blogspot.com

3/16-18/2009

mailto:[email protected]

http://practicalagilebydan.blogspot.com/

implementing scrum on a large scale

Technology