10,000 miles away: developing with a distributed...

Or, “How to be Agile at a Distance”

Instructor: Kevin Thompson, PhD, PMP, ACP, CSP, CSM

4100 E. Third Ave, Suite 205, Foster City, CA 94404 | 650-931-1651 | www.cprime.com

The leader in training and consulting for project management and agile development

10,000 Miles away: Developing

with a Distributed Team

http://www.cprime.com/

Copyright 2013, cPrime Inc. 2

Who is cPrime? ENGAGED FOR YOUR SUCCESS


Today’s Presenter

Kevin Thompson, Ph.D.

Agile Practice Lead

• Trains, coaches teams and companies in Agile

development

• Education and certifications

Certified ScrumMaster and Scrum Professional

PMI Project Management Professional

PMI Agile Certified Practitioner

Scaled Agile Framework Program Consultant

Certified Yellow Belt Collaboration Architect

Doctorate in Physics, Princeton University


Congratulations!

You are VP of Engineering for a small software company

with big ambitions, located in Palo Alto, California.

Your first product is GloCAD.

GloCAD enables mechanical engineers around the world to

collaborate in the production of complex electro-mechanical

designs

You need to build a world-class software-development

organization. You are going to start with one, small

Team, and grow over time. You have selected the

Scrum process to provide the flexibility, visibility, and

short time-to-market the company needs.

Your journey has begun…


What We Need to Know First

What does “Agile” mean?

What is an Agile process?

What is “Scrum?”


Agile Background


What is an Agile Process?

In principle:

Any process that adheres to the principles

of the Agile Manifesto

“We are uncovering better ways of developing software by doing it and

helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

Manifesto for Agile Software Development, www.agilemanifesto.org

The concepts arose in software project management, BUT…

Change “software” to “products” or “deliverables” to apply more generally


Adaptive Spectrum Drives Process Selection

All processes have their “sweet spots”

Based on scope, effort uncertainty

Predictive Adaptive Reactive

Plan-Driven Scrum Kanban

Waterfall XP

SDLC

PREDICTIVE REACTIVE

The Agile Zone

Adaptive processes

• Emphasize adaptability to

rapid change

• Enable detailed planning

Reactive processes

• Don’t require planning

• Handle unpredictable work

well

Predictive

Processes

• Emphasize

Efficiency

• Perform poorly

when

uncertainty

is high


Key Scrum Concepts

Product Owner provides ranked requirements, as short narrative descriptions (“Stories”), or bug-fix requests. Set of unscheduled requirements is the “Product Backlog.” Each requirement is a Product Backlog Item (PBI).

Small Teams (3—9 people) work in short “Sprints” (2—4 weeks) to implement stories in rank order.

• Requirements frozen when Sprint starts—no change requests allowed!

Teams self-organize to best apply member skill sets (coding, test development, testing, etc.).

• ScrumMaster does not assign tasks

ScrumMaster does whatever is needed to make Team as productive as possible

• Focuses on planning, tracking, mentoring, resolving issues, enforcing process

Schedule rules: Don’t extend Sprint to finish incomplete Stories


Agile Collaboration by Swarming

How many people can work on

Story #1?

How many people can …

They swarm on #1.

How many people can work on

Story #2?

They swarm on #2.


Tracking

• Scrum Taskboard shows plan, tracks status of work in Sprint

• E.g., write Stories, Tasks on sticky templates, put on

Taskboard

• Team members move Tasks to update status

• Burndown chart tracks status against plan


Scrum Ceremonies

Ceremony Time Box

Input Output Value

Backlog

Grooming* <1 hr

Draft User Stories,

Epics from Product

Owner

Finalized User Stories

Technical Stories

Ranking for top PBIs

Product Backlog & Team

are ready for Sprint

Planning

Sprint

Planning 2 - 8 hr

Ranked Product

Backlog with

Acceptance

Criteria

Sprint Backlog:

• Selected stories + estimates

• Tasks + estimates

Team has a plan to

implement Sprint

Backlog

Daily Scrum

(Stand-up) <15 min In-progress Tasks

Tasks updated

Impediments raised

Team members on same

page re: Sprint progress

and impediments

Sprint

Review < 1 hr

Demo prepared for

completed stories

New Stories, based on review

by Product Owner

Ranking may be revised

Deliverables reviewed;

feedback from stake-

holders, other teams

Retro-

spective

1 - 1.5

hr

Sprint performance

data, e.g.

Burndown chart

Short list of improvements for

next Sprint, with owners

Learn from experience,

enable continuous

improvement

* Not officially a Scrum Ceremony, but important


Sprints Repeat in a Cadence

• Cadence: Rhythmical motion or activity

• Requirements are completed before Sprint starts

• Planning is continuous, not phased

Working Days

Day 1

Backlog

Grooming

Day 31

Sprint Planning Meeting

Create Task Breakdowns

Begin Dev & Testing

Sprint Review

Day 3 Day 5 Day 7 Day 9 Day 11 Day 13 Day 15 Day 17 Day 19 Day 21 Day 23 Day 25 Day 27 Day 29

Sprint N-1 Sprint N Sprint N+1

Retrospective

Create Task Breakdowns

Begin Dev & Testing

Sprint Planning Meeting

Backlog

Grooming

Backlog

Grooming


The Journey


Starting out with the First Scrum Team

• This Scrum Team has

• One experienced ScrumMaster

• One inexperienced but committed Product Owner

• Seven skilled Team members (5 developers, 2 QA specialists)

• This Team

• Plans one Sprint at a time

• Tracks progress with a physical Scrum Taskboard

Stories & Tasks on sticky notes

States are Not Started, In Progress, or Complete

ScrumMaster draws daily Burndown chart by hand

• Is serious about test automation, using

JUnit for unit and integration testing

Cucumber for acceptance testing

Selenium for UI-level testing

Bamboo for Continuous Integration & test automation


How the First Few Sprints Go

• Most of Sprint 1 focuses on setting up tools and

environments for development and testing

• Estimates are very poor

• Tracking is erratic

• The plan (Sprint Backlog) and reality differ substantially

• Sprint 2 gets more done on the product

• Estimates are better

• Visibility into status is improving

• Reality resembles plan

• Sprints 3 and on improve

• Estimation and planning improve

• Execution becomes more predictable

• Productivity reaches a plateau of high effectiveness

• It’s working!


Time to Grow

• Funding from early-stage investor will help us move faster

• We hire seven more people, for a total of 14

• Maximum Team size in Scrum is 9, so

• Organize into two Teams

• New Team moves into building next door

• Have same ScrumMaster for both Teams

• Have same Product Owner for both Teams

• We’ve clearly mastered Scrum

• We’re ready for growth

• Nothing will go wrong now, right?

• Right?


Wrong!


Here’s Why

• Team A changes an interface.

• Team B’s code stops working

• Guess who else used that interface?

• Team B deletes some obsolete code they don’t need any

more

• Team A’s code stops working

• Maybe that code wasn’t obsolete after all

• We have a new kind of problem: Cross-Team impacts

• What should we do?


How we Implement Cross-Team Requirements

• Cross-Team “Scrum of Scrums” meetings

• Purpose: Identify & address cross-Team issues

• Frequency: As needed (daily, semi-weekly,…)

• Participants:

• Facilitator (e.g., Program Manager)

• Representative from each Team

Team Member, ScrumMaster

• Agenda:

• Each person describes

What my Team is doing that may affect other Teams

What issues my Team needs help to resolve

• Resolve issues in meeting, if possible

• Identify follow-up actions and owners


Adding Scrum of Scrums Meeting Helps

• Twice-weekly forum with representative from each Team

ensures cross-Team issues are addressed

• It becomes clear that a major contributing factor to cross-

Team issues is lack of visibility for each Team into the

status of work, plans for the other Team

• They use paper-based Scrum Taskboards

• They are in different buildings

• They can’t see each other’s Taskboard



Introduce Web-based Agile PM Tool

• Agile Project-Management tools provide global visibility for

requirements, plans, and status of work

• Example: View of Sprint Status, from Atlassian’s Jira Agile


Scrum Metrics for Tracking Progress

• Example of Jira Agile Burndown, Burn-up Charts


Better, but…

• Plans, status of work are visible to everyone.

• Easier to see what each Team is doing that might affect other

Teams

• Fixing this problem clarifies the existence of another one:

• It’s getting hard to find documents other than requirements

• API (Application Programming Interface) definitions

• User Interface Style Guides

• Coding standards

• Design, Architecture documents

• These are relevant for all Teams. Emailing them on request

isn’t satisfying need for availability, and it doesn’t make

sense to attach them to Stories in Jira.



Provide Repository for Information

• Use this for everything not tied to a specific Story

• Wikis, SharePoint, Basecamp

• Example: Confluence, from Atlassian


Now Teams are Running Smoothly

• We’ve overcome our growing pains

• Our multiple-Team environment is developing products

smoothly, reliably

• We’re handling cross-Team issues effectively

• Everyone can see status, plans, and information of general

interest

• We’re set, now.

• Ready for future growth.


• Right?


Time to Grow—Again

• We’ve shipped our first release of GloCAD!

• Sales are good, demand is growing

• We hire 14 more people, for a total of 28

• Maximum Team size in Scrum is 9, so

• Organize into four Teams

• Move everyone into a bigger office

• Have same ScrumMaster for all four Teams

• Have same Product Owner for four Teams


What Happens Next

• Everything seems fine, at first

• Then we start to notice problems

• Discipline deteriorates

• Planning becomes haphazard

• Some Task Breakdowns are late or missing for some Stories

• Teams end Sprints with some Stories partially completed

• Teams aren’t getting enough User Stories

• Product Owner can’t keep up with demand

• Teams make up things to do

Nice opportunity to keep technical debt low, but

Not the best use of time and resources

• Product Owner is overworked

• ScrumMaster can’t keep track of what is happening or

needs to happen, which leads to more problems and

dropped balls



Solution

• A ScrumMaster or Product Owner cannot support more

than three Teams!

• Introduce a second of each

• Now each ScrumMaster and Product Owner supports two Teams

• Best to pair them: Same SM and PO for a set of Teams

• Some things improve

• Discipline, effectiveness, quality improve

• Teams have enough Stories to fill their Sprints


Problems

• Other problems become clear

• Teams need things from other Teams that haven’t been

developed

• Confusion, delays, and disruption result

• Marketing department is frustrated at inability to get

advance notice of new features in time to develop

marketing campaigns



Create a Release Plan every Three Months

• A Release cycle is a set of Sprints

• Release Planning produces a Release Plan

• Estimates for Stories, Epics in Release

• Rough map of Stories to Sprints

• Dependencies between Stories, Teams

• Release Planning can be expensive

• 1—3 days for all Teams and members

• Value of Release Planning must justify the

cost

• Reduction in confusion due to planning cross-

Team dependencies

• Necessary to understand how external resources

may be engaged

• Required to plan for customer commitments


Set up for Release Planning

• Schedule Release Planning Meeting

• Before start of Release cycle

• As late as possible

Typical: 1—4 weeks in advance

• Allow enough time (1/2 – 3 days)

• Publish Agenda

• Prepare for Meeting

• Product Owners: Print Stories, Epics on paper or sticky notes

• ScrumMasters: Estimate Team Velocity for each Sprint in Release

• Others: Create presentations, etc.


How to do Release Planning

1. Have introductory presentations

2. Set up Release Planning board or table

1. Each Team has its own row

2. Sprint boundaries are drawn vertically

3. Conduct Release Planning session (2—4 hours)

1. Program Manager, Product Owners, ScrumMasters assist as

needed

2. Teams lay out Stories, Epics (“Items”) in their preferred sequence

3. Teams estimate Items not yet estimated

4. Teams map Items to Sprints

Stories do not cross Sprint boundaries, but Epics may

5. Teams collaborate to identify, sequence dependencies

Show dependencies with lines, masking tape, etc.

6. Teams collaborate to identify missing Items, create, and

incorporate them into the plan

4. Repeat Step 3 until done


The Flow of Release Planning

A1 A2 A3 A4 A5 A6 A7 A8

B1 B2 B3 B4

C1 C2 C3

B5 B6 B7 B8 B9 B10

C4 C5 C6 C7 C8 C9 C10 C11

EB1

EA1

yy Story: xx Epic:


Example Release Plan

Shows

Team capacity / Sprint

Stories per Team

per Sprint

Story sizes

Dependencies

Arrows, highlighting

Release Name: 3.1

Goal: Fulfill Orders

Start: 1-Jan-2012

End: 31-Mar-2012

Sprint A B C

1

Capacity 16 Capacity 26 Capacity 21

Story A1 3 Story B1 13 Story C1 13



2

Capacity 16 Capacity 21 Capacity 26




Team



• We’ve overcome our growing pains—again

• Our multiple-Team environment is developing products

smoothly, reliably



interest

• We’re planning dependencies and setting longer-term

expectations for other departments




• Right?




• We are kicking off our second product!

• BasRelief operates 3D printers to fabricate designs created

in GloCAD

• Integrates with GloCAD

• Customer often buy both

• We grow to 60 Team members

• Maximum Team size in Scrum is 9, so organize accordingly

• GloCAD

Five Teams, two ScrumMasters, two Product Owners

• BasRelief

Three Teams, one ScrumMaster, one Product Owner


New Problems Arise

• Product Owners are overstretched

• Need to write Stories for Teams, work with customers to identify

needs and solutions, and negotiate with each other about trade-

offs for development

• Cross-Team impacts become overwhelming

• Many Teams means many more cross-Team dependencies

N Teams have 𝑁 × (𝑁 − 1)/2 interfaces: Grows as 𝑁2.

• Scrum of Scrums helps, but isn’t enough

• Release planning helps, but is taking much longer

• We miss a key deadline

• Customers and management are not happy



New Role: Area Product Owner

Split Product-Owner responsibilities across two

Roles

Introduce one Area Product Owner per Product

• APO meets with customers to understand needs,

solutions

• APO meets weekly with Team Product Owners to build

understanding of what needs to be done

• APO provides guidance about tradeoffs

• APO participates in Release Planning

• Team Product Owners focus on writing Stories,

collaborating with Teams, and not on gathering

Customer requirements


How we Develop Cross-Team Requirements

• Product Owner “Scrum of Scrums” meetings

• Purpose: Develop requirements across Teams

• Frequency: As needed (weekly,…)

• Participants:

• Area Product Owner (facilitator)

• Team Product Owners

• Agenda:

• Each Team Product Owner describes

What I’ve done since last meeting

What I plan to do by the next meeting

What issues I need help to resolve

• All review status of Release work to date (Burn-Up

Chart!), make scope-change decisions

• Area Product Owner describes new big-picture

requirements

• All discuss, agree on follow-up actions

Team Product Owners write, revise Stories for Teams


New Role: Program Manager

Introduce one Program Manager per Product

• PgM facilitates Scrum of Scrums

• PgM facilitates Release Planning, records Release

Plans

• PgM tracks, manages cross-Team dependencies to

ensure they are satisfied

• PgM removes obstacles that impact multiple Teams

• PgM acts like ScrumMaster for a set of Scrum Teams



• We’ve overcome our growing pains—again

• Our multiple-Team, multiple-Product environment is developing

products smoothly, reliably



interest






• Right?



Changes

• Introduce new Area Product Owner, Program

Manager for new Teams

• Introduce Release Planning to Philadelphia office

• APO’s and PgM’s meet bi-weekly to identify,

negotiate cross-product dependencies

Decisions flow into Release Planning

• Investment in Jira, Confluence becomes even more

valuable


• We buy a small company in Philadelphia, Pennsylvania • They make a key add-on for GloCAD

• They have three Scrum Teams, with one SM and one PO

• Our California-based Scrum Teams need to collaborate with new people

in Philadelphia


Our World Now


New Problems: Hard to Collaborate at a

Distance!

From the Principles behind the Agile Manifesto:

We cannot collaborate at all until we solve some key problems!

Business people and developers must work together daily

throughout the project.

The most efficient and effective method of conveying information

to and within a development team is face-to-face conversation.

Agile processes promote sustainable development. The sponsors,

developers, and users should be able to maintain a constant pace

indefinitely.


Why Co-Location is Preferred

• Members work together easily throughout day

• Proximity encourages interaction

• Information propagates rapidly

• Communication is Osmotic

• Members absorb information from questions, answers in

background

• Members can chime in if they have something to contribute

• Agile projects favor in-person communication over

documentation

• Co-location encourages and enables good communication

• Distribution impairs it, requires more documentation


The Team Room: What Co-Location Looks Like Shown

Protected work

area

Room for Daily

Stand-up

Large whiteboard

Information

Radiators (Sprint

Status)

Comfortable

furniture

Not Shown Projector

Large monitors

Avoid

Speakerphones

Through traffic

People in the Room who aren’t on the Team


Distributed Teams: Best Case

• Scrum Teams are

distributed

• Scrum Team members are

co-located per Team

• Compared to total co-

location

• Intra-Team communication

is the same

• Cross-Team communication

somewhat more difficult, but

not hard

• Cross-Team work synced

via “Scrum of Scrums”

meetings


Solutions for Collaboration

• Hold distributed meetings during joint working hours

• 9 AM - 2 PM PST, 12 Noon - 5 PM EST

• Use video (Skype, GoToMeeting, WebEx, etc.) for all distributed

meetings (e.g., Scrum of Scrums), wherever possible.

• Use phone or equivalent only when video is not possible.

• Use chat (Skype, HipChat, etc.) for individual real-time Q&A

between Team members, others, across the country

• Use email when real-time communication is not needed, or not

possible. This is the last choice, not the first!



• We’ve overcome our growing pains–again

• Our multiple-Team, multiple-Product, distributed environment is

developing products smoothly, reliably



interest






• Right?




• We buy a small company in Dallas, Texas

• They make a key add-on for BasRelief

• New challenges encountered with the new people

• They have one very informal development team, with no Scrum

Roles

• The development people are very junior

• Their code quality and design are poor and won’t scale well

• They have much “technical debt” because they’ve never

implemented automated testing

• It is important to improve the add-on’s code quality,

design, and test automation now, before we add many new

features

• But the people in Dallas don’t know how to do these things!


How We Organize

• Create a new Scrum Team, containing Dallas people, with

focus on building their technical expertise and paying

down the technical debt ASAP

• Five Dev and QA people in Dallas

• Two senior developers, one test-automation expert in Palo Alto

• Product Owner in Dallas

• ScrumMaster in Palo Alto

• Introduce collaboration practices

• All Team meetings held in common working hours

9 AM - 3 PM PST, 11 AM - 5 PM CST

• All Team meetings use video

• Scrum Ceremonies conducted as usual, but via videoconference

• Scrum of Scrums, Release Planning conducted as usual, via

videoconference

• Usual techniques for real-time communication


Our World Now


How Well do They Perform?

• Things go well in Palo Alto

• Things do not go well in Dallas, where Team members

• Often do not update Task status in Jira, so we can't track progress

• Work on things that have low priority, and don't work on things that

have high priority

• Do not collaborate to complete Stories quickly (with each other or

Palo Alto members), but default to having one Team member work

on each Story

• Often come late to, or fail to participate in, our key Scrum meetings



Solution

• Introduce new Role: The ScrumMaster Proxy (SMP)

• ScrumMaster is in Palo Alto

• Does usual ScrumMaster things

• ScrumMaster Proxy is in Dallas

• Does 80% of what a ScrumMaster does, for the people in Dallas

Pays close attention to who is doing what

Redirects people to the right things when they are focusing on the

wrong things

Enforces our process and policies (including Task-status updates to

enable tracking)

Removes obstacles to Team productivity

• Has daily synchronization call with ScrumMaster in Palo Alto

• On-site presence of the SMP improves effectiveness of

Dallas office dramatically


Our World Now




• Our multiple-Team, multiple-Product, distributed

environment is developing products smoothly, reliably

• We’re handling cross-Team and distributed intra-Team

issues effectively


interest






• Right?



• New challenges encountered with the new

people

• They use a Waterfall process

• 15 Developers are in New York. 8 QA people

are in Shanghai.

Twelve time zones apart!

• No possibility of changing distribution of

people soon

• It is important fold new folks into our

Scrum process

• But geographic distribution makes this difficult


• We buy a company with offices in New York, NY, and Shanghai, China

• They make a solid-modeling program called Densify

• Densify performs physical simulations of GloCAD designs to validate them and

reduce need for building costly prototypes


What we do Next

• Train new people in our Scrum process

• Define three Scrum Teams

• Each has a product-area focus

• Each Team has some NY developers and Shanghai QA people

• Two ScrumMasters, two Product Owners in New York

• Program Manager, Area Product Owner in New York

• Two ScrumMaster Proxies in Shanghai

• Plan, launch Scrum process

• On-site coaching in both offices shortens path to proficiency


How Well do the New Teams Perform?

• Growing pains are expected

• Some get better

• Some do not

• Scrum Team members, ScrumMaster, and Product Owner cannot

sustain overlapping working hours, for meetings or real-time

collaboration.

Initial attempt to do this fell apart quickly. Impact threatened burnout

and possible loss of people who may quit and look for other jobs.

• Team members in Shanghai, who focus on QA work, cannot get

clarity on requirements from the Product Owner when they need it

(now), and have at best a one-day response time to their

questions.

If questions lead to more questions (often the case), resulting back-

and-forth email messages may take several days to provide the

necessary clarity. Progress is very slow, as a result.



Solution

• Introduce new Role: The Product Owner Proxy (POP)

• Product Owner is in New York

• Does usual Product Owner things

• Product Owner Proxy is in Shanghai

• POP gives real-time guidance about requirements to local people

• PO and POP write daily summary of decisions, actions for each

toher

• PO and POP have daily 15-minute call to synchronize

understanding

• POP may be wrong sometimes

Quick answers that are right 80% of the time are better than perfect

answers that take one or more days to get

Cost of occasional re-work is much smaller than cost of delay


Our World Now


Results

• On-site presence of the POP improves productivity, morale

of Shanghai office dramatically

• …but splitting Teams this way will always be costly

• More documentation, less real-time communication

• Latency issues (time from question to answer) can be minimized,

but will always be significant

• Impact on productivity, quality of life likely to outweigh perceived

cost-reduction benefits of “junior offshoring model”


In Summary

• Introduce Scrum of Scrum, Release Planning, Area Product

Owner, Program Manager as organization grows

• Don’t split Scrum Teams, if at all possible

• If organization is distributed, keep Teams co-located in different

offices

• This is a very reasonable and effective strategy

• If you can’t keep logically-organized Scrum Teams co-located

• Always have a ScrumMaster Proxy for each Team fragment not co-

located with the ScrumMaster

• Consider providing a Product Owner Proxy if Product Owner cannot

supply rapid turnaround of questions for remove Team members

• Use communication and collaboration tools

• Use effective patterns for Scrum Ceremonies per type of

distribution


Distributed Teams: Best Practices for Meetings

Working

Time

Sprint

Planning

Daily Stand-

Up

Sprint

Review

Retro-

spective

Comment

Full overlap All attend All attend All attend All attend Co-located

Full overlap All attend All attend All attend All attend Designate

one SM

proxy (SMP)

always, one

PO Proxy

(POP) as

needed, per

location for

distributed

Teams

Partial

overlap

All attend All attend All attend All attend

Adjacent All attend All attend All attend All attend

Far apart All attend Sub-groups,

SMPs,

POPs meet.

SMPs, SM

provide all

findings to

full Team.

Sub-group

nearest to

PO demos

Sub-groups,

SMPs,

POPs meet.

SMPs, SM

provide all

findings to

full Team.

And / Or:

Full Team

meets twice

/ week

And / Or:

Rotate

demo to PO

among sub-

groups


Common Reporting Structures - 1

PgM: Program Manager

USM: Uber ScrumMaster

PMO: Project / Program

Management Office

APO: Area Product Owner

CPO: Chief Product Owner

PMM: Product Management &

Marketing


Common Reporting Structures - 2

PgM: Program Manager

USM: Uber ScrumMaster

PMO: Project / Program

Management Office

APO: Area Product Owner

CPO: Chief Product Owner

PMM: Product Management &

Marketing


Final Thoughts on Geographic Distribution

• Geographic distribution is not a good way to do Scrum

• …because it is not a good way to do work in general

• Scrum is a good way to do geographically-distributed software

development

• No advantage to other processes, since all processes suffer similarly

when spread around the globe




• Our multiple-Team, multiple-Product, globe-spanning

environment is developing products smoothly, reliably

• We’re handling cross-Team and distributed intra-Team issues

effectively


interest






• Right?


Time will Tell

Because our journey stops here

10,000 miles away: developing with a distributed...

Documents