Queues: An Invisible Money Drain

Phil Sarin VP Engineering, GameChanger Media

@philsarin http://216ways.net

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.

— Mark Twain

Things I once believedYou shouldn’t release often because releases have high overhead.

You need to track all of your bugs.

Slack is bad. Your team should be busy.

Specialists will always be more productive than generalists.

The Biggest Misunderstanding: We think queues are free.

What we’ll cover tonightWhat a queue is

Identifying the queues in your organization

Tactics for handling queues

Some war stories

Part 1: Queues and why they matter

Flickr: Jeff Turner

“Queues are the root cause of the majority of economic waste in product development.”

— Donald G. Reinertsen

Examples of queuesFeature QueuesWaiting to get designed Waiting to get developed Waiting for testing Waiting to get released Waiting to get validated

Bug QueuesWaiting for triage Waiting to get fixed Waiting to get verified Waiting for release

What’s wrong with big queues?Big queues increase cycle time and delay cost

Big queues increase variability

Big queues require management

Big queues delay feedback

Big queues hurt morale

The conservative team vs

The aggressive team

Variability

Flickr: State Farm

Flickr: State Farm

Flickr: State Farm

Flickr: State Farm

Batch Sizes

Flickr: kbeil

1 year

3 months

vs

3 months 3 months 3 months

vs

Why small batches are (usually) better

Realize benefits sooner

Faster feedback

Higher motivation

Less schedule slippage

Old School: Releases are expensive so let’s batch up work for release.

New School: Let’s drive down the cost of releases so that we can release more often.

Queue takeaways

Queues introduce delay.

Queues have economic and psychological costs.

High capacity utilization drives up queue size.

Small batches can reduce the cost of queues.

Part 2: How we apply this theory at GameChanger

Flickr: Jeff Turner

How we’ve managed queues

1. Swarm

2. Generalize queue handling

3. Obliterate queues

4. Reduce testing batch sizes

1. Swarm

Flickr: State Farm

The stages of swarming

1. Skepticism/suspension of disbelief

2. Guilt and busywork

3. Scope blows up

4. The “surge capacity effect”

Announcements

Invite Tools

Alerts

Announcements Invite Tools Alerts

vs

No silver bullet

2. Generalize

Flickr: Kaleb Fulgham

Multiple queues, 1 server each

Single queue, Single server

Single queue, pool of servers

Multiple queues, 1 server each

Single queue, Single server

Single queue, pool of servers

Sustained huge queuesare least likely

Pressures to specializeA specialty differentiates your business

“I don’t want to touch that code!”

Initial builder owns it forever.

Early adopters become de-facto specialists.

Some people want specialist careers.

Resisting specialist queuesOutsource non-core specialties

Despecialize

Challenge someone

Staff up

Generalist teams and morale

3. Obliterate queues

http://provocateurs.ca/2014/11/16/delmar-dont-be-afraid-to-hit-delete/

4. Reduce testing batch sizes

That time we missed our deadline by four months

2 months development 2 weeks testing

2 months development 2 weeks testing 4 months bug-driven development

2 weeks dev test 2 weeks

dev test 2 weeks dev test 2 weeks

dev test 2 weeks dev test

Can we reduce this batch size further?

Our first batch size reduction

2 weeks dev test 2 weeks

dev test 2 weeks dev test 2 weeks

dev test 2 weeks dev test

Can we reduce this batch size further?

2 weeks dev+test T 2 weeks

dev+test T 2 weeks dev+test T 2 weeks

dev+test T 2 weeks dev+test T 2 weeks

dev+test T

Continuous pre-release delivery?

It didn’t work

Nobodyused these

How we fixed it

More automated test coverage

Instituted small batch manual testing process

Hired more testers

Release Days5.15 215.14 145.13 265.12 215.11 235.10 45.9 665.8 75.6 21

It worked, mostly, eventually

Parting Thoughts

Flickr: Jeff Turner

Things I once believedYou shouldn’t release often because releases have high overhead.

You need to track all of your bugs.

Slack is bad. Your team should be busy.

Specialists will always be more productive than generalists.

Mindset shifts

Being busy vs being productive

Individual vs team

Rituals vs principles

Takeaways

Queues are real and expensive

You should know where your queues are

There are a bunch of ways to reduce the cost of queues

Further Reading

The Principles of Product Development Flow, Donald G. Reinertsen

“Software Inventory” by Joel Spolsky: http://www.joelonsoftware.com/items/2012/07/09.html

“How we fixed more bugs by deleting our bug DB” and “Building around generalists” on my blog (216ways.net)