etsy leanstartup presentation at sxsw 2013

7/29/2019 Etsy LeanStartup presentation at SXSW 2013

1/46

Continuous Deployment at

March 9, 2013

Ross Snyder

[email protected]

@beamrider9

:A Tale of Two Approaches
mailto:[email protected]:[email protected]:[email protected]


2/46

A quick primer on


3/46

is:

The global marketplace we make together.


4/46

is:

The premier destination for handmade goods,

vintage items, and craft supplies.

simplertimestoys lacklusterco norwesterseaglass


5/46

quick facts:

22+ million members

800,000+ active shops

18+ million items currently for sale

20 cents to list item, 3.5% transaction fee

400+ employees (majority in Brooklyn)

(as of March 2013)


6/46

$0

$200

$400

$600

$800

$1,000

2005 2006 2007 2008 2009 2010 2011 2012

Gross Merchandise Sales ($MM)

Since opening its doors in June 2005, Etsy

has grown virtually non-stop.


7/46

A nice problem to have:

Our site is sosuccessful, how can we

move fast enough to

keep up with demand?


8/46

CONTINUOUSDEPLOYMENT


9/46

:The Early Years

(2005 - 2008)


10/46

: The Early Years

1. Spend significant time writing code


11/46

: The Early Years

2. Painful source control merge

1. Weeks writing code


12/46

: The Early Years

3. Hand off to someone else to deploy

1. Weeks writing code2. Painful merge


13/46

: The Early Years

4. Deploy, site goes down


3. Hand off to deployers


14/46

: The Early Years

5. Roll back deploy



4. Deploy, site down


15/46

: The Early Years

6. Spend hours (days?) fixing bugs



4. Deploy, site down5. Roll back deploy


16/46

: The Early Years

7. Go back to step 2



4. Deploy, site down5. Roll back deploy

6. Fix bugs


17/46

: The Early Years

WATERFALL!


18/46

: The Early Years

Pros:

Early Etsy engineers

used this releasecycle to bootstrap the

marketplace from

nothing.

Forever grateful.


19/46

: The Early Years

Large changesets

Infrequent deploys

Weak confidence in deploy success

Significant time spent deploying Low ability to experiment/iterate/react

Developer stress/unhappiness

Cons:


20/46

: The Early Years

By late 2008, Etsy is still a startup, but has thedeploy process of a much bulkier company.

Popularity is on the verge of outpacing capacity.


21/46

:

Today


22/46

: Today

1. Small changesets, deployed frequently


23/46

: Today

2. Engineers deploy the site

1. Small changesets


24/46

: Today

And not just engineers, but also:

Designers

Product Folks

Upper Management

Board Members

Dogs


25/46

: Today

3. Deploys are fast and near-effortless

1. Small changesets2. Engineers deploy


26/46

: Today

4. Most changes behind config flags(safer deploys)


3. Deploys are fast


27/46

: Today

5. Graphs/metrics to assess deploy


3. Deploys are fast

4. Changes behind flags


28/46

: Today

6. If issues, fix immediately & roll forward


3. Deploys are fast

4. Changes behind flags5. Copious graphs/metrics


29/46

This isnt license to break stuff, quickly.

Engineer-driven QA and solid unit testing

are integral parts of the process.

: Today


30/46

: Today

7. Repeat 25+ times per day, every day


3. Deploys are fast

4. Changes behind flags5. Copious graphs/metrics

6. Fix fast & roll forward


31/46

Then:

1. Small changesets

2. Engineers deploy

3. Deploys are fast

4. Changes behind flags

5. Copious graphs/metrics

6. Fix fast & roll forward

1. Weeks writing code

2. Painful merge3. Hand off to deployers

4. Deploy, site down

5. Roll back deploy

6. Fix bugs, go to step 2

Now:


32/46

Etsy Deploy Stats: 2012

Deployed to production 6,419 times

On average, 535/month, 25/day Additional 3,851 config-only deploys

196 diff

erent people deployed to prod Nov/Dec 2012: deployed 752 times


33/46

Why does it work?


34/46

N = # of deploys

P = probability of site degradation

S = average severity of degradation

T = time to detect/resolve

= N * P * S * TExpected

Downtime

Continuous Deployment Math


35/46

N = 1

P = 0.5

S = 0.7

T = 100

Before:

E.D. = 35

N = 250

P = 0.1

S = 0.05

T = 5

Now:

E.D. = 6.25

N = # of deploys

P = prob. of degradation

S = avg. severity of degradation

T = time to detect/resolve

(all numbers completely arbitrary)

Continuous Deployment Math


36/46

Etsy circa 2013 (400+ employees)

acts, in some ways, more like a startup than

Etsy circa 2008 (40+ employees).

Big Takeaway


37/46

Continuous Deployment makes possible:

Continuous Experimentation

http://etsy.me/continuous-experimentation
http://github.com/etsy/deployinatorhttp://github.com/etsy/deployinatorhttp://github.com/etsy/deployinator


38/46

Continuous Experimentation

1. Small changes

2. Run experiment (A/B test)

3. Analyze data

4. Re-examine assumptions

Repeat continuously in pursuit

of larger goals.


39/46

Heard since 2010:Neat experiment, but

this will never scale.

As of 2013, Etsy has

100+ engineers -still going strong.


40/46

Some Etsy Customizations

Deploying is a first-class feature. Inability todeploy is a P1 incident (same as site down).


41/46


Internal admin site

API

Big data

Search Blog

Deployinator itself

We continuously deploy not just the main

Etsy website, but as much as possible:


42/46


Database schema changes

PCI-DSS environment (credit cards)

In the rare case we cant continuously

deploy, we create alternative tools:

We do continuously deploy as much of our

payment processing as is safe & legal (98%).


43/46


Keeping deploys fast is paramount and worththe investment in manpower & hardware.


44/46


Continuous deployment is all about moving

forward, sometimes at the expense of the past.

Our solution: engineering-wide bug rotation,

one day a month, every engineer participates.


45/46

Continuous Deployment is a fantastic

recruitment tool for attracting engineers

who like to move fast and get stuff done.

Fun Fact:


46/46

Learn more:http://codeascraft.etsy.com/

Join the fun:http://www.etsy.com/careers

Etsy open source (Deployinator, StatsD)http://etsy.github.com/
http://etsy.github.com/http://etsy.github.com/http://www.etsy.com/careershttp://www.etsy.com/careershttp://codeascraft.etsy.com/http://codeascraft.etsy.com/

etsy leanstartup presentation at sxsw 2013

Documents