continuous deployment at etsy — timesopen nyc

Principles and Practices in Continuous Deployment

Mike Brittain Engineering Director, Etsy @mikebrittain

July 22, 2014

“Continuous Deployment”Process by which our team deploys software changes to production services over 30 times per day.

The Release Hurdle2-3 weeks of code changes per release

Highly-orchestrated releases (6+ hours)

^ Same for rollbacks

Scheduled downtime

Long, stressful days

Slow iterations

pro·duc·tion [pruh-duhk-shuhn] (n)

1. This complex system of application code, distributed services, servers, networking gear, etc., upon which we’re going to try to carefully apply a complicated set of changes and hope that nothing goes wrong. Cross your fingers… here goes.

Software for large-scale web sites has been traditionally written by one group of people, then released and operated by a different group.

These two groups have very different levels of visibility into how the software works.

Stagnation

“…frequent and prolonged outages.”2010 CAPACITY PLAN

First, PrinciplesEnable rapid product innovation Resolve scaling hurdles Reduce mean-time-to-recovery Grow a talented and engaged engineering team Minimal bureaucracy and process

http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

In a software release process Fail Fast means releasing undeployed code as fast as possible, instead of waiting for a weekly release to break.

http://youtu.be/LdOe18KhtT4

Continuous Deployment

Continuous Delivery~ vs ~

Frequent check-ins directly to mainline.

Continuous Deployment Continuous Delivery✓ ✓

Continuous Integration and Automated tests.


Keep the build green. We’re always ready to release.


“One button” deploys.


Business dictates when a build is deployed.

Continuous Deployment Continuous Delivery✓

Every passing build is deployed to production.

Continuous Deployment Continuous Delivery✓

All enhancements are gated by Config Flags. (“Branch in code”)

Continuous Deployment Continuous Delivery✓ ?

Most of the builds we deploy are “dark” changes.CSS rules, template copy, un-referenced code (classes, functions, templates), code paths behind disabled flags, etc.

Dev Team Version Control Build & Unit Tests

Automated Acceptance Tests

User Acceptance Tests Release

Check inTrigger

Feedback

Source: http://en.wikipedia.org/wiki/Continuous_delivery

Continuous Delivery release pipeline




Check inTrigger

Feedback

Check inTrigger

Feedback Trigger

Feedback






Check inTrigger

Feedback

Check inTrigger

Feedback Trigger

Feedback

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback






Check inTrigger

Feedback

Check inTrigger

Feedback Trigger

Feedback

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback


Dev / Integration Staging Production




Check inTrigger

Feedback

Check inTrigger

Feedback Trigger

Feedback

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback



Assumptions:

Deploy process is infallible.

Staging is a perfect reflection of Production, with respect to hardware, configurations, data, overall load, capacity, etc.

“What do you mean, ‘it’s not working in production?’ I TESTED IT BEFORE WE RELEASED!”

(DUN DUN DUUUUHHHNNN !!!)




Check inTrigger

TriggerApproval



Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval




Check inTrigger

TriggerApproval



Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Approval

Check inTrigger

TriggerApproval

Feedback

DUN DUN DUUUUHHHNNN !!!




Check inTrigger

TriggerApproval


Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Approval

Check inTrigger

TriggerApproval

Feedback

"Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong , so you can spend more time building features.” !

—ThoughtWorks !

!http://www.thoughtworks.com/continuous-integration


http://www.thoughtworks.com/continuous-integration




Check inTrigger

TriggerApproval


Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Approval

Check inTrigger

TriggerApproval

Feedback


Where’s the bug? !

In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests? !Data out of sync? Server configurations out of sync? Capacity vs. current load? Deployment script?




Check inTrigger

TriggerApproval


Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Check inTrigger

TriggerApproval

Approval

Check inTrigger

TriggerApproval

Feedback


How will we know when something is wrong in production? !

How long will it take to resolve the issue?

Check inTrigger

We aim to reduce fundamental surprise in every release, and we optimize for detecting and recovering from failures quickly.

Sarabbit on Flickr:

“Quality is not just testing pre-release. It also includes our adaptability and response time.”

- Jeff Sussna at ALM Forum, 2014

Pre-production validationCode deployed to de-pooled application (web) servers touching prod services and databases.

Smoke tests

Integration tests

Functional tests

User-Acceptance (ad hoc, targeted)

Production validationExactly the same server configs, services and data as pre-prod, but this is where we introduce application code to live traffic.

Smoke tests (esp. over public hostnames)

User-Acceptance testing behind config flags

Gratuitous monitoring

Customer support and forums

Monitoring

Monitoring

PHP Warnings Bug Reports and Help Requests

Deploy logs

Check inTrigger

Feedback Trigger

Feedback Approval

Approval

Smoke Tests User Acceptance!Tests ReleaseDev Team Version Control Build & Unit

TestsAutomated

Acceptance TestsUser Acceptance

Tests Deploy (Prod) Monitoring and Automated Alerts

Continuous Deployment release pipeline

Feedback

Dev Pre-Production (“Princess”)

Production

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback


TestsAutomated




CI


Production

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback


TestsAutomated




CI

ApprovalApproval

FeedbackFeedback

Feedback


Production

Check inTrigger

Feedback Trigger

Feedback Approval

ApprovalFeedback


TestsAutomated




CI

ApprovalApproval

FeedbackFeedback

Feedback ApprovalFeedback Approval

Feedback

“Allow buttons properly to inherit color from their parent node.”

The Release Hurdle2-3 weeks of code changes per release

Highly-orchestrated releases (6+ hours)

^ Same for rollbacks

Scheduled downtime

Long, stressful days

Slow iterations

@mikebrittain

Very end of 2009 Today

DEPLOYMENTS PER DAYAPP CODE CONFIG FILES

First, PrinciplesEnable rapid product innovation Resolve scaling hurdles Reduce mean-time-to-recovery Grow a talented and engaged engineering team Reduce bureaucracy and process

Admin-launch and whitelist

Ramp-up public traffic

date":"30\/Mar\/2014:12:49:48","locale_currency_code":"USD","pref_language":"en-

US","region":"US","detected_currency_code":"USD","detected_language":"en-

US","detected_region":"US","accept-languages":"en-US","cdn-

provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0"

,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst

antRequest":"0","isMozPrefetchRequest":"0","listing_ids":

[104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11

4767467,117266897,157579748],"scheduled_modules_content_ids":

[10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte

nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl

ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in

ternational.languages.ja;international.languages.no;international.languages.pl;internationa

l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_

toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r

ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo

ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c

srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_encAnalytics connected to config names

Observed impact

Time series data for duration of the experiment

“Catapult”

FrankProduct Manager

“I want to find out whether buyers will favor a single price for the product that includes shipping.”

https://www.etsy.com/shop/lucra

Eligibility requirements:

- Must be first page of visit

- Buyer & seller in same region

- etc…

Time: < 8 hours Staff: One !

Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.

We do not bundle the item price and shipping cost together today. !

https://www.etsy.com/shop/lucra

Ambitious Product Goal

Monolithic Building and measuring many things at once.

Ambitious Product Goal

Monolithic Building and measuring many things at once.

Iterative One thing at a time, our design goal is always in sight.

Time: < 8 hours Staff: One !

Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.

Deployed Deployed

http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

“Maybe this is just viable for a single developer … your site will be down. A lot.”

~ Informed reader

etsystatus.com

@mikebrittain

Very end of 2009 Today

DEPLOYMENTS PER DAYAPP CODE CONFIG FILES

$1.35 Billion Goods sold in 2013 60+ Million Unique visitors per month !

200+ Committers, everyone deploys

http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet

http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/

Thank you. Mike Brittain Engineering Director, Etsy @mikebrittain

July 22, 2014

mikebrittain.com/talks

continuous deployment at etsy — timesopen nyc

Engineering

trigger feedback

config flag

etsy mikebrittain

config flag

eligibility

automated

orchestrated

6 hours