distributed release management
DESCRIPTION
Full Stack Engineering Meetup in NYC, May 27, 2014.TRANSCRIPT
![Page 1: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/1.jpg)
Distributed Release Management Deploying etsy.com 40+ times per day
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks
![Page 2: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/2.jpg)
1st Day Assignment Put your face on etsy.com/about
![Page 3: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/3.jpg)
What I’m showing you tonight is the result of four years of iteration.
![Page 4: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/4.jpg)
Small incremental changes to the application “Dark” features: new classes, methods, controllers Graphics, stylesheets, templates Copy/content changes !
App deploys
Turning flags on, off, or % ramp up
Config deploys
![Page 5: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/5.jpg)
Latent bugs and security holes Traffic management, load shedding Adding and removing infrastructure !
Tweaking config flags or releasing patches.
“Operating” the site
![Page 6: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/6.jpg)
IRC, #push
![Page 7: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/7.jpg)
/topic mbrittain | jgoulah | rsnyder | ekastner
![Page 8: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/8.jpg)
/topic mbrittain, jgoulah, rsnyder | ekastner
![Page 9: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/9.jpg)
Keep real people in the loop
Queue, with max batch size of seven.
Automated deployment run by humans
![Page 10: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/10.jpg)
4 people in this deploy.
“I’ve pushed my changes to master.”
“Everyone has checked in.”
![Page 11: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/11.jpg)
Build QA and Pre-prod
Build progress
Status in #push
Git SHA1 in for each env.
Date, username, deploy log, changeset, link to dashboard from time of deploy
![Page 12: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/12.jpg)
![Page 13: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/13.jpg)
Reporting what’s going on in Deployinator, and who triggered
Status from build cluster
![Page 14: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/14.jpg)
Pre-prod (“princess”) has been deployed. !
SHA1 of the change Time it took to deploy Link to changeset in GitHub Log of the deploy script
![Page 15: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/15.jpg)
Btw, there are three bots talking in channel at this point. O_o
![Page 16: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/16.jpg)
Queuing for next deploy
Humans talk to other humans from time to time.
![Page 17: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/17.jpg)
Talking to pushbot. !
Pushbot knows some Spanish… because, ya know, why not?
![Page 18: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/18.jpg)
Link to test results for CI environment, along with how long the tests took.Alerting by name.
![Page 19: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/19.jpg)
8 minutes have elapsed… We’ve built and tested our release in the CI environment (“QA”). !
QA build failed our 5 min. SLA for tests.
![Page 20: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/20.jpg)
“Try” is our pre-commit testing cluster.
![Page 21: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/21.jpg)
Bots help reinforce our values. This is especially helpful for new people on the team.
![Page 22: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/22.jpg)
![Page 23: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/23.jpg)
Still 8 minutes elapsed… Pre-prod has been deployed and tested. !
This ran in parallel with our QA build and tests.
![Page 24: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/24.jpg)
![Page 25: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/25.jpg)
Cross-traffic: In a separate channel (#config), our app configs files were deployed to pre-prod.
![Page 26: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/26.jpg)
![Page 27: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/27.jpg)
![Page 28: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/28.jpg)
![Page 29: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/29.jpg)
![Page 30: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/30.jpg)
Cross-traffic: Ops team deployed a configuration change.
And, yes… another non-human.
![Page 31: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/31.jpg)
![Page 32: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/32.jpg)
Code is live Link to dashboard.
![Page 33: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/33.jpg)
![Page 34: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/34.jpg)
13 minutes elapsed… Code is now in production with public traffic.
![Page 35: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/35.jpg)
Who committed code in the last deploy? And how many lines did each of them change?
![Page 36: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/36.jpg)
![Page 37: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/37.jpg)
![Page 38: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/38.jpg)
Handoff for the next deploy.
![Page 39: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/39.jpg)
Entire app deploy took 15 minutes. !
4 people running the deployment 8 committers Config deploy and Chef change deployed in parallel.
![Page 40: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/40.jpg)
Optimal queue size
Normalized communication
Improved visibility
Historical record is ideal for post-mortems
Organic evolution
![Page 41: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/41.jpg)
Hold up the queue (.hold)
Work the issue with the people available in #push
Additional help always available in #sysops
Buddy-system for off-hours deploys
Ops-on-call, dev-on-call
When something goes wrong?
![Page 42: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/42.jpg)
25 Million Items listed 60+ Million Monthly unique visitors 200 Countries with annual transactions !
175+ Committers, everyone deploys
Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
![Page 43: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/43.jpg)
@mikebrittain
DEPLOYMENTS PER DAYAPP CODE CONFIG FILES
![Page 44: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/44.jpg)
Start small. (We did.)
Automated tests and production monitoring.
Have a story around maintaining quality.
“We can always go back to the old way.”
Demonstrate value to leadership.
![Page 45: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/45.jpg)
Go write your own story.
![Page 46: Distributed Release Management](https://reader034.vdocuments.site/reader034/viewer/2022052505/554bb16fb4c905ae618b591d/html5/thumbnails/46.jpg)
Thank you.
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks