from code to the monkeys: continuous delivery at netflix
Post on 17-Oct-2014
1.698 views
DESCRIPTION
At Netflix, we continue to improve upon our continuous delivery process. We thrive in a hybrid environment, where every developer is able to deploy code, and with that freedom comes the responsibility for ensuring that our customers are not negatively impacted. We have constructed Open Source tools toward a Continuous Delivery solution. In this presentation, from QConSF 2013, you will learn about our tool chain so that you can determine which make sense in your environment.TRANSCRIPT
BUILD
BAKE
DEPLOY
Continuous Delivery at Netflix: From Code to the Monkeys
Dianne MarshQCon San Francisco, November 2013http://www.linkedin.com/in/diannemarsh
BUILD
BAKE
DEPLOY
Netflix Goals
High Availability
But … Move fast
Tools encourage Best Practices
But … Freedom to do the right thing
Teams Deploy Their Own Code
Run What You Wrote Rapid Innovation Rapid Detection Rapid Response
= Freedom + Responsibility
http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013
BUILD
Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code
BUILD BAKE
BAKE
Aminator• Create AMI from Base AMI• Image contains service and
everything needed to run it• Unit of Deployment for Test and
Prod• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
BUILD BAKE DEPLOY
DEPLOY
Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push
http://www.infoq.com/presentations/asgard
CANARY ANALYSIS
Test, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue
Asgard Developer Portal
GLISTEN
Extending Asgard’s WorkflowAutomated Red/Black Push
Test, Int, Prod stacksRun canary/analysisScale up new instancesTurn on traffic to new ASGRun more testsTurn off traffic to old ASGWait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
BUILD
BAKE
DEPLOY
Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey• (and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
One Button Deployment?
Regional Isolation
Limit Impact of Human Error Stagger deployments Canary testing per
region
Multi-Region Consistency
Build Tooling to: Schedule
Deployments Prefer off peak Choose next available
region automatically Provide high visibility
per region
Send in the Conformity Monkey
http://techblog.netflix.com/2013/05/conformity-monkey-keeping-your-cloud.html
Have deployments diverged? Balance regional consistency
with regional isolation Provide meaningful
thresholds Build best practices into
tooling and reporting
Clean up with Janitor Monkey
Disassociate unused EIPs
Delete unassociated Amazon EBS volumes
Delete older Amazon EBS snapshots
Leverage Amazon S3 Object Expiration
https://github.com/Netflix/SimianArmy/wiki/Janitor-Home
Key Elements for Netflix
Value Self-service
Test Everywhere
Build Awareness of Multiple Regions
Avoid peak times
Roll back quickly and easily
Be Cloud Native
Put NetflixOSS to Work for You
Netflix Platform
AMINATOR
Glisten
** And 30+ more projects at http://netflix.github.io/
Keep the Conversation Going
Continuous Delivery Open Space
Ballroom B/C (here!)
1:35-2:25, immediately following lunch