Continuous Delivery at NetflixSpeed at Scale

Monday, December 23, 13

Rob Spieldenner

Senior Software Engineer on Engineering Tools at Netflix

@robspieldenner -> Twitter

Monday, December 23, 13

Netflix Culture

Monday, December 23, 13

Culture Deckhttp://jobs.netflix.com/

Monday, December 23, 13

Freedom & Responsibility

Monday, December 23, 13

Context, not Control

Monday, December 23, 13

Self Service

Monday, December 23, 13

Developers on call for their services

Monday, December 23, 13

Netflix OSShttp://netflix.github.io/

Monday, December 23, 13

OODA Loop - Observe, Orient, Decide, Act

Monday, December 23, 13

Commit Unit TestBuild Bake Deploy To Test

Integration & Acceptance

Tests

Red/Black Push

Continuous Delivery Pipeline

Monday, December 23, 13

Commit and Code Review

Monday, December 23, 13

Build

simplify build file with plugins

add metadata

calculate version

output jar/war, sources, javadoc, rpm/deb

Monday, December 23, 13

Ant

build.xml<project name="example"> <property environment="env"/> <import file="${env.CBF_HOME}/library.xml"/></project>

ivy.xml

list dependencies

list publications

Monday, December 23, 13

Gradle

build.gradleapply plugin: ‘nebula’publications { JAR JAVADOC SOURCES}dependencies { testCompile ‘junit:junit:4.+’}

Monday, December 23, 13

Jenkins

Monday, December 23, 13

Job-DSLhttps://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

Monday, December 23, 13

Job DSL Example

exampledef project = 'quidryan/aws-sdk-test'def branchApi = new URL(“https://api.github.com/repos/${project}/branches”)def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())branches.each { def branchName = it.name job { name “${project}-${branchName}”.replaceAll(‘/’, ‘-’) scm { git(“git://github.com/${project}.git”, branchName) } steps { gradle(‘test’) } }}

Monday, December 23, 13

Bake

Monday, December 23, 13

aminatorhttps://github.com/Netflix/aminator

Monday, December 23, 13

chef for base image

Monday, December 23, 13

Apply deb/rpm, snapshot

Monday, December 23, 13

Unit of deployment

Monday, December 23, 13

Deploy

Monday, December 23, 13

Asgardhttps://github.com/Netflix/asgard

Monday, December 23, 13

Monday, December 23, 13

UI or REST API

Monday, December 23, 13

Canary Deploys

Monday, December 23, 13

Red / Black Push

Monday, December 23, 13

Testing

Webdriver

Geb

Betamax

and manual

Monday, December 23, 13

Monitoring

How many people are watching?

Can the new version handle load?

Errors and logging

suro, jmx

Monday, December 23, 13

Alert on issues

Monday, December 23, 13

Maybe automatic rollback

Monday, December 23, 13

Resiliency / Anti-Fragility

Monday, December 23, 13

Simian Armyhttps://github.com/Netflix/SimianArmy

Chaos Monkey

Chaos Gorilla

Chaos Kong

Latency Monkey

Janitor Monkey

Conformity Monkey

Monday, December 23, 13

Assume services will break, disappear, slow

Monday, December 23, 13

Books

Monday, December 23, 13

Adrian Cockcroft Flowcon Keynote - Velocity and Volume (or Speed Wins) - http://www.youtube.com/watch?v=wyWI3gLpB8o

Roy Rapoport - Cloud Operations at Netflix - http://www.youtube.com/watch?v=7779Wrun5fo

AWS re:Invent - http://reinvent.awsevents.com/recap2013.html

Other Presentations

Monday, December 23, 13

Netflix is Hiringhttp://jobs.netflix.com/

Monday, December 23, 13