chicago devops meetup

7/18/2019 Chicago DevOps Meetup

1/110

How Netflix Delivers

Software

July 8th, 2014

Email: jedberg@{gmail,netflix}.comTwitter: @jedbergWeb: www.jedberg.netFacebook: facebook.com/jedbergLinkedin: www.linkedin.com/in/jedberg
http://www.linkedin.com/in/jedberghttp://www.jedberg.net/mailto:[email protected]


2/110

When your software fails...


3/110

will your system survive?


4/110


5/110

The Netflix way

Fully automated build tools

to test and make packages

Fully automated machineimage bakery

Fully automated imagedeployment


6/110

Everything is built for three Independent teams responsible

for both Dev and Ops Redundancy through multi-

region deployment

The Netflix way


7/110

Philosophy


8/110

We hire responsible adultsand keep rules and policiesto a minimum

Developers can change anycode in production at any

time And things dont break

(usually)

Freedom and

Responsibility


9/110

Automate all the things!

http://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html
http://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html


10/110

Application startup Configuration Code deployment

Systemde lo ment

Automate all the things!


11/110

Standard base image Tools to manage all

the systems

Reduce errorsthroughreproducibility

Automation


12/110

Shared state should

be stored in ashared service

Data on an instance

should be replicatedto other instances


13/110

Build for three

We hold a boot camp for newengineers to teach them how to

build for a highly distributed

environment.


14/110

Build for three

We hold a boot camp for newengineers to teach them how to

build for a highly distributedenvironment.


15/110


16/110

!"#$%

'()*+,-%.,"*(/$0()"*

1*+$*% 2,%. 3*4"

!"#$%

!%5(6(5(

7$8$/(.

!"#$%, '%#$%9,

:;< =%,5

1*+$*%

Discovery

API

Streaming

API


17/110

!"#$%

'()*+,

-%.,"*(/$0()"*

1*+$*% 2,%. 3*4"

!"#$%

!%5(6(5(

7$8$/(.

!"#$%,

'%#$%9,

:;< =%,5

1*+$*%

Discovery

API

Streaming

API

Content

EncodingCDN

Management

QOS

LoggingDRM

OpenConnect

Edge Locations

Browse

Play

Watch


18/110

Services are built by different

teams who work together tofigure out what each servicewill provide.

The service owner publishesan API that anyone can use.

Highly aligned, loosely

coupled


19/110

Easier auto-scaling Easier capacity planning Identify problematic code-paths

more easily Narrow in the effects of a change More efficient local caching

Advantages to a Service

Oriented Architecture


20/110

Developers deploy whenthey want

They also manage their owncapacity and autoscaling

And fix anything that breaksat 4am!

Freedom and

Responsibility


21/110

All systems

choices assumesome part will fail

at some point.


22/110

Simulate things

that go wrong

Find things thatare different

The Monkey Theory


23/110

Execution


24/110


25/110

AWS

Netflix OSS

Netflix Application Code


26/110

AWS

Netflix OSS

YOUR Application Code


27/110

Instances Machine Images Elastic IPs

Load Balancers

Security groups / Autoscaling

What AWS Provides

AWS


28/110

AWS

Netflix OSS

YOUR Application Code


29/110

Service OrientedArchitecture HTTP/Rest interfaces

between services

Netflix built a global PaaS

Netflix OSS


30/110

Supports all regions and zones Multiple accounts Cross region/account replication Internationalized, localized and GeoIP routed Advanced key management Autoscaling with 1000s of instances Monitoring and alerting on millions of metrics

Netflix PaaS featuresNetflix OSS


31/110


32/110

Open Source at Netflix


33/110

Netflix OSS


34/110


35/110


36/110


37/110

Simulate things

that go wrong Find things that

are different

The Monkey Theory


38/110


39/110

Netflix OSS


40/110


41/110

Blueprint for the rest ofthe platform libraries

Pluggable architecture


42/110


43/110

On instance software load balancer Zone aware / Zone affinity

Handles retry logic


44/110


45/110

Global variables Support for staged rollout Feature flags


46/110

Netflix OSS


47/110


48/110

Application to instance mapping

Heartbeat to keep track of health


49/110


50/110

DQ Transport Routing

Suro

etc

Eventbus

Druid

N fl OSS


51/110

Netflix OSS


52/110


53/110

Why Bake?

Generic AMI Instance

Traditional:launch OSinstall packagesinstall app

Netflix:launch OS+app

App AMI Instance


54/110

Getting Baked

Perforce / Git

libraries

source

Ant targets

Ivy

Groovy all over

app bundles

Jenkins

sync

resolve

buildcompile report

publishtest

Artifactory

snapshot / release

libraries / apps


55/110

Base

ImageBaking

Yum / Apt

Linux: CentOS, Fedora, Ubuntu

RPMs: Apache, Java...

ec2 slave instances

S3 / EBS

foundationAMI

base

AMI

Bakery

mount

install

Ready

for

app

bake

snapshot

AWS

A


56/110

App

Image

Baking

Jenkins / Yum /

Artifactory

Linux, Apache, Java, Tomcat

AWS

app bundle

ec2 slave instances

S3 / EBS

base AMI

app

AMI

Bakery

mount

install

Ready

to launch!

snapshot


57/110

app

AMI Linux Base AMI (CentOS or Ubuntu)

Java

Tomcat

OptionalApache

MonitoringLog Rotation

to S3

monitoring

GC andthread dump

logging

Application war file, base

servlet, platform, interfacejars for dependentservices

Healthcheck, status

servelets, JMX interface,Servo autoscale


58/110

Application war file


59/110

Linux Base AMI (CentOS or Ubuntu)

Java

JBoss

OptionalApache


to S3

monitoring

GC andthread dump

logging

Application war file, base

servlet, platform, interfacejars for dependentservices

Healthcheck, status

servelets, JMX interface,Servo autoscale


60/110

Linux Base AMI (CentOS or Ubuntu)

Python

Bottle

OptionalApache


to S3

monitoring

logging

Application file, base

server, platform, interfacelibs for dependent services

Netflix OSS


61/110

Netflix OSS


62/110


63/110


64/110


65/110

Deploying Code; Step 1


66/110


67/110


68/110

Auto Scaling

Group

LaunchConfiguration

Security

Group

Amazon Machine

Image

Instances

Load

Balancer


69/110


70/110

Netflix has moved

the granularityfrom the instance

to the cluster


71/110


72/110

Data is the most

important asset Netflixhas. Its what differentiatesus from our competitors.

Netflix OSS


73/110


74/110

EVCache

Wrapper on top of memcached Automatically replicates writes to

multiple regions

Pulls cache data intelligently via zoneaffinity


75/110

Cassandra


76/110

Availability over consistency Writes over reads We know Java Open source + support

Why Cassandra?


77/110

Priam Zero touch auto-config State management Token assignment Node replacement

Backup/restore to/from S3

Using Cassandra at Netflix

Astyanax

OO abstractionto Cassandra Multi-regionsupport

Cassandra Architecture


78/110

Cassandra Architecture


79/110

Going Multi-region


80/110

100% uptime is theoreticallypossible.

You have to replicate your data This will cost money

Leveraging Multi-region


81/110


82/110


83/110

1 2


84/110

us-east-1 us-west-2 etc

eu-west-1


85/110

1 2


86/110

us-east-1 us-west-2 etc

eu-west-1


87/110

Whats going

on?!

Alert Systems


88/110

Atlas

alerting

api

api

Central

EventGateway

PagingService

AmazonSES

COREAgent

OtherTeamsAgent

COREAgent

Alert Systems

Central


89/110

Event

Gateway

Parse raw alerts, match application to owner Add image captures and links to related

graphs for easy mobile use Send to the right service based on priority Register the event in Chronos, the timeline

application Correlate low priority alerts and generate

new high priority alerts


90/110


91/110

Metrics in Production 796B Daily metric

points Peaks at 1.4B /

min

50% daily metricchurn


92/110

What is a metric?com.netflix.eds.nccp.successful.requests.uiversion.nccprt-authorization.devtypid-101.clver-PHL_0AB.uiver-UI_169_mid.geo-US


93/110

How we built it Built our own big data

system Based on S3 and EMR Less copies, lower

resolution, and slowerspeed retrieval based onage of data


94/110

Self Serve is the Key Developers choose

what metrics tosubmit

What graphs theyput on their

dashboards What to alert on

E l Al C fi


95/110

Example Alert Config


96/110

Atlas


97/110

When something breaks..


98/110


99/110


100/110


101/110

Breakdown of an outage


102/110



103/110

Change control, the good Tells you what changed Tells you whats about tochange Great for coordination

when one change gatesanother change


104/110

Change control, the bad Its manual It expresses intent, notreality It forces you to

serialize your changesto an extent


105/110



106/110

J t i k i d


107/110

(Some of) Netflix is open source:

https://netflix.github.io

Just a quick reminder...

N tfli i hi i !
https://netflix.github.io/


108/110

Netflix is hiring!

If you like what you see here,feel free to reach out!

Q ti ?


109/110

Questions?

G tti i t h


110/110

Getting in touch

Email: jedberg@{gmail,netflix}.comTwitter: @jedbergWeb: www.jedberg.netFacebook: facebook.com/jedbergLinkedin: wwwlinkedin com/in/jedberg
http://www.linkedin.com/in/jedberghttp://www.jedberg.net/mailto:[email protected]

chicago devops meetup

Documents