devops at netflix (re:invent)

Rainmakers

How Netflix Operates Clouds for Maximum Freedom and Agility

Jeremy EdbergReliability Architect,

Netflix

Tweet @jedberg with feedback!

Do you have...

• A release Engineer?

• A QA department?

• Chef or Puppet to manage your systems?

Do you have...

• Upwards of 100 releases a day?

With more than 30 million streaming members in the United States,

Canada, Latin America, the United Kingdom, Ireland and the Nordics,

Netflix is the world's leading internet subscription service for enjoying

movies and TV programs streamed over the internet to PCs, Macs and TV.

Source: http://ir.netflix.com

The Netflix Way• Everything is “built for three”

• Fully automated build tools to test and make packages

• Fully automated machine image bakery

• Fully automated image deployment

• Independent teams responsible for both Dev and Ops

Philosophy

Automate all the things!

• Application startup

• Configuration

• Code deployment

• System deployment

Automation

• Standard base image

• Tools to manage all the systems

• Automated code deployment

Shared state should be stored in a shared

service

Data on an instance should be replicated to other instances

“Build for Three”We hold a boot camp for new engineers to teach

them how to build for a highly distributed environment.

Netflix on AWS2012IPv6

2012IPv6

Open Connect

Highly aligned, loosely coupled

• Services are built by different teams who work together to figure out what each service will provide.

• The service owner publishes an API that anyone can use.

Advantages to a Service Oriented Architecture

• Easier auto-scaling

• Easier capacity planning

• Identify problematic code-paths more easily

• Narrow in the effects of a change

• More efficient local caching

Freedom and Responsibility

• Developers deploy when they want

• They also manage their own capacity and autoscaling

• And fix anything that breaks at 4am!

All systems choices assume some part will fail

at some point.

The Monkey Theory

•Simulate things that go wrong

•Find things that are different

Execution

Photo from I, Robot, copyright 20th Century Fox

Netflix built a global PaaS

•Service Oriented Architecture

•HTTP/Rest interfaces between services

Netflix PaaS features• Supports all regions and zones

• Multiple accounts

• Cross region/account replication

• Internationalized, localized and GeoIP routed

• Advanced key management

• Autoscaling with 1000s of instances

• Monitoring and alerting on millions of metrics

What AWS Provides

• Instances

• Machine Images

• Elastic IPs

• Load Balancers

• Security groups / Autoscaling groups

• Availability zones and regions

Monitoring

Log Rotation to

Appdynamics Machine

Linux Base AMI (CentOS or Ubuntu)

Java (JDK 6 or 7)

Tomcat

Optional Apache

Appdynamics App Agent

monitoringApplication war file,

base servlet, platform, interface jars for dependent

services

GC and thread dump

logging

Healthcheck, status servelets, JMX

interface, Servo autoscale

The Netflix PlatformDiscovery

(Eureka)Entrypoints (Edda)Configuration

(Archaius)Zookeeper (Exhibitor)

logging (Blitz4j & Honu)NIWSGeoBase

Circut Breakers (Hystrix)

Cassandra (Priam & Astyanax &

CassJMeter) Cryptex AKMSEvCache

Proxiesi18nL10nOpen Source

NovNov Feb

DecDec 201

20122 M

arMar AprApr Ju

AugAug Sep

OctOct

CassJM

MayMay

Open Source at Netflix

GovernatorBlitz4jEdda

Hystrix

Finding things• Discovery (Eureka)

• Application to instance mapping• Heartbeat to keep track of health

• Entrypoints (Edda)• Local database of AWS resources

• NIWS (Netflix Internal Web Service)• On instance software load balancer• Handles retry logic

• Geo (Geolocation library)• Provides IP to Lat/Lon mapping for any service

that needs it.

Entrypoints (Edda)

• REST API

• GET /REST/v2/instance/$id

• Keeps track of all resources

• Autoscaling groups, EIPs, Instances, Applications, Clusters, History

Entrypoints ExplorationFind all active

instancesGET /REST/v2/view/instances

Find all instances in a cluster

GET /REST/v2/group/clusters

Show only ASG name, instance ID

and health

/v2/aws/autoScalingGroups/edda-v123;_pp:(autoScalingGroupName,instances:

(instanceId,lifecycleState))

Which ASG contains a particular instance?

/v2/aws/autoScalingGroups;instances.instanceId=i-96f3ca3a

Keeping it all Straight• Configuration (Archaius)

• Global variables (Fast properties)

• Base

• Base system. Prod vs. Test, etc

• Zookeeper (Curator)

• Locks, other similar coordination

• Logging (Blitz4j and Honu)

• Keep track of what happened and store it for post analysis.

Keeping it Secure• Cryptex

• Service for key management

• High, medium and low value keys

• AKMS (Amazon Key Management System)

• Hands out keys to instances (and dev boxes) so they don’t have to store the key on the instance

For more info, see SEC201: Security Panel

Storing it• Cassandra (Priam, astyanax)

• Configure and access Cassandra

• Provide OO abstractions handle connection pooling, discovery of hosts

• EVCache (Eccentric Volatile Cache)

• Wrapper for memcached to handle zone awareness and replication

• Proxies

• Get data out of the datacenter and into the cloud.

DataWhat do we do with it all?

We store it!

•Cache (memcached)

•Cassandra

•RDS (MySql)

Cassandra

Why Cassandra?

•Availability over consistency

•Writes over reads

•We know Java

•Open source + support

Using Cassandra at Netflix

• Priam

• Zero touch auto-config

• State management

• Token assignment

• Node replacement

• Backup/restore to/from S3

• Astyanax

• OO abstraction to Cassandra

• Multi-region support

Cassandra Architecture

For more info, see DAT202: Optimizing your Cassandra Database on AWS

• Asgard

• AWS usage

• Atlas

• Chronos

• Build system

• Explorers (Cassandra and SimpleDB)

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Elastic LoadBalancer

api-usprod-v007

api-frontend

api-usprod-v008

api-usprod-v007

api-frontend

api-usprod-v008

Netflix has moved the granularity from the

instance to the cluster

Why Bake?

Generic AMI

Instance

Traditional:•launch OS•install packages•install app

Netflix:•launch OS+app App AMI Instance

Getting Baked

Perforce / GitPerforce / GitPerforce / GitPerforce / Git

libraries

source

Ant targetsAnt targets

IvyIvy

Groovy all overGroovy all over

snapshot / release libraries / apps

app bundles

JenkinsJenkinsJenkinsJenkins

syncsyncsyncsync

resolveresolveresolveresolve

buildbuildbuildbuildcompilecompilecompilecompile reportreportreportreport

publishpublishpublishpublishtesttesttesttest

ArtifactoryArtifactoryArtifactoryArtifactory

Base ImageBaking

Yum / AptYum / AptYum / AptYum / Apt

Linux: CentOS, Fedora, UbuntuLinux: CentOS, Fedora, Ubuntu

AWSRPMs: Apache, Java...

ec2 slave instancesec2 slave instances

S3 / EBS

foundatiofoundation AMIn AMI

base base AMIAMIbase base AMIAMI

BakeryBakeryBakeryBakery

install

Ready forappbake

snapshot

App ImageBaking

Jenkins / Yum / Jenkins / Yum / ArtifactoryArtifactory

Linux, Apache, Java, TomcatLinux, Apache, Java, Tomcat

AWSapp bundle

ec2 slave instancesec2 slave instances

S3 / EBS

base AMIbase AMIbase AMIbase AMI

app app AMIAMIapp app AMIAMI

BakeryBakeryBakeryBakery

install

Ready to launch!

snapshot

Monitoring

Log Rotation to

Appdynamics Machine

Java (JDK 6 or 7)

Tomcat

Optional Apache

services

GC and thread dump

logging

Java (JDK 6 or 7)

Optional Apache

Monitoring

Log Rotation to

Appdynamics Machine

services

GC and thread dump

logging

Python

Django

Optional Apache

Monitoring

Log Rotation to

Appdynamics Machine

monitoring

Application file, base server, platform, interface libs for

dependent serviceslogging

The Monkey Theory

•Simulate things that go wrong

•Find things that are different

The simian army• Chaos -- Kills random instances

• Chaos Gorilla -- Kills zones

• Chaos Kong -- Kills regions

• Latency -- Degrades network and injects faults

• Conformity -- Looks for outliers

• Circus -- Kills and launches instances to maintain zone balance

• Doctor -- Fixes unhealthy resources

• Janitor -- Cleans up unused resources

• Howler -- Yells about bad things like Amazon limit violations

• Security -- Finds security issues and expiring certificates

For more info, see ARC301: Intro to Chaos Monkey & the Simian Army

What’s going on?!

{ "clusters": [ "epic_aggregator", "epic_aggregator-dev" ], "alerts": [ // you can use javascript style comments in the config { "metricName": "EpicPlugin_NumDropped", "applyTo": "cluster", "condition": { "type": "StaticThreshold", "max": 0.0 }, "severity": "major", "description": "plugin is dropping metrics" }, { "metricName": "EpicPlugin_NumDropped_Instance", "applyTo": "instance", "condition": { "type": "NumOccurrences", "num": 4, "condition": { "type": "StaticThreshold", "max": 0.0 } }, "overrides": { "service_key_override": "12345", "require_instance_status_not_in: ["DOWN", "OUT_OF_SERVICE"], "email_override": "devnull@netflix.com" }, "severity": "minor" },

{ "metricName": "EpicPlugin_MetricCount", "applyTo": "instance", "description": "${instanceId} is reporting too many metrics", "condition": { "type": "NumOccurrences", "num": 4, "condition": { "type": "StaticThreshold", "max": 0.0 } }, "additionalDetails": { "statusUrl": "http://${publicDnsName}:7001/Status", "nacClusterUrl": "nac${env}/${region}/cluster/show/${cluster}" } "overrides": { "subject": "${instanceId} is reporting too many metrics", "incident_key": "${metricName}:${instanceId}", "service_key_override": "12345", "email_override": "devnull@netflix.com" }, "severity": "minor" } ]}

Example Alert Config

Alert Tuning

Alert Systems

alertingalertingalertingalerting

apiapiapiapi

CORECOREEvent Event

GatewaGatewayy

CORECOREEvent Event

GatewaGatewayy

Paging Paging ServiceServicePaging Paging ServiceService

AmazoAmazonn

SESSES

AmazoAmazonn

SESSES

CORE CORE AgentAgentCORE CORE AgentAgent

Other Other TeamTeam’’s s AgentAgent

CORE CORE AgentAgentCORE CORE AgentAgent

Appdynamics

Chronos

Data Collection Pipeline

Data Processing Pipeline

For more info, see BDT303: Data Science with Elastic MapReduce

Chuckwa/Honu messages / min

63 billion

messages a day

Best Practices

Incident Reviews

• What went wrong?

• How could we have detected it sooner?

• How could we have prevented it?

• How can we prevent this class of problem in the future?

• How can we improve our behavior for next time?

Ask the key questions:

Best Practices for Data

• Have multiple copies of all data

• Keep those copies in multiple AZs

• Avoid keeping state on a single instance

• Take frequent snapshots of EBS disks

• No secret keys on the instance

Netflix autoscaling

Traffic Peak

Deployment

AWS UsageDollar amounts have been carefully removed

Going multi-zone

Benefits of Amazon’s Zones

• Loosely connected

• Low latency between zones

• 99.95% uptime guarantee per region

Going Multi-region

Leveraging Multi-region

• 100% uptime is theoretically possible.

• You have to replicate your data

• This will cost money

Circuit Breakers (Hystrix)Be liberal in what you accept, strict in what you send

Just a quick reminder...

• (Some of) Netflix is open source:

• https://github.com/netflix

We are sincerely eager to hear your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form when you have a

chance.

Questions?

Getting in touchEmail: jedberg@{gmail,netflix}.com

Twitter: @jedberg

Web: www.jedberg.net

Facebook: facebook.com/jedberg

Linkedin: www.linkedin.com/in/jedberg

devops at netflix (re:invent)

rainmakershow netflix

netflix way

whateach service

service owner

netflix paas features

shared service data

96f3ca3aparticular instance

s of instances

Technology

2012 re:invent netflix: embracing the cloud final

aws re:invent 2016: how i learned to embrace devops and...

beyond devops: how netflix bridges the gap · beyond...

(gam404) gaming devops: scopely's continuous deployment...

netflix: amazon s3 & amazon elastic mapreduce to monitor at...

(ent210) accelerating business innovation with devops on aws...

devops nirvana: seven steps to a peaceful life on aws...

slalom technology - amazon s318 re:invent 2016 recap 20...

devops devops devops, at froscon

aws re:invent 2016: enabling devops for an enterprise with...

aws re:invent 2016: netflix: using amazon s3 as the fabric...

aws re:invent 2016: from dial-up to devops - aol’s...

what an enterprise can learn from netflix, a cloud-native...

aws re:invent 2016: use aws to secure your devops pipeline...

embracing devops as a security professional · security @...

aws re:invent 2016: how netflix achieves email delivery at...

how netflix leverages multiple regions to increase...

how netflix thinks of devops. spoiler: we don’t

aws re:invent 2016: enterprise fundamentals: use aws to...

aws black belt online seminar 2018 re:invent recap iot and...