continuous integration and deployment best practices on aws

CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS

Danilo Poccia – Solutions Architect

Compute & Networking

Storage & Content Delivery

AWS Global Infrastucture

Database

App Services

Anal

ytic

s

Deployment & Management

API

API

API

API

API

API

Regions Availability Zones Edge Locations

EC2 VPC Route 53 Direct Connect

S3 Glacier CloudFront Storage Gateway

RDS DynamoDB ElastiCache Redshift

SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder

IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks

EMR

Dat

a Pi

pelin

eK

ines

is

WorkSpaces

Regions Availability Zones Edge Locations

S3 Glacier CloudFront Storage Gateway

RDS DynamoDB ElastiCache Redshift

SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder

IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks

EMR

Dat

a Pi

pelin

eK

ines

is

EC2 VPC Route 53 Direct Connect WorkSpaces

Regions

Availability ZonesEdge Locations

EC2

VPC Route 53 Direct Connect

S3Glacier

CloudFront

Storage Gateway

RDSDynamoDB

ElastiCache

RedshiftSQS

SNS

SWF

SES

AppStreamCloudSearch

Elastic Transcoder

IAM

CloudTrail

CloudWatch CloudFormation

Elastic

Beanstalk

OpsWorks

EMR

Data Pipeline

Kinesis

WorkSpaces

http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg

BUILDING BLOCKS TO IMPLEMENT YOUR IDEA

FILE SYNC

DATA ANALYSIS

DATA ANALYSIS COSTS < VALUE

AMAZON EC2 SPOT INSTANCES

2013  

280

2011  

82

2010  

612009  

48

2008  

24

2007  

9 Amazon  FPS  Red  Hat  EC2

SimpleDB  CloudFront  EBS  Availability  Zones  Elastic  IPs

Relational  Database  Service  Virtual  Private  Cloud  Elastic  Map  Reduce  

Auto  Scaling  Reserved  Instances  

Elastic  Load  BalancerSimple  Notification  Service  

Route  53  RDS  Multi-­‐AZ  

Singapore  Region  Identity  Access  Management  

Cluster  Instances

Elastic  Beanstalk  Simple  Email  Service  CloudFormation  RDS  for  Oracle  ElastiCache

Redshift  Kinesis  

WorkSpaces  Route  53  Health  Check  

OpsWorks  CloudTrail

number  of  released  features,  sample  services  described

2012  

159Glacier  

DynamoDB  Simple  Workflow  

CloudSearch  Storage  Gateway  

Route  53  Latency  Based  Routing

NEW SERVICES + NEW FEATURES = NEW IDEAS

AMAZON KINESIS REAL-TIME PROCESSING

OF STREAMING DATA AT MASSIVE SCALE

!Data!Sources!

App.4%!

[Machine!Learning]!

!!!!!!!!!!!!!!!

! ! !AW

S!En

dpoint!

App.1%%

[Aggregate!&!De:Duplicate]!

!Data!Sources!

Data!Sources!

!Data!Sources!

App.2%%

[Metric!Extrac=on]!

S3

DynamoDB

Redshift

App.3%[Sliding!Window!Analysis]!

!Data!Sources!

Availability Zone

Shard 1

Shard 2 Shard N

Availability Zone

Availability Zone

CLICKSTREAM ANALYTICS INTERNET OF THINGS METERING & BILLING EVENT PROCESSING SOCIAL NETWORKS LOG COLLECTION …

AMAZON SQS + AUTO SCALING + SPOT INSTANCES !

= FREEMIUM

AMAZON SNS MOBILE PUSH

NOTIFICATIONS

EVE

Online

ANY NEW

IDEA TO BUILD?

THE JOURNEY OF CODE TO

PRODUCTION

WRITTEN TESTED

CONSUMED DEPLOYED

~ MANUFACTURING ~

LEAN MANUFACTURING

Transport Inventory Motion Waiting

Over Production

Over Processing Defects Talent!

& Skills

INFORMATION TECHNOLOGY

Transport Inventory Motion Waiting

Over Production

Over Processing Defects Talent!

& Skills

OPTIMIZE WITH AWS

Transport Inventory Motion Waiting

Over Production

Over Processing Defects Talent!

& Skills

OPTIMIZE WITH AWS

OPPORTUNITY STRATEGY

USE ADJUST

= PROGRAMMABLE

PLATFORM

!

IF YOU CAN DO ITYOU CAN AUTOMATE IT

“Improving daily work is even more important than doing daily work.”

!

!

Gene Kim, Kevin Behr, George Spafford, The Phoenix Project: A Novel About IT, DevOps,

and Helping Your Business Win

CONTINUOUS INTEGRATION

DEVELOPER

SOURCE CODE REPOSITORY

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

PICK TASKS

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

SUBMITCODE

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

SCHEDULEBUILD

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

AUTOMATIC BUILDS

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

CODEFETCH

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

CODE QUALITY TESTS

TESTRESULTS

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

BUILD OUTPUT

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

DOCS

BINARIES& PACKAGES

TDD TEST-DRIVEN

DEVELOPMENT

BDD BEHAVIOR-DRIVEN

DEVELOPMENT

PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES

“Complexity arises when the dependencies among the elements

become important.” !

!

Scott E. Page, John H. Miller,Complex Adaptive Systems:

An Introduction to Computational Models of Social Life

REAL USERS =

REAL FEEDBACK

CLOSE THE FEEDBACK

LOOP AS QUICKLY

AS POSSIBLE

CONTINUOUS INTEGRATION

CONTINUOUS DEPLOYMENT

SOFTWARE DEPLOY

≠PRODUCT LAUNCH

CONTINUOUS DEPLOYMENT

SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO

PRODUCTION

KEY = ITERATION

ITERATION =

MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF

YOUR USERS

FEATURE SWITCH FEATURE FLIPPER FEATURE TOGGLE

FEATURE FLAG CONDITIONAL FEATURE

// To totally enabled a feature !$server_config[‘foo'] = ‘on’;!!

// To totally disabled a feature!$server_config['foo'] = ‘off’;!!

// Feature with variant turned on for everyone!$server_config['foo'] = 'blue_background';!!

// Enabled for 10% of regular users and all admins!$server_config[‘foo'] = array(!   'enabled' => 10,!   'admin' => 'on',!!);

https://github.com/etsy/feature

!if (Feature::isEnabled('my_feature')) {!!    // do stuff!!}!! !!if (Feature::isEnabled('my_feature')) {!! !!    switch (Feature::variant('my_feature')) {!!      case 'foo':!!          // do stuff appropriate for the foo variant!!          break;!!      case 'bar':!!          // do stuff appropriate for the bar variant!!          break;!!    }!!}

https://github.com/etsy/feature

A / B TESTING CONTENT

// Single-variant feature in 50/50 A/B test!!

$server_config['foo'] = array('enabled' => 50);!!

// Multi-variant feature in A/B test!// with 20% of users seeing each variant!// (and 40% left in control group)!!

$server_config['foo'] = array(!!   'enabled' => array(!!       'blue_background'   => 20,!!       'orange_background' => 20,!!       'pink_background'   => 20,!!   ),!!);

https://github.com/etsy/feature

AWS OPSWORKS MODEL AND MANAGE

THE ENTIRE APPLICATION

14 BILLION REQUESTS/MONTH50 000 DATABASE UPDATES / SEC

NO CACHE

CASE STUDY

A / B TESTING NOT JUST FOR CONTENT

USER INTERFACE ALGORITHMS

ARCHITECTURES COST OPTIMIZATION

AWS CLOUDFORMATION INFRASTRUCTURE IS CODE

CLOUDFORMATION TEMPLATE

APPLICATION VERSIONS

+INFRASTRUCTURE

VERSIONS

CLOUDFORMATION TEMPLATE

AWS ELASTIC BEANSTALK

QUICKLY DEPLOY AND MANAGE APPLICATIONS

Production Environment

Test Environment

master branch

test branch

Git Repository

git commit

git aws.push

git aws.push

Swap URLs

AMAZON EC2 API / SDK / CLI

for ip in $(!!

aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text!!

); do ssh -t ec2-user@$ip "/app/update.sh"; done!!

ALL TOGETHER NOW! !

!

!

!

!

DEPLOYMENTS AT AMAZON.COM

!

!

!

11.6s

Mean time between deployments (weekday)

1,079

Max number of deployments in a

single hour

10,000

Mean number of hosts

simultaneously receiving a deployment

30,000

Max number of hosts

simultaneously receiving a deployment

CONTINUOUS DEPLOYMENT

= CONTINUOUS

EXPERIMENTATION

CONTINUOUS DEPLOYMENT

= CONTINUOUS INNOVATION

http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg

« Want to increase innovation? Lower the cost of failure »

!

Joi Ito

CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS

Danilo Poccia – Solutions Architect