angrybirds - overview for a high performance shop

Challenges for a high performance Shop

Angrybirds in the Cloud

September 12 | E-HandelsSnack | Stockholm

Donnerstag, 19. September 13

@aoepeople

@

danpoetz

DanielPötzinger


@aoepeople

��)

@

danpoetz

DanielPötzinger


��$""�'�� !$��!��

� !��

�#)�'%' (��'��


2.5 billion internet users


360 495677 785 914 1022 1150

13641560

1764

2013

2265

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011



360 495677 785 914 1022 1150

13641560

1764

2013

2265

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011


�$� �!�� "%��) "$� !��#)�'#�)


https://github.com/fbrnc/EnvSettingsTool


152,2 Million €

68,5 Million € Consumer Products


> 24.000.000 likes > 500.000 followers

Social MediaImpact


Irregular Traffic Behaviour


Angrybirds frequent Releases



2012 2013



2012 2013

Giftca

rds

Geo IPAsy

nc R

epor

ts

Sear

ch

Targ

eting

Prod

ucts


The DevOps Conflict


The DevOps Conflict

Frequent development and releasing of new

features


The DevOps Conflict

Frequent development and releasing of new

features

Scalable and stable Operation


Biggest Problems


Biggest ProblemsChallenges



1. Handling the tremendous peaks



1. Handling the tremendous peaks

2. Quick Handling and deploymentof Changes and new Features


Handling the peaks...


How to deal with peaks


How to deal with peaks '�&* '�� #�'�()'*�)*'��

��%�� ).









��*(��

'�&* '�� #�'�()'*�)*'��

��%�� ).











��*(��

��,�()��

'�&* '�� #�'�()'*�)*'��

��%�� ).












✓ quick scale on demand✓ pay only for what you use


*)$(��! #�





��'�� $#(�, )��+� !�� ! ).�/$#�(

�!$)�$��„��(�'+�'(0��$'��$#)�#)��! +�'.

Global Cloud Infrastructure










Think cleanly about your operation infrastructureDonnerstag, 19. September 13

Setting up a new environment should always be faster

than fixing an existing oneDonnerstag, 19. September 13

What you should do


What you should do

Automate the setup of new servers


What you should do


Automate failover


What you should do


Automate failover

Automate scaling


What you should do


Automate failover

Automate scaling

Save work by using AWS concepts


Backend Array

Frontend Array

Worker Array

Varnish Array


Backend Array

Frontend Array

Worker Array

Varnish Array

Loadbalancer


Deployment X

Backend Array

Frontend Array

Worker Array

Varnish Array

Loadbalancer


Deployment X

Backend Array

Frontend Array

Worker Array

Varnish Array

DNS

shop.angrybirds.com

Loadbalancer


Deployment X

Backend Array

Frontend Array

Worker Array

Varnish Array

DNS

shop.angrybirds.com

File Storage

File Storage DatabaseCache

CDN CDN

Braintree

MailDRM

Giftcards

Loadbalancer


Load Balancer

Backend Array

Frontend Array

Worker Array

Varnish Array

Deployment X

DNS


Worker Array

Deployment X

DNS


Worker Array

Deployment X

DNS

Deployment X+1


Worker Array

Deployment X

DNS

Deployment X+1 Loadbalancer

Backend Array

Frontend Array

Worker Array

Varnish Array


Worker Array

Deployment X

DNS

Backend Array

Frontend Array

Worker Array

Varnish Array



Frontend Array

Worker Array

Varnish Array

Deployment X

DNS

Backend Array

Deployment X+1Loadbalancer Loadbalancer


DNS

Backend Array

Deployment X+1

Frontend Array

Worker Array

Varnish Array

Deployment XLoadbalancer Loadbalancer


DNS

Backend Array



0Donnerstag, 19. September 13

0Downtime


0Downtime

Number of lost carts


0Downtime

Number of lost carts

Number of lost sessions and

logins


Scaling is about finding the bottleneck


The fastest HTTP Request is..


The fastest HTTP Request is..

the one- never made!Donnerstag, 19. September 13

Traffic

HTTP Cacheing


App

licat

ion

Traffic

HTTP Cacheing


App

licat

ion

TrafficBr

owse

rTraffic

HTTP Cacheing


App

licat

ion

Traffic

CDN

Brow

ser

Traffic

HTTP Cacheing


App

licat

ion

Varnish (Reverse Proxy) Traffic

Traffic

CDN

Brow

ser

Traffic

HTTP Cacheing


App

licat

ion

Varnish (Reverse Proxy) Traffic

Traffic

CDN

Brow

ser

TrafficPurging

HTTP Cacheing


Caches„protect“ your

application



application

#�+�

'�%*

'��



application

#�+�

'�%*

'��

��

��,

�'"

#�


10.000+Possible Pages / Seconde

from ELB & Varnish


Traffic

Autoscaling


Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

Autoscaling


Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

EC2

EC2

EC2

Autoscaling


10.000possible Orders / Hour


Search and Layered Nav��'�� (�*#��'�

�$%��%��(

#��'!.�*#��!�

�-%�#( +�








Bro

wse

r

Enterprise Search

Mage

nto


Bro

wse

r

��!�) "��#��- #�

Enterprise Search

Mage

nto




Bro

wse

r

��!�) "��#��- #�

� +�(*��()��'��

��.�'��+ ��) $#��$""�#��) $#�� )(

Enterprise Search

Mage

nto












Be prepared for changes


Continuous Deployment


Don‘t be scared of releasing to production


Late night bugfixing?Donnerstag, 19. September 13

That’s how a deployment should look like


We want on-demand deployment



We want on-demand deploymentof tested features and fixes



We want on-demand deploymentof tested features and fixesin a quick and reliable way



We reach this by


We reach this by

Automating everything


We reach this by

Build



We reach this by

BuildTesting



We reach this by

BuildTesting

Deployment



..and


..and

doing it frequently


..and

flickr deploys 3

times a day...

doing it frequently


ContentMaster

Backup

Version

Deployment Pipeline


ContentMaster

Backup

Version

Build

Deployment Pipeline


ContentMaster

Backup

Version

Build

Unit TestsDeployment Pipeline


ContentMaster

Backup

Version

Build

Unit TestsAutomatic Deploy on „Latest“

Deployment Pipeline


ContentMaster

Backup

Version

Build

Unit TestsAutomatic Deploy on „Latest“ Acceptance

Tests

Deployment Pipeline


ContentMaster

Backup

Version

Build


Tests

Deploy on

„Deploy“

Deployment Pipeline


ContentMaster

Backup

Version

Build


Tests Deploy to Staging+ Test

Deploy on

„Deploy“

Deployment Pipeline


ContentMaster

Live

Backup

Version

Build



Deploy on

„Deploy“

Deployment Pipeline


ContentMaster

Live

Backup

Version

Build



Deploy on

„Deploy“

Deployment Pipeline

Test Strategy


ContentMaster

Live

Backup

Version

Build



Deploy on

„Deploy“

Deployment Pipeline

Test Strategy

Feedback for the TeamDonnerstag, 19. September 13

The real live


Day 1 - Release 1


Day 2 - Release 2 - “Get ready for Space”


Day 3 - „People want it so badly“


New release deployed




Space game launchedArray scales up




Space game launchedArray scales up

USA wakes up



Day 4 - Release 3 „Performance Improvement“


40% average CPU usage.

1-2 FE Servers are enough for normal operations

Day 4 - Release 3 „Performance Improvement“


Thanks!

Follow us on Twitter:

AOE: @aoemediaDaniel P: @danpoetz

Flickr photo contributions in the slide:http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/


http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/

http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/

http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/

http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/

http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

angrybirds - overview for a high performance shop

Technology

failover donnerstag

scaling donnerstag

r array nt x dns donnerstag

new features donnerstag

tremendous peaks donnerstag

biggest problems donnerstag

devops conflict donnerstag

internet users donnerstag