Download - Games for the Masses (QCon London 2012)
![Page 1: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/1.jpg)
GAMES FOR THE MASSESHow DevOps Affects Architecture
Jesper Richter-‐Reichhelm, @jrireiWednesday, March 7, 2012
![Page 2: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/2.jpg)
Wednesday, March 7, 2012
![Page 3: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/3.jpg)
Our games all look the same
Flash client Backend
Wednesday, March 7, 2012
![Page 4: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/4.jpg)
Our games all look the same
Flash client
Game Session
Asynch. CommunicaEon
Wednesday, March 7, 2012
![Page 5: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/5.jpg)
Our games all look the same
State Changes
ValidaEon
Persistence
Backend
Wednesday, March 7, 2012
![Page 6: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/6.jpg)
But the scale is interesEng
14 billion requests / month
Wednesday, March 7, 2012
![Page 7: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/7.jpg)
But the scale is interesEng
14 billion requests / month
Wednesday, March 7, 2012
![Page 8: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/8.jpg)
But the scale is interesEng
14 billion requests / month
>100,000 DB operaEons / second
Wednesday, March 7, 2012
![Page 9: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/9.jpg)
But the scale is interesEng
14 billion requests / month
>100,000 DB operaEons / second
>50,000 DB updates / second
Wednesday, March 7, 2012
![Page 10: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/10.jpg)
2 Developers to do it all
Typical team setup
4 product managers4 ar0sts4 frontend engineers2 backend engineers-‐ design, implementa6on, opera6on
Wednesday, March 7, 2012
![Page 11: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/11.jpg)
Wooga has dedicated game teams
Wednesday, March 7, 2012
![Page 12: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/12.jpg)
Wednesday, March 7, 2012
![Page 13: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/13.jpg)
Wednesday, March 7, 2012
![Page 14: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/14.jpg)
Oct 2009
Wednesday, March 7, 2012
![Page 15: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/15.jpg)
Oct 2009 Jan 2010
Wednesday, March 7, 2012
![Page 16: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/16.jpg)
Oct 2009 Jan 2010
Oct 2010
Wednesday, March 7, 2012
![Page 17: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/17.jpg)
Oct 2009 Jan 2010
Oct 2010 Aug 2012
Wednesday, March 7, 2012
![Page 18: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/18.jpg)
Architecture EvoluEon at Wooga
The Start
The Next Step
Best of Two Worlds
Company Values
Wednesday, March 7, 2012
![Page 19: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/19.jpg)
Oct 2009: 1st team wanted good code quality
Good code quality
Easy to understandEasy to testEasy to refactor
Wednesday, March 7, 2012
![Page 20: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/20.jpg)
Oct 2009: 1st team wanted good code quality
Good code quality
Easy to understandEasy to testEasy to refactor
Wednesday, March 7, 2012
![Page 21: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/21.jpg)
EvoluEon I: Use Ruby (on Rails)
Oct 2009
Wednesday, March 7, 2012
![Page 22: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/22.jpg)
A basic setup using sharding worked fine
app app app app app app app app app
lb
MySQL
MySQL
slave slave
Wednesday, March 7, 2012
![Page 23: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/23.jpg)
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good
Wednesday, March 7, 2012
![Page 24: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/24.jpg)
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good NO MORE
Wednesday, March 7, 2012
![Page 25: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/25.jpg)
Welcome to 6 weeks of pain!
Heavy opEmizaEons were necessary
Wednesday, March 7, 2012
![Page 26: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/26.jpg)
Welcome to 6 weeks of pain!
Heavy opEmizaEons were necessary
Numerous small fixes regarding DB config
Wednesday, March 7, 2012
![Page 27: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/27.jpg)
Welcome to 6 weeks of pain!
Heavy opEmizaEons were necessary
Numerous small fixes regarding DB configMore shards
Wednesday, March 7, 2012
![Page 28: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/28.jpg)
Welcome to 6 weeks of pain!
Heavy opEmizaEons were necessary
Numerous small fixes regarding DB configMore shardsEven more shards
Wednesday, March 7, 2012
![Page 29: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/29.jpg)
Welcome to 6 weeks of pain!
Heavy opEmizaEons were necessary
Numerous small fixes regarding DB configMore shardsEven more shardsSpliBng the model to get more shards
Wednesday, March 7, 2012
![Page 30: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/30.jpg)
Early sharding hell: 8 master and 8 slaves
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slaveslave slave slave slave
Wednesday, March 7, 2012
![Page 31: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/31.jpg)
At 500K daily users we were at a dead end
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Wednesday, March 7, 2012
![Page 32: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/32.jpg)
OUCH!
http://www.flickr.com/photos/billue_the_bear/
Wednesday, March 7, 2012
![Page 33: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/33.jpg)
Don’t break the bank
Make it fasterMake it cheaperMake it simpler
Jan 2010: Meanwhile at the 2nd team
Wednesday, March 7, 2012
![Page 34: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/34.jpg)
Don’t break the bank
Make it fasterMake it cheaperMake it simpler
Jan 2010: Meanwhile at the 2nd team
Wednesday, March 7, 2012
![Page 35: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/35.jpg)
EvoluEon II: Use Redis as main database
Oct 2009
Jan 2010
Wednesday, March 7, 2012
![Page 36: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/36.jpg)
If MySQL is a truck
Fast enough
Disk based
Robust
Fast enough disk based robust
Wednesday, March 7, 2012
![Page 37: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/37.jpg)
If MySQL is a truck, Redis is a race car
Super fast
RAM based
Fragile
Super fast RAM based fragile
Wednesday, March 7, 2012
![Page 38: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/38.jpg)
Bare metal for low latency!
app app app
lb
Re-‐dis
Re-‐dis
disk(S3)
app appapp app
Wednesday, March 7, 2012
![Page 39: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/39.jpg)
How could we apply that knowledge?
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Wednesday, March 7, 2012
![Page 40: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/40.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 41: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/41.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 42: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/42.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 43: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/43.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 44: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/44.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 45: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/45.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 46: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/46.jpg)
On-‐demand migraEons from MySQL to Redis
Wednesday, March 7, 2012
![Page 47: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/47.jpg)
Typical migraEon throughput over 3 days
Wednesday, March 7, 2012
![Page 48: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/48.jpg)
Big and staEc data in MySQL, rest goes to Redis
60 GB data
50% writes
256 GB data
10% writeshCp://www.flickr.com/photos/erix/245657047/
Wednesday, March 7, 2012
![Page 49: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/49.jpg)
One team saved the other one
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Wednesday, March 7, 2012
![Page 50: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/50.jpg)
One team saved the other one
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Wednesday, March 7, 2012
![Page 51: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/51.jpg)
We now have more than 2 million users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Wednesday, March 7, 2012
![Page 52: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/52.jpg)
We now have more than 2 million users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
AWS outagein Ireland
Wednesday, March 7, 2012
![Page 53: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/53.jpg)
10 single-‐points-‐of-‐failure -‐ no fun at all!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redis redis redis redisMySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slave slaveslave slave slave slave slave
Wednesday, March 7, 2012
![Page 54: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/54.jpg)
OUCH!
http://www.flickr.com/photos/wolfsavard/
Wednesday, March 7, 2012
![Page 55: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/55.jpg)
Architecture EvoluEon at Wooga
The Start: Ruby
The Next Step
Best of Two Worlds
Company Values
Wednesday, March 7, 2012
![Page 56: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/56.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 57: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/57.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 58: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/58.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 59: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/59.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 60: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/60.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 61: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/61.jpg)
Stateless servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 62: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/62.jpg)
Stateful servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 63: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/63.jpg)
Stateful servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 64: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/64.jpg)
Stateful servers and DBs
Server Database
Wednesday, March 7, 2012
![Page 65: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/65.jpg)
Stateful servers and DBs
Server Database
One Game Session
Wednesday, March 7, 2012
![Page 66: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/66.jpg)
Stateful servers and DBs
Server Database
One Game Session
Wednesday, March 7, 2012
![Page 67: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/67.jpg)
Oct 2010: 3rd team used a stateful server
If DBs are the problem
Don’t use themStore state in serverNeed to be robust
Wednesday, March 7, 2012
![Page 68: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/68.jpg)
Oct 2010: 3rd team used a stateful server
If DBs are the problem
Don’t use themStore state in serverNeed to be robust
Wednesday, March 7, 2012
![Page 69: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/69.jpg)
EvoluEon III: Use Erlang for a stateful server
Oct 2009
Jan 2010
Oct 2010
Wednesday, March 7, 2012
![Page 70: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/70.jpg)
Stateful servers are not as hard as you think
session
Wednesday, March 7, 2012
![Page 71: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/71.jpg)
Stateful servers are not as hard as you think
sessionsessionsessionsession
Wednesday, March 7, 2012
![Page 72: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/72.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
Wednesday, March 7, 2012
![Page 73: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/73.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 74: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/74.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 75: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/75.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 76: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/76.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 77: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/77.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 78: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/78.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 79: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/79.jpg)
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 80: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/80.jpg)
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
Wednesday, March 7, 2012
![Page 81: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/81.jpg)
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
700
Wednesday, March 7, 2012
![Page 82: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/82.jpg)
Deploying with a stateful server
In order to bring up a new version
Wednesday, March 7, 2012
![Page 83: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/83.jpg)
Deploying with a stateful server
In order to bring up a new version
Just deploy it
Hot code replacement is great!
Wednesday, March 7, 2012
![Page 84: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/84.jpg)
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
Wednesday, March 7, 2012
![Page 85: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/85.jpg)
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
Very few SPOFs-‐ ... and those are easy to recover
Wednesday, March 7, 2012
![Page 86: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/86.jpg)
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
Very few SPOFs-‐ ... and those are easy to recover
TransacEonal logic-‐ Invariants instead of explicit error handling
Wednesday, March 7, 2012
![Page 87: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/87.jpg)
Example “controller” in Erlang
Wednesday, March 7, 2012
![Page 88: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/88.jpg)
Example “controller” in Erlang
Wednesday, March 7, 2012
![Page 89: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/89.jpg)
Example “controller” in Erlang
Wednesday, March 7, 2012
![Page 90: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/90.jpg)
Example “controller” in Erlang
Wednesday, March 7, 2012
![Page 91: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/91.jpg)
Example “controller” in Erlang
Central handling of effects
Wednesday, March 7, 2012
![Page 92: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/92.jpg)
Example “controller” in Erlang
Central handling of effects
TransacEonal behavior
Wednesday, March 7, 2012
![Page 93: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/93.jpg)
Example model in Erlang
Wednesday, March 7, 2012
![Page 94: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/94.jpg)
Example model in Erlang
Wednesday, March 7, 2012
![Page 95: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/95.jpg)
Example model in Erlang
Wednesday, March 7, 2012
![Page 96: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/96.jpg)
Example model in Erlang
Wednesday, March 7, 2012
![Page 97: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/97.jpg)
Example model in Erlang
Wednesday, March 7, 2012
![Page 98: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/98.jpg)
Example model in Erlang
Erlang code is not that hard to read, isn’t it?
Wednesday, March 7, 2012
![Page 99: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/99.jpg)
http://www.flickr.com/photos/hotreactor/
Wednesday, March 7, 2012
![Page 100: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/100.jpg)
Architecture EvoluEon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds
Company Values
Wednesday, March 7, 2012
![Page 101: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/101.jpg)
Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustnessGreat for opera0on
Wednesday, March 7, 2012
![Page 102: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/102.jpg)
Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustnessGreat for opera0on
Ruby is great
Concise, expressive, testableGreat for development
Wednesday, March 7, 2012
![Page 103: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/103.jpg)
Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustnessGreat for opera0on
Ruby is great
Concise, expressive, testableGreat for development
Wednesday, March 7, 2012
![Page 104: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/104.jpg)
Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustnessGreat for opera0on
Ruby is great
Concise, expressive, testableGreat for development
Wednesday, March 7, 2012
![Page 105: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/105.jpg)
EvoluEon IV: The best out of two worlds
Oct 2009
Jan 2010
Oct 2010
Aug 2011
Wednesday, March 7, 2012
![Page 106: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/106.jpg)
The basic setup looks exactly like before
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
Wednesday, March 7, 2012
![Page 107: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/107.jpg)
Example controller in Ruby
Wednesday, March 7, 2012
![Page 108: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/108.jpg)
Example controller in Ruby
Wednesday, March 7, 2012
![Page 109: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/109.jpg)
Example controller in Ruby
Wednesday, March 7, 2012
![Page 110: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/110.jpg)
Example controller in Ruby
Wednesday, March 7, 2012
![Page 111: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/111.jpg)
Example controller in Ruby
DSL-‐like definiEon of game acEon
Wednesday, March 7, 2012
![Page 112: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/112.jpg)
Example controller in Ruby
DSL-‐like definiEon of game acEon
Skinny as controllers should be
Wednesday, March 7, 2012
![Page 113: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/113.jpg)
Example model in Ruby
Wednesday, March 7, 2012
![Page 114: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/114.jpg)
Example model in Ruby
Wednesday, March 7, 2012
![Page 115: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/115.jpg)
Example model in Ruby
Wednesday, March 7, 2012
![Page 116: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/116.jpg)
Example model in Ruby
Wednesday, March 7, 2012
![Page 117: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/117.jpg)
Example model in Ruby
Easily unit testable
Wednesday, March 7, 2012
![Page 118: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/118.jpg)
Example model in Ruby
Easily unit testable
Minimal amount of code
Wednesday, March 7, 2012
![Page 119: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/119.jpg)
Bringing 2 worlds together
Server
session
session
...
session
Wednesday, March 7, 2012
![Page 120: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/120.jpg)
Bringing 2 worlds together
Server
session
session
...
session
sender
Wednesday, March 7, 2012
![Page 121: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/121.jpg)
Bringing 2 worlds together
Server
session
session
...
session
Worker
Worker
Worker
Worker
Worker
sender
Wednesday, March 7, 2012
![Page 122: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/122.jpg)
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender
Wednesday, March 7, 2012
![Page 123: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/123.jpg)
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender
Wednesday, March 7, 2012
![Page 124: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/124.jpg)
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender
Wednesday, March 7, 2012
![Page 125: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/125.jpg)
Game state
Game state is split in mulEple parts
user, map, fruit_trees etc.
Wednesday, March 7, 2012
![Page 126: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/126.jpg)
Game state
Game state is split in mulEple parts
user, map, fruit_trees etc.
Erlang does not care about content
Serialized Ruby objects
Wednesday, March 7, 2012
![Page 127: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/127.jpg)
Game state
Game state is split in mulEple parts
user, map, fruit_trees etc.
Erlang does not care about content
Serialized Ruby objects
Erlang does know mapping of state parts to URLs
Mapping provided by Ruby on startup
Wednesday, March 7, 2012
![Page 128: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/128.jpg)
Looking back at the game acEon
Wednesday, March 7, 2012
![Page 129: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/129.jpg)
Looking back at the game acEon
Wednesday, March 7, 2012
![Page 130: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/130.jpg)
Looking back at the game acEon
Mapping of state parts to game acEons
Wednesday, March 7, 2012
![Page 131: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/131.jpg)
Looking back at the game acEon
Mapping of state parts to game acEonsWorker knows mapping
Wednesday, March 7, 2012
![Page 132: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/132.jpg)
Looking back at the game acEon
Mapping of state parts to game acEonsWorker knows mappingWorker pushes mapping to Erlang on startup
Wednesday, March 7, 2012
![Page 133: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/133.jpg)
Looking back at the game acEon
Mapping of state parts to game acEonsWorker knows mappingWorker pushes mapping to Erlang on startupErlang can query mapping if needed
Wednesday, March 7, 2012
![Page 134: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/134.jpg)
NICE!
http://www.flickr.com/photos/aigle_dore/
Wednesday, March 7, 2012
![Page 135: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/135.jpg)
Architecture EvoluEon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds
Company Values
Wednesday, March 7, 2012
![Page 136: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/136.jpg)
Each new game brought us innovaEon
Oct 2009
Jan 2010
Oct 2010
Aug 2011
Wednesday, March 7, 2012
![Page 137: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/137.jpg)
Small teams
overbig teams
We’ve learned to value
Wednesday, March 7, 2012
![Page 138: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/138.jpg)
Collaboration
overcompetition
We’ve learned to value
Wednesday, March 7, 2012
![Page 139: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/139.jpg)
Generalists
overspecialists
We’ve learned to value
Wednesday, March 7, 2012
![Page 140: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/140.jpg)
Effort reduction
overcost reduction
We’ve learned to value
Wednesday, March 7, 2012
![Page 141: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/141.jpg)
Innovation
overrisk mitigation
We’ve learned to value
Wednesday, March 7, 2012
![Page 142: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/142.jpg)
A good value system
We’ve learned to value
Small teams over
CollaboraEon over
Generalists over
Effort reducEon over
InnovaEon over
Big teams
Compe00on
Specialists
Cost reduc0on
Risk mi0ga0on
Wednesday, March 7, 2012
![Page 143: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/143.jpg)
It works!
Wednesday, March 7, 2012
![Page 144: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/144.jpg)
It works!
Be fast, be bold!
Wednesday, March 7, 2012
![Page 145: Games for the Masses (QCon London 2012)](https://reader033.vdocuments.site/reader033/viewer/2022052619/555066fab4c905c0448b54b9/html5/thumbnails/145.jpg)
QuesEons?
Jesper Richter-‐Reichhelm@jrirei
slideshare.net/woogawooga.com/jobs
Wednesday, March 7, 2012