jesper richter-reichhelm - continuous evolution at wooga - code.talks 2015

Continuous Evolution at Wooga

Jesper Richter-Reichhelm @jrirei

BackendClient

Game EvolutionIterative improvements

Wooga team

FE Dev

BE Dev

Art

Product

“You build it, you run it. - Werner Vogels

BackendClient

Monster World

SQL

app

lb

SQL

Browser

sql sql

appapp

Monster World

FE Dev

BE Dev

Art

Product

DevOps (just 2 devs involved)

Daily iterations (sometimes faster)

Monster World

Fixes

Optimizations

Bigger DBs

More DBs

app

lb

Browser

SQL

sql

appapp

lb

app appappapp appapp

SQL

sql

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql sql

SQL

sql

SQL

sql

SQL

sql

SQL

sql

SQL

sql

SQL

sql

lb

Stateless approach

1+ million users / day

10+ billion requests / month

~10 DB writes / request

> 100,000 DB writes / second

http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

Monster World

Monster World Happy Hospital

Added Redis (to MySQL)

Migrating users (on demand)

Monster World

Tests worked fine (using their code)

Live it worked badly (50K users corrupted)

Fixed it quickly (and repaired the users)

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

app appapp app app app app app app app app appapp app app







SQL

EBS EBS EBS EBS

lb lblblb

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd








SQL

EBS EBS EBS EBS

lb lblblb

38 single points of failure

flickr.com/photos/qubodup/9244593352

http://flickr.com/photos/photomonkey/12746989

Why did we survive?

Monster World

Boyd Cycle

Col. John Boyd, USA

Korea war: F-86 won 10:1 fight against MiG-15

Time competitive cycles of action

Effects of faster decisions are cumulative!

Orient

Observe

Decide

Act

O

O

D

A

OODAIndependent teams (autonomous decisions)

Experiments (ab tests)

Agile approach (Scrum, Kanban etc.)

Quick iterations (4h, 1d, 7d, 14d)

OODA loop

Tech EvolutionSpreading and mutating ideas

Wooga teams

Freedom

Magic Land

Server Database

Server Database

One Game Session

Stateful approach

1+ million users / day

4 sessions / user / day

1 DB write / session

~50 DB writes / second

Server

sessionsessionsessionsession

Server


Server


S3

http://www.flickr.com/photos/aigle_dore/

http://www.flickr.com/photos/aigle_dore/

How does that work?

Reproduction

flickr.com/groups/bestfavoriterelayrace/

Rotating jobs


Selection and recombination

Exchanging knowledge

flickr.com/photos/wili/233621595


Being transparent

flickr.com/photos/marcomagrini/698692268


Mutation

Freedom

Having trust

Evolution

Company EvolutionEmergent behaviour

Tech Stack

2009

Ruby

LAMP

statelessBB BI

MW

Redis HH

LAMP

statelessBB BI

Ruby MW MW 2

Ruby MW MW 2

2010

LAMP

statelessBB BI

Redis HH

Ruby MW MW 2

2010

Erlang

statefulML

2011

Redis

Ruby

LAMP

statelessBB BI

MW

HH

MW 2

Erlang

statefulML

offline

DD

Erlang

statefulML

2011

Redis

Ruby

Riak

Javascript

ev. Ruby

1 doc

LAMP

statelessBB BI

MW

HH

MW 2

PI

PP

MWM

2012

Erlang

Redis

Ruby

stateful

Riak

Javascript

offline

ev. Ruby

1 doc

JRuby

shared

LAMP

multiplay

statelessBB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

GX

FT

KB

BI 2

2013

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

MA

2014

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

MA …

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

MA …

…

…

…2015

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services2015

MySQLPHP

PHP MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

PHP MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

Wooga’stoolbox

Switch to Mobile

PI

MWM PV

Switch to Mobile HTML5

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

JS

GX

BI 2

DIV

HTML5

Obj. C

Apportable

ext. Android

BB BI

MW

HH

MW 2

ML

Switch to Mobile

DD

PI

PP

MWM PV

JS

GX

BI 2

DIV

HTML5

Obj. C

Apportable

ext. Android

SD / DYN

… …

…

…

…

AAUnity

BB BI

MW

HH

MW 2

ML

Switch to Mobile

Why do we do that?

Wooga benefits

The ability to quickly adapt to change Survive disruptions and benefit from them

Resiliency and robustness Long-term perspective vs. short-term goals

To attract strong people We give transparency and trust, we want impact

Continuous EvolutionThe right approach for everyone?

Reinventing the wheel?

Knowledge sharing is the key

Transparency is mandatory

Reusing

code / SDKs

services

best practices

FE Dev

BE Dev

Art

Product

FE Dev

Art

Product

BE Dev

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

MA …

…

…

…SBS

Flexibility Innovation

Individual Impact

Efficiency Planning

Scale benefits

Wooga

Autonomy Centralisation

Autonomy Centralisation

Flexibility Innovation

Individual Impact

Efficiency Planning

Scale benefits

Wooga

What’s right for your company?

Questions?

Jesper Richter-Reichhelm (@jrirei)

jesper richter-reichhelm - continuous evolution at wooga - code.talks 2015

Internet