rapidly developing farmville

Rapidly Developing FarmVilleHow we built and scaled a

#1 Facebook game in 5 weeks

Amitt MahajanLead Developer – FarmVille

March 9, 2010

FarmVille

6 developers (PHP/Flash)2 artists3 producers/designers

5 week development cycle

FarmVille

18K users/day after 24 hrs1M users/day after 4 days

At Launch

110M+ Installs 31M Players/Day

Today

My Goals

• Reduce the amount of time it takes you to develop your games

• Give you a blueprint for how to reliably scale your games to millions without your servers falling over

What slows down developers?

• Other developers• Design / Art• Production (Copy)• Lack of knowledge

Increasing Developer Efficiency• All developers know both

• Design DOESN’T rule all• Developers co-own features with designers

Flash (Client)PHP (Server)

Data Driven Design

• Shoot for a content pipeline that doesn’t need a developer

• Data driven can be as easy as a designer-editable XML file

No dev needed!

String Tables

• A string table is an external file that holds strings for the app

• Best practice to prepare for localization• Developers are not blocked by production• Allows quick response to Facebook TOS

changes

Abstracted Network Layer

Goal Feature developers get client/server communication and serialization for free

AMF Protocol (RPC)

Flash PHP

Network Layer

Validation Checks

Action Batching

Networked Actions

Feature A Feature B

Network Layer

Validation Checks

Result Batching

Feature A Service

Feature B Service

Action Dispatcher

Client

Network Layer Batching

• No need to do the work twice• Reduced server load in FarmVille by 50%!

Plow

Plant

Plow

PlantPlow

Only One Request

ServeronPlow()

onPlow()

onPlant()…

Network Layer Validation

• Solves problem of unfiltered input• Ensures data is received by server in-order it

was sent

Social Network Wrapper

• Single place to perform Facebook API calls• Working with a highly dynamic API can be

difficult• Abstracting FB calls makes them easier to

integrate• Allows for cross-platform games

Continuous Deployment

• Build latest version of source-repo• Deploy to test “auto app” on Facebook to

surface production issues early

Continuous Deployment

FarmVille QA Process

QuickSmoke

Full TestPass

Auto app Staging Production

FarmVille Traffic Growth

1 Million DAUsevery week for 20 weeks

source: developeranalytics.com

Scalable Server Architecture

• We had no choice but to scale on the cloud• Every part of the server architecture scales

horizontally• No single points of failure• Take the DB out of the equation

Memcache Pool

Auto-scaling web array (PHP)

Web Server #1 Web Server #M

User Data & Updates

…

DataReads

Lazy Writes

DB Layer

Round Robin DNS

Load Balancer #1 Load Balancer #N

…

… DB-#KSlave

DB-#1Slave

DB-#KMaster

DB-#1Master

…

Web Server #1 Web Server #M

…

…

All you need to implement is…

…everything else is off-the-shelf components!

Reducing Load-times

• Users want responsive pages• Show something immediately

• Player’s won’t wait for your app to load• Stream non-critical content

Avoid Remote Calls• Remote server calls are slow and unreliable

• Aim for no remote calls during load• Embed data into Javascript

Facebook

Iframe

Inline-JS here

Flash ClientEmbed

Caching Slow Calls

• Build in Facebook API caching within the social network wrapper

• Write a DB wrapper to generate and cache SQL

FarmVille Page Profiler

Goal Catch and eliminate all un-cached Facebook and DB calls

Fault Tolerance

• Servers do go down randomly• Build redundancy on all levels of server

architecture• Facebook is a dependency also

• Aim for DB-less & Facebook-less modes• “Defcon”-style error management

Runtime Config

• How do we progressively keep the game running?• Create all features with kill-switches• Create a web dashboard to allow non-

technical folks to help out

Is it still running?

• Notify your team when things break using server monitoring (Nagios/Munin)

• What to watch• Server load/traffic graphs• Memcache evictions

Stats Tracking

• How do we know what users are seeing?• Have the client send back statistics

• Number of errors• Load-times

• Take a metrics driven approach to error handling

Before Launching

• Get confidence that your stuff actually works• Perform load testing

• Social games are a marathon not a sprint• Sleep before launching!

Questions?

Thank You!