![Page 1: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/1.jpg)
Building the Server Software for Eliminate
![Page 2: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/2.jpg)
![Page 3: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/3.jpg)
Introduction
Stephen Detwiler Director of Engineering, ngmoco:)
James Marr Lead Engineer R&D, ngmoco:)
![Page 4: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/4.jpg)
Introduction
Build the definitive FPS for iPhone in only 5 months
Multiplayer deathmatch wifi and 3g
Free to play
With three engineers
![Page 5: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/5.jpg)
Outline
Gameplay Lobby Matchmaking Load Testing Live Tuning Deployment Monitoring
![Page 6: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/6.jpg)
Server Architecture
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
![Page 7: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/7.jpg)
Gameplay
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Topic 1 of 7
![Page 8: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/8.jpg)
Gameplay: Requirements
3G requirement drives decision ~100kbps, 150ms latency
Aggressive bandwidth optimization Prediction to hide latency UDP
![Page 9: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/9.jpg)
Gameplay: Options
Are there any opensource options? Shipping to clients, so no GPL
Are there any commercial options?
Yes, Quake 3
Dialup from 1999 looks a lot like 3G from 2009
![Page 10: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/10.jpg)
Gameplay: Q3 Cost
Source code plus full rights minus any technical support = $10k
Same cost as a man month
![Page 11: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/11.jpg)
Gameplay: Q3 Benefits
Graphics BSP + portals Dynamic lights, static lightmaps Keyframe animation
Tools Custom map editor (Radiant) 3DS Max model animation exporters
Lots of information online about how to extend the engine
![Page 12: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/12.jpg)
Gameplay: Moving On
Purchased solution for “mundane” gameplay networking
Able to focus on rest of experience
![Page 13: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/13.jpg)
Lobby
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Topic 2 of 7
![Page 14: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/14.jpg)
Lobby: Requirements
Handles everything outside of realtime gameplay Inventory and commerce Proxy to Plus+ services Chat Matchmaking requests Party management
Support 10K+ concurrent users
![Page 15: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/15.jpg)
Lobby: Approach
Rejected: Periodic HTTP polling Easy to scale
Lots of HTTP front ends Big database backend
Latency will be high in many cases TCP socket setup over 3G is slow
Sometimes over 2 seconds!
Hard to tell when users go away Must have timeout thresholds
![Page 16: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/16.jpg)
Lobby: Approach
Chosen: Persistent TCP socket Only one initial TCP setup User is gone when socket closes Much lower message delivery latency Can push messages Harder to scale
One socket per user
![Page 17: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/17.jpg)
Lobby: Implementation
This will take more than 5 months to build. What can we use off the shelf?
Yes, XMPP
![Page 18: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/18.jpg)
Lobby: XMPP
Jabber/IM/Google Talk Proven to be scalable
TCP with XML payloads Can also route custom messages Many off the shelf implementations
jabberd, jabberd 2.x, ejabberd , etc.
![Page 19: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/19.jpg)
Lobby: Evaluating
jabberd and jabberd 2.x C/C++ codebase Not actively supported Early testing showed it did not scale well past 1000 users
Implementation difficult to extend
![Page 20: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/20.jpg)
Lobby: Evaluating
ejabberd Highly scalable
Load tested to 30K concurrent users
Extendable Active community
But written in erlang
![Page 21: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/21.jpg)
Lobby: Erlang
{Priority, RepackGameServers, IsGameServer} = case FromSession#ng_session.is_admin of true ->
case lists:filter(fun({"isGameServer", _IsGS}) -> true; (_) -> false end, OriginalAttributes) of [{_, IsGS}] -> {"0", "0", IsGS}; _ -> {"0", "0", "1"} end;
false -> AnyEnergy = does_any_player_have_energy(Players), case AnyEnergy of true -> {"1", "0", "0"}; _ -> {"0", "1", "0”} end
end,
![Page 22: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/22.jpg)
Lobby: Erlang
Functional language Crazy syntax Distributed message passing built
into language Data persistence occurs in
database
![Page 23: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/23.jpg)
Lobby: Plus+ Integration
Users log into XMPP using Oauth credentials from Plus+
Plus+ Friends and Followers populate user’s XMPP roster
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Plus
+
![Page 24: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/24.jpg)
Lobby: Scaling
ejabberd clusters well Almost for free using erlang
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Plus
+
![Page 25: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/25.jpg)
Lobby: Inventory & Purchasing All persistent data stored in Plus+ XMPP validates and caches data XMPP nodes can start and stop at
anytime iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Plus
+
![Page 26: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/26.jpg)
Matchmaking
iPhone
Matchmaking
Adm
inis
trat
ion
Servers
Game Servers
Geographically Distributed
Lobby
Topic 3 of 7
![Page 27: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/27.jpg)
Matchmaking: Goals
Console quality matchmaking
Dirt simple user experience Press a button Play against fun opponents
![Page 28: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/28.jpg)
Matchmaking: Options
Are there commercial options? Microsoft? Infinity Ward? Blizzard?
Are there opensource alternatives?
No. We’re building our own
![Page 29: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/29.jpg)
Matchmaking: Overview
Matchmaking server Receives requests from Lobby server Finds a good grouping of players Launches game server instance Inform clients through Lobby server
![Page 30: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/30.jpg)
Matchmaking: Instances
Quake 3 dedicated server is one process per concurrent game
Game manager on each server Talks to matchmaking server Launches instances on-demand Reports max instance capacity
![Page 31: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/31.jpg)
Matchmaking: Approach
Rejected: SQL DB All state stored in DB Query DB, process results, repeat Easy to cluster, provide redundancy
High data latency Complicated
![Page 32: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/32.jpg)
Matchmaking: Approach
Accepted: In Memory All players kept in memory Higher performance Fast to implement
Won’t cluster, one box must do it all Server crashes lose some data
![Page 33: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/33.jpg)
Matchmaking: Qualities
Each player has qualities Estimated skill Character level Desired party size Ping times to datacenters Time waiting in matchmaking
Find others with similar qualities Start with narrow tolerances Over time, if can’t find a match, dilate
tolerances for qualities
![Page 34: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/34.jpg)
Matchmaking: Qualities
0
750
1500
2250
3000
0 3 6 9 12 15
Skill d
iffe
ren
ce t
ole
ran
ce
Seconds in matchmaking
0
1
2
3
4
5
0 3 6 9 12 15
Min
imu
m p
art
y si
ze
Seconds in matchmaking
![Page 35: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/35.jpg)
Matchmaking: Algorithm
Sort players by one quality We choose Estimated Skill
For each player: Find other candidate players by
iterating forward and backwards until outside of skill tolerance
Evaluate other quality tolerances for each candidate
Form match if enough candidates pass
![Page 36: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/36.jpg)
Skill
Matchmaking: Algorithm
Name: Me Skill: 1000 Level: 15 Loc: SFO
Name: A Skill: 200 Level: 2 Ping: 100ms
Name: D Skill: 1700 Level: 14 Ping: 80ms
Name: E Skill: 2200 Level: 21 Ping: 160ms
Name: B Skill: 750 Level: 13 Ping: 125ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
![Page 37: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/37.jpg)
Skill
Matchmaking: Algorithm
Name: Me Skill: 1000 Level: 15 Loc: SFO
Name: A Skill: 200 Level: 2 Ping: 100ms
Name: D Skill: 1700 Level: 14 Ping: 80ms
Name: E Skill: 2200 Level: 21 Ping: 160ms
Time: 1 second Skill Tolerance: 500 Level Tolerance: 2
Name: B Skill: 750 Level: 13 Ping: 125ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Candidate Players
![Page 38: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/38.jpg)
Skill
Matchmaking: Algorithm
Name: Me Skill: 1000 Level: 15 Loc: SFO
Name: D Skill: 1700 Level: 14 Ping: 80ms
Name: E Skill: 2200 Level: 21 Ping: 160ms
Time: 2 seconds Skill Tolerance: 1000 Level Tolerance: 4
Name: B Skill: 750 Level: 13 Ping: 125ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Candidate Players
Name: A Skill: 200 Level: 2 Ping: 100ms
Name: A Skill: 200 Level: 2 Ping: 100ms
![Page 39: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/39.jpg)
Skill
Matchmaking: Algorithm
Name: Me Skill: 1000 Level: 15 Loc: SFO
Name: D Skill: 1700 Level: 14 Ping: 80ms
Name: E Skill: 2200 Level: 21 Ping: 160ms
Time: 3 seconds Skill Tolerance: 1500 Level Tolerance: 6
Name: B Skill: 750 Level: 13 Ping: 125ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Name: C Skill: 1300 Level: 17 Ping: 370ms
Candidate Players
Name: A Skill: 200 Level: 2 Ping: 100ms
Name: A Skill: 200 Level: 2 Ping: 100ms
![Page 40: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/40.jpg)
Matchmaking: Algorithm
Name: Me Skill: 1000 Level: 15 Loc: SFO
Name: D Skill: 1700 Level: 14 Ping: 80ms
Name: E Skill: 2200 Level: 21 Ping: 160ms
Name: B Skill: 750 Level: 13 Ping: 125ms
![Page 41: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/41.jpg)
Matchmaking: Skill
Players start with skill of zero After match, update skill estimate
based on previous skill estimate and match outcome
Veteran beating noob veteran += little noob -= little
Noob beating veteran noob += big veteran -= big
![Page 42: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/42.jpg)
Matchmaking: Skill
Math loosely based on Halo 2 Early values are positive sum game Middle values are zero sum game Late values are negative sum game
-100%
0%
100%
0 2500 5000 7500 10000
Skill P
oin
ts A
dd
ed
/
Rem
ove
d f
rom
Sys
tem
Player Skill
![Page 43: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/43.jpg)
Matchmaking: Speed
Need < 10% wait / play ratio Status quo
~ 10+ minutes per match ~ 1+ minutes to find opponents
Eliminate ~ 3 minutes per match ~ 15 seconds to find opponents
![Page 44: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/44.jpg)
Matchmaking: Capacity
Can’t cluster, must be confident one box can handle load
Algorithm is worst case θ(n2), expected θ(n)
From unit testing, one box can handle 50k players / second <10% of player time in matchmaking,
so supports 500k concurrent users
![Page 45: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/45.jpg)
Matchmaking: Faults
Two matchmaking servers Primary, backup
Clients refresh match request every 4 seconds
System switches to backup if primary stops responding
Backup doesn’t know how long players had been in matchmaking
![Page 46: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/46.jpg)
Matchmaking: Wrinkle
Initially, character level was ignored by matchmaking Thinking: estimated skill = actual skill + character level
HUGE outcry from users Incorporated character level in 2.0
![Page 47: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/47.jpg)
Load Testing
iPhone
Matchmaking
Adm
inis
trat
ion
Game Servers
Geographically Distributed
Lobby
Topic 4 of 7
![Page 48: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/48.jpg)
Load Testing: Why
Not enough hardware at launch Users won’t come back
Spend all of your money hardware You don’t make a sequel
![Page 49: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/49.jpg)
Load Testing: How
Build tools to generate load for each component Measure CPU, memory and bandwidth
Build model to estimate requirements at different usage levels DAUs, Concurrent Users, Session
Length
Re-test often
![Page 50: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/50.jpg)
Load Testing: XMPP
Simulate player XMPP actions Login, chat, inventory, etc.
Reuse actual XMPP client code Repurposed game manager
hardware Ran up to 30K users
![Page 51: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/51.jpg)
Load Testing: Matchmaking Unit test code easily matched 50k
players / second on a laptop
![Page 52: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/52.jpg)
Load Testing: Game Managers Take 1 Needed to run actual game to
generate realistic load Only ran on iPhone
Built headless version for OS X Not enough resources available to
stress even one game manager
![Page 53: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/53.jpg)
Load Testing: Game Managers Take 2 Measured server load per single
game instance Created tool to generate matching
cpu load Continued spawning until OS
scheduler fell apart Reasonable results but not great
Learned more when we went live
![Page 54: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/54.jpg)
Live Tuning
iPhone
Matchmaking
Servers
Game Servers
Geographically Distributed
Lobby
Live
Tun
ing
Topic 5 of 7
![Page 55: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/55.jpg)
Live Tuning: Overview
Must be able to tune game experience based on user feedback Weapon and armor strength Items for sale and price in store Regulating stat frequency
![Page 56: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/56.jpg)
Live Tuning: Plists
Configuration stored in plist Client downloads latest version to
drive UI, modify gameplay Servers consume latest version to
configure behavior, validate purchases
![Page 57: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/57.jpg)
Live Tuning: Problem
Initial implementation did not scale XML plist used to make erlang parsing
easier Served as base64 encoded XMPP
message
![Page 58: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/58.jpg)
Live Tuning: Problem
80KB plist at launch Quickly grew past 200KB Bandwidth usage spikes when
change published 400+Mbps during update
0
100
200
300
400
500
Peak
Average
![Page 59: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/59.jpg)
Live Tuning: Fix
Eliminate 1.1 added more tuning plist exceeds 400KB New version announced via XMPP Downloaded over gzipped HTTP Bandwidth usage now about 120Mbps
0
100
200
300
400
500
Peak
Average
![Page 60: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/60.jpg)
Deployment
iPhone
Matchmaking
Game Servers
Geographically Distributed
Lobby
Dep
loym
ent
Topic 6 of 7
![Page 61: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/61.jpg)
Deployment: Overview
Eliminate uses lots of servers 4 XMPP 2 Matchmaking 8 Game Managers 2 Management
Production, Staging and Development deployments
How do we deploy and manage?
![Page 62: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/62.jpg)
Deployment: Release Management Servers run Ubuntu 9.04 64 bit Components deployed with apt-get
Versioned releases Software dependency tracking Robust upgrade path
24 packages for Eliminate
![Page 63: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/63.jpg)
Deployment: Release Management Control script knows about all
machines in the cluster Full system upgrades in under 1 minute $ ./control.py upgrade
Can upgrade subsystems easily $ ./control.py upgrade –c livefire-‐matchmaking
![Page 64: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/64.jpg)
Deployment: Geography
XMPP, matchmaking and management servers at ngmoco:)
Geographically distributed game managers
sfo ams ord
iad nrt
![Page 65: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/65.jpg)
Deployment: Scaling
We run hardware to meet our expected daily user load But concurrent user spikes occur
Promotions New content creates renewed interest
Disable energy timer Content updates 1.1 release
![Page 66: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/66.jpg)
Deployment: Scaling
XMPP deployment can handle 20k concurrent users Can add new capacity in 60 minutes if
required
Matchmaking overbuilt so it never has to scale
Match 50K requests/second
![Page 67: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/67.jpg)
Deployment: Scaling
Amazon EC2 is our safety valve for game managers
New game managers in 5 minutes High-CPU Extra Large (c1.xlarge)
EC2 Regions: US-East EU-West
![Page 68: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/68.jpg)
Deployment: Scaling
Why not use EC2 for everything? Compute time is cheap Bandwidth is not
EC2
Co-locate
![Page 69: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/69.jpg)
Monitoring
iPhone
Matchmaking
Mon
itor
ing
Servers
Game Servers
Geographically Distributed
Lobby
Topic 7 of 7
![Page 70: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/70.jpg)
Monitoring: Tools
Need to track health of the system nagios
Hardware health checks Text messages on component failure
munin Visually graphs trends over time Bandwidth CPU Memory
![Page 71: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/71.jpg)
Monitoring: Custom Tools
Custom munin plugins Players online People waiting to get in a game Estimated wait time Active games
Great for long term trends Not good for immediate feedback
![Page 72: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/72.jpg)
Conclusion
It took eight months Turns out this is hard
What we learned that you should know Reuse systems when possible Do load testing early and often Design a system that can scale
![Page 73: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/73.jpg)
We’re Hiring ;)
Did this sound fun? We’re looking for exceptional
engineers
![Page 74: Building the Server Software for Eliminatetwvideo01.ubm-us.net/o1/vault/gdc10/slides/... · jabberd, jabberd 2.x, ejabberd , etc. Lobby: Evaluating ... New version announced via XMPP](https://reader035.vdocuments.site/reader035/viewer/2022081612/5f2913e699b9365c5522821c/html5/thumbnails/74.jpg)
Thank You
Questions?