Slide 1

Dedicated Servers in Gears of War 3 Scaling to Millions of Players Michael Weilbacher Development Manager, Microsoft Studios Slide 2 Introductions Michael Weilbacher Technical Development Manager at Microsoft 1.5 years at Microsoft 16.5 years in the game industry Shipped games: Gears of War 3, Magic the Gathering: Tactical, Mortal Kombat: Deception to Mortal Kombat vs. DC Universe, John Woo presents Stranglehold, NBA Ballers, Blitz, MLB Slugfest, Psi-Ops: The Mindgate Conspiracy, NASCAR 02-03, Madden NFL 97-03, NCAA Football 97-02 and some more. Slide 3 Topics From the beginning to the end What are/why dedicated servers The consumer experience The associated cost Game code decisions Administering the servers Implementation rollout Out in the wild Trends and the future Gears of war 3 dedicated servers Slide 4 What are dedicated servers? 32-bit headless client instance without renderer/user input Multiple clients hosted on a single server Servers hosted in a datacenter Multiple datacenters worldwide support the community Software infrastructure that ties it all together What are / why dedicated servers Slide 5 Why dedicated servers? Best game experience Datacenters provide high bandwidth, low latency Increased host performance Consistency between games What are / why dedicated servers Addresses Gears 2 problems Prevent host latency advantages Reduce host quitting and game interruption Cheaters and lag switches Community perception/expectations Decided against distributing the server to the public Reduces problem scope Security concerns Control the experience with consistent performance/bandwidth The downside of hosting is the increase to the game cost Slide 6 Overview of our datacenters Four large datacenters Four small datacenters Over 900 servers worldwide Average ~70 users per core What are / why dedicated servers Slide 7 What is our latency tolerance? < 150ms is playable, 50-90 is best Average case after launch at datacenters were 75 ms Able to tweak by region Oceania / Asia requirements relaxed slightly after launch During development Playtest labs tested worst case Artificial latency >200ms Packet loss @ 5-10% The consumer experience Slide 8 Finding the server hosted game Each server hosted game is assigned an ID based on datacenter Each client is assigned one of these IDs based on an IP to location lookup In matchmaking query Client looks for a server hosted game with the ID Hosted game balance experience and TrueSkill rating based on players that join The consumer experience Slide 9 Consumers finding the best match Matchmaking returns servers in TrueSkill/XP range 4 types of queries Best looking for exactly my party size Any looking for any match that fits the party Empty configure a new host from the shared pool Default to peer to peer Lots of knobs to tweak allows much control over matchmaking experience The consumer experience Slide 10 Games always available Always fallback to a player hosted match Necessary if we ever phase out servers over the life of the project Underestimating server need should never affect players Some people are just not close to datacenters Favor server experience Die roll to balance between "host vs. client rich Servers can host migrate if needed Servers to peer to peer migration, but not server to server Tracking this metric shows host migration is rare ~0.17% of matches The consumer experience Slide 11 How much hardware do we need for day one? Historical data from previous game Gears 2 multiplayer data Gears supports 10 players max Sales forecast per region Formula driven Assumed 15% attach rate online and 30% concurrent rate Can be costly if you are wrong If too little, then community is unhappy If too much, the accountants are unhappy Easier to ask for the accountants forgiveness The associated cost Slide 12 How much hardware: should you buy versus rent? Purchased enough for long term needs (not peak) Rented over 45% in US Rented in regions which were hard to setup big deployments GameServers.Com The associated cost Slide 13 Monthly cost Hardware is not most expensive part About the graph: At our highest cost bandwidth facility Hardware amortized over 36 months The associated cost Slide 14 How much bandwidth do you need? Our average hosted game sends out ~7kb/sec Our average consumers sends in ~4kb/sec VOIP traffic is peer to peer to reduce host bandwidth requirement Cost savings: Pay for burst (more costly) versus committed (long term) More upfront, but cheaper over lifecycle The associated cost Slide 15 Match making: LSP or XLIVE/G4WL? Punch through LSP? Extra level of indirection Extra latency Roll your own matchmaking (no advertising on LIVE) Non-starter Games for Window Live? Acts as a headless client Codebase built around LIVE already (UE3 / Gears1 / Gears PC / Gears 2) Only minor and focused additions/changes required Game code decisions Slide 16 G4WL challenges Still beholden to client rules CD Key / Local admin account necessary per instance Need one local account for each game process on servers One live account for each hosted game 1 Gamertag for every 10 users Microsoft Platform created a custom tool to generate all the accounts Manually creating initial 50 Gamertags was no fun Over 100k Gamertags created! Game code decisions Platform did not maintain the accounts for us Manually accepting Terms of Service for every Gamertag Used a web testing solution to help upgrade accounts when account terms changed Very painful for all parties involved Talk to your Developer Account Manager before you go down this route Slide 17 Modifications to the existing UE3 dedicated server platform Sitting idle Needed to restart every 10 minutes to pull down possibly new information Dynamically need to configure themselves with new updates Transition period where clients and hosts are sync'd up Detecting "empty" and resetting People start to go into the game and do not make it People stop playing and server needs to become available again Server shutdown whitelist Need to be able to shutdown gracefully for upgrades/maintenance Game code decisions Auto configure when the first party joins and re-advertise Players make a request for what game mode they want to play and the game needs to setup Empty server pool shared across all playlists and configurations General robustness Needs a solid uptime, error conditions, shutdown Fortunately not a single crash during the beta However precision issues creep in after 48 hours, so we reboot as players roll off servers close to that mark Lots of memory leak testing Lots of logging, events, perf counters (more on that later) Most of these have been integrated back to UE3 Slide 182008R2 was 2x win (Vista -> Win7 kernel) "> Memory and Performance Memory was not as a big deal as performance Servers run under 150MB/instance Memory was cheap on the server Memory optimizations The associated cost Set a goal for a solid 30 fps network tick rate Simulated load with automated bot matches Charted fps via performance counters 2.5 hosted games per core (2009 Gears 2) 7.2 hosted games per core (2011 Gears 3) Major performance wins Stripped out the renderer Lots of time spent removing "visual effects" code paths Get the whole team thinking about dedicated servers Moving from Server 2008 -> 2008R2 was 2x win (Vista -> Win7 kernel) Slide 19 Lessons learned Servers load much faster than clients Server told clients to load things before they had unloaded previous maps -> higher watermark and occasional OOM errors Introduced configurable latency before loading next map No intrinsic first player assumption Slow to connect players were missing the game based on checks that assumed player host existed More code to check that at least one player existed before running existing checks Mixing client and server side optimizations Lots of animation optimizations "last render time" code had to be double checked Invisible collision in a few instances where the animation never played leaving collision in a bad state Make sure the "server" Gamertag was never exposed to the clients Made sure arbitrated sessions did not include server in the TrueSkill calculation Never registered a session for the server Game code decisions Slide 20 Reporting systems Created by Games IT at MS SCMM Monitoring system Tells Tier1 staff an issue is occurring Email reporting and graphing of health Monitoring DB for heartbeats in game process and launcher Most common issue is XLive not logging in. Administering the servers Slide 21 Control center Aimed at Tier 1 support Silverlight app that interfaces with Master services Lock down to datacenter and not accessible to the team Silverlight app that shows high level metrics Available through login Webservice only has three read only service calls Can fetch log files of game Administering the servers Slide 22 Master DB Master Service Launcher Servers Game Process Administering the servers Major components of infrastructure Slide 23 Master DB All components handshakes with the DB to accomplish work Size fixed after all machines and accounts are added Parameterized stored procedures only Separate DB for metrics No performance issues with proper indices in place Administering the servers Slide 24 Master service Writes to the master DB Configuration setting of the machines Datacenter setup with ID association Assigns accounts to each machine and each process (Account and 5x5 input) Installs and health monitor of launcher service on each machine Tracks and moves builds to the datacenter local cache Removed from DB and move to file caching Can inject into the ini for custom settings Can fetch log files from any game process or launcher service Administering the servers Slide 25 Gears of War 3 process Runs many per server All communication is asynchronous with database DB Status messages Game status (datacenter/game mode/playlist version/map name) Server status (launching/map cycling/restarting/shutting down/etc) DB Configuration options Query every time server restarts or idle threshold is reached Query returned various key/value pairs Very flexible Many performance counters exposed Frame rate, thread timings, number of players connected, client connection data (Ping, Incoming/Outgoing traffic, Packet loss) Administering the servers Slide 26 Launcher service DB Commands to interact with game Start (Install if needed from cache), Stop (Bleeds off clients), Kill, Kill All Restart server, and clean machine Gather and records state of the game processes Game status (datacenter/game mode/playlist version/map name) Server status (launching/map cycling/restarting/shutting down/etc) Administering the servers Runs one per server Owns game processes on server Health monitor of the process Reasons to restart, Every 48 hours In case game crashes Datacenter ID or playlist version does not match Server status hangs in any state for too long Hot swappable Allowed us to change health rules dynamically without stopping server hosted games Slide 27 Health monitoring good day Administering the servers Slide 28 Health monitoring bad day Administering the servers Slide 29 Lessons learned Restarting the process automatically is mandatory Many small things outside your control, allows you to come back online quickly Live connectivity Server hiccups Configuration issues G4WL cannot handle loading all processes at once We found the need of 10-15 seconds between the load of each game process to prevent XLIVE DLL issues All administrative applications need the ability to be updated without taking down the server hosted games From the game to the monitoring services, you never know when you need to make adjustment and this allows you to do a simple form of A/B testing Administering the servers Slide 30 Developer environment Client/Server Environment Could run multiple clients and servers on same machine Multiple Gamertags / local accounts required (runas.exe) Maintained GFWL PC client for rapid iteration Could run without admin tool from commandline UnrealConsole could talk to server through socket All the debugging functionality of UE3 Admin Environment One datacenter simulation for testing 5 servers with 1 SQL/webservice server Could run locally using Visual 2008 (for the game), Visual 2010 (admin tools), SQL, and Internet Information Services (ISS) Implementation rollout Slide 31 Retrofit game to support planned Gears 3 features Good way to introduce features with no expectations First test of new matchmaking flow First test of dedicated servers Limited run of dedicated servers Controlled environment, closely monitored Profiling servers in a real environment Tested CPU/Bandwidth usage in the wild on various hardware Found 2 otherwise irreproducible crashes in the wild Able to get minidumps and figure out the problems Phase 1 - Gears 2 title update (April 2010) Implementation rollout Slide 32 More than 100 people (mostly testers) Lock machines available and cores to create simulated overload Monitor CPU and bandwidth Will stress servers, but not infrastructure. Work with enterprise staff to look for flaws (outside of games devs) DB analysis Network sniffers Locking down cores on a PC Number of network cards, etc Phase 1.5 Large test in the labs (January 2011) Implementation rollout Slide 33 Real rollout of servers to datacenters First consumer trials of our server administration tools Phased rollout Huge success Solid uptime Gamers happy Phase 2 - Gears 3 Beta (April 2011) Implementation rollout Slide 34 Lack of communication issues (zombie games) Misconfigured servers Small number of game and balance issues Added more matchmaking tweaks to ease contention Good sampling of ping data from around the world Discover data points which we should capture during release A HTTPS Webservice is better than direct DB access Better caching of static data in the DB to offset the DB load Lessons LEARNED (BETA) Implementation rollout Slide 35 MS Cert needs to be able to run the server in their environment needs to be able to see the client attached to a server liked to see that the server is attached to the client Challenges of MS Cert Environment Closed environment Not accessible to our admin framework or network Reverse IP lookup cannot find their server Solutions Always keep the ability to run the server by itself without any DB connections Set cert environment to only one use datacenter therefore all IPs return one datacenter ID Submission process Implementation rollout Slide 36 Always kill the process on security concerns, better to be alerted than be exposed The game is signed, but we have exposed connections that must be protected!! Use SDL to examine how trustworthy those communications are and what happens if someone crashes your game process File and networking fuzzing can be difficult, but worthwhile Look for exposure of personal information especially in log files Get an enterprise developer to look at your SQL stored procedures Know your pattern for your game to help look for regularities Think of that pattern as credit cards look for fraud Security reviews of datacenters (Before you go out) Implementation rollout Slide 37 Ability to rev the dedicated servers faster / independent of client updates Servers have to have all the content Matchmaking can impose certain requirements on clients before searching Balance between value to those who purchase content vs. fragmenting our client base Plan for an update path for your servers DLC / Title updates Out in the wild Slide 38 If mandatory upgrade, servers will not work until upgraded Update requires a server to shut down all games Can be done in a rolling manner No matter how much communication, be prepared to be surprised when these happen Automated solution to deploy to reduce impact Releases of new G4WL client dlls Out in the wild Slide 39 Trends suggest that we have a lot of unused time on our servers everyday To the cloud With a cloud solution, you could possibly get the following: Pay for what you use (but more likely at higher hour rate) More volume upfront for day one demands Tier1 built in to the purchase (hardware issues, network issues) Could freeze VMs on machines to debug later The Future Slide 40 Hopefully your launch looks like ours. Slide 41 Questions? Email: mweilb@microsoft.com Special thanks to: Epic Games MS Core Pub Team MS Games IT Individual call outs: Josh Markiewicz Sam Zamani Wes Hunt Ian Thomas Joe Graf Vijay Krishnan Nur Sheikhassan Chris Kimmell Chris Wynn Microsoft Studios - Core Publishing is recruiting