devopscon 2015 - devops in mobile games

DevOps in Mobile GamesOr „How to serve millions of players“

Andreas Katzig@derFunk

Technical Director of Chimera Entertainment@ChimeraEnt

Andreas Katzig,Technical Director

About me

• Online user since 1994• BBS, FidoNet, Compuserve

• Online programmer since 1996• PHP 2, MySQL 3

• Founder of Photono Design andCo-Founder of Photono Software in 1999


About me

• Graduate Degree in Computer Science

• Co-Founder of Vulcando Games in 2006• Funded by Exist/BMWi in 2008

• Released 2 MMOG Titles

• Own OpenGL-based Game Engine with dozens of custom tools


About me

• Joined Chimera Entertainment in 2009

• Now overseeing the Tech Department

• Chimera is the flagship studio of theremote control productions family

• More than 14 other studios are part of it


Chimera. Mobile Games.

• Chimera is a Unity 3D studio

• First Unity Engine licensee in Germany

• Since 2010 more than 15 game releaseson more than 7 different platforms

• The most prominent ones being

• Angry Birds Epic (Mobile)

• Our own idea, pitched to Rovio in 2012

• John Woo‘s BloodStroke (Mobile)

• Bernd das Brot (PC)

• Dr. Kawashima (PC)

• Ravensburger Memory™ (PC, Mobile)


Chimera. Mobile Games.

> 65.000.000 Downloads worldwide


Chimera. Partners.


Chimera. Key Technology Partners.


Chimera. Platforms.


Chimera. Development Processes.

• Agile / Scrum since 2008• Define (your own) goals in a two-weeks sprint cycle.

• Daily standups with the Team. Post-Its, Burndown Charts, all that stuff.

• Dedicated DevOps Engineer working 90% from remote• Slack helps to stay in touch

• DevOps also happens from the Chimera HQ in Munich

• Regular Code Reviews• Utilizing Atlassian Crucible

• Peer Reviews

• Regular Dev Team Meetings, Focus Days, Social Events etc.


Chimera. Development Practices and Paradigms.

• „Move fast“• But try to not break things.

• Yes, originally from Facebook. They also removed „break things“ from their motto this year (because things broke too often).

• „Does it scale“?• Default question for every DevOps Engineer.


Chimera. Development Practices and Paradigms.

• „Convention over Configuration“• This is true for code, but not necessarily for DevOps, as we have a lot of configuration scripts here.

• “Configuration as Code“

• Prevents „Configuration Drift“

• „Code once, run everywhere“• That‘s where Docker comes in place

• „Version all the Things“

• „Automize everything“

• „Don‘t repeat yourself“

• „Separation of Concerns“

• „Single Responsibilities“

• „Loose coupling of modular components“

• Wow, can you still read that?


Chimera. Mobile Games. Tech. Infrastructure.

• Continuous Integration with Jenkins• One Jenkins head server

• > 5 headless slaves (OSX, Win, Linux)

• Building for mobile platforms, consoles, standalone

• Nightly project builds and integration tests of our own Cross Platform Library

• Parameterized manual builds

• Extended uses of Jenkins• Auto-install new Unity versions on all buildservers as they are released

• Jobs to manually upload assets to servers and services

• „Project setup script“ that generates SCM repositories and Jenkins build jobs for all mobile platforms

• According to Chimera conventions – Every project setup is similar

• Resigning-Jobs for IPAs and APKs


Chimera. Mobile Games. Tech. Infrastructure.

• Continuous Delivery• Apps: Mobile Over The Air distribution

• Simple custom solution

• Artifacts: Deployment via Amazon S3

• Repositories involved• SVN/GIT

• Docker Hub

• NuGet

Our Build Engineer Grisha will be here this evening! Take the chance to talk to him!


Chimera. Mobile Games. Tech Stack. Backend.

• We‘re sharing C# code between Unity and the backend• Of course we do! Data Models (DTOs), Game Logic, Serialization

• To run C# code at the server side, we‘re using Mono• As FastCGI processes behind NginX

• Communication layer is RESTful web services• Stateless Microservices

• Server OS is generally Debian

• C# SDKs for all thinkable server components are available• AWS, MySQL, Redis, Memcached etc

Hyper!


How we rolled in 2010.


How we roll today.

• AWS• Elastic Container Services

• Docker!

• Horizontal Scaling

• High Availability

• And more


How we roll today. Components.

• AWS• Classes of problems abstracted away:

Power and network topology, hardware costs andvendor differences, network storage systems, etc...

• Flexibility to start new servers or whole new environments in minutes.

• Auto scaling and zero setup costs. Wow.

Our DevOps Engineer Giannicola is here today! Take the chance to talk to him!



• ECS• Management of Docker containers on a cluster

• Basic scheduling of containers based on resourceneeds, isolation policies, and availability requirements

• Integrates with auto scaling and ELB

• Easy Blue/Green deployments solution

• We run our stateless game backends on it



• Terraform• A tool for building, changing, and versioning

infrastructure safely and efficiently

• Infrastructure as Code

• Change Automation

• We used it to create our whole AWS infrastructure

• We use it for basic configuration management too• but it cannot completely substitute Ansible or equivalent tool



• Docker• Developers have full control over the execution

environment of an application

• Reduced risks when deploying

• Infrastructure as Code

• Lightweight virtualization

• Standard packaging unit for builds, tests and deployments



• CoreOS• Operating system for the cloud age

• Everything runs in containers

• Does not ship with a package manager, basic shippable units are Docker containers

• The goal is to have the build system output a container as the final artifact: then systemd and fleet deal with containers lifecycle



• CoreOS• Fleet works by receiving systemd unit files and scheduling

them onto machines in the cluster based on declared conflictsand other properties

• Easy and robust (replicated) Service Discovery with etcd:services can scale seamlessly

• We use it to deploy our monitoring infrastructure, our applicationlogging facility and our chat cluster



• Consul• Service Registration and Discovery through

HTTP and DNS interfaces.

• Distributed and highly available system:gossip pool, distributed health checking

• We use it for our chat and ELK clusters

• We‘re also using „Registrator“ from Gliderlabs in connection with Consul• „Registrator automatically registers and deregisters services for any Docker container

by inspecting containers as they come online.”



• Consul Web Interface



• ELK• Main open source centralized logging solution

• Elasticsearch containers clusters together thanks to etcd

• Logstash and Kibana containers discover Elasticsearchquery nodes thanks to Consul

• Logstash is powerful enough to be able to parse logs fromvarious sources (nginx logs, log4net application logs, etc...)



• ELK



• Collectd• It gathers statistics about the system it is running on

• Simple and lightweight tool written in C, with more than 90 plugins

• Collected metrics can be sent to Logstash, Influxdb, Librato:it does not generate graphs, information must be sent to other tools

• We run it as a Docker container on every server we want to monitor



• Influxdb• Time series, metrics and analytics database

• Data can be queried with a SQL-like language

• HTTP(S) API for data ingestion and queries

• It has built-in support for other data protocols such as collectd



• Influxdb• Written in Go with no external dependencies

• Clustering, replication and high-availability is in an alpha state

• We store Collectd metrics from various hosts on it,running as a Docker container



• Grafana• Graph and dashboard builder for visualizing

time series metrics

• Built in support for many of the most populartime series data sources: Graphite, Elasticsearch,Influxdb, CloudWatch

• We run it as a Docker container: it's the perfect pair for Influxdb, we visualize Collectd metrics gathered on all servers



• Grafana



• Librato• SaaS real-time cloud monitoring

• Integrates with more than 100 open-source agents,turn-key solutions and 3rd-party services:CloudWatch, Heroku, Collectd, Statsd, etc...

• Fastest and easiest solution to collect, correlate and visualize data at any layer of the stack

• We send CloudWatch and Collectd metrics to it, in order to have another level of system dashboards



• Librato


Questions?


We‘re hiring!

http://jobs.chimera-entertainment.de@ChimeraEnt@derFunk