Maximizing Revenue Through Logging Albert Ho
About Rumble Funded by: Mission: ● Bring AAA quality to F2P ● Play together cross-device/platform
Rumble’s Platform ● Played in >200 countries ● Six data centers worldwide ● Hybrid infrastructure ● Play together on:
What this talk is about…
User Acquisition
Funnel Loss Retention Loss
Your Game Paid Users
Free Users
“Logging” Distinctions ● User behavior
● Game design, retention, virality, monetization
● System behavior ● Maximize potential ● Lower CPA
Thesis “You really aren’t maximizing revenue
unless you can measure system behavior”
Challenges ● 24/7 operations ● Game teams ship code daily ● Being “up” is not good enough
The Long Tail of Revenue Lost: Daily Site Problems Affect Weeks of Revenue
Outage
Problem
Recovery
New Installs/Sales
Revenue
Days
How to Detect Problems
● Solving is Harder than it sounds Signal to noise ratio + Indexing and search + Multiple game teams shipping code daily = BIG Headache and world’s biggest chore
The answers are in the logs!
Rumble Logging 1.0 ● Graylog, Elasticsearch ftw?
● 1 day to setup ● Collecting for 2 game servers ● Few hundred MB a day / 7-day retention time ● Scaling is easy… right?
- Bigger hard disks, more ES nodes
Rumble Logging 1.5 ● 7 days to setup ES cluster ● > 20 servers ● Logs @ TB a day / 3-day retention time ● Log noise incomprehensible ● Very expensive!!! ● Logging cluster broke every day
Key Learnings Log Management is really… “Undifferentiated Heavy Lifting”
We were spending more on LOGGING than HOSTING
There must be an expert that does it better, faster, and cheaper
Rumble Logging 2.0 Requirements ● Must support JSON ● Easy to integrate ● Can scale up and meet retention time goals ● Cost-effective
Enter…….
● No more agents! ● REST API – 21st century please/no syslogs ● Scalable and Responsive ● Superb partner relationship
Sample Log
Loggly Use Cases ● Log clean-up ● Release monitoring and debugging ● Customer service requests ● Trend analysis ● 3rd-party developers
Log Clean-up ● 1000’s errors per minute noise ● DEBUG, INFO, WARN, ERROR? ● Reduce TB’s of logs to GB’s
Release Monitoring ● Log snapshots before and after
Release Monitoring ● Game teams own
process ● All logs in one place ● Easy for PM’s
Customer Service Reactive problem solving ● “Why is customer x slow?” ● “Why did I not get my gems?” !
Proactive debugging ● “These 20 customers were affected by
today’s issue.”
Trend Analysis Game performance analysis over time
Client Lag
Days
3rd Party Development ● Bake logging into games at the beginning
● Including 3rd party ones…
Loggly Live Demo
Loggly Before And After Before After
Log Retention 3 days 15 days
Log Availability 50% uptime >99.9%
Debugging Speed Days Minutes
Happiness Factor
Questions? [email protected] KingsRoad http://kingsroad.com http://apps.facebook.com/kingsroadgame
Ballistic https://apps.facebook.com/ballistic
