metrics for mmp development and operations lessons learned: the sims online
DESCRIPTION
Metrics for MMP Development and Operations Lessons Learned: The Sims Online. Larry Mellon GDC, Spring 2004. Metrics: Catch-22. Useful. Careful…. GI / GO. Hard Data. Optimization Tool. Expensive. Importance of Metrics is Relative. Mark Twain. Lord Kelvin. Measure Everything. Measure - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/1.jpg)
Metrics forMMP Development and Operations
Lessons Learned: The Sims Online
Larry Mellon
GDC, Spring 2004
![Page 2: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/2.jpg)
Metrics: Catch-22
Useful
Hard Data
Optimization Tool
Careful…
GI / GO
Expensive
![Page 3: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/3.jpg)
Importance of Metrics is Relative
Lord Kelvin Mark Twain
MeasureEverything
Measure “Just Enough”
![Page 4: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/4.jpg)
Invest No More Than You Need
Kelvin Twain
Complexity
Scale
Precision
$$$ $MMP
![Page 5: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/5.jpg)
MMP: Measurement Focal Points
Infrastructure
Operational Costs
Player Actions
Economy
![Page 6: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/6.jpg)
Similar Use Case: Casinos(Harrah’s “Total Reward”)
Unified Player Action DB
CasinoCasino
Table / MachineTable / MachineTable / Machine
Player Player Player
Track every Player Action
Casino
![Page 7: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/7.jpg)
Highly Profitable, Highly Popular
Unified Player Action DBProfit (per Casino, per Player)
Patterns of Play
Analyze:
Casino Operations
Player Awards Program
Modify:
“This is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future.“ John Boushy (Harrah's CIO, 2000)
![Page 8: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/8.jpg)
TSO: Live Monitors, Summary Views
Player Actions & Persistent Data
Embedded Profiler (Server Side)Automated Report Generators
Designers CommunityManagers
Metrics System
Server Reactions
Engineers Operations
![Page 9: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/9.jpg)
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
![Page 10: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/10.jpg)
Implementation:Driving Requirements
Common Infrastructure
Ease of use
Low overhead
![Page 11: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/11.jpg)
Esper Architecture
esperProbeesperProbeesperProbe
Live Server CPUs
ProcessProcessProcess
UserUserUser
esperLogesperLogesperLog
![Page 12: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/12.jpg)
Event-level Sampling, Aggregated Reporting
esperStore
DBImporter
esperLogesperLogesperLog
esperViewesperViewesperView
esperFetch
t1 t2 t3 t4 t5
esperProbes
Min, Max, Av, Count
![Page 13: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/13.jpg)
Esper Probes• Self-organizing: “class” hierarchy
• Data driven: new probes and/or new game content immediately visible on web
• Example: ESPER_PROBE – (“Object.interaction.%s”, chair->picked)– (“Object.interaction.puppet.%s”, self->picked)
• Human-readable intermediate files
![Page 14: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/14.jpg)
EsperView: Web-Driven Presentation
Daily Reports
Graph Caching & Archiving
Filtering & Meta Data
Report Generator
![Page 15: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/15.jpg)
EsperView: Hierarchical Presentation
Process-Level Collection
Server Cluster
Process Class
Process Instance
![Page 16: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/16.jpg)
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
![Page 17: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/17.jpg)
Applications of Metrics
Load Testing(Realistic Inputs)
Beta Testing & Live Operations (Game Tuning, Community Management)
Load Testing & Live Operations(Server Performance)
![Page 18: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/18.jpg)
Test Servers
Load Testing:“Monkey See / Monkey Do”
AlphaVille Servers
Live Beta Testers
Sim Actions (Player Controlled) Sim Actions (Script Controlled)
![Page 19: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/19.jpg)
Applications of Metrics
Load Testing:Realistic Inputs
Beta Testing & Live Operations: Tuning/Mngmnt
Load Testing & Live Operations:Server Performance
![Page 20: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/20.jpg)
Make Friends, Shake Hands
beats outGive Money / Get Money
Least Used: Disco Dancing
History
Meta Data
![Page 21: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/21.jpg)
Top: TD Dance, WoohooBottom: Dance
![Page 22: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/22.jpg)
Players per Lot 0 to 70 players: < 2 / lot 70 to 400 players: > 3 /
lot
![Page 23: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/23.jpg)
Top: Metrics Bug (sorta)Next: Garden Gnome, Toilet
Bottom: Buffet Table
![Page 24: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/24.jpg)
Beta: numPlayers by numRMates
1. Most have one Roommate
2. Hard-core players: 8
![Page 25: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/25.jpg)
Highest
1. Using Skill Objects2. Using Pizza Maker3. Selling Objects
Lowest
1. Visitor Bonus2. New Avatar3. Object “Rentals”
![Page 26: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/26.jpg)
Economy: Detailed View
1. Using Objects = $$$2. Visitor Bonus == $
![Page 27: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/27.jpg)
Visitor Bonus: Who Makes Money?
1. Most getting no V. Bonus2. Hard-core players: $$
![Page 28: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/28.jpg)
4 of top 5: windows?
?
![Page 29: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/29.jpg)
House Categories (Beta Test)
1. Not a well-used feature2. Shopping least of all
![Page 30: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/30.jpg)
Community Management
Community Actions & Trends
Influencing Player Activity
Free Content
Tracking Problem Players
![Page 31: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/31.jpg)
Marketing
Press Release Teasers
In-Game Brand Exposure
Special Events
![Page 32: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/32.jpg)
NYEve: Kiss Count
Esper Cities All Cities (extrapolated)===========================================New Year's Kiss 32,560 271,333Be Kissed Hotly 7,674 63,950Be Kissed 5,658 47,150Be Kissed Sweetly 2,967 24,725Blow a Kiss 1,639 13,658Be Kissed Hello 1,161 9,675Have Hand Kissed 415 3,458===========================================
Total 52,074 433,949
![Page 33: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/33.jpg)
Applications of Metrics
Load Testing:Realistic Inputs
Beta Testing & Live Operations: Tuning/Mngmnt
Load Testing & Live Operations:Server Performance
![Page 34: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/34.jpg)
![Page 35: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/35.jpg)
DB byte count oscillates out of
control
![Page 36: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/36.jpg)
A single DB Request is clearly
at fault
![Page 37: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/37.jpg)
![Page 38: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/38.jpg)
Most-Used DB Queries (unfiltered)
11,000,000 level Queries need attention, and
drown out others
![Page 39: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/39.jpg)
DB Queries (Filtered)
Filters on 11,000,000 level Queries show patterns of 7,000
level Queries
![Page 40: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/40.jpg)
Incoming & Outgoing Packets
![Page 41: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/41.jpg)
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
WrapupLessons Learned
Conclusions
Questions
![Page 42: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/42.jpg)
Lessons Learned
• Implement early
• Ownership, senior engineers
• Aggregated probes vs event-level tracking
• Automation: collect / summarize / alarms
• “There can be only one”…
![Page 43: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/43.jpg)
Conclusion: Very Useful!
Data MiningOn Players:
Untapped Gold…
Game Design
Realistic Load Testing
Engine Fixes, Optimization
Server Cost, Launch Timing
Critical Feature: Accessibility
![Page 44: Metrics for MMP Development and Operations Lessons Learned: The Sims Online](https://reader031.vdocuments.site/reader031/viewer/2022013101/568134ec550346895d9c29e7/html5/thumbnails/44.jpg)
Questions
Slides available @
www.maggotranch.com/MMP