couchconf israel 2013_couchbase at playtika
TRANSCRIPT
Myself
• Many years @ 888 an online Casino & Poker company. Finished as Gaming Architect
• CTO @ UMOO, a financial market gaming platform
• CTO @ Playtika
Playtika
• Founded mid 2010• Social gaming company• Multiple social networks and sites• Multiple platforms (SNs, destination, Android,
iOS)• Acquired by by Caesar’s Interactive
Entertainment mid 2011
Our Games
• A few million daily active users.• A few (more) million monthly active users• Our biggest game today Slotomania – 200K+
concurrent users at peak hours. No less than 100K during off-peak
• At least one request per user every 5 seconds• At least one read-modify write on user state per
request• In many cases multiple read-modify-writes
2 years ago, Slotomania on Facebook
• 70K Concurrent users • A few million registered
users• One MySQL node
• Hit a glass ceiling Needed to scale “yesterday”
So what now?
• Had Sharding code branch that had to be dropped
• Within 3 weeks!!! MySQL only => MySQL + Couchbase
• Meantime – 3 ring MySQL circus
Why Couchbase
• Open source • Backed by a serious company• Indications all around that the tech is being
successfully used with some big-time players• Almost no comparison work. Online only!
Keep MySQL – Why?
• Fear is the mind killer• Site to site replication• Can still “SQL Query” the data• Easy as pie…• BUT, not everyone can keep everything on one
node.
What do we keep in couchbase?
• Everything key value• Anything we need to access fast and frequent– User Score– User Save– User Messages
• Only things where true ACIDness is not 100% mandatory or can be faked.
Keys and Values
• Be creative with keys and objects• Most common: user_id• E.g. user_id & session_id• E.g. user_id & game_type
“Schema” – SQL vs. Document based
• Sometimes you need to turn the data on its head
• 2 Table join in SQL = multi get in Couchbase• Excessive ‘multigetting’ = network saturation• Our Lesson: Don’t migrate as is.
Refactor/reimagine the schema for a document-oriented database
Serialization
• Right now, some JSON some Binary• Intend to move most if not all to JSON – 2.0!– Browse keys without having to have entire context– Easier to support object versioning
• Consider other serialization methods (thrift, protobuf)
Data Services or Direct Access?
• Encapsulate data access?• Direct Access => High Performance• Data layer Encapsulation => mitigate complex
SOA-ish environments
Data Services vs. Direct Access
Service A
DAL Lib
Service B
Couchbase
DAL Lib
Service A
RPC Lib
Service B
Couchbase
RPC Lib
Data Services
Locks
• Pessimistic vs. Optimistic• We use 2 different lock types in different cases:• CAS + spin X times (and fail)• Lock object with timeout (actually can be done
better than we do it!)• Avoid contention if possible (especially cross-process
) by old-school tricks– Queue stuff– IP based stickiness (you lose nothing by this)– Don’t be a nerd! Cut corners where possible.
Our cluster(s)
• 12 Couchbase buckets + 4 memcache buckets On two clusters
• The 2 “heavy” buckets are on their own cluster
• 40K ops/s on couchbase, 20K ops/s on memcache
• 30-35% Couchbase ops are writes• We still live in a pre-2.0 world (1.8.1)
Our hardware
• Couchbase boxes are physical• Latest “spec” – Dell R710 with • 2 CPUs (12 cores) – overkill!!• 6 SAS HDDs• 64 Gigs RAM• Running CentOS 5.8