cassandra at ebay - cassandra summit 2012

30
Cassandra at eBay August 8, 2012 Jay Patel Architect, Platform Systems @pateljay3001 Time left: 29m 59s

Upload: jay-patel

Post on 04-Dec-2014

26.205 views

Category:

Technology


2 download

DESCRIPTION

"Buy It Now! Cassandra at eBay" talk at Cassandra Summit 2012 http://www.datastax.com/events/cassandrasummit2012

TRANSCRIPT

Page 1: Cassandra at eBay - Cassandra Summit 2012

Cassandra at eBay

August 8, 2012

Jay Patel Architect, Platform Systems @pateljay3001

Time left: 29m 59s

Page 2: Cassandra at eBay - Cassandra Summit 2012

eBay Marketplaces

97 million active buyers and sellers

200+ million items

2 billion page views each day

80 billion database calls each day

5+ petabytes of site storage capacity

80+ petabytes of analytics storage capacity

2

Page 3: Cassandra at eBay - Cassandra Summit 2012

How do we scale databases? Shard

– Patterns: Modulus, lookup-based, range, etc.

– Application sees only logical shard/database

Replicate

– Disaster recovery, read availability/scalability

Big NOs

– No transactions

– No joins

– No referential integrity constraints

3

Page 4: Cassandra at eBay - Cassandra Summit 2012

We like Cassandra

Multi-datacenter (active-active)

Availability - No SPOF

Scalability

4

We also utilize MongoDB & HBase

Write performance

Distributed counters

Hadoop support

Page 5: Cassandra at eBay - Cassandra Summit 2012

Are we replacing RDBMS with NoSQL?

Some use cases don’t fit well - sparse data, big data, schema optional, real-time analytics, …

Many use cases don’t need top-tier set-ups - logging, tracking, …

5

Not at all! But, complementing.

Page 6: Cassandra at eBay - Cassandra Summit 2012

A glimpse on our Cassandra deployment

Dozens of nodes across multiple clusters

200 TB+ storage provisioned

400M+ writes & 100M+ reads per day, and growing

QA, LnP, and multiple Production clusters

6

Page 7: Cassandra at eBay - Cassandra Summit 2012

Use Cases on Cassandra

Social Signals on eBay product & item pages

Hunch taste graph for eBay users & items

Time series use cases (many):

Mobile notification logging and tracking

Tracking for fraud detection

SOA request/response payload logging

RedLaser server logs and analytics

7

Page 8: Cassandra at eBay - Cassandra Summit 2012

8

Served by Cassandra

Page 9: Cassandra at eBay - Cassandra Summit 2012

Manage signals via “Your Favorites”

9

Whole page is served by Cassandra

Page 10: Cassandra at eBay - Cassandra Summit 2012

Why Cassandra for Social Signals?

Need scalable counters

Need real (or near) time analytics on collected social data

Need good write performance

Reads are not latency sensitive

10

Page 11: Cassandra at eBay - Cassandra Summit 2012

Deployment

11

Topology - NTS RF - 2:2 Read CL - ONE Write CL – ONE

Data is backed up periodically to protect against human or software error

User request has no datacenter affinity

Non-sticky load balancing

Page 12: Cassandra at eBay - Cassandra Summit 2012

Data Model

depends on query patterns

12

Page 13: Cassandra at eBay - Cassandra Summit 2012

Data Model (simplified)

13

Page 14: Cassandra at eBay - Cassandra Summit 2012

Wait…

14

Duplicates!

Oh, toggle button! Signal --> De-signal --> Signal…

Page 15: Cassandra at eBay - Cassandra Summit 2012

Yes, eventual consistency!

One scenario that produces duplicate signals in UserLike CF:

1. Signal

2. De-signal (1st operation is not propagated to all replica)

3. Signal, again (1st operation is not propagated yet!)

15

So, what’s the solution? Later…

Page 16: Cassandra at eBay - Cassandra Summit 2012

Social Signals, next phase: Real-time Analytics

16

Most signaled or popular items per affinity groups (category, etc.)

Aggregated item count per affinity group

Example affinity group

Page 17: Cassandra at eBay - Cassandra Summit 2012

Initial Data Model for real-time analytics

Update counters for both individual item and all the affinity groups that item belongs to

Items in an affinitygroup is physically stored sorted by their signal count

Page 18: Cassandra at eBay - Cassandra Summit 2012

Deployment, next phase

Topology - NTS RF - 2:2:2

Page 19: Cassandra at eBay - Cassandra Summit 2012

19

item1

user2

user1

item2

buy

bid

sell watch

Page 20: Cassandra at eBay - Cassandra Summit 2012

Graph in Cassandra

Event consumers listen for site events (sell/bid/buy/watch) & populate graph in Cassandra

20

30 million+ writes daily 14 billion+ edges already

Batch-oriented reads (for taste vector updates)

Page 21: Cassandra at eBay - Cassandra Summit 2012

Mobile notification logging and tracking

Tracking for fraud detection

SOA request/response payload logging

RedLaser server logs and analytics

21

Page 22: Cassandra at eBay - Cassandra Summit 2012

A glimpse on Data Model

Page 23: Cassandra at eBay - Cassandra Summit 2012

RedLaser tracking & monitoring console

23

Page 24: Cassandra at eBay - Cassandra Summit 2012

Remember the duplicate problem in Use Case #1?

24

Let’s see some options we considered to solve this…

That’s all about the use cases..

Page 25: Cassandra at eBay - Cassandra Summit 2012

Option 1 – Make ‘Like’ idempotent for UserLike

Remove time (timeuuid) from the composite column name:

Multiple signal operations are now Idempotent

No need to read before de-signaling (deleting)

25

X Need timeuuid for ordering! Already have a user with more than 1300 signals

Page 26: Cassandra at eBay - Cassandra Summit 2012

Option 2 – Use strong consistency

Local Quorum

– Won’t help us. User requests are not geo-load balanced (no DC affinity).

Quorum

– Won’t survive during partition between DCs (or, one of the DC is down). Also, adds additional latency.

26

X Need to survive!

Page 27: Cassandra at eBay - Cassandra Summit 2012

Option 3 – Adapt to eventual consistency

27

If desire survival!

http://www.strangecosmos.com/content/item/101254.html

Page 28: Cassandra at eBay - Cassandra Summit 2012

Adjustments to eventual consistency

De-signal steps:

– Don’t check whether item is already signaled by a user, or not

– Read all (duplicate) signals from UserLike_unordered (new CF to avoid reading whole row from UserLike)

– Delete those signals from UserLike_unordered and UserLike

28

Still, can get duplicate signals or false positives as there is a ‘read before delete’.

Not a full story! To shield further, do ‘repair on read’.

Page 29: Cassandra at eBay - Cassandra Summit 2012

Lessons & Best Practices

• Choose proper Replication Factor and Consistency Level.

– They alter latency, availability, durability, consistency and cost.

– Cassandra supports tunable consistency, but remember strong consistency is not free.

• Consider all overheads in capacity planning.

– Replicas, compaction, secondary indexes, etc.

• De-normalize and duplicate for read performance.

– But don’t de-normalize if you don’t need to.

• Many ways to model data in Cassandra.

– The best way depends on your use case and query patterns.

More on http://ebaytechblog.com?p=1308

Page 30: Cassandra at eBay - Cassandra Summit 2012

30

@pateljay3001

#cassandra12

Thank You