lambda architecture with cassandra (vaibhav puranik, gumgum) | c* summit 2016
TRANSCRIPT
Vaibhav Puranik
Lambda Architecture with Cassandra
About me
Invented In Image advertising in 2008http://gumgum.com/gallery
Computer VisionProcessing 2.6B image impressions / month10B events / day2000 premium publishers165 employees in 3 continents
Lambda Architecture
Lambda Architecture
Architecture Before Cassandra
500 Ad Servers
Caps
Geo Filter
Keyword Filter
Cap Filter
Other Filters
Sort by revenue and pick one
Real Time Pipeline - The Problem500 Ad Servers
The Problem
6 minsroundtrip time
Enter Cassandra - Phase 1500 Ad Servers
Cassandra: US East - 21 Nodes, US West: 18 Nodes, EU West: 12 Nodes - r3.2xlarge
Caps
Geo Filter
Other Filters
Cassandra Cap Filter
Old Cap Filter
Sort by revenue and pick one
The Solution
6 mins 10 ms
Enter Cassandra - Phase 2500 Ad Servers
Cassandra: US East - 21 Nodes, US West: 18 Nodes, EU West: 12 Nodes - r3.2xlarge
Load data every 8 mins async way
Caps
Geo Filter
Other Filters
Cassandra RealtimeCap Filter
Cassandra Batch Data Filter
Sort by revenue and pick one
Scale
21 + 18 + 12 + 9 = 60 r3.2xlarge instances (Metadata)
9 + 6 + 3 = 18 r3.2xlarge instances (Tally)
400,000 records loaded every 8 mins by each Ad Server
20,000 read requests per second
10,000 write requests per second
Batch Layer
Realtime View
Batch Views
Lambda Architecture
Speed Layer
gumgum.com/careers