serving large-scale batch computed data with project voldemort · linkedin! linkedin! serving...
TRANSCRIPT
![Page 1: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/1.jpg)
USENIX Conference on File and Storage Technologies February, 2012
Serving Large-scale Batch Computed Data with Project Voldemort
Roshan Sumbaly, Jay Kreps, Lei Gao, Alex Feinberg, ���Chinmay Soman, and Sam Shah
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
![Page 2: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/2.jpg)
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
2 4 8
17
32
55
90
2004 2005 2006 2007 2008 2009 2010
LinkedIn Members (Millions)
150M+
75% Fortune 100 Companies ���use LinkedIn to hire
Company Pages
>2M
Searches in 2011
~4B
![Page 3: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/3.jpg)
Data Features on LinkedIn
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
People You May Know
LinkedIn Skills
Related Searches Viewers of this profile also viewed
Events you may be interested in Jobs you may be interested in
![Page 4: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/4.jpg)
Data Features on LinkedIn
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
People You May Know • Batch computed algorithms • MapReduce (Hadoop)
• Output • Large • Immutable • Key-value • Full refresh
![Page 5: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/5.jpg)
Problem
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
How do we serve these massive outputs to our 150 million members?
![Page 6: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/6.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 7: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/7.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Distributed key-value system
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 8: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/8.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Minimum performance impact during bulk loads • Offload index construction to processing system
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 9: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/9.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Error in algorithm à Bulk load bad data à Bad state till next push • Quick rollback capability
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 10: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/10.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
job.class=com.linkedin.jobs.BuildAndPushJob build.input.path=/algorithm/output push.store.name=people-you-may-know push.cluster=tcp://testing-cluster-url:6666 build.replication.factor=1
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 11: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/11.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
job.class=com.linkedin.jobs.BuildAndPushJob build.input.path=/algorithm/output push.store.name=people-you-may-know push.cluster=tcp://production-cluster-url:6666 build.replication.factor=2
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 12: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/12.jpg)
System features
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Apache License v2.0 • Project Voldemort – http://project-voldemort.com
• Fast, available and elastic • Bulk load massive data-sets • Minimum time in error • Easy to use • Open-source
![Page 13: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/13.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 14: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/14.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 15: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/15.jpg)
Project Voldemort
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Amazon’s Dynamo clone • Distributed key-value store
• Pluggable architecture • Initially written for read-write storage engines
• MySQL, BDB
![Page 16: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/16.jpg)
Project Voldemort
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Node 0 Node 1
Store 1 Store 1
Store 2 Store 2
• Multiple peers • Multiple stores ( ~ tables ) • Different replication factor ���
per store
![Page 17: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/17.jpg)
Project Voldemort
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Node 0 Node 1
• Hash ring per store • Ring split into partitions
Partition 0, 2, 4
Partition 0, 2, 4
Partition 1, 3, 5
Partition 1, 3, 5
Store 1 Store 1
Store 2 Store 2
Partition 0 (Node 0)
Partition 1 (Node 1)
Partition 2 (Node 0)
Partition 3 (Node 1)
Partition 4 (Node 0)
Partition 5 (Node 1)
0 232-1
![Page 18: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/18.jpg)
Existing approaches – Bulk load solution 1
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Node 0 Node 1
• Latency degradation during load
Store 1 Store 1
Processing system (Hadoop)
Staging
LOAD DATA
Staging
LOAD DATA
![Page 19: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/19.jpg)
Existing approaches – Bulk load solution 2
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Store 1
Processing system (Hadoop)
Staging
Live requests
Staging
LOAD DATA LOAD DATA
• Maintaining extra cluster
T1
View
T2
Store 1
T1
View
T2
![Page 20: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/20.jpg)
Existing approaches – Bulk load solution 2
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Store 1
Processing system (Hadoop)
Staging
Live requests
Staging
LOAD DATA LOAD DATA
T1
View
T2
Store 1
T1
View
T2
• Maintaining extra cluster
![Page 21: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/21.jpg)
Existing approaches – Bulk load solution 3
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Processing system (Hadoop)
HDFS
M M
R R
• Multiple copy operations
Live requests
T1
View
T2 T1
View
T2
![Page 22: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/22.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 23: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/23.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 24: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/24.jpg)
Bulk load extensions – Minimal impact on live system
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
HDFS Hadoop flow
![Page 25: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/25.jpg)
Bulk load extensions – Minimal impact on live system
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
HDFS Hadoop flow
Construct store
![Page 26: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/26.jpg)
Bulk load extensions – Minimal impact on live system
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• “Construct Store” step • Single MapReduce job • Map • Input - Output of algorithm • Output – Emit replication factor number of times
• Partitioner • Redirect to appropriate reducer
• Reducer • Output to Voldemort node based folders
![Page 27: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/27.jpg)
Bulk load extensions – Minimal impact on live system
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• What is output in the reducer phase?
• Store Partitions Chunk sets • One reducer = one chunk set • Chunk set = Index + data file
split into split into
![Page 28: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/28.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 29: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/29.jpg)
• Construction • MapReduce job
• Fetch • Pull chunk sets in parallel • Store into new version folder
• Swap • Close latest version’s index files • Change latest version link • Memory map new version’s index files
• Rollback • Close latest version’s index file • Change latest version link • Memory map old version’s index file
Bulk load extensions – Full pipeline
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Hadoop
HDFS
Driver / Scheduler
Voldemort cluster
Trigger ConstrucDon
![Page 30: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/30.jpg)
Bulk load extensions – Full pipeline
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Construction • MapReduce job
• Fetch • Pull chunk sets in parallel • Store into new version folder
• Swap • Close latest version’s index files • Change latest version link • Memory map new version’s index files
• Rollback • Close latest version’s index file • Change latest version link • Memory map old version’s index file
Hadoop
HDFS
Driver / Scheduler
Voldemort cluster
Trigger Fetch
Parallel pull
![Page 31: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/31.jpg)
Bulk load extensions - Rollback
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Voldemort node
store-‐1
/store-‐1
/version-‐(i)
/version-‐(i+1)
latest à version-‐(i+1)
Chunk sets
Chunk sets
![Page 32: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/32.jpg)
Bulk load extensions – Full pipeline
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Construction • MapReduce job
• Fetch • Pull chunk sets in parallel • Store into new version folder
• Swap • Close latest version’s index files • Change latest version link • Memory map new version’s index files
• Rollback • Close latest version’s index file • Change latest version link • Memory map old version’s index file
Hadoop
HDFS
Driver / Scheduler
Voldemort cluster
Trigger Fetch
Parallel pull
![Page 33: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/33.jpg)
• Construction • MapReduce job
• Fetch • Pull chunk sets in parallel • Store into new version folder
• Swap • Close latest version’s index files • Change latest version link • Memory map new version’s index files
• Rollback • Close latest version’s index file • Change latest version link • Memory map old version’s index file
Bulk load extensions – Full pipeline
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Hadoop
HDFS
Driver / Scheduler
Voldemort cluster
Trigger Swap
![Page 34: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/34.jpg)
Bulk load extensions - Rollback
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Voldemort node
store-‐1
/store-‐1
/version-‐(i+1)
latest à version-‐(i+2)
Chunk sets
/version-‐(i+2)
Chunk sets
![Page 35: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/35.jpg)
• Construction • MapReduce job
• Fetch • Pull chunk sets in parallel • Store into new version folder
• Swap • Close latest version’s index files • Change latest version link • Memory map new version’s index files
• Rollback • Close latest version’s index file • Change latest version link • Memory map old version’s index file
Bulk load extensions – Full pipeline
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Hadoop
HDFS
Driver / Scheduler
Voldemort cluster
Trigger Rollback
![Page 36: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/36.jpg)
Bulk load extensions - Rollback
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Voldemort node
store-‐1
/store-‐1
/version-‐(i+1)
latest à version-‐(i+1)
Chunk sets
/version-‐(i+2)
Chunk sets
![Page 37: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/37.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 38: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/38.jpg)
Bulk load extensions – Lookup
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Find partition and chunk set to read • Binary search in index file of chunk set • Jump to offset in data file • Go through all collided tuples
![Page 39: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/39.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 40: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/40.jpg)
Bulk load extensions – Rebalancing
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Adding new nodes with no downtime • Change ownership of partitions to new nodes • Simple move of corresponding chunk sets + swap
![Page 41: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/41.jpg)
Roadmap
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Background – Voldemort architecture • Custom Voldemort storage engine • Minimal impact on live system • Fast rollback • Fast lookups • Easy rebalancing
• Performance
![Page 42: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/42.jpg)
Performance
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Single node latency • Multi-node latency • Production
![Page 43: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/43.jpg)
Performance – Single node latency
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
throughput (qps)
late
ncy
(ms)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
median
● ●
●● ●
●
●
●●
●
●
● ●
●
●
100 200 300 400 500 600 700
0
50
100
150
200
250
99th percentile
●● ●
● ●●
●
● ●●
●
●●
●●
100 200 300 400 500 600 700
● MySQL ● Voldemort
throughput (qps)
late
ncy
(ms)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
median
● ●
●● ●
●
●
●●
●
●
● ●
●
●
100 200 300 400 500 600 700
0
50
100
150
200
250
99th percentile
●● ●
● ●●
●
● ●●
●
●●
●●
100 200 300 400 500 600 700
● MySQL ● Voldemort
100 GB data, 24 GB RAM
throughput (qps)
late
ncy
(ms)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
median
● ●
●● ●
●
●
●●
●
●
● ●
●
●
100 200 300 400 500 600 700
0
50
100
150
200
250
99th percentile
●● ●
● ●●
●
● ●●
●
●●
●●
100 200 300 400 500 600 700
● MySQL ● Voldemort
throughput (qps)
late
ncy
(ms)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
median
● ●
●● ●
●
●
●●
●
●
● ●
●
●
100 200 300 400 500 600 700
0
50
100
150
200
250
99th percentile
●● ●
● ●●
●
● ●●
●
●●
●●
100 200 300 400 500 600 700
● MySQL ● Voldemort
![Page 44: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/44.jpg)
Performance – Multi-node latency
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
32 nodes client side latency
output size (TB)
late
ncy
(ms)
0
10
20
30
40
median
●
●
●
●
●
●
●
●
●●
0 25 50 75 100
0
50
100
150
99th percentile
●
●
●
● ●
●
●
●●
●
0 25 50 75 100
● Uniform ● Zipfian
output size (TB)
late
ncy
(ms)
0
10
20
30
40
median
●
●
●
●
●
●
●
●
●●
0 25 50 75 100
0
50
100
150
99th percentile
●
●
●
● ●
●
●
●●
●
0 25 50 75 100
● Uniform ● Zipfian
output size (TB)
late
ncy
(ms)
0
10
20
30
40
median
●
●
●
●
●
●
●
●
●●
0 25 50 75 100
0
50
100
150
99th percentile
●
●
●
● ●
●
●
●●
●
0 25 50 75 100
● Uniform ● Zipfian
output size (TB)
late
ncy
(ms)
0
10
20
30
40
median
●
●
●
●
●
●
●
●
●●
0 25 50 75 100
0
50
100
150
99th percentile
●
●
●
● ●
●
●
●●
●
0 25 50 75 100
● Uniform ● Zipfian
![Page 45: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/45.jpg)
Performance – Production
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
People You May Know – 25 nodes client side latency
time (mins)
Med
ian
late
ncy
(ms)
0
2
4
6
8
10
12
14
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●●
●●
●●
●
●●●●●●●●●●●●●●●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
swap
0 50 100 150
![Page 46: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/46.jpg)
Related Work
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• Shared nothing databases • Yahoo’s PNUTS [Silberstein08] • Bulk load into range partitioned tables • Requires some compaction on serving system
• HBase [Konstantinou11], Cassandra [Lebresne11] • Offline tablet construction • Expensive rollback
• Search systems • Build index offline and pull • MapReduce[Dean04] use-case
![Page 47: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/47.jpg)
Conclusion
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
• LinkedIn • Serving ~120 stores in production for past 2 years • Fetching ~ 4 TB of data every day • 76 stores swapped every day
• Open-source • http://project-voldemort.com
![Page 48: Serving Large-scale Batch Computed Data with Project Voldemort · LinkedIn! LinkedIn! Serving Large-scale Batch Computed Data with Voldemort ! 2 4 8 17 32 55 90 2004 2005 2006 2007](https://reader036.vdocuments.site/reader036/viewer/2022062403/5fdb421ef10eac4586110343/html5/thumbnails/48.jpg)
Bibliography
LinkedIn Serving Large-scale Batch Computed Data with Voldemort
Images • Slide 6 - www.aviastar.org/air/inter/inter_concorde.php • Slide 15 - www.wright-brothers.org/Information_Desk/Just_the_Facts/Airplanes/Flyer_II.htm • Slide 17 - www.youtube.com/watch?v=oz-7wJJ9HZ0 • Slide 48 - www.wright-brothers.org/History_Wing/History_of_the_Airplane/ Century_Before/Road_to_Kitty_Hawk/Road_to_Kitty_Hawk.htm Papers [1] Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon's Highly Available Key-value Store. In Proceedings of 21st ACM SIGOPS symposium on Operating systems principles (SOSP '07) [2] Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation (OSDI ’04) [3] Adam Silberstein, Brian F. Cooper, Utkarsh Srivastava, Erik Vee, Ramana Yerneni, and Raghu Ramakrishnan. 2008. Efficient bulk insertion into a distributed ordered table. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data (SIGMOD '08) [4] Ioannis Konstantinou, Evangelos Angelou, Dimitrios Tsoumakos, and Nectarios Koziris. Distributed Indexing of Web Scale Datasets for the Cloud. In Proceedings of the 2010 Workshop on Massive Data Analytics on the Cloud (MDAC ’10) [5] Sasha Pachev. Understanding MySQL Internals. O’Reilly Media, 2007. [6] Tom White. Hadoop: The Definite Guide. O’Reilly Media, 2010���[7] Sylvain Lebresne. Using the Cassandra Bulk Loader. http://www.datastax.com/dev/blog/bulk-loading