pbs vldb 2012 - shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · soundcloud twitter mozilla yammer...
TRANSCRIPT
![Page 1: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/1.jpg)
Peter Bailis, Shivaram Venkataraman, Mike Franklin, Joe Hellerstein, Ion Stoica
VLDB2012
UC Berkeley
Probabilistically Bounded Stalenessfor Practical Partial Quorums
PBS
![Page 2: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/2.jpg)
watch a recording(of an earlier talk) at:http://vimeo.com/37758648
PBSNOTE TO READER
![Page 3: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/3.jpg)
play with a live demo at:http://pbs.cs.berkeley.edu/#demo
PBSNOTE TO READER
![Page 4: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/4.jpg)
R+W
strongconsistencyhigherlatency
eventualconsistency
lowerlatency
![Page 5: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/5.jpg)
consistencycontinuum is a choicebinary
strong eventual
![Page 6: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/6.jpg)
latency vs.consistency
informed by practice
our focus:
availability, partitions,failures
not in this talk:
![Page 7: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/7.jpg)
quantify eventual consistency:wall-clock time (“how eventual?”)versions (“how consistent?”)
analyze real-world systems:EC is often strongly consistentdescribe when and why
our contributions
![Page 8: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/8.jpg)
introsystem model
practicemetricsinsights
integration
![Page 9: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/9.jpg)
Apache, DataStax
Project Voldemort
Dynamo:Amazon’s Highly Available Key-value Store
SOSP 2007
![Page 10: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/10.jpg)
Adobe
Cisco
Digg
Gowalla
IBM
Morningstar
NetflixPalantir
Rackspace
Rhapsody
Shazam
Spotify
Soundcloud
Mozilla
Ask.comYammerAol
GitHubJoyentCloud
Best Buy
LinkedInBoeing
Comcast
Cassandra
RiakVoldemortGilt Groupe
![Page 11: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/11.jpg)
N replicas/keyread: wait for R replieswrite: wait for W acks
N=3R=2
![Page 12: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/12.jpg)
“strong”consistency
else:
R+W > Nif:
eventualconsistency
then:
![Page 13: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/13.jpg)
reads return the last acknowledged write or anin-flight write (per-key)
consistency___“strong”
regular register
R+W > N
![Page 14: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/14.jpg)
99th 99.9th1 1x 1x2 1.59x 2.35x3 4.8x 6.13xR
W99th 99.9th
1 1x 1x2 2.01x 1.9x3 4.96x 14.96x
LatencyLinkedIndisk-basedmodelN=3
![Page 15: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/15.jpg)
⇧consistency, ⇧latencywait for more replicas,
read more recent data
consistency, ⇧ ⇧
latencywait for fewer replicas,
read less recent data
![Page 16: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/16.jpg)
eventualconsistency“if no new updates are made to the object, eventually all accesses will return the last updated value”
W. Vogels, CACM 2008
R+W ≤ N
![Page 17: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/17.jpg)
HowHow long do I have to wait?
eventual?
![Page 18: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/18.jpg)
consistent?How
What happens if I don’t wait?
![Page 19: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/19.jpg)
R+W
strongconsistencyhigherlatency
eventualconsistency
lowerlatency
![Page 20: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/20.jpg)
introsystem model
practicemetricsinsights
integration
![Page 21: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/21.jpg)
Cassandra:R=W=1, N=3
by default(1+1 ≯ 3)
![Page 22: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/22.jpg)
eventual consistency
“maximum performance”
“very lowlatency”
okay for “most data”
“general case”
in the wild
![Page 23: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/23.jpg)
anecdotally, EC“good enough” for many kinds of data
How eventual?How consistent?
“eventual and consistent enough”
![Page 24: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/24.jpg)
Probabilistically Bounded Staleness
can’t make promisescan give expectations
Can we do better?
![Page 25: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/25.jpg)
introsystem model
practicemetricsinsights
integration
![Page 26: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/26.jpg)
HowHow long do I have to wait?
eventual?
![Page 27: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/27.jpg)
t-visibility: probability p of consistent reads after t seconds
(e.g., 10ms after write, 99.9% of reads consistent)
How eventual?
![Page 28: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/28.jpg)
t-visibility depends on
messaging and processing delays
![Page 29: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/29.jpg)
Coordinator Replicawrite
ack
read
response
wait for W responses
t seconds elapse
wait for R responses
response is stale
if read arrives before write
once per replica Time
![Page 30: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/30.jpg)
R1
R2
write
ack
read
W=1
R=1
N=2
response
Time
Alice
Bob
R2R1
R2inconsistent
![Page 31: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/31.jpg)
(R)
(W)write
ack
read
response
wait for W responses
t seconds elapse
wait for R responses
response is stale
if read arrives before write
once per replica
(A)
(S)
Coordinator Replica Time
![Page 32: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/32.jpg)
solving WARS: order statistics
dependent variables
instead: Monte Carlo methods
![Page 33: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/33.jpg)
to use WARS:
W53.244.5101.1
...
A10.38.211.3...
R15.322.419.8...
S9.614.26.7...
run simulationMonte Carlo, sampling
gather latency data
44.511.3
15.314.2
![Page 34: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/34.jpg)
real Cassandra cluster varying latencies:
t-visibility RMSE: 0.28%latency N-RMSE: 0.48%
WARS accuracy
![Page 35: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/35.jpg)
How eventual?
key: WARS modelneed: latencies
t-visibility: consistent reads with probability p after t seconds
![Page 36: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/36.jpg)
introsystem model
practicemetricsinsights
integration
![Page 37: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/37.jpg)
Yammer100K+ companies
uses Riak
LinkedIn 175M+ users
built and uses Voldemort
production latenciesfit gaussian mixtures
![Page 38: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/38.jpg)
10 ms
N=3
![Page 39: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/39.jpg)
Latency is combined read and write latency at 99.9th percentile
R=3, W=1100% consistent:Latency: 15.01 ms
LNKD-DISKN=3
16.5% faster
R=2, W=1, t =13.6 ms99.9% consistent:Latency: 12.53 ms
worthwhile?
![Page 40: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/40.jpg)
N=3
![Page 41: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/41.jpg)
Latency is combined read and write latency at 99.9th percentile
R=3, W=1100% consistent:Latency: 4.20 ms
LNKD-SSDN=3
59.5% faster
R=1, W=1, t = 1.85 ms99.9% consistent:Latency: 1.32 ms
![Page 42: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/42.jpg)
10�2 10�1 100 101 102 103
0.20.40.60.81.0
W=3
10�2 10�1 100 101 102 103
0.20.40.60.81.0
CD
F
W=1
10�2 10�1 100 101 102 103
Write Latency (ms)
0.20.40.60.81.0
W=2
LNKD-SSD LNKD-DISK YMMR WANLNKD-SSD LNKD-DISK YMMR WAN
N=3
![Page 43: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/43.jpg)
Coordinator Replica
write
ack(A)
(W)
response(S)
(R)
wait for W responses
t seconds elapse
wait for R responses
response is stale
if read arrives before write
once per replica
SSDs reducevariance
compared todisks!
read
![Page 44: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/44.jpg)
Yammer
latency81.1%
⇧
(187ms) 202 mst-visibility 99.9th
N=3
![Page 45: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/45.jpg)
k-staleness (versions)How consistent?monotonic reads
quorum load
in the paper
![Page 46: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/46.jpg)
in the paper
<k,t>-staleness:versions and time
![Page 47: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/47.jpg)
latency distributions
WAN model
varying quorum sizes
staleness detection
in the paper
![Page 48: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/48.jpg)
introsystem model
practicemetricsinsights
integration
![Page 49: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/49.jpg)
1.Tracing2. Simulation3. Tune N,R,W
Integration
Project Voldemort
![Page 50: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/50.jpg)
https://issues.apache.org/jira/browse/CASSANDRA-4261
![Page 51: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/51.jpg)
![Page 53: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/53.jpg)
Related WorkQuorum Systems• probabilistic quorums [PODC ’97]
• deterministic k-quorums [DISC ’05, ’06]
Consistency Verification• Golab et al. [PODC ’11]
• Bermbach and Tai [M4WSOC ’11]
• Wada et al. [CIDR ’11]
• Anderson et al. [HotDep ’10]
• Transactional consistency:Zellag and Kemme [ICDE ’11],Fekete et al. [VLDB ’09]
Latency-Consistency• Daniel Abadi [Computer ’12]
• Kraska et al. [VLDB ’09]
Bounded StalenessGuarantees
• TACT [OSDI ’00]
• FRACS [ICDCS ’03]
• AQuA [IEEE TPDS ’03]
![Page 54: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/54.jpg)
R+W
strongconsistencyhigherlatency
eventualconsistency
lowerlatency
![Page 55: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/55.jpg)
consistencycontinuum is a
strong eventual
![Page 56: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/56.jpg)
quantify eventual consistency
model staleness in time, versions
latency-consistency trade-offs
analyze real systems and hardware
pbs.cs.berkeley.eduPBSquantify which choice is best and explain
why EC is often strongly consistent
![Page 57: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/57.jpg)
Extra Slides
![Page 58: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/58.jpg)
PBS and apps
![Page 59: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/59.jpg)
staleness requires either:
staleness-tolerant data structurestimelines, logs
cf. commutative data structures logical monotonicity
asynchronous compensation codedetect violations after data is returned; see paper
cf. “Building on Quicksand” memories, guesses, apologies
write code to fix any errors
![Page 60: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/60.jpg)
minimize:(compensation cost)×(# of expected anomalies)
asynchronouscompensation
![Page 61: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/61.jpg)
Read only newer data?
client’s read rateglobal write rate
(monotonic reads session guarantee)
# versions tolerablestaleness
=
(for a given key)
![Page 62: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/62.jpg)
Failure?
![Page 63: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/63.jpg)
latency spikes
Treat failures as
![Page 64: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/64.jpg)
How l o n gdo partitions last?
![Page 65: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/65.jpg)
what time interval?99.9% uptime/yr ⇒ 8.76 hours downtime/yr
8.76 consecutive hours down⇒ bad 8-hour rolling average
hide in tail of distribution ORcontinuously evaluate SLA, adjust
![Page 66: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/66.jpg)
10�2 10�1 100 101 102 103
0.20.40.60.81.0
W=3
10�2 10�1 100 101 102 103
0.20.40.60.81.0
CD
F
W=1
10�2 10�1 100 101 102 103
Write Latency (ms)
0.20.40.60.81.0
W=2
LNKD-SSD LNKD-DISK YMMR WAN
LNKD-SSD LNKD-DISK YMMR WANLNKD-SSD LNKD-DISK YMMR WAN
LNKD-SSD LNKD-DISK YMMR WAN
N=3
![Page 67: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/67.jpg)
10�2 10�1 100 101 102 103
0.20.40.60.81.0
R=3
LNKD-SSD LNKD-DISK YMMR WAN
LNKD-SSD LNKD-DISK YMMR WANLNKD-SSD LNKD-DISK YMMR WAN
LNKD-SSD LNKD-DISK YMMR WAN
10�2 10�1 100 101 102 103
0.20.40.60.81.0
CD
F
W=1
10�2 10�1 100 101 102 103
Write Latency (ms)
0.20.40.60.81.0
W=2
(LNKD-SSD and LNKD-DISK identical for reads)N=3
![Page 68: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/68.jpg)
<k,t>-staleness:versions and time
approximation: exponentiate
t-staleness by k
![Page 69: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/69.jpg)
Synthetic,Exponential Distributions
W 1/4x ARS
W 10x ARS
N=3, W=1, R=1
![Page 70: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/70.jpg)
concurrent writes:deterministically choose
Coordinator R=2
(“key”, 1) (“key”, 2)
![Page 71: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/71.jpg)
![Page 72: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/72.jpg)
![Page 73: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/73.jpg)
![Page 74: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/74.jpg)
![Page 75: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/75.jpg)
N = 3 replicas
Coordinator
read(“key”)
(“key”, 1)(“key”, 1)(“key”, 1)
client
read(“key”)(“key”, 1)readR=3
R1 R2 R3(“key”, 1) (“key”, 1) (“key”, 1)
![Page 76: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/76.jpg)
N = 3 replicas
Coordinator
read(“key”)
(“key”, 1)(“key”, 1)(“key”, 1)
(“key”, 1)read(“key”)readR=3
R1 R2 R3(“key”, 1) (“key”, 1) (“key”, 1)
client
![Page 77: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/77.jpg)
N = 3 replicas
Coordinator
read(“key”)
(“key”, 1)(“key”, 1)(“key”, 1)
(“key”, 1)read(“key”)
send read to all
readR=1
R1 R2 R3(“key”, 1) (“key”, 1) (“key”, 1)
client
![Page 78: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/78.jpg)
(“key”, 2)
Coordinator Coordinator
read(“key”)
ack(“key”, 2)
write(“key”, 2)
write(“key”, 2)ack(“key”, 2)
W=1
R1(“key”, 1) R2(“key”, 1) R3(“key”, 1)(“key”, 2)
(“key”, 1)
read(“key”)
(“key”,1)
ack(“key”, 2) ack(“key”, 2)
(“key”, 2)
(“key”, 2) (“key”, 2)
R=1
![Page 79: PBS VLDB 2012 - Shivaramshivaram.org/talks/pbs-vldb12-talk.pdf · Soundcloud Twitter Mozilla Yammer Ask.com Aol GitHub JoyentCloud Best Buy LinkedIn Boeing Comcast Cassandra Voldemort](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f98e4db8f206109922300d1/html5/thumbnails/79.jpg)
N=3