WriteFast,ReadinthePast:CausalConsistencyforClient-sideApps
withSwiftCloud
PresentedbyMarekZawirskiInria/UPMC-LIP6,Paris
(nowatGoogle,Zürich)
MarekZawirski,NunoPreguiça,SérgioDuarte,AnnetteBieniusa,ValterBalegas,MarcShapiro
Challenge:DatabaseAccessforClient-side Apps
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 1
App
App
App
App
Challenge:DatabaseAccessforClient-side Apps
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 2
App
App
App
App
API
API
Limitedboundariesofserver-sidedatabaseguarantees
Challenge:DatabaseAccessforClient-side Apps
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 3
App
App
App
App
� ad-hocontheclient-side
API
API
Limitedboundariesofserver-sidedatabaseguarantees
Challenge:DatabaseAccessforClient-side Apps
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 4
API
App
App
API
App
App
API
API
ExtendedboundarieswithSwiftCloud
Challenge:DatabaseAccessforClient-side Apps
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 5
API
App
App
API
App
App
API
API
ExtendedboundarieswithSwiftCloud
• Consistent,available and convergentdataaccess
• Scalability with#objectsand#clients
• Fault-tolerance
StrongerthanEventual:CausalConsistency
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 6
bob_posts.add(“don’tthinkofvisitingVancouver…”)
bob_posts.add(“…justdoit!YOLO”)
replies.add(“Alice:totallyJ”)
clientorder
read
Defaultonclient-side:eventualconsistency� anomalies
StrongerthanEventual:CausalConsistency
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 7
bob_posts.add(“don’tthinkofvisitingVancouver…”)
bob_posts.add(“…justdoit!YOLO”)
replies.add(“Alice:totallyJ”)
clientorder
read
Defaultonclient-side:eventualconsistency� anomalies
read
StrongerthanEventual:CausalConsistency
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 8
!?!
bob_posts.add(“don’tthinkofvisitingVancouver…”)
bob_posts.add(“…justdoit!YOLO”)
replies.add(“Alice:totallyJ”)
clientorder
readread
Defaultonclient-side:eventualconsistency� anomalies
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 9
ü
Causalconsistency:readsfromcausally-closedsnapshot
bob_posts.add(“don’tthinkofvisitingVancouver…”)
replies.add(“Alice:totallyJ”)
readread
clientorder
StrongerthanEventual:CausalConsistency
bob_posts.add(“…justdoit!YOLO”)
Defaultonclient-side:eventualconsistency� anomalies
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 10
bob_posts.add(“don’tthinkofvisitingVancouver…”)
replies.add(“Alice:totallyJ”)
read
replies.add(“Eve:Bob�”)
replies:???
clientorder
ConvergentCausalConsistency:NoLostUpdates
bob_posts.add(“…justdoit!YOLO”)
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 11
replies:{“Alice:totallyJ”,“Eve:Bob�”}
High-levelconvergentobjects[CRDTs] resolveconcurrency
bob_posts.add(“don’tthinkofvisitingVancouver…”)
replies.add(“Alice:totallyJ”)
clientorder
read
replies.add(“Eve:Bob�”)
ConvergentCausalConsistency:NoLostUpdates
bob_posts.add(“…justdoit!YOLO”)
Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 12
objB.op
objB.op
objA.op
objC.op
[PRACTI,NSDI’06]Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 13
objB.op
objB.op
objA.op
objC.opobjAobjB
[PRACTI,NSDI’06]
currentstate
Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 14
objB.op
objB.op
objA.op
objC.opobjAobjB
objBobjC
[PRACTI,NSDI’06]Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 15
objB.op
objB.op
objA.op
objC.op
objAobjC
objAobjB
objBobjC
[PRACTI,NSDI’06]
Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 16
objB.op
objB.op
objA.op
objC.opobjAobjB
objBobjC
[PRACTI,NSDI’06]
objAobjC
Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 17
objB.op
objB.op
objA.op
objC.opobjAobjB
objBobjCdependencies
[PRACTI,NSDI’06]
objB.op
objA.op
objC.op
objB.op
objAobjC
dependencies
Challenge:CausalConsistencywithPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 18
objB.op
objB.op
objA.op
objC.opobjAobjB
objBobjC
[PRACTI,NSDI’06]
objB.op
objB.op1 0 2Inherenttrade-offsinthegeneralcase:scalabilityvs.availabilityvs.fault-tolerance
objAobjC
Approach:Cloud-backedPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 19
DCgeo-replication
DataCenterfullreplicas:
üProvideconsistentviewAssignsmallmetadataü4 9
DC
Approach:Cloud-backedPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 20
DCgeo-replication
4 9
DC
logcache
App
DataCenterfullreplicas:
üProvideconsistentviewAssignsmallmetadataü
Approach:Cloud-backedPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 21
DCgeo-replication
4 9
DC
logcache
App
update
DataCenterfullreplicas:
üProvideconsistentviewAssignsmallmetadataü
Approach:Cloud-backedPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 22
DCgeo-replication
4 9
DC
logcache
App
update
4 9
DataCenterfullreplicas:
üProvideconsistentviewAssignsmallmetadataü
5
Approach:Cloud-backedPartialReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 23
DCgeo-replication
4 9
DC
logcache
App
Clientreads:cachedfragmentofcloudversion� ownlog
ü HighavailabilityConsistencyw/read-your-writesü
DataCenterfullreplicas:
üProvideconsistentviewAssignsmallmetadataü
read
PotentialofCloud-backedClientReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 24
Setup:DCsin3AWSEC2 regions,YCSBworkload,cache=256objects
Responsetimeofoperation[ms]
CDFfor10
00clients
PotentialofCloud-backedClientReplicas
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 25
Responsetimeofoperation[ms]
CDFfor10
00clients
Objectsinthecache� immediate,consistentresponse
Setup:DCsin3AWSEC2 regions,YCSBworkload,cache=256objects
ChallengefortheCloudApproach:SafeDCFailover
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 26
cache
log
ChallengefortheCloudApproach:SafeDCFailover
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 27
cache
log
riskyread
ChallengefortheCloudApproach:SafeDCFailover
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 28
cache
log
riskyread
operationswithriskydependencies
ChallengefortheCloudApproach:SafeDCFailover
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 29
cache
log
riskyread
operationswithriskydependencies
newDCinincompatiblestate
SupportingFailoverbyConservativeReads
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 30
cacheK=2
log
Foreignupdates:readversionreplicatedinK>1DCsOwnwrites:readfromthelog,recovertoanewDC
SupportingFailoverbyConservativeReads
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 31
cacheK=2
log
Foreignupdates:readversionreplicatedinK>1DCsOwnwrites:readfromthelog,recovertoanewDC
conservativeread
SupportingFailoverbyConservativeReads
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 32
cacheK=2
log
ü
Foreignupdates:readversionreplicatedinK>1DCsOwnwrites:readfromthelog,recovertoanewDC
newDCincompatiblestate
Experiment:InjectionofShortDCDisconnection
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 33
respon
setim
e[m
s]
Experiment:InjectionofShortDCDisconnection
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 34
remotereads
fastconservativereads
respon
setim
e[m
s]
Experiment:InjectionofShortDCDisconnection
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 35
remotereads
fastconservativereads
remoteops:smoothfailover
…unaffected…
respon
setim
e[m
s]
Experiment:InjectionofShortDCDisconnection
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 36
Trade-offcontrolledbyK:stalenessvs.availability• StalenessnegligibleinmostK=2setups,<1%reads• Incherry-pickedunfavorablesetup,1.0–2.5%reads
remotereads
fastconservativereads
remoteops:smoothfailover
…unaffected…
respon
setim
e[m
s]ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 37
inc() 44
x=0 x=0x=0
x=1
time
x.inc()(x=1)
ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 38
inc() 44
inc() 3
x=0 x=0x=0
x=1
x=1
x.inc()(x=1)
ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 39
inc() 44
inc() 33
x=0 x=0x=0
x=1
x=1
x.inc()(x=1)
ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 40
inc() 44
inc() 33
inc() 3inc()4geo-replication
x.inc()(x=1)
x=0 x=0x=0
x=1
x=1
x=2
ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 41
duplicatedelivery!
inc() 44
inc() 33
inc() 3inc()4geo-replication
x.inc()(x=1)
x=0 x=0x=0
x=1
x=1
x=2
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 42
geo-replication
inc() 44
inc() 33
3inc()4
1
1
4
1~
1 ~
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
ü
safe:≤1delivery
3
x=0 x=0x=0
x.inc()(x=1) x=1
x=1
x=1
1
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 43
geo-replication
inc() 44
inc() 33
3inc()4
1
1
4
1~
1 ~
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
ü
safe:≤1delivery
3
x=0 x=0x=0
x.inc()(x=1) x=1
x=1
x=1
1
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 44
geo-replication
inc() 44
inc() 33
3inc()4
1
1
4
1~
1 ~
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
Extension:logpruningindependentofclientavailability
ü
safe:≤1delivery
3
x=0 x=0x=0
x.inc()(x=1) x=1
x=1
x=1
1
#clientreplicas
Experiment:SizeofMetadataonClient-DCLink
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 45
1
10
100
1K
metadata/up
date[B
]
SwiftCloud’s decoupledmetadata:const size
Setup:3DCs,YCSBBuniformworkload
Client-assignedvectors[PRACTI,NSDI’06]:unboundedoverhead
Summary
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 46
SwiftCloud providesclient-side apps:
• Consistent,available and convergentobjectdatabase
• Scalability:fullreplicasatDCbackpartialatclient� smallcausalitymetadata(<15B/update)
• Fastfailoverthankstoconservativereads(<1%stale)
• Saferetryofinterruptedtransferandsafelogpruningthankstodecoupledmetadata
Researchprototypeatgithub.com/SyncFree/SwiftCloud
SwiftCloud comparedto“LazyReplication”
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 47
• Assumeclient-sideapplicationlogic• Describecausalconsistencysupport• Supportcommunicationwithmultipleservers• Usedecoupledmetadata
• DB=RDTobjects+globaltransactions
• Supportspartialclientreplicas =>fastreadsandread-your-writes
• K-stability-driventrade-off• GCindependentofclients
• MonolithicDB
• Noclient-sidereplicas
• Stabilitydiscussion• Physical-clock-drivenGC• Moreconsistencychoices
ChallengefortheCloudApproach:ProtocolRetries
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 48
cache
log
inc 3
inc 4
duplicatedelivery!
inc 4
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 49
cache
log
inc 4
inc 3
1
14 ü
safe:≤1delivery
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 50
cache
log
inc 4
inc 3
1
14 ü
safe:≤1delivery
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
SafeRetrieswithDecoupledMetadata
Zawirski etal.,WriteFast,ReadinthePast:CausalConsistencyforClient-sideApplicationswithSwiftCloud 51
cache
log
inc 4
inc 3
1
14 ü
safe:≤1delivery
Solution:client-assignedtimestampsforsafety+1..NDCtimestampsforefficientsummary
Extension:logpruningindependentofclientavailability