![Page 1: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/1.jpg)
Speculations:Speculative Execution in a Distributed File System1
andRethink the Sync2
Edmund Nightingale12, Kaushik Veeraraghavan2, Peter Chen12, Jason Flinn12
Presentation by Ji-Yong Shin(Some slides are from Nightingale’s talk)
![Page 2: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/2.jpg)
Agenda
• CAP theorem, Consistency Semantics, Consistency Model
• Papers– Speculative Execution in a Distributed File System
(Award Paper from SOSP’05)– Rethink the Sync (Best Paper from OSDI’06)
![Page 3: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/3.jpg)
CAP Theorem by Eric Brewer
• At most two of CAP can be satisfied simultaneously– Consistency: correctness of data– Availability: guaranteed immediate access to data– Partition Tolerance: guaranteed functioning
despite network disruption or partition
N1 N2
A B
C
A
P
B ASync
![Page 4: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/4.jpg)
ACID vs BASE(Not exactly opposite but..)
ACID• Atomicity• Consistency• Isolation• Duration
BASE• Basically Available• Soft-state• Eventually consistent
Distributed File System Design
![Page 5: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/5.jpg)
Consistency Semantics by Leslie Lamport
• Atomic (single copy)– Every read returns the value of the most recent write
• Regular– Read not concurrent with any write returns the most recent
write– Read concurrent with some writes returns either the most
recent write or a value of concurrent write• Safe– Read not concurrent with any write returns the most recent
write– Read concurrent with some writes returns any value
![Page 6: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/6.jpg)
Consistency Models• Strict consistency
– All executions in strict order• Sequential consistency
– All execution results exposed in strict order• Causal consistency
– All executions results with causal dependency exposed in strict order
• Close-to-open consistency– All execution results of processes that closed the file should be exposed to process
opening the file
• Delta consistency– After fixed period of time all memory parts will be consistent
• Eventually consistency– After sufficiently long period of time all memory parts will be consistent
![Page 7: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/7.jpg)
• Consistency and CAP theorem• Papers– Speculative Execution in a Distributed File System
(Award Paper from SOSP’05)– Rethink the Sync (Best Paper from OSDI’06)
![Page 8: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/8.jpg)
Authors
• Edmund B Nightingale– PhD from UMich (Jason Flinn)– Microsoft Research– Both papers are part of PhD Thesis
• Kaushik Veeraraghavan– PhD Student in Umich (Jason Flinn)
• Peter M Chen– PhD fromUCB (David Patteron)– Faculty at UMich
• Jason Flinn– PhD at CMU (Mahadev Satyanarayanan)– Faculty at Umich
![Page 9: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/9.jpg)
Speculation
• Execute using assumption– If assumption holds
• performance gain
– If assumption fails• Restart execution• Not much performance overhead
• Example– Branch prediction– Transaction– Thread level speculation in multiprocessor (or multicore)
IF DEC EX WB
If (a == 1)
If (a == 1)
If (a == 1)
If (a == 1)
b = 0
c = 1 b = 0
c = 1 b = 0
c = 1 b = 0
c = 1
If (a == 1) {b = 0;c = 1;
} else {b = 1;c = 0;
}
Clk
cycl
e
IF DEC EX WB
If (a == 1)
If (a == 1)
If (a == 1)
If (a == 1)
b = 0
c = 1 b = 0
c = 1 b = 0
c = 1 b = 0
c = 1
IF DEC EX WB
If (a == 1)
b = 0 If (a == 1)
c = 1 b = 0 If (a == 1)
c = 1 b = 0 If (a == 1)
c = 1 b = 0
c = 1
IF DEC EX WB
If (a == 1)
b = 0 If (a == 1)
c = 1 b = 0 If (a == 1)
c = 1 b = 0 If (a == 1)
b = 1
c = 0 b = 1
c = 0 b = 1
c = 0 b = 1
c = 0
Sync Delay
Sync Complete
Rollback and
restart
![Page 10: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/10.jpg)
Motivation and Approach
• Distributed file system– Significant cost for consistency and safety
• Block and wait from sync msg and write
– Tradeoff between consistency and performance• Weak consistency for high performance
• Speculative distributed file system– Execute sync operations in async manner– While syncing execute next operation on cached files– Check correctness later and rollback if necessary– Guarantee single copy semantics
![Page 11: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/11.jpg)
11
Big Idea: Slow Way
RPC Req
Client
RPC Resp
Server
Block!2) Speculate!
1) Checkpoint
Big Idea: Speculator
3) Correct?
Yes: discard ckpt.No: restore process & re-execute RPC Req
RPC Resp
RPC Req
RPC Resp
![Page 12: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/12.jpg)
Conditions for Success
1. Highly predictive operations– Misprediction can worsen performance• Rare misprediction
2. Faster checkpointing compared to remote IO– Slow checkpointing is not worth doing• 52us for small process < network IO
3. Available spare resource for speculation– Speculation requires memory and CPU cycles• Modern computers have abundant resource
![Page 13: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/13.jpg)
13Undo log
Implementing SpeculationPro
cess
Checkpoint Spec
1) System call 2) Create speculation (create_speculation)
Time
Copy on write fork()Tracks kernel objects that depend on it
Ordered list of speculative operations
![Page 14: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/14.jpg)
14
Speculation Success
Undo log
Checkpoint
1) System call 2) Create speculation
Proce
ss
3) Commit speculation
Time
Spec
(commit_speculation)
Tracks kernel objects that depend on it
Ordered list of speculative operations
![Page 15: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/15.jpg)
15
Speculation Failure
Undo log
Checkpoint
1) System call 2) Create speculation 3) Fail speculation
Proce
ss
Time
Spec
(fail_speculation)
Tracks kernel objects that depend on it
Ordered list of speculative operations
Proce
ss
![Page 16: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/16.jpg)
Multi-Process Speculation
• Processes often cooperate– Example: “make” forks children to compile, link,
etc.– Would block if speculation limited to one task
• Supports– Propagate dependencies among objects– Objects rolled back to prior states when specs fail
![Page 17: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/17.jpg)
17
Spec 1Spec 1
Multi-Process Speculation
Spec 2
pid 8001
Checkpoint
Checkpoint
inode 3456
Chown-1
Write-1
pid 8000
CheckpointCheckpoint
Checkpoint
Chown-1
Write-1
Stat A Stat B
![Page 18: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/18.jpg)
Ensuring Correctness
• Speculative state must never be visible to 1. User or external device2. Process not depending on the it
• Controlling speculative process– Block access to external environment
• Read only (getpid) and private state updates (dup2) allowed
– Buffer write to external device– Propagate speculation if necessary
![Page 19: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/19.jpg)
19
Multi-Process Speculation
• Supports– Objects in distributed file system
• Will be explained in next slides
– Objects in local memory file system (RAMFS)– Objects in local disk file system
• Use buffering strategy for speculation• Shared on-disk metadata: only valid state committed using redo and undo• Journal: only commit non speculative operations
– Etc• Pipe, fifos, unix sockets, signals, fork, exit
• Doesn’t support– write-shared memory including V IPC, futex
![Page 20: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/20.jpg)
Using SpeculationClient 1
cat foo(0)
> bar(1)
Client 2
cat bar
Server
foo(1), bar(0) bar(1)foo(0), bar(0) foo(1), bar(1)
![Page 21: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/21.jpg)
Using SpeculationClient 1
cat foo(0)
> bar(1)
Client 2
cat bar
Server
foo(1), bar(0) bar(0)foo(0), bar(0)
foo(0)
foo(0), bar(1)
• Mutating Operation – Server determines speculation success/failure
• State at server never speculative• Can be durable to server crash
– Requires server to track failed speculations– Requires in-order processing of messages
![Page 22: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/22.jpg)
22
Group Commit
• Previously sequential ops now concurrent• Sync ops usually committed to disk• Speculator makes group commit possible
write
writecommit
commit
ClientClient Server ServerUpdating different files…
Can significantly improve disk throughput
![Page 23: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/23.jpg)
23
Implementation• SpecNFS
– Modified NFSv3 in Linux 2.4 kernel to support Speculator• Same RPCs issued (but many now asynchronous)• SpecNFS has same close-to-open consistency, safety as NFS
• BlueFS– new file system for Speculator
• Single copy semantics• Each file, directory, etc. has version number• Check server for every operation
• Two Dell Precision 370 desktops as the client and file server• Routed packet through NISTnet network emulator to insert delay.
![Page 24: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/24.jpg)
24
Apache Build
• With delays SpecNFS up to 14 times faster
0
50
100
150
200
250
300
No delay
Tim
e (s
eco
nd
s)
NFS
SpecNFS
BlueFS
ext3
0
500
1000
1500
2000
2500
3000
3500
4000
4500
30 ms delay
![Page 25: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/25.jpg)
25
The Cost of Rollback
• All files out of date SpecNFS up to 11x faster
0
20
40
60
80
100
120
140
NFS SpecNFS ext3
No delay
Tim
e (s
eco
nd
s)
0
200
400
600
800
1000
1200
1400
1600
1800
2000
NFS SpecNFS ext3
30ms delay
No files invalid10% files invalid
50% files invalid100% files invalid
![Page 26: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/26.jpg)
26
Group Commit & Sharing State
050
100150200250300350400450500
NFS SpecNFS BlueFS
0 ms delay
Tim
e (s
eco
nd
s)
0
500
1000
1500
2000
2500
3000
3500
4000
4500
NFS SpecNFS BlueFS
30ms delay
Default
No prop
No grp commit
No grp commit & no prop
![Page 27: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/27.jpg)
27
Conclusion
• Speculator greatly improves performance of existing distributed file systems
• Speculator enables new file systems to be safe, consistent in some sense and fast
![Page 28: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/28.jpg)
Discussion
• Starvation (Infinite rollback)?
• Overhead for maintaining speculation?– Memory or CPU?
• Multiple server environment?
• Consistency?– Consistency Semantics?– Consistency Model?– CAP Theorem?
![Page 29: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/29.jpg)
• CAP theorem, Consistency Semantics, Consistency Model
• Papers– Speculative Execution in a Distributed File System
(Award Paper from SOSP’05)– Rethink the Sync (Best Paper from OSDI’06)
![Page 30: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/30.jpg)
Synchronization
Asynchronous IO• High performance
– Non-blocking
• Low reliability– Vulnerable to crash– Ordering not guaranteed
Synchronous IO• Low performance
– Blocking
• High reliability – Resilient to crash – Guaranteed ordering of IO
External Synchrony• High performance close to async IO– Async-like execution until externalization
• High reliability close to synchronous – User centric view of guaranteed durability
![Page 31: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/31.jpg)
External Synchrony
• Delay commit of data until externally observable operation is necessary– Print to screen– Packet send
• Externally observable behavior implicates– Operation before the observed behavior are
committed
![Page 32: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/32.jpg)
Example: Synchronous I/O
OS Kernel DiskProcess
101 write(buf_1);102 write(buf_2);103 print(“work done”);104 foo();
Application blocksApplication blocks
%work done%
TEXT
%
![Page 33: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/33.jpg)
Example: External synchrony
OS Kernel DiskProcess
101 write(buf_1);102 write(buf_2);103 print(“work done”);104 foo();
TEXT
%work done%%
![Page 34: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/34.jpg)
Improving Performance
• Group commit of multiple modification– Atomic commit reduces disk access
• Buffering of output– Output function runs while committing– Buffered output is released after completion of
commit
![Page 35: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/35.jpg)
Multiprocess support
• Necessary functions– Tracking down causal dependencies• Speculator concept borrowed
– Output triggered commit• Buffering output borrowed from Speculator
![Page 36: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/36.jpg)
Multiprocess support
DiskProcess 1
101 write(file1);
102 do_something();
%hello%%
101 print (“hello”);
102 read(file1);
103 print(“world”);
Process 1 Process 2
Process 2
Commit Dep 1
Process 1
OS Kernel
Process 2 TEXTTEXTworld
![Page 37: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/37.jpg)
Limitation
• Application specific recovery is difficult– Delayed commit makes it difficult to track back
• Commit may be unlimitedly delayed– 5 second rule applied, users may not meet user’s
expectation• Data in multiple file system is difficult to
commit in single transaction– Journal in different locations
![Page 38: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/38.jpg)
Implementation
Speculator• Hide speculative state until
RPC response• Trace of causal dependency
for commit and roll back• Buffers output • Group commit
External Synchrony• Delay commit until
externalization • Trace of causal dependency
for commit• Buffers output• Group commit
• Implemented ext sync file system Xsyncfs– Based on the ext3 file system and Speculator– Use journaling to preserve order of writes– Use write barriers to flush volatile cache– Write to disk guaranteed
![Page 39: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/39.jpg)
Evaluation
• Compare Xsyncfs to 3 other file systems– Default asynchronous ext3– Default synchronous ext3– Synchronous ext3 with write barriers
![Page 40: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/40.jpg)
When is data safe?
File SystemConfiguration
Data durable on write()
Data durable on fsync()
Asynchronous NoNot on
power failure
SynchronousNot on
power failureNot on
power failure
Synchronousw/ write barriers Yes Yes
External synchrony Yes Yes
![Page 41: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/41.jpg)
Postmark benchmark
Xsyncfs within 7% of ext3 mounted asynchronously
1
10
100
1000
10000
Tim
e (S
eco
nd
s)
ext3-asyncxsyncfsext3-syncext3-barrier
![Page 42: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/42.jpg)
The MySQL benchmark
0500100015002000250030003500400045005000
0 5 10 15 20
Number of db clients
New
Ord
er T
ran
sact
ion
s P
er M
inu
te
xsyncfsext3-barrier
MySQL’s group commit can reach xsyncfs performance when # of client is large
![Page 43: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/43.jpg)
Specweb99 throughput
Xsyncfs within 8% of ext3 mounted asynchronously Lots of operations buffered, more externalization
0
50
100
150
200
250
300
350
400
Th
rou
gh
pu
t (K
b/s
)
ext3-async
xsyncfs
ext3-sync
ext3-barrier
![Page 44: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/44.jpg)
Conclusion
• New concept, external synchrony, proposed• External synchrony performs with 8% of async
![Page 45: Speculations: Speculative Execution in a Distributed File System 1 and Rethink the Sync 2 Edmund Nightingale 12, Kaushik Veeraraghavan 2, Peter Chen 12,](https://reader034.vdocuments.site/reader034/viewer/2022042615/56649d2f5503460f94a0663d/html5/thumbnails/45.jpg)
Discussion
• What happens when external synchrony system fails?
• Consistency?– Consistency Semantics?– Consistency Model?– CAP Theorem?