Ordering and Consistent Cuts

Presented by Chi H. Ho

Time, Clocks, and the Ordering of Events in a Distributed System

Leslie Lamport

Introduction

• 2000 PODC Influential Paper Award• Outline of the paper: not in presented order

– Partial and Total Orderings– Logical and Physical Clocks– Clock and Strong Clock Conditions– Synchronize Physical Clocks

• Beyond…

“Happened Before”

• a b: if– a and b are events in the same process and a

comes before b, or– a is the send event of some message, and b is

the receive event of the same message.

• Transitive:

(a b) & (b c) (a c)• Concurrent: (a b) & (b a).

Partial Ordering

Examples

• q5 p4

• q2 q3

• p1 r3

• q2 // p2

• q2 // p3

Partial Ordering

Logical Clock

• Clock Condition:

a,b:a b C(a) < C(b)

Partial Ordering Implementation

Logical Clock

• Implementation Rules:– IR1:

• Each process Pi increments Ci between any two successive events.

– IR2:• If event a is the sending of a message m by process Pi,

then the message contains a timestamp Tm = Ci(a).

• Upon receiving a message m, process Pj sets Cj greater than or equal to its present value and greater than Tm.

Partial Ordering Implementation

Examples

Partial Ordering Implementation

P0

P1

0

Examples

Partial Ordering Implementation

P0

P1

0

0

Examples

Partial Ordering Implementation

P0

P1

0

0

1

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1 2

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1 2 3

[3]

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1 2 3

[3]

2

[2]

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1 2 3

[3]

2

[2]

4

Examples

Partial Ordering Implementation

P0

P1

0

0

1

1 2 3

[3]

2

[2]

4

4

Extended “Happened Before”

• a => b: iff– Ci(a) < Cj(b), or

– (Ci(a) = Cj(b)) & (Pi ≺ Pj)

Total Ordering

Example Application

• Shared resource granting– Fixed number of processes– Single shared resource– Requirements:

I. Mutual Exclusive

II. Fair

III. Exhaustive

Total Ordering

Example Application

• Solution: Distributed algorithm• Model:

– Channels are FIFO– Each process maintains a process queue

• Algorithm– Request: broadcast Tm:Pi request resource– Release: broadcast Tm:Pi release resource– Receive request: enqueue– Receive release: dequeue– Resource granted (local decision): Pi

• Tm:Pi request resource w/ Tm min• Pi has received from every process a msg timestamped later than Tm

• Note: – Can be generalized to solve Replicated State Machine!

Total Ordering

Anomaly

Amazon.com[19]

Anomaly

Amazon.com[19]

Anomaly

Amazon.com[19] [7]

Anomaly

Amazon.com[19]

[7]

Anomaly

Amazon.com[19]

[7]

External event

Strong Clock Condition

• S = {events in the system}

• S = S ⋃ {relevant external events}

is “happened before” for S

∀a,b ∈ S:

a b C(a) < C(b)

Avoid Anomaly

Physical Clocks

• PC1: (drift rate bound)

∃ << 1 such that ∀i: |dCi(t)/dt – 1| <

• PC2: (drift bound)

i,j: |Ci(t) – Cj(t)| <

Avoid Anomaly

< shortest msg transmission time• ∀i,j,t:

Ci(t+) – Cj(t) > 0

Physical Clocks

/(1-)

Amazon.com

ji

Cj(t) > Ci(t+)

>

Implementation Rules

• IR1’: – For each i, if Pi does not receive a message at

physical time t, then Ci is differentiable at t and dCi(t)/dt > 0.

• IR2’:– (a) If Pi sends a message m at physical time t,

then m contains a timestamp Tm = Ci(t).– (b) Upon receiving a message m at time t’,

process Pj sets Cj(t’) equal to maximum (Cj(t’-0), Tm + m)

Physical Clocks

Synchronize Physical Clocks

Physical Clocks

• Problem statement:– IR1’ and IR2’ are followed,– Message delay is bounded,– Clocks satisfied PC1,– Goal: PC2

• Algorithm:– Every seconds, a message is sent over every arc.

• Guarantees:– Clocks are synchronized after t0 + d d(2 + )

Beyond…

• Shortcomings:– No gap-detection property– C(a) < C(b) ???– Bounds are not practical (So is PC!)

Gap Detection Property

• Problem statement:– Given: a, b, C(a), C(b), C(a) < C(b), – Determine if c exists, where

C(a) < C(c) < C(b)?

Beyond…

Another Strong Clock Condition

a b C(a) < C(b)

Beyond…

What clock, then?

• Causal histories:

Beyond…

• Vector Clocks:

More on Vector Clocks

Strong Clock ConditionConcurrentPair-wise InconsistentConsistent CutCountingGap Detection

Beyond…

More on Vector Clocks

Strong Clock ConditionConcurrentPair-wise InconsistentConsistent CutCountingGap Detection, but…

Beyond…

X Weak Gap-Detection

Given a, b, can detect existence of c such that

(c a) & (c b)

Reference

• O. Babaoglu and K. Marzullo. Consistent global states of distributed systems: Fundamental concepts and mechanisms. In Sape Mullender, editor, Distributed Systems, ch. 4, pages 55--96. Addison Wesley, 2nd ed., 1993. http://citeseer.ist.psu.edu/babaoglu93consistent.html

• Note: some materials in this paper are used to clarify a few concepts in the next paper.

Beyond…

Distributed Snapshots: Determining Global States of Distributed Systems

K. Mani Chandy

Leslie Lamport

Introduction

• Outline of the paper:– Motivation– Model– Algorithm– Correctness– Other issues

• Beyond…

Motivation

• Capture the global state of a system.

• Really?True global state:

Impossible!!!

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

Motivation

• Capture the global state of a system.

• Really?These are what

can be done

Are they useful?

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

Motivation

• Capture the global state of a system.

• Useful?

Equivalent!

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

Motivation

• Capture the global state of a system.

• Useful?

Consistent, but not happens in reality.

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

p1

e11 e1

2 e13 e1

4 e15 e1

6

p2e2

1 e22 e2

3 e24 e2

5

Motivation

• Capture the global state of a system.

• Useful?

Not even consistent!

p1

e11 e1

2 e13 e1

4 e15e1

6

p2e2

1 e22 e2

3 e24 e2

5

Motivation

• Capture the global state of a system.

• Useful? Yes:– To detect stable

properties of a system:y(S) y(S’)

for all S’ reachable from S.

– E.g.: • “computation has

terminated,” • “the system is

deadlocked,”• “all tokens in a token

ring have disappeared.”

Model

A distributed system

• A distributed system (on the right).

• A global state = set of processes’ and channels’ states.

• Event:– atomic– e = <p, S, S’, m,

c>• Computation:

– seq =(ei: 0 i n)

– Si+1 = next(Si, ei)

• Channels’ assumptions:– Singly directed– FIFO– Asynchronous– Error free– Infinite buffer

Algorithm

• Invoker: behave as if receiving a marker from a virtual node.• Receiving rule for process q receiving a marker along channel

c:if q has not recorded its state then

begin q records its state; q records the state c as the empty sequenceend

else q records the state of c as the sequence of messages received along c after q’s state was recorded and before q received the marker along c.

• Sending rule for a process p: for each outgoing channel c:p sends one marker along c after p records its

state and before p sends further messages along c.

Illustration

• Next 14 slides, courtesy of Professor Birman.

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

I want to start a

snapshot

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

p records local state

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

p starts monitoring incoming channels

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

“contents of channel p-y”

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

p floods message on outgoing channels…

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q is done

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q

zs

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q

v

z

x

u

s

Chandy/Lamport

p

qr

s

t

u

v

w

xy

z

A network

q

v

w

z

x

u

s

y

r

Chandy/Lamport

pq

r

s

t

u

v

w

xy

z

A snapshot of a network

q

x

u

s

v

r

t

w

p

y

z

Done!

Correctness

• Consistency

• Termination

Consistency

• m is recorded iff so is send(m):– sender’s state recording and marker

sending are done atomically.• m is not recorded more than once:

– if channel is recorded before receiver, it will be empty.

– if channel is recorded after receiver, none of the in-channel messages will be recorded as the receiver’s state.

Correctness:

Termination

• Assumptions:– L1: no marker remains forever in a channel.– L2: processes’ states are recorded in finite time.

• Every process either spontaneously records its state, or there is a path from such a process.

• Every channel is flushed by a marker after the sender records its state.

Correctness:

Remained Issues

• Property of recorded state:Si --> S* --> Sf

• Stable detection:– Stable property:

• y(Si) definite• definite y(Sf)

– Algorithm:begin

record a global state S*;definite := y(S*)

end.

Beyond…

• Channels’ assumptions:–Singly directed–FIFO–Asynchronous–Error free–Infinite buffer

Non-FIFO

• What is FIFO for?–Separate messages between

before-snapshot and after-snapshot.

• A snapshot counter piggybacked on messages would do just fine!

Beyond:

Beyond…

• Channels’ assumptions:–Singly directed–FIFO–Asynchronous–Error free–Infinite buffer

Messages can be corrupted/duplicated

Messages can be dropped

Unreliable channels

• How to deal with corruption?– Checksum/ECC; reduced to drop.

• How to deal with duplication?– Message ID

• How to deal with dropping?– Channel states are not needed

anymore.– Markers indicate completion.

Beyond:

Even More Aggressive…

• Don’t want to piggyback!• Step 1: no piggybacking:

– Block all messages sent after recording local state and before receiving marker from all neighbors.

• Step 2: no blocking, min piggybacking– Blocked messages are sent with

piggybacked snapshot info.

Beyond:

Conclusion

• Two influential papers.• Much work built upon these results.• Can be improved significantly when

being adopted to particular systems.

• Additional comments/suggestions?