1 4. synchronization clock synchronization logical clocks and global state election algorithms ...

1

4. Synchronization

Clock synchronization

Logical clocks and global state

Election Algorithms

Mutual exclusion

Distributed Transactions

2

Learning Objectives

To study how processes cooperate and synchronize with one another.

To examine the issues of synchronization based on real time, and to understand the key features of Cristian's algorithm, the Berkeley algorithm and the Network Time Protocol.

To understand the utility of logical clocks and global state, the rules for updating them and their limitations.

To be familiar with various election algorithms in DS to appoint a coordinator;

To study and understand the mutual exclusion and transactions methods in DS.

3

Clock Synchronization: Introduction

• We need to measure time accurately:• to know the time an event occurred at a computer• to do this we need to synchronize its clock with an authoritative

external clock

• Algorithms for clock synchronization useful for• concurrency control based on timestamp ordering• authenticity of requests e.g. in Kerberos (in late lecture)

• There is no global clock in a distributed system

• Logical time is an alternative• It gives ordering of events - also useful for consistency of replicated

data

4

Clock Synchronization: Sources of Accurate Timing Signals

International Atomic Time is based on very accurate physical clocks (drift rate 10-13).

Coordinated Universal Time (UTC) is an international standard for time keeping.

It is broadcast from radio stations on land and satellite (e.g. GPS).

Computers with receivers can synchronize their clocks with these timing signals.

Signals from land-based stations are accurate to about 0.1-10 millisecond.

Signals from GPS are accurate to about 1 microsecond.

5

Clock Synchronization

Difficulties in clock synchronization in DS– Not all sites have direct access to accurate time source such as a

GPS receivers due to the cost.– Sites has to synchronize their local clocks with those have more

accurate time. – Synchronization needs to be done periodically due to clock drift: rate

is change in the offset (difference in reading) between the clock and a normal perfect reference clock per unit of time measured (about 10-6 or 1 sec. in 11.6 days).

If a site’s clock is ahead of the reference (time server) to which it synchronizes to, it cannot be simply set back (why?).

This may have effect that a event that has happened but its time-stamp is in the future (Example: compiler and UNIX make).

6

Clock Synchronization

For process pi

– Ci(t) – software clock

– Hi(t) – Hardware clock, I.e., the time given by hardware clock

Thus: Ci(t) = H(t)+

Various notions of correctness for clocks have been suggested.

It is common to define a hardware clock H to be correct if its drifts fall within a known bound > 0 (e.g. for real-time t’ and t, t’>t).

(1- )(t’-t) ≤ H(t’) – H(t) ≤ (1+ )(t’-t)

7

External and Internal Clock Synchronization

External synchronization– A computer’s clock Ci is synchronized with an external authoritative

time source S, so that:– |S(t) - Ci(t)| < D for i = 1, 2, … N over an interval, I of real time – The clocks Ci are accurate to within the bound D.

Internal synchronization– The clocks of a pair of computers are synchronized with one another

so that:– | Ci(t) - Cj(t)| < D for i = 1, 2, … N over an interval, I of real time – The clocks Ci and Cj agree within the bound D.

Internally synchronized clocks are not necessarily externally synchronized, as they may drift collectively.

if the set of processes P is synchronized externally within a bound D, it is also internally synchronized within bound 2D.

8

Clock Synchronization: Cristian Algorithm

Client C requests time in Request message sent to time server S, then it receives time value t in message CUTC from S. t is the current time in S before sending back CUTC to C.

Let Ttrans be time taken for CUTC message from S to C, then C should set its time to t + Ttrans .

Ttrans can be variant. You may say: Ttrans = min+x, x ≥ 0 and min = time of message transmission if no interference of other process and no other messages, but x is unknown!

Solution (See the next slide): (1) Record total round trip time as Tround (between sending Request and receiving CUTC,, i.e., T1-T0) (2) If the time received in Request message is t, then C can estimate its time as: t + Tround/2.

9

Cristian's Algorithm

Getting the current time from a time server.

10

Clock Synchronization: Cristian Algorithm

Keep in mind: C’s time is t + Tround/2.

Let the minimum transmission time be min, and min is known.

Therefore, t’ (the time in S when C receives CUTC ) lies in the range [t+min, t+Tround min]

t+ Tround min (t+min) = Tround 2min

The accuracy for this time adjustment is

(Tround/2min)

Problem: single-server failure.

Solution: providing group synchronization time servers.

11

Clock Synchronization: Berkeley Algorithm

In Berkeley UNIX, one of a group sites (computers) is chosen as coordinator, (master or time server).

(1) The master (actually a time daemon in the master) periodically polls the other sites (slaves) to ask what time it is there, by telling its own time (the time in master) to these slaves.

(2) The slaves response with how far ahead or behind the time daemon they are, that is, the difference between its time and the time in master.

(3) Based on the replies from the slaves, the master averages the time obtained (including its own).

(4) The master sends time rate adjust value (+ or -) to slaves, requesting them to adjust their time rate.

Question: How the Cristian’s algorithm is applied to the above Berkeley algorithm?

12

The Berkeley Algorithm: An Example

a) The time daemon asks all the other machines for their clock values

b) The machines answer

c) The time daemon tells everyone how to adjust their clock

13

Clock Synchronization: Network Time Protocol (NTP)

NTP is used in the Internet. Time servers are arranged in levels called strata. NTP intends to:– provide a service enabling clients in Internet to be

synchronized accurately to UTC.– provide reliable service to losses of connectivity.– enable clients to resynchronize sufficiently and frequently.– provide protection against interference with the time

service.

14

Clock Synchronization: Network Time Protocol

Synchronization clock: Network Time Protocol (NTP)

Stratum 1: Primary server (PS) sync directly with UTC sources.

Stratum 2: Secondary servers (SS) sync directly to PS.

Stratum 3: Lowest servers (LS) execute in user sites sync with SS.

Accuracy: the number of levers (strata).

1

2

3

2

3 3

15

Clock Synchronization: Network Time Protocol

Multiple servers provide for redundancies and hence availability of time sources and fault-tolerance.

NTP synchronizes with one another in one of three modes:– Multicast: Used on LAN. One or more servers periodically

broadcast the time to servers running in workstations which set their clocks assuming a small delay. Accuracy is low.

– Procedure-call: Similar to Cristian’s method. Workstations ask the server the time. Accuracy is higher than multicast.

– Symmetric mode: Used by high level to achieve highest accuracy. A pair of servers exchange messages bearing timing information.

16

Logical Time and Logical Clock

For many purposes, it is sufficient that all machines agree on the same time. It is not essential that this time also agrees with the real time as announced on the radio every hour.

What usually matters is not that all processes agree on exactly what time it is, but rather that they agree on the order in which events occur.

17

Lamport causal ordering x-P->y means x occurs before y in process P. – happened-before “->”– HB1: If for process P: x-P->y, then x-> y.– HB2: For the same message m, send(m)->receive(m)– HB3: If x, y, and z are events, x->y, y->z then x->z.p1

p2

p3

a bm1

m2

c d

e f


18

Logical clock can capture happened-before ordering by logical clock.

A logical clock need not bear particularly relationship with physical clock.

Each process just keeps its own clock Cp. It is used to

timestamp events. Use Cp(a) and Cp(b) to timestamp events a and b happen

in process p and C(b) for event b at whatever process it occurred.


19

Logical clock update and transmission rules:– LC1:

Cp is incremented before each event is issued at p: Cp:= Cp+1;

– LC2:

a) When p sends a message m, it piggybacks on m the value t=Cp.

b) On receiving (m, t), a process q computes Cq:=max(Cq, t) and then applies LC1 before time-stamping the event receive(m).


20

Totally ordered logical clocks– for pair of distinct events and we take process id into

consideration.

– If a is an event occurring at pa with local timestamp Ta, and b is an event occurring at pb with local timestamp Tb.

– Define the global logical timestamp for those events as (Ta, a) and (Tb, b), respectively.

– (Ta, a) < (Tb, b) if and only if either

Ta < Tb or (Ta =Tb and pa < pb)


21

Example: Totally-Ordered Multicasting

Multicast: A message is sent to multiple receivers.

Totally-ordered multicast: all multicast messages are delivered in the same order to each receiver.

For example, to improve query performance, a bank may place copies of an account database in two different cities, say A and B.

* A customer in B wants to add $100 to his account that currently contains $1,000 (update 1);

* At the same time, a bank employee in A initiates an update by which the customer’s account is to increased with 1% interest (update 2).

22


Both updates should be carried out at both copies (in A and B) of the database.

* If update 2 is performed before update 1 in A, the A’ database records $1,110;

* If update 1 is performed before update 2 in B, the B’ database records $1,111;

An inconsistence occurs if the two updates are not performed in the same order!

Solution: Using Lamport timestamp (with logical time) to implement totally-ordered multicast (for update messages), so that update operations are performed in the same order at each copy.

23


Updating a replicated database and leaving it in an inconsistent state.

24

Totally-Ordered Multicasting Using Logical Time

For a group of processes multicasting messages to each other, we assume:

* Each message is timestamped with current logical time of its sender;

* The sender is also a receiver of its own sending message;

* The messages from the same sender are received in the order they were sent, and no messages are lost.

When process receives a message, it is put into a local queue, ordered according to its timestamp. The receiver multicasts an acknowledgement to the other processes.

25

Totally-Ordered Multicasting Using Logical Time

A process can deliver a queued message to the application it is running only when the message is at the head of the queue and has been acknowledged by each other process.

All processes will eventually have the same copy of the local queue by using Lamport’s timestamp..

Lamport’s clock ensures that no two messages have the same timestamp, and timestamps reflect a consistent global order of events.

Therefore, all message are delivered in the same order everywhere. That is, we have established totally-ordered multicasting.

26

Global State

It is often useful to know the global state in which a distributed system is currently residing, e.g., to detect the termination of a distributed computation, or deadlock etc.

Global state of a DS: The local state of each process, and the messages that are currently in transit.

Distributed snapshot: reflects a (consistent global) state in which the DS might have been. It is not desirable that a snapshot contains the recording of messages that have been received but never sent.

Cut: A graphical representation of global state, as shown in the next slide.

27

Global State (1)

a) A consistent cut

b) An inconsistent cut

28

Global State: Algorithm for Taking Snapshot I

Assume that the DS can be represented as a collection of processes connected to each other through unidirectional point-to-point communication channels (e.g., TCP connection).

Any initiating process, say P, may start by recording its own local state; then it sends a marker along each of its outgoing channels;

When a process Q receives a marker from an incoming channel C,

* If Q has not saved its local state, it first saves the state, then sends a marker along each of its own outgoing channels.

29

Global State: Algorithm for Taking Snapshot II

* If Q has recorded the local state, it records the state of channel C: the sequence of messages that have been received by Q since the last time Q recorded its local state, and before it received the marker.

When a process has received a marker along each of its incoming channels, and processed each one, its recorded local state and state of each channel are collected and sent to process P.

Because any process can initiate the algorithm, several snapshots can be constructed at the same time. To identify different processes of snapshot construction, a marker can be tagged with identifier (even version) of process that initiates the snapshot.

30

Global State (2)

a) Organization of a process and channels for a distributed snapshot

31

Global State (3)

b) Process Q receives a marker for the first time and records its local state

c) Q records all incoming message

d) Q receives a marker for its incoming channel and finishes recording the state of the incoming channel

32

Global State: Algorithm for Termination Detection I

The termination of a distributed computation can be detected using the algorithm of taking a snapshot.

A snapshot may show a global state in which message are still in transit, which means the distributed computation has not terminated.

The termination detection algorithm is a simple modification of the snapshot algorithm. Assume that a process Q receives the marker from process P, requesting a snapshot for the first time, then P is said to be the predecessor of Q.

When a process Q finishes its part of the snapshot, it either returns DONE or CONTINUE message to its predecessor:

33

Global State: Algorithm for Termination Detection II

* DONE: all of Q’s successors have returned a DONE message; and Q has not received any message between the point it recorded its state and the point it had received the marker along each of its incoming channel;

* CONTINUE: In all other cases.

When the original initiator of the snapshot, say P, receives only the DONE messages, it concludes that the distributed computation has terminated; Otherwise, P initiates another snapshot, and continues to do so until only DONE message are eventually returned.

34

Election Algorithms

Many distributed applications require one processor act coordinator, such as Berkeley algorithm for clock synchronization, or otherwise perform some special role.

If all processes are exactly the same, with no distinguishing characteristics, there is no way to select one of them to be special.

Assume that each processor has a unique number. The process with the highest number often could be elected.

The goal of an election algorithm is to ensure that when election starts, it concludes with all processes agreeing on who the new coordinator is to be.

35

Election Algorithms

Need to find one process that is the coordinator Assume

– Each process has a unique identifier• network address for example

– One process per machine

– Every process knows the process number of every other process

– Processes don’t know which processes are down and which ones are still running

End result of the algorithm: all processes agree on who is the new coordinator/leader

Bully algorithm & Ring Algorithm

36

Bully Algorithm (Garcia-Molina) A process notices that coordinator is not responding

– it starts an election (any process can start one) Election algorithm

– P sends an ELECTION message to processes with higher numbers– If no one responds, P wins the election– If some process with higher process number responds

• P’s job is done, that process takes over • the receiver sends an OK message to P• receiver starts an election process

Eventually all processes give up, except one This process sends out a message saying that it is the new

“COORDINATOR” A process that was down, when it comes back up starts a new election of

its own

37

The Bully Algorithm (1)

The bully election algorithm; pcs 7 as coordinator=> crashed

(a) Process 4 is the first on noticed the crashed >> send ELECTION process to 5, 6 and 7 (higher pcs)

(b) Process 5 and 6 respond, telling 4 to stop

(c) Now 5 and 6 each hold an election

38

The Bully Algorithm (2)

d) Process 6 tells 5 to stop

e) Process 6 wins and tells everyone

39

The Bully Algorithm (Summarize)

It is assumed that every process knows the processor numbers of every other process, but does not know which one is currently up or down.

When any process P notices that the current coordinator is no longer responding the request, it initiates an election:

1. P sends an ELECTION message to all processes with higher numbers;

2. If no one responds, P wins the lection and becomes coordinator;

3. If one of the highest-ups answers, it takes over. P’s job is done.

40

The Bully Algorithm (Summarize)

When a process gets a message form one of its lower-numbered processes, it sends an OK message back to the sender to indicate that it is alive and will take over, and it then holds an election (unless it is already doing so).

Eventually, all processes but one give up, and that one is the new coordinator.

The new coordinator announces its victory by sending all processes a message telling them that starting immediately it is the new coordinator.

If a process that was previously down comes back up, it holds an election, and may win the election if it happens to be highest number process among all the running processes.

41

Ring Algorithm (1)

Does NOT use a token Assume

– processes are ordered– each process knows its successor

• and the successor’s successor, and so on (needed in case of failures)

Process P detects that the coordinator is dead– sends an ELECTION message to its successor– includes its process number in the message– each process that receives it

• adds its own process number and then forwards it to its successor

– eventually it gets back that message• now what does it do?

42

Ring Algorithm (2)

The process that initiated it, then sends out a message saying “COORDINATOR”– the process with highest number in list is the leader– when this comes back, then process P deletes it

43

A Ring Algorithm(1)

Election algorithm using a ring.

We start with 6 processes, connected in a logical ring.

Process 6 is the leader, as it has the highest number.

44

A Ring Algorithm(2)

Process 6 fails.

45

A Ring Algorithm(3)

Process 3 notices that Process 6 does not respond

So it starts an election, sending a message containing its id to the next node in the ring.

46

A Ring Algorithm(4)

Process 5 passes the message on, adding its own id to the message

47

A Ring Algorithm(5)

Process 0 passes the message on, adding its own id to the message.

48

A Ring Algorithm(6)

Process 1 passes the message on, adding its own id to the message.

49

A Ring Algorithm(7)

Process 4 passes the message on, adding its own id to the message

50

A Ring Algorithm(8)

When Process 3 receives the message back, it knows the message has gone around the ring, as its own id is in the list.

Picking the highest id in the list, it starts the coordinator message "5 is the leader" around the ring

51

A Ring Algorithm(9)

Process 5 passes on the coordinator message

52

A Ring Algorithm(10)

Process 0 passes on the coordinator message.

53


Process 1 passes on the coordinator message.

54


Process 4 passes on the coordinator message

Process 3 receives the coordinator message, and stops it.

55

Ring Algorithm (Summarize)

Assume that the processes are physically or logically ordered, so that each process knows who its successor is.

When any process notices that the coordinator is not functioning, it builds an ELECTION message containing its process number and sends the message to its successor. If the successor is down, the sender skips over it and goes to the next number along the ring, or the next after that, until a running process is located.

At each step, the sender adds its own process number (as a candidate for coordinator) to the list in the message.

Eventually, the message gets back to the process that started it all. The process can recognize this event.

56

Ring Algorithm (Summarize)

The process changes the message type to COORDINATOR and circulates it once again, to inform everyone else who the new coordinator is (the list number with the highest number) and who the numbers of the new ring are.

When the COORDINATOR message has circulated once, it is removed and everyone goes back to work.

57

Summary I

There is no notion of a globally shared clock. Accurate timekeeping is important for distributed systems.

There are various ways to synchronize clocks in a DS, essentially all are based on exchanging clock values. The accuracy of clock synchronization depends on the variation in communication delays and how they are dealt with.

For ordering of an arbitrary pair of events at different computers, clock synchronization is not always practical.

The happened-before relation is a partial order on events that reflects a flow of information between them.

58

Summary II

Lamport clocks are counters that are updated according to the happened-before relationship between events.

There is no globally shared memory in DS. It is often hard to determine the system’s current state, which can be done by synchronizing all processes so that each collects its local state and the messages currently in transit (using snapshot).

Synchronization between processes often requires that one process acts as a coordinator. A decision on who is going to be the new coordinator is taken by means of election algorithms. These algorithms are primarily used in cases where the coordinator can crash.

59

Tutorial

Q1. Discuss the features and applications of Cristian algorithm, Berkeley algorithm, and network time protocol in clock synchronization.

Q2. In the example of the Berkeley algorithm in the lecture (see the next slide), the effect of transmission delay of the message is not considered. How the Cristian’s algorithm is applied to Berkeley method to improve its accuracy?

60

Tutorial: An Example of the Berkeley Algorithm

a) The time daemon asks all the other machines for their clock values

b) The machines answer

c) The time daemon tells everyone how to adjust their clock

61

Tutorial

Q3. (Cristian’s algorithm) A client attempts to synchronize with a time server. It records the following round-trip times and timestamps returned by the server:

Round trip (ms) Time (hr:min:sec)

22 10:54:23:674

25 10:54:25:450

20 10:54:28:342

Which of these times should it use to set its clock? To what time should it set? Estimate the accuracy of the setting with respect to the server’s clock. If it is known that the time between sending and receiving a message in the system concerned is at least 8 ms, do your answer change?

62

Tutorial

Q4. Given the event sequences in the next slide. Suppose each logical clock starts at 0. Show how Lamport’s clock mechanism adjusts the local clocks. Also, show how the totally ordered logical clock can be used so that each event has a unique timestamp using logical clock .

63

Tutorial: Event Sequences in Three Processes

p1

a3 b3 c3 d3 e3 f3

P2

P3

a1 b1 c1 d1 e1

a2 b2 c2

64

Tutorial

Q5. Suppose that there are three processes P1, P2, and P3. Each process multicasts two messages to the other processes, including itself as follows:

* P1 multicasts message m1 with timestamp (9, 1) and message m2 with timestamp (11, 1);

* P2 multicasts message m3 with timestamp (9, 2) and message m4 with timestamp (10, 2);

* P3 multicasts message m5 with timestamp (10, 3) and message m6 with timestamp (12, 3).

How the totally-ordered multicasting is achieved (under the same assumptions in Lecture Note 5), and what is the order that all the 6 messages are received in each process?

1 4. synchronization clock synchronization logical clocks and global state election algorithms ...

Documents