![Page 1: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/1.jpg)
1Distributed Systems ID2201
Distributed Systems ID2201
coordination and agreement IJohan Montelius
![Page 2: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/2.jpg)
2Distributed Systems ID2201
Coordination
• Coordinating several threads in one node is a problem, coordination in a network is of course worse:– no fixed coordinator– no shared memory– failure of nodes and networks
• Coordination is often the problem of:– deciding who is to decide – knowing who is alive.
![Page 3: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/3.jpg)
3Distributed Systems ID2201
Fundamental models
• Interaction model:– Is the system asynchronous or
synchronous?– Can we assume a node has crashed if it
does not reply?• Failure model:
– Will nodes crash?– Will crash nodes return to life?– Is crashing the only failure?
![Page 4: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/4.jpg)
4Distributed Systems ID2201
Distributed algorithms
• We will look at some distributed algorithms and assume: – that nodes are correct– that messages are delivered
![Page 5: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/5.jpg)
5Distributed Systems ID2201
Three sides of the same coin
• Mutual exclusion– Decide who is to enter a critical
section.• Leader election
– Decide who is to be the new leader.• Atomic multicast
Which messages, and in which order, should be deliver a message.
![Page 6: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/6.jpg)
6Distributed Systems ID2201
Distributed mutual exclusion
• Requirements– Safety: at most one process may be in
critical section at a time– Liveness: starvation free, deadlock free– Ordering: allowed to enter in request
happened-before order
![Page 7: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/7.jpg)
7Distributed Systems ID2201
Evaluation
• Number of messages needed.• Client delay:
– worst, – mean or, average time to enter critical
section• Synchronization delay: how long time
between exit and enter.
![Page 8: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/8.jpg)
8Distributed Systems ID2201
Central service algorithm
• Requirements?– safety– liveness – ordering req
release
grant
queue
![Page 9: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/9.jpg)
9Distributed Systems ID2201
Ordering - what is a request
A
B
Server
![Page 10: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/10.jpg)
10Distributed Systems ID2201
Performance
• messages– enter: request, grant– exit: release
• client delay – enter: message round trip plus waiting
in queue– exit: constant (asynchronous message)
• synchronization delay– round trip: release - grant
![Page 11: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/11.jpg)
11Distributed Systems ID2201
Ring-based algorithm
• Requirements– safety– liveness– ordering
![Page 12: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/12.jpg)
12Distributed Systems ID2201
Ring-based algorithm
• Performance– messages – client delay– synchronization delay
![Page 13: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/13.jpg)
13Distributed Systems ID2201
Distributed algorithm
• Send request to all peers.
• When all peers have acknowledged the request, enter the critical section.
• What could go wrong?
![Page 14: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/14.jpg)
14Distributed Systems ID2201
Distributed algorithm
• Break deadlock– introduce priority
• Fairness– Ricart and Agrawala
![Page 15: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/15.jpg)
15Distributed Systems ID2201
Ricart and Agrawala• Enter:
– enter state waiting and broadcast a request {T,i} containing a Lamport time stamp T and process id I to all peers
– wait for replies from all peers– enter state held
• Receiving a request {R,j}:– if held or (waiting and {T,i} < {R,j}) then queue
request, else reply ok
• Exit: – reply to all queued requests
![Page 16: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/16.jpg)
16Distributed Systems ID2201
Ricart and Agrawala• Requirements
– safety, liveness, ordering• Efficiency
– messages– client delay– synchronization delay
![Page 17: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/17.jpg)
17Distributed Systems ID2201
Maekawa's voting
• Why have permission from all peers, it's sufficient to have votes from a subset S if no one can enter with the votes from the complement of S.
• The subset S is called a quorum.
![Page 18: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/18.jpg)
18Distributed Systems ID2201
Maekawa's voting
• Requirements– safety– liveness– ordering
![Page 19: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/19.jpg)
19Distributed Systems ID2201
Maekawa's voting
• Efficiency– messages– client delay– synchronization delay
![Page 20: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/20.jpg)
20Distributed Systems ID2201
Election
• Many algorithms require a leader but if no node is assigned to be the leader one has to be elected.
• Assumptions:– any node can call an election, but it can
only call one at a time– a node is either participant or
non-participant– nodes have identifiers that are ordered
![Page 21: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/21.jpg)
21Distributed Systems ID2201
Election
• Requirements– safety: a participant is either non-decided or
decided with P, a unique non crashed node
– liveness: all nodes eventually participate and decide on a elected node
• Efficiency– number of messages– turnaround time: delay from call to close
![Page 22: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/22.jpg)
22Distributed Systems ID2201
Ring-based election12
3
18
9
11
23
14
v-18
12
3
18
9
11
23
14e-23
12
3
18
9
11
23
14
![Page 23: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/23.jpg)
23Distributed Systems ID2201
Ring-based election12
3
18
9
11
23
14
v-18
v-23
12
3
18
9
11
23
14
v-18
v-23
12
3
18
9
11
23
14
v-18
v-2312
3
18
9
11
23
14
v-23
![Page 24: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/24.jpg)
24Distributed Systems ID2201
Ring-based election
• Requirements– safety– liveness
• Efficiency– messages: best case, worst case?– turnaround:
![Page 25: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/25.jpg)
25Distributed Systems ID2201
Multicast communication
• Multicast: – Sending a message to a specified
group of n nodes.• Atomic multicast:
– All nodes see the same messages in the same order.
![Page 26: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/26.jpg)
26Distributed Systems ID2201
Model
send
deliver
receive
deliver
receive
group
m-cast
![Page 27: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/27.jpg)
27Distributed Systems ID2201
Requirements
• Integrity– a process delivers a message at most
once and only deliver messages that have been sent
• Validity– if a process multicast m then it will also
eventually deliver m• Agreement
– if a process delivers m then all processes in the group eventually delivers m
![Page 28: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/28.jpg)
28Distributed Systems ID2201
Basic multicast
• To b-multicast a message m: – send m to each process p
• If m is received:– b-deliver m
• What was the problem?
![Page 29: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/29.jpg)
29Distributed Systems ID2201
Ordered multicast
• The problem with the basic multicast is that multicast messages might arrive in different order at different nodes.
• Requirements:– FIFO order: delivered in order as sent by the
sender– Causal order: delivered in order as happened
before sent order– Total order: delivered in same order by all
processes
![Page 30: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/30.jpg)
30Distributed Systems ID2201
Sequencer
m-cast m
m
message queue
![Page 31: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/31.jpg)
31Distributed Systems ID2201
Distributed - ISIS
• Multicast a message and request a sequence number.
• When receiving a message, propose a sequence number (including process id) and place in an ordered hold-back queue.
• After collecting all proposals, select the highest and multicast agreement.
• When receiving agreement tag message as agreed and reorder hold-back queue.
• If first message in queue is decided then deliver.
![Page 32: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/32.jpg)
32Distributed Systems ID2201
Distributed - ISIS
{request, m1, ....}
![Page 33: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/33.jpg)
33Distributed Systems ID2201
Distributed - ISIS
{proposal, m1, 3}
{proposal, m1, 2}
{proposal, m1, 1}
![Page 34: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/34.jpg)
34Distributed Systems ID2201
Distributed - ISIS
{assign, m1, 3}
![Page 35: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/35.jpg)
35Distributed Systems ID2201
the hold-back queue
{m1, proposed <2,i>}
{m2, agreed <3,e>}
{m3, agreed <3,k>}
{m4, proposed <4,i>}
{m5, proposed <5,i>}
deliver
What happened here?
What will the agreed sequence number be?
![Page 36: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/36.jpg)
36Distributed Systems ID2201
Causal ordering
• How can we implement casual ordering?– multicast vector clock holds number of
multicast operations– tag each multicast message with multicast
clock– hold b-delivered messages until clock of
message is less (modulo sender) than own current message clock
– update own message clock
• Only multicasted messages are counted.
![Page 37: Distributed Systems · Distributed Systems ID2201 Election •Many algorithms require a leader but if no node is assigned to be the leader one has to be elected. •Assumptions: –any](https://reader036.vdocuments.site/reader036/viewer/2022071023/5fd855331025422c071b5512/html5/thumbnails/37.jpg)
37Distributed Systems ID2201
Summary
• Coordination in distributed systems is problematic.
• Three sides of the same coin:– mutual exclusion– leader election– atomic multicast
• If nodes fail– next lecture