![Page 1: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/1.jpg)
Distributed Network AlgorithmsWelcome to the GIAN Course!
Partha Sarathi Mandal
IIT Guwahati, India
Stefan Schmid
Aalborg University, Denmark
& TU Berlin, Germany
![Page 2: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/2.jpg)
Distributed Network AlgorithmsWelcome to the GIAN Course!
Partha Sarathi Mandal
IIT Guwahati, India
Here the next 10
days. Let‘s meet!
Goal: Get a feeling of the field and
techniques. Essentially: Classic TCS
problems from a distributed lens,
also an active research area.
Stefan Schmid
Aalborg University, Denmark
& TU Berlin, Germany
![Page 3: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/3.jpg)
3
What is a distributed system? A quote.
‘‘You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done.” (Leslie Lamport)
![Page 4: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/4.jpg)
4
Why distributed systems?Heard of Moore‘s law?
We start observing the limits of extent at which
we can increase transistor density and clock
speed. One solution: compensate by
combining multiple, distributed components!
![Page 5: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/5.jpg)
❏ Traditionally: astronomers and
physicists
❏ CERN LHC: 15 petabytes / year
❏ More recently: biologists and life
scientists join big data club
❏ Everything from ‘why coastal
algae bloom to what microbed
dwell where in the human body’
Science
❏ Facebook, Twitter, LinkedIn
❏ Operate pipelines of 100s
terabytes / day
❏ Logging user interactions,
monitoring compute
infrastructures, tracking
business-critical functions
❏ More and more machine-
driven data collection
❏ Advent of Internet-of-Things,
smart devices, …
Business
Another reason: Big Data
Roughly 1 TB/day!
Big data is relevant not only for
researchers, but also for businesses,
decision makers, journalists, crowd
workers, and “everyday consumers”.Processing this data requires
distributed systems and algorithms!
![Page 6: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/6.jpg)
6
What makes distributed systems challenging? An analogy.
Distributed systems are large beasts: and each individual
or constituting component can often only see a small part.
Often hard or impossible to obtain a global snapshot!
![Page 7: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/7.jpg)
7
What makes distributed systems challenging? An analogy.
❏ Human mind not used to think about concurrent processes:
In my opinion: better to go from theory to practice than vice versa!
Distributed systems are large beasts: and each individual
or constituting component can often only see a small part.
Often hard or impossible to obtain a global snapshot!
![Page 8: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/8.jpg)
Dis
trib
ute
d A
lgori
thm
s
How to perform computations in a distributed system?
Dynamic Resource Allocation
Allows to partition the work
among distributed entities
(e.g. processors, robots,
etc.). Goal: minimize
coordination overhead!
A new trend: virtualized
infrastructure itself can be
adapted (e.g., adding virtual
machines or bandwidth for
communication)!
![Page 9: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/9.jpg)
Dis
trib
ute
d A
lgori
thm
s
How to perform computations in a distributed system?
Dynamic Resource Allocation
Allows to partition the work
among distributed entities
(e.g. processors, robots,
etc.). Goal: minimize
coordination overhead!
A new trend: virtualized
infrastructure itself can be
adapted (e.g., adding virtual
machines or bandwidth for
communication)!
This course considers
both dimensions!
![Page 10: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/10.jpg)
Confluence: innovation!
Programmability and virtualization
Algorithms
”We are at interesting inflection point!”
Keynote by George Varghese
Modern Distributed Systems: Virtualized & Programmable
![Page 11: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/11.jpg)
Confluence: innovation!
Programmability and virtualization
Algorithms
”We are at interesting inflection point!”
Keynote by George Varghese
Modern Distributed Systems: Virtualized & Programmable
Flexibilities but also
challenges!
![Page 12: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/12.jpg)
App 1: Mobile Service App 2: Big Data Analytics
Realization and Embedding
Virtualization and Isolation
Quality-of-Service
& Resource
Requirements
Computational
& Storage
Requirements
Predictable Performance in Virtualized Environments?
![Page 13: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/13.jpg)
13
Spectrum of Distributed (Computer) Systems
small/synchronous/... wide-area/decoupled/...
E.g., tiny graphical processing
units (GPUs) and specialized
devices, in which large arrays of
simple processors work in lock-
step, PRAM, ...
cache
•BusBus
shared memory
cachecache
Multi-threaded + multi-core
servers/desktops with shared
memory for communication.
Loosely-coupled peer-to-peer
systems with message passing
communication
Even your laptop is a distributed
system (multiple cores etc.)!
![Page 14: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/14.jpg)
14
Spectrum of Distributed (Computer) Systems
small/synchronous/... wide-area/decoupled/...
E.g., tiny graphical processing
units (GPUs) and specialized
devices, in which large arrays of
simple processors work in lock-
step (“Gleichschritt”).
cache
•BusBus
shared memory
cachecache
Multi-threaded + multi-core
servers/desktops with shared
memory for communication.
Loosely-coupled peer-to-peer
systems with message passing
communication
Many commonalities! Communication, concurrent, failures, …
![Page 15: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/15.jpg)
15
Not a Panacea: Limitations of Distributed Computing
E.g., tiny graphical processing
units (GPUs) and specialized
devices, in which large arrays of
simple processors work in lock-
step (“Gleichschritt”).
Multi-threaded + multi-core
servers/desktops with shared
memory for communication.
Shared MemoryAmdahl’s Law
Some problems can be very hard in distributed setting (e.g., symmetry breaking)
And not everything is embarrassingly parallel: inherent speedup limits
S = speedup
p = fraction of work that can be done in parallel
n = number of processors
sequential part parallel part
Take-home message:
try to minimize sequential part!
You can only speedup the
parallel work share!
![Page 16: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/16.jpg)
16
Speedup Limits
Example: 5 friends want to paint new apartment, with 5 rooms
Speed-up 5 people vs 1?
5 times faster than alone!
![Page 17: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/17.jpg)
17
Speedup Limits
Example: 5 friends want to paint new apartment, with 5 rooms
What if last room is twice as large?
Assume there is space for at most one painter per room!
Speed-up 5 people vs 1?
5 times faster than alone!
![Page 18: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/18.jpg)
18
Speedup Limits
Example: 5 friends want to paint new apartment, with 5 rooms
What if last room is twice as large?
Speed-up 5 people vs 1?
5 times faster than alone!
Assume there is space for at most one painter per room!
Only p=5/6 of the work can be performed in parallel, rest sequential.
Parallel execution time = 5/(5∙6)+1/6 = 1/6+1/6 = 2/6 = 1/3. Only 3 times faster!
Would be better to parallelize painting of last room also!
![Page 19: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/19.jpg)
19
This Course
Distributed
SystemsNetworking
(Algorithms
Course
![Page 20: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/20.jpg)
20
This Course
Distributed
SystemsNetworking
(Algorithms
Course
Example: An efficient routing
algorithm lies at the heart of
any network. But how to
perform local routing, i.e.,
without global knowledge?
![Page 21: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/21.jpg)
21
This Course
Distributed
SystemsNetworking
(Algorithms
Course
Recall the elephant analogy: in a
distributed setting, a local node may only
have a partial view of the network! How to
do routing in this case?!
Example: An efficient routing
algorithm lies at the heart of
any network. But how to
perform local routing, i.e.,
without global knowledge?
![Page 22: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/22.jpg)
22
Example: Routing With A Local View
Example: How to get out of this maze in the dark?
It is dark and you do not have a torch!
![Page 23: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/23.jpg)
23
Famous trick:
Left-hand rule!
Example: Routing With A Local View
Example: How to get out of this maze in the dark?
![Page 24: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/24.jpg)
24
Example: Routing With A Local View
Nice: The algorithm does not
need much memory!
Famous trick:
Left-hand rule!
Example: How to get out of this maze in the dark?
![Page 25: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/25.jpg)
25
Argh......
Example: How to get out of the maze here?!
Example: Routing With A Local View
Homework: Local routing algorithm
which can deal with mazes containing
such a room?
![Page 26: Distributed Network Algorithms · 2016-06-27 · Distributed Network Algorithms Welcome to the GIAN Course! Partha Sarathi Mandal IIT Guwahati, India Here the next 10 days. Let‘s](https://reader034.vdocuments.site/reader034/viewer/2022042417/5f338a878603323f1245f160/html5/thumbnails/26.jpg)
26
Literature For This Course
David Peleg “Distributed Computing”
Newman “Networks”Upfal+Mitzenmacher “Probability
and Computing”
Attiya+Welch: “Distributed Computing”
+ Lecture Notes
+ Slides