Distributed computing
• Key idea– Buying 1000 machines of speed x is significantly cheaper
than buying one machine of speed 1000x– No one person has to buy all 1000 machines: A lot of
computational, communication and storage resources already in place and can be harvested for bigger things
• Key challenge– Making the machines work together for effective speedup.
Communication between machines is a key challenge.
• Approaches– Find problems that can be distributed easily
Distributed problems• Problems that can use decentralized computing
– Weather prediction• Weather in a location is most affected by weather nearby
– Movie generation• Individual frames can be generated separately
– Google search engine• 10,000s PC’s. all of them cheap, many of them identical• Can answer over 100,000,000 queries per day in ½ sec or less each
– Looking for the origin of the universe• Can be localized like weather prediction
– File swapping and access (distributed storage)– Looking for extra terrestrial intelligence– Content caching and distribution
Distributed computers
• Scales of distributed computing– Cluster-in-a-room hundreds of machines
• All dedicated to the task
– PCs on a campus thousands of machines• Using spare cycles
– SETI cluster millions of machines• Screen saver situation
Cluster in a Room
• Machines are dedicated to the network
• All machines run similar software
• Problem is divided into pieces– Each piece is assigned to a machine in the cluster
• Problem pieces should be loosely linked– Computation is faster than communication
PCs on a Campus
• Loosely coupled on a local-area-network
• PCs do other things some of the time
• When free cycles are available, they’re used
• Many more machines, but less of each machine available
Workstation Network at GoogleFront end100 machines called www.google.com
Searching machines Retrieving machines
Fit 40-80 machines in a 7’x2’x3’ rack
SETI
• Telescope at Arecibo, PR collects data
• Data is processed in real time by fast machines
• But, no one looks for weak signals– Too costly
• SETI@Home project built to do this
SETI@Home
• Receive data from Arecibo– 35 Gbytes per day by snail mail
• Break into Work Units– .25 Mbyte each, so 140,000 WU’s per day
• WU takes 20 hours to process
• Need about 117,000 dedicated machines to process one day
SETI@Home
• Get individual users to download software• Machine idle and screen saver runs software
– Download WU– Compute– When finished send back result
• Database at Berkeley reassembles results• Progress to date -- Seti@HomeStats
Medical/Biological Applications
• Peer-to-Peer Medicine• Cancer Research• …
Distributed databases
• Data spread across machines in different ways– Web pages
• E.g. HTTP
– MP3 collections• E.g. Napster, Gnutella
• E.g Morpheus, Kazaa, Music City
– Auction items• E.g. EBay
Client-Server Model
• Central server
• Clients store and retrieve data from server– File manager– HTTP
Napster Model
• Server is only used to find who has the file
• Communication is peer-to-peer (P2P)– Client to client transfer without a real server
How Napster works
• Initial registration – name, password, local directory for files, …
• When client connects to Napster server– Client provides list of files it will share
– Napster updates its central index of available files
• When client asks for a file– Napster gives client a list of online clients with that file
How Napster works (contd.)
• When client asks for download from given supplier– Napster asks supplier to accept a request
– Napster tells client how to contact supplier
– Client opens port and fetches file from supplier
– Supplier and client report progress/completion status to Napster
• Napster server directory continually updated• Client ranks potential servers by bandwidth and latency
Napster Model
• Server is only used to make connections
• Communication is peer-to-peer (P2P)– Client to client transfer without a real server
• Can we do this without a central server???
Gnutella
• Gnutella design has no central server
• Every machine is both client and server (called servant in Gnutella)
• To connect, you need to know any one machine already on the Gnutella network
How Gnutella works
• To connect to the network, – only need to know of any servant that is already connected. – Your servant announces your presence to all of the servants
it is already connected to, and so on until the message propagates throughout the entire network.
– Each of these servants then responds to this message with a bit of information about itself: how many files it is sharing, how many KBs of space they take up, etc.
• By connecting, you immediately know how much is available on the network to search through.
How Gnutella works - II
• To search– You send out a search request, it is propagated through
the network, and each servant that has matching terms passes back its result set
– Each servant handles the search query in its own way
– To save on bandwidth, a servant does not have to respond to a query if it has no matching items. The servant also has the option of returning only a limited result set.
How Gnutella works - III
• For file sharing, each servant acts as a miniature HTTP web server.
• To prevent searches from going on forever:– Gnutella messages have a TTL (Time To Live)
– The TTL starts off at some low number, like 5
– Each time a packet is routed through a servant, the servant lowers the TTL by 1
– Once the TTL hits 0 the packet is no longer forwarded
– Keeps messages from circling the network forever
What is KaZaA (now FastTrack)
• KaZaA Media Desktop17 Million downloads of KaZaA by April 2002! And 2nd place on C|Net Download.com. Why don't you join the world's largest online media community right now. 91% of users are recommending it. What are you waiting for? Download it now - it's free.
• KaZaA Media Desktop is a full featured peer-to-peer file sharing application. You can search, download, organise and play your media files - audio, video, images and documents with it. It has a powerful search engine where you can search on 'meta data' such as categories, artist etc.
Users and Usage
• 60M users of file-sharing in US– 8.5M logged in at a given time on average
• 814M units of music sold in US last year– 140M digital tracks sold by music companies
• As of Nov, 35% of all Internet traffic was for BitTorrent, a single file-sharing system
• Major legal battles underway between recording industry and file-sharing companies
Number of Users
Others includeBitTorrent, eDonkey, iMesh,Overnet, Gnutella
BitTorrent (and others) gains sharefrom FastTrack.Why?
What about:
• Copyrights– Who owns this stuff?
– Why is this not like going to a store and stealing?
• Breaches of trust– Try to download Friends but get pornography
– Can download serious viruses
• PU Bandwidth– This detracts from real work of the university?
What’s next
• Problems for which there are no algorithms
• Problems for which all algorithms run slowly
• Applications of problems where algorithms run slowly
The big picture
• We built a computer• We built an operating system • We attached the computer to a network• We wrote programs • We designed algorithms• We looked at distributed algorithms and systems
• Next, we want to see if there are limitations– Mathematical proofs that things cannot be done???
Simple unsolvable problem
This sentence is false.
Consider the sentence:
Problem: Can you say, correctly, if the sentence is true or false?
Limitations of Computers?
• Possible impossible tasks for computers?
• Emotion
• Creative thought
• Beating the world chess champion?
• Deep Blue v. Kasparov: Who won the 6-game rematch in 1997
• Others?
Limitations of Computers?
• How to show that such tasks are impossible?
• Find a cleanly defined problem
• Prove a theorem that says it can’t be solved
• Theorem is proved using axioms (like logic gates) that are assembled (like state machines and computers) to justify the result
• The underlying field is called mathematical logic
Simple task --The “Hello” Assignment
• You write a computer program which outputs “Hello!” and stops.
• Assignment promises full credit for ANY program that outputs “Hello!” and then halts, and no credit otherwise.
• Assignment is to be done on a computer that is as fast as you like and has as much memory as you like.
The “Hello” Assignment
• Can I write a program that automatically grades your homework?
• Sample program
Function GradeHELLO (Program P)
Execute P, and store output in output
If (output = “Hello!”) ThenReturn “Pass”
ElseReturn “Fail”
End If
End Function
The “Hello” Assignment
• Possible glitch– What if your homework program never
stops?Program P
Do While (1 > 0)Print “Hello!”
End While
End Program
The “Hello” Assignment
• Student’s program runs forever!
• This means the grading program would run forever.
• It would never tell me that the student should fail…
Program P
Do While (1 > 0)Print “Hello!”
End While
End Program
The “Hello” Assignment
I’m smarter than that:
Can just stop the Program after 1 hour and fail the student if the program hasn’t stopped yet.
The “Hello” Assignment• What if student’s program stops, but takes
a long time?Program P
Initialize x = 0Do While (x is not a proof of Fermat’s Last Theorem)
x = next mathematical statement in a sequenceEnd While
Print “Hello!”
End Program
The “Hello” Assignment• What if student’s program stops, but takes a
long time?
• Grading program might say to fail you
• But technically you deserve to Pass
• Unless you never reach a statement that is the proof
Program PInitialize x = 0Do While (x is not a proof of Fermat’s Last Theorem)
x = next mathematical statement in a sequenceEnd While
Print “Hello!”
End Program
The “Hello” Assignment• This is becoming a cat-and-mouse game…
• Could it be that writing a computer program to grade such a simple assignment is impossible?
• YES! Moreover we can prove that it is impossible.
The Halting Problem• Can we design a program, that tells whether
other programs ever halt on given inputs?
• Output of this program, called HALT, when applied to program P, is “halts” if P(input) eventually halts
• Output of this program is “never halts” if P(input) never halts
• Theorem (Alan Turing 1937): No computer program cansolve the halting problem.