![Page 1: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/1.jpg)
Distributed (Operating) Systems -Communication in Distributed Systems-
Fall 2011Kocaeli University Computer
Engineering Department
![Page 2: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/2.jpg)
Communication inDistributed Systems
• Communication is done through message passing• Expressing communication through message passing is
harder than using primitives based on shared memory
• Remote Procedure Calls– Transparency but poor for passing references– Ideal for client server applications
• Message-oriented Communication• Stream-oriented Communication
– Continuous media
![Page 3: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/3.jpg)
Communication Between Processes
• Unstructured communication– Use shared memory or shared data structures
• Structured communication– Use explicit messages (IPCs)
• Distributed Systems: both need low-level communication support (why?)
![Page 4: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/4.jpg)
Types of Communication -1
• Transient Communication– A message is stored by the communication system only as
long as the sending and the receiving application are executing
– Typically all transport level communication is transient communication
• Persistent Communication– A message that has been submitted for transmission is kept
by middleware as long as it takes to deliver it to the receiver– Receiving application don’t need to be executing when the
message when the message is submitted
![Page 5: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/5.jpg)
Types of Communication -2
• Asynchronous Communication– Sender continues immediately after it has submitted its
message for transmission– It means, message is temporarily stored in middleware
upon submission• Synchronous Communication
– Sender is blocked until its request is known to be accepted– Three points
• Sender informs message is delivered• Middleware informs message is delivered• Sender informs with the response after processing
![Page 6: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/6.jpg)
Types of Communication -3
• Connection-oriented (telephone)– TCP
• Connectionless (mailbox – dropping a letter)– IP, UDP
![Page 7: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/7.jpg)
Persistence and Synchronicity in Communication
![Page 8: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/8.jpg)
Persistence
• Persistent communication– Messages are stored until (next) receiver is ready– Examples: email, pony express
![Page 9: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/9.jpg)
Transient Communication
• Transient communication– Message is stored only so long as
sending/receiving application are executing– Discard message if it can’t be delivered to next
server/receiver– Example: transport-level communication services
offer transient communication– Example: Typical network router – discard
message if it can’t be delivered next router or destination
![Page 10: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/10.jpg)
Synchronicity
• Asynchronous communication– Sender continues immediately after it has submitted
the message– Need a local buffer at the sending host
• Synchronous communication– Sender blocks until message is stored in a local buffer
at the receiving host or actually delivered to sending– Variant: block until receiver processes the message
• Six combinations of persistence and synchronicity
![Page 11: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/11.jpg)
Persistence and Synchronicity Combinations
a) Persistent asynchronous communication (e.g., email)b) Persistent synchronous communication
![Page 12: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/12.jpg)
Persistence and Synchronicity Combinations
c) Transient asynchronous communication (e.g., UDP)d) Receipt-based transient synchronous communication
![Page 13: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/13.jpg)
Persistence and Synchronicity Combinations
e) Delivery-based transient synchronous communication at message delivery (e.g., asynchronous RPC)
f) Response-based transient synchronous communication (RPC)
![Page 14: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/14.jpg)
Layered Protocols 1
• Communication messaging rules are defined in protocols
• Due to the absence of shared memory, all communication in distributed systems is based on sending and receiving (low-level) messages
• Many different agreements– IBM’s EBCDIC and ASCII– How many volts should be used to signal 0– How does the receiver know which is the last bit– How can the receiver detect if a message has been damaged
![Page 15: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/15.jpg)
Layered Protocols 2
• ISO/OSI never widely used and are essentially dead.
• In the OSI model communication is divided up into 7 layers
• OSI layers were never popular. In contrast, protocols developed for the internet, such as TCO and IP, are mostly used
![Page 16: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/16.jpg)
Communication Protocols• There are some rules that communicating processes must adhere to –known as
protocols• Protocols are agreements/rules on communication• Protocols could be connection-oriented or connectionless
![Page 17: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/17.jpg)
Layered Protocols
• A typical message as it appears on the network.
![Page 18: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/18.jpg)
Layers -1
• Physical layer– Transmitting 0s and 1s– How many bits per sec transfer– Can transmission take place in both direction
• Data Link Layer– Puts special bit pattern on the start and end of each
frame– Computing checksum
• Network Layer– Routing
![Page 19: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/19.jpg)
Layers -2
• Transport Protocol– Turns the underlying network into something than an application
developer can use– Messages from the application is broken into small pieces-packets– Which packets have been sent, which have been received, which
should be retransmitted– Connection-oriented: messages arrive in the same order (as in TCP)– Connectionless: message can arrive in different order (as in UDP)– Another example of transport protocol is RTP– The combination of TCP/IP is now used as a de facto standard for
network communication
![Page 20: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/20.jpg)
Client-Server TCP
![Page 21: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/21.jpg)
Layers -3
• Session Layer– Enhanced version if transport layer– Keeps track of which party is currently talking and
it provides a synchronization– Puts checking points – in case of crash going to the
last check point• Presentation Layer
– Concerned with the meaning of the bits
![Page 22: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/22.jpg)
Middleware Protocols• Middleware: layer that resides between an OS and an application
– May implement general-purpose protocols that warrant their own layers
![Page 23: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/23.jpg)
OSI Model summary
• In this model, clear distinction between applications, application-specific protocols and general purpose protocols.
• Application-specific protocol– FTP– HTTP
• General-purpose protocols– Useful to many applications but cannot be qualified as
transport protocols. These protocols fall into the category of middleware protocols.
![Page 24: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/24.jpg)
Middleware Protocols 1
• Some services are not tied to any specific applications, but instead can be integrated into a middleware system as a general service– Authentication protocols
• Proof of claimed identity
– Atomicity protocols• Widely applied in transactions
– Distributed locking protocols• Resources can be protected against simultaneous
accesses
![Page 25: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/25.jpg)
Middleware Protocols 2
• Also supports high-level communication services– Remote Procedure Call (RPC) – Message Oriented Middleware (MOM)– RTP: Streams for transferring real-time data, such
as needed for multimedia applications
![Page 26: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/26.jpg)
Remote Procedure Calls
• Goal: Make distributed computing look like centralized computing
• Allow remote services to be called as procedures– Transparency with regard to location,
implementation, language• Issues
– How to pass parameters– Bindings– Semantics in face of errors
![Page 27: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/27.jpg)
Example of an RPCNo message passing at all is visible to the programmer.
![Page 28: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/28.jpg)
Divide programs up and add communication protocols
blah, blah, blah
bar = add(i,j);
blah, blah, blah
blah, blah, blah
bar = add(i,j);
blah, blah, blah
Int add(int x, int y ) { if (x>100)
return(y-2); else if (x>10)
return(y-x); else
return(x+y);}
Int add(int x, int y ) { if (x>100)
return(y-2); else if (x>10)
return(y-x); else
return(x+y);}
ClientServer
protocol
![Page 29: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/29.jpg)
RPC Semantics• Principle of RPC between a client and server program [Birrell&Nelson 1984]
![Page 30: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/30.jpg)
Other RPC Models
• Asynchronous RPC– Request-reply behavior often not needed– Server can reply as soon as request is received and execute
procedure later• Deferred-synchronous RPC
– Use two asynchronous RPCs– Client needs a reply but can’t wait for it; server sends reply via
another asynchronous RPC• One-way RPC
– Client does not even wait for an ACK from the server– Limitation: reliability not guaranteed (Client does not know if
procedure was executed by the server).
![Page 31: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/31.jpg)
Asynchronous RPC
a) The interconnection between client and server in a traditional RPCb) The interaction using asynchronous RPC
![Page 32: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/32.jpg)
Deferred Synchronous RPC
• A client and server interacting through two asynchronous RPCs
![Page 33: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/33.jpg)
Conventional Procedure Calla) Parameter passing in a local
procedure call: the stack before the call to read
b) The stack while the called procedure is active
Count = read(fd, buf, nbytes)
![Page 34: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/34.jpg)
Possible Issues
• Calling and called procedures run on different machines
• They execute in different address spaces• Parameters and results have to be passed, it can be
complicated when the machines are not identical.– How do you represent integers – big-endian little-endian
• Either or both machines can crash and each of the possible failures causes different problems.
![Page 35: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/35.jpg)
Parameter Passing
• Local procedure parameter passing– Call-by-value– Call-by-reference: arrays, complex data structures
• No sense BUT in RMI it is OK
• Remote procedure calls simulate this through:– Stubs – proxies– Flattening – marshalling
• Related issue: global variables are not allowed in RPCs
![Page 36: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/36.jpg)
Client and Server Stubs
• Client makes procedure call (just like a local procedure call) to the client stub
• Server is written as a standard procedure• Stubs take care of packaging arguments and
sending messages• Packaging parameters is called marshalling• Stub compiler generates stub automatically from
specs in an Interface Definition Language (IDL)– Simplifies programmer task
![Page 37: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/37.jpg)
Steps of a Remote Procedure Call
1. Client procedure calls client stub in normal way2. Client stub builds message, calls local OS3. Client's OS sends message to remote OS4. Remote OS gives message to server stub5. Server stub unpacks parameters, calls server6. Server does work, returns result to the stub7. Server stub packs it in message, calls local OS8. Server's OS sends message to client's OS9. Client's OS gives message to client stub10. Stub unpacks result, returns to client
![Page 38: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/38.jpg)
Marshalling
• Problem: different machines have different data formats– Intel: little endian, SPARC: big endian
• Solution: use a standard representation– Example: external data representation (XDR)
![Page 39: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/39.jpg)
Binding
• Problem: how does a client locate a server?– Use Bindings
• Server– Export server interface during initialization– Send name, version no, unique identifier, handle (address)
to binder• Client
– First RPC: send message to binder to import server interface– Binder: check to see if server has exported interface
• Return handle and unique identifier to client
![Page 40: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/40.jpg)
Binder: Port Mapper• Server start-up: create port• Server stub calls svc_register to register prog #, version # with local port mapper• Port mapper stores prog #, version #, and port• Client start-up: call clnt_create to locate server port• Upon return, client can call procedures at the server
![Page 41: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/41.jpg)
Case Study: SUNRPC• One of the most widely used RPC systems• Developed for use with NFS• Built on top of UDP or TCP• Multiple arguments marshaled into a single structure• At-least-once semantics if reply received, at-least-zero semantics if
no reply. With UDP tries at-most-once• Use SUN’s eXternal Data Representation (XDR)
– Big endian order for 32 bit integers, handle arbitrarily large data structures• XDR was originally designed for specifying eXternal Data
Representation• XDR has been extended to become Sun RPC IDL• An interface contains a program number, version number,
procedure definition and required type definitions
![Page 42: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/42.jpg)
Rpcgen: generating stubs
• Q_xdr.c: do XDR conversion• Detailed example: later in this course
![Page 43: Distributed (Operating) Systems -Communication in Distributed Systems- Fall 2011 Kocaeli University Computer Engineering Department](https://reader030.vdocuments.site/reader030/viewer/2022032612/56649ec85503460f94bd49b6/html5/thumbnails/43.jpg)