Message Passing Architecture

Introduction

• The architecture is used to communicate data among a set of processors without the need for a global memory.

• Each PE has its own local memory and communicates with other PEs using message.

Message Passing Systems

networks have received considerable attention over the years. As shown in Chapter 2,two important factors must be considered in designing message passing interconnec-tion networks: link bandwidth and the network latency. The link bandwidth is definedas the number of bits that can be transmitted per unit of time (bits/s). Network latencyis defined as the time to complete a message transfer through the network.

In executing a given application program, the program is divided into concurrentprocesses; each is executed on a separate processor. If the number of processes islarger than the number of processors, then more than one process will have to beexecuted on a processor in a time-shared fashion. Processes running on a given pro-cessor use what is called internal channels to exchange messages among them-selves. Processes running on different processors use the external channels toexchange messages. Data exchanged among processors cannot be shared; it israther copied (using send/receive messages). An important advantage of thisform of data exchange is the elimination of the need for synchronization constructs,such as semaphores, which results in performance improvement. In addition,a message passing scheme offers flexibility in accommodating a large numberof processors in addition to being readily scalable. It should be noted that a givennode can execute more than one process, each at a given time.

Figure 5.2 shows an example message passing system consisting of four pro-cesses. In this figure, a horizontal line represents the execution of each processand lines extended among processes represent messages exchanged among theseprocesses. A message is defined as a logical unit for internode communication;it is considered as a collection of related information that travels together asan entity. A message can be an instruction, data, synchronization, or interruptsignals. A message passing system interacts with the outside world by receivinginput message(s) and/or outputting message(s). It is essential that the outsideworld perceives a consistent behavior of a given message passing system.

Process Granularity The size of a process in a message passing system can bedescribed by a parameter called process granularity. This is defined as follows.

Process Granularity! computation time

communication time

1P 1M 2P 2M nP nM

Interconnection Network

Link 1 Link 2Link n

Figure 5.1 Message passing systems.

104 MESSAGE PASSING ARCHITECTURE

Message Passing Systems

networks have received considerable attention over the years. As shown in Chapter 2,two important factors must be considered in designing message passing interconnec-tion networks: link bandwidth and the network latency. The link bandwidth is definedas the number of bits that can be transmitted per unit of time (bits/s). Network latencyis defined as the time to complete a message transfer through the network.

In executing a given application program, the program is divided into concurrentprocesses; each is executed on a separate processor. If the number of processes islarger than the number of processors, then more than one process will have to beexecuted on a processor in a time-shared fashion. Processes running on a given pro-cessor use what is called internal channels to exchange messages among them-selves. Processes running on different processors use the external channels toexchange messages. Data exchanged among processors cannot be shared; it israther copied (using send/receive messages). An important advantage of thisform of data exchange is the elimination of the need for synchronization constructs,such as semaphores, which results in performance improvement. In addition,a message passing scheme offers flexibility in accommodating a large numberof processors in addition to being readily scalable. It should be noted that a givennode can execute more than one process, each at a given time.

Figure 5.2 shows an example message passing system consisting of four pro-cesses. In this figure, a horizontal line represents the execution of each processand lines extended among processes represent messages exchanged among theseprocesses. A message is defined as a logical unit for internode communication;it is considered as a collection of related information that travels together asan entity. A message can be an instruction, data, synchronization, or interruptsignals. A message passing system interacts with the outside world by receivinginput message(s) and/or outputting message(s). It is essential that the outsideworld perceives a consistent behavior of a given message passing system.

Process Granularity The size of a process in a message passing system can bedescribed by a parameter called process granularity. This is defined as follows.

Process Granularity! computation time

communication time

1P 1M 2P 2M nP nM

Interconnection Network

Link 1 Link 2Link n

Figure 5.1 Message passing systems.

104 MESSAGE PASSING ARCHITECTURE

external channel

MP Network

• Normally, a static-type network

• hypercubes

• nearest-neighbor 2D & 3D mesh

• Two important factors must be considered:

• link bandwidth - the number of bits that can be transmitted per unit of times (bits/s)

• network latency - the time to complete a message transfer through the network

Executing Program• The program is divided into concurrent

processes; each is executed on a separate processor.

• If the no. of processes is larger than the number of processors, then more than one process will have to be executed on a processor in a time-shared fashion.

networks have received considerable attention over the years. As shown in Chapter 2,two important factors must be considered in designing message passing interconnec-tion networks: link bandwidth and the network latency. The link bandwidth is definedas the number of bits that can be transmitted per unit of time (bits/s). Network latencyis defined as the time to complete a message transfer through the network.

In executing a given application program, the program is divided into concurrentprocesses; each is executed on a separate processor. If the number of processes islarger than the number of processors, then more than one process will have to beexecuted on a processor in a time-shared fashion. Processes running on a given pro-cessor use what is called internal channels to exchange messages among them-selves. Processes running on different processors use the external channels toexchange messages. Data exchanged among processors cannot be shared; it israther copied (using send/receive messages). An important advantage of thisform of data exchange is the elimination of the need for synchronization constructs,such as semaphores, which results in performance improvement. In addition,a message passing scheme offers flexibility in accommodating a large numberof processors in addition to being readily scalable. It should be noted that a givennode can execute more than one process, each at a given time.

Figure 5.2 shows an example message passing system consisting of four pro-cesses. In this figure, a horizontal line represents the execution of each processand lines extended among processes represent messages exchanged among theseprocesses. A message is defined as a logical unit for internode communication;it is considered as a collection of related information that travels together asan entity. A message can be an instruction, data, synchronization, or interruptsignals. A message passing system interacts with the outside world by receivinginput message(s) and/or outputting message(s). It is essential that the outsideworld perceives a consistent behavior of a given message passing system.

Process Granularity The size of a process in a message passing system can bedescribed by a parameter called process granularity. This is defined as follows.

Process Granularity! computation time

communication time

1P 1M 2P 2M nP nM

Interconnection Network

Link 1 Link 2Link n

Figure 5.1 Message passing systems.

104 MESSAGE PASSING ARCHITECTURE

Process Communication

• Processes running on a given processor use what is called internal channels to exchange messages among themselves.

• Processes running on different processors use the external channels to exchange messages.

Data Exchanged

• Data exchanged among processors cannot be shared; it is rather copied (using send/receive messages).

• An important advantage of this form of data exchange is the elimination of the need for synchronization constructs, such as semaphores, which results in performance improvement.

More advantages

• A message passing scheme offers flexibility in accommodating a large number of processors in addition to being readily scalable.

• A given node can execute more than one process, each at a given time.

Three types of granularity can be distinguished. These are:

1. Coarse granularity: Each process holds a large number of sequential instruc-tions and takes a substantial amount of time to execute.

2. Medium granularity: Since the process communication overhead increases asthe granularity decreases, medium granularity describes a middle groundwhere communication overhead is reduced.

3. Fine granularity: Each process contains a few sequential instructions (as fewas just one instruction).

Message passing multiprocessors uses mostly medium or coarse granularity.

5.2 ROUTING IN MESSAGE PASSING NETWORKS

Routing is defined as the techniques used for a message to select a path over thenetwork channels. Formally speaking, routing involves the identification of a setof permissible paths that may be used by a message to reach its destination, and afunction, h, that selects one path from the set of permissible paths.

A routing technique is said to be adaptive if, for a given source and destinationpair, the path taken by the message depends on network conditions, such as networkcongestion. Contrary to adaptive routing, a deterministic routing technique, alsocalled oblivious, determines the path using only the source and destination, regard-less of the network conditions. Although simple, oblivious routing techniques makeinefficient use of the bandwidth available between the source and destination.

Routing techniques can also be classified based on the method used to make therouting decision as centralized (self ) or distributed routing. In centralized routing,

Outside World

1P

1m2P

4m

3P

2m 3m

4P

Figure 5.2 An example of a message passing system.

5.2 ROUTING IN MESSAGE PASSING NETWORKS 105

Process Granularity

• It’s a parameter which describes the size of a process in a message passing system.

Process Granularitycomputation time

communication time=

Granularity Types

• There are 3 types of granularity:

• Coarse - each process holds a large number of sequential instructions and takes a substantial amount of time to execute.

• Medium - a middle ground where communication overhead is reduced

• Fine - each process contains a few sequential instructions

• Use mostly medium or coarse granularity

Routing in MP Networks

• Involves the identification of a set of permissible paths that may be used by a message to reach its destination, and a function (η) that selects one path from the set of permissible paths.

Routing Techniques

• Adaptive

• the path taken by the message depends on network conditions for a given source and destination pair

• Deterministic (oblivious)

• determines the path using only the source and destination regardless of the network conditions - simple but bandwidth inefficient

Routing for Broadcasting & Multicasting

• Communication operations: unicast and collective

• Unicast - communicate to only a single destination

• Collective - several routing operations are defined, broadcast & multicast are the most widely used.

Broadcast Multicast

one-to-all operation one-to-many operation

distribute data during computation of a

distributed memory program

uses in large-scale multiprocessors, inc.

parallel search algorithm

routing algorithms must be deadlock-freerouting algorithms must be deadlock-free

little time and short path is desirablelittle time and short path is desirable

multiple outgoing links towards disjoint subsets of destinations. Another method toimplement a multicast operation uses separate addressing. In this case, a separatecopy of the message is sent directly from the source to every destination. Clearly,this is an inefficient technique. A hypercube broadcast tree-based nearest-neighborcommunication is shown in Figure 5.3.

5.2.2 Routing Potential Problems

A number of possible problems can result from the use of certain routing mechan-isms in message passing systems. These include deadlock, livelock, and starvation,which are explained below.

Deadlock When two messages each hold the resources required by the other inorder to move, both messages will be blocked. This is called a deadlock. It is aphenomenon that occurs whenever there exists cyclic dependency for resources.Management of resources in a network is the responsibility of the flow controlmech-anism used. Resources must be allocated in a manner that avoids deadlock.A straightforward, but inefficient, way to solve the deadlock problem is to allowrerouting (maybe discarding) of the messages participating in a deadlock situation.Rerouting of messages gives rise to nonminimal routing, while discarding messagesrequires that messages be recovered at the source and retransmitted. This preemptivetechnique leads to long latency and, therefore, is not used by most message passingnetworks.

A more common technique is to avoid the occurrence of deadlock. This can beachieved by ordering network resources and requiring that messages request useof these resources in a strict monotonic order. This restricted way for using networkresources prevents the occurrence of circular wait, and hence prevents the occur-rence of deadlock. The channel dependency graph (CDG) is a technique usedto develop a deadlock-free routing algorithm. A CDG is a directed graphD ! G(C, E ), where the vertex set C consists of all the unidirectional channels inthe network and the set of edges E includes all the pairs of connected channels,

110 111

100

011

001000

010

000

001 010 100

011 101 110 011 101 110

101 111 010 100 111 100 111 001001

110 101 011 010

Figure 5.3 Hypercube broadcast tree-based communication.

5.2 ROUTING IN MESSAGE PASSING NETWORKS 107

• Deadlock

• When two messages each hold the resources required by the other in order to move, both message will be blocked.

• Livelock

• A situation in which a message keeps going around the network and never reaches its destination.

• Starvation

• A node is said to suffer from starvation if it has a message to inject into the network but is never allowed to do so.

Routing Potential Problems

Message Passing vs Shared Memory

• Shared memory systems may be easier to program, but the difficult to scale up to a large number of processors.

• If scalability to larger and larger systems was to continue, systems had to use message passing techniques.

• However, a number of problems associated with message passing systems: communication overhead and difficulty of programming.