introduction basics mpi interface topologies message passing...
Post on 29-Aug-2020
8 Views
Preview:
TRANSCRIPT
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
1 / 64
Message PassingInterface
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
2 / 64
• Google group – get me your email if youare not a member yet
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
3 / 64
Message Passing Paradigm
• Principles– Partitioned address space– Support only explicit parallelization
• Implications– Each data element belongs to one partition
in the address space– Encourages locality of access– All interactions require cooperation of two
or more processes
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
4 / 64
Quiz 2.1
• Can you use message passingparallelization if you do not know howto decompose your problem intoparallel parts?
No.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
5 / 64
Quiz 2.2
• Can you use shared memoryparallelization efficiently if you do notknow how to decompose your probleminto parallel parts?
No. Well maybe if you use aparallelizing compiler, but itprobably won’t be very efficient.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
6 / 64
Message Passing Paradigm
• Structure of Message Passing Programs– Synchronous– Asynchronous– Loosely synchronous– Single program multiple data
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
7 / 64
Building Blocks: Send/Receive
• Generic operation definitions• Send
• Receivereceive(void *recvbuf, int nelems, int source)
send(void *sendbuf, int nelems, int dest)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
8 / 64
Building Blocks: Send/Receive
• Simple Example
P0
A=100;send(&A, 1, 1);A=0;
P1
receive(&B, 1, 0);Printf(“%d\n”,B);
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
9 / 64
Building Blocks: Send/Receive
• Message Passing Operations– Blocking non-buffered
• Idling• Deadlocks
– Blocking buffered• Deadlocks
– Non-blocking• Caveats
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
10 / 64
Quiz 2.3
• Show a simple example that leads todeadlock if you have blocking sends?
P0 P1send (&A,1,1); send(&A,1,0);receive(&b,1,1); receive(&b,1,0);
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
11 / 64
MPI• Standard library for message passing• C or Fortran libraries• More than 125 routines• 6 basic routines
– MPI_Init– MPI_Finalize– MPI_Comm_size– MPI_Comm_rank– MPI_Send– MPI_Receive
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
12 / 64
MPI: Starting and Terminating• MPI_Init
– Called prior to any calls to any other MPIroutines
• MPI_Finalize
– Called after last other MPI call to clean-up and terminate the MPI environment
MPI_Finalize()
MPI_Init(int *argc, char ***argv)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
13 / 64
Quiz 2.4
• What happens if you call MPI_Sendbefore you call MPI_Initialize or afteryou call MPI_Finalize?
You have created an error andthe implementation can doanything – it is an error.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
14 / 64
MPI: Communicators• Communication Domain
– A set of processes that are allowed tocommunicate with each other.
– Each process can belong to multiplecommunication domains.
– type MPI_COMM• MPI_COMM_WORLD
– Includes all processes
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
15 / 64
Quiz 2.5
• Can an MPI process belong to 0, 1, ormany communicators?
1 or many; each processalways belongs toMPI_COMM_WORLD
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
16 / 64
MPI: Information Collection• MPI_Comm_size
– Returns (in the size variable) the numberof processes in the communicator comm.
• MPI_Comm_rank
– Returns (in the rank variable) theidentifier of the calling process.
int MPI_Comm_size(MPI_Comm comm, int *size)
int MPI_Comm_rank(MPI_Comm comm, int *rank)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
17 / 64
MPI: Creating a communicatorMPI_Group MPI_GROUP_WORLD, subgroup;int ranks[] = {2, 4, 6, 8};MPI_Comm the_comm;...MPI_Init(&argc, &argv);MPI_Comm_group(MPI_COMM_WORLD,
&MPI_GROUP_WORLD);MPI_Group_incl(MPI_GROUP_WORLD, 4, ranks, &subgroup); /*
local */MPI_Group_rank(subgroup, &me); /* local */MPI_Comm_create(MPI_COMM_WORLD, subgroup,
&the_comm);
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
18 / 64
MPI: Send & Receive• Send
• Receive
MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
19 / 64
MPI: Information Collection• Hello World Program
#include <mpi.h>main (int argc, char *argv[]){ int npes, myrank; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD,&npes); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); printf(“From process %d out of %d, Hello\n”, myrank, npes); MPI_Finalize();}
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
20 / 64
MPI: Status & Count• Status
• Countint MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count);
typedef struct MPI_Status { int MPI_SOURCE; int MPI_TAG; int MPI_ERROR; };
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
21 / 64
Why does MPI not guaranteebuffering?
• MPI does not guarantee to bufferarbitrary messages because memory is afinite resource on all computers. Thus, allcomputers will fail under sufficientlyadverse communication loads. Differentcomputers at different times are capableof providing differing amounts ofbuffering, so if a program relies onbuffering it may fail under certainconditions, but work correctly under otherconditions. This is clearly undesirable.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
22 / 64
Why does MPI not guaranteebuffering?
• Given that no message passing systemcan guarantee that messages will bebuffered as required under allcircumstances, it might be asked whyMPI does not guarantee a minimumamount of memory available forbuffering. One major problem is that it isnot obvious how to specify the amount ofbuffer space that is available, nor is iteasy to estimate how much buffer spaceis consumed by a particular program.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
23 / 64
Quiz 2.6
• List three MPI datatypes.
See next slide.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
24 / 64
MPI: DatatypesMPI Datatype C DatatypeMPI_CHAR Signed charMPI_SHORT Signed short intMPI_INT Signed intMPI_LONG Signed long intMPI_UNSIGNED_CHAR Unsigned charMPI_UNSIGNED_SHORT Unsigned short intMPI_UNSIGNED Unsigned intMPI_UNSIGNED_LONG Unsigned long intMPI_FLOAT FloatMPI_DOUBLE DoubleMPI_LONG_DOUBLE Long doubleMPI_BYTEMPI_PACKED
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
25 / 64
Slightly more complicated example#include "mpi.h"main( argc, argv )int argc;char **argv;{ char message[20]; int myrank; MPI_Status status; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &myrank ); if (myrank == 0) /* code for process zero */ { strcpy(message,"Hello, there"); MPI_Send(message, strlen(message), MPI_CHAR, 1, 99, MPI_COMM_WORLD); } else /* code for process one */ { MPI_Recv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &status); printf("received :%s:\n", message); } MPI_Finalize();}
Message send/receive;Extend this to N processors.
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
26 / 64
Quiz 2.7
• Can MPI_Send and MPI_Recv, asdescribed, lead to deadlock?
Yes, we will see anexample next…
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
27 / 64
MPI: Deadlockint a[10], b[10], myrank;MPI_Status status;…MPI_Comm_rank(MPI_COMM_WORLD, &myrank);if (myrank == 0) { MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); }else if (myrank == 1){ MPI_Recv(b, 10, MPI_INT, 0, 2, MPI_COMM_WORLD, &status); MPI_Recv(a, 10, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); }…
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
29 / 64
MPI: Deadlockint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); …
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
31 / 64
MPI: Deadlock⇒Safeint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);If (myrank%2 == 1) { MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); }else { MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); }…
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
32 / 64
MPI: SendrecvMPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, int dest, int sendtag,
void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int source, int recvtag, MPI_Comm comm, MPI_Status *status);
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
34 / 64
MPI: Deadlock⇒Safeint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);MPI_Sendrecv(a, 10, MPI_INT, (myrank+1)%npes, 1, b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status);…
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
35 / 64
MPI: Sendrecv_replace
MPI_Sendrecv_replace(void *sendbuf, int count, MPI_Datatype datatype, int dest, int sendtag, int source, int recvtag, MPI_Comm comm, MPI_Status *status);
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
37 / 64
MPI: Deadlock⇒Simple Safeint a[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD,&myrank);MPI_Sendrecv_replace(a, 10, MPI_INT, (myrank+1)%npes, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status);…
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
38 / 64
Topologies and Embedding
• Topology• Goodness of Mapping• Communication Domains & Ranks• Embedding
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
39 / 64
Cartesian Topologies• Virtual process topologies• Cartesian topologies
• Process naming
int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_cart)
int MPI_Cart_rank(MPI_Comm comm_cart, int *coords, int *rank)int MPI_Cart_coords(MPI_Comm comm_cart, int rank, int maxdims, int *coords)int MPI_Cart_shift(MPI_Comm comm_cart, int dir,int s_step, int *rank_source, int *rank_dest)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
41 / 64
Overlapping Computation andCommunication
• Non-blocking Communicationint MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request)
int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
43 / 64
Overlapping Computation andCommunication
• Non-blocking Communicationint MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)
int MPI_Wait(MPI_Request *request, MPI_Status *status)
int MPI_Request_free(MPI_Request request)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
45 / 64
Avoiding Deadlock• Using non-blocking communication
int a[10], b[10], npes, myrank;MPI_Status status;MPI_Request requests[2];…MPI_Comm_rank(MPI_COMM_WORLD, &myrank);If (myrank%2 == 0) { MPI_ISend(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_ISend(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); }else { MPI_IRecv(b, 10, MPI_INT, 0, 2, &requests[0], MPI_COMM_WORLD); MPI_IRecv(a, 10, MPI_INT, 0, 1, &requests[1], MPI_COMM_WORLD); } …
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
46 / 64
Collective Operations
• CollectiveCommunication/Synchronization– Barrier
int MPI_Barrier(MPI_COMM comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
48 / 64
Collective Operations
• Collective Communication– Broadcast
int MPI_Bcast(void *buf, int count, MPI_Datatype datatype, int source, MPI_Comm comm)
MPI_Bcast MPI_Bcast MPI_Bcast
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
50 / 64
Collective Operations• Collective Communication
– Reduction
int MPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op,int target, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
52 / 64
Predefined Reduction OperationsOperation Meaning DatatypesMPI_MAX Maximum C int or floatMPI_MIN Minimum C int or floatMPI_SUM Sum C int or floatMPI_PROD Product C int or floatMPI_LAND Logical AND C intMPI_BAND Bitwise AND C int or byteMPI_LOR Logical OR C intMPI_BOR Bitwise OR C int or byteMPI_LXOR Logical XOR C intMPI_BXOR Bitwise XOR C int or byteMPI_MAXLOC Max value location Data pairsMPI_MINLOC Min value location Data pairs
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
53 / 64
Collective Operations
• Collective Communication– Prefix
int MPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype, MPI_Op op, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
54 / 64
Collective Operations
• Collective Communication– Scatter
int MPI_Scatter(void *sendbuf, int sendcnt,MPI_Datatype sendtype, void *recvbuf, int recvcnt,MPI_Datatype recvtype, int source, MPI_Commcomm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
57 / 64
Collective Operations• Collective Communication
– Gatherint MPI_Gather(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int target, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
58 / 64
Collective Operations• Collective Communication
– Allgatherint MPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
59 / 64
Collective Operations
• Collective Communication– Gather Vector
int MPI_Gatherv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvdatatype, int target, MPI_Comm comm)
int MPI_Allgatherv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvdatatype, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
60 / 64
Collective Operations
• Collective Communication
int MPI_Scatter(void *sendbuf, void sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int source, MPI_Comm comm)
int MPI_Scatterv(void *sendbuf, void *sendcounts,int *displs, MPI_Datatype senddatatype, void*recvbuf, int recvcount, MPI_Datatype recvdatatype,int source, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
61 / 64
Collective Operations
• Collective Communication– All-to-All
int MPI_Alltoall(void *sendbuf, void sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, MPI_Comm comm)
int MPI_Alltoallv(void *sendbuf, void *sendcounts, int *sdispls, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *rdispls, MPI_Datatype recvdatatype, MPI_Comm comm)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
62 / 64
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
64 / 64 http://www.mpi-forum.org/docs/mpi-11-html/node64.html#Node64
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
65 / 64 http://www.mpi-forum.org/docs/mpi-11-html/node64.html#Node64
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
65 / 64
Communicators
• Splitting general communicators
int MPI_Comm_split(MPI_Comm comm, int color,int key, MPI_Comm *newcomm)
3
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
66 / 64
Communicators• Splitting Cartesian communicators
int MPI_Cart_sub(MPI_Comm comm_cart, int *keep_dims, MPI_Comm *comm_subcart)
1
2
3
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
67 / 64
Evaluating Parallel Programs
• Parallel Execution Time– tp = tcomp + tcomm
• Communication Time– tcomm = tstartup + ntdata
– Or for all communication…q messages– tcomm = q(tstartup + ntdata)
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
68 / 64
Timing in MPI• Inside the code
– Fortran – thread time• dtime• etime
– C• times()
– C – wallclock time• MPI_wtime
Debugging andEvaluating ParallelPrograms
Assignment
Evaluating ParallelPrograms
CS/IT 451 – Parallel Processing
Lecture 2 – Message Passing Computing
Introduction
Topologies
Overlapping C&C
Basics
Collective Comms.
Communicators
MPI Interface
69 / 64
Analyzing and DebuggingParallel Programs
top related