mpi presentation
TRANSCRIPT
![Page 1: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/1.jpg)
11/05/08 Parallel Programming Using MPI 1
Parallel Programmingusing
Message Passing Interface
(MPI)
metu-ceng [email protected]
25 April 2008
![Page 2: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/2.jpg)
11/05/08 Parallel Programming Using MPI 2/26
Outline
• What is MPI?• MPI Implementations• OpenMPI• MPI• References• Q&A
![Page 3: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/3.jpg)
11/05/08 Parallel Programming Using MPI 3/26
What is MPI?
• A standard with many implementations (lam-mpi and mpich, evolving into OpenMPI and MVAPICH).
• message passing API• Library for programming clusters• Needs to be high performing, scalable,
portable ...
![Page 4: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/4.jpg)
11/05/08 Parallel Programming Using MPI 4/26
MPI Implementations• Is it up for the challenge?
MPI does not have many alternatives (what about OpenMP, map-reduce etc?).
• Many implementations out there.• The programming interface is all the same. But
underlying implementations and what they support in terms of connectivity, fault tolerance etc. differ.
• On ceng-hpc, both MVAPICH and OpenMPI is installed.
![Page 5: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/5.jpg)
11/05/08 Parallel Programming Using MPI 5/26
OpenMPI
• We'll use OpenMPI for this presentation• It's open source, MPI2 compliant,
portable, has fault tolerance, combines best practices of number of other MPI implementations.
• To install it, for example on Debian/Ubuntu type:# apt-get install openmpi-bin libopenmpi-dev
openmpi-doc
![Page 6: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/6.jpg)
11/05/08 Parallel Programming Using MPI 6/26
MPI – General Information
• Functions start with MPI_* to differ from application
• MPI has defined its own data types to abstract machine dependent implementations (MPI_CHAR, MPI_INT, MPI_BYTE etc.)
![Page 7: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/7.jpg)
11/05/08 Parallel Programming Using MPI 7/26
MPI - API and other stuff
• Housekeeping (initialization, termination, header file)
• Two types of communication: Point-to-point and collective communication
• Communicators
![Page 8: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/8.jpg)
11/05/08 Parallel Programming Using MPI 8/26
Housekeeping
• You include the header mpi.h• Initialize using MPI_Init(&argc, &argv)
and end MPI using MPI_Finalize()• Demo time, “hello world!” using MPI
![Page 9: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/9.jpg)
11/05/08 Parallel Programming Using MPI 9/26
Point-to-point communication
• Related definitions – source, destination, communicator, tag, buffer, data type, count
• man MPI_Send, MPI_Recv int MPI_Send(void *buf, int count, MPI_Datatype
datatype, int dest,int tag, MPI_Comm comm)
• Blocking send, that is the processor doesn't do anything until the message is sent
![Page 10: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/10.jpg)
11/05/08 Parallel Programming Using MPI 10/26
P2P Communication (cont.)• int MPI_Recv(void *buf, int count, MPI_Datatype
datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)
• Source, tag, communicator has to be correct for the message to be received
• Demo time – simple send• One last thing, you can use wildcards in
place of source and tag. MPI_ANY_SOURCE and MPI_ANY_TAG
![Page 11: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/11.jpg)
11/05/08 Parallel Programming Using MPI 11/26
P2P Communication (cont.)
• The receiver actually does not know how much data it received. He takes a guess and tries to get the most.
• To be sure of how much received, one can use:
• int MPI_Get_count(MPI_Status *status, MPI_Datatype dtype, int *count);
• Demo time – change simple send to check the received message size.
![Page 12: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/12.jpg)
11/05/08 Parallel Programming Using MPI 12/26
P2P Communication (cont.)• For a receive operation, communication ends when
the message is copied to the local variables.• For a send operation, communication is completed
when the message is transferred to MPI for sending. (so that the buffer can be recycled)
• Blocked operations continue when the communication has been completed
• Beware – There are some intricaciesCheck [2] for more information.
![Page 13: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/13.jpg)
11/05/08 Parallel Programming Using MPI 13/26
P2P Communication (cont.)
• For blocking communications, deadlock is a possibility:
if( myrank == 0 ) {
/* Receive, then send a message */
MPI_Recv( b, 100, MPI_DOUBLE, 1, 19, MPI_COMM_WORLD, &status );
MPI_Send( a, 100, MPI_DOUBLE, 1, 17, MPI_COMM_WORLD );
}
else if( myrank == 1 ) {
/* Receive, then send a message */
MPI_Recv( b, 100, MPI_DOUBLE, 0, 17, MPI_COMM_WORLD, &status ); MPI_Send( a, 100, MPI_DOUBLE, 0, 19, MPI_COMM_WORLD );
}
• How to remove the deadlock?
![Page 14: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/14.jpg)
11/05/08 Parallel Programming Using MPI 14/26
P2P Communication (cont.)
• When non-blocking communication is used, program continues its execution
• A program can send a blocking send and the receiver may use non-blocking receive or vice versa.
• Very similar function callsint MPI_Isend(void *buf, int count, MPI_Datatype dtype, int dest,
int tag, MPI_Comm comm, MPI_Request *request);
• Request handle can be used latereg. MPI_Wait, MPI_Test ...
![Page 15: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/15.jpg)
11/05/08 Parallel Programming Using MPI 15/26
P2P Communication (cont.)
• Demo time – non_blocking• There are other modes of sending
(but not receiving!) check out the documentation for synchronous, buffered and ready mode send in addition to standard one we have seen here.
![Page 16: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/16.jpg)
11/05/08 Parallel Programming Using MPI 16/26
P2P Communication (cont.)
• Keep in mind that each send/receive is costly – try to piggyback
• You can send different data types at the same time – eg. Integers, floats, characters, doubles... using MPI_Pack. This function gives you an intermediate buffer which you will send.
• int MPI_Pack(void *inbuf, int incount, MPI_Datatype datatype, void *outbuf, int outsize, int *position, MPI_Comm comm)
• MPI_Send(buffer, count, MPI_PACKED, dest, tag, MPI_COMM_WORLD);
![Page 17: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/17.jpg)
11/05/08 Parallel Programming Using MPI 17/26
P2P Communication (cont.)
• You can also send your own structs (user defined types). See the documentation
![Page 18: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/18.jpg)
11/05/08 Parallel Programming Using MPI 18/26
Collective Communication
• Works like point to point except you send to all other processors
• MPI_Barrier(comm), blocks until each processor calls this. Synchronizes everyone.
• Broadcast operation MPI_Bcast copies the data value in one processor to others.
• Demo time - bcast_example
![Page 19: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/19.jpg)
11/05/08 Parallel Programming Using MPI 19/26
Collective Communication
• MPI_Reduce collects data from other processors, operates on them and returns a single value
• reduction operation is performed• Demo time – reduce_op example• There are MPI defined reduce
operations but you can define your own
![Page 20: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/20.jpg)
11/05/08 Parallel Programming Using MPI 20/26
Collective Communication - MPI_Gather
• gather and scatter operations• Like what their name implies• Gather – like every process sending
their send buffer and root process receiving
• Demo time - gather_example
![Page 21: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/21.jpg)
11/05/08 Parallel Programming Using MPI 21/26
Collective Communication - MPI_Scatter
• Similar to MPI_Gather but here data is sent from root to other processors
• Like gather, you can accomplish it by having root calling MPI_Send repeatedly and others calling MPI_Recv
• Demo time – scatter_example
![Page 22: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/22.jpg)
11/05/08 Parallel Programming Using MPI 22/26
Collective Communication – More functionality
• Many more functions to lift hard work from you.
• MPI_Allreduce, MPI_Gatherv, MPI_Scan, MPI_Reduce_Scatter ...
• Check out the API documentation• Manual files are your best friend.
![Page 23: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/23.jpg)
11/05/08 Parallel Programming Using MPI 23/26
Communicators
• Communicators group processors• Basic communicator
MPI_COMM_WORLD defined for all processors
• You can create your own communicators to group processors. Thus you can send messages to only a subset of all processors.
![Page 24: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/24.jpg)
11/05/08 Parallel Programming Using MPI 24/26
More Advanced Stuff
• Parallel I/O – when one node is used for reading from disk it is slow. You can have each node use its local disk.
• One sided communications – Remote memory access
• Both are MPI-2 capabilities. Check your MPI implementation to see how much it is implemented.
![Page 25: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/25.jpg)
11/05/08 Parallel Programming Using MPI 25/26
References[1] Wikipedia articles in general, including but not limited to:
http://en.wikipedia.org/wiki/Message_Passing_Interface
[2] An excellent guide at NCSA (National Center for Supercomputing Applications):
http://webct.ncsa.uiuc.edu:8900/public/MPI/
[3] OpenMPI Official Web site:
http://www.open-mpi.org/
![Page 26: MPI Presentation](https://reader034.vdocuments.site/reader034/viewer/2022042700/555043fbb4c9058f768b4c41/html5/thumbnails/26.jpg)
11/05/08 Parallel Programming Using MPI 26/26
The End
Thanks For Your Time.Any Questions
?