mpi 0802 mod collective

7/30/2019 Mpi 0802 Mod Collective

1/33

1

Collective Communication

Collective communication

Barrier synchronization

Broadcast*

Scatter*

Gather

Gather/Scatter variations

Summary illustration

Global Reductionoperations

Predefined reductionoperations

MPI _Reduce

Minloc and Maxloc*

User-Defined reduction

operations

Reduction operator functions

Registering a user-defined

reduction operator* Variants ofMPI _Reduce

Class Exercise: Last Ring

*includes sample C and Fortranprograms


2/33

2

Collective communication

Communications involving a group of processes

Called by all processes in a communicator

Examples: Barrier (synchronization)

Broadcast, scatter, gather (data distribution)

Global sum, global maximum, etc. (collectiveoperations)


3/33

3

Characteristics of collective communication

Collective communication will not interfere withpoint-to-point communication and vice-versa

All processes must call the collective routine Synchronization not guaranteed (except forbarrier)

No non-blocking collective communication No tags

Receive buffers must be exactly the right size


4/334

Barrier synchronization

Red light for each processor: turns green whenall processors have arrived

C: i nt MPI _Bar r i er ( MPI _Comm comm)

Fortran:I NTEGER COMM, I ERROR

CALL MPI _BARRI ER ( COMM, I ERROR)


5/335

Broadcast

One-to-all communication: same data sent from rootprocess to all the others in the communicator

C:i nt MPI _Bcast ( voi d *buf f er , i nt , count ,

MPI _Dat at ype dat at ype, i nt r oot , MPI _Comm comm)

Fortran:

BUFFER ( *)I NTEGER COUNT, DATATYPE, ROOT, COMM, I ERRORCALL MPI _BCAST( BUFFER, COUNT, DATATYPE, ROOT, COMM,

I ERROR)

All processes must specify same root rank andcommunicator


6/336

Sample Program #5 - C#i ncl udei nt mai n ( i nt ar gc, char *ar gv[ ] ) {

i nt r ank;

doubl e par am;

MPI _I ni t ( &ar gc, &ar gv) ;

MPI _Comm_r ank( MPI _COMM_WORLD, &r ank) ;

i f ( r ank==5) par am=23. 0;

MPI_Bcast(&param,1,MPI_DOUBLE,5,MPI_COMM_WORLD);

pr i nt f ( "P: %d af t er br oadcast par amet er i s %f \ n" , r ank, par am) ;

MPI _Fi nal i ze( ) ;

}

Progr am Out putP: 0 af t er br oadcast par amet er i s 23. 000000P: 6 af t er br oadcast par amet er i s 23. 000000P: 5 af t er br oadcast par amet er i s 23. 000000P: 2 af t er br oadcast par amet er i s 23. 000000P: 3 af t er br oadcast par amet er i s 23. 000000P: 7 af t er br oadcast par amet er i s 23. 000000P: 1 af t er br oadcast par amet er i s 23. 000000P: 4 af t er br oadcast par amet er i s 23. 000000


7/337

Sample Program #5 - FortranPROGRAM br oadcastI NCLUDE ' mpi f . h'

I NTEGER er r , r ank, si ze

r eal par am

CALL MPI _I NI T( er r )

CALL MPI _COMM_RANK( MPI _COMM_WORLD, r ank, er r )

CALL MPI _COMM_SI ZE( MPI _COMM_WORLD, si ze, er r )

i f ( r ank. eq. 5) par am=23. 0

call MPI_BCAST(param,1,MPI_REAL,5,MPI_COMM_WORLD,err)

pr i nt *, "P: ", r ank, " af t er br oadcast par am i s ", par am

CALL MPI _FI NALI ZE( er r )

ENDProgr am Out put

P: 1 af t er br oadcast par amet er i s 23.P: 3 af t er br oadcast par amet er i s 23.P: 4 af t er br oadcast par amet er i s 23P: 0 af t er br oadcast par amet er i s 23

P: 5 af t er br oadcast par amet er i s 23.P: 6 af t er br oadcast par amet er i s 23.P: 7 af t er br oadcast par amet er i s 23.P: 2 af t er br oadcast par amet er i s 23.


8/338

Scatter

One-to-all communication: different data sent to eachprocess in the communicator (in rank order)

C:i nt MPI _Scat t er ( voi d* sendbuf , i nt sendcount ,

MPI _Dat at ype sendt ype, voi d* r ecvbuf , i nt r ecvcount ,MPI _Dat at ype r ecvt ype, i nt r oot , MPI _Comm comm)

Fortran:CALL MPI _SCATTER( SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF,RECVCOUNT, RECVTYPE, ROOT, COMM, I ERROR)

SENDBUF( *) , RECVBUF( *)

sendcount is the number of elements sent to eachprocess, not the total number sent send arguments are significant only at the root process


9/33

9

Scatter example

A B C D E

A B C D E

BA C D E


10/33

10

#i ncl ude i nt mai n ( i nt ar gc, char *ar gv[ ] ) {

i nt r ank, si ze, i , j ;

doubl e param[ 4] , mi ne;

i nt sndcnt , r evcnt ;



MPI _Comm_si ze( MPI _COMM_WORLD, &si ze) ;

r evcnt =1;

i f ( r ank==3) {

f or ( i =0; i


11/33

11

Sample Program #6 - FortranPROGRAM scat t erI NCLUDE ' mpi f . h'


r eal par am( 4) , mi ne

i nt eger sndcnt , r cvcnt


CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )

r cvcnt =1

i f ( r ank. eq. 3) t hen

do i =1, 4

par am( i ) =23. 0+i

end do

sndcnt =1

end i f

call MPI_SCATTER(param,sndcnt,MPI_REAL,mine,rcvcnt,MPI_REAL,

& 3,MPI_COMM_WORLD,err)

pr i nt *, "P: ", r ank, " mi ne i s ", mi ne


END Progr amOut putP: 1 mi ne i s 25.P: 3 mi ne i s 27.P: 0 mi ne i s 24.P: 2 mi ne i s 26.


12/33

12

Gather

All-to-one communication: different data collectedby root process Collection done in rank order

MPI _GATHER &MPI _Gat her have samearguments as matching scatter routines

Receive arguments meaningful only at the rootprocess


13/33

13

Gather example

A B C D E

BA C D E

BA C D E


14/33

14

Gather/Scatter variations

MPI _Al l gat her MPI _Al l t oal l

No root process specified: all processes getgathered or scattered data

Send and receive arguments significant for all

processes


15/33

15

Summary


16/33

16

Global Reduction Operations

Used to compute a result involving datadistributed over a group of processes

Examples: Global sum or product Global maximum or minimum

Global user-defined operation


17/33

17

Predefined reduction operations

MPI Name Function

MPI_MAX Maximum

MPI_MIN Minimum

MPI_SUM SumMPI_PROD Product

MPI_LAND Logical AND

MPI_BAND Bitwise AND

MPI_LOR Logical OR

MPI_BOR Bitwise OR

MPI_LXOR Logical exclusive OR

MPI_BXOR Bitwise exclusive OR

MPI_MAXLOC Maximum and location

MPI_MINLOC Minimum and location


18/33

18

Global Reduction Operations

C:i nt MPI _Reduce( voi d* sendbuf , voi d* r ecvbuf , i nt count ,

MPI _Dat at ype dat at ype, MPI _Op op, i nt r oot ,MPI _Comm comm)

Fortran: SENDBUF( *) , RECVBUF( *)I NTEGER COUNT, DATATYPE, OP, ROOT, COMM, I ERRORCALL MPI _REDUCE( SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM,

I ERROR)

op is an associative operator that takes two operands oftype dat at ype and returns a result of the same type

count is the number of ops done on consecutiveelements ofsendbuf (it is also size ofr ecvbuf)


19/33

19

MPI_Reduce


20/33

20

MPI_MINLOC,MPI_MAXLOC

Designed to compute a global minimum/maximumand an index associated with the extreme value Common application: index is the processor rank (see

sample program)

If more than one extreme, get the first

Designed to work on operands that consist of a value

and index pair MPI _Dat at ypes include:

C:MPI _FLOAT_I NT, MPI _DOUBLE_I NT, MPI _LONG_I NT, MPI _2I NT,

MPI _SHORT_I NT, MPI _LONG_DOUBLE_I NT

Fortran:MPI _2REAL, MPI _2DOUBLEPRECI SI ON, MPI _2I NTEGER


21/33

21

Sample Program #7 - C#i ncl ude

/ * Run wi t h 16 pr ocesses * /

i nt mai n ( i nt ar gc, char *ar gv[ ] ) {

i nt r ank;

struct {

doubl e val ue;

i nt r ank;

} i n, out ;

i nt r oot ;



i n. val ue=r ank+1;

i n. r ank=r ank;

r oot =7;

MPI_Reduce(&in,&out,1,MPI_DOUBLE_INT,MPI_MAXLOC,root,MPI_COMM_WORLD);

i f ( r ank==r oot ) pr i nt f ( "PE: %d max=%l f at r ank %d\ n" , r ank, out . val ue, out . r ank) ;MPI_Reduce(&in,&out,1,MPI_DOUBLE_INT,MPI_MINLOC,root,MPI_COMM_WORLD);

i f ( r ank==r oot ) pr i nt f ( "PE: %d mi n=%l f at r ank %d\ n" , r ank, out . val ue, out . r ank) ;


}

Progr am Out putP: 7 max = 16. 000000 at r ank 15P: 7 mi n = 1. 000000 at r ank 0


22/33

22

PROGRAM MaxMi nC

C Run wi t h 8 processes

C

I NCLUDE ' mpi f . h'


i nt eger i n( 2) , out ( 2)


CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )

CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )

i n( 1) =r ank+1

i n( 2) =r ank

call MPI_REDUCE(in,out,1,MPI_2INTEGER,MPI_MAXLOC,


i f ( r ank. eq. 7) pr i nt *, "P: ", r ank, " max=", out ( 1) , " at r ank ", out ( 2)

call MPI_REDUCE(in,out,1,MPI_2INTEGER,MPI_MINLOC,


i f ( r ank. eq. 2) pr i nt *, "P: ", r ank, " mi n=", out ( 1) , " at r ank ", out ( 2)


END

Sample Program #7 - Fortran

Progr am Out putP: 2 mi n=1 at r ank 0

P: 7 max=8 at r ank 7


23/33

23

User-Defined Reduction Operators

Reducing using an arbitrary operator op

C function of type MPI _User _f unct i on:voi d my_oper at or ( voi d *i nvec, voi d *i nout vec, i nt *l en,

MPI _Dat at ype *dat at ype)

Fortran function of type:FUNCTI ON MY_OPERATOR ( I NVEC( *) , I NOUTVEC( *) , LEN, DATATYPE)

I NVEC( LEN) , I NOUTVEC( LEN)

I NTEGER LEN, DATATYPE


24/33

24

Reduction Operator Functions

Operator function for op must act as:f or ( i =1 t o l en)

i nout vec( i ) = i nout vec( i ) op i nvec( i )

Operator op need not commute

i nout vec argument acts as both a second input

operand as well as the output of the function

R i i U D fi d R d i O


25/33

25

Registering a User-Defined Reduction Operator

Operator handles have type MPI _Op orI NTEGER

Ifcommut e isTRUE, reduction may beperformed faster

C:i nt MPI _Op_cr eat e( MPI _User _f unct i on *f unct i on,i nt commut e, MPI _Op *op)

Fortran:EXTERNAL FUNCI NTEGER OP, I ERRORLOGI CAL COMMUTEMPI _OP_CREATE ( FUNC, COMMUTE, OP, I ERROR)

S l P #8 C


26/33

26

Sample Program #8 - C#i ncl ude t ypedef st r uct {

doubl e r eal , i mag;

} compl ex;

voi d cpr od( compl ex *i n, compl ex *i nout , i nt *l en, MPI _Dat at ype *dpt r ) {

i nt i ;compl ex c;

f or ( i =0; i


27/33

27

Sample Program #8 - C (cont.)MPI _Op myop;MPI _Datat ype ct ype;



MPI _Type_cont i guous( 2, MPI _DOUBLE, &ct ype) ;

MPI _Type_commi t ( &ct ype) ;MPI _Op_cr eat e( cpr od, TRUE, &myop) ;

r oot =2;

sour ce. r eal =r ank+1;

sour ce. i mag=r ank+2;

MPI _Reduce( &sour ce, &r esul t , 1, ct ype, myop, r oot , MPI _COMM_WORLD) ;i f ( r ank==r oot ) pr i nt f ( "PE: %d r esul t i s %l f + %l f i \ n", r ank, r esul t . r eal ,

r esul t . i mag) ;


}

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

P: 2 r esul t i s - 185. 000000 + - 180. 000000i

S l P #8 F


28/33

28

Sample Program #8 - FortranPROGRAM User OP

I NCLUDE ' mpi f . h'


i nt eger sour ce, r esl t

ext er nal di gi t

l ogi cal commut e

i nt eger myop


CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )

CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )

commut e=. t r ue.

cal l MPI _OP_CREATE( di gi t , commut e, myop, er r )

source=( r ank+1) **2

cal l MPI _BARRI ER( MPI _COM_WORLD, er r )cal l MPI _SCAN( sour ce, r esl t , 1, MPI _I NTEGER, myop, MPI _COMM_WORLD, er r )

pr i nt * , "P: " , r ank, " my r esul t i s " , r es l t


END

i nt eger f unct i on di gi t ( i n, i nout , l en, t ype)

i nt eger i n( l en) , i nout ( l en)i nt eger l en, t ype

do i =1, l en

i nout ( i ) =mod( ( i n( i ) +i nout ( i ) ) , 10)

end do

di gi t = 5

end

Progr am Out putP: 6 my r esul t i s 0P: 5 my r esul t i s 1P: 7 my r esul t i s 4P: 1 my r esul t i s 5P: 3 my r esul t i s 0P: 2 my r esul t i s 4P: 4 my r esul t i s 5P: 0 my r esul t i s 1

V ri nt f MPI REDUCE


29/33

29

Variants ofMPI_REDUCE

MPI _ALLREDUCE - no root process (all get results) MPI _REDUCE_SCATTER - multiple results are scattered

MPI _SCAN - parallel prefix

MPI ALLREDUCE


30/33

30

MPI_ALLREDUCE

MPI REDUCE SCATTER


31/33

31

MPI_REDUCE_SCATTER

MPI SCAN


32/33

32

MPI_SCAN

Class Exercise: Collective Ring


33/33

33

Class Exercise: Collective Ring

Rewrite the Structured Ring program to use MPIglobal reduction to perform its global sums

Extra credit: Rewrite it so that each processcomputes a partial sum

Extra extra credit: Rewrite this so that each

process prints out its partial result in rank order

mpi 0802 mod collective

Documents