parallel data transfer in the model coupling toolkit robert l. jacob j. walter larson mathematics...
TRANSCRIPT
Parallel Data Transferin the Model Coupling
ToolkitRobert L. JacobJ. Walter Larson
Mathematics and Computer Science DivisionArgonne National Laboratory
A Typical Coupled Model
AGCM
Temp, Rain, Radiation, Wind
Sea Ice
Temp, Albedo
OGCMTemp, Currents, Salinity
Land Surface
Temp, Soil Moisture
Coupler
CouplerCoupler
Coupler
Coupler
Basic Function of a Coupler:
Move data from a model that has it to a model that needs it
Other, minor functions
•Interpolate data onto different grids•Calculate data needed by one model but not calculated by any
other model•Control model integration•Ensure conservation•(many other misc. functions)
Component 1
Component 2Coupler
Memory
Coupling and Computer Memory: Shared memory
Component 1Component 2Coupler
Memory Memory Memory
(lightly distributed)
C
omponent 1
Com
ponent 2
Coupler
Coupling and Computer memory: Distributed memory
Memory
Memory
Memory
Memory
Ex: NCAR Parallel Climate Model
Com
ponent 1
Com
ponent 2
Coupler
Coupling and Computer memory: Distributed memory
Ex: NCAR Climate System Model
Memory
Memory
Memory
Memory
Memory
MemoryMemory
Com
ponent 1
Com
ponent 2
Coupler
Coupling and Computer memory: Distributed memory
Memory
Memory
Memory
Memory
Memory
MemoryMemory
Memory
How do you decide where the data goes? On any decomposition?
Idea: For efficiency and scalability, send only the data needed to each node.
MCT provides data types and algorithmsto find this data transfer pattern for any
two decompositions of any numerical grid
•GlobalSegmentMap: Describes decomposition•Router: Describes data transfer between GSMaps•Attribute Vector: Holds data to be sent/received
GlobalSegMap Example
Pe_loc start length
0 1 8
1 9 8
2 17 40
1
2
1 2 3 45 6 7 89 10 11 12
13 14 15 1617 18 19 20
1 2 3 45 6 7 89 10 11 12
13 14 15 1617 18 19 20
0 1
2 3
Pe_loc start length
0 1 2
0 5 2
0 9 2
1 3 2
1 7 2
1 11 2
2 13 2
2 17 2
3 15 2
3 19 2
Numbering of gridpoints
Processor Decomposition
Total number of segments = 10
Total numberof segments = 3
1 2 3 45 6 7 89 10 11 12
13 14 15 1617 18 19 20
0 1
2 3
4
5
6
1 2 3 45 6 7 89 10 11 12
13 14 15 1617 18 19 20
Pe_loc start length
0 1 2
0 5 2
0 9 2
1 3 2
1 7 2
1 11 2
2 13 2
2 17 2
3 15 2
3 19 2
GlobalSegMap
Pe_loc start length
0 1 8
1 9 8
2 17 4
GlobalSegMap
ProcessorDecomposition
Atmosphere Coupler
Router
Router Example
Initializing a Router
1 2 3 45 6 7 89 10 11 12
13 14 15 1617 18 19 20
0 1
2 3
3 Segments: (1,2) (5,6)
(9,10)
Each processor is given a copy of other component’s GlobalSegMap.Each processor compares each of its segments against other component’s entire GSMap.Each processor constructs a list of what processors it must send to, and which parts of its GSMap must be sent to eachThis information is stored in the Router datatypeRouter resembles a GSMap: it’s a list of remote processors and starts and counts of local index values
NOTE: Router is between two different GSMaps of the same grid
T q u
Real Data
Mask Veg typ
Integer Data
Attribute Vector
Pe_loc start length
0 1 2
0 5 2
0 9 2
Relationship Between GlobalSegMap and AttributeVector
Processor 0’s portionof GlobalSegMap
6One Attribute
from processor 0’sAttVect
1
2
3
4
5
6Local index
1
2
5
6
9
10
GSMap values
Rules for using GlobalSegmentMap and MCT Parallel Data Transfer
• Each grid’s gridpoints are numbered with unique integers
• Model’s which use the same grid must number points the same way
• Components can exchange only real and integer data as groups of vectors in an AttributeVector
Features
Any number of components, any grid, any decomposition, any number of processors-per-component is allowed
ATM (M nodes)
Call MCT World
Define GlobalSegMapDefine AttrVectDefine Router
Call MCT World
Define GlobalSegMapsDefine AttrVectsDefine RoutersDefine AccumulatorsRead Matrix elements
Call MCT World
Define GlobalSegMapDefine AttrVectDefine Router
CPL (N nodes) OCN (P nodes)
Initialization
Read Atmosphere Data
Read Ocean Data
MCT_Send(AtrVect, Router) MCT_Recv(AAtrVect, ARouter)MCT_Recv(OAtrVect, ORouter)InterpolateCompute FluxesMCT_Send(AAtrVect, ARouter)AccumulateMCT_Send(OAtrVect, ORouter)
MCT_Send(AtrVect, Router)
MCT_Recv(AtrVect, Router)
MCT_Recv(AtrVect, Router)
Data Transfer in the MCT unit tester
Identical decompositions
Atmosphere on 4 nodes, coupler on 4 nodes
Total transfer time for easy case
No. of atmosphere nodes = No. of coupler nodes
Two very different decompositions
Atmosphere on 16 nodes, coupler on 4 = 20 nodes total
Total transfer time for pathological case
Atmosphere on 8 nodes, 16 nodes, 32 nodes
Future of MCT Data Transfer
•Support for Parallel Transpose•Intra-model load balancing•Increase Performance/Scalability•Automatic GlobalSegMap generation for
standard decompositions
ATM (M nodes)
Call MCT World
Define GlobalSegMapDefine AttrVectDefine Router
Call MCT World
Define GlobalSegMapsDefine AttrVectsDefine RoutersDefine AccumulatorsRead Matrix elements
Call MCT World
Define GlobalSegMapDefine AttrVectDefine Router
CPL (N nodes) OCN (P nodes)
Initialization
Read Atmosphere Data
Read Ocean Data
MCT_Send(AtrVect, Router) MCT_Recv(AAtrVect, ARouter)MCT_Recv(OAtrVect, ORouter)InterpolateCompute FluxesMCT_Send(AAtrVect, ARouter)AccumulateMCT_Send(OAtrVect, ORouter)
MCT_Recv(AAtrVect, Router)InterpolateCompute FluxesMCT_Send(AAtrVect, Router)AccumulateMCT_Recv(AtrVect, Router)
Future of MCT Data Transfer?