parallel data transfer in the model coupling toolkit robert l. jacob j. walter larson mathematics...

Parallel Data Transferin the Model Coupling

ToolkitRobert L. JacobJ. Walter Larson

Mathematics and Computer Science DivisionArgonne National Laboratory

A Typical Coupled Model

AGCM

Temp, Rain, Radiation, Wind

Sea Ice

Temp, Albedo

OGCMTemp, Currents, Salinity

Land Surface

Temp, Soil Moisture

Coupler

CouplerCoupler

Coupler

Coupler

Basic Function of a Coupler:

Move data from a model that has it to a model that needs it

Other, minor functions

•Interpolate data onto different grids•Calculate data needed by one model but not calculated by any

other model•Control model integration•Ensure conservation•(many other misc. functions)

Component 1

Component 2Coupler

Memory

Coupling and Computer Memory: Shared memory

Component 1Component 2Coupler

Memory Memory Memory

(lightly distributed)

C

omponent 1

Com

ponent 2

Coupler

Coupling and Computer memory: Distributed memory

Memory

Memory

Memory

Memory

Ex: NCAR Parallel Climate Model

Com

ponent 1

Com

ponent 2

Coupler

Coupling and Computer memory: Distributed memory

Ex: NCAR Climate System Model

Memory

Memory

Memory

Memory

Memory

MemoryMemory

Com

ponent 1

Com

ponent 2

Coupler

Coupling and Computer memory: Distributed memory

Memory

Memory

Memory

Memory

Memory

MemoryMemory

Memory

How do you decide where the data goes? On any decomposition?

Idea: For efficiency and scalability, send only the data needed to each node.

MCT provides data types and algorithmsto find this data transfer pattern for any

two decompositions of any numerical grid

•GlobalSegmentMap: Describes decomposition•Router: Describes data transfer between GSMaps•Attribute Vector: Holds data to be sent/received

GlobalSegMap Example

Pe_loc start length

0 1 8

1 9 8

2 17 40

1

2

1 2 3 45 6 7 89 10 11 12

13 14 15 1617 18 19 20

1 2 3 45 6 7 89 10 11 12

13 14 15 1617 18 19 20

0 1

2 3

Pe_loc start length

0 1 2

0 5 2

0 9 2

1 3 2

1 7 2

1 11 2

2 13 2

2 17 2

3 15 2

3 19 2

Numbering of gridpoints

Processor Decomposition

Total number of segments = 10

Total numberof segments = 3

1 2 3 45 6 7 89 10 11 12

13 14 15 1617 18 19 20

0 1

2 3

4

5

6

1 2 3 45 6 7 89 10 11 12

13 14 15 1617 18 19 20

Pe_loc start length

0 1 2

0 5 2

0 9 2

1 3 2

1 7 2

1 11 2

2 13 2

2 17 2

3 15 2

3 19 2

GlobalSegMap

Pe_loc start length

0 1 8

1 9 8

2 17 4

GlobalSegMap

ProcessorDecomposition

Atmosphere Coupler

Router

Router Example

Initializing a Router

1 2 3 45 6 7 89 10 11 12

13 14 15 1617 18 19 20

0 1

2 3

3 Segments: (1,2) (5,6)

(9,10)

Each processor is given a copy of other component’s GlobalSegMap.Each processor compares each of its segments against other component’s entire GSMap.Each processor constructs a list of what processors it must send to, and which parts of its GSMap must be sent to eachThis information is stored in the Router datatypeRouter resembles a GSMap: it’s a list of remote processors and starts and counts of local index values

NOTE: Router is between two different GSMaps of the same grid

T q u

Real Data

Mask Veg typ

Integer Data

Attribute Vector

Pe_loc start length

0 1 2

0 5 2

0 9 2

Relationship Between GlobalSegMap and AttributeVector

Processor 0’s portionof GlobalSegMap

6One Attribute

from processor 0’sAttVect

1

2

3

4

5

6Local index

1

2

5

6

9

10

GSMap values

Rules for using GlobalSegmentMap and MCT Parallel Data Transfer

• Each grid’s gridpoints are numbered with unique integers

• Model’s which use the same grid must number points the same way

• Components can exchange only real and integer data as groups of vectors in an AttributeVector

Features

Any number of components, any grid, any decomposition, any number of processors-per-component is allowed

ATM (M nodes)

Call MCT World

Define GlobalSegMapDefine AttrVectDefine Router

Call MCT World

Define GlobalSegMapsDefine AttrVectsDefine RoutersDefine AccumulatorsRead Matrix elements

Call MCT World

Define GlobalSegMapDefine AttrVectDefine Router

CPL (N nodes) OCN (P nodes)

Initialization

Read Atmosphere Data

Read Ocean Data

MCT_Send(AtrVect, Router) MCT_Recv(AAtrVect, ARouter)MCT_Recv(OAtrVect, ORouter)InterpolateCompute FluxesMCT_Send(AAtrVect, ARouter)AccumulateMCT_Send(OAtrVect, ORouter)

MCT_Send(AtrVect, Router)

MCT_Recv(AtrVect, Router)

MCT_Recv(AtrVect, Router)

Data Transfer in the MCT unit tester

Identical decompositions

Atmosphere on 4 nodes, coupler on 4 nodes

Total transfer time for easy case

No. of atmosphere nodes = No. of coupler nodes

Two very different decompositions

Atmosphere on 16 nodes, coupler on 4 = 20 nodes total

Total transfer time for pathological case

Atmosphere on 8 nodes, 16 nodes, 32 nodes

Future of MCT Data Transfer

•Support for Parallel Transpose•Intra-model load balancing•Increase Performance/Scalability•Automatic GlobalSegMap generation for

standard decompositions

ATM (M nodes)

Call MCT World

Define GlobalSegMapDefine AttrVectDefine Router

Call MCT World

Define GlobalSegMapsDefine AttrVectsDefine RoutersDefine AccumulatorsRead Matrix elements

Call MCT World

Define GlobalSegMapDefine AttrVectDefine Router

CPL (N nodes) OCN (P nodes)

Initialization

Read Atmosphere Data

Read Ocean Data

MCT_Send(AtrVect, Router) MCT_Recv(AAtrVect, ARouter)MCT_Recv(OAtrVect, ORouter)InterpolateCompute FluxesMCT_Send(AAtrVect, ARouter)AccumulateMCT_Send(OAtrVect, ORouter)

MCT_Recv(AAtrVect, Router)InterpolateCompute FluxesMCT_Send(AAtrVect, Router)AccumulateMCT_Recv(AtrVect, Router)

Future of MCT Data Transfer?