session 3: nuopc interoperability layer instructors: rocky dunlap and fei liu noaa cooperative...

Session 3: NUOPC Interoperability Layer

Instructors: Rocky Dunlap and Fei LiuNOAA Cooperative Institute for Research in Environmental Sciences

University of Colorado, Boulder

Training at NRL MontereyAugust 5-6, 2015

2

Two Day Overview

• Day 1 morning– Overview of ESMF and component-based modeling– Coding exercise: Run a single-component ESMF application

• Day 1 afternoon– ESMF distributed data classes and wrapping your model data types

with ESMF data types– Regridding, LocStream– Coding exercise: Grid/Mesh construction

• Day 2 morning– Overview of NUOPC interoperability layer– Coding exercise: Run a NUOPC prototype application

• Day 2 afternoon– Questions and time for discussion of ESMF/NUOPC in COAMPS and

other NRL modeling applications

3

Reading an ESMF Componentsubroutine SetServices(comp, rc) type(ESMF_GridComp) :: comp ! must not be optional integer, intent(out) :: rc ! must not be optional

! Set the entry points for standard ESMF Component methods call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_INITIALIZE, & userRoutine=AtmInit, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_RUN, & userRoutine=AtmDyn, phase=1, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_RUN, & userRoutine=AtmPhys, phase=2, rc=rc) call ESMF_GridCompSetEntryPoint(comp, ESMF_METHOD_FINALIZE, & userRoutine=AtmFinal, rc=rc) rc = ESMF_SUCCESSend subroutine

subroutine AtmDyn(comp, importState, exportState, clock, rc) type(ESMF_GridComp) :: comp type(ESMF_State) :: importState, exportState type(ESMF_Clock) :: clock integer :: rc

call model_dyn() ! call the dynamics

end subroutine

4

Reading an ESMF Driverprogram AtmDriver

use ATM, only: ATM_SetServices => SetServices…call ESMF_GridCompSetServices(comp, userRoutine=ATM_SetServices, rc=rc)…call ESMF_GridCompInitialize(comp, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc)

do while (.not. ESMF_ClockIsStopTime(clock)) call ESMF_GridCompRun(comp, phase=1, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc) call ESMF_GridCompRun(comp, phase=2, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc)

…enddo

call ESMF_GridCompFinalize(comp, importState=importState, & exportState=exportState, clock=clock, userRc=userrc, rc=rc)

…end program

5

This Session…• Understand the goals of the NUOPC Layer.• Understand the purpose of each of the four generic

components: Driver, Model, Connector, and Mediator and why they need to be “specialized” to provide application-specific code.

• Learn how NUOPC Components can be assembled into different configurations with customized run sequences.

• Learn about the “handshaking” steps in the initialization sequence and which ones require or allow user specialization.

• Review a set of recommended steps for converting an existing model component into a NUOPC application.

6

National Unified Operational Prediction Capability

• ESMF– provides common data types (field, grid, mesh, clock, etc.) and

component interfaces (init, run, finalize)– BUT: there are few constraints on how to organize Components into

an application and what happens at each model phase– RESULT: even if two independent models are built with ESMF types,

interoperability is not guaranteed

• National Unified Operational Prediction Capability (NUOPC)Consortium of U.S. operational weather and water prediction centers– Both an agreement among agencies and a software layer

implementation– Accelerate research to operations and model reuse– Technical goal: Increase interoperability of ESMF-based applications

7

NUOPC Interoperability LayerNUOPC guarantees that Components provide: As in ESMF, a single, public entry point called

SetServices A set of standard initialize phases that can be

used to link fields across models and ensure model clock synchronization

A standard run phase (or phases) that ensures that incoming and outgoing fields are consistent with the Component’s clock.

A finalize method that deallocates memory and releases file handles.

A makefile fragment with a small number of variables used for compiling and linking against the Component.

ESMF API

NUOPC API

NUOPC is a layer on top of ESMF and is included with the ESMF distribution.

8

NUOPC Generic Components

Model: Implements a specific physical domain, e.g. atmosphere, ocean,

wave, ice

Mediator: Scientific coupling code (flux calculations, accumulation,

averaging, etc.) between (potentially multiple) Models

Connector: Connects pairs of components in one direction, e.g. Model

to/from Model, or Model to/from Mediator Executes simple transforms (Regrid/Redist)

Driver: Provides a harness for Models, Mediators, and Connectors

(supporting hierarchies) Coordinates initialize and run sequences

9

Example Configurations of NUOPC Components

a b c

a) A NUOPC Driver for two Models that share data via two Connectors

b) A four-Model system with centralized Mediator. The OCN and WAVE components share data directly without going through the Mediator

c) A complex system showing multiple Atmosphere components running as an ensemble. The highest level Mediator sees the ensemble as a single Model component.

10

Initialization Sequence• The Initialize Phase Definition (IPD) regulates the interaction of

NUOPC Components during their initialization process.• The IPD ensures that:

– Each Model advertises a set of required import and available export Fields, and that those Fields have standard names and associated units in the NUOPC Field Dictionary.

– Each Connector identifies matching Fields between Models and establishes a connection and communication RouteHandle.

– All Models have their import dependencies satisfied.– All Model Clock’s agree on the start time.– Two Models with connected Fields agree on who will provide the

grid structure (can be both).– Model Fields are initialized at the right time and a timestamped

according to the Model clock.– The Driver has established a run sequence.

11

Run Sequence• The Run Sequence is set dynamically

(during the initialize sequence) and controls the master time loop.

• Each Driver maintains a Run Sequence over its child Components.

• Run Sequences supported:– Simple loops in which all

Components exchange data at the same coupling interval

– Multiple timescales in which some Components communicate less frequently than others

– Explicit, semi-implicit, and implicit modes in which some Components may run ahead or lag behind to satisfy numerical constraints

12

Sequential and Concurrent Components• Drivers support both sequential and

concurrent execution of child Components.• The amount of concurrency is dependent

on:– whether Components are assigned disjoint

or overlapping PETs– data dependencies introduced by scientific

constraints, e.g., the frequency of coupling required

• Drivers have the option of assigning a PET List when a Component is added

13

Specialization

• Specialization is a way of customizing the generic NUOPC Components to meet your application’s requirements.

• A specialization either:– provides an implementation not provided by NUOPC,

or – overrides (replaces) a default behavior

• There are three ways to specialize a NUOPC Component:– by providing an entire initialize or run phase method– by registering a method at predefined specialization

points– by setting or modifying metadata attributes

14

Some Specialization Points

• NUOPC_Driver– label_SetModelServices – Add Components to a Driver– label_SetRunSequence – Change default run sequence

• NUOPC_Model– label_SetClock – Modify incoming clock, e.g., to change timestep length– label_DataInitialize – Initialize Fields in Export State– label_Advance – Take a timestep

• NUOPC_Connector– label_ComputeRouteHandle – Store custom redist, regrid, etc.– label_ExecuteRouteHandle – Execute redist, regrid, etc.

• NUOPC_Mediator– label_CheckImport – Verify import fields– label_Advance – Execute mediation code

15

Knowing When to Specialize

A key part of making a component NUOPC compliant is understanding the default behavior of the generic NUOPC components when it is necessary to specialize.

In this next part, we’ll walk through the initialize sequence of a NUOPC application with two Models and two Connectors.

This will give an idea of the basic control flow in a NUOPC application. Along the way, we’ll see important specialization points.

16

Walk through of 2-Model, Explicit Coupling Example

Architecture: One driver, two models, two connectors

Coupling: Explicit time dependency requires coupling exchange at beginning of each coupling interval.

Driver

2-Model, Explicit Coupling

17

Driver

ATM

OCN

Atm2O

cn

Ocn2Atm

start: 0 stop: 10

dt: 2

Init Phase 1* (Driver)(NUOPC provided)

MainApp

1. Set internal clock to copy of incoming clock, if present.

2. Specialization: SetModelServices

3. Optional specialization: SetRunSequence

4. Drive initialization sequence for child Components (Models, Mediators, Drivers, Connectors).

Run sequence:1. Atm2Ocn2. Ocn2Atm3. ATM4. OCN

* Some details left out to simplify.

Import: Export:

Export: Import:


18

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 2* (Model)(User provided)

MainApp


1. Advertise Fields in Import and Export States.

“Advertising” a Field does not allocate memory, but allows each Model to announce the Standard Names of required and provided Fields.

The NUOPC Field Dictionary is checked to verify that the advertised field names exist and have the correct units.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD



19

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 1* (Connector)(NUOPC provided)

MainApp


1. Construct the CplList attribute, which contains a list of Fields with matching Standard Names. These Fields are connected.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

CplList:SST

CplList:DNRAD



20

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 2* (Model)(User provided)

MainApp


1. Realize connected Import and Export Fields.

A connected field has a match in another Component.

“Realizing” a Field means that a Grid is defined and memory is allocated.

Fields that are not connected do not have to be realized.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

CplList:SST

CplList:DNRAD

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD



21

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 2* (Connector)(NUOPC provided)

MainApp


1. Construct srcFields and dstFields FieldBundles.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

2. Optional specialization: ComputeRouteHandle

3. If not specialized, compute default RouteHandle as bilinear Regrid between srcFields and dstFields.

Drivers also have the ability to modify CplList attributes to change connection options.

RH:Bilinear Regrid



22

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 3* (Model)(NUOPC provided)

MainApp


1. Set internal clock as copy of incoming clock.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

2. Optional specialization: SetClock (adjust timestep, set Alarms)

RH:Bilinear Regrid

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 2

3. Ensure all Import Fields are connected, raising an error if not.

start: 0 stop: 10

dt: 1Run sequence:1. Atm2Ocn2. Ocn2Atm3. ATM4. OCN


23

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Init Phase 4* (Model)(NUOPC provided)

MainApp


1. Optional specialization: Data Initialize (Fields in Export State).

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

2. Timestamp Fields in export state to current time on internal clock.

RH:Bilinear Regrid

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 1

Export:DNRAD[0]

Export:SST[0]



24

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Run* (Driver)(NUOPC provided)

MainApp


1. Override specialization: SetRunClock. Default is to set internal clock stop time to one timestep on incoming clock, if present.

Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

2. Drive timestepping loop from current time to stop time (internal clock), calling child Components according to Run Sequence.

RH:Bilinear Regrid

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 1

Export:DNRAD[0]

Export:SST[0]



25

Driver

ATM

OCN

start: 0 stop: 10

dt: 2

Run* (Connector)(NUOPC provided)

MainApp


Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

2. Default is to execute pre-computed RouteHandle between srcFields and dstFields

RH:Bilinear Regrid

start: 0 stop: 10

dt: 2

start: 0 stop: 10

dt: 1

Export:DNRAD[0]

Export:SST[0]

1. Optional specialization: ExecuteRouteHandle.


3. Update timestamp on dstFields to match srcFields.

Import:SST[0]

Import:DNRAD[0]


26

Driver

ATM

OCN

curr: 0 stop: 10

dt: 2

Run* (Model)(NUOPC provided)

MainApp


Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST[0]

Export:DNRAD

Export:SST

Import:DNRAD[0]

2. Override specialization: CheckImport.Default is to verify all Import Fields are at current time.

RH:Bilinear Regrid

curr: 0 stop: 10

dt: 2

curr: 0 stop: 10

dt: 1

Export:DNRAD[0]

Export:SST[0]

1. Override specialization: SetRunClock. Default is to set internal clock stop time to one timestep on incoming clock, if present.

curr: 0 stop: 2

dt: 1

curr: 0 stop: 2

dt: 2


3. Timestepping loop calls Advance specialization until stop time. Export Fields timestamped.

curr: 1 stop: 2

dt: 1

Export:DNRAD[1]

curr: 2 stop: 2

dt: 1

Export:DNRAD[2]

curr: 2 stop: 2

dt: 2

Export:SST[2]


27

Driver

ATM

OCN

curr: 0 stop: 10

dt: 2

Run* (Driver)(NUOPC provided)

MainApp


Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST[0]

Export:DNRAD

Export:SST

Import:DNRAD[0]

RH:Bilinear Regrid

curr: 0 stop: 10

dt: 2

curr: 0 stop: 10

dt: 1

Export:DNRAD[0]

Export:SST[0]

1. Driver advances its internal clock at end of run sequence.

The run sequence is repeated by the Driver until its internal clock reaches the stop time.

curr: 0 stop: 2

dt: 1

curr: 0 stop: 2

dt: 2


curr: 1 stop: 2

dt: 1

Export:DNRAD[1]

curr: 2 stop: 2

dt: 1

Export:DNRAD[2]

curr: 2 stop: 2

dt: 2

Export:SST[2]

curr: 2 stop: 10

dt: 2


28

Driver

ATM

OCN

curr: 2 stop: 10

dt: 2

Run* (Connector)(NUOPC provided)

MainApp


Atm2O

cn

Ocn2Atm

Import:SST

Export:DNRAD

Export:SST

Import:DNRAD

RH:Bilinear Regrid

Import:SST[0]

Export:DNRAD

Export:SST

Import:DNRAD[0]

RH:Bilinear Regrid

Export:DNRAD[0]

Export:SST[0]


Export:DNRAD[1]

curr: 2 stop: 2dt: 1

curr: 2 stop: 2dt: 2

Export:DNRAD[2]

Export:SST[2]

2. Default is to execute pre-computed RouteHandle between srcFields and dstFields

1. Optional specialization: ExecuteRouteHandle.

3. Update timestamp on dstFields to match srcFields.

Import:SST[2]

Import:DNRAD[2]

29

Some time later…


30

Driver

ATM

OCN

curr: 10 stop: 10

dt: 2

Finalize* (Driver)(NUOPC provided)

MainApp


Atm2O

cn

Ocn2Atm

RH:Bilinear Regrid

RH:Bilinear Regrid


Export:DNRAD[10]

curr: 10 stop: 10

dt: 1

curr: 10 stop: 10

dt: 2

Export:SST[10]

2. Optional specialization: Finalize (Driver itself)

1. Execute Finalize() on all child Components. (This can be specialized.)

3. Destroy Models and their States followed by Connectors. Internal cleanup.

Import:SST[8]

Import:DNRAD[8]

Import: Export:

Export: Import:

31

Making an Existing Model Component NUOPC Compliant

1. Create a NUOPC Model “cap” starting from a basic template.2. Do not try to implement the entire initialize sequencing up front –

instead, call your model’s existing initialize subroutine and provide stubs for any required NUOPC initialize phases.

3. Call your model’s update routine from the Advance specialization. If possible, use an offline mode (read from file) since the model will run in uncoupled mode.

4. Plug your Model “cap” into a generic NUOPC Driver and try to get through the run sequence.

5. Then, go back and split up the initialize phases as required by NUOPC. During the realize phase, you will need to create an ESMF_Grid and ESMF_Fields and hook them into your model variables.

6. Test the “cap” in coupled mode by adding another Model and Connectors to the generic Driver.

32

NUOPC Tools: Compliance Checker

The Compliance Checker is a run-time analysis tool provides detailed log output about the interaction of NUOPC components.

It is turned on a run time by setting the environment variable:$ export ESMF_RUNTIME_COMPLIANCECHECK=ON

It reports on:• which init, run, finalize phases are registered• timekeeping• fields passed between components via import/export states• component-, state-, and field-level metadata

33

Sample Compliance Checker Output

20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM:>START register compliance check.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: phase Zero for Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 5 phase(s) of Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 1 phase(s) of Run registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: 1 phase(s) of Finalize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM:>STOP register compliance check.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED:>START register compliance check.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: phase Zero for Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 3 phase(s) of Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 1 phase(s) of Run registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED: 1 phase(s) of Finalize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM2MED:>STOP register compliance check.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM:>START register compliance check.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: phase Zero for Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 3 phase(s) of Initialize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 1 phase(s) of Run registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM: 1 phase(s) of Finalize registered.20131108 172844.458 INFO PET0 COMPLIANCECHECKER:|->|->|->:MED2ATM:>STOP register compliance check.20131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: >START InitializePrologue for phase= 020131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState name: modelComp 1 Import State20131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState stateintent: ESMF_STATEINTENT_IMPORT20131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: importState itemCount: 020131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState name: modelComp 1 Export State20131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState stateintent: ESMF_STATEINTENT_EXPORT20131108 172844.459 INFO PET0 COMPLIANCECHECKER:|->|->|->:ATM: exportState itemCount: 0

34

NUOPC Tools:Component Explorer

The Component Explorer is a run-time tool that can be used to gain insight into a NUOPC Component.

It acts like a NUOPC Driver, but reports component information to standard out.

A configuration file (explorer.config) is used to set up the clock and has options for whether or not to enter the run phase. (Most relevant information is available during the initialize sequence.)

$ ./nuopcExplorerScript <component-mk-file>$ mpirun -np X ./nuopcExplorerApp

35

Component Explorer OutputNUOPC Component Explorer App ---------------------------- Exploring a component with a Fortran module front... Model component # 1 InitializePhaseMap: IPDv00p1=1 IPDv00p2=2 IPDv00p3=3 IPDv00p4=4 Model component # 1 // name = ocnA ocnA: <LongName> : Attribute is present but NOT set! ocnA: <ShortName> : Attribute is present but NOT set! ocnA: <Description> : Attribute is present but NOT set! -------- ocnA: importState // itemCount = 2 ocnA: importState // item # 001 // [FIELD] name = pmsl <StandardName> = air_pressure_at_sea_level <Units> = Pa <LongName> = Air Pressure at Sea Level <ShortName> = pmsl ocnA: importState // item # 002 // [FIELD] name = rsns <StandardName> = surface_net_downward_shortwave_flux <Units> = W m-2 <LongName> = Surface Net Downward Shortwave Flux <ShortName> = rsns -------- ocnA: exportState // itemCount = 1 ocnA: exportState // item # 001 // [FIELD] name = sst <StandardName> = sea_surface_temperature <Units> = K <LongName> = Sea Surface Temperature <ShortName> = sst

36

NUOPC Tools: CupidCupid is a static analysis and code generation tool the provides guidance to developers working with NUOPC application code.

Understand what’s there:Reverse Engineer an

ESMF/NUOPC Application

Check for issues:Static Compliance Checking

NUOPC View• compact outline of

NUOPC component source code

• linked with source code editor

• contextual reference documentation

• compliance issues shown in red

Write compliant code:In-place Code Generation

40

NUOPC Resources

• Website: https://www.earthsystemcog.org/projects/nuopc/• Reference Manual: https://

www.earthsystemcog.org/projects/nuopc/refmans• Prototype Codes description: https://

www.earthsystemcog.org/projects/nuopc/proto_codesPrototype Codes repository: http://sourceforge.net/p/esmfcontrib/svn/HEAD/tree/NUOPC/tags/ESMF_7_0_0_beta_snapshot_50/

• Technical support email: [email protected]

https://www.earthsystemcog.org/projects/nuopc/

https://www.earthsystemcog.org/projects/nuopc/

https://www.earthsystemcog.org/projects/nuopc/refmans

https://www.earthsystemcog.org/projects/nuopc/refmans

https://www.earthsystemcog.org/projects/nuopc/proto_codes



http://sourceforge.net/p/esmfcontrib/svn/HEAD/tree/NUOPC/tags/ESMF_7_0_0_beta_snapshot_50/



mailto:[email protected]

42

Extra Slides

43

NUOPC Generic Components

ESMF_GridComp ESMF_CplComp

NUOPC_Driver

NUOPC_Model NUOPC_Mediator NUOPC_Connector

NUOPC Generic Components are specialized versions of GridComp and CplComp.

session 3: nuopc interoperability layer instructors: rocky dunlap and fei liu noaa cooperative...

Documents

esmf types

rc typeesmf

rc enddo

morning overview of

rc end program

esmf data types regridding

exportstate typeesmf

clock integer