Introduction to the Earth System Modeling Framework
Nancy Collins [email protected] July 22, 2005
ClimateData Assimilation
Weather
2
Goals of this Tutorial1. To give future ESMF users an understanding of the background,
goals, and scope of the ESMF project2. To review the status of the ESMF software implementation and
current application adoption efforts 3. To outline the overall design and principles underlying the ESMF
software 4. To describe the major classes and functions of ESMF in sufficient
detail to give future users an understanding of how ESMF could be utilized in their own codes
5. To describe in steps how a user code prepares for using ESMF, incorporates ESMF, and runs under ESMF
6. To identify ESMF resources available to users such as documentation, mailing lists, and support staff
3
For More Basic Information …ESMF Website
http://www.esmf.ucar.edu
See this site for downloads, documentation, references, repositories, meeting schedules, test archives, and just about anything else you need to know about ESMF.
References to ESMF source code and documentation in this tutorial correspond to ESMF Version 2.2.0.
4
1 BACKGROUND, GOALS, AND SCOPE
• Overview• ESMF and the Community• Development Status• Exercises
5
Motivation and Context
In climate research and NWP... increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system
In computing technology... increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable computing architectures
In software …development of first-generation frameworks, such as FMS, GEMS, CCA and WRF, that encourage software reuse and interoperability
6
What is ESMF?• ESMF provides tools for turning model codes
into components with standard interfaces and standard drivers.
• ESMF provides data structures and common utilities that components use for routine services such as data communications, regridding, time management and message logging.
ESMF InfrastructureData Classes: Bundle, Field, Grid, Array
Utility Classes: Clock, LogErr, DELayout, Machine
ESMF SuperstructureAppDriver
Component Classes: GridComp, CplComp, State
User Code
ESMF GOALS
1. Increase scientific productivity by making model components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers.
2. Promote new scientific opportunities and services through community building and increased interoperability of codes (impacts in collaboration, code validation and tuning, teaching, migration from research to operations)
7
GEOS-5
surface fvcore gravity_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
• Each box is an ESMF component• Every component has a standard interface so that it is swappable• Data in and out of components are packaged as state types with user-defined fields• New components can easily be added to the hierarchical system• Coupling tools include regridding and redistribution methods
Application Example: GEOS-5 AGCM
8
Why Should I Adopt ESMF If I Already Have a Working Model?• There is an emerging pool of other ESMF-based science components that you
will be able to interoperate with to create applications - a framework for interoperability is only as valuable as the set of groups that use it.
• It will reduce the amount of infrastructure code that you need to maintain and write, and allow you to focus more resources on science development.
• ESMF provides solutions to two of the hardest problems in model development: structuring large, multi-component applications so that they are easy to use and extend, and achieving performance portability on a wide variety of parallel architectures.
• It may be better software (better features, better performance portability, better tested, better documented and better funded into the future) than the infrastructure software that you are currently using.
• Community development and use means that the ESMF software is widely reviewed and tested, and that you can leverage contributions from other groups.
9
1 BACKGROUND, GOALS, AND SCOPE
• Overview• ESMF and the Community• Development Status• Exercises
10
Growing ESMF Customer Base
• Original ESMF applications:NOAA GFDL atmospheresNOAA GFDL MOM4 oceanNOAA NCEP atmospheres, analysesNASA GMAO models and GEOS-5NASA/COLA Poseidon oceanLANL POP oceanNCAR WRFNCAR CCSMMITgcm atmosphere and ocean
• Other groups using ESMF:NASA GISSUCLACSUNASA Land Information Systems (LIS) projectNOAA Integrated Dynamics in Earth’s Atmosphere (IDEA) project, more…
• New applications coming in during FY05 through the newly funded, ESMF-based DoD Battlespace Environments Institute (BEI):DoD Navy HYCOM oceanDoD Navy NOGAPS atmosphereDoD Navy COAMPS coupled atm-ocean DoD Air Force GAIM ionosphereDoD Air Force HAF solar windDoD Army ERDC WASH123 watershed
• More new applications will begin adopting ESMF during FY06 through the ESMF-based NASA Modeling Analysis and Prediction (MAP) Climate Variability and Change program.
• Further growth of the customer base is anticipated through development of an ESMF-based Space Weather computational environment.
11
ESMF ImpactsESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: • Climate prediction• Weather forecasting• Seasonal prediction• Basic Earth and planetary system research at various time and spatial scales• Emergency response• Ecosystem modeling• Battlespace simulation and integrated Earth/space forecasting• Space weather (through coordination with related space weather frameworks)• Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks, e.g. CCA
12
Open Source Development• Open source license (GPL) • Open source environment (SourceForge)• Open repositories: web-browsable CVS repositories accessible from the
ESMF website ◦ for source code◦ for contributions (currently porting contributions and performance testing)
• Open development priorities and schedule: priorities set based on user meetings, telecons, and mailing list discussions, web-browsable task lists
• Open testing: 1000+ tests are bundled with the ESMF distribution and can be run by users
• Open port status: results of nightly tests on many platforms are web-browsable• Open metrics: test coverage, lines of code, requirements status are updated
regularly and are web-browsable
13
Open Source Constraints• ESMF does not allow unmoderated check-ins to its main source
CVS repository (though there is minimal check-in oversight for the contributions repository)
• ESMF has a co-located, line managed Core Team whose members are dedicated to framework implementation and support – it does not rely on volunteer labor
• ESMF actively sets priorities based on user needs and feedback• ESMF requires that contributions follow project conventions and
standards for code and documentation• ESMF schedules regular releases and meetings
The above are necessary for development to proceed at the pace desired by sponsors and users, and to provide the level of quality and customer support necessary for codes in this domain
14
1 BACKGROUND, GOALS, AND SCOPE
• Overview• ESMF and the Community• Development Status• Exercises
15
Latest Information
For scheduling and release information, see:
http://www.esmf.ucar.edu > Development
This includes latest releases, known bugs, supported platforms.
Task lists, bug reports, and support requests are tracked on the ESMF SourceForge site:
http://sourceforge.net/projects/esmf
16
ESMF Development Status• Overall architecture is well-defined and well-accepted• Components and low-level communications stable• Logically rectangular grids with regular and arbitrary distributions
implemented• On-line parallel regridding (bilinear, 1st order conservative) completed
and optimized• Other parallel methods, e.g. halo, redistribution, low-level comms
implemented• Utilities such as time manager, logging, and configuration manager
usable and adding features• Virtual machine with interface to shared / distributed memory
implemented, hooks for load balancing implemented
17
ESMF Platform Support• IBM AIX (32 and 64 bit addressing) • SGI IRIX64 (32 and 64 bit addressing) • SGI Altix (64 bit addressing) • Cray X1 (64 bit addressing)• Compaq OSF1 (64 bit addressing) • Linux Intel (32 and 64 bit addressing, with mpich and lam) • Linux PGI (32 and 64 bit addressing, with mpich) • Linux NAG (32 bit addressing, with mpich) • Linux Absoft (32 bit addressing, with mpich) • Linux Lahey (32 bit addressing, with mpich) • Mac OS X with xlf (32 bit addressing, with lam)• Mac OS X with absoft (32 bit addressing, with lam)• Mac OS X with NAG (32 bit addressing, with lam)
• User-contributed g95 support
18
ESMF Distribution Summary• Fortran interfaces and complete documentation• Many C++ interfaces, no manuals yet• Serial or parallel execution (mpiuni stub library)• Sequential or concurrent execution• Single executable (SPMD) support
19
Some Metrics …• Test suite currently consists of
◦ ~1200 unit tests◦ ~15 system tests◦ ~35 examples
runs every night on ~12 platforms• ~289 ESMF interfaces implemented, ~276 fully or partially tested,
~95% fully or partially tested.• ~160,000 SLOC• ~1000 downloads
20
ESMF Near-Term Priorities, FY05/06• Reworked design and implementation of array / grid / field
interfaces and array-level communications • Optimized regridding and low-level communications• Grid masks and merges• Unstructured grids• Read/write interpolation weights and grid specifications
21
Planned ESMF Extensions
1. Looser couplings: support for multiple executable and Grid-enabled versions of ESMF
2. Support for representing, partitioning, communicating with, and regridding unstructured grids and semi-structured grids
3. Support for advanced I/O, including support for asynchronous I/O, checkpoint/restart, and multiple archival mechanisms (e.g. NetCDF, HDF5, binary, etc.)
4. Advanced support for data assimilation systems, including data structures for observational data and adjoints for ESMF methods
5. Support for nested, moving grids and adaptive grids 6. Support for regridding in three dimensions and between different
coordinate systems7. Advanced optimization and load balancing
22
1 BACKGROUND, GOALS, AND SCOPE
• Overview• ESMF and the Community• Development Status• Exercises
23
Exercises1. Sketch a diagram of the major components in your
application and how they are connected.2. Introduction of tutorial participants.
24
Application Diagram
25
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
26
Computational Characteristicsof Weather/Climate• Mix of global transforms and local communications
• Load balancing for diurnal cycle, event (e.g. storm) tracking• Applications typically require 10s of GFLOPS,
100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs• Required Unix/Linux platforms span laptop to
Earth Simulator• Multi-component applications: component
hierarchies, ensembles, and exchanges;components in multiple contexts
• Data and grid transformations between components
• Applications may be MPMD/SPMD, concurrent/sequential, combinations
• Parallelization via MPI, OpenMP, shmem, combinations• Large applications (typically 100,000+ lines of source code)
Platforms
assim
sea iceocean
landatm
physics dycore
assim_atm
atmland
Seasonal Forecastcoupler
27
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
28
Design Strategy:Hierarchical ApplicationsSince each ESMF application is also a Gridded Component, entire ESMF applications can be nested within larger applications. This strategy can be used to systematically compose very large, multi-component codes.
GEOS-5
surface fvcore gravity_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
29
Design Strategy: ModularityGridded Components don’t have access to the internals of other Gridded Components, and don’t store any coupling information. Gridded Components pass their States to other components through their argument list.Since components are not hard-wired into particular configurations and do not carry coupling information, components can be used more easily in multiple contexts.
atm_comp
NWP application
Seasonal prediction
Standalone for basic research
30
Design Strategy: Flexibility• Users write their own drivers as well as their own Gridded Components and
Coupler Components• Users decide on their own control flow
Ocean
SeaIce
Land Coupler
Atmosphere
DATA DATA
D A T A
D A T A
Hub and Spokes Coupling
Land AtmLandCoupler Atmosphere DATA DATA Pairwise Coupling
31
Design Strategy:Communication Within ComponentsAll communication in ESMF is handled within components. This means that if an atmosphere is coupled to an ocean, then the Coupler Component is defined on both atmosphere and ocean processors.
ocn_comp atm_comp
processors
atm2ocn _coupler
32
Design Strategy:Uniform Communication API• The same programming interface is used for shared memory, distributed
memory, and combinations thereof. This buffers the user from variations and changes in the underlying platforms.
• The idea is to create interfaces that are performance sensitive to machine architectures without being discouragingly complicated.
• Users can use their own OpenMP and MPI directives together with ESMF communications
ESMF sets up communications in a way that is sensitive to the computing platform and the application structure
33
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
34
Elements of Parallelism: Serial vs. Parallel• Computing platforms may possess multiple processors, some
or all of which may share the same memory pools• There can be multiple threads of execution and multiple
threads of execution per processor• Software like MPI and OpenMP is commonly used for
parallelization• Programs can run in a serial fashion, with one thread of
execution, or in parallel on multiple threads of execution.• Because of these and other complexities, terms are needed
for units of parallel execution.
35
Elements of Parallelism: PETs Persistent Execution Thread (PET)• Path for executing an instruction sequence• For many applications, a PET can be thought of as a
processor• Sets of PETs are represented by the Virtual Machine (VM)
class• Serial applications run on one PET, parallel applications run
on multiple PETs
36
Elements of Parallelism: Sequential vs. ConcurrentIn sequential mode components run one after the other on the same set of PETs.
GridComp “Atmosphere”
GridComp “Hurricane Model”
GridComp “Ocean”
CplComp “Atm-Ocean Coupler”
LOOP Call Run
Run
Run
Run
Run
AppDriv er (“Main”)
Call Run
1 2 3 5 4 6
PETs
T i m
e
7 8 9
37
Elements of Parallelism: Sequential vs. ConcurrentIn concurrent mode components run at the same time on different sets of PETs
GridComp “Atmosphere”
GridComp “Hurricane Model”
GridComp “Ocean”
CplComp “Atm-Ocean Coupler”
LOOP Call Run
Run
Run Run
Run
AppDriver (“Main”)
Call Run
1 2 3 5 4 6
PETs
T i m
e
7 8 9
38
Elements of Parallelism: DEsDecomposition Element (DE) • In ESMF a data decomposition is represented as a set of Decomposition
Elements (DEs).• Sets of DEs are represented by the DELayout class.• DELayouts define how data is mapped to PETs.• In many applications there is one DE per PET.
1 2 3 5 4 6
PETs
7 8 9
1 2 3 5 4 6
DEs
7 8 9
Temperature Field T
VM with 9 PETs
1 x 9 DELay out
4 x 9 f ield T1 T10 T19 T28
T2 T11 T20 T29
T3 T12 T21 T30
T4 T13 T22 T31
T5 T14 T23 T32
T6 T15 T24 T33
T7 T16 T25 T34
T8 T17 T26 T35
T9 T18 T27 T36
39
Elements of Parallelism: DEsMore complex DELayouts:• Users can define more than one DE per PET for cache blocking and chunking• DELayouts can define a topology of decomposition (i.e., decompose in both x and y)
1 2 3 5 4 6
PETs
7 8 9
10 11 12 14 13 15
DEs
16 17 18
Temperature Field T
1 2 3 5 4 6 7 8 9
VM with 9 PETs
2 x 9 DELay out
4 x 9 f ield
T1 T10
T19 T28
T2 T11
T20 T29
T3 T12
T21 T30
T4 T13
T22 T31
T5 T14
T23 T32
T6 T15
T24 T33
T7 T16
T25 T34
T8 T17
T26 T35
T9 T18
T27 T36
40
Modes of Parallelism:Single vs. Multiple Executable• In Single Program Multiple Datastream (SPMD) mode the same
program runs across all PETs in the application - components may run sequentially or concurrently.
• In Multiple Program Multiple Datastream (MPMD) mode the application consists of separate programs launched as separate executables - components may run concurrently or sequentially, but in this mode almost always run concurrently
41
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
42
Framework-Wide BehaviorESMF has a set of interfaces and behaviors that hold across the entire framework. This consistency helps make the framework easier to learn and understand.
For more information, see Sections 6-8 in the Reference Manual.
43
Classes and Objects in ESMF• The ESMF Application Programming Interface (API) is based on
the object-oriented programming notion of a class. A class is a software construct that’s used for grouping a set of related variables together with the subroutines and functions that operate on them. We use classes in ESMF because they help to organize the code, and often make it easier to maintain and understand.
• A particular instance of a class is called an object. For example, Field is an ESMF class. An actual Field called temperature is an object.
44
Classes and Fortran• In Fortran the variables associated with a class are stored in a
derived type. For example, an ESMF_Field derived type stores the data array, grid information, and metadata associated with a physical field.
• The derived type for each class is stored in a Fortran module, and the operations associated with each class are defined as module procedures. We use the Fortran features of generic functions and optional arguments extensively to simplify our interfaces.
45
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
46
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct, etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
47
3 DESIGN AND PRINCIPLES OF ESMF
• Computational Characteristics of Weather and Climate• Design Strategies• Parallel Computing Definitions• Framework-Wide Behavior• Class Structure• Exercises
48
Exercises
Following instructions given during class:• ssh to login to the Linux cluster.• Find the ESMF distribution directory.• See which ESMF environment variables are set.• Browse the source tree.
49
4 CLASSES AND FUNCTIONS• ESMF Superstructure Classes• ESMF Infrastructure Classes: Data Structures• ESMF Infrastructure Classes: Utilities• Exercises
50
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct, etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
51
ESMF Superstructure Classes
See Sections 12-16 in the Reference Manual.
• Gridded Component ◦ Models, data assimilation systems - “real code”
• Coupler Component◦ Data transformations and transfers between Gridded
Components• State – Packages of data sent between Components• Application Driver – Generic driver
52
ESMF Components
• An ESMF component has two parts, one that is supplied by the ESMF and one that is supplied by the user. The part that is supplied by the framework is an ESMF derived type that is either a Gridded Component (GridComp) or a Coupler Component (CplComp).
• A Gridded Component typically represents a physical domain in which data is associated with one or more grids - for example, a sea ice model.
• A Coupler Component arranges and executes data transformations and transfers between one or more Gridded Components.
• Gridded Components and Coupler Components have standard methods, which include initialize, run, and finalize. These methods can be multi-phase.
53
ESMF States• All data passed between Components is in the form of States
and States only• Description/reference to other ESMF data objects• Data is referenced so does not need to be duplicated• Can be Bundles, Fields, Arrays, States, or name-placeholders
54
Application Driver
• Small, generic program that contains the “main” for an ESMF application.
55
4 CLASSES AND FUNCTIONS• ESMF Superstructure Classes• ESMF Infrastructure Classes: Data Structures• ESMF Infrastructure Classes: Utilities• Exercises
56
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct, etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
57
ESMF Infrastructure Data Classes
Model data is contained in a hierarchy of multi-use classes. The user can reference a Fortran array to an Array or Field, or retrieve a Fortran array out of an Array or Field.
• Array – holds a Fortran array (with other info, such as halo size) • Field – holds an Array, an associated Grid, and metadata• Bundle – collection of Fields on the same Grid bundled together
for convenience, data locality, latency reduction during communicationsSupporting these data classes is the Grid class, which represents a numerical grid
58
Grids
See Section 25 in the Reference Manual for interfaces and examples.
• The ESMF Grid class represents all aspects of the computational domain and its decomposition in a parallel-processing environment It has methods to internally generate a variety of simple grids
• The ability to read in more complicated grids provided by a user is not yet implemented
• ESMF Grids are currently assumed to be two-dimensional, logically-rectangular horizontal grids, with an optional vertical grid whose coordinates are independent of those of the horizontal grid
• Each Grid is assigned a staggering in its create method call, which helps define the Grid according to typical Arakawa nomenclature.
59
Arrays
See Section 22 in the Reference Manual for interfaces and examples.
• The Array class represents a multidimensional array. • An Array can be real, integer, or logical, and can possess up to
seven dimensions. The Array can be strided. • The first dimension specified is always the one which varies
fastest in linearized memory. • Arrays can be created, destroyed, copied, and indexed.
Communication methods, such as redistribution and halo, are also defined.
60
Fields
See Section 20 in the Reference Manual for interfaces and examples.
• A Field represents a scalar physical field, such as temperature. • ESMF does not currently support vector fields, so the components of a vector
field must be stored as separate Field objects. • The ESMF Field class contains the discretized field data, a reference to its
associated grid, and metadata. • The Field class provides methods for initialization, setting and retrieving data
values, I/O, general data redistribution and regridding, standard communication methods such as gather and scatter, and manipulation of attributes.
61
Bundles
See Section 18 in the Reference Manual for interfaces and examples.
• The Bundle class represents “bundles” of Fields that are discretized on the same Grid and distributed in the same manner.
• Fields within a Bundle may be located at different locations relative to the vertices of their common Grid.
• The Fields in a Bundle may be of different dimensions, as long as the Grid dimensions that are distributed are the same.
• In the future Bundles will serve as a mechanism for performance optimization. ESMF will take advantage of the similarities of the Fields within a Bundle in order to implement collective communication, IO, and regridding.
62
ESMF CommunicationsSee Section 27 in the Reference Manual for a summary of
communications methods.
• Halo◦ Updates edge data for consistency between partitions
• Redistribution ◦ No interpolation, only changes how the data is decomposed
• Regrid◦ Based on SCRIP package from Los Alamos ◦ Methods include bilinear, conservative
• Bundle, Field, Array-level interfaces
63
ESMF DataMap ClassesThese classes give the user a systematic way of expressing interleaving and memory layout, also hierarchically (partially implemented, rework expected)
• ArrayDataMap – relation of array to decomposition and grid, row / column major order, complex type interleave
• FieldDataMap – interleave of vector components • BundleDataMap – interleave of Fields in a Bundle
64
4 CLASSES AND FUNCTIONS• ESMF Superstructure Classes• ESMF Infrastructure Classes: Data Structures• ESMF Infrastructure Classes: Utilities• Exercises
65
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct, etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
66
ESMF Utilities
• Time Manager• Configuration Attributes (replaces namelists)• Message logging • Communication libraries• Regridding library (parallelized, on-line SCRIP)• IO (barely implemented)• Performance profiling (not implemented yet, may simply use
Tau)
67
Time Manager
See Sections 32-37 in the Reference Manual for more information.
Time manager classes are:• Calendar• Clock • Time • Time Interval • AlarmThese can be used independent of other classes in ESMF.
68
CalendarA Calendar can be used to keep track of the date as an ESMF Gridded Component advances in time. Standard calendars (such as Gregorian and 360-day) and user-specified calendars are supported. Calendars can be queried for quantities such as seconds per day, days per month, and days per year.
Supported calendars are:• Gregorian The standard Gregorian calendar, proleptic to 3/1/-4800.• no-leap The Gregorian calendar with no leap years.• Julian Day A Julian days calendar.• 360-day A 30-day-per-month, 12-month-per-year calendar.• no calendar Tracks only elapsed model time in seconds.
69
Clock and AlarmClocks collect the parameters and methods used for model time advancement into a convenient package. A Clock can be queried for quantities such as start time, stop time, current time, and time step. Clock methods include incrementing the current time, and determining if it is time to stop.Alarms identify unique or periodic events by “ringing” - returning a true value - at specified times. For example, an Alarm might be set to ring on the day of the year when leaves start falling from the trees in a climate model.
70
Time and Time Interval
A Time represents a time instant in a particular calendar, such as November 28, 1964, at 7:31pm EST in the Gregorian calendar. The Time class can be used to represent the start and stop time of a time integration.Time Intervals represent a period of time, such as 300 milliseconds. Time steps can be represented using Time Intervals.
71
Config AttributesSee Section 38 in the Reference Manual for interfaces and examples.
• ESMF Configuration Management is based on NASA DAO’s Inpak 90 package, a Fortran 90 collection of routines/functions for accessing Resource Files in ASCII format.
• The package is optimized for minimizing formatted I/O, performing all of its string operations in memory using Fortran intrinsic functions.
72
LogErrSee Section 39 in the Reference Manual for interfaces and examples.
• The Log class consists of a variety of methods for writing error, warning, and informational messages to files.
• A default Log is created at ESMF initialization. Other Logs can be created later in the code by the user.
• A set of standard return codes and associated messages are provided for error handling.
• LogErr will automatically put timestamps and PET numbers into the Log.
73
Virtual Machine (VM)
See Section 41 in the Reference Manual for VM interfaces and examples.
• VM handles resource allocation• Elements are Persistent Execution Threads or PETs• PETs reflect the physical computer, and are one-to-one
with Posix threads or MPI processes• Parent Components assign PETs to child Components• The VM communications layer does simple MPI-like
communications between PETs (alternative communication mechanisms are layered underneath)
74
DELayoutSee Section 40 in the Reference Manual for interfaces and
examples.
• Handles decomposition• Elements are Decomposition Elements, or DEs (decomposition
that’s 2 pieces in x by 4 pieces in y is a 2 by 4 DELayout)• DELayout maps DEs to PETs, can have more than one DE per
PET (for cache blocking, user-managed OpenMP threading) • Simple connectivity or more complex connectivity, with weights
between DEs - users specify dimensions where greater connection speed is needed
• Array, Field, and Bundle methods perform inter-DE communications
75
4 CLASSES AND FUNCTIONS• ESMF Superstructure Classes• ESMF Infrastructure Classes: Data Structures• ESMF Infrastructure Classes: Utilities• Exercises
76
Exercises1. On the Linux cluster, cd to $ESMF_DIR, which is the top of the ESMF distribution.2. Change directory to build_config, to view directories for supported platforms.3. Change directory to ../src and locate the Infrastructure and Superstructure
directories.4. Note that code is arranged by class within these directories, and that each class has a
standard set of subdirectories (doc, examples, include, interface, src, and tests, plus a makefile).
Web-based alternative:1. Go to the sourceforge site: http://sourceforge.net/projects/esmf2. Select Browse the CVS tree 3. Continue as above from number 2. Note that this way of browsing the ESMF source
code shows all directories, even empty ones.
77
5 RESOURCES
• Documentation• User Support• Testing and Validation Pages• Mailing Lists• Users Meetings• Exercises
78
Documentation• Users Guide
◦ Installation, quick start and demo, architectural overview, glossary• Reference Manual
◦ Overall framework rules and behavior◦ Method interfaces, usage, examples, and restrictions◦ Design and implementation notes
• Developers Guide◦ Documentation and code conventions◦ Definition of compliance
• Requirements Document• Implementation Report
◦ C++/Fortran interoperation strategy• (Draft) Project Plan
◦ Goals, organizational structure, activities
79
User Support
• All requests go through the [email protected] list so that they can be archived and tracked
• Support policy is on the ESMF website• Support archives and bug reports are on the ESMF website -
see http://www.esmf.ucar.edu > Development
Bug reports are under Bugs and support requests are under Lists.
80
Testing and Validation Pages• Accessible from the Development link on the ESMF website• Detailed explanations of system tests• Supported platforms and information about each• Links to regression test archives• Weekly regression test schedule
81
Mailing Lists To Join• [email protected]
Joint specification team discussion◦ Release and review notices◦ Technical discussion◦ Coordination and planning
• [email protected] information◦ Quarterly updates
• [email protected] announcements◦ Annual meeting announcements
82
Mailing Lists To Write• [email protected]
Project leads◦ Non-technical questions◦ Project information
• [email protected] questions and comments
83
Users Meetings andCommunity Meeting
Every six weeks ESMF Early Adopters meet at GFDLMeeting schedule is on the ESMF websitehttp://www.esmf.ucar.edu > Community
4th ESMF Annual Community Meeting
Yesterday here at MIT CampusSee the ESMF Website for future meeting announcements.
84
5 RESOURCES
• Documentation• User Support• Testing and Validation Pages• Mailing Lists• Users Meetings• Exercises
85
ExercisesLocate on the ESMF website:
1. The Reference Manual, User’s Guide and Developer’s Guide2. The ESMF Draft Project Plan3. The current task list4. The modules in the contributions repository5. The weekly regression test schedule6. Known bugs from the last public release7. The % of public interfaces tested8. The schedule of Early Adopter (Users Group) meetings9. The ESMF Support Policy10. Subscribe to the ESMF mailing lists
86
6 PREPARING FOR AND USING ESMF
• Adoption Strategies• Demo• Quickstart• Exercises
87
Adoption Strategies: Top Down1. Decide how to organize the application as discrete Gridded and Coupler
Components. The developer might need to reorganize code so that individual components are cleanly separated and their interactions consist of a minimal number of data exchanges.
2. Divide the code for each component into initialize, run, and finalize methods. These methods can be multi-phase, e.g., init_1, init_2.
3. Pack any data that will be transferred between components into ESMF Import and Export States in the form of ESMF Bundles, Fields, and Arrays. User data must match its ESMF descriptions exactly.
4. The user must describe the distribution of grids over resources on a parallel computer via the VM and DELayout.
5. Pack time information into ESMF time management data structures.6. Using code templates provided in the ESMF distribution, create ESMF
Gridded and Coupler Components to represent each component in the user code.
7. Write a set services routine that sets ESMF entry points for each user component’s initialize, run, and finalize methods.
8. Run the application using an ESMF Application Driver.
88
Adoption Strategies: Bottom UpAdoption of infrastructure utilities and data structures can follow many different paths. The calendar management utility is a popular place to start, since there is enough functionality in the ESMF time manager to merit the effort required to integrate it into codes and bundle it with an application.
89
6 PREPARING FOR AND USING ESMF
• Adoption Strategies• Demo• Quickstart• Exercises
90
ESMF Demo
• Overview of ESMF Demo program
91
6 PREPARING FOR AND USING ESMF
• Adoption Strategies• Demo• Quickstart• Exercises
92
ESMF Quickstart
• Created when ESMF is compiled• $ESMF_DIR/quick_start top level directory• Contains a makefile which builds the quick_start application• Running it will print out execution messages to standard output• Cat the output file to see messages
93
ESMF Quickstart Structure
94
ESMF QuickstartDirectory contains the skeleton of a full application:• 2 Gridded Components• 1 Coupler Component• 1 top-level Gridded Component• 1 AppDriver main program• A file for setting module names• README file• Makefile• sample.rc resource file
95
6 PREPARING FOR AND USING ESMF
• Adoption Strategies• Demo• Quickstart• Exercises
96
Exercises
Following the User’s Guide:1. Build and run Quickstart program.2. Find the output files and see the printout.3. Add your own print statements in the code.4. Rebuild and see the new output
97
7 APPLICATIONS
Users can discuss adoption of ESMF in their applications with ESMF staff.
98
Answers to Section 5 Exercises Starting from http://www.esmf.ucar.edu/ :1. The Reference Manual, User’s Guide and Developer’s Guide
Downloads & Documentation -> ESMF Documentation List2. The ESMF Draft Project Plan
Publications & Talks -> ESMF Publications and Other Documents (first item)3. The current task list
Development -> Entry Point to the ESMF Source Code Repository -> Go to Sourceforge Site -> Tasks -> Core Tasks
4. The modules in the contributions repositoryCommunity -> Entry Point to the ESMF Community Contributions Repository -> Go to Sourceforge Site
5. The weekly regression test scheduleDevelopment -> Test & Validation
99
Answers to Section 5 Exercises
Starting from http://www.esmf.ucar.edu/ :6. Known bugs from the last public release
Downloads & Documentation -> Download ESMF v2.2.0 -> Release Notes and Known Bugs
7. The % of public interfaces testedDevelopment -> Metrics
8. The schedule of Early Adopter (Users Group) meetingsCommunity -> Early Adopter Meetings
9. The ESMF Support PolicyUser Support & Contacts -> Support Requests
10. Subscribe to the ESMF mailing listsCommunity -> Mailing Lists