expanding regridding capabilities of the earth system modeling framework andrew scholbrock...

17
Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Upload: leila-paschal

Post on 29-Mar-2015

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Expanding Regridding Capabilities of the Earth System Modeling Framework

Andrew ScholbrockUniversity of Colorado – Boulder

Robert OehmkeNOAA/CIRES

1

Page 2: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Outline

• Introduction• Background Information• Incorporating LibCF Regridding into ESMF• Current Status• Future Work

2

Page 3: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Introduction

• This project aims to allow regridding software that follows a proposed metadata standard (Gridspec) to be used within the Earth System Modeling Framework (ESMF)

• As new regridding capabilities are developed they will be automatically available through ESMF

3

Page 4: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

ESMF

• Framework developed by a multi-agency consortium [1]• Provides standard interfaces for model components• Provides standard structure for transferring data between model

components• Provides a range of utilities to ease coupling

– Data regridding– Data redistribution– Time management– Error handling

4

Page 5: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Gridspec

• Metadata convention developed at NOAA GFDL [2]

• Proposed for inclusion in the Climate and Forecast (CF) [3] conventions

• Gridspec represents grids as mosaics and tiles– Mosaic files contain a list of tile files and

how they are connected– Tile files contain actual grid coordinate

information

5

http://www.gfdl.noaa.gov/~vb/gridstd/gridstdse2.html#x4-160002.7

Page 6: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

LibCF

• Utility library developed at Unidata• Built on top of NetCDF [4] Climate and Forecast (CF) conventions• LibCF [3] includes the capability to:

– Manipulate Gridspec metadata on a NetCDF file– Create Gridspec files– Regrid between different Gridspec files

• LibCF is still in alpha release mode

6

Page 7: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Regridding Overview• Climate and other models require coupling of modeling components (e.g. atmosphere, ocean)• Different modeling components may require different grids

– e.g. different shapes or different resolutions

• Regridding needs to be performed to move data between components on different grids • Steps involved in regridding:

– Generate interpolation weights – Apply weights to interpolate field using a sparse matrix multiply

7

Page 8: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

ESMF Regridding• Methods of accessing regridding:

– Online • Subroutine calls which calculate weights during a model run• Can get weights or feed directly into ESMF sparse matrix multiply

– Offline• Application which generates a netCDF interpolation weight file

from two netCDF grid files• Computation of weights can be done in parallel• Supported grids:

– 2D meshes composed of triangles or quadrilaterals– 3D meshes composed of hexahedra– 2D/3D logically rectangular grids– Cubed sphere

• Supported interpolation methods:– Bilinear– Higher order finite element patch recovery [5][6]– Conservative (beta version)

• Not all combinations of the above are supported

8

Page 9: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

LibCF Regridding

• LibCF takes Gridspec files and field data files as input to generate weights and interpolate data– If just Gridspec files are specified then only interpolation weights are

generated• LibCF supports regridding between mosaics of logically rectangular grid tiles

(e.g. cubed sphere, single tile logically rectangular)• Currently first order conservative interpolation methods are supported, and

some specific bilinear cases

9

Page 10: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Combining ESMF and LibCF

• LibCF was introduced into the ESMF online regridding software.• This was somewhat awkward because LibCF requires Gridspec files as

input, but ESMF online regridding holds grid descriptions in memory.• As a result, ESMF must convert its grid descriptions to Gridspec files

and then have LibCF read them in.• This works, but more efficient connections could be made with

regridding libraries that did not require file input.

10

Page 11: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Implementation Steps:Getting Interpolation Weights

1. Write ESMF grids to NetCDF filesa) Take an existing ESMF grid and get the coordinate information out of itb) Write out the coordinate information to a NetCDF file

• Done using ESMF I/O functionality

c) Create mosaic files from the NetCDF grid• Done using LibCF’s make_mosaic function

2. Get interpolation weights from LibCFa) Call LibCF’s regridding function through ESMF callsb) Read in LibCF’s interpolation weights into ESMFc) Convert interpolation data to a form that ESMF’s sparse matrix multiply can use

11

Page 12: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Implementation Steps:Weight Application

• Apply LibCF interpolation weights to field data using the ESMF sparse matrix multiply call

• Future regrid calls to updated field data can bypass the LibCF interpolation weight generation step– This holds as long as the grids involved remain the same

12

Page 13: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

LibCF/ESMF Regridding Flow

Write Tile

Source Grid

Create Mosaic

Source Tile

Create Weights Read Weights

Weight Matrix

make_mosaic fregrid

Source Mosaic Remap File

Source Variable Data

Destination Variable Data

ESMF

LibCF

Destination Tile

Destination Grid

Destination Mosaic

Regrid

File I/O

NetCDF File

ESMF Data

API Call

Legend:

13

Page 14: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

ESMF User Interface for Regridding with LibCF

• Actual interface that users employ contains two steps:1. ESMF_FieldRegridStore(): Provide source/destination grids and receive

interpolation weights2. ESMF_FieldRegrid(): Provide interpolation weights and source field data

to receive interpolated field data corresponding to the destination grid• This is the same sequence of calls used for pure ESMF regridding

14

Page 15: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Current Status

• Can read interpolation weight information generated by LibCF into ESMF sparse matrix representation and perform a regrid.

• Can write ESMF grid structures to Gridspec files• ESMF restriction:

– Writing Gridpsec files from ESMF is serial and thus restricts the scaling of the final code

• Intersection of LibCF/ESMF capabilities:– Only supports single tile logically rectangular grids

• LibCF restriction:– Only supports first order conservative remapping for this geometry

15

Page 16: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

Future Work

• Parallel I/O for writing Gridspec files• Switch to non-file based interface if LibCF develops one• Allow for regridding of more complicated grids including the cubed

sphere

16

Page 17: Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

References1. Hill, C., C. DeLuca, V. Balaji, M. Suarez, and A. da Silva (2004). Architecture of the Earth

System Modeling Framework. Computing in Science and Engineering, Volume 6, Number 1, pp. 18-28.

2. Balaji, V., Liang, Z. Gridspec: A Standard for the Description of Grids Used in Earth System Models. http://www.gfdl.noaa.gov/~vb/gridstd/gridstd.html

3. The CF Library Users Guide. http://www.unidata.ucar.edu/software/libcf/docs/libcf.html4. Hartnett, E. and Rew, R., Experience with an enhanced netCDF data model and interface

for scientific data access, 88th AMS Annual Meeting, 24th Conference on IIPS, 2008.

5. Khoei S.A., Gharehbaghi A. R. The superconvergent patch recovery technique and data transfer operators in 3d plasticity problems. Finite Elements in Analysis and Design, 43(8), 2007.

6. Hung K.C, Gu H., Zong Z. A modified superconvergent patch recovery method and its application to large deformation problems. Finite Elements in Analysis and Design, 40(5-6), 2004.

[email protected]

17