geoscience data analysis and visualization tools from ncar

Geoscience Data Analysis and Visualization

Tools from NCAR

HDF/HDF-EOS Workshop XI

Nov 6-8, 2007David Brown

HDF/HDEOS Workshop XI, November 2007

Topics• Two interfaces: the same capabilities (mostly)

– NCL - a self-contained scripting language– PyNGL and PyNIO - Python modules

• Quick overview: visualization and analysis• Data model • State of our I/O component development• Enabling access to HDF 5, HDFEOS 5, and

NetCDF 4

Brief HistoryBrief HistoryNCAR NCAR

GraphicsGraphicsHLUsHLUsNCLNCLNIONIO

PyNGLPyNGLPyNIOPyNIO

PyNIO and PyNGLJuly 2006

(Numeric 24.x/NumPy 1.0)

PyNIO2005

PyHLU2003

PyNGL 2004

Original FortranLate 60s

GKS/CGM/Fortran 77/UNIX conversion1980s NCAR

Graphics

1992: NCARG ConferenceC interface

Mid 90sHLUs

Mid 90s

NCLMid 90s

GSUN - Late 90s

1995: NCL Conference

2000-present: 30+ NCL Workshops

NIO2005

HDF/HDEOS Workshop XI, November 2007

NCAR Command Language (NCL)A scripting language designed for

scientific data analysis and visualization

1. Simple, robust file input and output

2. Hundreds of analysis functions

3. Visualizations (2D) are world class and highly customizable

http://www.ncl.ucar.edu/

HDF/HDEOS Workshop XI, November 2007

NCL: Data analysis

• Array-based syntax and operators• Hundreds of functions- Spherical harmonics- Scalar and vector regridding- Vertical interpolation- EOFs

• Many tailored to geosciences• Most handle missing data• Can call C and Fortran routines

HDF/HDEOS Workshop XI, November 2007

NCL: Visualization

• High-quality and customizable visualizations• Contours, XY, vectors, streamlines• Maps with most common map projections• Handles data on regular and irregular grids,

triangular meshes• Specialized scripts for skew-T, wind roses,

histograms, Taylor diagrams, panels, bar charts• GSUN interface: simplifies visualization • Over 1,400 visualization options available

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

PyNGL - Python module• Python NCL Graphics Library • Python version of popular GSUN interface• Same publication-quality graphics as NCL• Utilizes existing Python modules and

development tools (swig, Numeric, NumPy)• Contains some climate-specific data

analysis functions• Extensive and updated documentation

http://www.pyngl.ucar.edu/

HDF/HDEOS Workshop XI, November 2007

NCL Data Model

• Based on netCDF 3 • Language variables can have:• Named dimensions• Attributes• Coordinates variables

• A convention in NetCDF but a language feature in NCL

• 1d array with the same name as a dimension• Contains coordinate values for dimension elements

HDF/HDEOS Workshop XI, November 2007

Supported file format I/O• One function reads all supported data

formats: • NetCDF (now including NetCDF4 classic),

HDF4, HDF-EOS 2, GRIB 1 and 2• Writes NetCDF and HDF4

• All file formats massaged into same model: file variables have basically the same features as internal variables

• Flat name space: HDF group names are appended to variable names

HDF/HDEOS Workshop XI, November 2007

PyNIO - Python module• Same I/O library as NCL (libnio)• Reads and writes same formats as NCL• Same NetCDF-like view of all formats• But interfaces with NumPy for data access• Modeled on the Scientific NetCDF module

– NioVariable type is a reference to the variable in the file (metadata is attached to variable)

– Dereferencing it results in a NumPy variable with no metadata

• *Numpy is a Python array processing module

http://www.pyngl.ucar.edu/Nio.shtml

HDF/HDEOS Workshop XI, November 2007

2D Coordinates• Traditional NetCDF coordinate variables are

1D vectors• However, satellite data and model data on

modern grids require 2D coordinate grids• 2D coordinate variables created on the fly for

GRIB grids and HDFEOS Grid-type data• Added value variables that allow data to be

plotted on any map projection

HDF/HDEOS Workshop XI, November 2007

Enabling NetCDF 4 ‘classic’ • Recompile with NetCDF 4 and HDF 5 1.8

beta (we also included szip)• Add options for specifying format and

compression level• Tests using ncl_convert2nc on GRIB files

show reduction in file size by ~1/2 over GRIB.• Caveats:

– Beta version NetCDF 4 not yet supported on some architectures (64 bit mainly)

– Not supported for OPeNDAP NetCDF client library

HDF/HDEOS Workshop XI, November 2007

Path to HDF, HDFEOS 5 and full NetCDF 4 support

• size_t for dimension sizes to enable large variables on 64-bit hosts

• Add support for more atomic types (int64) • Support for groups in the file context• User control of chunking• Compound and variable length data types• Components to read HDF 5 and HDFEOS 5• Current NetCDF module extended to handle v4• More support for aggregation

HDF/HDEOS Workshop XI, November 2007

Current status• NCL 5.0.0 released this week

– NCSA-style Open Source license (finally)– Binaries available under slightly more restrictive

license (because of included 3rd party software)

• PyNGL/PyNIO 1.2.0 available– Supports NumPy 1.0.x and Python 2.5– Binaries (easy installation) available for:

• Various flavors of Linux, Mac OSX, Cygwin

• Other Unix systems, 32 and 64 bit

Distribution of PyNGL/PyNIO sites

Distribution of NCL sites

HDF/HDEOS Workshop XI, November 2007

Support

• Websites with extensive documentation

• Tutorials

• Hundreds of examples with downloadable scripts

• Active email lists– ncl-talk@ucar.edu, pyngl-talk@ucar.edu

• Hands-on training workshops

HDF/HDEOS Workshop XI, November 2007

Download locations

• NCL

• PyNGL and PyNIO

http://www.pyngl.ucar.edu/download.html

http://www.ncl.ucar.edu/download.html

HDF/HDEOS Workshop XI, November 2007

Questions? Email me: dbrown@ucar.edu

HDF/HDEOS Workshop XI, November 2007

Resources (a.k.a. attributes)

• Same control mechanism for NCL and PyNGL

• Detailed control of the appearance of a visualization

• Example using POP grid ocean current data

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

HDF/HDEOS Workshop XI, November 2007

NCL and PyNGL visualization examples

• scatter and line plots, contours, vectors, skew-t, meteograms, maps, wind barbs, bar charts, streamlines, trajectories, filled polygons, paneled visualizations

Skew-T graphicCourtesy ofDennis Shea

A Skew-T plot is used by

meteorologists to analyze data from

a balloon

sounding.

Based on a visualizationof Adam Phillips

Based on a visualizationof Joel Norris (Scripps)

using dummy data

Data from Climate Analysis SectionChristian Guillemot

THORPEX - an international research and development program responding to the weather related challenges of

the 21st century to accelerate improvements in the accuracy of 1-day to 2- week high impact weather

forecasts for the benefit of society, the economy and the

environment.

GRIB 2 TIGGE (THORPEX Interactive Grand Global Ensemble) data

Japan Meteorological Agency

Courtesy Jeff YinNCAR CGD

Image courtesy of Nan Rosenbloom, CGD

First two map databases built-in; high-resolution available as simple download

Christophe Cassou (CNRS/CERFACS)

This grid could be described as a tripole grid that is further modified by the arbitrary displacement of some portions of the grid to achieve finer resolution over areas of interest (typically, ocean areas).

Courtesy Mark Stevens, NCAR CGD

Grid from Tom GrossNOAA/NOS/CSDL/MMAP