gridded data interoperability for the integrated ocean observing system rich signell usgs woods...

Gridded Data Interoperability for the Integrated Ocean Observing SystemGridded Data Interoperability for the Integrated Ocean Observing System

Rich Signell USGS Woods Hole/NOAA Silver Spring USA

USGS CDI Web Meeting: Oct 22, 2009

Rich Signell USGS Woods Hole/NOAA Silver Spring USA

USGS CDI Web Meeting: Oct 22, 2009

US Integrated Ocean Observing System (IOOS®)

IOOSIOOS®® Plan defines: Plan defines:• Global ComponentGlobal Component• Coastal ComponentCoastal Component

17 Federal Agencies17 Federal Agencies 11 Regional Associations 11 Regional Associations

IOOS® Subsystems

Observing SystemsObserving Systems Data Management and Communications Data Management and Communications (DMAC)(DMAC)

Modeling and Modeling and AnalysisAnalysis

In-situ & remotely In-situ & remotely sensed ocean sensed ocean

informationinformation

Decision support tools to Decision support tools to aid aid informed and timelyinformed and timely

decision makingdecision making

Effectively & Efficiently links ocean Effectively & Efficiently links ocean information to user applicationsinformation to user applications

DMAC Core PrinciplesDMAC Core Principles

• Adopt open standards & practices

• Avoid proprietary protocols or technologies

• Avoid customer-specific stovepipes

• Adopt open standards & practices

• Avoid proprietary protocols or technologies

• Avoid customer-specific stovepipes

66

CustomerCustomer

Web accessWeb accessserviceservice

DataDataProviderProviderObservationsObservations

ModelsModels

Sensor SystemsSensor Systems

In situ observationsIn situ observations

OGC SOSOGC SOS(Sensor(Sensor

ObservationObservationService)Service)

Table ofTable ofContentsContents

MetadataMetadata

Data ValuesData Values

DataDataProviderProvider

MetadataMetadata

QualityQualityControlControl

ObservationsObservations

Maps of DataMaps of Data

OGC WMSOGC WMS(Web(WebMapMap

Service)Service)


GeoreferencedGeoreferencedImagesImages


satellite satellite

HF radarHF radarmodelmodeloutputsoutputs

In situIn situobservationsobservations

HF radarHF radar

Gridded Data and Model OutputsGridded Data and Model Outputs


MetadataMetadata

Data ValuesData Values


CF MetadataCF Metadata

QualityQualityControlControl

satellite satellite

griddedgriddedmodelmodeloutputsoutputs

OGC WCSOGC WCS(Web(Web

CoverageCoverageService)Service)

CF+OPeNDAPCF+OPeNDAP

The Problem: Stovepiped Model AccessThe Problem: Stovepiped Model Access

Result: Inability to compare and assess model results & underutilization of model products

Result: Inability to compare and assess model results & underutilization of model products

The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)

Issue: Ocean grids are not regularly spaced!Issue: Ocean grids are not regularly spaced!

Stretched surface and terrain following vertical coordinates

Curvilinear orthogonal horizontal coordinates

NetCDF Climate and Forecast (CF) Conventions provide a solution

NetCDF Climate and Forecast (CF) Conventions provide a solution

Groups using CF:

GO-ESSP: Global Organization for Earth System Science Portal

IOOS: Integrated Ocean Observing System

ESMF: Earth System Modeling Framework

OGC: Open Geospatial Consortium (GALEON: WCS profile)

NcML to the Rescue! (XML markup language for NetCDF)

NcML to the Rescue! (XML markup language for NetCDF)

ROMS

POM

WW3

WRF

ECOM

NcML

NcML

NcML

NcML

NcML

UnidataCommon Data Model

OPeNDAP

OGC WMS

NetCDF Subset Service

THREDDS Data Server

Standardized (CF) & Aggregated Virtual Datasets

NonstandardOutput Files

Web Services

Data Interoperability System DesignData Interoperability System Design

Matlab

Godiva2

Panoply

IDV

ERDDAP

StandardClients

NetCDF-Java

API

NcML, NetCDF-Java and THREDDS Data NcML, NetCDF-Java and THREDDS Data Server built and supported by Unidata Server built and supported by Unidata under NSF-supportunder NSF-support

ArcGIS

OGC WCS

GOMODP THREDDS CatalogGOMODP THREDDS Catalog

Web Mapping Service with ncWMSWeb Mapping Service with ncWMS

3D visualization of data with IDV3D visualization of data with IDV

Comparing Models with Data in MatlabComparing Models with Data in Matlab

Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32

Buoy_comp.mBuoy_comp.m

Web Mapping Service with ncWMSWeb Mapping Service with ncWMS

Working with Data in ArcGIS using the NOAA/ASA Environmental Data Connector

Working with Data in ArcGIS using the NOAA/ASA Environmental Data Connector

Extract to multiple formats using ERDDAP

Extract to multiple formats using ERDDAP

Extracting Bathymetry Data:Step1: Browse http://coast-enviro.er.usgs.gov/thredds

for OpenDAP URL

Extracting Bathymetry Data:Step1: Browse http://coast-enviro.er.usgs.gov/thredds

for OpenDAP URL

Extracting Bathy Data: Step 2: Use cf_subsetGrid.m with OpenDAP URL

Extracting Bathy Data: Step 2: Use cf_subsetGrid.m with OpenDAP URL

bathy_comp.mbathy_comp.m

Using Mirone for BathymetryUsing Mirone for Bathymetry

Exported from Mirone to Iview3DExported from Mirone to Iview3D

SummarySummary

• What did we get through the use of this approach? – Standards-based delivery of aggregated model data

with minimal burden on the data provider – Single point of access for regional model results– Model data interoperability via standard software– Multiple modes of access for users– More eyes on the model results– More usage of model results– Faster feedback to modelers– Improved models!

• We now have an IOOS THREDDS Catalog that points to 17 THREDDS servers running in all 11 IOOS Regions

• This approach can be applied to any gridded data

• What did we get through the use of this approach? – Standards-based delivery of aggregated model data

with minimal burden on the data provider – Single point of access for regional model results– Model data interoperability via standard software– Multiple modes of access for users– More eyes on the model results– More usage of model results– Faster feedback to modelers– Improved models!

• We now have an IOOS THREDDS Catalog that points to 17 THREDDS servers running in all 11 IOOS Regions

• This approach can be applied to any gridded data

Questions, Feedback?Questions, Feedback?

Dr. Richard P. Signell+1 (508) 457-2229

[email protected]@noaa.gov

Try the NJ Toolkit for Matlab:Try the NJ Toolkit for Matlab:http://njtbx.sourceforge.nethttp://njtbx.sourceforge.net

Glossary (1/6)Glossary (1/6)

• CF Conventions: NetCDF Climate and Forecast Metadata Conventionshttp://cf-pcmdi.llnl.gov/(Standards for identifying geospatial coordinates in NetCDF files andgrid relationships)

• DMAC: Data Management And Communications subsystem http://dmac.ocean.us/ (One of three subsystems of IOOS, providing the infrastructure that will link the Observing and the Modeling and Applications subsystems)

• ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html(3D ocean circulation model developed principally by Alan Blumberg)

• ERDDAP: Environmental Research Division's Data Access Programhttp://coastwatch.pfeg.noaa.gov/erddap/index.html(A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data)

• CF Conventions: NetCDF Climate and Forecast Metadata Conventionshttp://cf-pcmdi.llnl.gov/(Standards for identifying geospatial coordinates in NetCDF files andgrid relationships)

• DMAC: Data Management And Communications subsystem http://dmac.ocean.us/ (One of three subsystems of IOOS, providing the infrastructure that will link the Observing and the Modeling and Applications subsystems)

• ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html(3D ocean circulation model developed principally by Alan Blumberg)

• ERDDAP: Environmental Research Division's Data Access Programhttp://coastwatch.pfeg.noaa.gov/erddap/index.html(A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data)


• ESMF: Earth System Modeling Frameworkhttp://www.esmf.ucar.edu/

(Software for building and coupling weather, climate, and related models)

• GO-ESSP: Global Organization for Earth System Science Portal http://go-essp.gfdl.noaa.gov/(Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities)

• IDV: Integrated Data Viewerhttp://www.unidata.ucar.edu/software/idv/(Java-based software for analyzing and visualizing geoscience data)

• IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/(Multidisciplinary system designed to collect, deliver, and use ocean information)

• IWGOO: Interagency Working Group on Ocean Observations

• ESMF: Earth System Modeling Frameworkhttp://www.esmf.ucar.edu/

(Software for building and coupling weather, climate, and related models)

• GO-ESSP: Global Organization for Earth System Science Portal http://go-essp.gfdl.noaa.gov/(Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities)

• IDV: Integrated Data Viewerhttp://www.unidata.ucar.edu/software/idv/(Java-based software for analyzing and visualizing geoscience data)

• IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/(Multidisciplinary system designed to collect, deliver, and use ocean information)

• IWGOO: Interagency Working Group on Ocean Observations


• MAST: Modeling and Analysis Steering Teamhttp://www.ocean.us/Models(Guide the development of ocean and coastal modeling activities in support of the goals of IOOS)

• NcWMS: Web Map Servicehttp://www.resc.rdg.ac.uk/trac/ncWMS/(ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. )

• NcML: NetCDF Markup Languagehttp://www.unidata.ucar.edu/software/netcdf/ncml/(Used to standardize and aggregate non-standard NetCDF files)

• NetCDF-Java: NetCDF-Java Libraryhttp://www.unidata.ucar.edu/software/netcdf-java/(Library that takes advantage of CF conventions, NcML to facilitatebuilding standards-based clients)

• MAST: Modeling and Analysis Steering Teamhttp://www.ocean.us/Models(Guide the development of ocean and coastal modeling activities in support of the goals of IOOS)

• NcWMS: Web Map Servicehttp://www.resc.rdg.ac.uk/trac/ncWMS/(ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. )

• NcML: NetCDF Markup Languagehttp://www.unidata.ucar.edu/software/netcdf/ncml/(Used to standardize and aggregate non-standard NetCDF files)

• NetCDF-Java: NetCDF-Java Libraryhttp://www.unidata.ucar.edu/software/netcdf-java/(Library that takes advantage of CF conventions, NcML to facilitatebuilding standards-based clients)


• NSF: National Science Foundationhttp://www.nsf.gov/(independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense…“)

• OpenDap: Open-source Project for a Network Data Access Protocolhttp://opendap.org(allows efficient hyperslabbing of geospatial data)

• POM: Princeton Ocean Modelhttp://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/(Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability)

• RCOOS: Regional Coastal Ocean Observing System(Generic term for one of the 11 regional observing systems in IOOS)

• NSF: National Science Foundationhttp://www.nsf.gov/(independent federal agency created by Congress in 1950 "to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense…“)

• OpenDap: Open-source Project for a Network Data Access Protocolhttp://opendap.org(allows efficient hyperslabbing of geospatial data)

• POM: Princeton Ocean Modelhttp://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/(Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability)

• RCOOS: Regional Coastal Ocean Observing System(Generic term for one of the 11 regional observing systems in IOOS)


• ROMS: Regional Ocean Modeling System

www.myroms.org

(A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications)

• SCCOOS: Southern California Coastal Ocean Observing System

http://www.sccoos.org/

(Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.)

• THREDDS (Thematic Realtime Environmental Distributed Data Services)

http://www.unidata.ucar.edu/projects/THREDDS

(Services for delivering standards-based geospatial data via OpenDAP,

WCS and more, works with NcML)

• ROMS: Regional Ocean Modeling System

www.myroms.org

(A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications)

• SCCOOS: Southern California Coastal Ocean Observing System

http://www.sccoos.org/

(Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.)

• THREDDS (Thematic Realtime Environmental Distributed Data Services)

http://www.unidata.ucar.edu/projects/THREDDS

(Services for delivering standards-based geospatial data via OpenDAP,

WCS and more, works with NcML)


• WCS: Web Coverage Servicehttp://www.opengeospatial.org/standards/wcs(OGC standard for serving gridded information: time series, profiles,hyperslabs of remote sensing data, bathymetry, model output)

• WW3: WaveWatch III

http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html

(Ocean wave model developed by Hendrik Tolman)• WRF: Weather Research and Forecasting Model

http://www.wrf-model.org/index.php

(Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs)

• WCS: Web Coverage Servicehttp://www.opengeospatial.org/standards/wcs(OGC standard for serving gridded information: time series, profiles,hyperslabs of remote sensing data, bathymetry, model output)

• WW3: WaveWatch III

http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html

(Ocean wave model developed by Hendrik Tolman)• WRF: Weather Research and Forecasting Model

http://www.wrf-model.org/index.php

(Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs)

Path from research to operationsPath from research to operations

Recommendation:- R01 funding, to support (a) Education & Training and (b) Research Grade modeling (broadly defined) activities – open to all academia- R02 funding, to support semi-operational systems at regional associations- R03 funding to support operational systems at federal agencies- R04 funding, to support translation and evaluation

Towards an integrated modeling system for the U.S. coastal margin …

… with clearly delineated roles, responsibilities and sustaining mechanisms



== Draft document == Prepared by A. Baptista with input from D. Schwab

and many others, for discussion by the NFRA modeling sub-committeeVersion 1.0, March 2009



Background: Hot off the press IOOS legislationBackground: Hot off the press IOOS legislation

(1) establish a national integrated System of ocean, coastal, and Great Lakes observing systems, comprised of Federal and non-Federal components coordinated at the national level by the National Ocean Research Leadership Council and at the regional level by a network of regional information coordination entities, and that includes in situ, remote, and other coastal and ocean observation, technologies, and data management and communication systems, …

(2) improve the Nation's capability to measure, track, explain, and predict events related directly and indirectly to weather and climate change, natural climate variability, and interactions between the oceanic and atmospheric environments, including the Great Lakes; and

(3) authorize activities to promote basic and applied research to develop, test, and deploy innovations and improvements in coastal and ocean observation technologies, modeling systems, and other scientific and technological capabilities to improve our conceptual understanding of weather and climate, ocean-atmosphere dynamics, global climate change, physical, chemical, and biological dynamics of the ocean, coastal and Great Lakes environments, and to conserve healthy and restore degraded coastal ecosystems.




Background: Hot off the press IOOS legislationBackground: Hot off the press IOOS legislation







Anecdotal background: modeling post-doc search

Anecdotal background: modeling post-doc search

Stale e-mailNo responseCourtesy responseSuggested alternative contact/listSuggested candidateApplied

Findings/implications:• There is a significant dearth of expertise:

• most responses knew of no candidates• most of suggested candidates were not own students (several duplicates)• a significant number or responses asked for access to any good “leftover list”

• There is a need for investment in education and training of modelers

Based on 71 direct e-mails

GoalGoal

• Establish the need for a sustainable, integrated U.S. coastal margin modeling system as key technology within IOOS

• Establish the framework for such a system …– … with clearly defined roles and responsibilities for

the federal government and for academia …– … in a partnership effectively facilitated through

IOOS and its Regional Associations





Who does ?Who does ?

DAILY FORECASTS

Operational Semi-operational Research grade

Basins Navy N/A N/A

Shelves NOAA RA partnerships RAs, RA partnerships, academia at large

Major estuaries/bays

NOAA RAs RAs, academia at large

Other estuaries/bays

-- RAs RAs, academia at large

Semi-operational


Research grade

Operational

Education and training

Translation

Moving across operational levelsMoving across operational levels

Research grade

Operational

Semi-operation

al

?

? ??

Need mechanism to make evaluation/review objective , and to

facilitate evolution of modeling technology and strategy.

Open Columbia River benchmarkOpen Columbia River benchmark

Goals

Enablers

Phases

Concept

Color Key

Goals Enable continuous enhancement of multiple models and exploration of diverse modeling strategies Maximize value-added expertise of model developers/expert users, while minimizing their time investment

Illustr

ation only.

Think of a

ny (or

selecte

d?) IOOS si

te of interest


Recommendation:- R01 funding, to support (a) Education & Training and (b) Research Grade modeling (broadly defined) activities – open to all academia- R02 funding, to support semi-operational systems at regional associations- R03 funding to support operational systems at federal agencies- R04 funding, to support translation and evaluation

DefinitionsDefinitions

• Operational– Runs reliably and robustly, at prescribed frequency– Distributes consistent products at scheduled times– Is regularly quality controlled– Has 24/7 maintenance

• Semi-operational– Runs reliably and robustly, at prescribed frequency– Distributes consistent products at scheduled times– Is regularly quality controlled

• Research grade– May run only an ad hoc basis– Has no mandate for product distribution– May be quality controlled, but perhaps only on an exploratory basis– May be bottom-up– …

• Operational– Runs reliably and robustly, at prescribed frequency– Distributes consistent products at scheduled times– Is regularly quality controlled– Has 24/7 maintenance

• Semi-operational– Runs reliably and robustly, at prescribed frequency– Distributes consistent products at scheduled times– Is regularly quality controlled

• Research grade– May run only an ad hoc basis– Has no mandate for product distribution– May be quality controlled, but perhaps only on an exploratory basis– May be bottom-up– …

GoalGoal









Basic principles / assumptionsBasic principles / assumptions

• Operational systems need a long-term institutional framework– The federal government does this best

• No operational system will be sustainable without an adequate workforce– Academia does this best

• No operational system will be effective without the benefits of regional knowledge and regular innovation– Academia does this best

• A healthy operational system requires an effective and sustained “translation” layer between academia and the federal government– IOOS, specifically through Regional Associations, has the

opportunity to do this best– Note: “Translation” needs definition. It includes more than one-

way export of software and data. It is a funded exchange, of experiences, technologies, and evaluation methods

• Operational systems need a long-term institutional framework– The federal government does this best

• No operational system will be sustainable without an adequate workforce– Academia does this best

• No operational system will be effective without the benefits of regional knowledge and regular innovation– Academia does this best

• A healthy operational system requires an effective and sustained “translation” layer between academia and the federal government– IOOS, specifically through Regional Associations, has the

opportunity to do this best– Note: “Translation” needs definition. It includes more than one-

way export of software and data. It is a funded exchange, of experiences, technologies, and evaluation methods

gridded data interoperability for the integrated ocean observing system rich signell usgs woods...

Documents

data dissemination

data providerscustomers

gridded data interoperability

different ocean

system ioos ioos plan

us integrated ocean

stovepiped model accessresult

public safety