2 3 roms/coawst ncml file 4 5 exploiting ioos: a distributed, standards-based framework and software...
TRANSCRIPT
22
33
ROMS/COAWST NcML fileROMS/COAWST NcML file
44
55
Exploiting IOOS: A Distributed, Standards-Based Framework and
Software Stack for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data
Exploiting IOOS: A Distributed, Standards-Based Framework and
Software Stack for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data
Rich Signell (USGS-CMG)Filipe Fernandes (SECOORA)
Kyle Wilcox (Axiom Data Science)Andrew Yan (USGS-CIDA)
Rich Signell (USGS-CMG)Filipe Fernandes (SECOORA)
Kyle Wilcox (Axiom Data Science)Andrew Yan (USGS-CIDA)
Regional IOOS DMAC Meeting: Silver Spring, 5/28/2015
ObjectivesObjectives
• Set up a standards-based framework for easy and efficient access to insitu and ocean model data
• Provide a high-level search and browse web interface for program datasets, for scientists, end users and program managers
• Contribute to a growing standardized data search, access and use infrastructure that supports all geoscience
• Set up a standards-based framework for easy and efficient access to insitu and ocean model data
• Provide a high-level search and browse web interface for program datasets, for scientists, end users and program managers
• Contribute to a growing standardized data search, access and use infrastructure that supports all geoscience
Why not just use ERDDAP?Why not just use ERDDAP?
• Two reasons:• 1. Unstructured grid models• 2. Curvilinear grid models
• Two reasons:• 1. Unstructured grid models• 2. Curvilinear grid models
UGRID Conventions on GitHubUGRID Conventions on GitHub
SGRID Conventions: github/sgridSGRID Conventions: github/sgrid
IOOS Model Data Interoperability DesignIOOS Model Data Interoperability Design
ROMS
ADCIRC
HYCOM
SELFE
NCOM NcML
NcML
NcML
NcML
NcML
Common Data Model
OPeNDAP+CF
WCS
NetCDF Subset
THREDDS Data Server
Standardized (CF-1.6, UGRID-0.9) Virtual Datasets
NonstandardModel Output Data Files
Web Services Matlab
Panoply
IDV
Clients
NetCDF-Java
Libraryor Broker
WMS
ncISO
ArcGIS
NetCDF4-Python
FVCOM
PythonERDDAP
NetCDF-Java
SOS
Geoportal Server
GeoNetwork
GI-CAT
Observed data (buoy, gauge, ADCP, glider)
Godiva2
pycsw-CKANNcML
GridUgridTimeSeriesProfileTrajectoryTimeSeriesProfile
NonstandardData Files
CatalogServices
Interoperable Model Comparison in Matlab(using nctoolbox)
Interoperable Model Comparison in Matlab(using nctoolbox)
compare_secoora_model_sections.mcompare_secoora_model_sections.m
3D visualization of data with IDV3D visualization of data with IDV
NECOFS Access in ArcGIS(using the dap2arc python toolbox)
NECOFS Access in ArcGIS(using the dap2arc python toolbox)
USGS CMG PortalUSGS CMG Portal
NetCDF Point Subset ServiceNetCDF Point Subset Service
Iris Python tools from the UK Met OfficeIris Python tools from the UK Met Office
Automated model comparisonAutomated model comparison
Getting your model results connectedGetting your model results connected
• Find someone with a THREDDS Data Server or install your own
• Drop your files in a directory, and add an NcML file that starts with “00_dir” (e.g. “00_dir_roms.ncml”) to aggregate, standardize and describe the dataset: Sample ROMS NcML file
• If you want your data to end up in the portal, add “CMG_Portal” to the “project” attribute: <attribute name=“project”value=“CMG_Portal”/>
• If you want your datasets to be discoverable, submit a PR on list of thredds catalogs being scanned on github
• Full instructions on the USGS-CMG Portal Github Wiki
• Find someone with a THREDDS Data Server or install your own
• Drop your files in a directory, and add an NcML file that starts with “00_dir” (e.g. “00_dir_roms.ncml”) to aggregate, standardize and describe the dataset: Sample ROMS NcML file
• If you want your data to end up in the portal, add “CMG_Portal” to the “project” attribute: <attribute name=“project”value=“CMG_Portal”/>
• If you want your datasets to be discoverable, submit a PR on list of thredds catalogs being scanned on github
• Full instructions on the USGS-CMG Portal Github Wiki
A few problems… PackagingA few problems… Packaging
• Ipython notebooks are a great way to document model skill assessment workflows (Filipe will talk about this)
• But python environment uses a lot of tricky packages. How to make this easy for folks?
• Conda and binstar to the rescue! (Filipe will talk about this)
• Ipython notebooks are a great way to document model skill assessment workflows (Filipe will talk about this)
• But python environment uses a lot of tricky packages. How to make this easy for folks?
• Conda and binstar to the rescue! (Filipe will talk about this)
A few problems… WMSA few problems… WMS
• ncWMS works great for CF compliant data• Unstructured grids are not CF compliant.• Staggered grids are not CF compliant. • ncWMS doesn’t work for unstructured grid data
(FVCOM, ADCIRC, SELFE), and doesn’t work for staggered grid velocities in models like ROMS, WRF and Delft3D
• sci-wms to the rescue, using UGRID conventions for unstructured grid (pyugrid), and SGRID conventions for staggered grid (pysgrid). (Kyle will talk about this)
• ncWMS works great for CF compliant data• Unstructured grids are not CF compliant.• Staggered grids are not CF compliant. • ncWMS doesn’t work for unstructured grid data
(FVCOM, ADCIRC, SELFE), and doesn’t work for staggered grid velocities in models like ROMS, WRF and Delft3D
• sci-wms to the rescue, using UGRID conventions for unstructured grid (pyugrid), and SGRID conventions for staggered grid (pysgrid). (Kyle will talk about this)
Key Infrastructure ComponentsKey Infrastructure Components
• Common data models for “feature types” (structured, staggered and unstructured grids, time series, profiles, swaths) (Unidata CDM, UGRID, SGRID)
• Standard web data services for delivering these common data model “feature types” (OPeNDAP/CF/UGRID/SGRID, WMS, SOS, WFS, ERDDAP/tabledap, ERDDAP/griddap)
• Standard catalog services for the metadata (OGC CSW, OpenSearch)
• Tools for easy delivery of data in standard services
• Tools for easy search, access and use of data in standard services (in all major environments: Python, ArcGIS, R, Matlab, JavaScript)
• Common data models for “feature types” (structured, staggered and unstructured grids, time series, profiles, swaths) (Unidata CDM, UGRID, SGRID)
• Standard web data services for delivering these common data model “feature types” (OPeNDAP/CF/UGRID/SGRID, WMS, SOS, WFS, ERDDAP/tabledap, ERDDAP/griddap)
• Standard catalog services for the metadata (OGC CSW, OpenSearch)
• Tools for easy delivery of data in standard services
• Tools for easy search, access and use of data in standard services (in all major environments: Python, ArcGIS, R, Matlab, JavaScript)
Infrastructure BenefitsInfrastructure Benefits
• What are the benefits?– Less time wasted messing with data,
more time spent on science– More skill assessment of models– More usage and more appropriate
useage of model results – Faster feedback to modelers =>
improved models– Better science, better models =>better
world
• What are the benefits?– Less time wasted messing with data,
more time spent on science– More skill assessment of models– More usage and more appropriate
useage of model results – Faster feedback to modelers =>
improved models– Better science, better models =>better
world