thredds, cdm, opendap, netcdf and related conventions john caron unidata/ucar sep 2007
TRANSCRIPT
![Page 1: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/1.jpg)
THREDDS, CDM, OPeNDAP, netCDF and Related
Conventions
John Caron
Unidata/UCAR
Sep 2007
![Page 2: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/2.jpg)
Contents
• Overview
• THREDDS Data Server
• Unidata’s Common Data Model
![Page 3: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/3.jpg)
2) Server0) Client
1) Request
3) Response
![Page 4: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/4.jpg)
0) The Client
What functionality is needed?
1. Scientific User– Raw data– Drill down to arbitrary detail
2. Decision Support– “best effort” Visualization– operational
![Page 5: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/5.jpg)
1) The Request
What functionality is possible?• Analogous to SQL language for RDBMS• Implies a Data Model• OGC vs File access APIs
– NetCDF/OPeNDAP/HDF5 : index space– WXS : coordinate space
• Higher semantic level trumps if no significant extra cost.– File APIs become implementation, not interface
![Page 6: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/6.jpg)
1) WCS Request
• Functionality – Subsetting (bounding box, time range, variable)– Optional reprojection/resample
• Variants: KML/XML/SOAP+XML/REST• Optional Functionality : 42 flavors• Bad news for interoperability• Is there an elephant to dictate a standards?
– Eg IBM chose SQL/Relational model (1984)
![Page 7: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/7.jpg)
2) Server
How do I serve my data?
• Do I need specialized personnel?– $$, resource consumption, core competency
• What are the common requests?– (that I should optimize for)?
![Page 8: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/8.jpg)
3) Response
What comes back?
• Has to be a representation of the “answer” in the Data Model
• WCS allows anything– Cant write a generic client
• Communities will form around a small number of variants– No elephants in sight
![Page 9: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/9.jpg)
3) Response : XML vs. binary
• Extensibility vs. Efficiency
• Binary: netCDF/GeoTIFF/HDF/etc – reflect favorite formats of committee members– Different data models : ideally need a formal
mapping (but there arent any yet)– Domain experts can make use
• GML closely follows the OGC/ISO data model (WFS requires GML)
![Page 10: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/10.jpg)
3) Response : XML vs. binary
• GML is waaaay too complex– Ambitious– OGC/ISO models are complex– Reality is complex– XML Schema is a disaster
• Google KML – “visualization format” not “data storage”
![Page 11: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/11.jpg)
HTTP Tomcat Server
THREDDS Data Server
Datasets
catalog.xml
motherlode.ucar.edu
THREDDS Server
NetCDF-Javalibrary
Application
IDD Data
•HTTPServer
•NetcdfSubset
•WCS
•OPeNDAP
configCatalog.xml
![Page 12: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/12.jpg)
THREDDS Catalogs
• XML over HTTP• Hierarchical listing of online resources (datasets)• Container for arbitrary search metadata
– Standard set maps to DC, GCMD, ADN – Unidata/NCAR-CDP
• Metadata can be inherited• Design goal: Make it easy for data providers• TDS uses extended version for configuration• Data Access URLS
– “Crossing the protocol boundary”
![Page 13: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/13.jpg)
THREDDS OPeNDAP Server
• OPenDAP is protocol for remote access to CDM• Current version 2.0; NASA ESE standard
– Working on new 4.0 protocol spec
• Based on Java-OPeNDAP library – shared development by Unidata/opendap.org
• Any CDM dataset can be served• Server4 (Hyrax):
– latest version of opendap.org C++ library – THREDDS Catalogs replace dods_dir
![Page 14: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/14.jpg)
THREDDS WCS service
• CDM files that have Grid coordinate system– evenly spaced x,y
• Allow to subset the dataset by:– Lat/lon or projection bounding box– time and vertical coordinate range– list of Variables
• Return formats– GeoTIFF floating point, grayscale– NetCDF/CF-1.0
• No reprojections, resamplings• Uses WCS 1.0, work on WCS 1.1 in progress
![Page 15: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/15.jpg)
NetCDF Subset Service
• Experiment with REST style web service• Allow to subset the dataset by:
– Lat/lon bounding box– time and vertical coordinate range– list of Variables
• NetCDF/CF, XML, CSV (spreadsheet)• Gridded Data
– Output is a CF-1.0 netCDF file– Variation of WCS (simplified request protocol)
• Grid as Point Datasets (experimental)– Extract vertical profile, time series from one point in model data
• Station Data: metars (7 day rolling archive)
![Page 16: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/16.jpg)
![Page 17: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/17.jpg)
HTTP Tomcat Server
Common Data Model
catalog.xml
hostname.edu
THREDDS Server Application
NetCDF-Javalibrary
IDD Data
•HTTPServer
•NetcdfSubset
•WCS
•OPeNDAP
Then a miracle
happens
Datasets
![Page 18: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/18.jpg)
NetcdfDataset
ApplicationScientific Datatypes
NetCDF-Java architecture
OPeNDAP
THREDDS
Catalog.xml NetCDF-3
HDF5
I/O service provider
GRIB
GINI
NIDS
NetcdfFile
NetCDF-4
…Nexrad
DMSP
CoordSystem Builder
Datatype Adapter
NcMLNcML
![Page 19: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/19.jpg)
Common Data Model File Formats
• General: NetCDF, HDF5, OPeNDAP
• Gridded: GRIB-1, GRIB-2 • Radar: NEXRAD level II and level III, DORADE,
Chinese NEXRAD
• Point: BUFR
• Satellite: DMSP, GINI
• In Progress: NetCDF4, McIdas AREA, NPOESS, NOAA CLASS legacy files, Barrowdale DataBlade, others
![Page 20: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/20.jpg)
Coordinate Systems
Common Data Model Layers
Data Access
Scientific Datatypes
Grid
Point
Radial
Trajectory
Swath
Station Profile
![Page 21: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/21.jpg)
Common Data Model(Data Access Layer)
![Page 22: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/22.jpg)
Coordinate Systems UML
![Page 23: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/23.jpg)
NetCDF-4 file format
• NetCDF-4 C library – 4.0 Beta implements CDM access layer
• Persistence format for complete CDM
– 4.1: adding Coordinate Systems • Optional layer, focus on CF-1 (libcf)
– 4.?: merge OPeNDAP access
• NetCDF-Java library will read, maybe write
![Page 24: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/24.jpg)
TDS / NcML aggregation
<dataset name="WEST-CONUS_4km Aggregation" urlPath="satellite/3.9/WEST-CONUS_4km">
<netcdf > <aggregation dimName="time" type="joinNew"> <scan location="/data/ldm/pub/satellite/3.9/WEST-CONUS_4km/"
suffix=".gini" /> </aggregation> </netcdf>
</dataset>
![Page 25: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/25.jpg)
Forecast Model Run Collection (FMRC)
![Page 26: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/26.jpg)
Scientific DataTypes
• Based on datasets Unidata is familiar with– APIs are evolving
• How are data points connected?• Intended to scale to large, multifile
collections• Intended to support “specialized queries”
– Space, Time
• Intend to create “standard” NetCDF file encoding conventions
![Page 27: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/27.jpg)
Scientific DataTypes
• Grids– Structured– Swath– Unstructured
• Point Observation– Unconnected– Station / Time Series– Trajectory– Profile
• Radial
![Page 28: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/28.jpg)
Climate and Forecast (CF) Conventions
• Conventions for encoding coordinate systems, other semantics in netCDF
• Working for 10 years– Version 1.0 in 2003– Good for gridded data
• Current working goups– Point/Station/Trajectory/Profile observations– CRS (map to OGC)
• Governance in place• Volunteer: motivated, practical, real
![Page 29: THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649cc95503460f94990df4/html5/thumbnails/29.jpg)
Summary: Unidata’s directions
• Client: both Scientific User and Decision Support
• Request in coordinate space– WCS is fine, not a big architectural decision
• Server: TDS– Files in native format, augmented by
indexing/DB
• Response: netCDF/CF and GeoTIFF/KML or WMS/JPEG