data interoperability

www.hdfgroup.org

The HDF Group

Data Interoperability

The HDF Group Staff

Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 1

www.hdfgroup.org

Interoperability

• Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation.

(From Wikipedia)


www.hdfgroup.org

Background for data interoperability

• Focus on Earth Sciences

• Data needs to be shared across communities• Self-described data format

• Many different types of data• Grids, moving-sensor multidimensional

fields, time series, profiles, trajectories, geospatial framework data, points


www.hdfgroup.org

Why data interoperability?

• Data format

• Several data formats available HDF4/5, netCDF3/4, GRIB, BUFR, Binary

• Data are created with some data formats by following different physical models

• End users would like to use familiar tools to access data via different formats


www.hdfgroup.org

Why data interoperability?

• Semantic/content interoperability

• An example: the missing value of a physical variable• Tools or applications need to know the fill

value of a physical variable• How can they identify? Normally via an attribute

that stores the fill value • The name can be

“Fill_value, _FillValue, badValue,……”


www.hdfgroup.org

Goals

• Discuss a few cases about data interoperability

• Hear opinions or issues about data interoperability from audience.


www.hdfgroup.org

The HDF Group

Data Format Interoperability


www.hdfgroup.org

Case 1

• One set of APIs to handle multiple data formats


www.hdfgroup.org

Case 1

• netCDF4• Combine powerful model and simplicity of

netCDF with features of HDF5• netCDF interface on top of HDF5

• CDM(Common Data Model)• Harmonize netCDF, HDF4, HDF5, OPeNDAP,

GRIB and others


www.hdfgroup.org

Discussions

• Any comments/sharings about the experiences with netCDF4 and CDM?


www.hdfgroup.org

Case 2

• Format conversions and translations


www.hdfgroup.org

Data format Conversions/Translations

• Format conversions• The HDF4 to HDF5 conversion tool• The HDF-EOS2 to netCDF3/4 conversion tool• HDF-EOS to GeoTIFF conversion tool• ……

• Data format Translations• netCDF tools to access HDF4/5 files via

OPeNDAP• ……


www.hdfgroup.orgSep. 28-30, 2010 HDF/HDF-EOS Workshop XIV

Use netCDF tools to access HDF via OPeNDAP

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4/5 Files

AIRS/OMI

DAP

libnc-dap

Visualization Tools

HDF4/5 Handlers

Aqua/Aura

13

Translation Layers

www.hdfgroup.orgSep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 14

netCDF4 to access HDF-EOS5 files

• Augmentation• One file can be used for both EOS5 and NetCDF-4.• Note that EOS5 users are not affected at all.

HDF5

HDF-EOS5HDF-EOS5

fileHDF5

NetCDF4NetCDF-4

file

AugmentedHDF-EOS5

file

Augmentation

www.hdfgroup.org

Discussions

• Any comments/sharings about the experiences with data format conversions and translations


www.hdfgroup.org

The HDF Group

Semantic/Content Interoperability


www.hdfgroup.org

An example

• Degree of latitude and longitude• Champaign’s latitude is 40°6´38" N • It can be represented in two formats:

• 400638 in DDDMMSS format

• 40.1105556 in decimal format

• How can the applications know which format the file is used?

• Better to have a common standard to facilitate

the exchange this kind of information


www.hdfgroup.org

Semantic/Content Interoperability

• CF conventions become such a standard

• Many applications/tools follow CF conventions to interpret the metadata


www.hdfgroup.orgSep. 28-30, 2010 HDF/HDF-EOS Workshop XIV

Use netCDF tools to access HDF via OPeNDAP

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4/5 Files

AIRS/OMI

DAP

libnc-dap

Visualization Tools

HDF4/5 Handlers

Aqua/Aura

19

Not follow CF conventions

www.hdfgroup.org

Discussions

• Any comments about semantic/content interoperability

• Any comments about the usage of CF conventions


www.hdfgroup.org

The HDF Group

HDF5 and netCDF-4 Libraries Interoperability

Two sources of interoperability problems and how to deal with them


www.hdfgroup.org

netCDF-4 library and arbitrary HDF5 files

• netCDF-4 is built on top of HDF5• netCDF-4 files are HDF5 files with specific

characteristics1. Tracked creation order of objects and their

attributes

2. Absence of datasets with complex compound, and region and object references

3. Presence of dimension scales

• netCDF-4 expects to find properties 1 and 2 in an HDF5 file and fails to open or read the file when those are not present (e.g., the NPOESS files)

• Solution: TB discussed with the netCDF-4 folks04/20/23Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 22

www.hdfgroup.org

Programming models and libraries settings

• Use case 1 (libraries settings): Simultaneous access to an HDF5 file via HDF5 and netCDF-4 • One of the libraries will not be able to open the file due

to the different access properties used by each library (H5F_CLOSE_WEAK in HDF5 vs.

H5F_CLOSE_ STRONG in netCDF-4)

• Solution: • Design improvements in HDF5

• New APIs to detect situation

• Automatic detection and correction of the access properties

• Document the best practices for applications

04/20/23Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 23

www.hdfgroup.org

Programming models and libraries settings

• Use case 2 (programming models): Usage of HDF5 wrapper libraries and netCDF-4 library in the same application• C++ H5Topen call fails due to the netCDF-4 library

shutting down the HDF5 library underneath by calling H5close

• Solution: • Resolved in the newest netCDF-4• Document the best practices for applications

• For more detailed discussion see “NetCDF-4/HDF5 Libraries Interoperability Issues” at

http://www.hdfgroup.uiuc.edu/RFC/HDF5/netCDF4-HDF5/

• Let us know your use case to improve testing! 04/20/23Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 24

www.hdfgroup.orgSep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 25

Thank you !

data interoperability

Documents