access hdf5 datasets via opendap's data access protocol (dap)
DESCRIPTION
The Data Access Protocol(DAP) is a protocol for requesting and transporting data across the web. By using OPeNDAP servers data can be made accessible to remote locations regardless of local storage format. A prototyped HDF5-OPeNDAP server was implemented several years ago. With the HDF5 being selected as the primary data format of NASA Aura satellite, a robust, fully supported HDF5-OPeNDAP server is necessary. Recently NASA Headquarters Science Mission Directorate Advancing Collaborative Connections For Earth-Sun System Science (ACCESS) program has awarded a joint proposal to implement a fully supported HDF5-OPeNDAP server. This poster will discuss the design and implementation plan for this project.TRANSCRIPT
![Page 1: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/1.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
1
Access HDF5 Datasets via OPeNDAP’s Data
Access Protocol (DAP)MuQun Yang
The HDF Group
![Page 2: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/2.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
2
OPeNDAP
• A software framework that allows simple access to remote data• Client-server model• Servers make the local data accessible remotely• Clients make applications easily access to remote served data• Widely used by Earth Science Community• URL: http://www.opendap.org/
![Page 3: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/3.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
3
?sst[10:10][0:90][0:180]
Constraint
The Data RequestThe Data Request
Data are requested with a URL.
http://www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Reynolds_sst
Protocol Machine name OPeNDAP server Directory File name
The user can impose a constraint on the data that s/he would like to acquire from a data set by appending a constraint expression to the end of the URL.
![Page 4: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/4.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
4
OPeNDAPOPeNDAP Data Types Data Types
OPeNDAP provides three basic data objects:
data descriptor structure (.dds) - syntactic metadata - rigid
data attribute structure (.das) - semantic metadata - flexible
data (.dods) - the actual data in a binary structure
![Page 5: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/5.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
5
OPeNDAP ServersOPeNDAP Servers
Data Data Data Data Data Data Data
Matlab
HDF4 JDBC
FreeFromFITS
CDF CEDAR
Data
netCDF
netCDF HDF4 Matlab
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
CODAR
Data
ESML
GeneralCODAR
No fully-supported HDF5 server
![Page 6: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/6.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
6
OPeNDAP ClientsOPeNDAP Clients
netCDF C netCDF Java
IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access
MatlabClient
IDLClient
No HDF5 client
![Page 7: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/7.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
7
Why HDF5-OPeNDAP server and client?
Users!
Users!!
Users!!!
![Page 8: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/8.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
8
Task:
•10 elements of 17 MB HDF5 data•200 elements of 10 MB NetCDF data•150 elements of 5 MB HDF4 data
A User Case
![Page 9: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/9.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
9
General approach
FTP
HDF5 data
FTP
HDF4 data
FTP
NetCDF data
![Page 10: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/10.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
10
Approach with OPeNDAP
HDF4 data
NetCDF data
FTP
HDF5 data
server
server
DAP
DAP
client
![Page 11: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/11.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
11
Conclusion
• HDF5-DAP server is required
• HDF5-DAP client is strongly recommended
![Page 12: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/12.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
12
• Incomplete mapping from HDF5 to OPeNDAP
• No comprehensive test suite• Only tested a few examples on two
platforms• Incomplete documentation
Previous work - A Prototype of DAP2-HDF5
server
![Page 13: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/13.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
13
An example from SSMI
From HDF4 server at JPL
From HDF5 server at NCSA
![Page 14: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/14.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
14
Current HDF5-OPeNDAP Support
DAP
Prototyped Server No Client
HDF5
![Page 15: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/15.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
15
HDF5-OPeNDAP Project
• Two-year project awarded by NASA ROSES NRA
• Funding much less than expected• Original proposed work kept
• with significant contributions from ESDIS
• Starting date: Nov. 15th, 2006
![Page 16: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/16.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
16
Proposed Work – software development
• Implementation• HDF5-DAP2 server• HDF5-DAP4 server
• Deployment • Release servers to OPeNDAP
community
![Page 17: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/17.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
17
HDF5-DAP2 server
• Update and complete the prototype, including make files, instructions, documentation, etc
• Add a few selected features to handle NASA Aura data and NPOESS data
- Map HDF5 group to DAP(?)- Map Object/regional reference to DAP(?)
• Implement a comprehensive test suite• Test on platforms supported by OPeNDAP• Demonstrate with sample NASA data
![Page 18: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/18.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
18
Fully-Supported HDF5 Server
DAP
HDF5HDF5 server
DAP
HDF5HDF5 serverMatlab/Ferret Client
![Page 19: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/19.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
19
Proposed Work – software development
• Implementation• DAP4 to HDF5 conversion utility
• Deployment • Release utility to OPeNDAP
community
• Design study • Feasibility of DAP-aware HDF5 library
![Page 20: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/20.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
20
Proposed Work - DAP to HDF5 utility
• A command line tool that converts DAP to HDF5 file• individual DAP variable• subsetting
DAP
HDF5HDF5 serverConversion tool
![Page 21: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/21.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
21
Proposed work - documentations
• DAP4 specification• HDF5-DAP4 mapping document• DAP4 to HDF5 specification• RFC for DAP-aware HDF5 library• Submit specifications to ESDSWG
![Page 22: Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)](https://reader035.vdocuments.site/reader035/viewer/2022062319/55639466d8b42ad2128b5588/html5/thumbnails/22.jpg)
04/12/23 HDF and HDF-EOS Workshop X, Landover, MD
22
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNG05GC60A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.