ensuring long term access to remotely sensed hdf4 data with layout maps
DESCRIPTION
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps. Ruth Duerr, NSIDC Christopher Lynnes, GES DISC Mike Folk, Kent Yang, Peter Cao, The HDF Group. HDF4 files are complex. How do we save HDF users from having to deal with all of the complexity under the hood?. - PowerPoint PPT PresentationTRANSCRIPT
www.hdfgroup.org
The HDF Group
Ensuring Long Term Access to Remotely Sensed HDF4 Data
with Layout MapsRuth Duerr, NSIDC
Christopher Lynnes, GES DISC
Mike Folk, Kent Yang, Peter Cao, The HDF Group
November 3-5, 2009 1HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
HDF4 files are complex
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 2
www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 3
How do we save HDF users from having to deal with all of
the complexity under the hood?
www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 4
Through the HDF software libraries, either by using the
HDF APIs directly or by using HDF tools that depend on the
HDF libraries.
But what about the future…
www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 5
There is a risk in depending solely on HDF libraries to access HDF-
formatted data over the long term.
It is possible, especially in the distant future, that the libraries may
not be available.
www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 6
“If only we could read HDF data with an read HDF data with an independent program that does not rely on independent program that does not rely on
the HDF API… the HDF API… A possible approach [would be to create] a
map of a data file, [and] utilities to find, assemble and write out SDSes and vdatas.”
“Leveraging HDF Utilities”Christopher LynnesHDF Workshop X.
www.hdfgroup.org
User’s view of the HDF4 SD model
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 7
www.hdfgroup.org
Mapping SDS to file offset/length
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 8
HDF4 file layout
www.hdfgroup.org
Mapping with chunks
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 9
HDF4 file layout
www.hdfgroup.org
Recap
• Problem• The complex byte layout of HDF files makes
long-term readability of HDF data dependent on long-term availability HDF software.
• Solution• Create a map of the layout of data objects in
an HDF file, allowing a simple reader to be written to access the data.
November 3-5, 2009 10HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
The HDF Group
The project – phase 12007-2008
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 11
www.hdfgroup.org
HDF4 mapping project activities
1. Assess and categorize HDF4 data held by NASA • Determine what types of objects to map.
• Get an idea of the magnitude of the project.
2. Develop prototype for proof of concept• Develop markup-language based layout
specification.
• Develop tool to produce layout for an HDF4 file.
• Develop and test two independent tools to read HDF4 data based solely on the map files
November 3-5, 2009 12HDF/HDF-EOS Workshop XIII
www.hdfgroup.orgHDF/HDF-EOS Workshop XIII
How many HDF4 products?
Data Center HDF4 Products
ASF 0
GES-DISC 236
GHRC 54
ASDC 63
LP-DAAC 67
NSIDC 47
ORNL-DAAC 2
PO.DAAC 22
SDAC 0
MrDC 95
Total 586
November 3-5, 2009 13
www.hdfgroup.orgHDF/HDF-EOS Workshop XIII
Data characteristics
• Product Identification• Product Name• Data Level• Archive Location• Product Version
• Whether the product was multi-file• For HDF-EOS products
• HDF-EOS version• For point data
• Number of point data sets• Maximum number of levels
• For swath data• Number of swaths• Maximum number of dimensions• Organized by time, space, both, or other• Whether dimension maps were used
• For gridded data• Number of grids• Max number of dimensions in a grid• Number of projections used• Whether any grids were indexed
• HDF Version
• For raster data• Number of 8-bit rasters• Number of 24-bit rasters• Number of general rasters• Whether any rasters had attributes• Whether any rasters were compressed• Whether any rasters were chunked• Whether there were any palettes
• For SDS data• Number of SDSs• Maximum number of dimensions• Did any SDS have attributes• Was any SDS annotated• Were dimension scales used• Was compression used and if so what kind• Was chunking used
• For Vdata• Number of Vdata structures• Did any Vdata have attributes• Did any Vdata fields have attributes• Was compression used and if so what kind• Was chunking used
Product Characteristics Examined
November 3-5, 2009 14
www.hdfgroup.org
HDF4 mapping prototype workflow
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 15
HDF4 File “H4.hdf”
HDF4 File “H4.hdf”
HDF4 Mapping File (XML document)“H4.hdf.map.xml”
HDF4 Mapping File (XML document)“H4.hdf.map.xml”
hmaplinked with HDF4 library
hmaplinked with HDF4 library
Reader 1(C program)
Object DataObject Data Groups, Data Objects, Structural and Application
Metadata; Locations of Object Data
Reader 2(Perl Script)Reader 2
(Perl Script)
www.hdfgroup.org
The HDF Group
Phase 2: 2009-2011
Productizing HDF4 Mapping schema and tools for deployment
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 16
www.hdfgroup.org
Phase 2 tasks
• Revise schema• Investigate integration of mapping schema with
existing standards• Analyze what’s needed to include HDF-EOS 2• Revise the XML schema
• Implement production quality HDF4 map writer• Develop demo HDF4 map reader• Deploy• Optional tasks
• Implement general purpose reader• Develop validation utilities
November 3-5, 2009 17HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
How you can help
• Project page at The HDF Group website:• http://www.hdfgroup.org/projects/hdf4mapping/
• Consider what it might take to implement this for your archive - contact us if you’d like support
• Let us know if you are interested in participating in any capacity.
November 3-5, 2009 18HDF/HDF-EOS Workshop XIII
www.hdfgroup.org
The HDF Group
Thank You!
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 19
www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or recommendations expressed in this material are
those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space
Administration.
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 20
www.hdfgroup.org
The HDF Group
Questions/comments?
November 3-5, 2009 HDF/HDF-EOS Workshop XIII 21