hdf5 updatemedsbio.org/meetings/bnl_may16_hdrmx_meeting_files/hdf5... · 2016-06-20 · hdrmx...
TRANSCRIPT
![Page 1: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/1.jpg)
TheHDFGroup
www.hdfgroup.org
HDF5Update
High-DataRateMXMee9ngNSLS-IIatBNLMay26,2016
1
![Page 2: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/2.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Outline
2
• HDF5featuresrequestedandsponsoredbySynchrotronCommunity(2012-2016)• DirectchunkI/O(PSI,Dectris)• Dynamicallyloadedfilters(DESY)• SingleWrite/Mul9pleReaderorSWMR(DLS,ESRF,DESY)
• VirtualDataset(VDS)(DLS,DESY,XFEL)• Newrequirementsandfeatures• HDF5roadmapfor2016
![Page 3: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/3.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Directchunkwrite:H5DOwrite_chunk
5/25/16 3
![Page 4: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/4.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
PerformanceresultsforH5DOwrite_chunk
5/25/16 4
1 Speed in MB/s 2 Time in seconds
Test result on Lunux 2.6, x86_64 Each dataset contained 100 chunks, written by chunks
![Page 5: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/5.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Dynamicallyloadedfilters
• Problemswithusingcustomfilters• “Offtheshelf”toolsdonotworkwiththethird-partyfilters• h5dump,MATLABandIDL,etc.
• HDF5toolscannotreadfilecreatedby• h5py,PyTables,etc.
• Solu9on• ModifyHDF5sourcewithyourcodeanddistributeit
• Andwhatwillhappenifauserwantsfiltersfromthedifferentdistribu9ons????Oh….No….
• Usea1.8.11andlater• ProvidemaintainedlibraryofHDF5compressionfilters
55/25/16
![Page 6: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/6.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Approach
• Therearepredefineddefaultloca9onswheretheHDF5librarysearchesthesharedlibrariesorDLLswiththeHDF5filterfunc9ons.
• Thedefaultloca9onmaybeoverwrigenbyanenvironmentvariable.
• Encoding:Standardprogrammingmodeltoregistercompressionfilteronwriteopera9on
• Decoding:Transparentonread;nouserac9onrequired.
• Worksgreat,but….
65/25/16
![Page 7: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/7.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
User’sheadacheandpossiblesolu9ons
• WheretofindHDF5plugins?• Noofficialdistribu9onsite
• ExampleofHDF5filtersdistribu9onsonGithub! hgps://github.com/dectris/HDF5Plugin! hgps://github.com/Blosc/hdf5-blosc! hgps://github.com/nexusformat/HDF5-External-Filter-Plugins! hgps://svn.hdfgroup.org/hdf5_plugins/
• TheHDFGroupcanhelpwith• Maintainingcommonrepository• Tes9ngpluginwiththereleases• Distribu9ngsourceandbinariesforLinux,MacandWindows• Freeandlicensed(forfee)binarydistribu9ons
75/25/16
![Page 8: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/8.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
SWMR:Dataaccesstofilebeingwrigen
HDF5File
Writer Reader
…whichcanbereadbyareader…
withnoIPCnecessary.
Newdataelements…
…areaddedtoadatasetinthefile…
04/01/16
![Page 9: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/9.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
SWMR
9
• ReleasedinHDF51.10.0• Restrictedtoappend-dataonlyscenario• SWMRdoesn’tworkonNFS• Filesarenotcompa9blewithHDF51.8.*libraries
• Useh5format_converttool• ConvertsHDF5metadatainplace• Norawdataisrewrigen
![Page 10: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/10.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDS
10
• Datastoredinmul9plefilesanddatasetscanbeaccessedviaonedataset(VDS)usingstandardHDF5read/write
![Page 11: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/11.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDSExample
11
Seriesofimages
A
B
C
D
EF
a.h5
c.h5
e.h5
b.h5
d.h5
f.h5
A
C
E
B
D
F
t1 t2
t3 VDS.h5
111Imageat9met2
A
B
C
D
EF
A
B
C
D
EF
AADataset A
Dataset C
Dataset E
CC
Dataset B
DatasetD
Dataset FF
F
DD
EE
BB
M
n
kn
nk
k
n k n
n k
k
M
t2
t2
t2
t2
t2
t2
![Page 12: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/12.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
VDS
12
• VDSworkswithSWMR• FilewithVDScannotbeaccessedbyHDF51.8.*libraries
• Useh5repacktooltorewritedata(1.10.0-patch1)
![Page 13: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/13.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Newrequirementsandfeatures?
13
• Tellusyourneeds:• Mul9-threadedcompressionfilters• H5DOread_chunkfunc9on• FullSWMRimplementa9on• Performance• Backward/forwardcompa9bility
• Otherrequests?
![Page 14: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/14.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
HDF5Roadmapfor2016
14
• May31-HDF51.10.0-patch1• H5repack,Windowsbuilds,FortranissuesonHPCsystems
• Mid-summerHDF51.10.1• Someinternalissuesfoundfor1.10.0
• December• HPCfeaturesthatdidn’tmakeitinto1.10.0release
![Page 15: HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community](https://reader034.vdocuments.site/reader034/viewer/2022050605/5facfd983d961c425312bfb1/html5/thumbnails/15.jpg)
www.hdfgroup.orgHDRMXNSLS-IIatBNL
Ques9ons?
Thankyou!
15