overview of noaa/nesdis goes-r hyperspectral sounder data compression study bormin huang, allen...
Post on 14-Jan-2016
216 Views
Preview:
TRANSCRIPT
Overview of NOAA/NESDIS GOES-R
Hyperspectral Sounder Data Compression Study
Bormin Huang, Allen Huang, Alok Ahuja
Cooperative Institute for Meteorological Satellite Studies
University of Wisconsin-Madison
4th MURI Workshop, April 27-28
What is hyperspectral sounding data?• It is generated from an interferometer (e.g. HIS, AERI, CrIS, IASI) or a grating sounder (e.g. AIRS).• It consists of several thousand spectral channels that span the infrared region on the order of one wavenumber or lessWhat is hyperspectral sounding data for?• to retrieve - atmospheric temperature, water vapor, and trace gases profiles; - cloud & aerosol properties, - surface temperature, emissivities, etc.,• to derive wind from radiance or retrieved water vapor fields,for better weather and climate prediction.Why does it require compression?• Unprecedented volume of 3D data that consists of one spectral and two spatial dimensions (~10-100 GB per day) ;• Beneficial to efficient data transfer and archive. What is new for the data compression society?• High correlation among remote disjoint channels due to the absorption of the same absorbing gases.• Lossy compression needs retrieval impact studies, i.e. interdisciplinary knowledge in data compression and remote sensing is needed!Why is lossless or near-lossless compression desired?Physical retrieval of atmospheric temperature and absorbing gases is a mathematically ill-posed problem, i.e. sensitive to data error and noise!
Lossless Compression Study
• 2D Wavelet-Based Compression SchemeJPEG2000: 2D IWT → Bitplane Coding → Entropy Coding
• 3D Wavelet-Based Compression Schemes3D IWT → 3D EZW → Entropy Coding3D IWT → 3D SPIHT → Entropy Coding3D IWT → 1D BWT → Entropy Coding
• 2D Predictor-Based Compression SchemesCALIC : 2D Gradient-adjusted Prediction → Entropy CodingJPEG-LS : 2D Nonlinear Prediction → Entropy Coding
2D Wavelet Transform
Integer Wavelet Transform (Lifting Scheme)
JPEG2000
A new ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) compression standard.
• Successor to the DCT (discrete cosine transform)-based JPEG algorithm.
Wavelet based Schemes
IWT with 3 stages (Taubman et. al. 2000)
3D Wavelet Tree Coding
3D EZW: It uses the spatial hierarchical tree relationship of the wavelet transform coefficients for efficient compression.
3D SPIHT: Refinement of the EZW scheme that provides better compression while having faster encoding and decoding times.
Parent-child interband relationship and locations for EZW and SPIHT coding
Binarymode
ContextQuantization
Context Formation
Two-line Buffer
Error Modeling
Ternary EntropyCoder
ConditionalProbabilitiesEstimation
CodingHistogramSharpening
EntropyCoder
GradientAdjusted
Prediction
-
+
yes
no
I
I
e
e
I
codestream
2D Context-based Adaptive Lossless Image Codec (CALIC)
• Among the nine proposals in the initial ISO/JPEG evaluation in July 1995, CALIC was ranked first.• It is considered the benchmark for lossless compression of continuous-tone images.
Predictor-Based Schemes
n ne
nnenn
nw
www ?
i
j
Schematic description of the CALIC encoderNeighboring pixels used in prediction (Wu et. al. 1997)
PredictionContext
ModelingError
Encoding
Run-lengthCoder
Imagepixels
RegularMode
compressedbitstream
RunMode
FlatRegion?
No
Yes
2D JPEG-LS
• Published in 1999 as a lossless compression standard of the ISO/IEC.
c b
x
d
a
Schematic description of the JPEG-LS encoder
Neighborhood of JPEG-LS used in prediction
Burrows Wheeler Transform• Block-sorting compression scheme [Burrows et al, 1994]• Rearranges the positions of the data such that the few distinct values under the same previous context are grouped together in position.
tennessee* tennessee*ennessee*t *tennesseennessee*te ssee*tennenessee*ten e*tennesseessee*tenn nnessee*tessee*tenne nessee*tensee*tennes essee*tennee*tenness see*tennese*tennesse ee*tenness*tennessee ennessee*t
An example of the Burrows-Wheeler transform.
bwt(tennessee*) = t*sennesee. The matrix on the right is obtained by sorting the rows of the left matrix in right-to-left lexicographic order. * denotes end of the data block and can be considered as the smallest symbol.
Granule 9 00:53:31 UTC -12 H (Pacific Ocean, Daytime)
Granule 16 01:35:31 UTC +2 H (Europe, Nighttime)
Granule 60 05:59:31 UTC +7 H (Asia, Daytime)
Granule 82 08:11:31 UTC -5 H (North America, Nighttime)
Granule 120 11:59:31 UTC -10 H (Antarctica, Nighttime)
Granule 126 12:35:31 UTC -0 H (Africa, Daytime)
Granule 129 12:53:31 UTC -2 H (Arctic, Daytime)
Granule 151 15:05:31 UTC +11 H (Australia, Nighttime)
Granule 182 18:11:31 UTC +8 H (Asia, Nighttime)
Granule 193 19:17:31 UTC -7 H (North America, Daytime)
Ten selected AIRS granules on Sept. 6, 2002
AIRS radiance field at wavenumber 900.3cm-1 for the selected granules
AIRS radiance field at wavenumber 900.3cm-1 for the selected granules
1.55
1.60
1.65
1.70
1.75
1.80
1.85
1.90
1.95
2.00
2.05
9 16 60 82 120 126 129 151 182 193
Granule
Com
pres
sion
Rat
io
JPEG-LS
JPEG2000
3D SPIHT
CALIC
BWT
3D EZW
Compression ratios of different algorithms for the 10 selected AIRS granules
Bias-Adjusted Reordering (BAR)* Scheme for Data Preprocessing
• Hyperspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases at various altitudes.
• The Bias-Adjusted Reordering (BAR) scheme is used for exploring the correlation among remote disjoint channels.
• The technique can be used to improve the compression ratio of any existing scheme.
•The BAR scheme paper is accepted to be published in Optical Engineering. We are in the process of patent application.
Effect of the BAR scheme on various compression algorithms for the 10 selected AIRS granules
1.55
1.60
1.65
1.70
1.75
1.80
1.85
1.90
1.95
2.00
2.05
2.10
9 16 60 82 120 126 129 151 182 193
Granule
Com
pres
sion
Rat
io
JPEG-LS
BAR+JPEG-LS
JPEG2000
BAR+JPEG2000
3D SPIHT
BAR+3D SPIHT
CALIC
BAR+CALIC
BWT
BAR+BWT
3D EZW
BAR+3D EZW
• In support of the NOAA/NESDIS GOES-R data processing studies, we investigated lossless compression of 3D hyperspectral sounding data using wavelet-based schemes (3D EZW, 3D SPIHT, JPEG2000) and predictor-based schemes (CALIC, JPEG-LS).
• The performance rank from best to worst in terms of compression ratios before the BAR scheme is given in the order of JPEG-LS, 3D SPIHT, JPEG2000, CALIC, BWT and 3D EZW.
• The performance rank from best to worst in terms of compression ratios after the BAR scheme is given in the order of JPEG-LS, JPEG2000, CALIC, 3D SPIHT, BWT and 3D EZW.
• To take advantage of the spectral correlations, we applied the BAR scheme to significantly improve the compression performance of all the compression algorithms.
Acknowledgement: This research is supported by NOAA NESDIS OSD under grant NA07EC0676.
Summary
top related