Overview of NOAA/NESDIS GOES-R

Hyperspectral Sounder Data Compression Study

Bormin Huang, Allen Huang, Alok Ahuja

 Cooperative Institute for Meteorological Satellite Studies

University of Wisconsin-Madison

4th MURI Workshop, April 27-28

What is hyperspectral sounding data?• It is generated from an interferometer (e.g. HIS, AERI, CrIS, IASI) or a grating sounder (e.g. AIRS).• It consists of several thousand spectral channels that span the infrared region on the order of one wavenumber or lessWhat is hyperspectral sounding data for?• to retrieve - atmospheric temperature, water vapor, and trace gases profiles; - cloud & aerosol properties, - surface temperature, emissivities, etc.,• to derive wind from radiance or retrieved water vapor fields,for better weather and climate prediction.Why does it require compression?• Unprecedented volume of 3D data that consists of one spectral and two spatial dimensions (~10-100 GB per day) ;• Beneficial to efficient data transfer and archive. What is new for the data compression society?• High correlation among remote disjoint channels due to the absorption of the same absorbing gases.• Lossy compression needs retrieval impact studies, i.e. interdisciplinary knowledge in data compression and remote sensing is needed!Why is lossless or near-lossless compression desired?Physical retrieval of atmospheric temperature and absorbing gases is a mathematically ill-posed problem, i.e. sensitive to data error and noise!

Lossless Compression Study

• 2D Wavelet-Based Compression SchemeJPEG2000: 2D IWT → Bitplane Coding → Entropy Coding

• 3D Wavelet-Based Compression Schemes3D IWT → 3D EZW → Entropy Coding3D IWT → 3D SPIHT → Entropy Coding3D IWT → 1D BWT → Entropy Coding

• 2D Predictor-Based Compression SchemesCALIC : 2D Gradient-adjusted Prediction → Entropy CodingJPEG-LS : 2D Nonlinear Prediction → Entropy Coding

2D Wavelet Transform

Integer Wavelet Transform (Lifting Scheme)

JPEG2000

A new ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) compression standard.

• Successor to the DCT (discrete cosine transform)-based JPEG algorithm.

Wavelet based Schemes

IWT with 3 stages (Taubman et. al. 2000)

3D Wavelet Tree Coding

3D EZW: It uses the spatial hierarchical tree relationship of the wavelet transform coefficients for efficient compression.

3D SPIHT: Refinement of the EZW scheme that provides better compression while having faster encoding and decoding times.

Parent-child interband relationship and locations for EZW and SPIHT coding

Binarymode

ContextQuantization

Context Formation

Two-line Buffer

Error Modeling

Ternary EntropyCoder

ConditionalProbabilitiesEstimation

CodingHistogramSharpening

EntropyCoder

GradientAdjusted

Prediction

-

+

yes

no

I

I

e

e

I

codestream

2D Context-based Adaptive Lossless Image Codec (CALIC)

• Among the nine proposals in the initial ISO/JPEG evaluation in July 1995, CALIC was ranked first.• It is considered the benchmark for lossless compression of continuous-tone images.

Predictor-Based Schemes

n ne

nnenn

nw

www ?

i

j

Schematic description of the CALIC encoderNeighboring pixels used in prediction (Wu et. al. 1997)

PredictionContext

ModelingError

Encoding

Run-lengthCoder

Imagepixels

RegularMode

compressedbitstream

RunMode

FlatRegion?

No

Yes

2D JPEG-LS

• Published in 1999 as a lossless compression standard of the ISO/IEC.

c b

x

d

a

Schematic description of the JPEG-LS encoder

Neighborhood of JPEG-LS used in prediction

Burrows Wheeler Transform• Block-sorting compression scheme [Burrows et al, 1994]• Rearranges the positions of the data such that the few distinct values under the same previous context are grouped together in position.

tennessee*          tennessee*ennessee*t          *tennesseennessee*te          ssee*tennenessee*ten          e*tennesseessee*tenn          nnessee*tessee*tenne          nessee*tensee*tennes          essee*tennee*tenness          see*tennese*tennesse          ee*tenness*tennessee          ennessee*t

            An example of the Burrows-Wheeler transform.

bwt(tennessee*) = t*sennesee. The matrix on the right is obtained by sorting the rows of the left matrix in right-to-left lexicographic order. * denotes end of the data block and can be considered as the smallest symbol.

Granule 9 00:53:31 UTC -12 H (Pacific Ocean, Daytime)

Granule 16 01:35:31 UTC +2 H (Europe, Nighttime)

Granule 60 05:59:31 UTC +7 H (Asia, Daytime)

Granule 82 08:11:31 UTC -5 H (North America, Nighttime)

Granule 120 11:59:31 UTC -10 H (Antarctica, Nighttime)

Granule 126 12:35:31 UTC -0 H (Africa, Daytime)

Granule 129 12:53:31 UTC -2 H (Arctic, Daytime)

Granule 151 15:05:31 UTC +11 H (Australia, Nighttime)

Granule 182 18:11:31 UTC +8 H (Asia, Nighttime)

Granule 193 19:17:31 UTC -7 H (North America, Daytime)

Ten selected AIRS granules on Sept. 6, 2002

AIRS radiance field at wavenumber 900.3cm-1 for the selected granules

AIRS radiance field at wavenumber 900.3cm-1 for the selected granules

1.55

1.60

1.65

1.70

1.75

1.80

1.85

1.90

1.95

2.00

2.05

9 16 60 82 120 126 129 151 182 193

Granule

Com

pres

sion

Rat

io

JPEG-LS

JPEG2000

3D SPIHT

CALIC

BWT

3D EZW

Compression ratios of different algorithms for the 10 selected AIRS granules

Bias-Adjusted Reordering (BAR)* Scheme for Data Preprocessing

• Hyperspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases at various altitudes.

• The Bias-Adjusted Reordering (BAR) scheme is used for exploring the correlation among remote disjoint channels.

• The technique can be used to improve the compression ratio of any existing scheme.

•The BAR scheme paper is accepted to be published in Optical Engineering. We are in the process of patent application.

Effect of the BAR scheme on various compression algorithms for the 10 selected AIRS granules

1.55

1.60

1.65

1.70

1.75

1.80

1.85

1.90

1.95

2.00

2.05

2.10

9 16 60 82 120 126 129 151 182 193

Granule

Com

pres

sion

Rat

io

JPEG-LS

BAR+JPEG-LS

JPEG2000

BAR+JPEG2000

3D SPIHT

BAR+3D SPIHT

CALIC

BAR+CALIC

BWT

BAR+BWT

3D EZW

BAR+3D EZW

• In support of the NOAA/NESDIS GOES-R data processing studies, we investigated lossless compression of 3D hyperspectral sounding data using wavelet-based schemes (3D EZW, 3D SPIHT, JPEG2000) and predictor-based schemes (CALIC, JPEG-LS).

• The performance rank from best to worst in terms of compression ratios before the BAR scheme is given in the order of JPEG-LS, 3D SPIHT, JPEG2000, CALIC, BWT and 3D EZW.

• The performance rank from best to worst in terms of compression ratios after the BAR scheme is given in the order of JPEG-LS, JPEG2000, CALIC, 3D SPIHT, BWT and 3D EZW.

• To take advantage of the spectral correlations, we applied the BAR scheme to significantly improve the compression performance of all the compression algorithms.

Acknowledgement: This research is supported by NOAA NESDIS OSD under grant NA07EC0676.

Summary