LANL Hyperspectral Image LANL Hyperspectral Image ProcessingProcessing

Begin periodic

evaluation of clutter

Subtract mean

spectral vector

NxJxL ops

1 row of i-grams is complete

N pixels x 2LInterferogram accumulator

N x (2L)2 buffer w/ rolling pointer

Readouts: N spatial pixels

by 2L i-gram samples

Alarms

N FFTs

N x 2L ln(2L) ops

Divide sum

by NxJ

L opsIncrement

mean vector sums

N x L ops

Fast on-board processing for broad area search

Factor covariance

matrix

Order L3 opsForm

covariance matrices

NxJxL2 ops

Average: L2 ops

Buffer spectra

Evaluate matched filter(s)

NxJxLxM ops

NxJxL

Buffer full

Every J timesteps

M(2L2+L) ops

Form matched filter(s)

M signatures

Threshold exceeded

MxNxJ ops

Record detection

L channels; J rows in buffer; M spectral signatures; N pixels in swath

Every timestep: Every J steps:

Goals•Collect•Analyze•Reacton Deployed Platform

Barriers:•data intensive •compute intensive

HIRIS Example:Sensor Data at 67MB/sec

Problems with conventional solution:It takes 3.75 minutes to process 3 sec. of data on a high end multiprocessor server after the data has been returned to the ground.There is a critical need for real-time feedback at the sensor.

HIRIS data processing chain

Solution•Reconfigurable Computers

Parallel OperationsHigh Memory Bandwidth

•Algorithms tailored for hardware

HIRIS - Hyperspectral InfraRed Imaging System

HIRIS is an airborne technology demonstration of an imaging Fourier transform spectrometer.

Goal: Broad Area Search with Passive Hyperspectral ImagerFly a Michelson interferometer with an add-on imaging system on an airplane to detect chemical plumes at 20km.

•Imaging Fourier transform spectrometry from 8 to 17 microns.•Need pointing and tracking system to stare for several seconds with very high

accuracy and low jitter.•Need extremely low NESR to detect and distinguish effluent gases in LWIR

Status:•Numerous HIRIS flight campaigns with two instruments on two airplanes.•Successful detection of gaseous chemical plumes by spatial/spectral analysis.•Active program to refine analysis and exploitation algorithms and test them on

both flight and modeled data.•LANL modeling and exploitation has produced an end-to-end model, analysis

of HIRIS data, and a number of new algorithms.

Data Source:Fourier Transform Imaging Spectrometer

Instrument Data Characteristics:

• Spatial Resolution: 128x128 pixels• 14 Bits of data per pixel• Mirror Positions: 4000+• Number of mirror positions not

necessarily a power of 2• Mirror Scan Time over all locations:

3 seconds• Each spatial image is collected before

the mirror is moved again.• Image Data cube of contains an

Interferogram of theoptical spectrum at each

spatial pixel.• The FOV of the instrument may move

slightly between image collections. Such translations need to be removed, preferably at the sub-pixel level

Raw Data Rate: 67 Megasamples/second

Difficulties with hyperspectral image analysis

Complexity of spectra - spectral clutter:•Atmosphere: To model all features at high resolution is a huge job.•Background surfaces: Variety of spectral emissivities.•Gases: Molecular band structures are function of temperature.

Mitigation: Construct a matched filter for each gas incl. Water vapor.

Complexity of background - spatial clutter•Material variation, both under plume and on plume•Variations of water vapor over short distances

Mitigation: Preclassify scene, perform spectral analysis by class Fit plume shape iteratively to physical dispersion model.

Spectrum of 1Pixel

Background

Categorize hyperspectral image analysis

1. Precise Chemical Analysis: (2 alternative approaches)1. Statistical

Principal Component Analysis followed by Adaptive Matched FilterPast and current main focus of LANL effortProcessing chain validated and being used

2. ExplicitClassify and find spectral end-members followed by Spectral FittingCurrent LANL research effortClassify with K-Means Alg. End Member with Pixel Purity Index Alg and Nfindr Alg.

2. Detection1. Real-time analysis with no human analyst input

Based on further Hardware Acceleration of Statistical Approach but reduced precision because:• Front End data reduction• Less precise background compensation since no analyst

Future LANL research effort

Desired: All of the following must be performed in less than 3 seconds:• Step 1: Register each image slice with the next• Step 2: Perform a FT on each pixel’s Interferogram

• -> 1634 4096 pt FTs• Step 3: Combine and average symmetric portions of transforms

• Actual number of valid wavenumber bins is 251• Step 4: Apply wavelength and pixel dependent gains and offsets

• Calibration table generated from subset of data and instrument parameters• Step 5: Sum the datacube in the wavenumber dimension and produce a

broadband image• Step 6: Compute the * Covariance Matrix of the image background• Step 7: Use the covariance matrix on 32 library spectral templates to produce 32

matched filters.• Step 8: Produce 32 images by performing a * Spectral Dot Product of the matched

filters on the datacube.

Statistical Analysis Overview (Example)

* Focus of ACS acceleration to date

(a) Original Image

(b) Image plus 5% plume of SO2

(c) Using SO2 spectrum to look for the plume

(d) Using orthogonalized SO2 spectrum to look for the plume

Example of Statistical Analysis with Simulated SO2 plume

L=630

N=128

J=128

DP1 DP2 DPM

L(n

)

...

Covariance

MACS = NJL(L/2) = 3.25e9

RCC Computation Time = NJL(L/2)/Mfclk = 1.3s fclk = 50MHz, M=50

(M is number of DPs in FPGA)

Benchmark = 71.5s (733MHz PC)

Acceleration 55X

…L2,L1

Local mem

Work in Progress – Acceleration of Statistical Analysis

L=630

N=128

J=128

DP1 DP2 DPM

F1

co

effs

...

...

Matched Filter

Total MACS = NJLM = 516e6

TCOMP = NJL/fclk = 0.2s fclk = 50MHz

Benchmark = 20.9s (733MHz PC)

Acceleration 105X

…p2,p1

MACs

M Matched filters

F2

co

effs

FM

co

effs

Local mem

Work in Progress – Acceleration of Statistical Analysis

class 0

class 1

class 2

class 3

class 4

Completed Work – Acceleration of Explicit Analysis

K-MEANS: Unsupervised image classification: ‘cluster’ the pixels into a few number of classes such that pixels belonging to the same class have similar spectral properties

Initialization: Randomly assign pixels to clusters.

Step 1: Find new cluster centers using current cluster assignments.

Use the mean of pixels assigned to that cluster.

Step 2: Assign pixels to clusters.

Use minimum distance from pixel to cluster center.

Iterate Steps 1 and 2

Completed Work – Acceleration of Explicit Analysis

Pixel Purity Index (PPI): Identify “pure” pixels (or end-members) from multi-spectral images

•Set N random D-vectors: skewers

•for each skewer:

–compute a dot-product with all the pixels

–the two pixels having produced the highest and the lowest dot-product are potential pure pixels

•Select the “pure” pixels

–i.e. the pixels which have been selected several times as potential pure pixels

Future Work – Acceleration of Detection Algorithm

Goal: Chemical Detection at the frame rate of the sensor

Approach: Evaluate ways to accelerate statistical processing chain if we are willing to give up a little in sensitivity.

Plan:

Work with HIRIS experts to identify end to end processing steps

Modify processing by

a) performing data reduction at front end

b) Matching processing chain to FPGA strengths

c) More automated background compensation

Allow some reduction in sensitivity because

a) we remove the human analyst from the process

b) front end data reduction

Develop software model of hardware processing chain

Take advantage of BYU JHDL Module Generators

Target Hardware: New USC/ISI Virtex2 based RCC “Osiris”

Future Work – Acceleration of Detection Algorithm

Tasks• Generate Detailed Task List

• Explore Architectural Trades of a Heterogeneous System

• Develop Software Model of Detection Algorithm

• Code an Implementation on “Osiris”

• Caveat: Acceleration will depend on a FFT. While the FFT is feasible, development of a high performance FFT is beyond the scope of this effort. An early task is to address this issue.

Level of Effort• LANL - 1 FTE: Herb Fry, Kevin McCabe, Tony Nelson

• ISI - 1 FTE: Ron Riley, Matt French

• BYU – 1 student?

Future Work – Acceleration of Detection Algorithm

Schedule• June : Detailed Task List

• June – August : Develop initial software model

• June – August : Identify Modules

• July : Address high performance FFT issue (what size FFTs are available and how fast are they?)

• End of July : Checkpoint and Demo definition

• August – November : Refine software model and begin detailed design

• November : Workshop

• Spring ’02 : JHDL Module and VHDL coding for “Osiris”

• August ’02 : Demo: probably a detection “movie”

Future Work – Acceleration of Detection Algorithm

Dependencies

• FFT Solution

• New ISI “Osiris” RCC

• BYU resources