1

Recent Advances in Real-Time Recent Advances in Real-Time

Hyperspectral Image ProcessingHyperspectral Image Processing

Mingkai HsuehMingkai Hsueh

Remote Sensing Signal and Image Processing LaboratoryRemote Sensing Signal and Image Processing Laboratory

Department of Computer Science and Electrical EngineeringDepartment of Computer Science and Electrical Engineering

University of Maryland Baltimore CountyUniversity of Maryland Baltimore County

1000 Hilltop Circle, Baltimore, MD 212501000 Hilltop Circle, Baltimore, MD 21250

2

Outline

Introduction to Hyperspectral Image Processing and its Introduction to Hyperspectral Image Processing and its

ApplicationsApplications

Anomaly DetectionAnomaly Detection

Anomaly Detection Anomaly Detection

Real-time implementationReal-time implementation

Speed-up of Adaptive Causal Anomaly DetectionSpeed-up of Adaptive Causal Anomaly Detection

ConclusionsConclusions

ProjectsProjects

3

0

1000

2000

3000

4000

5000

300 600 900 1200 1500 1800 2100 2400

Wavelength (nm)R

efl

ecta

nce

0

1000

2000

3000

4000

300 600 900 1200 1500 1800 2100 2400

Wavelength (nm)

Refl

ecta

nce

Water

Mixed pixel(soil + mineral)

Mixed pixel(trees + soil)

0

1000

2000

3000

4000

300 600 900 1200 1500 1800 2100 2400

Wavelength (nm)

Refl

ecta

nce

Hyperspectral Image

4

Applications of Hyperspectral Image Processing

ApplicationsApplications Man-made objects: canvas, camouflage, Man-made objects: canvas, camouflage,

military vehicles in defense applicationsmilitary vehicles in defense applications Toxic waste, oil spills in environmental Toxic waste, oil spills in environmental

monitoringmonitoring LandminesLandmines Trafficking in law enforcementTrafficking in law enforcement Chemical/biological agent detectionChemical/biological agent detection Special species in agriculture, ecologySpecial species in agriculture, ecology

5

Types of Signatures

Endmembers: Endmembers: Pure signatures for a spectral class used Pure signatures for a spectral class used

for spectral unmixingfor spectral unmixing Anomalies:Anomalies: Signals/signatures spectrally distinct fromSignals/signatures spectrally distinct from

their surroundings, i.e., abnormality.their surroundings, i.e., abnormality. rare minerals in geologyrare minerals in geology abnormal activities in military abnormal activities in military

applications.applications.

6

RX Algorithm

RX algorithm basically performs RX algorithm basically performs the Mahalanobis distance that is the Mahalanobis distance that is specified by specified by

(r(rii--))TT ×× (K) (K)-1 -1 ×× (r (ri i --))

The required mean vector The required mean vector μμ hinder the possibility of hinder the possibility of implementing the algorithm in implementing the algorithm in real-time fashion.real-time fashion.

7

Causal RX Filter (CRXF)

By replacing the covariance matrix by By replacing the covariance matrix by correlation matrix, we can achieve the correlation matrix, we can achieve the real-time real-time processingprocessing..

The functional form of CRXFThe functional form of CRXF

rriiTT ×× ( (RRii))-1 -1 ×× r rii

The major drawback is that if a detected The major drawback is that if a detected anomaly remains on the image to be processed, it anomaly remains on the image to be processed, it may decrease the detectability of the following may decrease the detectability of the following anomalies.anomalies.

8

Adaptive Causal Anomaly Detector (ACAD)

ACAD has the same functional form as does ACAD has the same functional form as does CRXF, except the sample correlation matrix CRXF, except the sample correlation matrix R’R’ is is formed by all the arrived pixel vectors except the formed by all the arrived pixel vectors except the detected anomalous target pixel vectors that have detected anomalous target pixel vectors that have been removed.been removed.

rriiTT ×× ( (R’R’ii))-1 -1 ×× r rii

An anomalous target map is generated at the same An anomalous target map is generated at the same time as the detection process takes place.time as the detection process takes place.

9

HYDICE Data

HYDICE (Hyperspectral Digital Imagery Collection Experiment)HYDICE (Hyperspectral Digital Imagery Collection Experiment) 15 panels of five types with three different materials.15 panels of five types with three different materials. They are arranged into a matrix in such a way that each row represents 3 They are arranged into a matrix in such a way that each row represents 3

panels of the same type with three different sizes, 3panels of the same type with three different sizes, 3mm33mm, 2, 2mm22mm, , 11mm11mm. Each column represents 5 panels of different types with the same . Each column represents 5 panels of different types with the same size.size.

Original image Target masked image

Anomaly

10

CRXF Results

row 8 row 16 row 24 row 32

row 40 row 48 row 56 row 64

11

ACAD Results

row 8 row 16 row 24 row 32

row 40 row 48 row 56 row 64

12

ACAD Target Map

row 8 row 16 row 24 row 32

row 40 row 48 row 56 row 64

13

ACAD Hardware Design

Ri = Ri-1 + ri × riT

(Ri)-1 = (Qi × Riupper )-1

= ( Riupper )-1 × Qi

T

δACAD (ri) = riT × (Ri

T)-1 × ri

tK ≤ τ

Auto CorrelatorAuto Correlator

QR Matrix InverseQR Matrix Inverse

Abundance CalculationAbundance Calculation

Anomalous Target DiscriminatorAnomalous Target Discriminator

14

(A+BCD)-1 = A-1 – A-1B(C-1+DA-1B)-1 DA-1

(A+rrT)-1 = A-1 – (A-1rrT A-1) / (1+rTA-1r)

By Woodbury’s identity, set B a column vector, C a scalar of unity, and D a row vector

Let Let AA be the current correlation matrix and be the current correlation matrix and rr be the incoming pixel vector.be the incoming pixel vector.

Matrix Inversion Lemma

15

Matrix Inversion Lemma (Cont’d)

With Matrix Inversion Lemma (MIL), we With Matrix Inversion Lemma (MIL), we

only need to computeonly need to compute

Using MIL the matrix inversion is reduced Using MIL the matrix inversion is reduced

to matrix multiplications. to matrix multiplications.

Simulation is provided to evaluate the Simulation is provided to evaluate the

performance of MIL.performance of MIL.

rRr1

RrrR1T

1T1

16

ACAD Hardware Design

Ri = Ri-1 + ri × riT

(Ri)-1 = (Qi × Riupper )-1

= ( Riupper )-1 × Qi

T

δACAD (ri) = riT × (Ri

T)-1 × ri

tK ≤ τ

i

1i

Ti

-1i

Tii

-1i1-

i

1Tiii rRr1

RrrRRrrR

Auto CorrelatorAuto Correlator

QR Matrix InverseQR Matrix Inverse

Matrix Inversion LemmaMatrix Inversion Lemma

Abundance CalculationAbundance Calculation

Anomalous Target DiscriminatorAnomalous Target Discriminator

17

Speed-up of MIL

We use two versions of the MATLAB program to We use two versions of the MATLAB program to

perform the ACAD on the same image cube. One uses perform the ACAD on the same image cube. One uses

the MATLAB inv() function and another one uses the the MATLAB inv() function and another one uses the

MIL.MIL.

As we can see, the speed-up is about “2” times As we can see, the speed-up is about “2” times

faster for the 64x64 HYDICE image than the one faster for the 64x64 HYDICE image than the one

without MIL. without MIL.

With MILWith MIL Without MILWithout MIL

Computation timeComputation time 26.609026.6090 45.656045.6560

18

The Matrix Inversion Lemma has been successfully The Matrix Inversion Lemma has been successfully applied to reduce the matrix inversion performed by applied to reduce the matrix inversion performed by Adaptive Causal Anomaly Detection (ACAD) into Adaptive Causal Anomaly Detection (ACAD) into matrix multiplications.matrix multiplications.

Since the Causal RX Filter (CRXF) and Real-time CEM Since the Causal RX Filter (CRXF) and Real-time CEM (Constrained Energy Minimization) previously proposed (Constrained Energy Minimization) previously proposed in Wang [2003] also involve inverse matrix in Wang [2003] also involve inverse matrix computation, the same MIL-based approach can be also computation, the same MIL-based approach can be also applied to reduce the computational load.applied to reduce the computational load.

ConclusionsConclusions

19

Future Work

An effective Dimensionality Reduction (DR) or An effective Dimensionality Reduction (DR) or Band Selection (BS) may need to reduce the Band Selection (BS) may need to reduce the number of bands to an acceptable range so that we number of bands to an acceptable range so that we can further reduce the computation cost in both can further reduce the computation cost in both applications.applications.

Heterogeneous platform may be also considered to Heterogeneous platform may be also considered to reduce the design time and possibly achieve better reduce the design time and possibly achieve better performance.performance.

20

Projects Conducted in RSSIPL

Joint Service Agent Water Monitor Joint Service Agent Water Monitor MissionMission

Develop GUI image analysis software for detecting Develop GUI image analysis software for detecting Biological Threat Agent on Handheld Assays Biological Threat Agent on Handheld Assays

Ported developed algorithms onto embedded system, Ported developed algorithms onto embedded system, Stargate Gateway (SPB400, Linux single board Stargate Gateway (SPB400, Linux single board computer) with external hand held scanner device. computer) with external hand held scanner device.

SponsorSponsor US Army Edgewood Chemical and Biological Center US Army Edgewood Chemical and Biological Center

(ECBC) (ECBC) ANP Technologies, Inc.ANP Technologies, Inc.

21

Projects Conducted in RSSIPL (Cont’d)

22

Projects Conducted in RSSIPL (Cont’d)

Multi-band Multi-threat warning sensor Multi-band Multi-threat warning sensor MissionMission

Developed detection algorithms for missile and grenade Developed detection algorithms for missile and grenade images captured from real-time Multispectral imaging images captured from real-time Multispectral imaging system. system.

Developed MATLAB based GUI for image analysis.Developed MATLAB based GUI for image analysis. SponsorSponsor

Surface Optics Corporation (SOC)Surface Optics Corporation (SOC)

23

Software for Detecting Agents

24

Embedded System

25

Projects Conducted in RSSIPL (Cont’d)

Multi-band Multi-threat warning sensor Multi-band Multi-threat warning sensor MissionMission

Developed detection algorithms for missile and grenade Developed detection algorithms for missile and grenade images captured from real-time Multispectral imaging images captured from real-time Multispectral imaging system. system.

Developed MATLAB based GUI for image analysis.Developed MATLAB based GUI for image analysis. SponsorSponsor

Surface Optics Corporation (SOC)Surface Optics Corporation (SOC)

26

Publication

Book ChapterBook Chapter

J. Wang, M. Hsueh and C.-I Chang, “FPGA Design for Second-order J. Wang, M. Hsueh and C.-I Chang, “FPGA Design for Second-order Statistics Based Target Detection Algorithm for Hyperspectral Imagery Statistics Based Target Detection Algorithm for Hyperspectral Imagery Applications,” High Performance Computing in Remote Sensing, Applications,” High Performance Computing in Remote Sensing, Chapman & Hall/CR, Oct 2007.Chapman & Hall/CR, Oct 2007.

J. Wang, M. Hsueh and C.-I Chang, “FPGA Implementation for Real-J. Wang, M. Hsueh and C.-I Chang, “FPGA Implementation for Real-time Orthogonal Subspace Projection for Hyperspectral Imagery time Orthogonal Subspace Projection for Hyperspectral Imagery Applications,” High Performance Computing in Remote Sensing, Applications,” High Performance Computing in Remote Sensing,

Chapman & Hall/CR, Oct 2007.Chapman & Hall/CR, Oct 2007.

27

Publication (cont’d)

JournalJournal

C.-I Chang and M. Hsueh, “Characterization of Anomaly Detection in C.-I Chang and M. Hsueh, “Characterization of Anomaly Detection in Hyperspectral Imagery,” Hyperspectral Imagery,” Sensor ReviewSensor Review, Volume 26, Issue 2, pp. 137-, Volume 26, Issue 2, pp. 137-146, 2006.146, 2006.

M. Hsueh and C.-I Chang, “Field Programmable Gate Arrays for Pixel M. Hsueh and C.-I Chang, “Field Programmable Gate Arrays for Pixel Purity Index Using Blocks of Skewers for Endmember Extraction in Purity Index Using Blocks of Skewers for Endmember Extraction in Hyperspectral Imagery,” International Journal of High Performance Hyperspectral Imagery,” International Journal of High Performance Computing Applications, Dec 2007. (to appear)Computing Applications, Dec 2007. (to appear)

C.-I Chang, M. Hsueh, F. Chaudhry, W. Liu, C.-C. Wu, G. Solyar, “A C.-I Chang, M. Hsueh, F. Chaudhry, W. Liu, C.-C. Wu, G. Solyar, “A pyramid-based block of skewers for pixel purity index for endmember pyramid-based block of skewers for pixel purity index for endmember Extraction in hyperspectral imagery,” International Journal of High Speed Extraction in hyperspectral imagery,” International Journal of High Speed Electronics and Systems. (to appear)Electronics and Systems. (to appear)

M. Hsueh and C.-I Chang, “Adaptive Causal Anomaly Detection on M. Hsueh and C.-I Chang, “Adaptive Causal Anomaly Detection on Reconfigurable Computing,” Reconfigurable Computing,” IEEE Transaction on Industrial ElectronicsIEEE Transaction on Industrial Electronics. . (To be submitted) (To be submitted)

28

Publication (cont’d)

ConferenceConference

M. Hsueh and C.-I Chang, “FPGA implementation of Adaptive Causal M. Hsueh and C.-I Chang, “FPGA implementation of Adaptive Causal Anomaly Detection,” Anomaly Detection,” 2006 CIE Annual Convention2006 CIE Annual Convention, Newark, NJ, Sep 16, , Newark, NJ, Sep 16, 2006.2006.

C.-I Chang, M. Hsueh, F. Chaudhry, W. Liu, C. C. Wu, A. Plaza and G. C.-I Chang, M. Hsueh, F. Chaudhry, W. Liu, C. C. Wu, A. Plaza and G. Solyar, “A Pyramid-based Block of Skewers for Pixel Purity Index for Solyar, “A Pyramid-based Block of Skewers for Pixel Purity Index for Endmember Extraction in Hyperspectral Imagery,” Endmember Extraction in Hyperspectral Imagery,” 2006 International 2006 International Symposium on Spectral Sensing ResearchSymposium on Spectral Sensing Research, Bar Harbor, ME, May 29 to , Bar Harbor, ME, May 29 to Jun 2, 2006.Jun 2, 2006.

D. Valencia, A. Plaza, M. A. Vega-Rodriguez, R. M. Perez and M. D. Valencia, A. Plaza, M. A. Vega-Rodriguez, R. M. Perez and M. Hsueh, “FPGA Design and Implementation of a Fast Pixel Purity Index Hsueh, “FPGA Design and Implementation of a Fast Pixel Purity Index Algorithm for Endmember Extraction in Hyperspectral Imagery,” Algorithm for Endmember Extraction in Hyperspectral Imagery,” SPIE SPIE Optics East, Optics East, Boston, MA, Oct 23-26 2005.Boston, MA, Oct 23-26 2005.

L. Wu, J. Wang, B. Ramakrishna, M. Hsueh, J. Liu, Q. Wu, C. Wu, M. L. Wu, J. Wang, B. Ramakrishna, M. Hsueh, J. Liu, Q. Wu, C. Wu, M. Cao, C. Chang, J. L. Jensen, J. O. Jensen, H. Knapp, R. Daniel, R. Yin, Cao, C. Chang, J. L. Jensen, J. O. Jensen, H. Knapp, R. Daniel, R. Yin, “An embedded system developed for hand held assay used in water “An embedded system developed for hand held assay used in water monitoring,” monitoring,” SPIE Optics East, SPIE Optics East, Boston, MA, Oct 23-26, 2005.Boston, MA, Oct 23-26, 2005.

29

Publication (cont’d)

ConferenceConference

M. Hsueh and C.-I Chang, “Adaptive Causal Anomaly Detection for M. Hsueh and C.-I Chang, “Adaptive Causal Anomaly Detection for Hyperspectral Imagery”, Hyperspectral Imagery”, IEEE International Geoscience and Remote IEEE International Geoscience and Remote Sensing Symposium,Sensing Symposium, Alaska, Sep 19-26, 2004. Alaska, Sep 19-26, 2004.

M. Hseuh, A. Plaza, J. Wang, S. Wang, W. Liu, C.-I Chang, J. L. Jensen M. Hseuh, A. Plaza, J. Wang, S. Wang, W. Liu, C.-I Chang, J. L. Jensen and J. O. Jensen, “Morphological algorithms for processing tickets by and J. O. Jensen, “Morphological algorithms for processing tickets by hand held assay,” hand held assay,” OpticsEast, Chemical and Biological Standoff OpticsEast, Chemical and Biological Standoff Detection IIDetection II (OE120), Vol. 5584, Philadelphia, PA, Oct 25-28, 2004. (OE120), Vol. 5584, Philadelphia, PA, Oct 25-28, 2004.

C.-I Chang, H. Ren, M. Hsueh, F. D’Amico and J.O. Jensen, “A Revisit C.-I Chang, H. Ren, M. Hsueh, F. D’Amico and J.O. Jensen, “A Revisit to Target-Constrained Interference-Minimized Filter,” to Target-Constrained Interference-Minimized Filter,” 48th Annual 48th Annual Meeting, SPIE International Symposium on Optical science and Meeting, SPIE International Symposium on Optical science and Technology, Imaging Spectrometry IX ( AM110), Technology, Imaging Spectrometry IX ( AM110), San Diego, CA, Aug 3-San Diego, CA, Aug 3-8, 2003.8, 2003.

S. T. Sheu, M. Hsueh, “An Intelligent Cell Checking Policy for S. T. Sheu, M. Hsueh, “An Intelligent Cell Checking Policy for Promoting Data Transfer Performance in Wireless ATM Networks,” Promoting Data Transfer Performance in Wireless ATM Networks,” IEEE ATM Workshop '99, Kochi City, Kochi, Japan, May 24-27, 1999. IEEE ATM Workshop '99, Kochi City, Kochi, Japan, May 24-27, 1999.

30

Thank you!!