פברואר 1968, בסיום מעלה האיסיים פברואר 1968, בסיום מסע...

, בסיום מעלה האיסיים1968פברואר , בסיום מסע למצדה1968פברואר

, עובדת )מסע גדנ"ע(1969פברואר , אודיטוריום פיסיקה, 1969יוני בית-בירם

Image Registration of Remotely Sensed Data

Nathan S. Netanyahu

Dept. of Computer Science, Bar-Ilan Universityand Center for Automation Research, University of Maryland

Collaborators:

Jacqueline Le Moigne NASA / Goddard Space Flight CenterDavid M. Mount University of Maryland Arlene A. Cole-Rhodes, Kisha L. Johnson Morgan State University, MarylandRoger D. Eastman Loyola College of MarylandArdeshir Goshtasby Wright State University, OhioJeffrey G. Masek, Jeffrey Morisette NASA / Goddard Space Flight CenterAntonio Plaza University of Extremadura, SpainHarold Stone NEC Research Institute )Ret.(Ilya Zavorin CACI International, MarylandShirley Barda, Boris Sherman Applied Materials, Inc., IsraelYair Kapach Bar-Ilan University

Akavia Memorial Lecture, Haifa Univ., 11.01.20073

What is Image Registration / Alignment / Matching?

The above image is rotated and shifted with respect to the left image.


Motivation

• A crucial, fundamental step in image analysis tasks, where final information is obtained by the combination / integration of multiple data sources.


Motivation / Applications

• Computer Vision )target localization, quality control, stereo matching(

• Medical Imaging )combining CT and MRI data, tumor growth monitoring, treatment verification(

• Remote Sensing )classification, environmental monitoring, change detection, image mosaicing, weather forecasting, integration into GIS(


Application Examples

• Global Wafer Alignment


Wafer Alignment (cont’d)

Courtesy: Shirley Barda and Boris Sherman, Applied Materials, Inc.


Application Examples (cont’d)

• Integration of medical images

Registration of MR and PET images of the same person (courtesy: A. Goshtasby)


Application Examples (cont’d)

• Change Detection

1975 2000

Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website


Change Detection (cont’d)

1990 2005

Satellite images of Amona hilltop, Peace Now website


Change Detection (cont’d)

IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org


What is the “Big Deal”?

How do humans solve this?By matching control points, e.g., corners, high-curvature points.

Zitova and Flusser, IVC 2003


Automatic Image Registration

• Books:– Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill )Eds.(, CRC 2001– Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 2004– 2-D and 3-D Image Registration, A. Goshtasby, Wiley 2005– Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman )Eds.(, Cambridge

University Press, in preparation.

• Surveys:– A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992– A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998– Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003

• Sample Papers:– Image Sequence Enhancement Using Sub-pixel Displacements, CVPR, D. Keren, S. Peleg, and R. Brada, 1988– Improving Resolution by Image Registration, CVGIP, M. Irani and S. Peleg, 1991– Computing Correspondence Based on Regions and Invariants without Feature Extraction and Segmentation,

CVPR, C. Lee, D. Cooper, and D. Keren, 1993– Robust Multi-Image Sensor Image Alignment, ICCV, M. Irani and P. Anandan, 1998– Fast Block Motion Estimation Using Gray Code Kernels, Israel CV Workshop, Y. Moshe and H. Hel-Or, 2006– Image Matching Using Photometric Information, Israel CV Workshop, M. Kolomenkin and I. Shimshoni, 2006


Automatic Image Registration Components

0. Preprocessing

– Image enhancement, cloud detection, region of interest masking

1. Feature extraction )control points(

– Corners, edges, wavelet coefficients, segments, regions, contours

2. Feature matching

– Spatial transformation )a priori knowledge(

– Similarity metric )correlation, mutual information, Hausdorff distance(

– Search strategy )global vs. local, multiresolution, optimization(

3. Resampling

I1

I2Tp


Example of Image Registration Steps

Feature extraction

Feature matching

Resampling Registered images after transformation

Zitova and Flusser, IVC 2003


Automatic Image Registrationfor Remote Sensing

• Sensor webs, constellation, and exploration

• Selected NASA Earth science missions

• Domain-dependent characteristics


Automatic Multiple Source

IntegrationSatellite/Orbiter,and In-Situ Data

Planning andScheduling

Sensor Webs, Constellation, and Exploration

Intelligent Navigationand Decision Making


0.1 0.4 0.5 0.6 1.0 1.3 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 15.0

Instrument Number of(Spat. Resol.) Channels

AVHRR (D) 5 Channels(1.1 km)

TRMM/VIRS 5 Channels(2 km)

Landsat4-MSS 4 Channels(80 m)

Landsat5&7-TM&ETM+(30 m) 7 Channels

Landsat7-Panchromatic(15m)

IRS-1 4 ChannelsLISS-I (73m) - LISS-2 (36.5m)

JERS-1 8 Channels(Ch1-4:18m; Ch5-8:24m)

SPOT-HRV Panchromatic(10m) 1 ChannelSpot-HRV Multispectral(20 m) 3 ChannelsMODIS 36 Channels(Ch1-2:250 m;3-7:500m;8-36:1km)

EO/1

ALI-MultiSpectr. 9 Channels (30m)

ALI-Panchrom. 1 Channel (10m)

Hyperion 220 Channels

(30m)

LAC 256 Channels

(250m)

IKONOS-Panchromatic

(1m) 1 Channel

IKONOS-MS 4 Channels (4m)

ASTER 14 Channels

(Ch1-3:15m;4-9:30m;10-14:90m)

CZCS 6 Channels(1 km)

SeaWiFS (D) 8 Channels(1.1 km)

TOVS-HIRS2 (D) 20 Channels

(15 km)

GOES 5 Channels(1 km:1, 4km:2,4&5, 8km:3)METEOSAT 3 Channels(V:2.5km,WV&IR:5km)

.

13.0 14.00.7 10.0 11.0 12.0

Near-IR Mid-IR Thermal-IR

1 2 4 5

2 3 4 5

1 2 3 4 5 7 6

1

1 2 3

21 3 4 5

1

3

1 2 3 4 5 6 7 8

20 19 18

17 to 13

12 11 10 9 8 7 to 1

UltraViolet

1 2 3 4 5

Visible IRWaterVapor

Visible

1 2 3 4

1 23

&465 7 8

1,13,14

2,16-19

3,8-10

11,4,12

5 6 7 20-2515 26 27 28 29 30 31 32 33-36

1 2 3 4

1

1 to 220

5'1' 1 2 3 4 5 7

1

1 to 256

1 2 3 4 5-9 10,11 12 13,14

1

1 2 3 4

Selected NASA Earth Science Missions


MODIS Satellite System

From the NASA MODIS website


MODIS Satellite Specifications

Orbit: 705 km, 10:30 a.m. descending node )Terra( or 1:30 p.m. ascending node )Aqua(, sun-synchronous, near-polar, circular

Scan Rate: 20.3 rpm, cross track

Swath Dimensions: 2330 km )cross track( by 10 km )along track at nadir(

Telescope: 17.78 cm diam. off-axis, afocal )collimated(, with intermediate field stop

Size: 1.0 x 1.6 x 1.0 m

Weight: 228.7 kg

Power: 162.5 W )single orbit average(

Data Rate: 10.6 Mbps )peak daytime(; 6.1 Mbps )orbital average(

Quantization: 12 bits

Spatial Resolution: 250 m )bands 1-2(500 m )bands 3-7(1000 m )bands 8-36(

Design Life: 6 years


Landsat 7 Satellite System

New Orleans, before and after Katrina 2005 (from the USGS Landsat website)


Landsat 7 Satellite Specifications

Launch

Date April 15, 1999

Vehicle Delta II

Site Vandenberg AFB

Orbit Characteristics

Reference system WRS-2

Type Sun-synchronous, near-polar

Altitude 705 km )438 mi(

Inclination 98.2°

Repeat cycle 16 days

Swath width 185 km )115 mi(

Equatorial crossing time 10:00 AM +/- 15 minutes


IKONOS Satellite System


IKONOS Satellite Specifications

Launch Date24 September 1999

Vandenberg Air Force Base, California, USA

Operational Life Over 7 years

Orbit 98.1 degree, sun synchronous

Speed on Orbit 7.5 kilometers per second

Speed Over the Ground 6.8 kilometers per second

Number of Revolutions Around the Earth 14.7 every 24 hours

Orbit Time Around the Earth 98 minutes

Altitude 681 kilometers

Resolution

Nadir:0.82 meters panchromatic3.2 meters multispectral26° Off-Nadir1.0 meter panchromatic4.0 meters multispectral

Image Swath11.3 kilometers at nadir

13.8 kilometers at 26° off-nadir

Equator Crossing Time Nominally 10:30 a.m. solar time

Revisit Time Approximately 3 days at 40° latitude

Dynamic Range 11-bits per pixel

Image Bands Panchromatic, blue, green, red, near IR


Domain-Dependent Characteristics

• Very large images )~ 6200 x 5700 of typical Landsat 7 scene(

• Practically “flat”, 2D images

• Rigid/similar transformations

• A priori knowledge )e.g., small rotation and scale(


Challenges in Processing of Remotely Sensed Data

• Multisource data• Multi-temporal data• Various spatial resolutions• Various spectral resolutions

• Subpixel accuracy• 1 pixel misregistration ≥ 50% error in NDVI classification

• Computational efficiency• Fast procedures for very large data sets

• Accuracy assessment• Synthetic data• “Ground Truth" )manual registration?(• Consistency )"circular" registrations( studies


Fusion of Multi-temporal Images

Improvement of NDVI classification accuracy due to fusion of multi-temporal SAR and Landsat TM over farmland in The Netherlands (source: The Remote Sensing Tutorial by N.M. Short, Sr.)


Integration of Multiresolution Sensors

Registration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site

(source: LeMoigne et al., IGARSS 2003)


Feature Extraction

Top 10% of wavelet coefficients )due to Simoncelli( of Landsat image over Washington, D.C. (source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)

Gray levels BPF wavelet coefficients Binary feature map


Feature Extraction (cont’d)

Image features )extracted from two overlapping scenes over D.C.( to be matched


Feature Matching / Transformations

• Given a reference image, I1(x, y), and a sensed image I2(x, y), find the

mapping )Tp, g( which “best” transforms I1 into I2, i.e.,

where Tp denotes spatial mapping and g denotes radiometric mapping.

• Spatial transformations:Translation, rigid, affine, projective, perspective, polynomial

• Radiometric transformations )resampling(:Nearest neighbor, bilinear, cubic convolution, spline

2 1( , ) ( ( ( , ), ( , ))),p pI x y g I T x y T x y


Transformations (cont’d)

cos sin

sin cos

0 0 1

x

p y

s s t

T s s t

' cos sin

' sin cosx

y

x s x s y t

y s x s y t

Objective: Find parameters of a transformation Tp (consisting of a translation,

a rotation, and an isometric scale) that maximize similarity measure.


Similarity Measures (cont’d)

• L2 norm:

Minimize sum of squared errors over overlapping subimage

• Normalized cross correlation (NCC):Maximize normalized correlation between the images

1 1 2 2

1 22 2

1 1 2 2

( , ) ( , )

( , )

( , ) ( , )

x y

x y x y

I x y I I x y I

NCC I I

I x y I I x y I

2

2 1pI T I



• Mutual information (MI):Maximize the degree of dependence between the images

or using histograms, maximize

1 2

1 2

1 2 1 2

, 1 21 2 , 1 2

1 2

,, , log ,I I

I Ig g I I

p g gMI I I p g g

p g p g

1 2

1 2

1 2 1 2

, 1 21 2 , 1 2

1 2

,1, , log I I

I Ig g I I

Mh g gMI I I h g g

M h g h g


Similarity Measures (cont’d), An Example

MI vs. L2-norm and NCC applied to Landsat 5 images

(source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)


Similarity Measures (cont’d), an MI Example

Source: Cole-Rhodes et al., IEEE-TIP, 2003



• (Partial) Hausdorff distance (PHD):

where

1 1 2 21 2 1 2, min dist , ,th

K p I p IH I I K p p

11 K I

1K

2K

1K I


Similarity Measures (cont’d), PHD Example

PHD-based matching of Landsat images over D.C. (source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004)


Feature Matching / Search Strategy

• Exhaustive search

• Fast Fourier transform )FFT(

• Optimization )e.g., gradient descent; Thévenaz, Ruttimann, and Unser )TRU(, 1998; Spall, 1992(

• Robust feature matching )e.g., efficient subdivision and pruning of transformation space(


Computational Efficiency

• Extraction of corresponding regions of interest )ROI(

• Hierarchical, pyramid-like approach

• Efficient search strategy


Computational Efficiency (cont’d), ROI Extraction

UTM of 4 scene corners known from systematic correction

Input Scene

1. Extract reference chips and corresponding input windows using mathematical morphology

2. Register each )chip-window( pair and record pairs of registered chip corners )refinement step(

3. Compute global registration from multiple local ones

4. Compute correct UTM of 4 scene corners of input scene

Reference Scene

Advantages:• Eliminates need for chip database• Cloud detection can easily be included in process• Process any size images• Initial registration closer to optimal registration => reduces computation time and increases accuracy.

Source: Plaza, LeMoigne, and Netanyahu, MultiTemp, 2005


Computational Efficiency (cont’d),An Example of a Pyramid-Like Approach

0

1

2

3

32 x 32

64 x 64

128 x 128

256 x 256

2

2x x

y y

s s

t t

t t


IR Example Using Partial Hausdorff Distance

64 x 64

128 x 128

256 x 256


IR Example Using PHD (cont’d)

Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004


IR Components (Revisited)

Gray Levels EdgesFeatures

SimilarityMeasure

Strategy

Correlation L2-Norm Mutual Information Hausdorff Distance

Fast Fourier TransformGradient Descent

Spall’sOptimization

RobustFeature

Matching

Thevenaz,Ruttimann,

Unser Optimization

Wavelets or Wavelet-Like


IR Components (Revisited)

Features

SimilarityMeasure

Strategy

Correlation L2-Norm MI Hausdorff Distance

FFT

RobustFeature

MatchingGradient Descent


Thevenaz,Ruttimann,

Unser Optimization

Gray Levels Spline or SimoncelliLPF

Simoncelli BPF

L2-Norm MI

Gradient Descent


Thevenaz,Ruttimann,

Unser Optimization


Goals of a Modular Image Registration Framework

• Testing framework to:– Assess various combinations of components – Assess a new registration component

• Web-based registration tool would allow user to “schedule” combination of components, as a function of:– Application– Available computational resources– Required registration accuracy

• Prototype of web-based registration toolbox:– Several modules based on wavelet decomposition– Java implementation; JNI-wrapped functions


Web-Based Image Registration ToolboxTARA (“Toolbox for Automated Registration & Analysis”)


Current and Future Work

• Conclude component evaluation – Sensitivity to noise, radiometric transformations, initial

conditions, and computational requirements– Integration of digital elevation map )DEM( information

• Build operational registration framework/toolbox– Web-based– Applications:

• EOS validation core sites• Other EOS satellites )e.g., Hyperion vs. ALI registration( and beyond• Image fusion, change detection

View of the moon, as seen from Apollo 11 )1969)

“Icarus fell because he flew too close to the sun. Columbia - and the whole American manned space program today - fell because it flies too close to the Earth” – Charles Krauthammer (Feb. 2003)

Back to the Moon

פברואר 1968, בסיום מעלה האיסיים פברואר 1968, בסיום מסע...

Documents

akavia memorial lecture

medical image registration

d image registration

goshtasby slide

gis slide

image mosaicing

left image

image analysis tasks