פברואר 1968, בסיום מעלה האיסיים

1968, 1968, 1969, ( ") 1969, , -

Nathan S. Netanyahu

Dept. of Computer Science, Bar-Ilan Universityand Center for Automation Research, University of Maryland

Collaborators:

Jacqueline Le MoigneNASA / Goddard Space Flight CenterDavid M. MountUniversity of Maryland Arlene A. Cole-Rhodes, Kisha L. JohnsonMorgan State University, MarylandRoger D. EastmanLoyola College of MarylandArdeshir GoshtasbyWright State University, OhioJeffrey G. Masek, Jeffrey Morisette NASA / Goddard Space Flight CenterAntonio PlazaUniversity of Extremadura, SpainHarold StoneNEC Research Institute (Ret.)Ilya ZavorinCACI International, MarylandShirley Barda, Boris Sherman Applied Materials, Inc., IsraelYair KapachBar-Ilan UniversityImage Registration of Remotely Sensed Data

Akavia Memorial Lecture, Haifa Univ., 11.01.2007*

What is Image Registration / Alignment / Matching?The above image is rotated and shifted with respect to the left image.


MotivationA crucial, fundamental step in image analysis tasks, where final information is obtained by the combination / integration of multiple data sources.


Motivation / ApplicationsComputer Vision (target localization, quality control, stereo matching)

Medical Imaging (combining CT and MRI data, tumor growth monitoring, treatment verification)

Remote Sensing (classification, environmental monitoring, change detection, image mosaicing, weather forecasting, integration into GIS)


Application ExamplesGlobal Wafer Alignment


Wafer Alignment (contd)Courtesy: Shirley Barda and Boris Sherman, Applied Materials, Inc.


Application Examples (contd)Integration of medical imagesRegistration of MR and PET images of the same person (courtesy: A. Goshtasby)


Application Examples (contd)Change Detection

19752000Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website


Change Detection (contd)19902005Satellite images of Amona hilltop, Peace Now website


Change Detection (contd)IKONOS images of Irans Bushehr nuclear plant, GlobalSecurity.org


What is the Big Deal?How do humans solve this?By matching control points, e.g., corners, high-curvature points.Zitova and Flusser, IVC 2003


Automatic Image RegistrationBooks:Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill (Eds.), CRC 2001Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 20042-D and 3-D Image Registration, A. Goshtasby, Wiley 2005Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman (Eds.), Cambridge University Press, in preparation.

Surveys:A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003

Sample Papers:Image Sequence Enhancement Using Sub-pixel Displacements, CVPR, D. Keren, S. Peleg, and R. Brada, 1988Improving Resolution by Image Registration, CVGIP, M. Irani and S. Peleg, 1991Computing Correspondence Based on Regions and Invariants without Feature Extraction and Segmentation, CVPR, C. Lee, D. Cooper, and D. Keren, 1993Robust Multi-Image Sensor Image Alignment, ICCV, M. Irani and P. Anandan, 1998Fast Block Motion Estimation Using Gray Code Kernels, Israel CV Workshop, Y. Moshe and H. Hel-Or, 2006Image Matching Using Photometric Information, Israel CV Workshop, M. Kolomenkin and I. Shimshoni, 2006


Automatic Image Registration Components0. PreprocessingImage enhancement, cloud detection, region of interest masking

1. Feature extraction (control points)Corners, edges, wavelet coefficients, segments, regions, contours

2. Feature matchingSpatial transformation (a priori knowledge)Similarity metric (correlation, mutual information, Hausdorff distance)Search strategy (global vs. local, multiresolution, optimization)

3. ResamplingI1I2Tp


Example of Image Registration StepsFeature extractionFeature matchingResamplingRegistered images after transformationZitova and Flusser, IVC 2003


Automatic Image Registrationfor Remote SensingSensor webs, constellation, and exploration

Selected NASA Earth science missions

Domain-dependent characteristics


Automatic Multiple Source IntegrationSatellite/Orbiter,and In-Situ DataPlanning andSchedulingSensor Webs, Constellation, and ExplorationIntelligent Navigationand Decision Making


Selected NASA Earth Science Missions

Table1

0.10.40.50.60.71.01.32.03.04.05.06.07.08.09.010.011.012.013.014.015.0

InstrumentNumber of

(Spat. Resol.)Channels

AVHRR (D)5 Channels

(1.1 km)

TRMM/VIRS5 Channels

(2 km)

Landsat4-MSS4 Channels

(80 m)

Landsat5&7-TM&ETM+

(30 m)7 Channels

Landsat7-Panchromatic

(15m)

IRS-14 Channels

LISS-I (73m) - LISS-2 (36.5m)

JERS-18 Channels

(Ch1-4:18m; Ch5-8:24m)

SPOT-HRV Panchromatic

(10m)1 Channel

Spot-HRV Multispectral

(20 m)3 Channels

MODIS36 Channels

(Ch1-2:250 m;3-7:500m;8-36:1km)

EO/1

ALI-MultiSpectr.9 Channels (30m)

ALI-Panchrom.1 Channel (10m)

Hyperion220 Channels

(30m)

LAC256 Channels

(250m)

IKONOS-Panchromatic

(1m)1 Channel

IKONOS-MS4 Channels (4m)

ASTER14 Channels

(Ch1-3:15m;4-9:30m;10-14:90m)

CZCS6 Channels

(1 km)

SeaWiFS (D)8 Channels

(1.1 km)

TOVS-HIRS2 (D)20 Channels

(15 km)

GOES5 Channels

(1 km:1, 4km:2,4&5, 8km:3)

METEOSAT3 Channels

(V:2.5km,WV&IR:5km)

.

`

&C&"Geneva,Bold"&12Table 1Summary of the Main Current Earth Science Satellite Data Operating from UV to Thermal IR("D": Direct Broadcast)

Near-IR

Mid-IR

Thermal-IR

1

2

4

5

2

3

4

5

1

2

3

4

5

7

6

1

1

2

3

2

1

3

4

5

1

3

1

2

3

4

5

6

7

8

20

19

18

17 to 13

12

11

10

9

8

7 to 1

UltraViolet

1

2

3

4

5

Visible

IR

WaterVapor

1

2

3

4

2) 0.52-0.59

3) 0.62-0.68

4) 0.77-0.86

1) 0.45-0.52

1

2

3&4

2) 0.63-0.69

3) and4) 0.76-0.86

1) 0.52-0.60

5) 1.60-1.71

6) 2.01-2.12

7) 2.13-2.25

8) 2.27-2.40

Visible

6

5

7

8

1

2

3

4

1

2

3&4

6

5

7

8

1,13,14

2,16-19

3,8-10

11,4,12

5

6

7

20-25

15

26

27

28

29

30

31

32

33-36

1

2

3

4

1

1 to 220

5'

1'

1

2

3

4

5

7

1

1 to 256

1

2

3

4

5-9

10,11

12

13,14

1

1

2

3

4


MODIS Satellite SystemFrom the NASA MODIS website


MODIS Satellite Specifications


Landsat 7 Satellite SystemNew Orleans, before and after Katrina 2005 (from the USGS Landsat website)


Landsat 7 Satellite Specifications


IKONOS Satellite System


IKONOS Satellite Specifications


Domain-Dependent CharacteristicsVery large images (~ 6200 x 5700 of typical Landsat 7 scene)

Practically flat, 2D images

Rigid/similar transformations

A priori knowledge (e.g., small rotation and scale)


Challenges in Processing of Remotely Sensed DataMultisource dataMulti-temporal dataVarious spatial resolutionsVarious spectral resolutions

Subpixel accuracy1 pixel misregistration 50% error in NDVI classification

Computational efficiencyFast procedures for very large data sets

Accuracy assessmentSynthetic dataGround Truth" (manual registration?)Consistency ("circular" registrations) studies


Fusion of Multi-temporal ImagesImprovement of NDVI classification accuracy due to fusion of multi-temporal SAR and Landsat TM over farmland in The Netherlands (source: The Remote Sensing Tutorial by N.M. Short, Sr.)


Integration of Multiresolution SensorsRegistration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site(source: LeMoigne et al., IGARSS 2003)


Feature ExtractionTop 10% of wavelet coefficients (due to Simoncelli) of Landsat image over Washington, D.C. (source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)Gray levelsBPF wavelet coefficientsBinary feature map


Feature Extraction (contd)Image features (extracted from two overlapping scenes over D.C.) to be matched


Feature Matching / TransformationsGiven a reference image, I1(x, y), and a sensed image I2(x, y), find the mapping (Tp, g) which best transforms I1 into I2, i.e.,

where Tp denotes spatial mapping and g denotes radiometric mapping.

Spatial transformations:Translation, rigid, affine, projective, perspective, polynomial

Radiometric transformations (resampling):Nearest neighbor, bilinear, cubic convolution, spline


Transformations (contd)Objective: Find parameters of a transformation Tp (consisting of a translation, a rotation, and an isometric scale) that maximize similarity measure.


Similarity Measures (contd)L2 norm:Minimize sum of squared errors over overlapping subimage

Normalized cross correlation (NCC):Maximize normalized correlation between the images


Similarity Measures (contd)Mutual information (MI):Maximize the degree of dependence between the images

or using histograms, maximize


Similarity Measures (contd), An ExampleMI vs. L2-norm and NCC applied to Landsat 5 images(source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)


Similarity Measures (contd), an MI ExampleSource: Cole-Rhodes et al., IEEE-TIP, 2003


Similarity Measures (contd)(Partial) Hausdorff distance (PHD):

where


Similarity Measures (contd), PHD ExamplePHD-based matching of Landsat images over D.C. (source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004)


Feature Matching / Search StrategyExhaustive search

Fast Fourier transform (FFT)

Optimization (e.g., gradient descent; Thvenaz, Ruttimann, and Unser (TRU), 1998; Spall, 1992)

Robust feature matching (e.g., efficient subdivision and pruning of transformation space)


Computational EfficiencyExtraction of corresponding regions of interest (ROI)

Hierarchical, pyramid-like approach

Efficient search strategy


Computational Efficiency (contd), ROI ExtractionUTM of 4 scene corners known from systematic correctionExtract reference chips and corresponding input windows using mathematical morphology

Register each (chip-window) pair and record pairs of registered chip corners (refinement step)

Compute global registration from multiple local ones

Compute correct UTM of 4 scene corners of input sceneReference SceneAdvantages: Eliminates need for chip database Cloud detection can easily be included in process Process any size images Initial registration closer to optimal registration => reduces computation time and increases accuracy.Source: Plaza, LeMoigne, and Netanyahu, MultiTemp, 2005


Computational Efficiency (contd),An Example of a Pyramid-Like Approach012332 x 3264 x 64128 x 128256 x 256


IR Example Using Partial Hausdorff Distance64 x 64128 x 128256 x 256


IR Example Using PHD (contd)Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004


IR Components (Revisited)Gray LevelsEdgesFeatures

SimilarityMeasure

StrategyWavelets or Wavelet-Like


IR Components (Revisited)Features

SimilarityMeasure

StrategyFFTRobustFeatureMatchingGradient DescentSpallsOptimizationThevenaz,Ruttimann,Unser Optimization


Goals of a Modular Image Registration FrameworkTesting framework to:Assess various combinations of components Assess a new registration component

Web-based registration tool would allow user to schedule combination of components, as a function of: Application Available computational resourcesRequired registration accuracy

Prototype of web-based registration toolbox:Several modules based on wavelet decompositionJava implementation; JNI-wrapped functions


Web-Based Image Registration ToolboxTARA (Toolbox for Automated Registration & Analysis)


Web-Based Image Registration ToolboxTARA (Toolbox for Automated Registration & Analysis)


Current and Future WorkConclude component evaluation Sensitivity to noise, radiometric transformations, initial conditions, and computational requirementsIntegration of digital elevation map (DEM) information

Build operational registration framework/toolboxWeb-basedApplications: EOS validation core sitesOther EOS satellites (e.g., Hyperion vs. ALI registration) and beyondImage fusion, change detection

View of the moon, as seen from Apollo 11 (1969(Icarus fell because he flew too close to the sun. Columbia - and the whole American manned space program today - fell because it flies too close to the Earth Charles Krauthammer (Feb. 2003) Back to the Moon

פברואר 1968, בסיום מעלה האיסיים

Documents

image registration methods

d image registration

gis akavia memorial

image mosaicing

left image

medical image analysis

image analysis tasks

johnsonmorgan state