image matching, point transfer, dsm generationhome.agh.edu.pl/~krisfoto/lib/exe/...matching.pdf ·...

1

International Summer School “Digital Recording and 3D Modeling”, Aghios Nikolaos, Crete, Greece, 24-29 April 2006

Commission VI Special Interest Group “Technology Transfer Caravan”

Image matching, point transfer, DSM generation

Dr. Maria Pateraki

Department of Rural and Surveying Engineering

Aristotle University of Thessalonikitel:+30 2310 996407, email: [email protected], URL: www.topo.auth.gr



Introduction � Definition

match: �A suitable conjunction or pairing� (Oxford English Dictionary)

The matching problem is also referred to as the correspondence problem.

The data sets can be:+ images+ maps+ object models+ GIS data

� What is image matching?

Finding (measuring) automatically conjugate points in two or more images.

� Synonyms – alternative termsImage matching ~ automatic stereo matching ~ correlation ~ correspondence problem

2



Introduction � What does image matching needs?

+ digital stereo imagery (>=2) + grey value variations (good texture) + smooth object surface, small slopes+ non-moving objects

Matching strategy for DSM generation is similar to ARO / AAT, plus:

� dense surface description� hierarchical procedure with 3D points as result� integration of matching and surface interpolation

Potential problem areas

� urban areas� forests� water bodies



� Where is image matching being used (in which procedures)?

+ target measurements

+ interior orientation

+ relative and absolute orientation of steropairs

+ aerotriangulation

+ DSM generation

+ image Fusion (or Registration).

Introduction template

search

3



Matching – problematic areas



Matching tools� Area based matchingHigh accuracy potential but smoothing effectsSensitivity to discontinuity

Methods:

a. Cross correlation

b. Least squares matching

� Feature based matchingUse of abstract image representation derived from feature extraction algorithmAvoid smoothing effectsNo subpixel precision

Methods:

a. points

b. edges

c. blobs�Relational matchingBased on relationship between objects (distances, angles, collinearity)

4



Area based matching� Intensities are used as input to solve the correspondence problem

� Geometric transformation (object surface):a) Translation b) Translation + rotation c) Affine (6 parameters - locally planar surface)d) Projective (8 parameters � globally planar surface)e) Smooth (smooth surface without occlusions)f) Piecewise smooth (possibly with occlusions)

� Similarity measure (most frequently applied)a) Distance (Haussdorf, Euclidean)b) Sum of products (2nd moments, covariance)c) Sum of squared differences (least squares)d) Sum of absolute differencese) Normalized cross correlation

� Algorithmic solutiona) Sequential b) Heuristicc) Iteratived) Dynamic programming



Area based matching - Cross Correlation

� Measure the similarity of the template with the matching window by computing the correlation factor.

� Highest cross correlation coefficient in the search image

NCC = normalized cross correlation coefficient

Std. Dev. of f Std. Dev. of g Covariance of f and g

5




� No correlation between the two image patches NCC = 0

� Identical image patches NCC = 1

� Inverse correlation NCC = -1 (positive and negative)

� Geometric differences are modeled only by translation

� Radiometric differences exist only due to brightness and contrast

� No generalization for Multi-image matching




� Problematic cases

(a) Little contrast

low reliability of the match

(b) Multiple solutions

Repetitive patterns(c) Lower correlation than (a),

But possibly better match

a = flatness or the angle between the tangents to the parabola next to the maximum

6




� Different cases (Putockova 2004)



Template

Area based matching – Multi-pass Cross Correlation3 passes of matching using different parameters NCC as similarity measure

improve coarse approximations for position

Large patch

! aims at reliability of the solution

! less sensitive to noise, occlusions, multiple solutions

Small patch

! aims at precision of the solution

! better preserves height discontinuities

search range y

search range xsearch range x search range y

search range x search range y

Search

7



Area based matching - LSM

Minimize the grey level differences between template and search image.

Position and shape of the matching window are parameters to be determined in the adjustment process

Using Affine transformation (locally planar surface)

2 shifts, 2 rotations, 2 scales , 1 radiometric parameter

Precision: Can reach 1/25 pixel

It requires GOOD initial approximations

template search search




8







The matching is ideal if :

f(x,y) = g (x,y) Template = Search

Because of random effects (noise), a noise vector e(x,y) is added:

f(x,y) – e(x,y) = g (x,y) non-linear observation equation

f(x,y) = vector of observations (in the template image)g(x,y) = function model

Non linear system => linearization of the equations =>Approximations required => Iteration necessary

Taylor linearization:

9



Parameter vector:

Least squares estimation (Gauss-Markov model)

l - e = Ax


l = f(x,y) - g0(x,y)e = error vectorA = design matrix (coefficients of the parameters)

= solution vector for transformationv = residuals (grey-values differences Template Search)




10



Convergence (e.g. the residual vector is close to 0) vs. divergence

� Possible cases in LSM:- FAST AND CORRECT CONVERGENCE- SLOWLY BUT CORRECT CONVERGENCE- CONVERGENCE TO FALSE POSITION- NO CONVERGENCE (singular matrix)


Fast convergence Slow convergence, partial divergence



In general Geometrical Constraints are used to:+ strengthen matching + improve computational stability+ convergencyPairwise or simultaneous using all available images.Geometrical constraints: Collinearity equations

Area based matching – Multiphoto Geometrically Constrained LSM

11



Pairwise or simultaneous using all available images.Geometrical constraints: Collinearity equations


� 1 Template and n Search images (Patches)� n+1 collinearity conditions for object point P

Parameters to be estimated:�Shape and position of the patch in each search image�Object coordinates X, Y, Z



Least squares estimation (Gauss-Markov model)


solution vector

residual vector for intensity observationsresidual vector for collinearity constraint observations

variance factor

12



Area based matching – NCC vs. LSM

Positive and negative properties of NCC and LSM



� Feature based matching uses symbolic descriptions for establishing correspondences

� The representation using features is invariant with respect to distortions

(illumination, reflectance, geometry)

� Density of features is usually not sufficient. Usually FBM are combined with ABM methods to improve surface representation

Features

a) Interest points

� Type: corners, center of circular features

� Operators: contour-, intensity-based and parametric- model based

Intensity - based methods: Easy to implement, less complex, better localization accuracy

E.g. Foerstner, Harris, Moravec, Susan, etc.

b) Edges

� Detecting edge pixels (or edgels) -> Linking edge pixels into edges -> Grouping edges

� Operators: gradient- based, surface fitting, model matching, Laplacian of Gaussian and moment-based algorithm

Feature based matching (FBM)

13



Feature extraction� Edgel extraction - Canny extractor

1. Smoothing: using a gaussian smoothing operator

2. Gradient

3. Non-maximum suppression

4. Hysteresis threshold

1 2 3 4

dy =dx =



Feature extraction

1

Canny Grad. Thresh

14



Feature extraction

Foerstner Harris Susan

Canny Grad. Thresh Susan

Points

Edgels



Feature extraction

All Edges Straight Edges Straight Edges

> 10 pixels

15



Feature extractionEdgels All Edges Long straight Edges



Feature extractionComparison of operators (qualitative analysis)

++00-0+Method

-0-+--+Localization

+

+

+

+

Canny

-

+

+

-

Foerstner

+

-

+

+

Susan

+++-Time & memory performance

+--+Reliability

-

-

Harris

0

-

Susan

0+Distinct features

+-Completeness

Grad. Thresh.Foerstner

Remarks:

-Thresholds are not comparable

- No quantitative analysis

- Performance is related to implementation

PointsEdgels

16



Edgel matching – example modified LSM

Patch analysis derive interest values for signal ellipse (qa , qb ) + direction (φ)

Method A: - pre-rotation of patch with respect to the template- small patch size

vs. standart LSM: less iterations, decrease of oscillations for scales and shears

Method B:- pre-rotation of template and patch- thin rectangle patch (long dimension is aligned with the edge direction

vs. standart LSM: false matching could be almost eliminated, as long as the derived edge directions in template and patch are reliable.



Edge matching – example modified LSM

Advantages of method:- fast convergence (3-7 iterations) - less good approximations are required - danger of multiple solutions is reduced- safer determination of rotation

Nadir Backward Forward

Patch dimension expanded to contain whole edge- Center of template at the middle point of the edge- Pixels off-the-edge assigned a small weight (0.1)

17



Matching options according to features



Hierarchical processing

Use of doublets:

- avoid propagation of mismatches

- reduce processing time

Use of image pyramids: small parallaxes small search range

18



� determine parameters of a 2-D transformation between pixel and image coordinate system (usually affine)

� Example (a): approximate position of each fiducialuse fiducial as templatecross correlation

� Example (b): precisely locate fiducial centrecross correlation & subpixel fit (e.g. LSM) compute transformation parametersrobust least squares adjustment

Interior orientation

UL UR

LL LR



� Examples of problematic fiducials


poor contrast

(center – background)

poor contrast

(fiducial – image)

scratches

19



� Current status+ included in most DPWS+ autonomous, fast, accurate+ often manual measurements only + automatic estimation is problematic (reliability)

� for images acquired from digital cameras Automatic Interior Orientation is not required, because the projection centre remains stable with respect to the image




� recovery of the position and orientation of one imaging system relative to another from correspondences between 5 or more ray pairs

� Option (a): tie point distribution in six von-Gruber positions only+ most important areas for geometric stability+ similar to analytical relative orientation+ processing of relatively few data only- patches possibly unsuitable for matching- no full exploitation of available information

� Option (b): tie point distribution in the whole overlapping area+ avoids disadvantages of option (a) - higher computational cost

Relative orientation

(a)

(b)

20



ideal tie point positions




Point transfer– search space

step 1: transformation into object spacestep 2: computation of predicted position

size of search space depends on:+ accuracy of initial values for surface height+ image orientations+ location of selected position

21



Point transfer – search space

- Main source of uncertainty results from the terrain surface

Maximum Error Δpx in the predicted x-parallax in dependecy of :1. Base to height ratio B/Z2. focal length c3. tolerance Δφ in the two angles φ of the left and the right camera,4. the tolerance ΔZ/Z of the height Z related to Z,5. the tolerance ΔB/B of the basis related to B

Δpx = 2 c {1+ 1/4 (B/Z) } Δφ + c (B/Z) ΔZ/Z + c (B/Z) ΔB/B

- In mountainous terrain the maximum error can reach nearly 6 cm.+ Increasing overlap, thus reducing B/Z, also reduces the prediction errors (80%-90%)

Most approaches for point transfer contain module for generating a rough digital elevation model.



Point transfer – search space

Example � computation of search space

Normal case

Different overlap in and across base direction

� Z has largest influence on search space size (base direction only) � influence of exterior orientation can be further reduced by GPS / IMU

22



� tie point selection criteria

+ located in stable areas (no vegetation, water, moving objects, shadows) the position of the points should be accurate enough for the final adjustment process

+ evenly distributed across the whole image + located in open, horizontal areas in order to be visible in both images+ lie in as many overlapping images as possible, for stability reasons,+ be distinct for supporting efficient matching,

More robust selection when knowledge about the scene content and geometry exist(image orientation, object surface)





23



Tie points on moving objects

1

1

22



Tie points on moving objects

1

1

24



Tie points and repetitive texture



Tie points and repetitive texture

25




� Current status+ various systems available+ autonomous black box systems+ accuracy: between 0.2 and 0.4 pixel+ large redundancy (50 .. 100 points per model)+ much faster than manual measurements

� Problems in Automatic Realtive Orientation and Point Transfer- large and unknown rotation differences in k (> 20 degrees)- large scale differences (> 1.5)- close range imagery, in particular convergent imagery




� accuracy, reliability and redundancy+ �replace intelligence by redundancy� (Ackermann 1996), at least partly compensates for

lack of knowledge of scene content (not of geometry!)

+ use of many features per image ( 100, high redundancy)+ low accuracy σ0 of a single observation can be compensated by high redundancy (large n)

single measurement

repeated measurements

+ high reliability and easy methods for blunder detection (robust adjustment)

26



Automatic Aerial Triangulation




Matching Strategy

� tie point distribution

� tie point selection

� accuracy, reliability, and redundancy

� search space

� hierarchical processing (image pyramids)

27




Matching Strategy

� pairwise matching in all combinations followed by tuple generation

+ ABM and/or FBM methods available+ mismatches can often be identified at an early stage- ambiguous solutions across more than two images possible- generation of image tuples via forward intersection

is a combinatorial search problem, computational complexity must be controlled




Multi � image matching

+ most accurate matching technique available+ combinatorial complexity is avoided a priori (ABM only)+ mismatches possibly distributed across all related images

28




AAT system characteristics

� high redundancy, many tie points per image

� hierarchical image matching, followed by robust bundle adjustment

� sub pixel accuracy for tie pint coordinates, often through least-squares image matching

�some systems integrate determination of tie points and computation of orientation parameters



� compute image pyramids

� extract features in both images separately (attention to feature distribution)

� interest operators (Moravec, Förstner), edge operators (Canny, Deriche)

Point transfer – Generic approach

29



� match features

� feature description (interest value, length, curvature, contrast, ...)

� approximate epipolar constraint

� flight navigation data




� compute approximate orientation parameters

� robust least squares adjustment

� refine results through image pyramid


30



Strategies for point transfer

Footprints FootprintsStandard positionsSelection areas

0.3-0.4<0.10.3-0.4Local Accuracy [pel]

XX

XX

Local Constraints�collinearity�affinity�smooth surface

X

250-500

X

dense

Schenk (1993)

XXX

Local Strategy�sequential�all pairs�simultaneous

200-300250-300Points/Image

X3-4

X3

Technique�LSM�FB points�hierarchy (levels)

Ackermann (1995)Tsingas (1992)Author




XXXExterior orientation (BA)

X

YesXXX

Yes

X

index mapsno

Schenk (1993)

XRough DEM

No

YesX

No

Yes

X

Block formationa. Sequential� relat. orientation� scale tranfer� link of stripsb. Parallel (constraints)� local relat. Orientation� local affine� local smooth

GPSno

index mapsno

GLOBAL STRATEGYApproximate Values�orientation index �DEM

Ackermann (1995)Tsingas (1992)

Author

31




� Use of image pyramids in all approaches

� Small pull in range in LSM -> use all levels of the image pyramid

� Large pull in range in FBM -> 3- 4 levels of the image pyramid

� Multi � image matching in all approaches

� Tsingas relies on good navigational data and NOT too hilly terrain,

Ackermann and Schenk determine footprints

� Tsingas and Ackermann use the selected and correctly matched feature points in the final triangulation

(σ0 is a good indicator for the accuracy of the feature points).

� Schenk�s tie points are iteratively determined in the local multi-image matching procedure and transferred from the reference surface via the digital elevation model to the image planes.

(σ0 can be expected to be in the range comparable or superior to manual measurements)



Point transfer (Schenk)

32



DSM generation

� Hierarchical processing

Image pyramid Features DSM



DSM Generation

� Matching modules exist in various commercial photogrammetric systems.� Methods used are often based on cross-correlation, and match at a regular object or image grid. Better methods have been developed at research labs.

� Matching results, espec. with commercial systems, can vary a lot depending on the selection of the matching parameters (which have sometimes an unclear definition or at least effect).Below 3 automatically generated DSMs with DPW770, SocetSet. Left and right ATE, middleAdaptive ATE (effect of different matching strategies and matching parameters is clear)

33



DSM Generation (one of the methods developed in ETHZ)



DSM Generation

34



Matching quality

� Example of some measures+ Normalized correlations coefficient (NCC)+ Second best NCC+ Change of NCC when using different masks+ Consistency in backmatching (left-right and right-left match)+ Local paralax consistecy check+ angle of dominant edge direction to the epipolar line+ residuals from forward intersection

...LSM specific+ number of iterations+ changes in parameters

image matching, point transfer, dsm generationhome.agh.edu.pl/~krisfoto/lib/exe/...matching.pdf ·...

Documents