Theory and Practice of Projective Rectification

Richard I. Hartley

Presented by

Yinghua Hu

Goal

Apply 2D projective transformations on a pair of stereo images so that the epipolar lines in resulting images match and are parallel to the x-axis.

Stereo ConstraintsEpipolar Geometry

X1

Y1

Z1

O1

Image plane

Focal plane

M

u u’

Y2

X2

Z2O2

Epipolar Line

Epipole

Before rectification

Epipolar lines

After rectification

Some terms

Cofactor matrix A* A*A = AA*=det(A)I If A is an invertible matrix, A*≈(AT)-1

Skew symmetrix matrix Given a vector t = (tx, ty, tz)T

[t]× = [ 0 -tz ty

tz 0 -tx

- ty tx 0 ] [t]×s = t×s

Fundamental matrix

x’TFx = 0

Some Properties

F is the fundamental matrix corresponding to an ordered pair of images (J, J’) and p and p’ are the epipoles, then FT is the fundamental matrix corresponding to ima

ges (J’, J) F=[p’]×M=M* [p]×, M is non-singular and not unique Fp=0 and p’TF=0

Outline of the algorithm

Identify image-to-image matches ui ↔ ui’ between the two images.

Compute the fundamental matrix F and find epipoles p and p’ by Fp=0 and p’TF=0.

Select a projective transformation H’ that maps the epipole p’ to the point at infinity.

Find the matching projective transformation H that minimizes the disparity of the corresponding points.

Resample the two images J and J’ and generate the rectified images

Mapping epipole p’ to infinity

Required mapping H’ = GRT T translate a selected point u0 to the origin R rotate about the origin taking p’ to (f,0,1)T

G maps rotated p’ to infinity (f,0,0) The composite mapping H’ is to the first order

a rigid transformation in the neighborhood of u0

Choice of u0, center of the image or the center of all the corresponding points

Matching Transformations

The matching transformation H on the first image is of the form H = (I+H’p’aT)H’M, in which M can be computed by s

olving F=[p’]×M and a is a vector. It is shown that I+H’p’aT is an affine transformation of t

he form [x y z 0 1 0 0 0 1]

Least square minimization is used to find x,y,z which will minimize ∑d(Hui, H’ui’)2, so the disparity in x direction is also minimized.

Non Quasi-affine Transformation

Quasi-affine Transformation The line at infinity L∞ in the projective plane P2 consists of all

points with last coordinate equal to 0. A view window is a convex subset of the image plane. A projective transformation H is quasi-affine with respect to view

window W if H(W)∩L∞ = Ø Theorem 5.7 proves that for a quasi-affine transformation H’ on

W’ mapping p’ to infinity, and H be any matching projective transformation of H’ on W, there exists a subset W+ of view window W, such that H is quasi-affine with respect to W+ and W+ contains all the matching points in W corresponding to W’.

W+

W-

Resampling images

Firstly, determine the dimensions of the output images, a rectangle R containing H’(W’)∩H(W+). (When I implement it, I use a region containing H’(W’) U H(W+)).

For each pixel location in R, applying separately inverse transformation H-1 and H’-1 to find the corresponding location in images J and J’, interpolate the color values to get the rectified image pair.

Linear interpolation is adequate in most cases.

Scene Reconstruction

Without camera parameters, the scene can be reconstructed but there will remain projective distortion.

It is possible that the reconstructed images will be disconnected. To avoid this problem, a translation is done in x

direction in one of the images before reconstruction.

Applications Rectification may be used to detect changes in the

two images, simplifying image matching problem.

Main contribution

Rectify images based on corresponding points alone

Give a firm mathematical basis as well as a rapid practical algorithm

Doubt in this paper

Page 8, below equation 7Hp = (I+H’p’aT)H’Mp = (I+H’p’aT)H’p’

How? → = (I+aTH’p’)H’p’ ≈ H’p’

H’- 3×3 p’- 3×1 a - 3×1 H’p’aT - 3×3 aT- 1×3 aTH’p’ - 1×1