lecture 9h

7/31/2019 Lecture 9h

1/13

COMPUTER VISION: TWO-V IE W GEOMETRY

IIT Kharagpur

Computer Science and Engineering,

Indian Institute of Technology

Kharagpur.

1 77

Two View Geometry

Epipolar Geometry: is the intrinsic projective geometry betwe

two views.

Fundamental Matrix: F is a 3 3 matrix of rank 2.

Internal parameters of cameras

Relative pose

Intrinsic Projective Geo

x image of X on image 1 x image of X on image 2

xTFx = 0

ipolar Geometry Two-View Geometry

OMETRY COMPONENTS:Baseline: is the line joining the two camera centres.

Image planes of the two cameras P P.

Pencil of planes having baseline as the axis.

The 3D point X which gets projected as x and x on the two

cameras

Plane passing through x, x and the 3D point X.

3 77

Epipolar Geometry Two-View Geom

GEOMETRY COMPONENTS:

5 77



Rays back projected from x and x are coplanar lie on ) and inte

at X


2/13


OMETRY COMPONENTS:

x x are the corresponding points.

Plane : can be specified by the baseline and the ray

back-projected from x.

The line of intersection of with the second image is l

l is the epipolar line corresponding to the point x.

The corresponding point x lies on this epipolar line l.

7 77




OMETRY COMPONENTS:

9 77


GEOMETRY COMPONENTS:Epipole: is the pointof intersection of the line joining the cam

centres the baseline) with the image plane.

Epipole: is the imageof the camera centre of the other view.

Epipolar plane: is the plane containing the baseline. There is

one-parameter family a pencil) of epipolar planes.

Epipolar line: is the line of intersection of the epipolar plane w

the image plane.

ALL EPIPOLAR LINES INTERSECT AT THE EPIPOLE .


OMETRY COMPONENTS:

11 77




3/13


OMETRY COMPONENTS:

tion parallel to the image plane

13 77

Fundamental Matrix Epipolar Geom

FUNDAMENTAL MATRIX: F is the algebraic representation of the

epipolar geometry.

Point to line mapping: A point x has a corresponding epipola

l in the second image.

x l

This mapping is the fundamental matrix F. It is a projective

mapping from points to lines.

The corresponding point x which matches to x must lie on l.

ndamental Matrix Epipolar Geometry

GEOMETRIC DERIVATION :

Consider a plane not passing through either of the two camera

centres.

The ray back-projected from point x intersects plane at point X.

The point X gets projected to point x in the second image.

The projected point x lies on the epipolar line l.

15 77



The set of all points xi in the first image and the correspondin

points xi in the second image are projectively equivalent, sin

they are each projectively equivalent to the planar point set X

There is a 2-D homography H mapping each xi to x

i

H is the transfer mapping from image 1 to image 2 via plane



Given the point x the epipolar line l passes through x and

epipole e

l = [e]Hx = Fx

Fundamental matrix F = [e]H

17 77


Cross product matrix: e = e1 e2 e3)

[e] =

0 e3 e2e3 0 e1

e2 e1 0

Any skew symmetric 3 3 matrix may be written in the form [

for a suitable vector e.

Matrix [e] is singular, and e is its null vector right or left).

The cross product of two 3-vectors a b

a b = [a] b = aT [b]



4/13




[e] has rank 2, H has rank 3, F is a matrix of rank 2.

F is a mapping from IP2

onto a IP1

.F is a point map. It maps x l.

The pencil of epipolar lines through e forms IP1.

19 77


ALGEBRAIC DERIVATION:

The ray back-projected from x by P is obtained by solving P X

The ray is parametrized by the scalar .

X) = Px + C

P is the pseudo inverse of P, i.e. PP = I , C is the cameracentre given by PC = 0

Two points on the ray are Px at = 0) and camera centre C = ).

These two points are imaged by the second camera P

atPx PPx

C PC


GEBRAIC DERIVATION:The epipolar line joins these two projected points:

l = PC) PPx)

The epipole e = PC, we have l = e PP)x = Fx

F = [e]PP

Comparing this with the previously derived formula F = [e]H we

have H = PP.

21 77


IN TERMS OF CAMERA MATRICES:

P = K[ I 0] P = K[R t]

P =

K1

0

C =

0

1

F = [PC] PP

= [Kt] KRK1

= K [t] RK1

= KRRt

K1

= KRKKRt

Using result:

[t] M = M

M1t

= M

M1t

up to s

t is any vector

M non-singular matrix

M = detM)M


TERMS OF CAMERA MATRICES:

poles are given by images of the camera centres:

e= P

R

t

1

= KR

Tt e

= P

01

= K

t

F = [PC] PP

= [Kt] KRK1

= K [t] RK1

= KR

Rt

K1

= KRKKRt

F = [PC] PP

= [e]KRK1

= K [t] RK1

= KRRt

K1

= KRK [e]

23 77


CORRESPONDENCE CONDITION:

The epipolar line l = Fx. Since point x lies on this line, we havex

l = 0. This gives xFx = 0.

The fundamental matrix satisfies the condition that for any

pair of corresponding points x x in the two images xFx = 0F can be characterized without reference to camera matrix, o

terms of x x) point correspondences.

F can be computed from image correspondences.

At least 7 point correspondences are required to compute F.


5/13


OPERTIES:

F is unique for two views.

F is 3 3 homogeneous matrix with rank 2.

If F is the fundamental matrix of the pair of cameras P P), then

F is the fundamental matrix of the pair in opposite order P P).

Epipolar line l = Fx contains the epipole e.

eFx) = eF)x = 0 for all x eF = 0

Epipolar line l = F

x

contains the epipole e.

eFx) = eF)x = 0 for all x Fe = 0

25 77


PROPERTIES:

F has 7 degrees of freedom. A 3 3 homogeneous matrix ha

independent ratios. F also satisfies the constraint detF = 0 w

removes one degree of freedom.

F is a correlation: a projective map taking point to a line. l =

Any point x on l is mapped to the same epipolar line l. This

means there is no inverse mapping, and F is not of full rank.

F is not invertible. Hence F is not a proper correlation.


POLAR LIN E HOMOGRAPHY:The set of epipolar lines in each of the images forms a pencil of

lines passing through the epipoles.

Such pencil of lines may be considered as a 1-D projective space.

27 77


EPIPOLAR LIN E HOMOGRAPHY:The set of epipolar lines in each of the images forms a pencil

lines passing through the epipoles.

Such pencil of lines may be considered as a 1-D projective sp

The corresponding epipolar lines are perspectively related.

There is a homography between the pencil of lines centered a

the 1st view and the pencil of lines centered at e in the 2nd vie

A homography between two such 1-D projective spaces has 3

degrees of freedom.

Degrees of freedom

for F

2 for e

2 for e

3 for epipolar line homography

= 7


POLAR LIN E HOMOGRAPHY:

Suppose l and l are corresponding epipolar lines.

Suppose k is any line passing through epipole e.

The point of intersection of two lines l and k is x = [k]l = k l.This point lies on the epipolar line l.

The epipolar line corresponding to x is l = Fx = F[k]l

Likewise we have l = Fx = F[k]l

29 77


Next

What is F for special special motions between twoviews.


6/13


ECIAL MOTIONS BETWEEN VIEWS:

Pure translation between the two views.

Pure planar motion between the two views: the translation t is

orthogonal to the direction of rotation axis a

We assume there is no change in the internal parameters of the

camera viewing the scene.

31 77


SPECIAL MOTIONS BETWEEN VIEWS:

Pure translation

The camera undergoes a translation t.

Equivalently, the camera is assumed stationary and the world

points undergo translation t.

Points in 3-space move on straight lines parallel to t.

On the image plane these parallel lines appear to intersect at

vanishing point v in the direction of t.

Both the views have a common epipole v.

The imaged parallel lines are the epipolar lines.

33 77



Pure translation

Camera translating along principal axis

mera translating along principal axis

35 77


Pure translation

The two cameras can be chosen as:

P = K[ I 0] P = K[ I t]

Given that the camera coordinate system is aligned with the world

coordinate system and the camera is looking at the Z axis.


7/13

jection on the 1st camera Pure Translation

e inhomogeneous space point X gets projected to the

homogeneous) image point x.

X =

X

Y

Z

1

x =

x

y

1

Zx = PX = K[ I 0] X

ZK1x =

X

Y

Z

37 77

Projection on the 2nd camera Pure Trans

X =

X

Y

Z

1

x =

x

y

1

Zx = PX = K[ I t] X

Zx = [K Kt] X Zx = K

X

Y

Z

+ Kt

Zx = KZK1x) + Kt Zx = ZKK1x) + Kt

x = x + Kt/Z

The epipoles e e are the same in both the views and they are

vanishing points of the imaged parallel lines in the direction t.

e translation Fundamental Matrix

The situation whenthe object translates

by t is the same as

camera translating

by t

The epipoles e e are

the same in both the

views and they are

the vanishing points

of the imaged

parallel lines in the

direction t.

39 77

Pure translation Fundamental M

SOME OBSERVATIONS:

x = x + Kt/Z

The extent of motion depends on the magnitude of translation

and the inverse depth Z.

In the case of pure translation:

P = K[ I 0] P = K[ I t]

F = [PC] PP = [e]K

RK1 = [e]KK1 = [e]

F = [e]

e translation Fundamental Matrix

ME OBSERVATIONS: x = x + Kt/Z F = [e]

For camera translating parallel to x axis:

e =

1

0

0

F =

0 0 0

0 0 1

0 1 0

xFx = 0 and thus y = y

The fundamental matrix has 2 dofs which correspond to the

position of the epipole.

l = Fx = [e]x and x

[e]x = 0. Hence x lies on line [e]x = l

.

Implying that x x e = e are collinear.This collinearity property is termed as auto-epipolarand does not

hold for general motion.

y x[e]x = 0 ? Verify: x

0 0 0

0 0 1

0 1 0

x

41 77



General Motion

We are given two arbitrary views:

Correction 1: Rotate the camera used for the first image so th

is aligned with the second camera. This rotation may be simu

by applying a projective transformation to the first image.

Correction 2: Apply further correction can be applied to the fir

image to account for any difference in the calibration matrices

K K of the two cameras.

The result of the two corrections is a projective transformation

the first image.

Now the two cameras are related by a pure translation.


8/13

neral Motion Fundamental Matrix

43 77

General Motion Fundamental M

After applying the two corrections we have F as the fundamen

matrix between the corrected first image x and the second im

i.e. x x

F = [e] x = x xFx = 0

x[e]x = 0

Hence the fundamental matrix corresponding to the initial poi

correspondences x x is

F = [e]

trieving the camera matrices Fundamental matrix

The fundamental matrix F can be used to determine the cameramatrices of the two views.

The relations l = Fx and xFx = 0 are projectiverelationships.They make use of the projective coordinates in the image.

Euclidean measurements such as angles are not used.

If the images undergo a projective transformation,

x = x x = Hx

there is a corresponding map

l

= Fx F = H FH1

Fx = Hx)Fx) = xHFx = xFx

HF = F hence F = HFH1

45 77

Retrieving the camera matrices Fundamental m

The fundamental matrix F only depends on the projectiveproperties of the cameras P P.

F does not depend on the choice of the world coordinate fram

Rotation of world coordinates changes P P and not F.

If the 3-space undergoes a projective transformation using a

4 4 H1)

X = H1X

then the fundamental matrices corresponding to the pairs of

cameras P P) and P P) are the same.

PX = P)H1X) PX = P)H1X)

F = [PC] PP = P

H1C P

)H1P)

Fundamental matrix remains unchanged.


A pair of cameras can uniquely determine F.

A fundamental matrix determines the two cameras at best up to a

right multiplication by a 3D projective transformation.

en two camera matrices P P), it is always possible to identify amography such that P P) will form a canonical camera pair.

P = [ I 0] P = [M m]

e fundamental matrix corresponding to a pair of camera matrices

= [ I 0] P = [M m] is equal to F = [m] M

call

F = [e]PP

47 77


A fundamental matrix determines the two cameras at best up

right multiplication by a 3D projective transformation.

It will now be shown that if two pairs of camera matrices P P

and P

P

) have the same fundamental matrix F, then the pacamera matrices are related up to a right multiplication by aprojective transformation .

There always exists a non-singular 4 4 matrix such that

P = P and P

= P.

We can assume that the two pairs of cameras P P) and P P)

provided in the canonical form.

P = [ I 0] P = [A a] P = [ I 0] P

= [A a]


9/13


P = [ I 0] P = [A a] P = [ I 0] P

= [A a]

F = [a]A = [a]A

have

aF = a[a]A = 0 and a

F = a[a]A = 0

ce F is rank 2, it has a 1-D null space. Hence a = ka

ce [a]A = [a]A,

[a

]A = k[a

]A [

a]k

A A) =

0

re k is any constant.

49 77


[a]A = k[a]A [a]kA A) = 0

Now, [a]kA A) is a 3 3 matrix.

If we substitute kA A) by a 3 3 matrix of form av then wethat [a]av

= 0

Hence kA A) = av where v is any 3-vector.Thus,

A = k1A + av)


en the two substitutions: a = ka and A = k1A + av)e camera matrices now become:

P = [ I 0]

P = [A a]

P = [ I 0]

P

= [A a]

P = [ I 0]

P

= [k1A + av ka]

here any which will now give

P = P and P = P

51 77


P = [ I 0]

P = [A a]

P = [ I 0]

P

= [k1A + av ka]

We choose =

k1 I 0

k1v k

P = [ I 0]

P = [A a]

P = k1P

P = [A a]

= [k1A + av) ka

= [A a] = P

Thus we have P = P and P = P

grees of Freedom Fundamental matrix

Each of the two camera matrices P P) have 11 degrees offreedom. Total: 22 dofs.

Specifying a projective world frame requires 15 dofs.

22-15 = 7.The fundamental matrix F has 7 degrees of freedom.

53 77

Computing camera matrices Fundamental m

F can determine the camera pair up to a projective transforma

of 3-space.

If any matrix, say M is skew symmetric, we have xMx = 0

Consider the composite matrix P

FP

we have XPFPX = 0 since xFx = 0

A non-zero matrix F is the fundamental matrix corresponding

pair of camera matrices P and P if and only if PFP is ske

symmetric.


10/13

mputing camera matrices Fundamental matrix

Consider F to be the given Fundamental matrix.

Consider a pair of 3 4 matrices

P = [ I 0] P = [SF e] such that e

F = 0

Assume that P P have rank 3.

We need to verify if P P are indeed the camera matrices

corresponding to F. Following conditions need to be checked:

We need to verify that PFP is skew symmetric.

We need to choose a skew symmetric matrix S such that P has

rank 3.

55 77


Check PFP is skew symmetric

[SF e] F [ I 0] =

FS

F 0

eF 0

=

FS

F 0

0 0

This is indeed skew symmetric if S is skew symmetric.

Choosing a suitablematrix S

S is skew symmetric. In terms of its null vector S = [s].

P = [SF e] = [ [s]F e]


oosing a suitablematrix S

oose S = [s].P = [SF e] = [ [s]F e

]

need to verify that P = [ [s]F e] has rank 3. [ [s]F e] will have rank 3 provided se 0. Why?

57 77


Choosing a suitablematrix S [ [s]F e] will have rank 3 provided se 0.[s]F has rank 2. The column space of [s]F is spanned by

cross product of s with the columns of F, and equals the pla

perpendicular to s.

If se 0 then e is not perpendicular to s, and hence it does

lie in this plane.

Thus [ [s]F e] has rank 3.

A suitable choice for s can be e since we have ee 0.

Thus we take S = [s] = [e]


The camera matrices corresponding to the Fundamental matrix F

can be chosen as:

P = [ I 0] P = [[e]F e] such that e

F = 0

The left 3 3 sub-matrix of P i.e. [e]F has rank 2. Thiscorresponds to a camera with centre at .

59 77


FAMILY OF CAMERAS WHICH HAVE THE SAME F

We can identify a family of cameras:

P = [ I 0] P = [[e]F e

v ke]

vk are parameters

v is any 3-vector and k is a non-zero scalar.


11/13

Fundamental matrix

xt

After Fundamental matrix: .....

The Essential Matrix

61 77

Essential Matrix A special case

The Essential matrix is a special case of fundamental matrix f

pair of normalized cameras.

It has fewer degrees of freedom and additional properties.

It makes use of normalized image coordinates.

Normalized Coordinates

P = K[R t] x = PX x = K1x = [R t]X

The matrix K1P = [R t] is the normalized camera matrix.

We have the normalized camera pair: P = [ I 0] and P = [R

sential Matrix A special case of F

P = K[R t] x = PX x = K1x = [R t]X

The Fundamental matrix corresponding to the normalized camera

pair P = [ I 0] and P = [R t] is called as the Essential matrix:

E = [t]R = R

Rt

We have

xEx = 0

For the corresponding points x x, the normalized image

coordinates are x x

63 77


xEx = 0

Substituting for x = K1x and x = K1x gives

xKEK1x = 0 F = KEK1 E = KFK

Properties of Essential Matrix

Has 5 dofs: 3 for R and 3 for t and -1 for overall scale.

A 3 3 matrix is an essential matrix if and only if two of itssingular values are equal and third is zero.


xt

We show that E has TWO singular values whichare equal and the third is zero

65 77


Consider decomposition of E as

E = SR = [t]R

S is a skew-symmetric matrix which can be decomposed as

S = kUZU where U is orthogonal

Matrix Z is a block diagonal matrix of the form

Z =

0 1 0

1 0 0

0 0 0

as a matrix product =

1 0 0

0 1 0

0 0 0

0 1

1 0

0 0


12/13


nsider decomposition of E as

E = SR = kUZU R = [t]R

s skew symmetric and

Z = diag1,1,0) W where W =

0 1 0

1 0 0

0 0 1

urns out to be an orthogonal matrix. S = UZU

S = U diag1,1,0) W U E = SR = U diag1,1,0) WUR)

67 77


Consider decomposition of E as up to scale)

E = SR = UZU R = [t]R

SVD decomposition of

E = UDV = U diag1,1,0) WUR) where V = WUR)

Thus E has two singular values which are equal.

SVD of E is not unique. Alternate SVDs are given as:

E = UdiagR22 1)) diag1,1,0) diagR22 1)V)

where R22 is any rotation matrix.


xt

How to extract cameras from E

69 77

Extraction of Cameras Essential ma

Consider decomposition of E as up to scale)

E = SR = [t]R

The two cameras can be chosen as: P = [ I 0] P = [R t

The vector t has to be chosen such that S t = 0.

SVD of E is not unique. Alternate SVDs are given as:

E = UDV = U diag1,1,0) WUR) where V = WUR

W =

0 1 0

1 0 0

0 0 1

R = UW

V

raction of Cameras Essential matrix E

nsider decomposition of E as

E = SR = [t]R

e two cameras can be chosen as: P = [ I 0] P

= [R t]The vector t has to be chosen such that S t = 0.

We choose

t = U0 0 1) = u

It can be verified that St = 0

St = U diag1,1,0) W U) t

= U diag1,1,0) W U) u

= 0

71 77


It can be verified that St = 0

St = U diag1,1,0) W U) t

= U diag1,1,0) W U) u

a1 a2 a3b1 b2 b3c1 c2 c3

1 0 0

0 1 0

0 0 0

0 1 0

1 0 0

0 0 1

a1 b1 c1a2 b2 c2a3 b3 c3

a

b

ca1 a2 a3b1 b2 b3c1 c2 c3

0 1 0

1 0 0

0 0 0

a1 b1 c1a2 b2 c2a3 b3 c3

a

b

ca2 a1 0

b2 b1 0

c2 c1 0

a1 b1 c1a2 b2 c2a3 b3 c3

a

b

c


13/13

raction of Cameras Essential matrix Ea2a1 a1a2 a2b1 a1b2 a2c1 a1c2b2a1 b1a2 b2b1 b1b2 b2c1 b1c2c2a1 a2c1 c2b1 c1b2 c2c1 c1c2

a3b3c3

0 a2b1 a1b2 a2c1 a1c2b2a1 b1a2 0 b2c1 b1c2c2a1 a2c1 c2b1 c1b2 0

a3b3c3

a2b1b3 a1b2b3 + a2c1c3 a1c2c3a3b2a1 a3b1a2 + b2c1c3 b1c3c2c2a3a1 a2a3c1 + c2b1b3 c1b2b3

a2b1b3 + c1c3) a1b2b3 + c2c3)b2a3a1 + c1c3) b1a3a2 + c3c2)c2a3a1 + b1b3) c1a2a3 + b2b3)

=

a2a1a3 a1a2a3b2b1b3 b1b2b3c2c1c3 c1c2c3

= 0

73 77


Consider decomposition of E as

E = SR = [t]R

The two cameras can be chosen as: P = [ I 0] P = [R t

We choose

t = U0 0 1) = u R = UWV

There are 4 possible pairs of cameras:

P

= [R t]= [UWV + u]

= [UWV u]

P

= [R t]= [UWV + u]

= [UWV u]

raction of Cameras Essential matrix E

There are 4 possible pairs of cameras:

P = [R t]

= [UWV + u]

= [UWV u]

P = [R t]

= [UWV + u]

= [UWV u]

W and W are related by a rotation througth 1800 about the

base-line.

75 77

mmary

Intrinsic projective geometry of 2-views.

Epipolar geometry

Fundamental matrix

Deriving the fundamental matrix from camera matrices.Deriving the fundamental matrix from point correspondences.

Deriving the camera matrices from the fundamental matrix.

Essential matrix

Deriving the camera matrices from the essential matrix.

77 77

lecture 9h

Documents