cs 558 computer vision

CS 558 COMPUTER VISIONLecture VIII: Single View and Epipolar Geometry

Slide adapted from S. Lazebnik

OUTLINE Single view geometry Epiploar geometry

SINGLE-VIEW GEOMETRY

Odilon Redon, Cyclops, 1914

OUR GOAL: RECOVERY OF 3D STRUCTURE• Recovery of structure from one image is

inherently ambiguous

x

X?X?

X?

OUR GOAL: RECOVERY OF 3D STRUCTURE• Recovery of structure from one image is

inherently ambiguous

OUR GOAL: RECOVERY OF 3D STRUCTURE• Recovery of structure from one image is

inherently ambiguous

AMES ROOM

http://en.wikipedia.org/wiki/Ames_room

OUR GOAL: RECOVERY OF 3D STRUCTURE• We will need multi-view geometry

RECALL: PINHOLE CAMERA MODEL

• Principal axis: line from the camera center perpendicular to the image plane

• Normalized (camera) coordinate system: camera center is at the origin and the principal axis is the z-axis

)/,/(),,( ZYfZXfZYX

10100

1ZYX

ff

ZYfXf

ZYX

RECALL: PINHOLE CAMERA MODEL

PXx

PRINCIPAL POINT

• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)

• Normalized coordinate system: origin is at the principal point

• Image coordinate system: origin is in the corner• How to go from normalized coordinate system to image

coordinate system?

px

py

)/,/(),,( yx pZYfpZXfZYX

10100

1ZYX

pfpf

ZpZYfpZXf

ZYX

y

x

y

x

PRINCIPAL POINT OFFSET

principal point: ),( yx pp

px

py

1010101

1ZYX

pfpf

ZZpYfZpXf

y

x

y

x

PRINCIPAL POINT OFFSET

0|IKP

1y

x

pfpf

K calibration matrix

principal point: ),( yx pp

111yy

xx

y

x

y

x

pfpf

mm

K

PIXEL COORDINATES

yx mm11

mx pixels per meter in horizontal direction, my pixels per meter in vertical direction

Pixel size:

pixels/m m pixels

C~-X~RX~ cam

CAMERA ROTATION AND TRANSLATION

• In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation

coords. of point in camera frame

coords. of camera center in world frame

coords. of a pointin world frame (nonhomogeneous)

C~-X~RX~ cam

X10

C~RR1X~

10C~RR

Xcam

XC~R|RKX0|IKx cam ,t|RKP C~Rt

CAMERA ROTATION AND TRANSLATION

In non-homogeneouscoordinates:

Note: C is the null space of the camera projection matrix (PC=0)

• Intrinsic parameters Principal point coordinates Focal length Pixel magnification factors Skew (non-rectangular pixels) Radial distortion

CAMERA PARAMETERS

111yy

xx

y

x

y

x

pfpf

mm

K

• Intrinsic parameters Principal point coordinates Focal length Pixel magnification factors Skew (non-rectangular pixels) Radial distortion

• Extrinsic parameters Rotation and translation relative to world

coordinate system

CAMERA PARAMETERS

CAMERA CALIBRATION

Source: D. Hoiem

1************

ZYX

yx

XtRKx

CAMERA CALIBRATION• Given n points with known 3D coordinates Xi

and known image projections xi, estimate the camera parameters

? P

Xi

xi

ii PXx

CAMERA CALIBRATION: LINEAR METHOD

0PXx ii 0XPXPXP

1 3

2

1

iT

iT

iT

i

i

yx

0PPP

0XXX0X

XX0

3

2

1

Tii

Tii

Tii

Ti

Tii

Ti

xyxy

Two linearly independent equations

CAMERA CALIBRATION: LINEAR METHOD

• P has 11 degrees of freedom (12 parameters, but scale is arbitrary)

• One 2D/3D correspondence gives us two linearly independent equations

• Homogeneous least squares• 6 correspondences needed for a minimal

solution

0pA 0PPP

X0XXX0

X0XXX0

3

2

1111

111

Tnn

TTn

Tnn

Tn

T

TTT

TTT

xy

xy

CAMERA CALIBRATION: LINEAR METHOD

• Note: for coplanar points that satisfy ΠTX=0,we will get degenerate solutions (Π,0,0), (0,Π,0), or (0,0,Π)

0Ap 0PPP

X0XXX0

X0XXX0

3

2

1111

111

Tnn

TTn

Tnn

Tn

T

TTT

TTT

xy

xy

CAMERA CALIBRATION: LINEAR METHOD• Advantages: easy to formulate and solve• Disadvantages

Doesn’t directly tell you camera parameters Doesn’t model radial distortion Can’t impose constraints, such as known focal

length and orthogonality

• Non-linear methods are preferred Define error as difference between projected

points and measured points Minimize error using Newton’s method or other

non-linear optimization

Source: D. Hoiem

• Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates of that point

Camera 3R3,t3 Slide credit:

Noah Snavely

?

Camera 1Camera 2R1,t1 R2,t2

MULTI-VIEW GEOMETRY PROBLEMS

MULTI-VIEW GEOMETRY PROBLEMS• Stereo correspondence: Given a point in one of

the images, where could its corresponding points be in the other images?

Camera 3R3,t3

Camera 1Camera 2R1,t1 R2,t2 Slide credit:

Noah Snavely

MULTI-VIEW GEOMETRY PROBLEMS• Motion: Given a set of corresponding points in two

or more images, compute the camera parameters

Camera 1Camera 2 Camera 3

R1,t1 R2,t2 R3,t3? ? ? Slide credit:

Noah Snavely

TRIANGULATION• Given projections of a 3D point in two or

more images (with known camera matrices), find the coordinates of the point

O1 O2

x1x2

X?

TRIANGULATION• We want to intersect the two visual rays

corresponding to x1 and x2, but because of noise and numerical errors, they don’t meet exactly

O1 O2

x1x2

X?R1R2

TRIANGULATION: GEOMETRIC APPROACH• Find shortest segment connecting the two

viewing rays and let X be the midpoint of that segment

O1 O2

x1x2

X

TRIANGULATION: LINEAR APPROACH

baba ][0

00

z

y

x

xy

xz

yz

bbb

aaaaaa

XPxXPx

222

111

0XPx0XPx

22

11

0XP][x0XP][x

22

11

Cross product as matrix multiplication:

TRIANGULATION: LINEAR APPROACH

XPxXPx

222

111

0XPx0XPx

22

11

0XP][x0XP][x

22

11

Two independent equations each in terms of three unknown entries of X

TRIANGULATION: NONLINEAR APPROACH

),(),( 222

112 XPxdXPxd

Find X that minimizes

O1 O2

x1x2

X?

x’1

x’2

TWO-VIEW GEOMETRY

• Epipolar Plane – plane containing baseline (1D family)• Epipoles = intersections of baseline with image planes = projections of the other camera center

• Baseline – line connecting the two camera centers

EPIPOLAR GEOMETRYX

x x’

THE EPIPOLE

Photo by Frank Dellaert

• Epipolar Plane – plane containing baseline (1D family)• Epipoles = intersections of baseline with image planes = projections of the other camera center• Epipolar Lines - intersections of epipolar plane with image

planes (always come in corresponding pairs)

• Baseline – line connecting the two camera centers

EPIPOLAR GEOMETRYX

x x’

EXAMPLE: CONVERGING CAMERAS

EXAMPLE: MOTION PARALLEL TO IMAGE PLANE

EXAMPLE: MOTION PERPENDICULAR TO IMAGE PLANE

EXAMPLE: MOTION PERPENDICULAR TO IMAGE PLANE

e

e’

EXAMPLE: MOTION PERPENDICULAR TO IMAGE PLANE

Epipole has same coordinates in both images.Points move along lines radiating from e: “Focus of expansion”

EPIPOLAR CONSTRAINT

• If we observe a point x in one image, where can the corresponding point x’ be in the other image?

x x’

X

• Potential matches for x have to lie on the corresponding epipolar line l’.

• Potential matches for x’ have to lie on the corresponding epipolar line l.

EPIPOLAR CONSTRAINT

x x’

X

x’

X

x’

X

EPIPOLAR CONSTRAINT EXAMPLE

X

x x’

EPIPOLAR CONSTRAINT: CALIBRATED CASE

• Assume that the intrinsic and extrinsic parameters of the cameras are known

• We can multiply the projection matrix of each camera (and the image points) by the inverse of the calibration matrix to get normalized image coordinates

• We can also set the global coordinate system to the coordinate system of the first camera. Then the projection matrix of the first camera is [I | 0].

X

x x’

EPIPOLAR CONSTRAINT: CALIBRATED CASE

R

t

The vectors x, t, and Rx’ are coplanar

= RX’ + t

Essential Matrix(Longuet-Higgins, 1981)

EPIPOLAR CONSTRAINT: CALIBRATED CASE

0)]([ xRtx RtExExT ][with0

X

x x’

The vectors x, t, and Rx’ are coplanar

X

x x’

EPIPOLAR CONSTRAINT: CALIBRATED CASE

• E x’ is the epipolar line associated with x’ (l = E x’)• ETx is the epipolar line associated with x (l’ = ETx)• E e’ = 0 and ETe = 0• E is singular (rank two)• E has five degrees of freedom

0)]([ xRtx RtExExT ][with0

EPIPOLAR CONSTRAINT: UNCALIBRATED CASE

• The calibration matrices K and K’ of the two cameras are unknown

• We can write the epipolar constraint in terms of unknown normalized coordinates:

X

x x’

0ˆˆ xExT xKxxKx ˆ,ˆ

EPIPOLAR CONSTRAINT: UNCALIBRATED CASEX

x x’

Fundamental Matrix(Faugeras and Luong, 1992)

0ˆˆ xExT

xKx

xKx

1

1

ˆ

ˆ

1with0 KEKFxFx TT

EPIPOLAR CONSTRAINT: UNCALIBRATED CASE

0ˆˆ xExT 1with0 KEKFxFx TT

• F x’ is the epipolar line associated with x’ (l = F x’)• FTx is the epipolar line associated with x (l’ = FTx)• F e’ = 0 and FTe = 0• F is singular (rank two)• F has seven degrees of freedom

X

x x’

THE EIGHT-POINT ALGORITHMx = (u, v, 1)T, x’ = (u’, v’, 1)T

Minimize:

under the constraintF33

= 1

2

1

)( i

N

i

Ti xFx

THE EIGHT-POINT ALGORITHM• Meaning of error

sum of Euclidean distances between points xi

and epipolar lines F x’i (or points x’i and epipolar lines FTxi) multiplied by a scale factor

• Nonlinear approach: minimize

:)( 2

1i

N

i

Ti xFx

N

ii

Tiii xFxxFx

1

22 ),(d),(d

PROBLEM WITH EIGHT-POINT ALGORITHM

PROBLEM WITH EIGHT-POINT ALGORITHM

Poor numerical conditioning Can be fixed by rescaling the data

THE NORMALIZED EIGHT-POINT ALGORITHM

• Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels

• Use the eight-point algorithm to compute F from the normalized points

• Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value)

• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is TT F T’

(Hartley, 1995)

COMPARISON OF ESTIMATION ALGORITHMS

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel

FROM EPIPOLAR GEOMETRY TO CAMERA CALIBRATION

• Estimating the fundamental matrix is known as “weak calibration”

• If we know the calibration matrices of the two cameras, we can estimate the essential matrix: E = KTFK’

• The essential matrix gives us the relative rotation and translation between the cameras, or their extrinsic parameters