what does calibration give? an image line l defines a plane through the camera center with normal...

What does calibration give?

xKd 1

0d0]|K[Ix

21-T-T

211-T-T

1

2-1-TT

1

2T

21T

1

2T

1

)xK(Kx)xK(Kx

)xK(Kx

dddd

ddcos

An image line l defines a plane through the camera center

with normal n=KTl measured in the camera’s Euclidean

frame. In fact the backprojection of l is PTl n=KTl

The image of the absolute conic

KRd0

dO]|KR[IPXx

mapping between ∞ to an image is given by the planar

homogaphy x=Hd, with H=KR

absolute conic (IAC), represented by I3 within ∞ (w=0)

1-T-1T KKKKω 1TCHHC

(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT

(iv) K (Cholesky factorization)(v) image of circular points belong

to (image of absolute conic)

2T

21T

1

2T

1

ωxxωxx

ωxxcos

its image (IAC)

A simple calibration device

(i) compute Hi for each square

(corners (0,0),(1,0),(0,1),(1,1))

(ii) compute the imaged circular points Hi [1,±i,0]T

(iii) fit a conic to 6 imaged circular points

(iv) compute K from K-T K-1 through Cholesky factorization (= Zhang’s calibration method)

Orthogonality relation

2T

21T

1

2T

1

ωvvωvv

ωvvcos

0ωvv 2T

1

0lωl 2*T

1

Calibration from vanishing points and lines

Calibration from vanishing points and lines

Two-view geometry

Epipolar geometry

F-matrix comp.

3D reconstruction

Structure comp.

(i) Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the

corresponding point x’ in the second image?

(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?

(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?

Three questions:

The epipolar geometry

C,C’,x,x’ and X are coplanar

The epipolar geometry

What if only C,C’,x are known?

The epipolar geometry

All points on project on l and l’

The epipolar geometry

Family of planes and lines l and l’ Intersection in e and e’

The epipolar geometry

epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction

an epipolar plane = plane containing baseline (1-D family)

an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)

Example: converging cameras

Example: motion parallel with image plane

The fundamental matrix F

algebraic representation of epipolar geometry

l'x

we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

The fundamental matrix F

algebraic derivation

λC0

xMλX

-1

1MM'e'F

xMM'Cm'M'0

xMm''MC'm'Ml' 1

1

e’ is the image of C taken by the second camera

xP

λX0Fxx'0 l' x' l' x' T

The fundamental matrix F

correspondence condition

0Fxx'T

The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images 0l'x'T

The fundamental matrix F

F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’

(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x

e’TF=0, similarly Fe=0;( vi e’ is left null space of F(v) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(vi) F is a correlation, projective mapping from a point x to

a line l’=Fx (not a proper correlation, i.e. not invertible)

Projective transformation and invariance

-1-T FHH'F̂ x'H''x̂ Hx,x̂

Derivation based purely on projective concepts

X̂P̂XHPHPXx -1

F invariant to transformations of projective 3-space

X̂'P̂XHHP'XP'x' -1

FP'P,

P'P,F

unique

not unique

canonical form

m]|[MP'0]|[IP

MmF

Projective ambiguity of cameras given Fprevious slide: at least projective ambiguitythis slide: not more!

Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’

~ ~

~ ~

]a~|A~

['P~ 0]|[IP

~ a]|[AP' 0]|[IP

A~

a~AaF

T1 avAA~

kaa~ klemma:

kaa~Fa~0AaaaF2rank

TavA-A~

k0A-A~

kaA~

a~Aa

kkIkH T1

1

v0

'P~

]a|av-A[v

0a]|[AHP' T1T1

1

kk

kkIk

(22-15=7, ok)

Canonical cameras given F

F matrix corresponds to P,P’ iff P’TFP is skew-symmetric

X0,FPXP'X TT

F matrix, S skew-symmetric matrix

]e'|[SFP' 0]|[IP

000FSF

0Fe'0FSF0]|F[I]e'|[SF

TT

T

TTT

(fund.matrix=F)

Possible choice:

]e'|F][[e'P' 0]|[IP

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

Triangulation of points in 3D

Ransac: - 8 point correspondence samples

Computation of F1i

-Compute viewing rays from cameras-Intersect viewing rays associated to corresponding points

Computation of canonical camerasfrom F1i

PROJECTIVE RECONSTRUCTION

The essential matrix

~fundamental matrix for calibrated cameras (remove K)

t]R[RRtE T

0x̂E'x̂ T

FKK'E T

x'K'x̂ x;Kx̂ -1-1

5 d.o.f. (3 for R; 2 for t up to scale)

E is essential matrix if and only iftwo singularvalues are equal (and third=0)

T0)VUdiag(1,1,E SVD

100

001

010

WGiven

Motion from E

TTVUWR'

TUWV'R'

and

3Ut'

3U't'

Four solutions

Four possible reconstructions from E

(only one solution where points is in front of both cameras)

Self-calibration

Motivation

• Avoid explicit calibration procedure– Complex procedure– Need for calibration object – Need to maintain calibration

Motivation

• Allow flexible acquisition– No prior calibration necessary– Possibility to vary intrinsics– Use archive footage

Constraints ?

• Scene constraints– Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...

Unknown scene no constraints

• Camera extrinsics constraints– Pose, orientation, ...

Unknown camera motion no constraints • Camera intrinsics constraints

– Focal length, principal point, aspect ratio & skew

Perspective camera model too general some constraints

Constraints on intrinsic parameters

Constant e.g. fixed camera:

Knowne.g. rectangular pixels:

square pixels:

principal point known:

21 KK

0s

1yy

xx

uf

usf

K

0, sff yx

2,

2,

hwuu yx

Self-calibration

Upgrade from projective structure to metric structure using constraints on intrinsic camera parameters– Constant intrinsics

– Some known intrinsics, others varying

– Constraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion) (Moons et al.´94, Hartley ´94, Armstrong ECCV´96)

(Faugeras et al. ECCV´92, Hartley´93,

Triggs´97, Pollefeys et al. PAMI´98, ...)

(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)

A counting argument

• To go from projective (15DOF) to metric (7DOF) at least 8 constraints are needed

• Minimal sequence length should satisfy

• Independent of algorithm• Assumes general motion (i.e. not critical)

8#1# fixednknownn

known internal parameters

-1-TKKω

0ωω 2112 0s

rectangular pixels

yx ff

square pixels

2211 ωω

Same camera for all images

same intrinsics same image of the absolute conic

e.g. moving cameras

given sufficient images there is in general only oneconic that projects to the same image in all images,i.e. the absolute conic

This approach is called self-calibration, see later

ωHHω' Ttransfer of IAC:

(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i

Obtain projective reconstruction (P,P‘,{Xi})

0]|[IP1 ]a|[AP iii

e.g. ]e|veFe[P iT

ixii

t]|[A1v

tA0]|[I0]H|[I0]|[KP T1M1

The metric reconstruction will be

t=0 1KA

Let the unknown plane at the infinity in the starting projective reconstruction be

1

pπ*

From1

pπH

1ππ *T*

0M

1

vK-

110

vK-K-

1H

1

pπ

-T1

-T1

-T1T-*

00

thus

v-Kp -T1 and p-Kv T

1Therefore,

1Kp

0KH

1T

1

iiiii1T

iiiiM tKRKa)Kpa-(AHaAPi

Now

8 unknowns:

Tiiii

Tii )R)(KR(KKK

TTii

T11

Tii

Tii )pa-(AK)Kpa-(AKK

Therefore, since

Self-calibration equation

1K and p

Equations come from constraints on intrinsics:-known parameters (e.g. s=0 or square pixels)-fixed parameters (e.g. s, or aspect ratio)

8#1# fixednknownn

Solution: nonlinear algorithms (Cipolla)

needed views: n, with

Uncalibrated visual odometry Uncalibrated visual odometry for ground plane motionfor ground plane motion

(joint work with Simone Gasparini)

Problem formulation

• Given:– an uncalibrateduncalibrated camera mounted on a robot– the camera is fixedfixed and aims at the floor– the robot moving on a planarplanar floor (ground

plane)

• Determine:– the estimate of robot motion from observed

features on the floor

Technique involvedEstimate the ground plane transformation (homography) between images taken before and after robot displacement

Motivations• Dead reckoning techniques are not reliable and diverge after few steps

[Borenstein96]

• Visual odometry techniques exploits cameras to recover motion

• We use a single uncalibrated camera– 3D reconstruction with uncalibrated camera usually require auto-calibration– Non-planar motion is required [Triggs98]

– Planar motion with different camera attitudes [Knight03]

– Special devices is required (e.g. PTZ cameras)

• Stereo cameras approaches [Nister04, Takaoka04, Agrawal06, Cheng06]

• Catadioptric cameras approaches [Bunschoten03, Corke04]

– Assume Single View Point (difficult setup)

• Our method similar in spirit to [Wang05] and [Benhimane06]

– We do not assume camera calibration

Problem formulation

• Fixed uncalibrated camera mounted on a robot

• The pose of the camera w.r.t. the robot is unknown

• The projective transformation between ground plane and image plane is a homography T (3x3)

• T does not change with robot motion

• T is uknownT is uknown

Image plane

Ground Plane

X

x x=TX

Problem formulation

• Robot and camera undergo a planar motion– rotation of unknown angle about unknown

vertical axis– 2D rotation matrix R (3x3 with homogeneous

coordinates)

2x2ˆ ˆ orthogonal matrix

0 translation vector (3x1 )

R R

R tt

Problem formulation

• Given 2 images before and after robot displacement determine:– Rotation centre C– Rotation angle – Determine unknown transformation T between

ground plane and image plane

Ground reference frame

• We define a ground reference frame– O is the origin of the projected reference frame

• E.g. O is the backprojection of image point O’(0,0)

– The vector connecting the origin to a point A is the unit vector along the x-axis

• E.g. A is the backprojection of image point A’(100,0)

Image plane

Ground Plane

O

A

O'

A’

Estimation of robot displacement

• Transformation relating the two images is still a homography

• Eigenvectors of H:

1 ( unknown)H = TRT T

' associated to the eigval, C rotation center

' associated to the eigval, I and J circular points

'

C TC

eigvec I TI

J TJ

H =

• Since eig(H)=eig(R)– Angle can be calculated through the ratio of

imaginary and real part of the complex eigenvalue of H

– E.g. the eigenvalue associated to I’ is e±i

Estimation of robot displacement: degenerate motion

Pure translation: a frequent motion on planar floor

• Under pure translation, the whole line at the infinity is invariant

ground plane can be only rectified modulo an affine transformation: e.g., orientation of translaton wrt ground reference

can not be determined

• In practice: – Use a motion including rotations to estimate

ground plane to image plane transformation T– Use knowledge of T while translating

Estimation of the transformation T

• Estimating T allows to determine the shape

of observed features

• Four pairs of corresponding points are

needed

– The 2 circular points I and J

– The 2 points A and O used for ground reference

where 1, ,0

where 1, ,0

where 0,0,1 and ' 0,0,1

where 1,0,1 and ' 100,0,1

T

T

T T

T T

i

i

I' = TI I

J' = TJ J

O' = TO O O

A' = TA A A

Experimental set up

• Fixed perspective camera placed on turntable ( ground truth)

• Optical distortion is negligible

• Camera pointing towards the ground floor

• Camera viewpoint in a generic position wrt rotation axis

• Basic algorithm:– Feature (corner) extraction from floor texture

[Harris88]– Feature tracking and matching in order to

determine correspondences– Corresponding features used to fit the

homography H– Eigendecomposition of matrix H

• Rotation angle• Image of circular points• Image of rotation centre• Ground plane to image plane transformation

Experiments on sequences of images

Sequence of large displacements

• Image displacements of the order of 10°

• Features extracted and matched [Torr93]

• Use best matching pairs to fit a homography using RANSAC [Fischler81]

• Good overall accuracy (error < 1°)

• Larger displacements affect matching

Experiments on sequences of images

Sequence of small displacements

• Image displacements of about 5°

• Small rotations may lead to numerical instability

• Track features over three images– H13 fitted with correspondences of 1 and 3

• Good overall accuracy (error < 1°)

HH1313

1 2 3

Experiments on a mobile platform

Feature extraction and matching between images

Rotation and relevant centre of rotation determination