Camera models and calibration
Read tutorial chapter 2 and 3.1http://www.cs.unc.edu/~marc/tutorial/
Szeliski’s book pp.29-73
Schedule (tentative)
2
# date topic
1 Sep.17 Introduction and geometry
2 Sep.24 Camera models and calibration
3 Oct.1 Invariant features
4 Oct.8 Multiple-view geometry
5 Oct.15 Model fitting (RANSAC, EM, …)
6 Oct.22 Stereo Matching
7 Oct.29 Structure from motion
8 Nov.5 Segmentation
9 Nov.12 Shape from X (silhouettes, …)
10 Nov.19 Optical flow
11 Nov.26 Tracking (Kalman, particle filter)
12 Dec.3 Object category recognition
13 Dec.10 Specific object recognition
14 Dec.17 Research overview
Brief geometry reminder
3
0=xlT
l'×l=x x'×x=l2D line-point coincidence relation:
Point from lines:
2D Ideal points ( )T0,, 21 xx
2D line at infinity ( )T1,0,0=l∞
0=XπT
0π
X
X
X
3
2
1
T
T
T
0X
π
π
π
3
2
1
T
T
T3D plane-point coincidence relation:
Point from planes: Plane from points:
Line from points:
3D line representation:
(as two planes or two points) 2x2
PA B 0
Q
T
T
3D Ideal points 1 2 3, , ,0X X XT
3D plane at infinity 0,0,0,1 T
Conics and quadrics
4
l=Cx
l
xC
1-* CC
0=QXXT 0=πQπ *T
Conics
Quadrics
2D projective transformations
A projectivity is an invertible mapping h from P2 to itself such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Definition:
A mapping h:P2P2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P2 reprented by a vector x it is true that h(x)=Hx
Theorem:
Definition: Projective transformation
3
2
1
333231
232221
131211
3
2
1
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
x=x' Hor
8DOF
projectivity=collineation=projective transformation=homography
Transformation of 2D points, lines and conics
Transformation for lines
l=l' -TH
Transformation for conics-1-TCHHC ='
Transformation for dual conicsTHHCC ** ='
x=x' HFor a point transformation
Fixed points and lines
e=e λH (eigenvectors H =fixed points)
l lH-T (eigenvectors H-T =fixed lines)
(1=2 pointwise fixed line)
Hierarchy of 2D transformations
1002221
1211
y
x
taa
taa
1002221
1211
y
x
tsrsr
tsrsr
333231
232221
131211
hhh
hhh
hhh
1002221
1211
y
x
trr
trr
Projective8dof
Affine6dof
Similarity4dof
Euclidean3dof
Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio
Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l∞
Ratios of lengths, angles.The circular points I,J
lengths, areas.
invariantstransformed squares
The line at infinity
00
l l 0 lt 1
1A
AH
A
TT
T T
The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity
Note: not fixed pointwise
Affine properties from images
projection rectification
APA
lll
HH
321
010
001
0≠,l 3321∞ llll T
Affine rectificationv1 v2
l1
l2 l4
l3
l∞
21∞ vvl
211 llv 432 l×l=v
The circular points
0
1
I i
0
1
J i
I
0
1
0
1
100
cossin
sincos
II
iseitss
tssi
y
x
S
H
The circular points I, J are fixed points under the projective transformation H iff H is a similarity
The circular points
“circular points”
0=++++ 233231
22
21 fxxexxdxxx 0=+ 2
221 xx
l∞
T
T
0,-,1J
0,,1I
i
i
( ) ( )TT 0,1,0+0,0,1=I i
Algebraically, encodes orthogonal directions
03 x
Conic dual to the circular points
TT JIIJ*∞ C
000
010
001*∞C
TSS HCHC *
∞*∞
The dual conic is fixed conic under the projective transformation H iff H is a similarity
*∞C
Note: has 4DOF
l∞ is the nullvector
*∞C
l∞
I
J
Angles
( )( )22
21
22
21
2211
++
+=cos
mmll
mlmlθ
( )T321 ,,=l lll ( )T
321 ,,=m mmmEuclidean:
Projective:
(orthogonal)
Transformation of 3D points, planes and quadrics
Transformation for lines
( )l=l' -TH
Transformation for quadrics
( )-1-TCHHC ='
Transformation for dual quadrics
( )THHCC ** ='
( )x=x' H
For a point transformation
X=X' H
π=π' -TH
-1-TQHH=Q'
THHQ=Q' **
(cfr. 2D equivalent)
Hierarchy of 3D transformations
vTv
tAProjective15dof
Affine12dof
Similarity7dof
Euclidean6dof
Intersection and tangency
Parallellism of planes,Volume ratios, centroids,The plane at infinity π∞
Angles, ratios of length
The absolute conic Ω∞
Volume
10
tAT
10
tRT
s
10
tRT
The plane at infinity
0
00π π π
0- t 1
1
A
AH
A
TT
T T
The plane at infinity π is a fixed plane under a projective transformation H iff H is an affinity
1. canonical position2. contains directions 3. two planes are parallel line of intersection in π∞
4. line // line (or plane) point of intersection in π∞
( )T1,0,0,0=π∞
( )T0,,,=D 321 XXX
The absolute conic
The absolute conic Ω∞ is a fixed conic under the projective transformation H iff H is a similarity
04
23
22
21
X
XXX
The absolute conic Ω∞ is a (point) conic on π.
In a metric frame:
T321321 ,,I,, XXXXXXor conic for directions:(with no real points)
1. Ω∞ is only fixed as a set2. Circle intersect Ω∞ in two circular points3. Spheres intersect π∞ in Ω∞
The absolute dual quadric
00
0I*∞ T
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. 8 dof2. plane at infinity π∞ is the nullvector of Ω∞
3. Angles:
( )( )2
*∞21
*∞1
2*∞1
πΩππΩπ
πΩπ=cos
TT
T
θ
Camera model
Relation between pixels and rays in space
?
Pinhole camera
Gemma Frisius, 1544
23
Distant objects appear smaller
24
Parallel lines meet
• vanishing point
25
Vanishing points
VPL VPRH
VP1VP2
VP3
To different directions correspond different vanishing points
26
Geometric properties of projection
• Points go to points• Lines go to lines• Planes go to whole image
or half-plane• Polygons go to polygons
• Degenerate cases:– line through focal point yields point– plane through focal point yields line
Pinhole camera model
TT ZfYZfXZYX )/,/(),,(
101
0
0
1
Z
Y
X
f
f
Z
fY
fX
Z
Y
X
linear projection in homogeneous coordinates!
Pinhole camera model
101
0
0
Z
Y
X
f
f
Z
fY
fX
101
01
01
1Z
Y
X
f
f
Z
fY
fX
PXx
0|I)1,,(diagP ff
Principal point offset
Tyx
T pZfYpZfXZYX )+/,+/(),,(
principal pointT
yx pp ),(
Principal point offset
camX0|IKx
1y
x
pf
pf
K calibration matrix
Camera rotation and translation
( )C~
-X~
R=X~
cam
X10
RC-R
1
10
C~
R-RXcam
Z
Y
X
camX0|IKx XC~
-|IKRx
t|RKP C~
R-t PX=x
~
CCD camera
1yy
xx
p
p
K
11y
x
y
x
pf
pf
m
m
K
General projective camera
1yx
xx
p
ps
K
1yx
xx
p
p
K
C~
|IKRP
non-singular
11 dof (5+3+3)
t|RKP
intrinsic camera parametersextrinsic camera parameters
Radial distortion
• Due to spherical lenses (cheap)• Model:
R
2 2 2 2 21 2( , ) (1 ( ) ( ) ...)
xx y K x y K x y
y
R
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore
Camera model
Relation between pixels and rays in space
?
Projector model
Relation between pixels and rays in space
(dual of camera)
(main geometric difference is vertical principal point offset to reduce keystone effect)
?
Meydenbauer camera
vertical lens shiftto allow direct ortho-photographs
Affine cameras
Action of projective camera on points and lines
forward projection of line
μbaμPBPAμB)P(AμX
back-projection of line
lPT
PXlX TT PX x0;xlT
PXx projection of point
Action of projective camera on conics and quadricsback-projection to cone
CPPQ Tco 0CPXPXCxx TTT
PXx
projection of quadric
TPPQC ** 0lPPQlQ T*T*T
lPT
ii xX ? P
Resectioning
Direct Linear Transform (DLT)
ii PXx ii PXx
rank-2 matrix
Direct Linear Transform (DLT)
Minimal solution
Over-determined solution
5½ correspondences needed (say 6)
P has 11 dof, 2 independent eq./points
n 6 points
Apminimize subject to constraint
1p
use SVD
Degenerate configurations
(i) Points lie on plane or single line passing through projection center
(ii) Camera and points on a twisted cubic
Scale data to values of order 1
1. move center of mass to origin2. scale to yield order 1 values
Data normalization
D3
D2
Line correspondences
Extend DLT to lines
ilPT
ii 1TPXl
(back-project line)
ii 2TPXl (2 independent eq.)
Geometric error
Gold Standard algorithm
ObjectiveGiven n≥6 2D to 3D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P
Algorithm
(i) Linear solution:
(a) Normalization:
(b) DLT
(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:
(iii) Denormalization:
ii UXX~ ii Txx~
UP~
TP -1
~ ~~
Calibration example
(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners
typically precision <1/10
(H&Z rule of thumb: 5n constraints for n unknowns)
Errors in the world
Errors in the image and in the world
ii XPx
iX
Errors in the image
iPXx̂
i
(standard case)
Restricted camera estimation
Minimize geometric error impose constraint through parametrization
Find best fit that satisfies• skew s is zero• pixels are square • principal point is known• complete camera matrix K is known
Minimize algebraic error assume map from param q P=K[R|-RC], i.e. p=g(q)minimize ||Ag(q)||
Restricted camera estimation
Initialization • Use general DLT• Clamp values to desired values, e.g. s=0, x= y
Note: can sometimes cause big jump in error
Alternative initialization• Use general DLT• Impose soft constraints
• gradually increase weights
Image of absolute conic
A simple calibration device
(i) compute H for each square (corners (0,0),(1,0),(0,1),(1,1))
(ii) compute the imaged circular points H(1,±i,0)T
(iii) fit a conic to 6 circular points(iv) compute K from through cholesky factorization
(≈ Zhang’s calibration method)
Some typical calibration algorithmsTsai calibration
Zhangs calibration
http://research.microsoft.com/~zhang/calib/
Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.
Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.
http://www.vision.caltech.edu/bouguetj/calib_doc/
81 from Szeliski’s book
82
Next week:Image features