![Page 1: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/1.jpg)
Last Lecture
(optical center)
origin
principal point
P
(X,Y,Z)
p
(x,y)
Z
Y
X
yf
xf
Z
fY
fX
y
x
100
0
0
~
10
0
x
y
![Page 2: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/2.jpg)
Camera calibration
![Page 3: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/3.jpg)
Camera calibration
• Estimate both intrinsic and extrinsic parameters
• Mainly, two categories:1. Using objects with known geometry as
reference 2. Self calibration (structure from motion)
11100
0
0
~
1Z
Y
X
lkji
hgfe
dcba
Z
Y
X
yf
xf
y
x
c
c
tR
![Page 4: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/4.jpg)
One app of camera pose application
• Virtual gaming– http://www.livestream.com/emtech/video?
clipId=pla_74103098-95fb-4704-99f0-d07339dc16a1
![Page 5: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/5.jpg)
Camera calibration approaches
• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
MXXtRKx ~
![Page 6: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/6.jpg)
Linear regression
![Page 7: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/7.jpg)
Linear regression
i
i
iiiiiiiii
iiiiiiiii
v
u
m
m
m
m
m
m
m
m
m
m
m
YvYvXvZYX
ZuYuXuZYX
22
21
20
13
12
11
10
03
02
01
00
10000
00001
![Page 8: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/8.jpg)
Linear regression
Solve for Projection Matrix M using least-square techniques
i
i
NNNNNNNNN
NNNNNNNNNv
u
m
m
m
m
m
m
m
m
m
m
m
YvYvXvZYX
ZuYuXuZYX
YvYvXvZYX
ZuYuXuZYX
22
21
20
13
12
11
10
03
02
01
00
111111111
111111111
10000
00001
10000
00001
![Page 9: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/9.jpg)
Normal equation (Geometric Interpretation)
Given an overdetermined system
bAx
0)(T bAxA
the normal equation is that which minimizes the sum of the square differences between left and right sides
2min bAx
bx T1T AAA
![Page 10: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/10.jpg)
Normal equation (Differential Interpretation)
2)( bAxx E
n
m
nmn
m
b
b
x
x
aa
aa
:
:
:
:
...
::
::
::
... 1
1
1
111
nxm, n equations, m variables
![Page 11: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/11.jpg)
Normal equation
bbxbAxbAAxAx
bbbAxAxbAxAx
bAxbAx
bAxbAx
bAxx
TTTTTTT
TTTTTT
TTT
T
E2
)(
bAAxAx
TT 22 E
Carl Friedrich Gauss
![Page 12: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/12.jpg)
Any issues with the method?
![Page 13: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/13.jpg)
Nonlinear optimization
• A probabilistic view of least square• Feature measurement equations
• Likelihood of M given {(ui,vi)}
![Page 14: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/14.jpg)
Optimal estimation
• Log likelihood of M given {(ui,vi)}
• It is a least square problem (but not necessarily linear least square)
• How do we minimize C?
2
222120
131211102
2
1 222120
030201002 1
1
1
1
iii
iiii
i
N
i iii
iiii
i ZmYmXm
mZmYmXmv
ZmYmXm
mZmYmXmu
![Page 15: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/15.jpg)
Nonlinear least square methods
![Page 16: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/16.jpg)
Least square fitting
number of data points
number of parameters
![Page 17: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/17.jpg)
Nonlinear least square fitting
![Page 18: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/18.jpg)
Function minimization
It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.
Least square is related to function minimization.
![Page 19: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/19.jpg)
Function minimization
![Page 20: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/20.jpg)
Quadratic functions
Approximate the function with a quadratic function within a small neighborhood
![Page 21: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/21.jpg)
Function minimization
![Page 22: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/22.jpg)
Computing gradient and Hessian
hxx
hhx
xx
hxxx
xhhx
xx
hxxx
xhhx
xx
hx
hhx
x
hxhx
m
i
iim
i
ii
m
ii
m
i
iiii
m
i
ii
m
ii
m
i
iiii
iii
m
i
iii
m
ii
fffff
ffff
fff
ffff
fff
fff
fF
1
TT
T
11
2
1
T
2
2T
T
11
2
1
T
2
2T
T2
1
2
2
2T
T
1
2
2
1)()(
2
1
)(2
1)()(
2
1
)()(2)(2
1
2
1)(
2
1
)(2
1)(
Gradient Hessian
![Page 23: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/23.jpg)
Computing gradient and Hessian
hxx
hhx
xx
hxxx
xhhx
xx
hxxx
xhhx
xx
hx
hhx
x
hxhx
m
i
iim
i
ii
m
ii
m
i
iiii
m
i
ii
m
ii
m
i
iiii
iii
m
i
iii
m
ii
fffff
ffff
fff
ffff
fff
fff
fF
1
TT
T
11
2
1
T
2
2T
T
11
2
1
T
2
2T
T2
1
2
2
2T
T
1
2
2
1)()(
2
1
)(2
1)()(
2
1
)()(2)(2
1
2
1)(
2
1
)(2
1)(
Gradient Hessian
![Page 24: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/24.jpg)
Computing gradient and Hessian
hxx
hhx
xx
hxxx
xhhx
xx
hxxx
xhhx
xx
hx
hhx
x
hxhx
m
i
iim
i
ii
m
ii
m
i
iiii
m
i
ii
m
ii
m
i
iiii
iii
m
i
iii
m
ii
fffff
ffff
fff
ffff
fff
fff
fF
1
TT
T
11
2
1
T
2
2T
T
11
2
1
T
2
2T
T2
1
2
2
2T
T
1
2
2
1)()(
2
1
)(2
1)()(
2
1
)()(2)(2
1
2
1)(
2
1
)(2
1)(
Gradient Hessian
![Page 25: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/25.jpg)
Computing gradient and Hessian
hxx
hhx
xx
hxxx
xhhx
xx
hxxx
xhhx
xx
hx
hhx
x
hxhx
m
i
iim
i
ii
m
ii
m
i
iiii
m
i
ii
m
ii
m
i
iiii
iii
m
i
iii
m
ii
fffff
ffff
fff
ffff
fff
fff
fF
1
TT
T
11
2
1
T
2
2T
T
11
2
1
T
2
2T
T2
1
2
2
2T
T
1
2
2
1)()(
2
1
)(2
1)()(
2
1
)()(2)(2
1
2
1)(
2
1
)(2
1)(
Gradient Hessian
![Page 26: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/26.jpg)
Computing gradient and Hessian
hxx
hhx
xx
hxxx
xhhx
xx
hxxx
xhhx
xx
hx
hhx
x
hxhx
m
i
iim
i
ii
m
ii
m
i
iiii
m
i
ii
m
ii
m
i
iiii
iii
m
i
iii
m
ii
fffff
ffff
fff
ffff
fff
fff
fF
1
TT
T
11
2
1
T
2
2T
T
11
2
1
T
2
2T
T2
1
2
2
2T
T
1
2
2
1)()(
2
1
)(2
1)()(
2
1
)()(2)(2
1
2
1)(
2
1
)(2
1)(
Gradient Hessian
![Page 27: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/27.jpg)
Searching for update h
Hhhhgxhx TT
1
2
2
1)(
2
1)(
m
iifF
Gradient Hessian
Idea 1: Steepest Descent
Hgg
gg
Hggggxhx
gh
T
T
T2
T
1
2
2)(
2
1)(
Let
m
iifF
![Page 28: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/28.jpg)
Steepest descent method
isocontour gradient
![Page 29: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/29.jpg)
Steepest descent method
It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.
![Page 30: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/30.jpg)
Searching for update h
Hhhhgxhx TT
1
2
2
1)(
2
1)(
m
iifF
Gradient Hessian
Idea 2: minimizing the quadric directly
gHh
Hhghxh
1
0)(Let
F
Converge faster but needs to solve the linear system
![Page 31: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/31.jpg)
Recap: Calibration
• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
MXXtRKx ~
Camera Model:
![Page 32: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/32.jpg)
Recap: Calibration
• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
MXXtRKx ~
Linear Approach:
![Page 33: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/33.jpg)
Recap: Calibration
• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
MXXtRKx ~
NonLinear Approach:
2
222120
13121110
2
1 222120
03020100
11
iii
iiii
N
i iii
iiii ZmYmXm
mZmYmXmv
ZmYmXm
mZmYmXmu
![Page 34: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/34.jpg)
Practical Issue
is hard to make and the 3D feature positions are difficult to measure!
![Page 35: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/35.jpg)
A popular calibration tool
![Page 36: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/36.jpg)
Multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage• Only requires a plane• Don’t have to know positions/orientations• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
– Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
![Page 37: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/37.jpg)
Step 1: data acquisition
![Page 38: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/38.jpg)
Step 2: specify corner order
![Page 39: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/39.jpg)
Step 3: corner extraction
![Page 40: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/40.jpg)
Step 3: corner extraction
![Page 41: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/41.jpg)
Step 4: minimize projection error
![Page 42: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/42.jpg)
Step 4: camera calibration
![Page 43: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/43.jpg)
Step 4: camera calibration
![Page 44: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/44.jpg)
Step 5: refinement
![Page 45: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/45.jpg)
Next
Image Mosaics and Panorama
• Today’s Readings– Szeliski and Shum paper
http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski/
Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html
2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html
![Page 46: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/46.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°
Slide from Brown & Lowe
![Page 47: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/47.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°
Slide from Brown & Lowe
![Page 48: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/48.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°
Slide from Brown & Lowe
![Page 49: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/49.jpg)
Mosaics: stitching images together
Creating virtual wide-angle camera
![Page 50: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/50.jpg)
Auto Stitch: the State of Art Method
• Demo
• Project 2 is a striped-down AutoStitch
![Page 51: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/51.jpg)
How to do it?
• Basic Procedure– Take a sequence of images from the same
position• Rotate the camera about its optical center
– Compute transformation between second image and first
– Transform the second image to overlap with the first
– Blend the two together to create a mosaic– If there are more images, repeat
![Page 52: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/52.jpg)
Geometric Interpretation of Mosaics
Image 1
Image 2
• If we capture all the 360º rays in different images, we can assemble them into a panorama.
• The basic operation is projecting an image from one plane to another
• The projective transformation is scene-INDEPENDENT
Optical Center
![Page 53: Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e885503460f94b8d33d/html5/thumbnails/53.jpg)
What is the transformation?
Translations are not enough to align the images
left on top right on top