![Page 1: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/1.jpg)
Peripheral drift illusion
![Page 2: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/2.jpg)
![Page 3: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/3.jpg)
Multiple views
Hartley and Zisserman
Lowestereo visionstructure from motionoptical flow
![Page 4: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/4.jpg)
Why multiple views?• Structure and depth are inherently ambiguous from
single views.
Images from Lana Lazebnik
![Page 5: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/5.jpg)
Why multiple views?• Structure and depth are inherently ambiguous from
single views.
Optical center
P1P2
P1’=P2’
![Page 6: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/6.jpg)
Geometry for a simple stereo system
• First, assuming parallel optical axes, known camera parameters (i.e., calibrated cameras):
![Page 7: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/7.jpg)
baseline
optical center (left)
optical center (right)
Focal length
World point
image point (left)
image point (right)
Depth of p
![Page 8: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/8.jpg)
• Assume parallel optical axes, known camera parameters (i.e., calibrated cameras). What is expression for Z?
Similar triangles (pl, P, pr) and (Ol, P, Or):
Geometry for a simple stereo system
Z
T
fZ
xxT rl
lr xx
TfZ
disparity
![Page 9: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/9.jpg)
Depth from disparity
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y), y)
So if we could find the corresponding points in two images, we could estimate relative depth…
![Page 10: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/10.jpg)
What did we need to know?
• Correspondence for every pixel. Sort of like project 2, but project 2 is “sparse” and we need “dense” correspondence.
• Calibration for the cameras.
![Page 11: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/11.jpg)
Where do we need to search?
![Page 12: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/12.jpg)
How do we calibrate a camera?312.747 309.140 30.086305.796 311.649 30.356307.694 312.358 30.418310.149 307.186 29.298311.937 310.105 29.216311.202 307.572 30.682307.106 306.876 28.660309.317 312.490 30.230307.435 310.151 29.318308.253 306.300 28.881306.650 309.301 28.905308.069 306.831 29.189309.671 308.834 29.029308.255 309.955 29.267307.546 308.613 28.963311.036 309.206 28.913307.518 308.175 29.069309.950 311.262 29.990312.160 310.772 29.080311.988 312.709 30.514
880 214 43 203270 197886 347745 302943 128476 590419 214317 335783 521235 427665 429655 362427 333412 415746 351434 415525 234716 308602 187
![Page 13: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/13.jpg)
World vs Camera coordinates
![Page 14: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/14.jpg)
Image formation
Let’s design a camera– Idea 1: put a piece of film in front of an object– Do we get a reasonable image?
Slide source: Seitz
![Page 15: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/15.jpg)
Pinhole camera
Idea 2: add a barrier to block off most of the rays– This reduces blurring– The opening known as the aperture
Slide source: Seitz
![Page 16: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/16.jpg)
Projection: world coordinatesimage coordinates
Camera Center (tx, ty, tz)
Z
Y
X
P.
.
. f Z
Y
v
up
.Optical Center (u0, v0)
v
u
X
![Page 17: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/17.jpg)
Homogeneous coordinates
Conversion
Converting to homogeneous coordinates
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
![Page 18: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/18.jpg)
World vs Camera coordinates
![Page 19: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/19.jpg)
Slide Credit: Saverese
Projection matrix
XtRKx x: Image Coordinates: (u,v,1)K: Intrinsic Matrix (3x3)R: Rotation (3x3) t: Translation (3x1)X: World Coordinates: (X,Y,Z,1)
Ow
iw
kw
jwR,T
![Page 20: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/20.jpg)
X0IKx
10100
000
000
1z
y
x
f
f
v
u
w
K
Slide Credit: Saverese
Projection matrix
Intrinsic Assumptions• Unit aspect ratio• Optical center at (0,0)• No skew
Extrinsic Assumptions• No rotation• Camera at (0,0,0)
![Page 21: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/21.jpg)
Remove assumption: known optical center
X0IKx
10100
00
00
10
0
z
y
x
vf
uf
v
u
w
Intrinsic Assumptions• Unit aspect ratio• No skew
Extrinsic Assumptions• No rotation• Camera at (0,0,0)
![Page 22: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/22.jpg)
Remove assumption: square pixels
X0IKx
10100
00
00
10
0
z
y
x
v
u
v
u
w
Intrinsic Assumptions• No skew
Extrinsic Assumptions• No rotation• Camera at (0,0,0)
![Page 23: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/23.jpg)
Remove assumption: non-skewed pixels
X0IKx
10100
00
0
10
0
z
y
x
v
us
v
u
w
Intrinsic Assumptions Extrinsic Assumptions• No rotation• Camera at (0,0,0)
Note: different books use different notation for parameters
![Page 24: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/24.jpg)
Oriented and Translated Camera
Ow
iw
kw
jw
t
R
![Page 25: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/25.jpg)
Allow camera translation
XtIKx
1100
010
001
100
0
0
10
0
z
y
x
t
t
t
v
u
v
u
w
z
y
x
Intrinsic Assumptions Extrinsic Assumptions• No rotation
![Page 26: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/26.jpg)
3D Rotation of Points
Rotation around the coordinate axes, counter-clockwise:
100
0cossin
0sincos
)(
cos0sin
010
sin0cos
)(
cossin0
sincos0
001
)(
z
y
x
R
R
R
p
p’
g
y
z
Slide Credit: Saverese
![Page 27: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/27.jpg)
Allow camera rotation
XtRKx
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
v
us
v
u
w
z
y
x
![Page 28: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/28.jpg)
Degrees of freedom
XtRKx
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
v
us
v
u
w
z
y
x
5 6
![Page 29: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/29.jpg)
Beyond Pinholes: Radial Distortion• Common in wide-angle lenses or for
special applications (e.g., security)• Creates non-linear terms in
projection• Usually handled by through solving
for non-linear terms and then correcting image
Image from Martin Habbecke
Corrected Barrel Distortion
![Page 30: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/30.jpg)
How to calibrate the camera?
1****
****
****
Z
Y
X
s
sv
su
XtRKx
![Page 31: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/31.jpg)
Calibrating the Camera
Use an scene with known geometry– Correspond image points to 3d points– Get least squares solution (or non-linear solution)
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
su
![Page 32: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/32.jpg)
How do we calibrate a camera?
312.747 309.140 30.086305.796 311.649 30.356307.694 312.358 30.418310.149 307.186 29.298311.937 310.105 29.216311.202 307.572 30.682307.106 306.876 28.660309.317 312.490 30.230307.435 310.151 29.318308.253 306.300 28.881306.650 309.301 28.905308.069 306.831 29.189309.671 308.834 29.029308.255 309.955 29.267307.546 308.613 28.963311.036 309.206 28.913307.518 308.175 29.069309.950 311.262 29.990312.160 310.772 29.080311.988 312.709 30.514
880 214 43 203270 197886 347745 302943 128476 590419 214317 335783 521235 427665 429655 362427 333412 415746 351434 415525 234716 308602 187
![Page 33: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/33.jpg)
0
0
0
0
10000
00001
10000
00001
34
33
32
31
24
23
22
21
14
13
12
11
1111111111
1111111111
m
m
m
m
m
m
m
m
m
m
m
m
vZvYvXvZYX
uZuYuXuZYX
vZvYvXvZYX
uZuYuXuZYX
nnnnnnnnnn
nnnnnnnnnn
Method 1 – homogeneous linear system
• Solve for m’s entries using linear least squaresAx=0 form
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
su
[U, S, V] = svd(A);M = V(:,end);M = reshape(M,[],3)';
![Page 34: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/34.jpg)
Method 2 – nonhomogeneous linear system
• Solve for m’s entries using linear least squares
n
n
nnnnnnnnn
nnnnnnnnn
v
u
v
u
m
m
m
m
m
m
m
m
m
m
m
ZvYvXvZYX
ZuYuXuZYX
ZvYvXvZYX
ZuYuXuZYX
1
1
33
32
31
24
23
22
21
14
13
12
11
111111111
111111111
10000
00001
10000
00001
Ax=b form
11333231
24232221
14131211
Z
Y
X
mmm
mmmm
mmmm
s
sv
su
M = A\Y;M = [M;1];M = reshape(M,[],3)';
![Page 35: Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f1d5503460f94c34810/html5/thumbnails/35.jpg)
Calibration with linear method• Advantages
– Easy to formulate and solve– Provides initialization for non-linear methods
• Disadvantages– Doesn’t directly give you camera parameters– Doesn’t model radial distortion– Can’t impose constraints, such as known focal length
• Non-linear methods are preferred– Define error as difference between projected points and measured points– Minimize error using Newton’s method or other non-linear optimization