![Page 1: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/1.jpg)
Lecture 5 - Silvio Savarese 21-Jan-15
Professor Silvio Savarese Computational Vision and Geometry Lab
Lecture 5Epipolar Geometry
![Page 2: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/2.jpg)
Lecture 5 - Silvio Savarese 21-Jan-15
• Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix • Estimating F • Examples
Lecture 5Epipolar Geometry
Reading: [HZ] Chapter: 4 “Estimation – 2D perspective transformations Chapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation” Chapter: 11 “Computation of the Fundamental Matrix F” [FP] Chapter: 7 “Stereopsis” Chapter: 8 “Structure from Motion”
![Page 3: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/3.jpg)
Pinhole perspective projectionRecovering structure from a single view
COw
Pp
Calibration rigScene
Camera K
From calibration rig
From points and lines at infinity + orthogonal lines and planes → structure of the scene, K
→ location/pose of the rig, K
Knowledge about scene (point correspondences, geometry of lines & planes, etc…
![Page 4: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/4.jpg)
Pinhole perspective projectionRecovering structure from a single view
COw
Pp
Calibration rigScene
Camera K
Why is it so difficult?Intrinsic ambiguity of the mapping from 3D to image (2D)
![Page 5: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/5.jpg)
Recovering structure from a single view
Intrinsic ambiguity of the mapping from 3D to image (2D)
Courtesy slide S. Lazebnik
![Page 6: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/6.jpg)
Two eyes help!
![Page 7: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/7.jpg)
O1 O2
pp’
?
Two eyes help!
This is called triangulation
K2 =knownK1 =known
R, T
P= l × "l P
l 'l[Eq. 1]
![Page 8: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/8.jpg)
• Find P’ that minimizesd(p,M1P')+d(p',M2P')
O1 O2
pp’
P
Triangulation
[Eq. 2]
M1 P’M2 P’
P’
image 1 image 2
![Page 9: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/9.jpg)
Stereo-‐view geometry
• Correspondence: Given a point p in one image, how can I find the corresponding point p’ in another one?
• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.
• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.
![Page 10: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/10.jpg)
• Epipolar Plane • Epipoles e, e’
• Epipolar Lines• Baseline
Epipolar geometry
O1 O2
p’
P
p
e e’
= intersections of baseline with image planes = projections of the other camera center
![Page 11: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/11.jpg)
Example of epipolar lines
![Page 12: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/12.jpg)
O1 O2
P
e’p p’e
Example: Parallel image planes
• Baseline intersects the image plane at infinity• Epipoles are at infinity• Epipolar lines are parallel to x axis
![Page 13: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/13.jpg)
Example: Parallel Image Planes
e’ at
infinity
e at
infinity
![Page 14: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/14.jpg)
Example: Forward translation
• The epipoles have same position in both images• Epipole called FOE (focus of expansion)
O2
e
e’
O1
P
![Page 15: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/15.jpg)
- Two views of the same object - Suppose I know the camera positions and camera matrices- Given a point on left image, how can I find the corresponding point on right image?
Epipolar Constraint
p
![Page 16: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/16.jpg)
Epipolar geometry
O1 O2
P
p p’
Epipolar line 2
![Page 17: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/17.jpg)
Epipolar Constraint
![Page 18: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/18.jpg)
!!!
"
#
$$$
%
&
=→
1vu
PMP
O1 O2
pp’
P
R, T
Epipolar Constraint
!!!
"
#
$$$
%
&'
'
='→
1vu
PMP[Eq. 3] [Eq. 4]
M = K I 0!"
#$
M ' = K ' RT −RTT"#$
%&'
![Page 19: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/19.jpg)
M = K I 0!"
#$
O1 O2
pp’
P
R, T
Epipolar Constraint
K = K’ are known (canonical cameras)
[ ]0IM =
Kcanonical =1 0 00 1 00 0 1
!
"
###
$
%
&&&
M ' = K ' RT −RTT"#$
%&'
M ' = RT −RTT"#$
%&'[Eq. 5] [Eq. 6]
![Page 20: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/20.jpg)
T
O1
O2
T = O2 in the camera 1 reference system
R is the rotation matrix such that a vector p’ in the camera 2 is equal to R p’ in camera 1.
The cameras are related by R, T
p’
Camera 1
Camera 2
![Page 21: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/21.jpg)
O1 O2
pp’
P
R, T
[ ] 0)( =!×⋅→ pRTpT
Epipolar Constraint
is perpendicular to epipolar plane)( pRT !×
pR !=p’ in first camera reference system is
[Eq. 7]
![Page 22: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/22.jpg)
baba ][0
00
×=
"""
#
$
%%%
&
'
"""
#
$
%%%
&
'
−
−
−
=×
z
y
x
xy
xz
yz
bbb
aaaaaa
Cross product as matrix multiplication
![Page 23: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/23.jpg)
O1 O2
pp’
P
R, T
[ ] 0)pR(TpT =!×⋅ [ ] 0pRTpT =!⋅⋅→ ×
E = Essential matrix(Longuet-Higgins, 1981)
Epipolar Constraint
[Eq. 8] [Eq. 9]
![Page 24: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/24.jpg)
• l = E p’ is the epipolar line associated with p’ • l’ = ET p is the epipolar line associated with p • E e’ = 0 and ET e = 0 • E is 3x3 matrix; 5 DOF• E is singular (rank two)
Epipolar Constraint
O1 O2
p’
P
p
e e’l l’
pT ⋅ E p' = 0[Eq. 10]
![Page 25: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/25.jpg)
Epipolar Constraint
O1 O2
pp’
P
[ ]0IKM =
R, T
M ' = K ' RT −RTT"#$
%&'
pc = K−1 p !pc = K '
−1 !p[Eq. 11] [Eq. 12]
![Page 26: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/26.jpg)
O1 O2
pp’
P
Epipolar Constraint
pTc ⋅ T×[ ]⋅R #pc = 0 [ ] 0pKRT)pK( 1T1 =!!⋅⋅→ −×
−
[ ] 0pKRTKp 1TT =!!⋅⋅ −×
− 0pFpT =!→ [Eq. 13]
[Eq.9]
pc = K−1 p
!pc = K '−1 !p
![Page 27: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/27.jpg)
O1 O2
pp’
P
Epipolar Constraint
0pFpT =!
F = Fundamental Matrix(Faugeras and Luong, 1992)
[ ] 1−×
− #⋅⋅= KRTKF TEq. 13
Eq. 14
![Page 28: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/28.jpg)
Epipolar Constraint
O1 O2
p’
P
p
e e’
• l = F p’ is the epipolar line associated with p’ • l’= FT p is the epipolar line associated with p • F e’ = 0 and FT e = 0• F is 3x3 matrix; 7 DOF • F is singular (rank two)
pT ⋅ F p' = 0
l’l
![Page 29: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/29.jpg)
Why F is useful?
- Suppose F is known- No additional information about the scene and camera is given- Given a point on left image, how can I find the corresponding point on right image?
l’ = FT pp
![Page 30: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/30.jpg)
Why F is useful?
• F captures information about the epipolar geometry of 2 views + camera parameters
• MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!)
• Powerful tool in: • 3D reconstruction • Multi-‐view object/scene matching
![Page 31: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/31.jpg)
O1 O2
pp’
P
Estimating FThe Eight-Point Algorithm
!!!
"
#
$$$
%
&'
'
='→
1vu
pP!!!
"
#
$$$
%
&
=→
1vu
pP 0pFpT =!
(Hartley, 1995)(Longuet-Higgins, 1981)
![Page 32: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/32.jpg)
0pFpT =!
Estimating F
Let’s take 8 corresponding points
( )11 12 13
21 22 23
31 32 33
', ,1 ' 0
1
F F F uu v F F F v
F F F
! "! "# $# $ =# $# $# $# $% &% &
( )
11
12
13
21
22
23
31
32
33
', ', , ', ', , ', ',1 0
FFFF
uu uv u vu vv v u v FFFFF
! "# $# $# $# $# $# $ =# $# $# $# $# $# $# $% &
[Eq. 13]
[Eq. 14]
![Page 33: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/33.jpg)
Estimating F
![Page 34: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/34.jpg)
Estimating F
Lsq. solution by SVD!• Rank 8 A non-zero solution exists (unique)
• Homogeneous system
• If N>8 F̂
f
1=f
W 0=f
W
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '
u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v
11
12
13
21
22
23
317 7 7 7 7 7
328 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1' ' ' ' ' ' 1
FFFFFFF
u v v v u vFu u u v u v u v v v u vF
! "! "# $# $# $# $# $# $# $# $# $# $# $ =# $# $# $# $# $# $# $# $# $# $# $# $% &# $
% &
[Eqs. 15]
![Page 35: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/35.jpg)
0pF̂pT =!
and estimated F may have full rank (det(F) ≠0)
0F̂F =−Find F that minimizes
Subject to det(F)=0
But remember: fundamental matrix is Rank2
Frobenius norm (*)
SVD (again!) can be used to solve this problem
^ ^
(*) Sq. root of the sum of squares of all entries
satisfies:F̂
![Page 36: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/36.jpg)
0F̂F =−Find F that minimizes
Subject to det(F)=0Frobenius norm (*)
F =Us1 0 00 s2 00 0 0
!
"
####
$
%
&&&&
VT
Us1 0 00 s2 00 0 s3
!
"
####
$
%
&&&&
VT = SVD(F̂)
Where:
[HZ] pag 281, chapter 11, “Computation of F”
![Page 37: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/37.jpg)
Data courtesy of R. Mohr and B. Boufama.
![Page 38: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/38.jpg)
Mean errors:10.0pixel9.1pixel
![Page 39: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/39.jpg)
Problems with the 8-Point Algorithm
- Recall the structure of W: - do we see any potential (numerical) issue?
F1=f,0=fW Lsq solution
by SVD
![Page 40: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/40.jpg)
1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4
5 5 5 5 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 6 6 6 6
7 7 7 7 7 7
' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '
u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v
11
12
13
21
22
23
317 7 7 7 7 7
328 8 8 8 8 8 8 8 8 8 8 8
33
0
' ' ' ' 1' ' ' ' ' ' 1
FFFFFFF
u v v v u vFu u u v u v u v v v u vF
! "! "# $# $# $# $# $# $# $# $# $# $# $ =# $# $# $# $# $# $# $# $# $# $# $# $% &# $
% &
Problems with the 8-Point Algorithm
• Highly un-‐balanced (not well conditioned)• Values of W must have similar magnitude• This creates problems during the SVD decomposition
0=Wf
HZ pag 108
![Page 41: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/41.jpg)
Normalization
IDEA: Transform image coordinates such that the matrix W becomes better conditioned (pre-conditioning)
For each image, apply a following transformation T (translation and scaling) acting on image coordinates such that:
• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels
ii pTq = ii pTq !!=! (normalization)
![Page 42: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/42.jpg)
Coordinate system of the image before applying T
Coordinate system of the image after applying T
T2 pixels
Example of normalization
• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels
![Page 43: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/43.jpg)
1. Normalize coordinates in images 1 and 2:
2. Use the eight-point algorithm to compute from the corresponding points q and q’ .
1. Enforce the rank-2 constraint.
2. De-normalize Fq:
i i
ii pTq = ii pTq !!=!
0qFq qT =!
The Normalized Eight-Point Algorithm
qF→
'TFTF qT= 0)Fdet( q =
0. Compute T and T’ for image 1 and 2, respectively
F̂q
such that:
![Page 44: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/44.jpg)
With
tran
sfor
mat
ion
With
out t
rans
form
atio
n Mean errors:10.0pixel9.1pixel
Mean errors:1.0pixel0.9pixel
![Page 45: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/45.jpg)
Lecture 5 - Silvio Savarese 21-Jan-15
• Why is stereo useful? • Epipolar constraints • Essential and fundamental matrix • Estimating F • Examples
Lecture 5Epipolar Geometry
Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformations Chapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation” Chapter: 11 “Computation of the Fundamental Matrix F” [FP] Chapter: 7 “Stereopsis” Chapter: 8 “Structure from Motion”
![Page 46: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/46.jpg)
O1 O2
P
e’p p’
e
Example: Parallel image planes
E=?
K K
K1=K2 = knownHint :
x
y
z
x parallel to O1O2
R = I T = (T, 0, 0)
![Page 47: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/47.jpg)
!!!
"
#
$$$
%
&
−=!!!
"
#
$$$
%
&
−
−
−
=
0000
000
00
0
TT
TTTTTT
xy
xz
yz
IE
Essential matrix for parallel images
T = [ T 0 0 ]R = I
[ ] RTE ⋅= ×
[Eq. 20]
![Page 48: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/48.jpg)
O1
P
e’p p’e
Example: Parallel image planes
K K
x
y
z
horizontal!What are the directions of epipolar lines?
l = E p' =0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
uv1
"
#
$$$
%
&
'''=
0−TTv
"
#
$$$
%
&
'''
l l’
![Page 49: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/49.jpg)
Example: Parallel image planes
How are p and p’ related?
pT ⋅ E p' = 0
P
e’p p’
e
K Ky
z
xO1
![Page 50: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/50.jpg)
Example: Parallel image planes
( ) ( )T
0 0 0 0p E p 0 1 0 0 0 1 0
0 0 1
uu v T v u v T Tv Tv
T Tv
!" # $ % $ %& ' & '( )! ! != ⇒ − = ⇒ − = ⇒ =& ' & '( )& ' & '!( ), - . / . /
How are p and p’ related?
P
e’p p’
e
K Ky
z
O1x
⇒ v = "v
![Page 51: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/51.jpg)
Example: Parallel image planes
Rectification: making two images “parallel” Why it is useful? • Epipolar constraint → v = v’
• New views can be synthesized by linear interpolation
P
e’p p’
e
K Ky
z
O1x
![Page 52: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/52.jpg)
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
![Page 53: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/53.jpg)
Rectification
u
u
u
v
v
v
î2
î1
îS
I1
I2
Is
P
C2
Cs
C1
H1
H2
![Page 54: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/54.jpg)
![Page 55: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/55.jpg)
![Page 56: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/56.jpg)
Morphing without rectifying
![Page 57: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/57.jpg)
![Page 58: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/58.jpg)
![Page 59: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/59.jpg)
From its reflection!
![Page 60: lecture5 epipolar geometry - Silvio Savarese...Silvio Savarese Lecture 5 - 21-Jan-15 Professor’Silvio’Savarese’ Computational+Vision+and+Geometry+Lab Lecture’5 Epipolar’Geometry](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39c3de3a06344b51603ea4/html5/thumbnails/60.jpg)
http://danielwedge.com/fmatrix/
The Fundamental Matrix Song