lecture 9: epipolargeometry - artificial...
Post on 30-May-2020
2 Views
Preview:
TRANSCRIPT
Lecture 9 -Fei-Fei Li
Lecture 9:
Epipolar Geometry
Professor Fei-Fei Li
Stanford Vision Lab
24-Oct-111
Lecture 9 -Fei-Fei Li
What we will learn today?
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F (Problem Set 2 (Q2))
• Rectification
24-Oct-112
Reading:
[HZ] Chapters: 4, 9, 11
[FP] Chapters: 10
Lecture 9 -Fei-Fei Li 24-Oct-113
Recovering structure from a single view
Pinhole perspective projection
C
Ow
Pp
Calibration rig
Scene
Camera K
Why is it so difficult?
Intrinsic ambiguity of the mapping from 3D to image (2D)
Lecture 9 -Fei-Fei Li 24-Oct-114
Recovering structure from a single view
Pinhole perspective projection
Intrinsic ambiguity of the mapping from 3D to image (2D)
Co
urt
esy
sli
de
S.
Laze
bn
ik
Lecture 9 -Fei-Fei Li 24-Oct-115
Two eyes help!
Lecture 9 -Fei-Fei Li 24-Oct-116
Two eyes help!
O2O1
x2
x1
?
This is called triangulation
K =knownK =known
R, T
Lecture 9 -Fei-Fei Li 24-Oct-117
Triangulation
• Find X that minimizes ),(),( 222
112 XPxdXPxd +
O1O2
x1
x2
X
Lecture 9 -Fei-Fei Li 24-Oct-118
Stereo-view geometry
• Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?
• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.
• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.
This lecture (#9)
Lecture 9 -Fei-Fei Li 24-Oct-119
Stereo-view geometry
• Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?
• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.
• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.
Next lecture (#10)
Lecture 9 -Fei-Fei Li
What we will learn today?
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F
• Rectification
24-Oct-1110
Reading:
[HZ] Chapters: 4, 9, 11
[FP] Chapters: 10
Lecture 9 -Fei-Fei Li 24-Oct-1111
Epipolar geometry
• Epipolar Plane • Epipoles e1, e2
• Epipolar Lines
• Baseline
O1O2
p2
P
p1
e1 e2
= intersections of baseline with image planes
= projections of the other camera center
= vanishing points of camera motion direction
Lecture 9 -Fei-Fei Li 24-Oct-1112
Example: Converging image planes
Lecture 9 -Fei-Fei Li 24-Oct-1113
Example: Parallel image planes
O1O2
P
e2
p1 p2
e2
• Baseline intersects the image plane at infinity
• Epipoles are at infinity
• Epipolar lines are parallel to x axis
Lecture 9 -Fei-Fei Li 24-Oct-1114
Example: Parallel image planes
Lecture 9 -Fei-Fei Li 24-Oct-1115
Example: Forward translation
• The epipoles have same position in both images
• Epipole called FOE (focus of expansion)
O2
e1
e2
O2
Lecture 9 -Fei-Fei Li 24-Oct-1116
Epipolar Constraint
- Two views of the same object
- Suppose I know the camera positions and camera matrices
- Given a point on left image, how can I find the corresponding point on right image?
Lecture 9 -Fei-Fei Li 24-Oct-1117
Epipolar Constraint
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
Lecture 9 -Fei-Fei Li 24-Oct-1118
Epipolar Constraint
=→1
v
u
PMp
OO’
pp’
P
R, T
′′
=′→1
v
u
PMp
[ ]0IKM = [ ]TRK'M =
Lecture 9 -Fei-Fei Li 24-Oct-1119
Epipolar Constraint
[ ]0IKM =
OO’
pp’
P
R, T
[ ]TRK'M =K1 and K2 are known
(calibrated cameras)
[ ]0IM = [ ]TRM ='
Lecture 9 -Fei-Fei Li 24-Oct-1120
Epipolar Constraint
OO’
pp’
P
R, T
[ ] 0)pR(TpT =′×⋅Perpendicular to epipolar plane
)( pRT ′×
Lecture 9 -Fei-Fei Li 24-Oct-1121
Cross product as matrix multiplication
baba ][
0
0
0
×=
−−
−=×
z
y
x
xy
xz
yz
b
b
b
aa
aa
aa
“skew symmetric matrix”
Lecture 9 -Fei-Fei Li 24-Oct-1122
Triangulation
OO’
pp’
P
R, T
[ ] 0)pR(TpT =′×⋅ [ ] 0pRTpT =′⋅⋅→ ×
E = essential matrix(Longuet-Higgins, 1981)
Lecture 9 -Fei-Fei Li 24-Oct-1123
Triangulation
• E p2 is the epipolar line associated with p2 (l1 = E p2)
• ET p1 is the epipolar line associated with p1 (l2 = ET p1)
• E is singular (rank two)
• E e2 = 0 and ET e1 = 0
• E is 3x3 matrix; 5 DOF
O1O2
p2
P
p1
e1e2
l1 l2
Lecture 9 -Fei-Fei Li 24-Oct-1124
Triangulation
PMP→
=→
v
up [ ]0IKM =
unknown
OO’
pp’
P
Lecture 9 -Fei-Fei Li 24-Oct-1125
Triangulation
OO’
pp’
P
[ ] 0pRTpT =′⋅⋅ ×
pKp 1−→The image part with relationship ID rId8 was not found in the file.
[ ] 0pKRT)pK( 1T1 =′′⋅⋅→ −×
−
[ ] 0pKRTKp 1TT =′′⋅⋅ −×
− 0pFpT =′→
Lecture 9 -Fei-Fei Li 24-Oct-1126
Triangulation
OO’
pp’
P
0pFpT =′
F = Fundamental Matrix(Faugeras and Luong, 1992)
Lecture 9 -Fei-Fei Li 24-Oct-1127
Triangulation
O1O2
p2
P
p1
e1e2
• F p2 is the epipolar line associated with p2 (l1 = F p2)
• FT p1 is the epipolar line associated with p1 (l2 = FT p1)
• F is singular (rank two)
• F e2 = 0 and FT e1 = 0
• F is 3x3 matrix; 7 DOF
Lecture 9 -Fei-Fei Li 24-Oct-1128
Why is F useful?
- Suppose F is known
- No additional information about the scene and camera is given
- Given a point on left image, how can I find the corresponding point on right image?
l’ = FT pp
Lecture 9 -Fei-Fei Li 24-Oct-1129
Why is F useful?
• F captures information about the epipolar geometry of
2 views + camera parameters
• MORE IMPORTANTLY: F gives constraints on how the
scene changes under view point transformation
(without reconstructing the scene!)
• Powerful tool in:
• 3D reconstruction
• Multi-view object/scene matching
Lecture 9 -Fei-Fei Li
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F (Problem Set 2 (Q2))
• Rectification
24-Oct-1130
What we will learn today?
Reading:
[HZ] Chapters: 4, 9, 11
[FP] Chapters: 10
Lecture 9 -Fei-Fei Li 24-Oct-1131
Estimating F
OO’
pp’
PThe Eight-Point Algorithm
′′
=′→1
v
u
pP
=→1
v
u
pP 0pFpT =′
(Hartley, 1995)
(Longuet-Higgins, 1981)
Lecture 9 -Fei-Fei Li 24-Oct-1132
Estimating F
0pFpT =′
Let’s take 8 corresponding points
Lecture 9 -Fei-Fei Li 24-Oct-1133
Estimating F
Lecture 9 -Fei-Fei Li 24-Oct-1134
Estimating F
Lsq. solution by SVD!
• Rank 8 A non-zero solution exists (unique)
• Homogeneous system
• If N>8 F̂
f
1=f
W 0=f
W
Lecture 9 -Fei-Fei Li 24-Oct-1135
Estimating F
0pF̂pT =′The estimated F may have full rank (det(F) ≠0)
(F should have rank=2 instead)
0F̂F =−Find F that minimizes
Subject to det(F)=0
Frobenius norm (*)
SVD (again!) can be used to solve this problem
^ ^
(*) Sqrt root of the sum pf squares of all entries
Lecture 9 -Fei-Fei Li 24-Oct-1136
ExampleD
ata
cou
rtes
y o
f R
. Mo
hr
and
B. B
ou
fam
a.
Lecture 9 -Fei-Fei Li 24-Oct-1137
Example
Mean errors:
10.0pixel
9.1pixel
Lecture 9 -Fei-Fei Li 24-Oct-1138
Normalization
Is the accuracy in estimating F
function of the ref. system in
the image plane?
iii pTq = iii pTq ′′=′
E.g. under similarity transformation
(T = scale + translation):
Does the accuracy in estimating F change if a
transformation T is applied?
Lecture 9 -Fei-Fei Li 24-Oct-1139
Normalization
Why?
•The accuracy in estimating F does
change if a transformation T is applied
•There exists a T for which accuracy is
maximized
Lecture 9 -Fei-Fei Li 24-Oct-1140
Normalization
- SVD enforces Rank(W)=8
- Recall the structure of W:Highly un-balance
(not well conditioned)
-Values of W must have similar magnitude
F1=f
,0=fW Lsq solution
by SVD
More details HZ pag 108
Lecture 9 -Fei-Fei Li 24-Oct-1141
Normalization
IDEA: Transform image coordinate
system (T = translation + scaling) such
that:
• Origin = centroid of image points
• Mean square distance of the data points
from origin is 2 pixels
iii pTq = iii pTq ′′=′ (normalization)
Lecture 9 -Fei-Fei Li 24-Oct-1142
The Normalized Eight-Point Algorithm
1. Normalize coordinates:
2. Use the eight-point algorithm to compute F’q from the
points q and q’ .
3. Enforce the rank-2 constraint.
4. De-normalize Fq:
i i
iii pTq = iii pTq ′′=′
0qFq qT =′
qF→
TFTF qT′=
0)Fdet( q =
0. Compute Ti and Ti’
Lecture 9 -Fei-Fei Li 24-Oct-1143
Wit
h
tra
nsf
orm
ati
on
Wit
ho
ut
tra
nsf
orm
ati
on
Mean
errors:
10.0pixel
9.1pixel
Mean
errors:
1.0pixel
0.9pixel
Example
Lecture 9 -Fei-Fei Li
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F
• Rectification
24-Oct-1144
What we will learn today?
Reading:
[HZ] Chapters: 4, 9, 11
[FP] Chapters: 10
Lecture 9 -Fei-Fei Li 24-Oct-1145
Rectification
p’
P
p
O O’
• Make two camera images “parallel”
• Correspondence problem becomes easier
Lecture 9 -Fei-Fei Li 24-Oct-1146
Rectification
O O
P
e2
p p’e2
K Kx
y
z
u
v
u
v
• Parallel epipolar lines
• Epipoles at infinity
• v = v’
Let’s see why….
Lecture 9 -Fei-Fei Li 24-Oct-1147
Rectification
K1=K2 = known
x parallel to O1O2
RtE ][ ×=
O O
P
e2
p p’e2
K Kx
y
z
u
v
u
v
Lecture 9 -Fei-Fei Li 24-Oct-1148
baba ][
0
0
0
×=
−−
−=×
z
y
x
xy
xz
yz
b
b
b
aa
aa
aa
Cross product as matrix multiplication
Lecture 9 -Fei-Fei Li 24-Oct-1149
Rectification
K1=K2 = known
x parallel to O1O2
−== ×
00
00
000
][
T
TRtE
O O
P
e2
p p’e2
K Kx
y
z
u
v
u
v
→ v = v’?
Lecture 9 -Fei-Fei Li 24-Oct-1150
Rectification
( ) ( ) vTTv
vT
Tvuv
u
T
Tvu ′==
′−=
′′
− 0
0
10
100
00
000
1
O O
P
e2
p p’e2
K Kx
y
z
u
v
u
v
0pEpT =′
→ v = v’
Lecture 9 -Fei-Fei Li 24-Oct-1151
Rectification
p’p
OO’
H
GOAL of rectification : Estimate a perspective transformation H that makes images
parallel Impose v’=v
• This leaves degrees of freedom for determining H
• If not appropriate H is chosen, severe projective distortions on image take place
• We impose a number of restriction while computing H [HZ] Chapters: 11 (sec. 11.12)
Lecture 9 -Fei-Fei Li 24-Oct-1152
Rectification
H
Courtesy figure S. Lazebnik
Lecture 9 -Fei-Fei Li 24-Oct-1153
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
Lecture 9 -Fei-Fei Li 24-Oct-1154
Application: view morphing
If rectification is not applied, the morphing procedure does
not generate geometrically correct interpolations
Lecture 9 -Fei-Fei Li 24-Oct-1155
Application: view morphing
Lecture 9 -Fei-Fei Li 24-Oct-1156
Application: view morphing
Lecture 9 -Fei-Fei Li 24-Oct-1157
Application: view morphing
Lecture 9 -Fei-Fei Li 24-Oct-1158
Application: view morphing
Lecture 9 -Fei-Fei Li 24-Oct-1159
The Fundamental Matrix Songhttp://danielwedge.com/fmatrix/
Lecture 9 -Fei-Fei Li
• Why is stereo useful?
• Epipolar constraints
• Essential and fundamental matrix
• Estimating F (Problem Set 2 (Q2))
• Rectification
24-Oct-1160
What we have learned today?
Reading:
[HZ] Chapters: 4, 9, 11
[FP] Chapters: 10
Lecture 9 -Fei-Fei Li
Supplementary materials
24-Oct-1161
Lecture 9 -Fei-Fei Li
Making image planes parallel
T21
T ]1ee[TRKe == TKe ′=′
Making image planes parallel
p’
P
p
e e’K K’
R,T
[ ] PTRKP ′→[ ] P0IKP→
O O’
0. Compute epipoles
24-Oct-1162
Lecture 9 -Fei-Fei Li
Making image planes parallel
1. Map e to the x-axis at location [1,0,1]T (normalization)
[ ]T101
→= T21 ]1ee[e
p’
P
p
e’
O O’
HHTRH =1
e
24-Oct-1163
Lecture 9 -Fei-Fei Li
Making image planes parallel
2. Send epipole to infinity:
p’p
[ ]T101e=
−=
101
010
001
H2
→
P
[ ]T001
e
O O’
Minimizes the distortion in a
neighborhood (approximates id. mapping)
24-Oct-1164
Lecture 9 -Fei-Fei Li
Making image planes parallel
4. Align epipolar lines
p’p
H = H2 H1
P
e
O O’
e’
3. Define: H = H2 H1
24-Oct-1165
Lecture 9 -Fei-Fei Li
Projective transformation of a line (in 2D)
lHl T−→
=
bv
tAH
24-Oct-1166
Lecture 9 -Fei-Fei Li
Making image planes parallel
p’p
H = H2 H1
P
e
O O’
e’
4. Align epipolar lines
3. Define: H = H2 H1
lHlH TT −− =′′These are called matched pair of transformation
l l′
[HZ] Chapters: 11 (sec. 11.12)
24-Oct-1167
top related