eecs 442 computer vision multiple view geometry affine ...x o d d projective case affine case...
TRANSCRIPT
EECS 442 – Computer vision
Multiple view geometry
Affine structure from Motion
Reading: [HZ] Chapters: 6,14,18
[FP] Chapter: 12
Some slides of this lectures are courtesy of prof. J. Ponce,
prof FF Li, prof S. Lazebnik & prof. M. Hebert
- Affine structure from motion problem
- Algebraic methods
- Factorization methods
Applications Courtesy of Oxford Visual Geometry Group
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
•xij = Mi Xj , i = 1, … , m, j = 1, … , n
From the mxn correspondences xij, estimate:
•m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motion
structure
M1
M2
Mm
Structure from motion problem
Affine structure from motion (simpler problem)
Image World
Image
From the mxn correspondences xij, estimate:
•m projection matrices Mi (affine cameras)
•n 3D points Xj
Finite cameras
p
q
r
R
Q
P
O
XTRKx
M
??10
0100
0010
0001
TR
Question:
Finite cameras
p
q
r
R
Q
P
O
XTRKx
10
TR
0100
0010
0001
KM 3x3
Canonical perspective projection matrix
Affine homography
(in 3D)
Affine
Homography
(in 2D)
100
y0
xs
K oy
ox
Projective & Affine cameras
XTRKx
10
TR
0100
0010
0001
KM
100
y0
xs
K oy
ox
Projective case
Affine case
Scaling function of the distance (magnification)
myy
mxx
'
'
Weak perspective projection When the relative scene depth is small compared to its distance from the camera
Orthographic (affine) projection
y'y
x'x
When the camera is at a (roughly constant) distance from the scene
–Distance from center of projection
to image plane is infinite
Transformation in 2D
Affinities:
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
Projective & Affine cameras
XTRKx
100
yα0
xsα
oy
ox
K
10
TR
1000
0010
0001
KM
10
TR
0100
0010
0001
KM
100
y0
xs
K oy
ox
Projective case
Affine case
Parallel projection matrix
(points at infinity are mapped as points at infinity) Magnification (scaling term)
Affine cameras
XTRKx
100
00
00
y
x
K
10
TR
1000
0010
0001
KM
10
bA
1000
]affine44[
1000
0010
0001
]affine33[ 2232221
1131211
baaa
baaa
M
12
1
232221
131211 XbAXx EucM
b
b
Z
Y
X
aaa
aaa
y
x
[Homogeneous]
[non-homogeneous
image coordinates] bAMMEuc
;1
PEucM
P p
p’
;1
PbAPp M
v
u bAM
Affine cameras
M = camera matrix
To recap: from now on we define M as the camera matrix for the affine case
The Affine Structure-from-Motion Problem
Given m images of n fixed points Pj (=Xi) we can write
Problem: estimate the m 24 matrices Mi and
the n positions Pj from the mn correspondences pij .
2m n equations in 8m+3n unknowns
Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points)
- Factorization method
How many equations and how many unknown?
N of cameras N of points
Algebraic analysis (2-view case)
- Derive the fundamental matrix FA for the
affine case
- Compute FA
- Use FA to estimate projection matrices
- Use projection matrices to estimate 3D
points
1. Deriving the fundamental matrix FA
P p
p’
Homogeneous system
u
v
Dim= ? 4x4
The Affine Fundamental Matrix!
where
Deriving the fundamental matrix FA
Are the epipolar lines parallel or converging?
Affine Epipolar Geometry
Estimating FA
• From n correspondences, we obtain a linear system on
the unknown alpha, beta, etc…
• Measurements: u, u’, v, v’
0
1vuvu
1vuvu
nnnn
1111
f
• Computed by least square and by enforcing |f|=1
• SVD
P p
p’
Estimating projection matrices from FA
Affine ambiguity
PQQMPMp A
-1
A
Affine
P p
p’
Estimating projection matrices from FA
Estimating projection matrices from FA
Choose Q such
that…
'~
M
Where a,b,c,d can be expressed as function of the parameters of FA
See HZ page 348
A factorization method – Tomasi & Kanade algorithm
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization
method. IJCV, 9(2):137-154, November 1992.
• Centering the data
• Factorization
Centering: subtract the centroid of the image points
ji
n
1k
kji
n
1k
ikiiji
n
1k
ikijij
ˆn
1
n
1
n
1ˆ
XAXXA
bXAbXAxxx
A factorization method - Centering the data
Xk
xik xi ^
Centering: subtract the centroid of the image points
ji
n
1k
kji
n
1k
ikiiji
n
1k
ikijij
ˆn
1
n
1
n
1ˆ
XAXXA
bXAbXAxxx
A factorization method - Centering the data
Centering: subtract the centroid of the image points
n
k
kji
n
k
ikiiji
n
k
ikijij
n
nn
1
11
1
11ˆ
XXA
bXAbXAxxx
jiij XAx ˆ
A factorization method - Centering the data
Assume that the origin of the world coordinate system is at the centroid of
the 3D points
After centering, each normalized point xij is related to the 3D point Xi by
X x
x’
A factorization method - Centering the data
jiij XAx ˆ
Let’s create a 2m n data (measurement) matrix:
mnmm
n
n
xxx
xxx
xxx
D
ˆˆˆ
ˆˆˆ
ˆˆˆ
21
22221
11211
cameras
(2 m )
points (n )
A factorization method - factorization
Let’s create a 2m n data (measurement) matrix:
n
mmnmm
n
n
XXX
A
A
A
xxx
xxx
xxx
D
21
2
1
21
22221
11211
ˆˆˆ
ˆˆˆ
ˆˆˆ
cameras
(2 m × 3)
points (3 × n )
The measurement matrix D = M S has rank 3 (it’s a product of a 2mx3 matrix and 3xn matrix)
A factorization method - factorization
(2 m × n)
M
S
Factorizing the measurement matrix
Source: M. Hebert
Factorizing the measurement matrix
Singular value decomposition of D:
Source: M. Hebert
Factorizing the measurement matrix
Singular value decomposition of D:
Source: M. Hebert
Since rank (D)=3, there are only 3 non-zero singular values
Factorizing the measurement matrix
Obtaining a factorization from SVD:
M = Motion (cameras)
S = structure
What is the issue here?
D has rank>3 because of - measurement noise
- affine approximation
Factorizing the measurement matrix
Obtaining a factorization from SVD:
S = structure
D
D
M = motion
Affine ambiguity
The decomposition is not unique. We get the same D by
using any 3×3 matrix C and applying the
transformations M → MC, S →C-1S
We can enforce some Euclidean constraints to resolve
this ambiguity (more on next lecture!)
Algorithm summary
1. Given: m images and n features xij
2. For each image i, center the feature coordinates
3. Construct a 2m × n measurement matrix D:
• Column j contains the projection of point j in all views
• Row i contains one coordinate of the projections of all the n
points in image i
4. Factorize D:
• Compute SVD: D = U W VT
• Create U3 by taking the first 3 columns of U
• Create V3 by taking the first 3 columns of V
• Create W3 by taking the upper left 3 × 3 block of W
5. Create the motion and shape matrices:
• M = M = U3 and S = W3V3T (or U3W3
½ and S = W3½ V3
T )
6. Eliminate affine ambiguity
Reconstruction results
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography:
A factorization method. IJCV, 9(2):137-154, November 1992.
Next lecture
Multiple view geometry
Perspective structure from Motion