Download - Projective structure from motion
ComputerVision
Projective structure from motion
Marc PollefeysCOMP 256
Some slides and illustrations from J. Ponce, …
ComputerVision Last class: Affine camera
• The affine projection equations are
1
~
~
j
j
j
yi
xi
ij
ij
ZYX
P
Py
x
how to find the origin? or for that matter a 3D reference point?
affine projection preserves center of gravity
i
ijijij xxx~ i
ijijij yyy~
2
1
tt
~
~
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
ComputerVision Orthographic factorization
• The orthographic projection equations are
• where
njmijiij ,...,1,,...,1,Xx P
• All equations can be collected for all i and j
• where
n
mmnmm
n
n
X,...,X,X , ,
xxx
xxxxxx
212
1
21
22221
11211
X
P
P
P
Px
XPx
X ~
~x
j
j
j
jyi
xi
iij
ijij
Z
Y
X
,P
P,
y
xP
Note that P and X are resp. 2mx3 and 3xn matrices and
therefore the rank of x is at most 3
2
1
tt
~
~
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
ComputerVision Orthographic factorization
• Factorize m through singular value decomposition
• An affine reconstruction is obtained as follows
TVUx
T~,
~VXUP
n
mmnmm
n
n
X,...,X,X
xxx
xxxxxx
min 212
1
21
22221
11211
P
P
P
Closest rank-3 approximation yields MLE!
ComputerVision
Aug 26/28 - Introduction
Sep 2/4 Cameras Radiometry
Sep 9/11 Sources & Shadows Color
Sep 16/18 Linear filters & edges
(hurricane Isabel)
Sep 23/25 Pyramids & Texture Multi-View Geometry
Sep30/Oct2 Stereo Project proposals
Oct 7/9 Tracking (Welch) Optical flow
Oct 14/16 - -
Oct 21/23 Silhouettes/carving (Fall break)
Oct 28/30 - Structure from motion
Nov 4/6 Project update Proj. SfM
Nov 11/13 Camera calibration Segmentation
Nov 18/20 Fitting Prob. segm.&fit.
Nov 25/27 Matching templates (Thanksgiving)
Dec 2/4 Matching relations Range data
Dec ? Final project
Tentative class schedule
ComputerVision Further Factorization work
Factorization with uncertainty
Factorization for indep. moving objects
Factorization for dynamic objects
Perspective factorization
Factorization with outliers and missing pts.
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
(Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))
(Sturm & Triggs 1996, …)
ComputerVision
Structure from motion of multiple moving objects
one object:
multiple objects:
ComputerVision SfM of multiple moving objects
ComputerVision
Structure from motion of deforming objects
Extend factorization approaches to deal with dynamic shapes
(Bregler et al ’00; Brand ‘01)
ComputerVision Representing dynamic shapes
represent dynamic shape as varying linear combination of basis shapes
k
kk (t)ScS(t)
(fig. M.Brand)
ComputerVision Projecting dynamic shapes
PtScR k
kk
(figs. M.Brand)Rewrite:
ComputerVision Dynamic image sequences
One image:
Multiple images(figs. M.Brand)
ComputerVision Dynamic SfM factorization?
Problem: find J so that M has proper structure
ComputerVision Dynamic SfM factorization
(Bregler et al ’00)
Assumption: SVD preserves order and orientation of basis shape components
ComputerVision Results
(Bregler et al ’00)
ComputerVision Dynamic SfM factorization
(Brand ’01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute J
hard!
(different methods are possible, not so simple and also not optimal)
ComputerVision Non-rigid 3D subspace flow
• Same is also possible using optical flow in stead of features, also takes uncertainty into account
(Brand ’01)
ComputerVision Results
(Brand ’01)
ComputerVision
(Brand ’01)
Results
ComputerVision Results
(Bregler et al ’01)
ComputerVision
PROJECTIVE STRUCTURE FROM MOTION
Reading: Chapter 13.
• The Projective Structure from Motion Problem• Elements of Projective Geometry• Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices• Projective Structure and Motion from Multiple Images
ComputerVision The Projective Structure-from-Motion Problem
Given m perspective images of n fixed points P we can write
Problem: estimate the m 3x4 matrices M andthe n positions P from the mn correspondences p .
i
j ij
2mn equations in 11m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
j
ComputerVision
The Projective Ambiguity of Projective SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and Q is an arbitrary non-singular 4x4 matrix.
When the intrinsic and extrinsic parameters are unknown
Q is a projective transformation.
ComputerVision Projective Spaces: (Semi-Formal) Definition
ComputerVision
A Model of P( R )3
ComputerVision Projective Subspaces and Projective Coordinates
ComputerVision
Projective Subspaces and Projective Coordinates
Projectivecoordinates
P
ComputerVision
Projective Subspaces
Given a choice of coordinate frame
Line: Plane:
ComputerVision
Affine and Projective Spaces
ComputerVision
Affine and Projective Coordinates
ComputerVision
Cross-Ratios
Collinear points
Pencil of coplanar lines Pencil of planes
{A,B;C,D}=sin(+)sin(+)
sin(++)sin
ComputerVision
Cross-Ratios and Projective Coordinates
Along a line equipped with the basis
In a plane equipped with the basis
In 3-space equipped with the basis
*
*
ComputerVision
Projective Transformations
Bijective linear map:
Projective transformation:( = homography )
Projective transformations map projective subspaces ontoprojective subspaces and preserve projective coordinates.
Projective transformations map lines onto lines andpreserve cross-ratios.
ComputerVision
Perspective Projections induce projective transformationsbetween planes.
ComputerVision
Geometric SceneReconstruction
Idea: use (A,O”,O’,B,C)as a projective basis.
ComputerVision
Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. ReportUIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).
ComputerVision
Motion estimation from fundamental matrices
Q
Facts:
b’ can be found using LLS.
Once M and M’ are known,P can be computed with LLS.
ComputerVision
Projective Structure from Motion and Factorization
Factorization??
Algorithm (Sturm and Triggs, 1996) • Guess the depths;• Factorize D;• Iterate.
Does it converge? (Mahamud and Hebert, 2000)
ComputerVision
Bundle adjustment - refining structure and motion
• Minimize reprojection error
– Maximum Likelyhood Estimation (if error zero-mean Gaussian
noise)– Huge problem but can be solved
efficiently (exploit sparseness)
m
i
n
jjiijD
ji 1 1
2
X̂,P̂X̂P̂,xmin
ComputerVision
• Developed in photogrammetry in 50´s
Bundle adjustment
ComputerVision Non-linear least squares
• Linear approximation of residual• allows quadratic approximation of sum- of-squares
J0e
0T1-T
0TT
JJJ
0J2J2J
e
e
JJ 00 ee T
J)λdiag(JJJN' TT
N
(extra term = descent term)
Minimization corresponds to finding zeros of derivative
Levenberg-Marquardt: extra term to deal with singular N(decrease/increase if success/failure to descent)
ComputerVision Bundle adjustment
U1
U2
U3
WT
W
V
P1 P2 P3 M
• Jacobian of has sparse block structure– cameras independent of other cameras,– points independent of other points
J JJN T
12xm 3xn(in general
much larger)
im.pts. view 1
m
i
n
jjiijD
1 1
2M̂P̂,m
Needed for non-linear minimization
ComputerVision Bundle adjustment
Eliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m
WT V
U-WV-1WT
NI0WVI 1
11xm 3xn
Allows much more efficient computations
e.g. 100 views,10000 points,
solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms
ComputerVision Sequential SfM
• Initialize motion from two images• Initialize structure• For each additional view
– Determine pose of camera– Refine and extend structure
• Refine structure and motion
ComputerVision
Initial projective camera motion
• Choose P and P´compatible with F
Reconstruction up to projective ambiguity
(reference plane;arbitrary)
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
Same for more views?
different projective basis
ComputerVision Initializing projective structure
• Reconstruct matches in projective frame by minimizing the reprojection error
Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
ComputerVision Projective pose estimation
• Infere 2D-3D matches from 2D-2D matches
• Compute pose from (RANSAC,6pts)
F
X
x
Inliers: inx,X x X DD iii P
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
ComputerVision
• Refining structure
• Extending structure2-view triangulation
X~
P
1
3
(Iterative linear)
•Initialize motion•Initialize structure•For each additional view
•Determine pose of camera•Refine and extend structure
•Refine structure and motion
Refining and extending structure
ComputerVision Refining structure and motion
• use bundle adjustment
Also model radial distortion to avoid bias!
ComputerVision
Hierarchical structure and motion recovery
• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction
FT
H
PM
ComputerVision Metric structure and motion
Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)
(solution possible based on model selection)
use self-calibration (see next class)
ComputerVision Dealing with dominant planes
ComputerVision
PPPgric
HHgric
ComputerVision Farmhouse 3D models
(note: reconstruction much larger than camera field-of-view)
ComputerVision Application: video augmentation
ComputerVision
Next class: Camera calibration (and self-calibration)
Reading: Chapter 2 and 3