projective structure from motion

ComputerVision

Projective structure from motion

Marc PollefeysCOMP 256

Some slides and illustrations from J. Ponce, …

ComputerVision Last class: Affine camera

• The affine projection equations are

1

~

~

j

j

j

yi

xi

ij

ij

ZYX

P

Py

x

how to find the origin? or for that matter a 3D reference point?

affine projection preserves center of gravity

i

ijijij xxx~ i

ijijij yyy~

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

ComputerVision Orthographic factorization

• The orthographic projection equations are

• where

njmijiij ,...,1,,...,1,Xx P

• All equations can be collected for all i and j

• where

n

mmnmm

n

n

X,...,X,X , ,

xxx

xxxxxx

212

1

21

22221

11211

X

P

P

P

Px

XPx

X ~

~x

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and X are resp. 2mx3 and 3xn matrices and

therefore the rank of x is at most 3

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

ComputerVision Orthographic factorization

• Factorize m through singular value decomposition

• An affine reconstruction is obtained as follows

TVUx

T~,

~VXUP

n

mmnmm

n

n

X,...,X,X

xxx

xxxxxx

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

ComputerVision

Aug 26/28 - Introduction

Sep 2/4 Cameras Radiometry

Sep 9/11 Sources & Shadows Color

Sep 16/18 Linear filters & edges

(hurricane Isabel)

Sep 23/25 Pyramids & Texture Multi-View Geometry

Sep30/Oct2 Stereo Project proposals

Oct 7/9 Tracking (Welch) Optical flow

Oct 14/16 - -

Oct 21/23 Silhouettes/carving (Fall break)

Oct 28/30 - Structure from motion

Nov 4/6 Project update Proj. SfM

Nov 11/13 Camera calibration Segmentation

Nov 18/20 Fitting Prob. segm.&fit.

Nov 25/27 Matching templates (Thanksgiving)

Dec 2/4 Matching relations Range data

Dec ? Final project

Tentative class schedule

ComputerVision Further Factorization work

Factorization with uncertainty

Factorization for indep. moving objects

Factorization for dynamic objects

Perspective factorization

Factorization with outliers and missing pts.

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

(Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))

(Sturm & Triggs 1996, …)

ComputerVision

Structure from motion of multiple moving objects

one object:

multiple objects:

ComputerVision SfM of multiple moving objects

ComputerVision

Structure from motion of deforming objects

Extend factorization approaches to deal with dynamic shapes

(Bregler et al ’00; Brand ‘01)

ComputerVision Representing dynamic shapes

represent dynamic shape as varying linear combination of basis shapes

k

kk (t)ScS(t)

(fig. M.Brand)

ComputerVision Projecting dynamic shapes

PtScR k

kk

(figs. M.Brand)Rewrite:

ComputerVision Dynamic image sequences

One image:

Multiple images(figs. M.Brand)

ComputerVision Dynamic SfM factorization?

Problem: find J so that M has proper structure

ComputerVision Dynamic SfM factorization

(Bregler et al ’00)

Assumption: SVD preserves order and orientation of basis shape components

ComputerVision Results

(Bregler et al ’00)

ComputerVision Dynamic SfM factorization

(Brand ’01)

constraints to be satisfied for M

constraints to be satisfied for M, use to compute J

hard!

(different methods are possible, not so simple and also not optimal)

ComputerVision Non-rigid 3D subspace flow

• Same is also possible using optical flow in stead of features, also takes uncertainty into account

(Brand ’01)

ComputerVision Results

(Brand ’01)

ComputerVision

(Brand ’01)

Results

ComputerVision Results

(Bregler et al ’01)

ComputerVision

PROJECTIVE STRUCTURE FROM MOTION

Reading: Chapter 13.

• The Projective Structure from Motion Problem• Elements of Projective Geometry• Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices• Projective Structure and Motion from Multiple Images

ComputerVision The Projective Structure-from-Motion Problem

Given m perspective images of n fixed points P we can write

Problem: estimate the m 3x4 matrices M andthe n positions P from the mn correspondences p .

i

j ij

2mn equations in 11m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

ComputerVision

The Projective Ambiguity of Projective SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and Q is an arbitrary non-singular 4x4 matrix.

When the intrinsic and extrinsic parameters are unknown

Q is a projective transformation.

ComputerVision Projective Spaces: (Semi-Formal) Definition

ComputerVision

A Model of P( R )3

ComputerVision Projective Subspaces and Projective Coordinates

ComputerVision

Projective Subspaces and Projective Coordinates

Projectivecoordinates

P

ComputerVision

Projective Subspaces

Given a choice of coordinate frame

Line: Plane:

ComputerVision

Affine and Projective Spaces

ComputerVision

Affine and Projective Coordinates

ComputerVision

Cross-Ratios

Collinear points

Pencil of coplanar lines Pencil of planes

{A,B;C,D}=sin(+)sin(+)

sin(++)sin

ComputerVision

Cross-Ratios and Projective Coordinates

Along a line equipped with the basis

In a plane equipped with the basis

In 3-space equipped with the basis

*

*

ComputerVision

Projective Transformations

Bijective linear map:

Projective transformation:( = homography )

Projective transformations map projective subspaces ontoprojective subspaces and preserve projective coordinates.

Projective transformations map lines onto lines andpreserve cross-ratios.

ComputerVision

Perspective Projections induce projective transformationsbetween planes.

ComputerVision

Geometric SceneReconstruction

Idea: use (A,O”,O’,B,C)as a projective basis.

ComputerVision

Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. ReportUIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

ComputerVision

Motion estimation from fundamental matrices

Q

Facts:

b’ can be found using LLS.

Once M and M’ are known,P can be computed with LLS.

ComputerVision

Projective Structure from Motion and Factorization

Factorization??

Algorithm (Sturm and Triggs, 1996) • Guess the depths;• Factorize D;• Iterate.

Does it converge? (Mahamud and Hebert, 2000)

ComputerVision

Bundle adjustment - refining structure and motion

• Minimize reprojection error

– Maximum Likelyhood Estimation (if error zero-mean Gaussian

noise)– Huge problem but can be solved

efficiently (exploit sparseness)

m

i

n

jjiijD

ji 1 1

2

X̂,P̂X̂P̂,xmin

ComputerVision

• Developed in photogrammetry in 50´s

Bundle adjustment

ComputerVision Non-linear least squares

• Linear approximation of residual• allows quadratic approximation of sum- of-squares

J0e

0T1-T

0TT

JJJ

0J2J2J

e

e

JJ 00 ee T

J)λdiag(JJJN' TT

N

(extra term = descent term)

Minimization corresponds to finding zeros of derivative

Levenberg-Marquardt: extra term to deal with singular N(decrease/increase if success/failure to descent)

ComputerVision Bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

• Jacobian of has sparse block structure– cameras independent of other cameras,– points independent of other points

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

i

n

jjiijD

1 1

2M̂P̂,m

Needed for non-linear minimization

ComputerVision Bundle adjustment

Eliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

ComputerVision Sequential SfM

• Initialize motion from two images• Initialize structure• For each additional view

– Determine pose of camera– Refine and extend structure

• Refine structure and motion

ComputerVision

Initial projective camera motion

• Choose P and P´compatible with F

Reconstruction up to projective ambiguity

(reference plane;arbitrary)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Same for more views?

different projective basis

ComputerVision Initializing projective structure

• Reconstruct matches in projective frame by minimizing the reprojection error

Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches

• Compute pose from (RANSAC,6pts)

F

X

x

Inliers: inx,X x X DD iii P

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

• Refining structure

• Extending structure2-view triangulation

X~

P

1

3

(Iterative linear)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Refining and extending structure

ComputerVision Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!

ComputerVision

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

ComputerVision Metric structure and motion

Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)

(solution possible based on model selection)

use self-calibration (see next class)

ComputerVision Dealing with dominant planes

ComputerVision

PPPgric

HHgric

ComputerVision Farmhouse 3D models

(note: reconstruction much larger than camera field-of-view)

ComputerVision Application: video augmentation

ComputerVision

Next class: Camera calibration (and self-calibration)

Reading: Chapter 2 and 3