structure from motion (with a digression on svd)

Structure from MotionStructure from Motion

(With a digression on SVD)(With a digression on SVD)

Structure from MotionStructure from Motion

• For now, static scene and moving cameraFor now, static scene and moving camera– Equivalently, rigidly moving scene andEquivalently, rigidly moving scene and

static camerastatic camera

• Limiting case of stereo with many Limiting case of stereo with many camerascameras

• Limiting case of multiview camera Limiting case of multiview camera calibration with unknown targetcalibration with unknown target

• Given Given nn points and points and NN camera positions, camera positions, have 2have 2nNnN equations and 3 equations and 3nn+6+6NN unknowns unknowns

ApproachesApproaches

• Obtaining point correspondencesObtaining point correspondences– Optical flowOptical flow– Stereo methods: correlation, feature Stereo methods: correlation, feature

matchingmatching

• Solving for points and camera motionSolving for points and camera motion– Nonlinear minimization (bundle Nonlinear minimization (bundle

adjustment)adjustment)– Various approximations…Various approximations…

OrthographicOrthographic ApproximationApproximation

• Simplest SFM case: camera Simplest SFM case: camera approximated by orthographic approximated by orthographic projectionprojection

PerspectivePerspective OrthographicOrthographic

Weak PerspectiveWeak Perspective

• An orthographic assumption is An orthographic assumption is sometimes well approximated by a sometimes well approximated by a telephoto lenstelephoto lens

Weak PerspectiveWeak Perspective

Consequences ofConsequences ofOrthographic ProjectionOrthographic Projection

• Scene can be recovered up to scaleScene can be recovered up to scale

• Translation perpendicular to image Translation perpendicular to image planeplanecan never be recoveredcan never be recovered

Orthographic Structure from Orthographic Structure from MotionMotion

• Method due to Tomasi & Kanade, Method due to Tomasi & Kanade, 19921992

• Assume Assume nn points in space points in space pp11 … … ppnn

• Observed at Observed at NN points in time at points in time at image coordinates (image coordinates (xxijij, , yyijij))– Feature tracking, optical flow, etc.Feature tracking, optical flow, etc.


• Write down matrix of dataWrite down matrix of data

NnN

n

NnN

n

yy

yy

xx

xx

1

111

1

111

D

NnN

n

NnN

n

yy

yy

xx

xx

1

111

1

111

D

Points Points Fra

mes

Fram

es


• Step 1: find translationStep 1: find translation

• Translation parallel to viewingTranslation parallel to viewingdirection can not be obtaineddirection can not be obtained

• Translation perpendicular to viewing Translation perpendicular to viewing direction equals motion of average direction equals motion of average position of all pointsposition of all points


• Subtract average of each rowSubtract average of each row

NNnNN

n

NNnNN

n

yyyy

yyyy

xxxx

xxxx

1

11111

1

11111

~D

NNnNN

n

NNnNN

n

yyyy

yyyy

xxxx

xxxx

1

11111

1

11111

~D


• Step 2: try to find rotationStep 2: try to find rotation

• Rotation at each frame defines local Rotation at each frame defines local coordinate axes , , andcoordinate axes , , and

• ThenThen

ii jj kk

jiijjiij yx pjpi ~ˆ~,~ˆ~ jiijjiij yx pjpi ~ˆ~,~ˆ~


• So, can write where R is a So, can write where R is a “rotation” matrix and S is a “shape” “rotation” matrix and S is a “shape” matrixmatrix

RSD~ RSD~

n

N

N ppS

j

j

i

i

R ~~

ˆ

ˆ

ˆ

ˆ

1

T

T1

T

T1

n

N

N ppS

j

j

i

i

R ~~

ˆ

ˆ

ˆ

ˆ

1

T

T1

T

T1


• Goal is to factor Goal is to factor

• Before we do, observe that Before we do, observe that rankrank( ) = 3( ) = 3(in ideal case with no noise)(in ideal case with no noise)

• Proof:Proof:– Rank of Rank of RR is 3 unless no rotation is 3 unless no rotation– Rank of Rank of SS is 3 iff have noncoplanar points is 3 iff have noncoplanar points– Product of 2 matrices of rank 3 has rank 3Product of 2 matrices of rank 3 has rank 3

• With noise, With noise, rankrank( ) might be > 3( ) might be > 3

D~D~

D~D~

D~D~

Digression:Digression:Singular Value Decomposition Singular Value Decomposition

(SVD)(SVD)• Handy mathematical technique that Handy mathematical technique that

has application to many problemshas application to many problems

• Given any Given any mmnn matrix matrix AA, algorithm to , algorithm to find matrices find matrices UU, , VV, and , and WW such that such that

AA = = UU WW VVTT

UU is is mmnn and orthonormal and orthonormalV V is is nnnn and orthonormal and orthonormalWW is is nnnn and diagonal and diagonal

SVDSVD

• Treat as black box: code widely Treat as black box: code widely available (available (svd(A,0)svd(A,0) in Matlab) in Matlab)

T

1

00

00

00

VUA

nw

w

T

1

00

00

00

VUA

nw

w

SVDSVD

• The The wwii are called the are called the singular valuessingular values of of

AA

• If If AA is singular, some of the is singular, some of the wwii will be 0will be 0

• In general In general rankrank((AA) = number of nonzero ) = number of nonzero wwii

• SVD is mostly unique (up to SVD is mostly unique (up to permutation of singular values, or if permutation of singular values, or if some some wwii are equal) are equal)

SVD and InversesSVD and Inverses

• Why is SVD so useful?Why is SVD so useful?

• Application #1: inversesApplication #1: inverses

• AA-1-1=(=(VVTT))-1-1 WW-1-1 UU-1-1 = = VV WW-1-1 UUTT

• This fails when some This fails when some wwii are 0 are 0– It’s It’s supposedsupposed to fail – singular matrix to fail – singular matrix

• Pseudoinverse: if Pseudoinverse: if wwii=0, set 1/=0, set 1/wwii to 0 (!) to 0 (!)– ““Closest” matrix to inverseClosest” matrix to inverse– Defined for all (even non-square) Defined for all (even non-square)

matricesmatrices

SVD and Least SquaresSVD and Least Squares

• Solving Solving AxAx==bb by least squares by least squares

• xx=pseudoinverse(=pseudoinverse(AA) times ) times bb

• Compute pseudoinverse using SVDCompute pseudoinverse using SVD– Lets you see if data is singularLets you see if data is singular– Even if not singular, ratio of max to min Even if not singular, ratio of max to min

singular values (condition number) tells singular values (condition number) tells you how stable the solution will beyou how stable the solution will be

– Set 1/Set 1/wwii to 0 if to 0 if wwii is small (even if not is small (even if not

exactly 0)exactly 0)

SVD and EigenvectorsSVD and Eigenvectors

• Let Let AA==UWVUWVTT, and let , and let xxii be be iithth column of column of VV

• Consider Consider AATTAA xxii::

• So elements of So elements of WW are squared are squared eigenvalues and columns of eigenvalues and columns of VV are are eigenvectors of eigenvectors of AATTAA

iiiii xwwxx 222T2TTTT

0

0

0

1

0

VVWVVWUWVUVWAA iiiii xwwxx 222T2TTTT

0

0

0

1

0

VVWVVWUWVUVWAA

SVD and EigenvectorsSVD and Eigenvectors

• Eigenvectors of Eigenvectors of AATTAA turned up in turned up in finding least-squares solution to finding least-squares solution to AxAx=0=0

• Solution is eigenvector corresponding Solution is eigenvector corresponding to smallest eigenvalueto smallest eigenvalue

SVD and Matrix SimilaritySVD and Matrix Similarity

• One common definition for the norm of a One common definition for the norm of a matrix is the Frobenius norm:matrix is the Frobenius norm:

• Frobenius norm can be computed from Frobenius norm can be computed from SVDSVD

• So changes to a matrix can be evaluated So changes to a matrix can be evaluated by looking at changes to singular valuesby looking at changes to singular values

i j

ija2

FA

i jija2

FA

i

iw2

FA

iiw2

FA

SVD and Matrix SimilaritySVD and Matrix Similarity

• Suppose you want to find best rank-Suppose you want to find best rank-kk approximation to approximation to AA

• Answer: set all but the largest Answer: set all but the largest kk singular values to zerosingular values to zero

• Can form compact representation by Can form compact representation by eliminating columns of eliminating columns of UU and and VV corresponding to zeroed corresponding to zeroed wwii

SVD and PCASVD and PCA

• Principal Components Analysis (PCA): Principal Components Analysis (PCA): approximating a high-dimensional approximating a high-dimensional data set with a lower-dimensional data set with a lower-dimensional subspacesubspace

Original axesOriginal axes

****

******

**** **

**

********

**

**

****** **

**** ******

Data pointsData points

First principal componentFirst principal componentSecond principal componentSecond principal component

SVD and PCASVD and PCA

• Data matrix with points as rows, take Data matrix with points as rows, take SVDSVD

• Columns of Columns of VVkk are principal are principal

componentscomponents

• Value of Value of wwii gives importance of each gives importance of each

componentcomponent

SVD and OrthogonalizationSVD and Orthogonalization

• The matrix The matrix UU is the “closest” is the “closest” orthonormal matrix to orthonormal matrix to AA

• Yet another useful application of the Yet another useful application of the matrix-approximation properties of SVDmatrix-approximation properties of SVD

• Much more stable numerically thanMuch more stable numerically thanGraham-Schmidt orthogonalizationGraham-Schmidt orthogonalization

• Find rotation given general affine matrixFind rotation given general affine matrix

End of DigressionEnd of Digression

• Goal is to factor into Goal is to factor into RR and and SS

• Let’s apply SVD!Let’s apply SVD!

• But should have rank 3 But should have rank 3 all but 3 of the all but 3 of the wwii should be 0 should be 0

• Extract the top 3 Extract the top 3 wwii, together with , together with

the corresponding columns of the corresponding columns of UU and and VV

D~D~

T~UWVD T~UWVD

D~D~

Factoring for Factoring for Orthographic Structure from Orthographic Structure from

MotionMotion• After extracting columns, After extracting columns, UU33 has has

dimensions 2dimensions 2NN3 (just what we 3 (just what we wanted for wanted for RR))

• WW33VV33TT has dimensions 3 has dimensions 3nn (just what (just what

we wanted for we wanted for SS))

• So, let So, let RR**==UU33, , SS**==WW33VV33TT

Affine Structure from MotionAffine Structure from Motion

• The The ii and and jj entries of entries of RR** are not, in are not, in general, general, unit length and perpendicularunit length and perpendicular

• We have found motion (and therefore We have found motion (and therefore shape)shape)up to an affine transformationup to an affine transformation

• This is the best we could do if we didn’tThis is the best we could do if we didn’tassume orthographic cameraassume orthographic camera

Ensuring OrthogonalityEnsuring Orthogonality

• Since can be factored as Since can be factored as RR** SS**, it , it can also be factored as (can also be factored as (RR**QQ)()(QQ-1-1SS**), ), for any for any QQ

• So, search for So, search for QQ such that such that RR == RR** QQ has the properties we wanthas the properties we want

D~D~


• Want orWant or

• Let Let TT = = QQQQTT

• Equations for elements of Equations for elements of TT – solve by – solve byleast squaresleast squares

• Ambiguity – add constraints Ambiguity – add constraints

1ˆˆ T*T* QiQi ii 1ˆˆ T*T* QiQi ii

0ˆˆ

1ˆˆ

1ˆˆ

*TT*

*TT*

*TT*

ii

ii

ii

jQQi

jQQj

iQQi

0ˆˆ

1ˆˆ

1ˆˆ

*TT*

*TT*

*TT*

ii

ii

ii

jQQi

jQQj

iQQi

0

1

0ˆ,

0

0

1ˆ *

1T*

1T jQiQ

0

1

0ˆ,

0

0

1ˆ *

1T*

1T jQiQ


• Have found Have found TT = = QQQQTT

• Find Find Q Q by taking “square root” of by taking “square root” of TT– Cholesky decomposition if Cholesky decomposition if TT is positive is positive

definitedefinite– General algorithms (e.g. General algorithms (e.g. sqrtmsqrtm in in

Matlab)Matlab)

Orthogonal Structure from Orthogonal Structure from MotionMotion

• Let’s recap:Let’s recap:– Write down matrix of observationsWrite down matrix of observations– Find translation from avg. positionFind translation from avg. position– Subtract translationSubtract translation– Factor matrix using SVDFactor matrix using SVD– Write down equations for orthogonalizationWrite down equations for orthogonalization– Solve using least squares, square rootSolve using least squares, square root

• At end, get matrix At end, get matrix RR == RR** QQ of camera of camera positionspositionsand matrix and matrix SS == QQ-1-1SS** of 3D points of 3D points

ResultsResults

• Image sequenceImage sequence

[Tomasi & Kanade][Tomasi & Kanade]

ResultsResults

• Tracked featuresTracked features


ResultsResults

• Reconstructed shapeReconstructed shape


Front viewFront viewTop viewTop view

structure from motion (with a digression on svd)

Documents