structure from motionece631web.groups.et.byu.net/lectures/ecen631 25...3d reconstruction...

Structure from Motion

  Givenamo*onfieldes*matedfromanimagesequence

  Computetheshape,orstructure,ofthevisibleobjects

  Theirmo*onwithrespecttotheviewingcamera

3-D Motion and Object Structure

3/31/11 2ECEn631

3D Reconstruction   CorrespondenceProblemEpipolarGeometry3DReconstruc9on

  Bothintrinsicandextrinsicparametersareknown–  Easytocalibrateforstereovision(fixedcameras)–  Unambiguous(Eculideanstructure)

  Onlytheintrinsicparametersareknowandes9mateextrinsicparameters(unknowncameramo9o)–  Structurefrommo9onusingasinglecamera–  Uptoanunknownscalingfactor(affinestructure)

  Onlythepixelcorrespondencesaretheonlyinforma9onavailable(videofromunknownsources)–  Uptoanunknownprojec9vetransforma9onoftheenvironment

3/31/11 ECEn631 3

3-D Reconstruction

  EuclideanStructure  AffineStructure(uptoascalefactor)  Projec*veTransforma*on

3/31/11 4ECEn631

Reconstruction from a Sparse Motion Field

  Largeaveragedisparitybetweenframes(slowframerates,fastmovingcamera,orveryshortobjectdistance)– Usestereovision‐likemethod,e.g.,8‐pointalgorithmappliedtoapairofframesata*me.

  Smallaveragedisparitybetweenframes–  Thereconstruc*ongainsstabilityandrobustnessfromthe*meintegra*onoflongsequencesofframes.

3/31/11 ECEn631 5

Large average disparity between frames

1.  Calibrate camera 2.  Acquire features (corners, SIFT, SURF, etc) 3.  Match features (matchTemplate()) 4.  Undistort feature points (not the entire image) 5.  Estimate fundamental matrix F (use data

normalization and RANSAC) 6.  Calculate essential matrix E from F and intrinsic

parameters 7.  Calculate R and T 8.  Bundle adjustment (i.e., estimation of 3D points and/

or cameras based on minimization of reprojection errors across all images and points)

3/31/11 ECEn631 6

8-point Linear Algorithm

Compute SVD of the new E=UDVT

Calculate R and T as

€

RZT (±π

2) =

0 ±1 01 0 00 0 1

Σ =σ 0 00 σ 00 0 0

=1 0 00 1 00 0 0

ˆ T =0 −Tz TyTz 0 −Tx−Ty Tx 0

€

E = ˆ T R ∈ R3×3Essential Matrix

3/31/11 7ECEn631€

R = URzT (±π

2)V T

ˆ T = URz (±π2

)ΣU T

Normalized the unknown scale factor

can be calculated if object size is

known.

Limitations

There is no guarantee that the estimated pose (R,T) is as close as possible to the true solution.

Even if we were to accept such an (R,T), a noisy image pair would not necessary give rise a consistent 3-D reconstruction.

3/31/11 8ECEn631

Small Average Disparity   Factoriza*onMethod  Assump*ons:

– Orthographiccameramodel– nimagepointscorrespondingtothenon‐coplanarscenepointshavebeentrackedinN>=3frames

  Assump*on2almostmeansthattheprocessingstartsaVertheen*resequenceisacquired(notacceptableforsomeapplica*ons)

  Cameracalibra*oncanbeignored(becauseassumingorthographicmodel),ifreconstructthe3‐Dpointsonlyuptoascalefactor

3/31/11 ECEn631 9

Pj

X

Y

Z

ii

ji ki

xij,yij

P1 P2

xi2,yi2 xi1,yi1

From a Sparse Motion Field

ii and ji are the unit vectors of the image reference frame, expressed in the world reference frame at time instant i. Ki is the cross product of the two and is the optical axis.

3/31/11 10ECEn631

Factorization Method

€

pij = [xij ,yij ]T jth image points at ith frame

j =1....n and i =1....NThe measurement matrix

W = XY where X and Y are two N×n

matrices with xijand yij as the entries.

It is simple to implement and gives numerical stable results.

3/31/11 11ECEn631

€

The registered measurement matrix

˜ W =˜ X ˜ Y

where ˜ X and ˜ Y are the X and Y

offset by the centroid ˜ x i and ˜ y i.

˜ x i =1n

xijj=1

n∑ and ˜ y i =

1n

yijj=1

n∑

3/31/11 12ECEn631

Factorization

3/31/11 ECEn631 13

€

The registered measurement matrix ˜ W can be factorized into a 2N × 3 rotation matrix R and an n× 3 shape matrix.˜ W = RS (proof skipped)

The factorization of ˜ W is not unique.˜ W = RQQ−1S is always true for any

invertible 3× 3 matrix Q.

Two constraints to help compute a factorization that is unique up to an unknown initial orientation of the world reference frame with respect to the camera frame:

The rows of R must have unit norm.

The first n rows of R (iiT) must be orthogonal to the corresponding n rows of (jiT) .

3/31/11 14ECEn631

€

The registered measurement matrix should have at most rank 3 : ˜ W = UDV T . But due to noise or imperfect matching, the rank of ˜ W is usually greater than 3 (More than three nonzero singular values along the diagonal.) This can be corrected by simply settingall but the three largest singular values in D to zero and recomputing a new ˜ W .SVD the new ˜ W to get new UDV T . Form D' by selecting the 3× 3 top left submatrix of the new D corresponding to the 3 argest singular values. Find columns corresponding to these singular value to form U' (2N× 3) and V' (n× 3), submatrics of U (2N×2N) and V (n ×n), respectively.

3/31/11 15ECEn631

€

ˆ R = U' D'1

2 =

i1T

i2T

.

.iNT

j1T

j2T

.

.jNT

ˆ S = D'1

2 V 'T = P1 P2 . . Pn[ ]

3/31/11 16ECEn631

2Nx3 rotation matrix

nx3 shape matrix

€

The rotation and shape matrices can be calaulated as R = ˆ R Q and S = Q−1 ˆ S to satisfy the constraints, where Q meets therequirements :ˆ i i

TQQTˆ i iT =1

ˆ j iTQQTˆ j i

T =1ˆ i i

TQQTˆ j iT = 0

3/31/11 17ECEn631

€

The rotation and shape matrices : R and SWhat about the translation?

€

Translation component parallel to the image plane ispropotional to the motion of the centroid of the data points on the image plane.Translation along the optical axis cannot be determined

3/31/11 18ECEn631

Reconstruction from a Dense Motion Field (Optical Flow)

  StructureDensebutinaccuratees*matesofthemo*onfieldfromop*calflow

  Itisaninstantaneousanalysisnotintegratedovermanyframes

3/31/11 19ECEn631

Method

  Determinethedirec*onoftransla*onusingmo*onparallax

  Determinealeast‐squaresapproxima*onoftherota*onalcomponentoftheop*calflowanduseittocomputedepth.

3/31/11 ECEn631 20

Motion Parallax - review

€

The relative motion field of two instantaneous conincident points does not depend on the rotational component of motion in 3 - D space.

Motion Parallax can be used to compute structure and motion from optical flow.

3/31/11 21ECEn631

Remember:

3/31/11 22ECEn631

Motion Parallax - review

€

Two points P and P in 3 - D Motion fields for P and P arevx = vx

T + vxω and vy = vy

T + vyω

v x = v xT + v x

ω and v y = v yT + v y

ω

If at an instant t the two points havethe same projection p = p = [x,y]T

, then vxω = v x

ωand vyω = v y

ω .

3/31/11 23ECEn631

Motion Parallax - review

€

By taking the difference between v and v

Δvx = vxT − v x

T = (Tz x −Tx f )( 1Z−

1Z

)

Δvy = vyT − v y

T = (Tz y−Ty f )( 1Z−

1Z

)

Δvx and Δvy (relative motion field) increase with the separation in depth between the two points P and P .

Rotation components cancel out. Relative motion field does not depend on rotation components.

3/31/11 24ECEn631

Motion Parallax - review

€

Δvx = vxT − v x

T = (Tz x −Tx f )( 1Z−

1Z

)

Δvy = vyT − v y

T = (Tz y−Ty f )( 1Z−

1Z

)

Δvy

Δvx=

Tz y−Ty fTz x −Tx f

=

y− fTy

Tz

x − f TxTz

=y− y0x − x0

For all possible rotational motions, the vector(Δvx

T ,ΔvyT ) points in the direction of p0 (the vanishing

point of th translation direction).

3/31/11 25ECEn631

Motion Parallax - review

Determine the direction of translation using motion parallax. Use two almost coincident points (approximation) to find the instantaneous epipole.

For two coincident points

For two very close points p and

€

ΔvxT =

Tz x −Tx fZ

−Tz x −Tx f

Z

ΔvyT =

Tz y−Ty fZ

−Tz y −Ty f

Z €

ΔvxT = vx

T − v xT = (Tz x −Tx f )( 1

Z−1Z )

ΔvyT = vy

T − v yT = (Tz y−Ty f )( 1

Z−1Z )

3/31/11 26ECEn631

The rotational component of the relative motion field

For two coincident points

For two very close points p and

€

vxω = v x

ω = −ω y f +ωz y+ω x xy

f−ω y x2

f

vyω = v y

ω =ω x f −ωz x −ω y xy

f+ω x y2

f

€

Δvxω = 0 and Δvy

ω = 0

€

Δvxω =ωz (y− y )+ω x

f(xy− x y )−

ω y

f(x2 − x 2 )

Δvyω = −ωz (x − x )+

ω y

f(xy− x y )−ω x

f(y2 − y 2 )

3/31/11 27ECEn631

€

ΔvxT =

Tz x −Tx fZ

−Tz x −Tx f

Z

ΔvyT =

Tz y−Ty fZ

−Tz y −Ty f

Z

€

ΔvxT = (Tz x −Tx f )( 1

Z−1Z )+ Tz

Z (x − x )

ΔvyT = (Tz y−Ty f )( 1

Z−1Z )+ Tz

Z (y− y )

3/31/11 28ECEn631

= 0 for two coincident points

Ifpandarecloseenough,alargerela*vemo*onfieldcanonlybeduetoalargedifferenceindepthbetweenthe3‐DpointsPand

3/31/11 29ECEn631

SFW from Optical Flow

  Input–  Intrinsicparametersfromtheviewingcamera– Adenseop*calflowfromasinglerigidmo*on

  Convertimageplanevxandvytoimagepixelsusingcameraintrinsicparameters

3/31/11 ECEn631 30

€

vx =Tzx −Tx f

Z−ω y f +ωz y+

ω x xyf

−ω y x

2

f

vy =Tzy−Ty f

Z+ω x f −ωz x −

ω y xyf

+ω x y

2

f

Step 1 – Motion Field

Step 2 – Flow Differences Compute the flow differences between a point pi and all its neighbors in a small patch Qi and determine the eigenvalues and eigenvectors of the matrix.

Let λi be the greater eigenvalue and the unit eigenvector corresponding to λi .

is the direction of the relative motion field

3/31/11 31ECEn631

Step 3 – Epipoles Since relative motion field and the two points are coplanar,

€

( ˆ I i × pi )T p0 = 0

If there are N patches, then

€

ˆ I 1× p1T

ˆ I 2 × p2T

••

ˆ I N × pNT

p0 = Bp0 = 0

3/31/11 32ECEn631

Step 3 – Epipoles

Epipole p0 can be solved by using SVD on

€

λ1 0 0 0 00 λ2 0 0 00 0 • 0 00 0 0 • 00 0 0 0 λN

ˆ I 1× p1T

ˆ I 2 × p2T

••

ˆ I N × pNT

= UDV T

Estimate the epipole p0 as the olumn of V corresponding to the smallest singular value.

3/31/11 33ECEn631

Step 4 – Components of Angular Velocity

Using the intrinsic parameters and at each point pi of the image plane, we form the dot product of all N points

€

v⊥ = vxω (yi − y0 )− vy

ω (xi − x0 )Determine the angular velocity components as the least-squares solution of a system of N simultaneous equations (the above dot product equation).

3/31/11 34ECEn631

Step 5 – Translational Components

Use the intrinsic parameters and epipole coordinates to determine the translational components

€

x0 = f TxTz

y0 = fTyTz

3/31/11 35ECEn631

Step 6 – Depth

Solve for depth Z for each point using

€

vx =Tzx −Tx f

Z−ω y f +ωz y+

ω x xyf

−ω y x

2

f

vy =Tzy−Ty f

Z+ω x f −ωz x −

ω y xyf

+ω x y

2

f

3/31/11 36ECEn631