EECS 274 Computer Vision

Affine Structure from Motion

Affine structure from motion

• Structure from motion (SFM)• Elements of affine geometry• Affine SFM from two views

– Geometric approach– Affine epipolar geometry– Affine SFM from multiple views– From affine to Euclidean images

• Reading: FP Chapter 12

Affine structure from motion

Given a sequence of images• Find out feature points in 2D images• Find out corresponding features• Find out their 3D positions• Find out their affine motion

Affine Structure from Motion

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A,8:377-385 (1990). 1990 Optical Society of America.

Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations (projection matrices)?

(structure)(motion)

Scene relief

• When the scene relief is small (compared with the overall distance separating it from the observing camera), affine projection models can be used to approximate the imaging process

Orthographic Projection

Parallel Projection

consider the points off optical axis

viewing raysare parallel

R is a scene reference point

Weak-Perspective Projection (generalizes orthographic projection)

Paraperspective Projection (generalizes parallel projection)

R is a scene reference point

Affine projection models

consider thedistortions for points off the optical axis

Affine projection equations

• Consider weak perspective projection and let zr denote the depth of a reference point R, then P P’ p

2220022

02222

020

0

10

11,

1,

1,,

1

10100

sin0

cot

,110

0

0

0

0

0

0

1

0

0

0

11

1

)coordinate d(normalize

1

0

0

0

0

0

0

1

0

0

0

11

1

ˆ

ˆ

,

1

/

/

1

ˆ

ˆ

tRsk

zMaK

zppatt

ptKz

bRKz

AbAMP

Mv

up

pKv

u

KPtR

z

Kz

v

u

z

y

x

zz

v

u

zy

zx

v

u

z

y

x

z

y

x

rr

rr

TT

rr

rr

r

r

r

R2 is a 2 × 3 matrix of the first 2 rows of R and t2 is a vector formed by first 2 elements of t

p is the non-homogenous coordinate

Weak perspective projection

• k and s denote the aspect ratio and skew of the camera

• M is a 2 × 4 matrix defined by– 2 intrinsic parameters– 5 extrinsic parameters

– 1 scene-dependent structure parameter zr

2220022

02222

10

11,

1,

1,,

1

tRsk

zMaK

zppatt

ptKz

bRKz

AbAMP

Mp

rr

rr

See Chapter 2.3 of FP

The Affine Structure-from-Motion Problem

Given m images of n matched points Pj we can write

Problem: estimate the m 2 × 4 affine projection matrices Mi andthe n positions Pj from the mn correspondences pij

2mn equations in 8m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

Here pij is 2 × 1 non-homogenous coordinate, and Mi = (Ai bi)

The Affine Ambiguity of Affine SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and

Q is an affinetransformation.

When the intrinsic and extrinsic parameters are unknown

C is a 3 × 3 non-singular matrix and d is in R3

Affine Structure from Motion

• Any solution of the affine structure from motion (sfm) can only be defined up to an affine transformation ambiguity

• Taking into account the 12 parameters define general affine transformation, for 2 views (m=2), we need at least 4 point correspondences to determine the projection matrices and 3D points

2mn ≥ 8m + 3n - 12

With known intrinsic parameters • Exploit constraints of Mi = (Ai bi) (See

Chapter 2.3) to eliminate ambiguity

• First find affine shape• Use additional views and constraints

to determine Euclidean structure

2210

1tR

sk

zM

r

2D planar transformations

Preserve parallelism and ratio of distance between colinear points

Affine Spaces: (Semi-Formal) Definition

Example: R as an Affine Space2

T

TT

Tu

TT

yyxxPQPQu

yxQyxP

byaxuPP

bauyxP

)','( vector unique

)','( and ),(Given

),()(

),(,),(

In General

The notation

is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP, i.e.u=OP , Φu(O)=P

Warning: P+u and Q-P are defined independently of O!!

Barycentric Combinations

• Can we add points? R=P+Q NO!

• But, when we can define

• Note by introducing an arbitrary origin O:

• Can “add” a vector to a point and “subtract” two points

Affine Subspaces

Can be defined purely in terms of points

defined by y a point O and a vector subspace U

m+1 points define a m-dimensional subspace

Affine Coordinates

• Coordinate system for U:

• Coordinate system for Y=O+U:

• Coordinate system for Y:

• Affine coordinates:

• Barycentric coordinates:

Affine coordinates of P in the basis formed by points Ai

Affine Transformations

Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.

•The affine coordinates of D in the basis of A,B,C are the same as those of D’ in the basis of A’,B’, and C’ – namely 2/3 and ½. •In E3 they are combinations of rigid transformations, non-uniform scalings and shears

Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.

Affine Transformations II

• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).

• Given an affine transformation from X to Y, one can always write:

• When coordinate frames have been chosen for X and Y,this translates into:

Affine projections induce affine transformations from planesonto their images.

Preserve ratio of distance between colinear points, parallelism,and affine coordinatesWeak- and paraperspective projections are affine transformations

Affine Shape

Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).

Affine structure from motion = affine shape recovery.

= recovery of the corresponding motion equivalence classes.

Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).

)',','( of basis in the '' and ''

potins theare ),( and ),( ''''

cbaqped

ppdd

AEADED

affine coordinates of P inthe basis (A,B,C,D)

•4 points define 2 affine views•Affine projection of a plane onto •another plane is an affine transformation•Affine coordinates in π can be measured by other two images

Affine Structure from Motion

(Koenderink and Van Doorn, 1991)

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A,8:377-385 (1990). 1990 Optical Society of America.

Given 2 affine views of 4 non-coplanar ponits, the affine shape ofthe scene is uniquely determined

Algebraic motion estimation using affine epipolar constraint

Note: the epipolar lines are parallel.

A, A’, b, b’ are knownα,β, α’,β’ are constants of A, A’, b, b’

0''

''

'''

'''

2

1

232221

131211

2

1

232221

131211

bv

bu

aaa

aaa

bv

bu

aaa

aaa

Affine Epipolar Geometry

Given point p=(u,v)T, the matching point p’=(u’,’v)T lies on α’u’+β’v’+(αu+βv+δ)=0

The Affine Fundamental Matrix

where

0''

'

'100

010

001

det :constraintEpipolar

0100'

~,

0010

0001~),,( If

, choosecan wesingular,-non is When

'',',,

'

,

'

'det

'''

''''det

'''det0

)(~

,''''~

,~

10,''

~,

~

''',

11

222

3

2

1

1

2

1

222

111

222

111

1

dvcubvau

dvcba

u

v

u

dcbaMMcbac

rSdSCS

bdadaCc

b

b

b

r

u

v

u

q

a

a

a

S

dvc

rSdqSC

bdavCa

bdauCa

bdavCa

bdauCa

bdApCA

bAdpAC

dpCPbdACAMbAdACM

dCQQMMMQM

bAMbAM

T

TT

T

T

T

T

TT

TT

TT

TT

T

Algebraic Scene Reconstruction Method

An Affine Trick.. Algebraic Scene Reconstruction Method

The Affine Structure of Affine Images

Suppose we observe a scene with m fixed cameras..

The set of all images of a fixed scene is a 3D affine space!

has rank 4!

From Affine to Vectorial Structure

Idea: pick one of the points (or their center of mass)as the origin.

What if we could factorize D? (Tomasi and Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

From uncalibrated to calibrated cameras

Weak-perspective camera:

Calibrated camera:

Problem: what is Q ?

Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

Photo tourism/photosynth