eecs 274 computer vision
DESCRIPTION
EECS 274 Computer Vision. Affine Structure from Motion. Affine structure from motion. Structure from motion (SFM) Elements of affine geometry Affine SFM from two views Geometric approach Affine epipolar geometry Affine SFM from multiple views From affine to Euclidean images - PowerPoint PPT PresentationTRANSCRIPT
EECS 274 Computer Vision
Affine Structure from Motion
Affine structure from motion
• Structure from motion (SFM)• Elements of affine geometry• Affine SFM from two views
– Geometric approach– Affine epipolar geometry– Affine SFM from multiple views– From affine to Euclidean images
• Image-based rendering• Reading: FP Chapter 8
Affine structure from motion
Given a sequence of images• Find out feature points in 2D images• Find out corresponding features• Find out their 3D positions• Find out their affine motion
Affine Structure from Motion
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A,8:377-385 (1990). 1990 Optical Society of America.
Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations (projection matrices)?
(structure)(motion)
Scene relief
• When the scene relief is small (compared with the overall distance separating it from the observing camera), affine projection models can be used to approximate the imaging process
Orthographic Projection
Parallel Projection
consider the points off optical axis
viewing raysare parallel
R is a scene reference point
Weak-Perspective Projection (generalizes orthographic projection)
Paraperspective Projection (generalizes parallel projection)
R is a scene reference point
Affine projection models
consider thedistortions for points off the optical axis
Affine projection equations
• Consider weak perspective projection and let zr denote the depth of a reference point R, then P P’ p
2220022
02222
020
0
10
11,
1,
1,,
1
10100
sin0
cot
,110
0
0
0
0
0
0
1
0
0
0
11
1
)coordinate d(normalize
1
0
0
0
0
0
0
1
0
0
0
11
1
ˆ
ˆ
,
1
/
/
1
ˆ
ˆ
tRsk
zMaK
zppatt
ptKz
bRKz
AbAMP
Mv
up
pKv
u
KPtR
z
Kz
v
u
z
y
x
zz
v
u
zy
zx
v
u
z
y
x
z
y
x
rr
rr
TT
rr
rr
r
r
r
R2 is a 2 × 3 matrix of the first 2 rows of R and t2 is a vector formed by first 2 elements of t
p is the non-homogenous coordinate
t2, p0 does not change with a translation
K : calibration matrixM : projection matrix
Weak perspective projection
• k and s denote the aspect ratio and skew of the camera
• M is a 2 × 4 matrix defined by– 2 intrinsic parameters– 5 extrinsic parameters
– 1 scene-dependent structure parameter zr
2220022
02222
10
11,
1,
1,,
1
tRsk
zMaK
zppatt
ptKz
bRKz
AbAMP
Mp
rr
rr
See Chapter 1.2 of FP
The Affine Structure-from-Motion Problem
Given m images of n matched points Pj we can write
Problem: estimate the m 2 × 4 affine projection matrices Mi andthe n positions Pj from the mn correspondences pij
2mn equations in 8m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
Here pij is 2 × 1 non-homogenous coordinate, and Mi = (Ai bi)
The Affine Ambiguity of Affine SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and
Q is an affinetransformation.
When the intrinsic and extrinsic parameters are unknown
C is a 3 × 3 non-singular matrix and d is in R3
Affine Structure from Motion
• Any solution of the affine structure from motion (sfm) can only be defined up to an affine transformation ambiguity
• Consider the 12 parameters for affine transformations, for 2 views (m=2), we need at least 4 point correspondences to determine the projection matrices and 3D points
2mn ≥ 8m + 3n - 12
With known intrinsic parameters • Exploit constraints of Mi = (Ai bi) (See
Chapter 1.2) to eliminate ambiguity
• First find affine shape• Use additional views and constraints
to determine Euclidean structure
2210
1tR
sk
zM
r
2D planar transformations
Preserve parallelism and ratio of distance between colinear points
Affine Spaces: (Semi-Formal) Definition
Left identityAssociativityUniqueness
P, Q: pointsu: vector
Example: R as an Affine Space2
T
TT
Tu
TT
yyxxPQPQu
yxQyxP
byaxuPP
bauyxP
)','( vector unique
)','( and ),(Given
),()(
),(,),(
P, Q: pointsu: vector
In General
The notation
is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP, i.e.u=OP , Φu(O)=P
Warning: P+u and Q-P are defined independently of O!!Namely, no distinguished point serves as an origin
Barycentric Combinations
• Can we add points? R=P+Q NO!
• But, when we can define
• Note by introducing an arbitrary origin O:
• Can “add” a vector to a point and “subtract” two points
Define a point by adding a vector to a point (independent of any choice)
Affine Subspaces
Can be defined purely in terms of points
defined by a point O and a vector subspace U
m+1 points define a m-dimensional subspace
Affine Coordinates
• Coordinate system for U:
• Coordinate system for Y=O+U:
• Coordinate system for Y:
• Affine coordinates:
• Barycentric coordinates:
Affine coordinates of P in the basis formed by points Ai
vector subspace
affine subspace
Define A0P in (u1, …um)
Affine Transformations
Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.
• The affine coordinates of D in the basis of A,B,C are the same as those of D’ in the basis of A’,B’, and C’ – namely 2/3 and ½. • In R3 they are combinations of rigid transformations, non-uniform scaling and shear
Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.
Affine Transformations II
• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).
• Given an affine transformation from X to Y, one can always write:
• When coordinate frames have been chosen for X and Y,this translates into:
Affine projections induce affine transformations from planesonto their images.
Preserve ratio of distance between colinear points, parallelism,and affine coordinatesWeak- and paraperspective projections are affine transformations
Affine Shape
Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).
Affine structure from motion = affine shape recovery.
= recovery of the corresponding motion equivalence classes.
Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).
)',','( of basis in the '' and ''
potins theare ),( and ),( ''''
cbaqped
ppdd
A EA DE D
affine coordinates of P in the basis (A,B,C,D), i.e., 4 points are sufficient
• 4 points define 2 affine views• Affine projection of a plane onto another plane is an affine transformation• Affine coordinates in π can be measured by other two images
Affine Structure from Motion
(Koenderink and Van Doorn, 1991)
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A,8:377-385 (1990). 1990 Optical Society of America.
Given 2 affine views of 4 non-coplanar ponits, the affine shape ofthe scene is uniquely determined
Algebraic motion estimation using affine epipolar constraint
α,β, α’,β’ are constants depending on A, A’, b, b’
0''
''
'''
'''
2
1
232221
131211
2
1
232221
131211
bv
bu
aaa
aaa
bv
bu
aaa
aaa
affine epipolar constraint
Affine Epipolar Geometry
Given point p=(u,v)T, the matching point p’=(u’,v’)T lies on α’u’+β’v’+γ’ = 0 where γ’=αu+βv+δ
Note: the epipolar lines are parallel. Moving p change γ’, or equivalent, the distance from the origin to the epipolar line l’, but does not change the direction.
The Affine Fundamental Matrix
where
affine fundamental matrix
0''
'
'100
010
001
det :constraintEpipolar
0100'
~,
0010
0001~),,( If
, choosecan wesingular,-non is When
'',',,
'
,
'
'det
'''
''''det
'''det0
)(~
,''''~
,~
10,''
~,
~
''',
11
222
3
2
1
1
2
1
222
111
222
111
1
dvcubvau
dvcba
u
v
u
dcbaMMcba
SdSCS
bdC
b
b
b
u
v
u
S
dvc
SSC
bdvC
bduC
bdvC
bduC
ACA
AAC
CPACAMAACM
CQQMMMQM
AMAM
T
TT
T
T
T
T
TT
TT
TT
TT
T
c
r
daacrq
a
a
a
rdq
aa
aa
aa
aa
bdp
bdp
dpbdbd
d
bb
Algebraic Scene Reconstruction Method
Q is an arbitrary affine transformation
Given enough point correspondences, a, b, c, d can be estimated by linear least squares
An Affine Trick.. Algebraic Scene Reconstruction Method
The Affine Structure and Motion from Multiple Images
Suppose we observe a scene with m fixed cameras.
The set of all images of a fixed scene is a 3D affine space!
has rank (at most) 4!
From Affine to Vectorial Structure (Vector Space)
Idea: pick one of the points (or their center of mass)as the origin.
What if we could factorize D? (Tomasi and Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
From uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera: take k=1, s=0 using normalized coordinate and zr=1
Problem: what is affine transformation Q ?
Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.
Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.
Photo tourism/photosynth