digital visual effects yung-yu chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · structure...
TRANSCRIPT
![Page 1: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/1.jpg)
Structure from motion
Digital Visual EffectsYung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz, Zhengyou Zhang and Marc Pollefyes
![Page 2: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/2.jpg)
Outline
• Epipolar geometry and fundamental matrix• Structure from motion• Factorization method• Bundle adjustment• Applications
![Page 3: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/3.jpg)
Epipolar geometry & fundamental matrix
![Page 4: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/4.jpg)
The epipolar geometry
C,C’,x,x’ and X are coplanar
epipolar geometry demo
![Page 5: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/5.jpg)
The epipolar geometry
What if only C,C’,x are known?
![Page 6: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/6.jpg)
The epipolar geometry
All points on project on l and l’
![Page 7: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/7.jpg)
The epipolar geometry
Family of planes and lines l and l’ intersect at eand e’
![Page 8: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/8.jpg)
The epipolar geometry
epipolar plane = plane containing baselineepipolar line = intersection of epipolar plane with image
epipolar pole= intersection of baseline with image plane = projection of projection center in the other image
epipolar geometry demo
![Page 9: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/9.jpg)
The fundamental matrix F
C C’T=C’-C
Rp p’
TRp'p Two reference frames are related via the extrinsic parameters
![Page 10: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/10.jpg)
The fundamental matrix F
0'Epp essential matrix
TRp'p
00
0
xy
xz
yz
TTTT
TTT
Multiply both sides by TpT
TRp'TppTp TT
Rp'Tp T0
![Page 11: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/11.jpg)
The fundamental matrix F
0'Epp
Let M and M’ be the intrinsic matrices, then
xMp 1 ''' 1 xMp
0)''()( 11 xMExM0'' 1 xEMMx
0'Fxx fundamental matrix
![Page 12: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/12.jpg)
The fundamental matrix F
• The fundamental matrix is the algebraic representation of epipolar geometry
• The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images
0Fx'xT 0lxT
![Page 13: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/13.jpg)
F is the unique 3x3 rank 2 matrix that satisfies xTFx’=0 for all x↔x’
1. Transpose: if F is fundamental matrix for (x,x’), then FT
is fundamental matrix for (x’,x)2. Epipolar lines: l=Fx’ & l’=FTx3. Epipoles: on all epipolar lines, thus eTFx’=0, x’
eTF=0, similarly Fe’=04. F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)5. F is a correlation, projective mapping from a point x to
a line l=Fx’ (not a proper correlation, i.e. not invertible)
The fundamental matrix F
![Page 14: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/14.jpg)
The fundamental matrix F
• It can be used for – Simplifies matching– Allows to detect wrong matches
![Page 15: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/15.jpg)
Estimation of F — 8-point algorithm
• The fundamental matrix F is defined by
0'Fxxfor any pair of matches x and x’ in two images.
• Let x=(u,v,1)T and x’=(u’,v’,1)T,
333231
232221
131211
fffffffff
F
each match gives a linear equation
0'''''' 333231232221131211 ffvfuvffvvfvuuffuvfuu
![Page 16: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/16.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
fffffffff
vuvvvuvuvuuu
vuvvvuvuvuuuvuvvvuvuvuuu
nnnnnnnnnnnn
• In reality, instead of solving , we seek fto minimize subj. . Find the vector corresponding to the least singular value.
0AfAf 1f
![Page 17: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/17.jpg)
8-point algorithm
• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to . 'FF 0'det F
• It is achieved by SVD. Let , where
, let
then is the solution.
VUF Σ
3
2
1
000000
Σ
0000000
Σ' 2
1
VUF Σ''
![Page 18: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/18.jpg)
8-point algorithm% Build the constraint matrix
A = [x2(1,:)'.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
% Extract fundamental matrix from the column of V % corresponding to the smallest singular value.
F = reshape(V(:,9),3,3)';
% Enforce rank2 constraint [U,D,V] = svd(F);F = U*diag([D(1,1) D(2,2) 0])*V';
![Page 19: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/19.jpg)
8-point algorithm
• Pros: it is linear, easy to implement and fast• Cons: susceptible to noise
![Page 20: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/20.jpg)
Problem with 8-point algorithm
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differencebetween column of data matrix least-squares yields poor results
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
fffffffff
vuvvvuvuvuuu
vuvvvuvuvuuuvuvvvuvuvuuu
nnnnnnnnnnnn
![Page 21: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/21.jpg)
Normalized 8-point algorithm
1. Transform input by ,2. Call 8-point on to obtain3.
ii Txx ˆ 'i
'i xTx 'ˆ
'ii xx ˆ,ˆ
TFTF ˆΤ'F̂
0Fxx'
0ˆ'ˆ 1 xFTTx'
F̂
![Page 22: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/22.jpg)
Normalized 8-point algorithm
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
normalized least squares yields good resultsTransform image to ~[-1,1]x[-1,1]
![Page 23: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/23.jpg)
Normalized 8-point algorithm
A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
F = reshape(V(:,9),3,3)';
[U,D,V] = svd(F);F = U*diag([D(1,1) D(2,2) 0])*V';
% DenormaliseF = T2'*F*T1;
[x1, T1] = normalise2dpts(x1);[x2, T2] = normalise2dpts(x2);
![Page 24: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/24.jpg)
Normalizationfunction [newpts, T] = normalise2dpts(pts)
c = mean(pts(1:2,:)')'; % Centroidnewp(1,:) = pts(1,:)-c(1); % Shift origin to centroid.newp(2,:) = pts(2,:)-c(2);
meandist = mean(sqrt(newp(1,:).^2 + newp(2,:).^2));scale = sqrt(2)/meandist;
T = [scale 0 -scale*c(1)0 scale -scale*c(2)0 0 1 ];
newpts = T*pts;
![Page 25: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/25.jpg)
RANSAC
repeatselect minimal sample (8 matches)compute solution(s) for Fdetermine inliers
until (#inliers,#samples)>95% or too many times
compute F based on all inliers
![Page 26: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/26.jpg)
Results (ground truth)
![Page 27: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/27.jpg)
Results (8-point algorithm)
![Page 28: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/28.jpg)
Results (normalized 8-point algorithm)
![Page 29: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/29.jpg)
Structure from motion
![Page 30: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/30.jpg)
Structure from motion
structure for motion: automatic recovery of camera motionand scene structure from two or more images. It is a self calibration technique and called automatic camera trackingor matchmoving.
Unknowncamera
viewpoints
![Page 31: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/31.jpg)
Applications
• For computer vision, multiple-view shape reconstruction, novel view synthesis and autonomous vehicle navigation.
• For film production, seamless insertion of CGI into live-action backgrounds
![Page 32: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/32.jpg)
Matchmove
example #1 example #2 example #3
![Page 33: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/33.jpg)
Structure from motion
2D featuretracking 3D estimation optimization
(bundle adjust)geometry
fitting
SFM pipeline
![Page 34: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/34.jpg)
Structure from motion
• Step 1: Track Features– Detect good features, Shi & Tomasi, SIFT– Find correspondences between frames
• Lucas & Kanade-style motion estimation• window-based correlation• SIFT matching
![Page 35: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/35.jpg)
KLT tracking
http://www.ces.clemson.edu/~stb/klt/
![Page 36: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/36.jpg)
Structure from Motion• Step 2: Estimate Motion and Structure
– Simplified projection model, e.g., [Tomasi 92]– 2 or 3 views at a time [Hartley 00]
![Page 37: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/37.jpg)
Structure from Motion• Step 3: Refine estimates
– “Bundle adjustment” in photogrammetry– Other iterative methods
![Page 38: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/38.jpg)
Structure from Motion• Step 4: Recover surfaces (image-based
triangulation, silhouettes, stereo…)
Good mesh
![Page 39: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/39.jpg)
Factorization methods
![Page 40: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/40.jpg)
Problem statement
![Page 41: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/41.jpg)
Notations
• n 3D points are seen in m views• q=(u,v,1): 2D image point• p=(x,y,z,1): 3D scene point• : projection matrix• : projection function• qij is the projection of the i-th point on image j• ij projective depth of qij
)( ijij pq )/,/(),,( zyzxzyx zij
![Page 42: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/42.jpg)
Structure from motion
• Estimate and to minimize
));((log),,,,,(1 1
11 ijij
m
j
n
iijnm Pw qpΠppΠΠ
otherwisej in view visibleis if
01 i
ij
pw
• Assume isotropic Gaussian noise, it is reduced to
2
1 111 )(),,,,,( ijij
m
j
n
iijnm w qpΠppΠΠ
j ip
• Start from a simpler projection model
![Page 43: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/43.jpg)
Orthographic projection• Special case of perspective projection
– Distance from the COP to the PP is infinite
– Also called “parallel projection”: (x, y, z) → (x, y)
Image World
![Page 44: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/44.jpg)
SFM under orthographic projection
2D image point
Orthographic projectionincorporating 3D rotation 3D scene
point
imageoffset
tΠpq 12 32 13 12
• Trick– Choose scene origin to be centroid of 3D points– Choose image origins to be centroid of 2D points– Allows us to drop the camera translation:
Πpq
![Page 45: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/45.jpg)
factorization (Tomasi & Kanade)
n332n2
n21n21 pppqqq
projection of n features in one image:
n3
32mn2m
212
1
21
22221
11211
n
mmnmm
n
n
ppp
Π
ΠΠ
qqq
qqqqqq
projection of n features in m images
W measurement M motion S shape
Key Observation: rank(W) <= 3
![Page 46: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/46.jpg)
n33m2n2m''
SMW
• Factorization Technique– W is at most rank 3 (assuming no noise)– We can use singular value decomposition to factor W:
Factorization
– S’ differs from S by a linear transformation A:
– Solve for A by enforcing metric constraints on M
))(('' ASMASMW 1
n33m2n2m SMWknown solve for
![Page 47: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/47.jpg)
Results
![Page 48: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/48.jpg)
Extensions to factorization methods
• Projective projection• With missing data• Projective projection with missing data
![Page 49: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/49.jpg)
Bundle adjustment
![Page 50: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/50.jpg)
Bundle adjustment
• n 3D points are seen in m views• xij is the projection of the i-th point on image j• aj is the parameters for the j-th camera• bi is the parameters for the i-th point• BA attempts to minimize the projection error
Euclidean distance
predicted projection
![Page 51: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/51.jpg)
Levenberg-Marquardt method
• LM can be thought of as a combination of steepest descent and the Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.
![Page 52: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/52.jpg)
Bundle adjustment
![Page 53: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/53.jpg)
MatchMove
![Page 54: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/54.jpg)
Applications of matchmove
![Page 55: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/55.jpg)
Jurassic park
![Page 56: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/56.jpg)
2d3 boujou
Enemy at the Gate, Double Negative
![Page 57: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/57.jpg)
2d3 boujou
Enemy at the Gate, Double Negative
![Page 58: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/58.jpg)
Photo Tourism
![Page 59: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/59.jpg)
VideoTrace
http://www.acvt.com.au/research/videotrace/
![Page 60: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/60.jpg)
Video stabilization
![Page 61: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/61.jpg)
References• Richard Hartley, In Defense of the 8-point Algorithm, ICCV, 1995. • Carlo Tomasi and Takeo Kanade, Shape and Motion from Image
Streams: A Factorization Method, Proceedings of Natl. Acad. Sci., 1993.
• Manolis Lourakis and Antonis Argyros, The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm, FORTH-ICS/TR-320 2004.
• N. Snavely, S. Seitz, R. Szeliski, Photo Tourism: Exploring Photo Collections in 3D, SIGGRAPH 2006.
• A. Hengel et. al., VideoTrace: Rapid Interactive Scene Modelling from Video, SIGGRAPH 2007.
![Page 62: Digital Visual Effects Yung-Yu Chuangcyy/courses/vfx/20spring/... · 2020. 2. 21. · Structure from motion Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski,](https://reader035.vdocuments.site/reader035/viewer/2022062609/610296839b3c906e6f145d1a/html5/thumbnails/62.jpg)
Project #3 MatchMove
• It is more about using tools in this project• You can choose either calibration or structure
from motion to achieve the goal• Calibration • Voodoo/Icarus
• Examples from previous classes, #1, #2• https://www.youtube.com/user/theActionMovieKid/videos