lecture 8.2 structure from motion - universitetet i oslo · 2 â ðžâð. 1,ð. 2 = ðŸ. 1....
TRANSCRIPT
Lecture 8.2 Structure from Motion
Thomas Opsahl
More-than-two-view geometry
2
ðï¿œðð = ððð¿ï¿œð
Correspondences (matching) ⢠More views enables us to reveal and remove
more mismatches than we can do in the two-view case
⢠More views also enables us to predict correspondences that can be tested with or without the use of descriptors
Scene geometry (structure) ⢠Effect of more views on determining the 3D
structure of the scene?
Camera geometry (motion) ⢠Effect of more views on determining camera
poses?
Structure from Motion
Problem Given ð images of ð fixed 3D points, estimate the ð projection matrices ðð and the ð points ð¿ð from the ð â ð correspondences ððð â ððð
3
ðï¿œðð = ððð¿ï¿œð
Structure from Motion
Problem Given ð images of ð fixed 3D points, estimate the ð projection matrices ðð and the ð points ð¿ð from the ð â ð correspondences ððð â ððð ⢠We can solve for structure and motion when
2ðð ⥠11ð + 3ð â 15 ⢠In the general/uncalibrated case, cameras and
points can only be recovered up to a projective ambiguity (ðï¿œðð = ðððâ1ðð¿ï¿œð)
⢠In the calibrated case, they can be recovered up to a similarity (scale)
â Known as Euclidean/metric reconstruction
4
ðï¿œðð = ððð¿ï¿œð
Structure from motion
Problem Given ð images of ð fixed 3D points, estimate the ð projection matrices ðð and the ð points ð¿ð from the ð â ð correspondences ððð â ððð ⢠We can solve for structure and motion when
2ðð ⥠11ð + 3ð â 15 ⢠In the general/uncalibrated case, cameras and
points can only be recovered up to a projective ambiguity (ðï¿œðð = ðððâ1ðð¿ï¿œð)
⢠In the calibrated case, they can be recovered up to a similarity (scale)
â Known as Euclidean/metric reconstruction
5 Images courtesy of Hartley & Zisserman http://www.robots.ox.ac.uk/~vgg/hzbook/
Projective reconstruction
Metric reconstruction
Structure from motion
Problem Given ð images of ð fixed 3D points, estimate the ð projection matrices ðð and the ð points ð¿ð from the ð â ð correspondences ððð â ððð ⢠We can solve for structure and motion when
2ðð ⥠11ð + 3ð â 15 ⢠In the general/uncalibrated case, cameras and
points can only be recovered up to a projective ambiguity (ðï¿œðð = ðððâ1ðð¿ï¿œð)
⢠In the calibrated case, they can be recovered up to a similarity (scale)
â Known as Euclidean/metric reconstruction
6
⢠This problem has been studied extensively and several different approaches have been suggested
⢠We will take a look at a couple of these â Sequential structure from motion â Bundle adjustment
Sequential structure from motion
7
Structure ð¿ð
Cameras ðð
ð
ð ⢠Initialize motion from two images
â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
Sequential structure from motion
8
Structure ð¿ð
Cameras ðð
ð
ð ⢠Initialize motion from two images
â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
9
Structure ð¿ð
Cameras ðð
ð
New points observed
New observations of old points
ð ⢠Initialize motion from two images
â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
10
Structure ð¿ð
Cameras ðð
ð
Extended structure
Refine old structure by triangulation over 3 views
ð ⢠Initialize motion from two images
â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
11
Structure ð¿ð
Cameras ðð
ð
ð ⢠Initialize motion from two images
â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
12
Structure ð¿ð
Cameras ðð
ð
Extend structure
ð
Refine structure
⢠Initialize motion from two images â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
13
Structure ð¿ð
Cameras ðð
ð
ð
⢠Initialize motion from two images â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
Sequential structure from motion
⢠Initialize motion from two images â ð¹ â ð1,ð2 â ðž â ð1,ð2 = ðŸ1 ðŒ ð ,ðŸ2 ð 21 ð21
⢠Initialize the 3D structure by triangulation
⢠For each additional view
â Determine the projection matrix ðð, e.g. from 2D-3D correspondences ððð â ð¿ð
â Refine and extend the 3D structure by triangulation
⢠The resulting structure and motion can be
refined in a process known as bundle adjustment
14
Structure ð¿ð
Cameras ðð
ð
ð
Bundle adjustment
⢠Non-linear method that refines structure and motion by minimizing the sum of squared reprojection errors
ð = ᅵᅵð ðï¿œðð ,ððð¿ï¿œð2
ð
ð=1
ð
ð=1
⢠Camera calibration can be solved as part of bundle
adjustment by including intrinsic parameters and skew parameters in the cost function
⢠Need initial estimates for all parameters! â 3 per 3D point â ~12 per camera depending on parameterization â Some intrinsic parameters, like the focal length, can
be initialized from image EXIF data
15
Bundle adjustment
16
⢠There are several strategies that deals with the potentially extreme number of parameters
⢠Reduce the number of parameters by not including all the views and/or all the points
â Perform bundle adjustment only on a subset and compute missing views/points based on the result
â Divide views/points into several subsets which are bundle adjusted independently and merge the results
⢠Interleaved bundle adjustment
â Alternate minimizing the reprojection error by varying only the cameras or only the points
â This is viable since each point is estimated independently given fixed cameras, and similarly each camera is estimated independently from fixed points
Bundle adjustment
17
The sparse structure of the Jacobian matrix for a bundle adjustment problem with 3 cameras and 4 3D points
Figure courtesy of Hartley & Zisserman http://www.robots.ox.ac.uk/~vgg/hzbook/
⢠Sparse bundle adjustment â For each iteration, iterative minimization methods
need to determine a vector ð« of changes to be made in the parameter vector
â In Levenberg-Marquardt each such step is determined from the equation
ðœððœ + ððŒ ð« = âðœðð where ðœ is the Jacobian matrix of the cost function and ð is the vector of errors
â For the bundle adjustment problem the Jacobian matrix has a sparse structure that can be exploited in computations
Bundle adjustment
⢠Sparse bundle adjustment â For each iteration, iterative minimization methods
need to determine a vector ð« of changes to be made in the parameter vector
â In Levenberg-Marquardt each such step is determined from the equation
ðœððœ + ððŒ ð« = âðœðð where ðœ is the Jacobian matrix of the cost function and ð is the vector of errors
â For the bundle adjustment problem the Jacobian matrix has a sparse structure that can be exploited in computations
⢠Combined with parallel processing the before
mentioned strategies has made it possible to solve extremely large SfM problems
18
Bundle adjustment
⢠Sparse bundle adjustment â For each iteration, iterative minimization methods
need to determine a vector ð« of changes to be made in the parameter vector
â In Levenberg-Marquardt each such step is determined from the equation
ðœððœ + ððŒ ð« = âðœðð where ðœ is the Jacobian matrix of the cost function and ð is the vector of errors
â For the bundle adjustment problem the Jacobian matrix has a sparse structure that can be exploited in computations
⢠Combined with parallel processing the before
mentioned strategies has made it possible to solve extremely large SfM problems
⢠S. Agarwal et al, Building Rome in a Day, 2011 â Cluster of 62-computers â 150 000 unorganized images from Rome â ~37 000 image registered â Total processing time ~21 hours â SfM time ~7 hours
⢠J. Heinly et al, Reconstructing the World in Six
Days, 2015 â 1 dual processor PC with 5 GPUâs (CUDA) â ~96 000 000 unordered images spanning the
globe â ~1.5 000 000 images registered â Total processing time ~5 days â SfM time ~17 hours
19
Bundle adjustment
⢠SBA â Sparse Bundle Adjustment â A generic sparse bundle adjustment C/C++
package based on the Levenberg-Marquardt algorithm
â Code (C and Matlab mex) available at http://www.ics.forth.gr/~lourakis/sba/
â CVSBA is an OpenCV wrapper for SBA www.uco.es/investiga/grupos/ava/node/39/
⢠Ceres
â By Google (used in production since 2010) â A C++ library for modeling and solving large,
complicated optimization problems like SfM â Homepage: www.ceres-solver.org â Code available on GitHub
https://github.com/ceres-solver/ceres-solver
20
⢠GTSAM â Georgia Tech Smoothing and Mapping â A C++ library based on factor graphs that is well
suited for SfM ++ â Code (C++ library and Matlab toolbox) available at
https://borg.cc.gatech.edu/borg/download
⢠g2o â General Graph Optimization â Open source C++ framework for optimizing graph-
based nonlinear error functions â Homepage: https://openslam.org/g2o.html â Code available on GitHub
https://github.com/RainerKuemmerle/g2o
Bundle adjustment
⢠Bundler â A structure from motion system for unordered
image collections written in C and C++ â SfM based on a modified version SBA (default) or
Ceres â Homepage:
http://www.cs.cornell.edu/~snavely/bundler/ â Code available on GitHub
https://github.com/snavely/bundler_sfm
⢠VisualSfM
â A GUI application for 3D reconstruction using structure from motion
â Output works with other tools that performs dense 3D reconstruction
â Homepage: http://ccwu.me/vsfm/
21
⢠RealityCapture â A state-of-the-art photogrammetry software that
automatically extracts accurate 3D models from images, laser-scans and other input
â Homepage: https://www.capturingreality.com/
Example Holmenkollen 2-view SfM
22
More than 10 000 points are registered in addition to the 2 cameras
Example Holmenkollen 2-view SfM
23
Example Holmenkollen 2-view SfM
24
Example Holmenkollen 3-view SfM
25
More than 10 000 points are registered in addition to the 2 cameras More than 16 000 points are registered in addition to the 3 cameras
Example Holmenkollen 4-view SfM
26
More than 10 000 points are registered in addition to the 2 cameras
More than 16 000 points are registered in addition to the 3 cameras
More than 20 000 points are registered in addition to the 4 cameras
Example Holmenkollen 110-view SfM
27
More than 10 000 points are registered in addition to the 2 cameras
More than 16 000 points are registered in addition to the 3 cameras
More than 20 000 points are registered in addition to the 4 cameras ~470 000 points are registered in addition to the 110 cameras
Example Holmenkollen 110-view SfM
28
More than 10 000 points are registered in addition to the 2 cameras
More than 16 000 points are registered in addition to the 3 cameras
More than 20 000 points are registered in addition to the 4 cameras ~470 000 points are registered in addition to the 110 cameras Maximal reprojection error ~3 pixels Mean reprojection error ~0.7 pixels
Summary
⢠Structure from motion â Sequential SfM â Bundle adjustment
⢠Additional reading:
â Szeliski: 7.3-7.5
⢠Optional reading: â Snavely N. Seitz S. M., Szeliski R., Modeling the
World from Internet Photo Collections, 2007 â S. Agarwal et al, Building Rome in a Day, 2011 â J. Heinly et al, Reconstructing the World in Six
Days, 2015
29
ð = ᅵᅵð ðï¿œðð ,ððð¿ï¿œð2
ð
ð=1
ð
ð=1