recent work in image-based rendering from unstructured image collections and remaining challenges...
TRANSCRIPT
Recent work in image-based rendering from unstructured image collections and remaining challenges
Sudipta N. Sinha Microsoft Research, Redmond, USA
• http://www.photosynth.net/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e
Image-based maps
• Structure from motion (Sfm)
• Robust depth-map estimation
• Rendering
Key Steps
• Structure from motion (Sfm)
• Robust depth-map estimation
• Image-based navigation
Recent results
A multi-stage linear approach to structure from motionSinha, Steedly & Szeliski, RMLE –ECCV workshop 2010
Piecewise planar stereo for image-based renderingSinha, Steedly & Szeliski, ICCV 2009
Image-based walkthroughs from incremental and partial scene reconstructions Kumar, Ahsan, Sinha & Jawahar, BMVC 2010
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pairPose estimation, triangulationRefinement
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pairPose estimation, triangulationRefinement
Contributions• Vanishing point (VP) constraints reduces drift in rotations
– more accurate than [Govindu’04, Martinec’07] for urban scenes.– Faster pairwise matching + geometric verification
• New practical linear structure and translation estimation– more stable than the known linear method [Rother’03]– robust to outliers in 2D observations– easy to parallelize– faster than sequential Sfm
– much faster than L∞ - methods
Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE
workshop)
Vanishing Point (VP) Detection
Pair Matching2 – VP + 2 point RANSAC VP tracks
relative rotationsFeature Extraction
VPs
interest pts
Images
GlobalRotation
Estimation
Linear Reconstruction
2-view Reconstruction
Robust Alignment
Global Scale & Translation
Estimation
VP tracks
relative rotations
global cameraorientations
relative pose estimates
Full Sfm initialization Final Bundle Adjustment
Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE
workshop)
Results
Timings
Break-up of Timings
Comparison with sequential Sfm
STREET sequence
HALLWAY sequence
OURS (65 cams, 52K pts)
before Bundle Adjustment BUNDLER (65 cams, 22K pts)
BUNDLER (139 cams, 13K pts) OURS (184 cams, 27K pts)
Comparison with sequential Sfm
Piecewise Planar Stereo for image-based rendering
Graph-cut based energy minimization
Sinha et. al. ICCV 2009
Piecewise Planar Stereo for image-based rendering
Sinha et. al. ICCV 2009
Planar Stereo Results
also handle non-planar scenes now ...
Piecewise Planar Stereo for image-based rendering
• Skip global scene reconstruction (Sfm) step, • Generate several overlapping, partial
reconstructions instead.• During navigation, jump
between local coordinate frames.• Scales easily, also parallelizable• Incremental matching & reconstruction
(images appear over time)
Image-based walkthroughs from incremental and partial scene reconstructions Kumar et. al. BMVC
2010
Fort sequence (~5800 images)
• Accuracy vs. Connectedness• Reliable results from sparse, unstructured imagery
– wide-baseline matching is still difficult
• Representations: – metric vs. topological reconstructions ? hybrid ?
• Reconstructing Indoors– Bottlenecks: doorways, corridors.– fewer features, non-Lambertian surfaces
Existing issues in unstructured Sfm
• Acquisition– Images vs. video– Short-term dynamics vs. long-term dynamics
• Need truly incremental Sfm– Start with scratch but keep going … ?– Interleaving matching, Sfm and dense stereo– Hybrid matching (2D—2D , 2D – 3D, 3D – 3D)
Dynamic Image-based Maps: Challenges
• Temporal appearance changes– Illumination:
• day/night, seasons, weather, lights on/off• Cyclic, predictable
– Albedo changes• Store-fronts, ads-billboards,• irreversible
• Geometric changes: – temporary vs. permanent
• Mid-level features for higher level recognition
Dynamic Image-based Maps: Challenges
Questions ?