computer vision group technical university of munich jakob...
TRANSCRIPT
![Page 1: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/1.jpg)
1
Computer Vision Group
Technical University of MunichJakob Engel
Semi-Dense Direct SLAMJakob Engel, Daniel Cremers
David Caruso, Thomas Schöps, Lukas von Stumberg, Vladyslav Usenko, Jörg Stückler, Jürgen Sturm
Technical University Munich
Jakob Engel Semi-Dense Direct SLAM
![Page 2: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/2.jpg)
2
Overview
Jakob Engel Semi-Dense Direct SLAM
LSD-SLAM Omnidirectional Stereo (+affine lighting)
Stereo + IMU Quadcopter!
![Page 3: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/3.jpg)
3
OverviewInput Video640x480 @ 30Hz
TrackingSE(3) alignment
to current KF
Depth Estimation
Current KF
Take KF?
Createnew KF
Refine KF
Add to MapSim(3) alignmentto all nearby KFs
Optional: FabMap for large loops
Map OptimizationSim(3) pose-graph
Jakob Engel Semi-Dense Direct SLAM
![Page 4: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/4.jpg)
4
Direct Tracking
KF image KF depth back-warped new frame
Camera Pose
minimize using Gauss-Newton / LM Algorithm
Coarse-to-fine + Huber norm + statistical norm.
Jakob Engel Semi-Dense Direct SLAM
![Page 5: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/5.jpg)
filter over many (small-baseline) stereo-correspondences.
small baseline + epipolar constraint + prior-> small search region („track “ instead of „detect “)
only use „good“ (sufficiently constrained) pixel.
Edge-preserving smoothing
Distance-based KF selection
5
Depth Estimation
image inverse depth inverse depth variance
Engel, Sturm, Cremers; ICCV ´13Jakob Engel Semi-Dense Direct SLAM
![Page 6: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/6.jpg)
with (warped point)
6
Global Mapping
Direct Tracking with scale (on Sim(3)):
Optimize pose-graph on Sim(3)
+ GN optimization + multi-resolution + Huber norm + statistical norm.
Jakob Engel Semi-Dense Direct SLAM
![Page 7: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/7.jpg)
7
LSD-SLAM Result
ECCV-sequence: 7 minutes, 640x480@50fps:25.000 Tracked Frames, 450 Keyframes; 7.000 Constraints; 51 Million Points
Engel, Schöps, Cremers; ECCV ´14
Jakob Engel Semi-Dense Direct SLAM
https://www.youtube.com/watch?v=isHXcv_AeFg
![Page 8: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/8.jpg)
8
Omni LSD-SLAM
A wider field of view is useful...but pinhole model does not allow it.
Pinhole, 120° horiz. FoV
Can we use a different model?
• central system• closed-form projection• closed form unprojection• large field of view (>180°)
Caruso, Engel, Cremers; IROS ´15Jakob Engel Semi-Dense Direct SLAM
![Page 9: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/9.jpg)
9
Omni LSD-SLAM1. Unified Omnidirectional Model 2. Piecewise Pinhole Model
185° FoV185° FoV
+ radio-tangential distortion, removed by pre-processingJakob Engel Semi-Dense Direct SLAM
![Page 10: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/10.jpg)
10
Omni LSD-SLAM
1. estimate and filter distance instead of depth2. inverse compositional LK instead of forward compositional
Piecewise Pinhole Model: Tracking: straight-forwardMapping: ep. lines are straightUgly edge-handling, artificial model.
Unified Model: Tracking: straight-forward (more costly!)Mapping: ep. lines are conics -> costly.Clean model, close to physical lenses.
Caruso, Engel, Cremers; IROS ´15Jakob Engel Semi-Dense Direct SLAM
![Page 11: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/11.jpg)
11
Omni LSD-SLAM
Large-Scale Direct SLAM for Omnidirectional CamerasCaruso, Engel, Cremers; IROS ´15
Jakob Engel Semi-Dense Direct SLAM
https://youtu.be/v0NqMm7Q6S8
![Page 12: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/12.jpg)
12
Omni LSD-SLAM
T1 T2 T3 T4 T5
Pinhole 57.8 22.8 8.3 60.5 55.0
Multi-Pinhole 4.9 6.6 4.6 4.2 5.4
Unified 5.3 5.1 4.6 4.5 3.6
Absolute translational RMSE (cm) to MoCap groundtruthdataset + groundtruth at http://vision.in.tum.de/omni-lsdslam
Caruso, Engel, Cremers; IROS ´15Jakob Engel Semi-Dense Direct SLAM
![Page 13: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/13.jpg)
13
Stereo LSD-SLAMHow about Stereo SLAM / VO?
- Get absolute scale- Initialization instantaneous- No issues with strong rotation- Often very practical
Add stereo disparity observations Tracking still „monocular“ Pose-graph again in SE(3)
Engel, Stückler, Cremers; IROS ´15Jakob Engel Semi-Dense Direct SLAM
![Page 14: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/14.jpg)
14
Stereo LSD-SLAM
(left) image (left) inverse depth inverse depth variance
Gaussian on inverse depth for each pixel in left image. Fusing stereo observations from Temporal Stereo (left image to next left image ) Static Stereo (left image to right image ) Temporal-Static Stereo (static stero in next
frame to ) Matching, Bayesian Fusion and Regularization as in LSD-SLAM.
Jakob Engel Semi-Dense Direct SLAM
![Page 15: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/15.jpg)
15
Stereo LSD-SLAM
no information from static stereo
no information from temporal stereo
Jakob Engel Semi-Dense Direct SLAM
![Page 16: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/16.jpg)
16
Affine Lighting
Approach: Make error function invariant to affine lighting changes:
Need to be careful with outliers!
Jakob Engel Semi-Dense Direct SLAM
Engel, Stückler, Cremers; IROS ´15
![Page 17: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/17.jpg)
17
Stereo LSD-SLAM
Large-Scale Direct SLAM with Stereo CamerasEngel, Stückler, Cremers; IROS ´15
Jakob Engel Semi-Dense Direct SLAM
https://youtu.be/oJt3Ln8H03s
![Page 18: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/18.jpg)
18
Stereo-Inertial LSD-SLAM
Tight IMU integration:
+ windowed marginalization
Usenko, Engel, Stückler, Cremers
How about Visual-Inertial Integration?
Jakob Engel Semi-Dense Direct SLAM
![Page 19: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/19.jpg)
19
Stereo-Inertial LSD-SLAM
Stereo Camera + tight IMU integrationUsenko, Engel, Stückler, Cremers
Jakob Engel Semi-Dense Direct SLAM
Soon onhttps://youtu.be/X_Wv-55Ulv8?list=PLTBdjV_4f-
EJnwIMsY1T0LwylRM7s_HcL
![Page 20: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/20.jpg)
20
Stereo-Inertial LSD-SLAM
Usenko, Engel, Stückler, Cremers
Jakob Engel Semi-Dense Direct SLAM
slow motion
fast motion
![Page 21: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/21.jpg)
21
Stereo-Inertial LSD-SLAM
* Results, dataset & calibration from „Keyframe-based visual–inertial odometry using nonlinear optimization”, Leutenegger, Lynen, Bosse, Siegwart, Furgale, IJRR’14
Usenko, Engel, Stückler, Cremers
Jakob Engel Semi-Dense Direct SLAM
![Page 22: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/22.jpg)
22
Autonomous Quadrotor
von Stumberg, Usenko, Engel, Stückler, Cremers
Jakob Engel Semi-Dense Direct SLAM
Soon onhttps://youtu.be/X_Wv-55Ulv8?list=PLTBdjV_4f-
EJnwIMsY1T0LwylRM7s_HcL
![Page 23: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/23.jpg)
Extract & MatchFeatures
(SIFT / SURF / BRIEF /...)
Input Images
23
Track:min. reprojection error
(point distances)
Map:est. feature-parameters
(3D points / normals)
abstract images to feature observations
Input Images
Track:min. photometric error
(intensity difference)
Map: est. per-pixel depth
(semi-dense depth map)
keep full image
ComparisonFeature-Based Direct
Chiuso ’02, Nistér ’04, Eade ’06, Klein ’06, Davison ’07, Strasdat ’10, Mur-Artal ’14, ....
Matthies ’88, Hanna ’91, Comport ’06, Newcombe ’11, Engel ’13, ...
Jakob Engel Semi-Dense Direct SLAM
![Page 24: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/24.jpg)
24
Comparison
can only use & reconstruct corners can use & reconstruct whole image
Feature-Based Direct
faster slower (but good for parallelism)
flexible: outliers can be removedretroactively.
inflexible: difficult to remove outliers retroactively.
robust to inconsistencies in themodel/system (rolling shutter).
not robust to inconsistencies inthe model/system (rolling shutter).
decisions (KP detection) based on less complete information.
decision (linearization point) based on more complete information.
~20+ years of intensive research ~4 years of research (+5years 25 years ago)
no need for good initialization. needs good initialization.
Jakob Engel Semi-Dense Direct SLAM
![Page 25: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/25.jpg)
25
Resolution vs. Accuracy
Accuracy gracefully declines with resolution, while runtime greatly improves with resolution.
Where does the remaining error (1%) come from?
error vs. resolution on KITTI (00-10)
Jakob Engel Semi-Dense Direct SLAM
Engel, Stückler, Cremers; IROS ´15
![Page 26: Computer Vision Group Technical University of Munich Jakob ...wp.doc.ic.ac.uk/thefutureofslam/wp-content/uploads/sites/93/2015/1… · Omni LSD-SLAM 1. estimate and filter distance](https://reader034.vdocuments.site/reader034/viewer/2022050114/5f4b84e51af5ea08183fc3a4/html5/thumbnails/26.jpg)
26
Semi-Dense SLAM
Questions
Jakob Engel Semi-Dense Direct SLAM