dataset generation and benchmarking of slam algorithms for robotics...
TRANSCRIPT
![Page 1: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/1.jpg)
Dataset Generation and Benchmarking of SLAM Algorithms for Robotics and VR/AR
(ICRA 2019 workshop)
Jieum HyunKAIST Urban Robotics Lab.
![Page 2: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/2.jpg)
Invited talk: Prof. Davide Scaramuzza
Benchmarking SLAM: Current Status and the Road Ahead
There are more and more VIO-VISLAM algorithms
But, how do we compare them?
1
![Page 3: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/3.jpg)
Invited talk: Prof. Davide Scaramuzza
Example Real-world Datesets
2
![Page 4: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/4.jpg)
Invited talk: Prof. Davide Scaramuzza
What metrics should be used?
3
![Page 5: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/5.jpg)
Invited talk: Prof. Davide Scaramuzza
Metric 1: Absolute Trajectory Error (ATE)
4Zhang, Zichao, and Davide Scaramuzza. "A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry." 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018.
![Page 6: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/6.jpg)
Invited talk: Prof. Davide Scaramuzza
Metric 2: Relative Trajectory Error (RTE)
5Zhang, Zichao, and Davide Scaramuzza. "A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry." 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018.
![Page 7: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/7.jpg)
Invited talk: Prof. Davide Scaramuzza
Accuracy: Trajectory Evaluation Toolbox
Designed to make trajectory evaluation easy Implements different alignment methods depending on the sensing
modalities: SE(3) for stereo VO, sim(3) for monocular, 4DOF for VIO Implements Absolute Trajectory Error and Relative Error
6
![Page 8: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/8.jpg)
Invited talk: Prof. Davide Scaramuzza
Benchmarking Efficiency Memory, CPU load, Processing time Depends not only on algorithm design but also implementation, platforms,
etc.
7
![Page 9: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/9.jpg)
Invited talk: Prof. Davide Scaramuzza
Robustness is the greatest challenge for SLAM
How can we quantify the robustness of algorithms to such situations?
8
![Page 10: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/10.jpg)
Invited talk: Prof. Davide Scaramuzza
Robustness
9
![Page 11: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/11.jpg)
Invited talk: Prof. Davide Scaramuzza
Data bias
10
![Page 12: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/12.jpg)
Invited talk: Prof. Davide Scaramuzza
UZH-FPV Drone Racing Dataset Contains data recorded by a drone flying up to over 20m/s indoors and
outdoors frown by a professional pilot. Contains frames, events, IMU, and Ground Truth from a Robotic Total Station
Video link : https://youtu.be/G5w4ZcEzvoo
11
![Page 13: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/13.jpg)
Invited talk: Prof. Davide Scaramuzza
Conclusion
12
![Page 14: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/14.jpg)
Invited talk: Prof. Davide Scaramuzza
Checklist for reproducible SLAM results
13
![Page 15: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/15.jpg)
Invited talk: Prof. Davide Scaramuzza
How should we report results in papers?
14
![Page 16: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/16.jpg)
InteriorNet
Mega-scale, Multi-sensor, Photo-realistic Indoor Scene Dataset
15
Li, Wenbin, et al. "InteriorNet: Mega-scale multi-sensor photo-realistic indoor scenes dataset." arXiv preprint arXiv:1809.00716 (2018).
![Page 17: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/17.jpg)
InteriorNet
Mega-scale, Multi-sensor, Photo-realistic Indoor Scene Dataset
Associated with NYU40 labels: wall, floor, bed, chair, table etc.
16
Li, Wenbin, et al. "InteriorNet: Mega-scale multi-sensor photo-realistic indoor scenes dataset." arXiv preprint arXiv:1809.00716 (2018).
![Page 18: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/18.jpg)
AI Habitat
AI Habitat enables training of embodied AI agents (virtual robots) in a highly photorealistic & efficient 3D simulator
17
Savva, Manolis, et al. "Habitat: A platform for embodied ai research." arXiv preprint arXiv:1904.01201 (2019).
![Page 19: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/19.jpg)
AI Habitat
AI Habitat enables training of embodied AI agents (virtual robots) in a highly photorealistic & efficient 3D simulator
18
Savva, Manolis, et al. "Habitat: A platform for embodied ai research." arXiv preprint arXiv:1904.01201 (2019).
![Page 20: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics …urserver.kaist.ac.kr/seminar/2019_ICRA2019_JieumHyun.pdf · 2020-02-13 · Dataset Generation and Benchmarking](https://reader033.vdocuments.site/reader033/viewer/2022042712/5fa5e24ed7b7a5224e2fe383/html5/thumbnails/20.jpg)
For more information
ICRA 2019 workshop/tutorial page link
InteriorNet homepage
Habitat homepage
19