object goal navigation using goal-oriented semantic...
TRANSCRIPT
![Page 1: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/1.jpg)
Object Goal Navigation using Goal-oriented Semantic Exploration
Devendra Singh Chaplot
AbhinavGupta
RuslanSalakhutdinov
DhirajGandhi
Winner CVPR 2020 Habitat ObjectNav Challenge Team Arnold (SemExp)
Webpage: https://devendrachaplot.github.io/projects/semantic-exploration
![Page 2: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/2.jpg)
Object Goal Navigation
2
![Page 3: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/3.jpg)
Object Goal Navigation
2
Object Goal: dining table
![Page 4: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/4.jpg)
Object Goal Navigation
2
Object Goal: dining table
Semantic Scene Understanding
Object detection and segmentation
Geometric Scene Understanding
Understanding navigable space
Passive
![Page 5: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/5.jpg)
Object Goal Navigation
2
Object Goal: dining table
Learning Semantic Priors Episodic Memory
Keeping track of explored and unexplored areas
Where is ‘dining table’ more likely to be found?
Semantic Scene Understanding
Object detection and segmentation
Geometric Scene Understanding
Understanding navigable space
Passive
Active
![Page 6: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/6.jpg)
Active Neural SLAM
Sensor Pose Reading (x′ t)
Observation (st)
Action (at)
Neural SLAM ( fSLAM)
Local Policy (πL)
Global Policy (πG)
Pose Estimate ( ̂xt)
Short-term goal (gs
t )
Long-term goal (gl
t)
fPlan
Map (mt)
3
![Page 7: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/7.jpg)
Active Neural SLAM
Sensor Pose Reading (x′ t)
Observation (st)
Action (at)
Neural SLAM ( fSLAM)
Local Policy (πL)
Global Policy (πG)
Pose Estimate ( ̂xt)
Short-term goal (gs
t )
Long-term goal (gl
t)
fPlan
Map (mt)
3
[Chaplot el al. ICLR-20]
![Page 8: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/8.jpg)
Incorporating Semantics
4
Obstacle Map Representation(Active Neural SLAM)
Obstacle Map (2 × M × M)
ObstaclesExplored Area
![Page 9: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/9.jpg)
Incorporating Semantics
4
Obstacle Map Representation(Active Neural SLAM)
Obstacle Map (2 × M × M)
ObstaclesExplored Area
Semantic Map (K × M × M)
Semantic categories (C)
ObstaclesExplored Area
Semantic Map Representation(SemExp)
K = C + 2
![Page 10: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/10.jpg)
RGB (It)
Depth (Dt)
Semantic Mapping
![Page 11: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/11.jpg)
Mask RCNN
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Semantic Mapping
![Page 12: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/12.jpg)
Mask RCNN
X Y Z
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Point Cloud
Semantic Mapping
![Page 13: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/13.jpg)
Mask RCNN
X Y Z
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Point Cloud
C3C1 C2
Semantic Labels
Semantic Mapping
![Page 14: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/14.jpg)
Mask RCNN
X Y Z
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Point Cloud
C3C1 C2
Semantic Labels
Semantic Mapping
Voxel (C + 1) × H × M × M
![Page 15: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/15.jpg)
All cells
Mask RCNN
X Y Z
sum across height
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Point Cloud
C3C1 C2
Semantic Labels
Category-wise
All obstacles
Projection Map (C + 2) × M × M
Semantic categories (C)
ObstaclesExplored Area
Semantic Mapping
Voxel (C + 1) × H × M × M
![Page 16: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/16.jpg)
All cells
Mask RCNN
X Y Z
sum across height
RGB (It)
Depth (Dt)
First-person Semantic Predictions
Point Cloud
C3C1 C2
Semantic Labels
Category-wise
All obstacles
Projection Map (C + 2) × M × M
Semantic categories (C)
ObstaclesExplored Area
Semantic Map Prediction (C + 2) × M × M
Denoising Network
Semantic Mapping
Voxel (C + 1) × H × M × M
![Page 17: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/17.jpg)
SemExp Model Overview
Sensor Pose Reading (xt)
Object Goal ( “chair”)G =
Observation (RGBD)
(st)
![Page 18: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/18.jpg)
SemExp Model Overview
Sensor Pose Reading (xt)
Semantic Mapping
Semantic Map (mt)
Object Goal ( “chair”)G =
Observation (RGBD)
(st)
![Page 19: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/19.jpg)
SemExp Model Overview
Sensor Pose Reading (xt)
Semantic Mapping
Goal-Oriented Semantic Policy
Long-term goal (gt)
Semantic Map (mt)
Object Goal ( “chair”)G =
Observation (RGBD)
(st)
![Page 20: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/20.jpg)
SemExp Model Overview
Sensor Pose Reading (xt)
Semantic Mapping
Deterministic Local Policy (πL)
Goal-Oriented Semantic Policy
Long-term goal (gt)
Semantic Map (mt)
Object Goal ( “chair”)G =
Observation (RGBD)
(st)
Action (at)
![Page 22: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/22.jpg)
ObjectGoal Navigation Results
8
![Page 23: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/23.jpg)
ObjectGoal Navigation Results
Gibson
Random
RGBD + RL [1]
RGBD + Semantics + RL [2]
Classical Map + FBE
Active Neural SLAM [3]
SemExp
0 0.15 0.3 0.45 0.6
0.5440.446
0.4030.159
0.0820.004
Gibson
Random
RGBD + RL [1]
RGBD + Semantics + RL [2]
Classical Map + FBE
Active Neural SLAM [3]
SemExp
0 0.15 0.3 0.45 0.6
0.5440.446
0.4030.159
0.0820.004 0.004
0.027
0.049
0.124
0.145
0.199
Success Rate
*Adapted from [1] Savva et al. ICCV-19, [2] Mousavian et al. ICRA-19, [3] Chaplot el al. ICLR-20
SPL
8
![Page 24: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/24.jpg)
ObjectGoal Navigation Results
MP3D
Random
RGBD + RL [1]
RGBD + Semantics + RL [2]
Classical Map + FBE
Active Neural SLAM [3]
SemExp
0 0.15 0.3 0.45 0.6
0.360.321
0.3110.0310.037
0.005
MP3D
Random
RGBD + RL [1]
RGBD + Semantics + RL [2]
Classical Map + FBE
Active Neural SLAM [3]
SemExp
0 0.15 0.3 0.45 0.6
0.360.321
0.3110.0310.037
0.005 0.004
0.027
0.049
0.124
0.145
0.199
Success Rate
0.004
0.027
0.049
0.124
0.145
0.199
*Adapted from [1] Savva et al. ICCV-19, [2] Mousavian et al. ICRA-19, [3] Chaplot el al. ICLR-20
SPL
8
![Page 25: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/25.jpg)
Habitat Challenge Leaderboard
9
Test-standard Minival
Method SPL Success Dist SPL Success Dist
Arnold (SemExp) 0.071 0.179 8.818 0.246 0.467 3.334
Active Exploration 0.041 0.089 9.461 0.108 0.167 5.079
DD-PPO 0.021 0.062 9.316 - - -
Blue Ox 0.017 0.060 8.903 0.083 0.133 4.254
SRCB-robot-sudoer 0.002 0.004 10.276 0.124 0.233 4.848
PPO RGBD - - - 0 0 6.055
Random 0.000 0.000 10.330 0 0 6.379
![Page 26: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/26.jpg)
Real-world Transfer
10
See video at https://devendrachaplot.github.io/projects/semantic-exploration
![Page 27: Object Goal Navigation using Goal-oriented Semantic ...dchaplot/talks/cvpr20-semantic-exploration.pdfObject Goal Navigation using Goal-oriented Semantic Exploration Devendra Singh](https://reader034.vdocuments.site/reader034/viewer/2022050516/5f9ff144740ff301d5543e23/html5/thumbnails/27.jpg)
11
Object Goal Navigation using Goal-oriented Semantic ExplorationDevendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan SalakhutdinovCVPR 2020
Webpage: https://devendrachaplot.github.io/projects/semantic-exploration
Devendra Singh ChaplotWebpage: http://devendrachaplot.github.io/Email: [email protected]: @dchaplot
Thank you