local-level 3d deep learningandyz/slides/3dmatch_marvin_tutorial_talk... · datasets which should...
TRANSCRIPT
Local-Level 3D Deep Learning Andy Zeng
1
Matching 3D Data
2Image Credits: Song and Xiao, Tevs et al.
Matching 3D Data
Reconstruction
Image Credits: Song and Xiao, Tevs et al. 3
Matching 3D Data
Reconstruction Shape retrievalObject pose estimation
4Image Credits: Song and Xiao, Tevs et al.
Matching 3D Data
Reconstruction Shape retrievalObject pose estimation
Aligning deformable shapes
5Image Credits: Song and Xiao, Tevs et al.
Matching 3D DataEstablish 3D geometric correspondences
6
Matching 3D DataEstablish 3D geometric correspondences
Find interesting 3D features Match 3D features7
Matching 3D Features in Scanning Data is Hard
Partial and noisy scan data
8
Matching 3D Features in Scanning Data is Hard
Viewpoint variancePartial and noisy scan data
9
Matching 3D Features in Scanning Data is Hard
Viewpoint variancePartial and noisy scan data
10
Matching 3D Features in Scanning Data is Hard
Viewpoint variancePartial and noisy scan data
Traditional hand-crafted 3D feature descriptors do not work well!11
Solution: Let the data speak for itself!
12
Solution: Let the data speak for itself!http://3dmatch.cs.princeton.edu/
3DMatch: 3D ConvNet that recognizes correspondences in 3D scan data
1
2
13
3D Data Representation
Use truncated distance fields (TDF)
Image Credits: Song and Xiao14
3D Data Representation
Use truncated distance fields (TDF)
Intermediate 3D Representation15
Image Credits: Song and Xiao
3D Data Representation
Use truncated distance fields (TDF)
Intermediate 3D Representation Enables 3D Convolution16
Image Credits: Song and Xiao
3D Data Representation
17
3D Data Representation
18
Metric Network vs. L2 Distance
19
Metric Network vs. L2 Distance
L2 contrastive loss
20
Metric Network vs. L2 Distance
L2 contrastive loss
Use metric network for accuracy, use L2 distance for speed21
Generating Training Data Automatically
Manually label geometric correspondences? Too much work!
22
Generating Training Data Automatically
Manually label geometric correspondences? Too much work!
“Think of all those maps that we've built using large-scale SLAM and all those correspondences that these systems provide — isn’t that a clear path for building terascale image-image "association" datasets which should be able to help deep learning?”
“The basic idea is that today's SLAM systems are large-scale correspondence engines which can be used to generate large-scale datasets, precisely what needs to be fed into a deep ConvNet.”
Newcombe’s Proposal: Use SLAM to help Deep Learning
23Image Credits: Malisiewicz et al.
Generating Training Data Automatically
Manually label geometric correspondences? Too much work!
“The basic idea is that today's SLAM systems are large-scale correspondence engines which can be used to generate large-scale datasets, precisely what needs to be fed into a deep ConvNet.”
Newcombe’s Proposal: Use SLAM to help Deep LearningTomasz Malisiewicz’s Computer Vision BlogICCV’s Future of Real-Time SLAM Workshop
Solution: Use existing 3D reconstructions to fuel correspondence labels!
Image Credits: Malisiewicz et al.24
Generating Training Data Automatically
Solution: Use existing 3D reconstructions to fuel correspondence labels!
Image Credits: Shotton et al.25
26
3DMatch for Reconstruction
27
3DMatch for Loop Closures
28
3DMatch for Loop Closures
29
3DMatch for Loop Closures
30
3DMatch for Loop Closures
31
3DMatch for 3D Reconstruction
32
3DMatch for Other Applications
Shape retrievalObject pose estimation
33
Evaluation: 3DMatch vs. Others
Correspondence
34
Evaluation: 3DMatch > Others
Correspondence Geometric Registration
35
Keypoint Selection Does Not Matter
36
Keypoint Selection Does Not Matter
The choice of keypoints do not matter much.
37
Conclusion
38
3DMatchhttp://3dmatch.cs.princeton.edu/
3D ConvNet that recognizes correspondences in 3D scan data
39