sensor fusion for semantic segmentation of urban...
TRANSCRIPT
![Page 1: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/1.jpg)
Sensor Fusion for Semantic Segmentation of Urban Scenes
Richard Zhang1, Stefan A. Candra1, Kai Vetter12, Avideh Zakhor1
1University of California, Berkeley2Lawrence Berkeley National Laboratory
Presented at ICRA, May 2015
1
![Page 2: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/2.jpg)
Goal
• Extract semantic information by fusing multiple modalities• Image
• LiDAR scan
• Challenges• Incorporate information from
multiple scales
• Fuse information from multiple modalities with non-overlapping sensor coverage
2
![Page 3: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/3.jpg)
Top Level Pipeline
3
![Page 4: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/4.jpg)
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
4
![Page 5: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/5.jpg)
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
Cadena and Košecká. ICRA 2014.
Single segmentationimage only
Simple Simple Early fusion
5
![Page 6: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/6.jpg)
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
Cadena and Košecká. ICRA 2014.
Single segmentationimage only
Simple Simple Early fusion
Oursmultiple segmentations
both domainsDescriptive Descriptive Late fusion
6
![Page 7: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/7.jpg)
Segmentation
• Treating each pixel/point separately is computationally intensive
• Under-segmentation & Over-segmentation errors
Multiple segmentations performed to get superpixels/supervoxels
7
![Page 8: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/8.jpg)
Segmentation
• Treating each pixel/point separately is computationally intensive
• Under-segmentation & Over-segmentation errors
Multiple segmentations performed to get superpixels/supervoxels
8
![Page 9: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/9.jpg)
Feature Extraction
9
Point cloud supervoxel featuresImage superpixel features
![Page 10: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/10.jpg)
Early FusionImage
features
Point cloudfeatures
Image-only segmentation
10
Imageinformation
only
![Page 11: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/11.jpg)
Early FusionImage
features
Point cloudfeatures
Image-only segmentation
Point cloud-onlysegmentation
11
Imageinformation
only
Point cloudinformation
only
![Page 12: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/12.jpg)
Early FusionImage
features
Point cloudfeatures
Overlappingsensor
coverage
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
12
Imageinformation
only
Point cloudinformation
only
![Page 13: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/13.jpg)
Early FusionImage
features
Point cloudfeatures
Overlappingsensor
coverage
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
Large feature spaceLess training data
13
Imageinformation
only
Point cloudinformation
only
![Page 14: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/14.jpg)
Late Fusion
Imagefeatures
Point cloudfeatures
Image-onlysegmentation
Overlappingsensor
coverage
Point cloud onlysegmentation
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
14
![Page 15: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/15.jpg)
Late Fusion
Imagefeatures
Point cloudfeatures
Image-onlysegmentation
Overlappingsensor
coverage
Point cloud onlysegmentation
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
15
Mid-level
feature
![Page 16: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/16.jpg)
Fusion
PC onlyPoint cloud unimodal segmentation
Ground Truth
16
Image unimodal segmentation
![Page 17: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/17.jpg)
Fusion
PC only
Image unimodal segmentation
Ground Truth
Point cloud unimodal segmentation(projected onto image)
17
![Page 18: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/18.jpg)
Fusion
Ground Truth
Late-fused multi-modal segmentation
18
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
![Page 19: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/19.jpg)
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
Fusion
Ground Truth
Late-fused multi-modal segmentation
19
![Page 20: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/20.jpg)
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
Fusion
Ground Truth
Late-fused multi-modal segmentation
20
![Page 21: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/21.jpg)
Fusion
Ground Truth
Late-fused multi-modal segmentation
Fused segmentation & Post-processingConditional Random Field (CRF) 21
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
![Page 22: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/22.jpg)
Fusion
22
PC only Image only FusedPerformance on overlapping region
![Page 23: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/23.jpg)
Fusion
23
PC only Image only FusedPerformance on overlapping region
![Page 24: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/24.jpg)
Fusion
24
PC only Image only FusedPerformance on overlapping region
![Page 25: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/25.jpg)
Fusion
25
PC only Image only FusedPerformance on overlapping region
![Page 26: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/26.jpg)
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 26
![Page 27: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/27.jpg)
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 27
![Page 28: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/28.jpg)
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 28
![Page 29: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/29.jpg)
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 29
![Page 30: Sensor Fusion for Semantic Segmentation of Urban Scenesrichzhang.github.io/index_files/pres_icra2015.pdf · Sensor Fusion for Semantic Segmentation of Urban Scenes Richard Zhang 1,](https://reader034.vdocuments.site/reader034/viewer/2022051913/6003b82ae2f4d218c301565d/html5/thumbnails/30.jpg)
Thank you!
30