segmentation based multi- v iew stereo
DESCRIPTION
Segmentation Based Multi- V iew Stereo. Michal Jan čošek, Tomáš Pajdla. Problem formulation : input. Input. Output. Problem formulation : output. Input. Output. Previous work. [Furukawa07] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis , CVPR 2007 - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/1.jpg)
Center for Machine PerceptionDepartment of Cybernetics
Faculty of Electrical EngineeringCzech Technical University in Prague
Segmentation Based Multi-View Stereo
Michal Jančošek, Tomáš [email protected], [email protected]
![Page 4: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/4.jpg)
/
Previous work
[Furukawa07] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis, CVPR 2007 prematching, growing, filtering
[Tao00] H. Tao and H.S. Sawhney. Global Matching Criterion and Color Segmentation Based Stereo, ACV 2000 using segmentation for hypothesizing the continuous parts of a
scene
[Felzenszwalb04] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation, IJCV 2004. color segmantation
[Jancosek08] Jancosek M. and Pajdla T. Effective seed generation for 3D reconstruction, CVWW 2008 optimal 3D segment orientation
34
[email protected], [email protected]
4
![Page 5: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/5.jpg)
/
Pipeline overview : Prematching34
[email protected], [email protected]
5
InputInput Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)Filtering
(3D patches)OutputOutput =
![Page 6: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/6.jpg)
/
Pipeline overview : 3D segments34
[email protected], [email protected]
6
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)Filtering
(3D patches)OutputOutput =
![Page 7: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/7.jpg)
/
Pipeline overview : Filtering34
[email protected], [email protected]
7
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)OutputOutput =
![Page 8: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/8.jpg)
/
Pipeline overview : Final mesh construction 34
[email protected], [email protected]
8
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh construction
Filtering(3D patches)
Filtering(3D patches)Output =
Poisson Surface Reconstruction
![Page 9: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/9.jpg)
/
Pipeline overview : Prematching34
[email protected], [email protected]
9
InputInput Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)Filtering
(3D patches)OutputOutput =
![Page 10: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/10.jpg)
/
Prematching
Cameras are known (or computed [Martinec et al.]) Feature points detection (harris on a grid [Furukawa07]) Feature points are matched in each image pair (guided matching
by epigeoms). Matching is based on the NCC score of two 5x5 windows Mutually best matches are selected Tracks are constructed by grouping mutually best matches
34
[email protected], [email protected]
10
r = ²i( )
= center of gravity of the points in S
S
![Page 11: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/11.jpg)
/
Pipeline overview : 3D segments34
[email protected], [email protected]
11
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)Filtering
(3D patches)OutputOutput =
![Page 12: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/12.jpg)
/
3D segments34
[email protected], [email protected]
12
For each segment with some seed projections, we create an optimal 3D segment and set it as explored
Next we do a greedy searching of unexplored segments to explore more
![Page 13: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/13.jpg)
/
Optimal 3D segment creation34
[email protected], [email protected]
13
TP FPTP
TP – true positiveFP – false positive - explored segment - unexplored segment
![Page 14: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/14.jpg)
/
Optimal 3D segment creation34
[email protected], [email protected]
14
MNCC( )=
Ф = 45
Ѳ = 90
Ф = 75
Ѳ = 120
,
The goal is to find the global maximum of the criteria function
)),,(),,,((,,
,180,0180,0:
kvpTkvpTMNCCkf
RKKf
mkd
d
k = 0
![Page 15: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/15.jpg)
/
Optimal 3D segment creation
First, we estimate 3D segment orientation 3D segment orientation is estimated by gradient descent optimization
from the best sample out of 10x10 regular orientation samples [Jancosek08]
Next, we find the optimum position of 3D segment on the ray from reference camera center for the estimated orientation
34
[email protected], [email protected]
15
![Page 16: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/16.jpg)
/
Greedy searching of unexplored segments34
[email protected], [email protected]
16
TP FPTPTPTP
TP – true positiveFP – false positive - explored segment - unexplored segment
![Page 17: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/17.jpg)
/
Greedy searching of unexplored segments34
[email protected], [email protected]
17
TP FPTPTP TP TP
TP – true positiveFP – false positive - explored segment - unexplored segment
![Page 18: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/18.jpg)
/
Pipeline overview : Filtering34
[email protected], [email protected]
18
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh constructionFinal mesh
constructionFiltering
(3D patches)OutputOutput =
![Page 19: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/19.jpg)
/
Filtering34
[email protected], [email protected]
19
TP FPTP
TP – true positiveFP – false positive - explored segment - unexplored segment
![Page 20: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/20.jpg)
/
Pipeline overview : Final mesh construction 34
[email protected], [email protected]
20
InputInput Prematching(3D seeds)
Prematching(3D seeds)
Hypothesizing(3D segments)Hypothesizing(3D segments)
Final mesh construction
Filtering(3D patches)
Filtering(3D patches)Output =
Poisson Surface Reconstruction
![Page 33: Segmentation Based Multi- V iew Stereo](https://reader033.vdocuments.site/reader033/viewer/2022051316/56814a96550346895db7a3ff/html5/thumbnails/33.jpg)
/
Conclusions
Advantages Complete models Lack of texture is explained by planes Speed Possible to implement on GPU
Disadvantages Low accuracy
Future work MRF on volume around 3D segments
34
[email protected], [email protected]
33