image segmentation and normalized cutscvcl.mit.edu/sunslides/suns_c9_l1_imagesegmentation.pdf ·...
TRANSCRIPT
![Page 1: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/1.jpg)
Image segmentation and normalized cuts
Bryan C. Russell
![Page 2: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/2.jpg)
Gestalt school (1938)
Max Wertheimer
![Page 3: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/3.jpg)
Forsyth and Ponce. Computer Vision: a modern approach. Prentice Hall, 2003.
![Page 4: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/4.jpg)
Forsyth and Ponce. Computer Vision: a modern approach. Prentice Hall, 2003.
![Page 5: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/5.jpg)
Forsyth and Ponce. Computer Vision: a modern approach. Prentice Hall, 2003.
![Page 6: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/6.jpg)
![Page 7: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/7.jpg)
![Page 8: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/8.jpg)
Image segmentation
• Can group based on brightness, color, texture, spatial location, shape, size, orientation, motion, etc.
• How do we realize this notion of grouping computationally?
• Here’s one way…
![Page 9: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/9.jpg)
Agglomerative/merge clustering
![Page 10: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/10.jpg)
Popular segmentation methods
• Mean shift– Comanicu et al. 2002
• Spectral clustering– Shi et al. 2000
• Bayesian, MRF– Felzenszwalb 2004, Borenstein et al. 2004
![Page 11: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/11.jpg)
Image segmentation issues
• What is a good segmentation?
• What are we optimizing? Is the returned image segmentation optimal?
• Let us consider a graph-theoretic approach
![Page 12: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/12.jpg)
G = (V;E)
v1
v2
v3
v4
v5
e1,4
e1,3
e2,3e3,5
e2,1 e5,3
![Page 13: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/13.jpg)
G = (V;E)
v1
v2
v3
v4
v5
e1,4
e1,3
e2,3e3,5
e2,1 e5,3
vi - feature, e.g. pixel intensity, spatial locationei,j - indicates two features have nonzero similarity
![Page 14: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/14.jpg)
G = (V;E)
v1
v2
v3
v4
v5
e1,4
e1,3
e2,3e3,5
e1,2
ei,j; ej,i2 E
![Page 15: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/15.jpg)
v1
v2
v3
v4
v5
w(1; 4)
w(1; 2)
w(2; 3)w(3; 5)
w(1; 3)
- similarity score of pixels i and jw(i; j)
![Page 16: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/16.jpg)
Visualizing similarities
![Page 17: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/17.jpg)
More complex graph
![Page 18: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/18.jpg)
Toy problem: spatial grouping
![Page 19: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/19.jpg)
Toy problem: spatial grouping
w(i; j) / exp(−dist(i; j))
![Page 20: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/20.jpg)
Toy problem: spatial grouping
Task: cut the graph to produce a meaningful segmentation
![Page 21: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/21.jpg)
Toy problem: spatial grouping
One possible cut…
![Page 22: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/22.jpg)
Toy problem: spatial grouping
One idea: find set of edges with minimum total weightthat partitions the graph - MinCut
![Page 23: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/23.jpg)
Toy problem: spatial grouping
Problem: MinCut prefers isolated points
![Page 24: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/24.jpg)
Problem with MinCut
![Page 25: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/25.jpg)
Normalize the cut
Use ratio of cut to the volume of the set
BA B
![Page 26: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/26.jpg)
Normalize the cut
B
Ncut(A;B) = cut(A ,B )vol(A ) + cut(A ,B )
vol(B )
BA
![Page 27: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/27.jpg)
Normalized Cuts caveats
• Finding the exact solution is NP-hard
• Need to relax the problem to be continuous-valued and use iterative methods
![Page 28: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/28.jpg)
Laplacian matrix
D −W
Wi,j = w(i; j)
Find generalized eigenvectors
![Page 29: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/29.jpg)
Toy problem: synthetic image
![Page 30: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/30.jpg)
Real image
![Page 31: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/31.jpg)
![Page 32: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/32.jpg)
![Page 33: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/33.jpg)
![Page 34: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/34.jpg)
Automatic Photo Popup
Derek HoiemAlexei A. EfrosMartial Hebert
Carnegie Mellon University
![Page 35: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/35.jpg)
Robust Spatial Support
RGB Pixels Superpixels
[Felzenszwalb and Huttenlocher 2004]
• Safe oversegmentation of image• Better but not still not enough spatial support
![Page 36: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/36.jpg)
Cutting and Folding
• Construct 3D model• Texture map
![Page 37: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/37.jpg)
Noise Estimation from a Single Image
Ce Liu William T. Freeman Richard Szeliski Sing Bing Kang
![Page 38: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/38.jpg)
Segmentation-based Approach
Over-segmentation
![Page 39: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/39.jpg)
Test on Low and High Noiselow noise high noise
Red Green Blue Red Green Blue
![Page 40: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/40.jpg)
Segment scores for a given topic
White indicates low KL divergence
Car topic
Car topic
Russell et al. CVPR 2006.
![Page 41: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/41.jpg)
Results II. – LabelMe dataset1,554 images
Learn 20 topics
4 topics shown
Each topic shown by top 25 segments
Russell et al. CVPR 2006.
![Page 42: Image segmentation and normalized cutscvcl.mit.edu/SUNSlides/SUNS_C9_L1_ImageSegmentation.pdf · Image segmentation • Can group based on brightness, color, texture, spatial location,](https://reader031.vdocuments.site/reader031/viewer/2022022015/5b5204257f8b9a35278cc64c/html5/thumbnails/42.jpg)
Thank you
• Ncuts software:– http://www.cis.upenn.edu/~jshi/software/
• Pedro Felzenszwalb software:– http://people.cs.uchicago.edu/~pff/segment/