images as graphs
DESCRIPTION
Images as graphs. Fully-connected graph node for every pixel link between every pair of pixels, p , q similarity w ij for each link. i. w ij. c. j. Source: Seitz. Segmentation by Graph Cuts. w. Break Graph into Segments Delete links that cross between segments - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/1.jpg)
i
Images as graphs
• Fully-connected graph– node for every pixel– link between every pair of pixels, p,q– similarity wij for each link
j
wijc
Source: Seitz
![Page 2: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/2.jpg)
Segmentation by Graph Cuts
• Break Graph into Segments– Delete links that cross between segments– Easiest to break links that have low cost (low similarity)
• similar pixels should be in the same segments• dissimilar pixels should be in different segments
w
A B C
Source: Seitz
![Page 3: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/3.jpg)
![Page 4: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/4.jpg)
Cuts in a graph
• Link Cut– set of links whose removal makes a graph disconnected– cost of a cut:
A B
One idea: Find minimum cut• gives you a segmentation• fast algorithms exist for doing this
Source: Seitz
![Page 5: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/5.jpg)
But min cut is not always the best cut...
![Page 6: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/6.jpg)
Cuts in a graph
A B
Normalized Cut• a cut penalizes large segments• fix by normalizing for size of segments
• volume(A) = sum of costs of all edges that touch A
Source: Seitz
![Page 7: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/7.jpg)
7
Finding Minimum Normalized-Cut• Finding the Minimum Normalized-Cut is NP-Hard.• Polynomial Approximations are generally used for
segmentation
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 8: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/8.jpg)
8
Finding Minimum Normalized-Cut
wherematrix, symmetric NNW
otherwise0 if,
22
iNjeejiWXjiFji XXFF
Proximity Spatial
similarity feature Image
ji
ji
XX
FF
wherematrix, diagonal NND j
jiWiiD ,,
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 9: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/9.jpg)
9
• It can be shown that
such that
• If y is allowed to take real values then the minimization can be done by solving the generalized eigenvalue system
Finding Minimum Normalized-Cut
Dyy
yWDyT
T
y
minmin cutN
0 and ,10 ,,1 D1yTbbiy
DyyWD
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 10: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/10.jpg)
10
Algorithm
• Compute matrices W & D• Solve for eigen vectors with the smallest eigen
values• Use the eigen vector with second smallest eigen value to
bipartition the graph• Recursively partition the segmented parts if necessary.
DyyWD
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 11: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/11.jpg)
Recursive normalized cuts1. Given an image or image sequence, set up a weighted
graph: G=(V, E)– Vertex for each pixel– Edge weight for nearby pairs of pixels
2. Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy
– Use the eigenvector with the second smallest eigenvalue to bipartition the graph
– Note: this is an approximation
4. Recursively repartition the segmented parts if necessary
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdfDetails:
![Page 12: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/12.jpg)
Normalized cuts results
![Page 13: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/13.jpg)
Graph cuts segmentation
![Page 14: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/14.jpg)
![Page 15: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/15.jpg)
![Page 16: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/16.jpg)
![Page 17: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/17.jpg)
![Page 18: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/18.jpg)
![Page 19: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/19.jpg)
![Page 20: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/20.jpg)
![Page 21: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/21.jpg)
![Page 22: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/22.jpg)
![Page 23: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/23.jpg)
Markov Random Fields
edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( y
Node yi: pixel label
Edge: constrained pairs
Cost to assign a label to each pixel
Cost to assign a pair of labels to connected pixels
![Page 24: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/24.jpg)
Solving MRFs with graph cuts
edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( y
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 25: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/25.jpg)
Solving MRFs with graph cuts
edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( y
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 26: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/26.jpg)
![Page 27: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/27.jpg)
![Page 28: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/28.jpg)
Graph cuts Boykov and Jolly (2001)
Image Min Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Foreground (source)
Background(sink)
Source: Rother
![Page 29: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/29.jpg)
Optimization Formulation - Boykov & Jolly ‘01
A – Proposed Segmentation
E(A) – Overall Energy R(A) – Degree to which
pixels fits model B(A) – Degree to which
the cuts breaks up similar pixels
- Balance A() and B()• Goal: Find Segmentation,
A, which minimizes E(A) 30
![Page 30: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/30.jpg)
Link Weights
• Pixel links based on color/intensity similarities
• Source/Target links based on histogram models of fore/background
31
![Page 31: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/31.jpg)
![Page 32: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/32.jpg)
Grab cuts and graph cuts
User Input
Result
Magic Wand (198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
Regions Boundary Regions & Boundary
Source: Rother
![Page 33: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/33.jpg)
![Page 34: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/34.jpg)
![Page 35: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/35.jpg)
![Page 36: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/36.jpg)
![Page 37: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/37.jpg)
![Page 38: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/38.jpg)
![Page 39: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/39.jpg)
![Page 40: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/40.jpg)
Colour Model
Gaussian Mixture Model (typically 5-8 components)
Foreground &Background
Background
Foreground
BackgroundG
R
G
RIterated graph cut
Source: Rother
![Page 41: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/41.jpg)
![Page 42: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/42.jpg)
![Page 43: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/43.jpg)
![Page 44: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/44.jpg)
![Page 45: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/45.jpg)
Moderately straightforward
examples
… GrabCut completes automatically GrabCut – Interactive Foreground Extraction 10
![Page 46: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/46.jpg)
Difficult Examples
Camouflage & Low Contrast Harder CaseFine structure
Initial Rectangle
InitialResult
GrabCut – Interactive Foreground Extraction 11
![Page 47: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/47.jpg)
Border Matting
Hard Segmentation Automatic Trimap Soft Segmentation
GrabCut – Interactive Foreground Extraction 23
to
![Page 48: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/48.jpg)
Input Bayes MattingChuang et. al. (2001)
Knockout 2Photoshop Plug-In
Comparison
GrabCut – Interactive Foreground Extraction 24
With no regularisation over alpha
Shum et. al. (2004): Coherence matting in “Pop-up light fields”
![Page 49: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/49.jpg)
Solve
Mean Colour Foreground
Mean ColourBackground
GrabCut – Interactive Foreground Extraction 25
Natural Image Matting
Ruzon and Tomasi (2000): Alpha estimation in natural images
![Page 50: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/50.jpg)
![Page 51: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/51.jpg)
Noisy alpha-profile
Border Matting
GrabCut – Interactive Foreground Extraction 26
1
0
Foreground
Mix
Back-ground
Foreground Mix Background
Fit a smooth alpha-profile with parameters
![Page 52: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/52.jpg)
Result using DP Border Matting
DP
t
Dynamic Programming
GrabCut – Interactive Foreground Extraction 27
Noisy alpha-profile Regularisation
t+1
![Page 53: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/53.jpg)
GrabCut Border Matting - Colour
Compute MAP of p(F|C,alpha) (marginalize over B)
To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work)
[Chuang et al. ‘01] Grabcut Border Matting
GrabCut – Interactive Foreground Extraction 28
![Page 54: Images as graphs](https://reader036.vdocuments.site/reader036/viewer/2022062310/56816487550346895dd65ebb/html5/thumbnails/54.jpg)
Results
GrabCut – Interactive Foreground Extraction 29