normalized cuts and image segmentation · 1 normalized cuts and image segmentation jianbo shi,...
TRANSCRIPT
1
Normalized Cuts and Image Segmentation
Jianbo Shi, UpennJitendra Malik, Berkeley
10/7/2003 2
Perceptual Grouping and Organization (Wertheimer)
SimilarityProximityCommon fateGood continuationPast experience
=Visual grouping
2
10/7/2003 3
History
10/7/2003 4
How to Partition?
There is no single answerPrior World Knowledge:
Low LevelCoherence of BrightnessColorTextureMotion
Mid or High LevelSymmetries of Objects
3
10/7/2003 5
How to Partition? (Cont’d)
inherently hierarchicalTREE STRUCTURE instead of flat partitioningOBJECTIVE:
low level cues for hierarchical partitions. High level knowledge for further partitioning.
10/7/2003 6
Tools
G=(V,E)An edge is formed between every pair (Complete graph)w(i,j) function of similarity.Partition V into disjoint sets V1, V2,…, Vm
Similarity within sets maximum.Similarity between sets minimum.
4
10/7/2003 7
Graph Vs. Image
What is the best criterion i.e. the weighting functionHow to compute efficiently
The graph is huge
10/7/2003 8
Definitions
Optimal bipartition is minimum cutMinimum cut
computed efficientlypartitions out small chunks
5
10/7/2003 9
Example of a Bad Partition
Similarity decreases as the distance increases
10/7/2003 10
New Definition: Normalized Cut
Normalize the cut.Still a disassociation (dissimilarity) measure.
Smaller the better.
6
10/7/2003 11
New Definition: Normalized Association
Total association (similarity) within groups.The bigger the better.
assoc(a,a): total connectivity within A.
10/7/2003 12
Normalized Cut and Normalized Association Are Related
7
10/7/2003 13
Formulation for Normal Cut
10/7/2003 14
System Linearization
8
10/7/2003 15
Setting up the matrices
d1 0 0 0 0 0 …… 0 0 d2 0 0 0 0 …… 0 D-W=
D=0 0 d3 0 0 0 …… 0……………………….0 0 0 0 0 0 …… dN
0 w12 w13 …. w1NW= w21 0 w23 …. w2N
……………………..wN1 wN2 ….. …. 0
dN…-wN3-wN2-wN1
……………
-w3N…d3-w32-w31
-w2N…-w23d2-w21
-w1N…-w13-w12d1
10/7/2003 16
Rayleigh Quotient
Generalized eigenvalue systemMinimized if y can be real.
Convert generalized eigensystem to a standard eigensystem:
Where
9
10/7/2003 17
Results
D-W is positive semidefiniteeigenvectors are perpendicular.
Second smallest eigenvector minimizes the normalized cut.
Rayleigh quotient
Constraints (to be justified later):y can take on either 1 or –b
10/7/2003 18
Problem
First constraint not satisfiedThe solution gives real valued eigenvectors.
Which pixels are above threshold, which ones are not?
10
10/7/2003 19
Algorithm
Set up G=(V,E)All the points in the image as nodesComplete graph.
Weights proportional to the similarity between two pixels.
Similarity measure is yet to be decided.
10/7/2003 20
Algorithm (cont’d)
Solve (D-W)x=λDx for smallest eigenvectorsMethod is yet to be decided.
Second smallest eigenvector bipartitions the graph.
Must decide the break point.Decide if current partition is good, if not recursively subdivide.
11
10/7/2003 21
Example: Brightness Image
Define the weighting function To build matrices D and W:
Sigma: 10 to 20 percent of the total range of the distance function.
10/7/2003 22
Example (Cont’d)
Solve for Transform into a standard eigenvalue problem
Normally takes O(n^3) operationsUse Lanczos method to reduce computation.
12
10/7/2003 23
Lanczos Method
Graphs are locally connected, thus sparseEigensystems are therefore sparse
Only few smallest eigenvectors neededPrecision requirement for eigenvectors is loose.
10/7/2003 24
Dividing Point for Eigenvector
Ideal case: second smallest eigenvector is integer. But in reality it takes real values
Use the median, orUse 0, orUse the value that makes Ncut(A,B) smallest.
13
10/7/2003 25
Stability Criterion
Sometimes an eigenvector takes on continues values.Not suitable for partitioning purpose
Hard to find a cut point Similar Ncut values.
Compute the histogram ratio between the minimum and maximum values in the bins.
10/7/2003 26
Drawbacks of 2-Way Cut
Stability criterion is good but it gets rid of subsequent eigenvectors
Recursive 2-way cut is inefficientuses only the second smallest eigenvector
14
10/7/2003 27
Simultaneous K-Way Cut with Multiple Eigenvectors
k-means is used for oversegmentation.Second step can vary
Greedy PruningGlobal Recursive Cut
10/7/2003 28
Greedy Pruning
Iteratively merge two segments until k segments are left.Each iteration choose the pair when merged minimizes the k-way Ncut criterion:
15
10/7/2003 29
Global Recursive Cut
Build a condensed graphEach node of the graph corresponds to one initial segmentWeights Defined as the total connection from one initial partition to anotherUse exhaustive search or generalized eigenvalue system for solution.
10/7/2003 30
Experiments
BrightnessColorTextureMotionFollowing equation always holds; it is F(i) that changes according to the application
16
10/7/2003 31
Similarity * Spatial Proximity
F(i)=1, when segmenting point setsI(i), the intensity value for segmenting scalar images[v,v.s.sin(h), v.s.cos(h)](i) for color images[|I*f1|, … , |I*fn|](i), for texture.
The weight is always zero if proximity criterion is not met, i.e. The pixels are more than r pixels apart.
10/7/2003 32
MOTION
Spatiotemporal neighborhood
Dm: motion distance
17
10/7/2003 33
Experimental Results
10/7/2003 34
Experimental Results
18
10/7/2003 35
Experimental Results
10/7/2003 36
19
10/7/2003 37
10/7/2003 38
Motion
20
10/7/2003 39
10/7/2003 40
21
10/7/2003 41
10/7/2003 42
Linearize the System
Xi > 0 if node i belongs to A, Xi<0 otherwise.
is the connection from node i to all nodes.
22
10/7/2003 43
Linearize the System (2)
The equation linearizes to:
D is a diagonal NxN matrix with d on its diagonal, W is the NxN weight matrix where w(i,j) is the weight of the edge between node i and j.1 is a vector of all ones.
10/7/2003 44
Linearize the System (3)
23
10/7/2003 45
Linearize the System (4)
10/7/2003 46
Linearize the System (5)
24
10/7/2003 47