spectral clustering eyal david image processing seminar may 2008
TRANSCRIPT
Spectral Clustering
Eyal David
Image Processing seminar
May 2008
2
Lecture Outline
Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations
3
A Tutorial on Spectral Clustering\Arik Azran
4
Spectral Clustering Example – 2 Spirals
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Dataset exhibits complex Dataset exhibits complex cluster shapescluster shapes
K-means performs very K-means performs very poorly in this space due poorly in this space due bias toward dense bias toward dense spherical clusters.spherical clusters.
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
-0.709 -0.7085 -0.708 -0.7075 -0.707 -0.7065 -0.706In the embedded space In the embedded space given by two leading given by two leading eigenvectors, clusters eigenvectors, clusters are trivial to separate.are trivial to separate.
Spectral Clustering - Derek Greene
5
Lecture Outline
Motivation Graph overview and construction Graph demo Spectral Clustering Spectral Clustering demo Cool implementation
6
Matthias Hein and Ulrike von Luxburg August 2007
7
Matthias Hein and Ulrike von Luxburg August 2007
8
Matthias Hein and Ulrike von Luxburg August 2007
9
Matthias Hein and Ulrike von Luxburg August 2007
10
Matthias Hein and Ulrike von Luxburg August 2007
11
Matthias Hein and Ulrike von Luxburg August 2007
12
Matthias Hein and Ulrike von Luxburg August 2007
13
Matthias Hein and Ulrike von Luxburg August 2007
14
Matthias Hein and Ulrike von Luxburg August 2007
15
Matthias Hein and Ulrike von Luxburg August 2007
16
Matthias Hein and Ulrike von Luxburg August 2007
17
Matthias Hein and Ulrike von Luxburg August 2007
Demo
(Live example)
19
Lecture Outline
Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations
20
Matthias Hein and Ulrike von Luxburg August 2007
21
Matthias Hein and Ulrike von Luxburg August 2007
22
Matthias Hein and Ulrike von Luxburg August 2007
23
Matthias Hein and Ulrike von Luxburg August 2007
24
Matthias Hein and Ulrike von Luxburg August 2007
25
Matthias Hein and Ulrike von Luxburg August 2007
26
Matthias Hein and Ulrike von Luxburg August 2007
27
Matthias Hein and Ulrike von Luxburg August 2007
28
Matthias Hein and Ulrike von Luxburg August 2007
29
Eigenvectors & Eigenvalues
30
Matthias Hein and Ulrike von Luxburg August 2007
31
Matthias Hein and Ulrike von Luxburg August 2007
Demo
(Live example)
33
Spectral Clustering Algorithm Ng, Jordan, and Weiss
Motivation Given a set of points
We would like to cluster them into k subsets
1,...,l
nS s s R
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
34
Algorithm
Form the affinity matrix Define if
Scaling parameter chosen by user
Define D a diagonal matrix whose
(i,i) element is the sum of A’s row i
nxnW Ri j
0iiW
2 2|| || / 2i js s
ijW e
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
35
Algorithm
Form the matrix
Find , the k largest eigenvectors of L These form the the columns of the new
matrix X Note: have reduced dimension from nxn to nxk
1/ 2 1/ 2L D WD
1 2, ,..., kx x x
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
36
Algorithm
Form the matrix Y Renormalize each of X’s rows to have unit length Y
Treat each row of Y as a point in Cluster into k clusters via K-means
2 2/( )ij ij ijj
Y X X kR
nxkR
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
37
Algorithm
Final Cluster Assignment Assign point to cluster j iff row i of Y was
assigned to cluster jis
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
38
Why?
If we eventually use K-means, why not just apply K-means to the original data?
This method allows us to cluster non-convex regions
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
39
Some Examples
40
Ng et al On Spectral clustering: analysis and algorithm
41
Ng et al On Spectral clustering: analysis and algorithm
42
Ng et al On Spectral clustering: analysis and algorithm
43
Ng et al On Spectral clustering: analysis and algorithm
44
Ng et al On Spectral clustering: analysis and algorithm
45
Ng et al On Spectral clustering: analysis and algorithm
46
Ng et al On Spectral clustering: analysis and algorithm
47
Ng et al On Spectral clustering: analysis and algorithm
48
User’s Prerogative
Affinity matrix construction Choice of scaling factor
Realistically, search over and pick value that gives the tightest clusters
Choice of k, the number of clusters Choice of clustering method
2
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm
49
0
5
10
15
20
25
30
35
40
45
50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
K
Eig
enva
lue
Largest Largest eigenvalueseigenvalues
of Cisi/Medline of Cisi/Medline datadata
λ1
λ2
How to select k? Eigengap: the difference between two consecutive eigenvalues. Most stable clustering is generally given by the value k that
maximises the expression
1k k k
Choose Choose k=2k=2
12max k
Spectral Clustering - Derek Greene
50
Matthias Hein and Ulrike von Luxburg August 2007
Recap – The bottom line
51
Summary
Spectral clustering can help us in hard clustering problems
The technique is simple to understand The solution comes from solving a simple
algebra problem which is not hard to implement
Great care should be taken in choosing the “starting conditions”
The End