linked based clustering and its theoretical foundations paper written by margareta ackerman and shai...

Linked Based Clustering and Its Theoretical Foundations

Paper written by Margareta Ackerman and Shai Ben-David

Presented by Yan T. YangYan T. Yang

Outline

Hierarchical clustering Linkage based clustering Theoretical foundation of clustering Characterization of linkage based

clustering

Hierarchical clustering

Agglomerative Divisive

Hierarchical clustering

A

D

E

B

C

A B C D E

Raw Data Dendrogram

F

F

Agglomerative clustering

Raw Data Dendrogram

A B C D E F

A

D

E

B

C

F


A

D

E

B

C

Raw Data Dendrogram

F

A B C D E F


Raw Data Dendrogram

A B C D E F

A

D

E

B

C

F


Raw Data Dendrogram

A B C D E F

A

D

E

B

C

F

General Complexity: O(n3)


Raw Data Dendrogram


Linkage Based: O(n2)

Linkage Based Clustering



Whenever a pair of points share the same cluster,

they are connected via a tight chain of points.

A

D

E

B

C

F

Linkage Based Clustering



Whenever a pair of points share the same cluster,

they are connected via a tight chain of points.

A

D

E

B

C

F

3. Repeat until only k clusters remain.

Steps:

1. Start: singletons

2. At each step, merge the closest pair of clusters

Linkage-based clustering


Dissimilarity Measure Linkage Criteria

A measure of distance between pairof observations.

A measure of distance between setsof observations

as function of dissimilarity measure.



Euclidean distance

Minkowski distance (generalized Euclidean)

Mahalahobis distance (scaled Euclidean)

City block distance ( |*| version of Euclidean)





Between two sets A and B

},:),(min{ :Linkage Single BbAabad

},:),(max{ :Linkage Complete BbAabad

Aa Bb

bad ),(|B||A|

1 :Linkage Average



Linkage Criteria




Aa Bb

bad ),(|B||A|

1 :Linkage Average

Minimum Spanning Tree

Successive Cliques

UPGMA tree



Linkage Criteria




Aa Bb

bad ),(|B||A|

1 :Linkage Average

SLINK [Defays]

CLINK [Sibson]

UPGMA [Murtagh]

Linkage-based clustering},:),(min{ :Linkage Single BbAabad

Add to cluster as long as there is a link

Linkage-based clustering},:),(max{ :Linkage Complete BbAabad

Add to cluster when a clique is formed

Theoretical Foundation

Many different clustering techniques: which one to use ?

Identify properties that separate input-output behavior of different clustering paradigms: [Ackerman, Ben-David, Loker]

1) Be intuitive and meaningful

2) Differentiate algorithms

The properties should


[Kleinberg, 2002]: abstract properties OR axioms.

[Zadeh, Ben-David 2009]: properties to characterize single linkage.

[Ackerman, Ben-David, Loker] : properties to characterize all linkage.


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

Clustering function

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

A clustering C is a refinement of clustering C’ if every cluster in C’ is a union of some clusters in C.

CC’

Refinement

Hierarchical

Clustering function

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

A clustering function F(X,d,k):

Clustering function

Hierarchical

1. Take input X, all data points2. Distance measure, d

3. Number of clusters desired, k.4. Output the clusters


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

A clustering function is hierarchical if forall X and all d and every 1<= k <= k’ <= |X| F(X,d,k’) is a refinement of F(X,d,k)

Clustering function

A B C D E F

6

43

2

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

Clustering function)||,,(

:any and any for if local is F

Cc

CdcFC

F(X,d,k)C X, d, k

C

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

Clustering function . and , allfor then distances,

cluster -between increasingfor except ,equals If

kd, X(X,d’,k)F(X,d,k)=F

dd’

d d’

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

Clustering function

Locality and outer consistency

Not axioms

Spectral clustering with

Ratio region

Normalized cut

does not satisfy these two.

Hierarchical


Properties

Extended richness

Outer consistency

Locality

Definition

Refinement

Hierarchical

Clustering function ),( 11 dX ),( 22 dX ),( 33 dX

),( 321 dXXX Hierarchical


Properties

Extended richness

Outer consistency

Locality

Theorem:

The clustering function is linkage based

If and only if

All of the properties are satisfied.

Hierarchical


Theorem:


If and only if


properties are satisfied.

Properties

Extended richness

Outer consistency

Locality

Hierarchical

linkage based function


Theorem:


If and only if


linkage based function properties are satisfied.

Straightforward

Properties

Extended richness

Outer consistency

Locality

Hierarchical


Theorem:


If and only if


properties are satisfied.

Properties

Extended richness

Outer consistency

Locality

Hierarchical



Properties

Extended richness

Outer consistency

Locality

Hierarchical


A linkage function is a function l: {(X1,X2,d) : d is a dissimilarity function over X1 U X2} → R+

that satisfies the following:},:),(min{ :Linkage Single BbAabad


Aa Bb

bad ),(|B||A|

1 :Linkage Average

- Representation independent

- Monotonic

- Any pair of clusters can be made arbitrarily distant


Properties

Extended richness

Outer consistency

Locality

Hierarchical


Goal

Given a clustering function F that satisfies the properties, define a linkage function l so that the linkage-based clustering based on l coincides with F (for every X, d and k).

F

Linkage-based

Theoretical FoundationGiven a clustering function F that satisfies the properties, define a linkage function l so that the linkage-based clustering based on l coincides with F (for every X, d and k).

F

Linkage-based

Define an operator <F: (A,B,d1) <F (C,D,d2) if there exists d that extends d1 and d2 such that when we run F on (A U B U C U D), A and B are merged before C and D.

A

B

C

D

F(AUBUCUD,d,4) A

B

C

D

F(AUBUCUD,d,3)

Theoretical Foundation- Prove that <F can be extended to a partial ordering - Use the ordering to define l

Lemma (<F is cycle-free)

Given a function F that is hierarchical, local, outer-consistent and satisfies extended richness, there are no (A1,B1,d1), (A2,B2,d2),… (An,Bn,dn) so that

(A1,B1,d1) <F (A2,B2,d2) … <F (An,Bn,dn)and (A1,B1,d1) = (An,Bn,dn)

By the above Lemma, the transitive closure of <F is a partial ordering.

There exists an order preserving function l that maps pairs of data sets to R (since <F is defined over a countable set)

It can be shown that l satisfies the properties of a linkage function.

linked based clustering and its theoretical foundations paper written by margareta ackerman and shai...

Documents