Fast SDP Relaxations of Graph Cut Clustering,

Transduction, and Other Combinatorial Problems

(JMLR 2006)

Tijl De Bie and Nello Cristianini

Presented by Lihan HeMarch 16, 2007

Outline

Statement of the problem

Spectral relaxation and eigenvector

SDP relaxation and Lagrange dual

Generalization: between spectral and SDP

Transduction and side information

Experiments

Conclusions

Statement of the problem

Data set S: niixS 1}{

Affinity matrix A: ),(),( ji xxajiA

Objective: graph cut clustering -- divide the data points into two set, P and N, such that NPSNP ,

No label: clusteringWith some labels: transduction

Statement of the problem

Normalized graph cut problem (NCut)

where

Cut costHow well the clusters are balanced

Cost function:

Statement of the problem

Normalized graph cut problem (NCut)

ny }1 ,1{Unknown label vector

Let )( , ddiagDAd 1

Write ,2/)('),( ydSPassocs 1 2/)('),( ydSNassocs 1

1 'dsss

Rewrite the NCut problem as a combinatorial optimization problem

.

,'

,}1 ,1{ s.t.

)('4

min,,

sss

ssyd

y

yADyss

s

n

ssy

NP-complete problem, the exponent is very high.

(1)

Spectral Relaxation

Let ys

dI

ss

sy

'

4~ 1

the problem becomes

sss

ys

dI

ss

sy

'

4~ 1

1~ '~ yDy

Relax the constraints by adding and dropping the combinatorial constraints on , we obtain the spectral clustering relaxationy~

(2)

1~ '~ yDy

Spectral Relaxation: eigenvector

Solution: the eigenvector corresponding to the second smallest generalized eigenvalue.

Solve the constrained optimization by Lagrange dual:

)1~'~(~)('~),~( yDyyADyyL

0~~)(2~

),~(

yDyADy

yL yDyAD ~~)(

The second constraint is automatically satisfied:

1 11 ,0 v

212 ,~ vvvy 0~' i.e., ,0~' ydyD1

SDP Relaxation

Let 'yy

the problem becomes

1 )( ,0 diagNote that

Relax the constraints by adding the above constraints and dropping

'yy ny }1 ,1{and

Let

ss

sq

4

2

and ,ˆ q we obtain the SDP relaxation

(3)

nnR ̂

SDP Relaxation: Lagrange dual

Lagrangian:

0),,,,ˆ(

0ˆ

),,,,ˆ(

q

qL

qL

We obtain the dual problem (strong dual is hold):

(4)n+1 variables

Generalization: between spectral and SDP

A cascade of relaxations tighter than spectral and looser than SDP

1qdiag )ˆ( 1')ˆ(' qWdiagW .1 , nmRW mn where

m+1 variables

n constraints m constraints, Looser than SDP

Design the structure of W design how to relax the constraints

nm 1

Generalization: between spectral and SDP

rank(W)=n: original SDP relaxation.

rank(W)=1: m=1, W=d: spectral relaxation.

A relaxation is tighter than another if the column space of the matrix W used in the first one contains the full column space of W of the second.

If choose d within the column space of W, then all relaxations in the cascade are tighter than the spectral relaxation.

One approach of designing W proposed by the author:

Sort the entries of the label vector (2nd eigenvector) from spectral relaxation;Construct partition: m subsets are roughly equally large;Reorder the data points by this sorted order;W

~ n/m

mn

W=

1…1

1…1

1…1

…

1 2 m…

Transduction

Given some labels, written as label vector yt -- transductive problem

Reparameterize 'ˆ :ˆ LML

Label constraints are imposed:

L=

yt 0

0 I

Labeled

Unlabeled

)1( testnn

1

1

ty

Rows (columns) corresponding to oppositely labeled training points then automatically are each other’s opposite;

Rows (columns) corresponding to same-labeled training points are equal to each other.

Transduction

Transductive NCut relaxation:

ntest+2 variables)1()1( testtest nnRM

General constraints

An equivalence constraint between two sets of data points specifies that they belong to the same class;

An inequivalence constraint specifies two set of data points to belong to opposite classes.

No detailed label information provided.

Experiments

1. Toy problems Affinity matrix: )2/||||exp(),( 22 ji xxjiA

Experiments

2. Clustering and transduction on text

Data set: 195 articles4 languages

several topics

Affinity matrix: 20-nearest neighbor: A(i,j)= 1

0.50

Distance of two articles: cosine distance on the bag of words representation

NwNwwi Rfffv ]',...,,[ 21

},...,,{ 21 NwwwDefine dictionary

),cos(1 jiij vvd

Experiments

2. Clustering and transduction on text: cost

By language By topic

Spectral (randomized rounding)

SDP (randomized rounding)

Spectral (lower bound)

SDP (lower bound)

Cost: randomized rounding ≥ opt ≥ lower bound

Cos

t

Cos

t

Fraction of labeled data points Fraction of labeled data points

Experiments

2. Clustering and transduction on text: accuracy

By language By topic

Spectral (randomized rounding)

SDP (randomized rounding)

Acc

urac

y

Acc

urac

y

Fraction of labeled data points Fraction of labeled data points

Conclusions

Proposed a new cascade of SDP relaxations of the NP-complete normalized graph cut optimization problem;

One extreme: spectral relaxation;

The other extreme: newly proposed SDP relaxation;

For unsupervised and semi-supervised learning, and more general constraints;

Balance the computational cost and the accuracy.