complex graph algorithms - clemson universityisafro/na13/l12.pdf · 60 chapter 6. complex graph...

R O M M E L J A L A S U T R A M 0 2 / 2 7 / 2 0 1 3

B A S E D O N A C H A P T E R F R O M : G R A P H

A L G O R I T H M S I N T H E L A N G U A G E O F L I N E A R A L G E B R A

Complex Graph Algorithms

Linear Algebra

�  What is linear algebra? ¡  Branch of mathematics studying vector spaces ¡  Techniques used in various subjects including analytic

geometry, physics, computer science, and NETWORK ANALYSIS

�  What are of interest to us? ¡  Adjacency matrices, Laplacians, Eigenvalues and Eigenvectors,

Dimensionality reduction ¡  Parallelization of different architectures

Algebraic Graph Theory

�  Studies graphs from the view point of Linear Algebra.

�  Graph G = (V , E) ¡  V = set of vertices ¡  E = set of edges ¡  Can be represented using adjacency matrices

�  Adjacency matrix = A(n x n) ¡  Square matrix i.e. # of rows = # of columns – n ¡  For every edge i à j Aij= 1 ¡  Aii = 0

Adjacency Matrix

1 2

4 3

0 1 1 0

1 0 1 1

1 1 0 1

0 1 1 0

Adjacency Matrix

Laplacians

�  L = D – A ¡  D is a degree matrix ¡  Diagonal entries have degrees of each node ¡  Lij = deg(node vi) if i = j ¡  = -1 if i ≠ j and end edge between (vi,vj) ¡  = 0 otherwise ¡  L is positive-semidefinite

�  Can be used to calculate the the number of spanning trees.

1 2

4 3

Eigenvalues and Eigenvectors

�  Computed for a square matrix ¡  Remember something??? Adjacency matrix

�  Av = λv , A is the adjacency matrix ¡  v = eigenvector ¡  λ = eigenvalue

�  Principal eigenvector used to measure centrality of vertices.

Dimensionality Reduction

�  Points are mapped in high dimensional space.

�  Curse of Dimensionality

�  Subset of features sufficient to classify.

�  Project points to a lower dimensional space while retaining the geometric characteristics.

�  Principal component analysis, singular value decomposition, eigen decomposition, non-negative matrix factorization, etc.

What is the relation with Linear Algebra?

�  Full text search: Matrix multiplication of term document frequency matrix by the query vector.

�  Computing co-occurrences in a collaborative filtering context: Squaring the user interaction matrix.

�  Calculating users who are k-degrees separated from each other in a social network or web-graph can be found by looking at the k-fold product of the graph adjacency matrix.

Aha!! Matrix Multiplication

�  Everything boils down to Matrix multiplication.

�  Dimensionality reduction needed when using large sparse matrices.

Outline

�  Peer Pressure Clustering.

�  Vertex Betweenness Centrality and its Matrix equivalent.

�  Edge Betweenness Centrality and its Matrix equivalent.

Peer Pressure Clustering

�  Goal: Determine natural groups in a graph with high connectivity.

�  Useful in machine learning, data mining, pattern recognition, etc.

�  In a decent cluster approximation, a vertex’s cluster assignment is similar to a majority of its neighbors.

�  Performed iteratively.

�  Algorithm terminates when cluster assignments do not change.

Improper Clustering 6.1. Graph clustering 61

8

2

6 7

5

3

14

Figure 6.1. Sample graph with vertex 4 clustered improperly.

B: 0R: 3

B: 1R: 3

B: 1R: 2

B: 4R: 1

B: 3R: 0

B: 3R: 1

B: 5R: 0

B: 2R: 3

Figure 6.2. Sample graph with count of edges from each clusterto each vertex.

B: 0R: 3

B: 0R: 4

B: 0R: 3

B: 4R: 1

B: 3R: 0

B: 3R: 1

B: 4R: 1

B: 1R: 4

Figure 6.3. Sample graph with correct clustering and edge counts.

Downloaded 09 Dec 2011 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Taken from the book “Graph Algorithms in the Language of Linear Algebra” by Jeremy Kepner and John Gilbert

Proper Clustering

6.1. Graph clustering 61

8

2

6 7

5

3

14

Figure 6.1. Sample graph with vertex 4 clustered improperly.

B: 0R: 3

B: 1R: 3

B: 1R: 2

B: 4R: 1

B: 3R: 0

B: 3R: 1

B: 5R: 0

B: 2R: 3

Figure 6.2. Sample graph with count of edges from each clusterto each vertex.

B: 0R: 3

B: 0R: 4

B: 0R: 3

B: 4R: 1

B: 3R: 0

B: 3R: 1

B: 4R: 1

B: 1R: 4

Figure 6.3. Sample graph with correct clustering and edge counts.




�  At the beginning, each cluster is in its own cluster.

�  Assuring vertices have equal votes.

�  Preserving small clusters and un-clustered vertices.

�  Settling ties.

�  Convergence not guaranteed for un-clustered graphs.


60 Chapter 6. Complex Graph Algorithms

cluster assignment of the majority of its neighbors. Consider the graph in Figure 6.1,where each vertex except vertex 4 is in the proper cluster.

If the incoming edges to each vertex in this graph are examined to determinefrom which cluster they originate, this clustering can be considered suboptimal. Asseen in Figure 6.2, all vertices except vertex 4 have the most incoming edges fromthe cluster they are in.

In addition, if the clustering is adjusted accordingly by placing vertex 4 inthe red cluster, the result now shows the clustering to be optimal. Each vertex hasa majority of incoming edges originating from its own cluster. This is shown inFigure 6.3.

Traditionally, peer pressure clustering is performed by iteratively refining acluster approximation for the graph. Given a cluster approximation, each vertex inthe graph first votes for all of its neighbors to be in its current cluster. These votesare then tallied and a new cluster approximation is formed by moving vertices tothe cluster for which they obtained the most votes. The algorithm typically ter-minates when a fixed point is reached (when two successive cluster approximationsare identical).

Algorithm 6.1 shows the recursive definition for peer pressure clustering. Thisalgorithm can also be performed in a loop, keeping track of the current and previouscluster approximation and terminating when they agree.

Algorithm 6.1. Peer pressure. Recursive algorithm for clustering vertices.!

PeerPressure(G = (V,E), Ci)

1 for (u, v, w) ! E2 do T (v)(C(u))" T (v)(C(u)) + w3 for n ! V4 do Cf (n)" i : #j ! V : T (n)(j) $ T (n)(i)5 if Ci == Cf

6 then return Cf

7 else return PeerPressure(G,Cf )

In this algorithm, the loop at line 1 is responsible for the voting, and the loop atline 3 tallies those votes to form a new cluster approximation. It is assumed thatthe structure T is stored as an array of lists, which keeps track of, for each vertex,the number of votes that vertex gets for each cluster for which it receives votes.

Unfortunately, convergence is not guaranteed for unclustered graphs and patho-logical cases do exist where the algorithm must make O(n) recursive calls before arepetition. In any well-clustered graph, however, this algorithm will terminate aftera small number of recursive calls, typically on the order of five.

*V.B. Shah, An Interactive System for Combinatorial Scientific Computing with an Emphasison Programmer Productivity, Ph.D. thesis, Computer Science, University of California, SantaBarbara, June 2007.



Clustering Process 6.1. Graph clustering 65

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 0.25 0.25 0.25 0.332 0.25 0.25 0.20 0.253 0.25 0.25 0.254 0.25 0.25 0.25 0.20 0.255 0.25 0.25 0.336 0.25 0.20 0.257 0.25 0.25 0.25 0.20 0.338 0.25 0.20 0.25

Figure 6.5. Initial clustering and weights.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.50 0.832 0.25 0.503 0.25 0.25 0.504 0.25 0.50 0.20 0.255 0.25 0.586 0.25 0.25 0.207 0.25 0.20 0.838 0.50 0.20

Figure 6.6. Clustering after first iteration.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 1.082 0.20 0.753 0.25 0.754 0.20 0.75 0.255 0.836 0.20 0.507 0.20 1.088 0.20 0.50

Figure 6.7. Clustering after second iteration.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 1.082 0.953 0.25 0.754 0.95 0.255 0.836 0.707 0.20 1.088 0.70

Figure 6.8. Final clustering.



Clustering Process

6.1. Graph clustering 65

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 0.25 0.25 0.25 0.332 0.25 0.25 0.20 0.253 0.25 0.25 0.254 0.25 0.25 0.25 0.20 0.255 0.25 0.25 0.336 0.25 0.20 0.257 0.25 0.25 0.25 0.20 0.338 0.25 0.20 0.25

Figure 6.5. Initial clustering and weights.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.50 0.832 0.25 0.503 0.25 0.25 0.504 0.25 0.50 0.20 0.255 0.25 0.586 0.25 0.25 0.207 0.25 0.20 0.838 0.50 0.20

Figure 6.6. Clustering after first iteration.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 1.082 0.20 0.753 0.25 0.754 0.20 0.75 0.255 0.836 0.20 0.507 0.20 1.088 0.20 0.50

Figure 6.7. Clustering after second iteration.

80.25

20.25

60.20

70.33

50.25

30.25

10.25

40.25

V Color-Coded Votes1 0.25 1.082 0.953 0.25 0.754 0.95 0.255 0.836 0.707 0.20 1.088 0.70

Figure 6.8. Final clustering.



Matrix Formulation


6.1.2 Matrix formulationWhere the graph is represented as a weighted adjacency matrix, G = A : RN!N

+ ,the clustering algorithm can be performed in the same manner. Let C : BN!N bethe cluster approximation, where if cij == 1, then vertex j is in cluster i.

With this representation, voting can be expressed as a simple matrix matrixmultiplication

T = CA

Here T represents a tallying matrix where if tij == k, then there are k votes forvertex j to be in cluster i.

Once the votes have been performed, the new clusters need to be selected.This can be done with the following operations

m = T max.

C = m .== T

Here the max vote for each vertex in each cluster is found, then the cluster approx-imation is set appropriately according to that value.

From this approximation, the peer pressure algorithm can be formed (seeAlgorithm 6.2). Line 2 performs the voting, and lines 3 and 4 tally those votes toform a new cluster approximation.

Algorithm 6.2. Peer pressure matrix formulation.

PeerPressure(G = A : RN!N+ ,Ci : BN!N )

1 T : RN!N+ Cf : BN!N m : RN

+

2 T = CiA3 m = T max.4 Cf = m .== T5 if Ci == Cf

6 then return Cf

7 else return PeerPressure(G,Cf )

Starting approximation

As before, an initial approximation must be selected. If each vertex is in a clusterby itself, with the cluster number being equal to the vertex number, then Ci = I,or the identity matrix.

Assuring vertices have equal votes

Normalizing the out-degrees of the vertices in the graph corresponds to normalizingthe rows of the adjacency matrix. This can be done easily as shown below

w = A +.

A = 1/w .! A




�  Time Complexity: O(m).

�  Other approaches – Markov clustering.

�  Has coarseness and inflation parameters.

�  Can be used to discover clusters of varied coarseness.

�  Time O(n3), space O(n2)

Vertex Betweenness Centrality

�  A measure of the number of shortest paths a vertex appears in.

�  Commonly used in fault tolerance and key component analysis.

�  Run all-pairs shortest paths �  Takes O(n3) time and requires O(n2)

�  Today’s Goal: Reduce the computation time to O(nm) for un-weighted graphs.

Brandes Algorithm(2001) Overview

�  Single source shortest paths from all the vertices

�  Number of shortest paths to all the vertices recorded.

�  Predecessors to each vertex stored.

�  Trivia: Used in networkx for computing centrality measures.

�  Good when used with sparse graphs.

The Algorithm -- BFS


Algorithm 6.4. Betweenness centrality.

BetweennessCentrality(G = (V,E))

1 CB[v]! 0, "v # V2 for "s # V3 do4 S ! empty stack5 P [w]! empty list, "w # V6 ![t]! 0, "t # V , ![s]! 17 d[t]! $1, "t # V , d[s]! 08 Q! empty queue9 enqueue(Q, s)10 while ¬empty(Q)11 do12 v ! dequeue(Q)13 push(S, v)14 for "w # neighbors(v)15 do16 if d[w] < 017 then18 enqueue(Q,w)19 d[w]! d[v] + 120 if d[w] = d[v] + 121 then22 ![w]! ![w] + ![v]23 append(P [w], v)24 "[v]! 0, "v # V25 while ¬empty(S)26 do27 w ! pop(S)28 for v # P [w]

29 do "[v]! "[v] + ![v]![w] % (1 + "[w])

30 if w &= s31 then CB[w]! CB[w] + "[w]

The first step in computing the betweenness centrality updates induced bypaths originating at vertex 1 is to perform a breadth-first search from that vertex,keeping track of the number of shortest paths to each vertex as the search progresses.It is easy to see that the number of shortest paths to a vertex v at depth d is simplythe sum of the number of shortest paths to all vertices with edges going to v atdepth d$ 1.

This progression is shown in Figure 6.10. The two interesting steps are seen inFigure 6.10(c) and Figure 6.10(e). Joins of multiple predecessors with shortest pathsoccur there. In Figure 6.10(c), three vertices, all with one shortest path, combine toform a shortest path count of three for their neighboring vertex. In Figure 6.10(e),two vertices, all with three shortest paths, combine to form a shortest path countof six for their neighboring vertex.



The Algorithm—Centrality Update





29 do "[v]! "[v] + ![v]![w] % (1 + "[w])






Example 6.2. Vertex betweenness centrality 71

Figure 6.9. Sample graph.

(a) Step 1 (b) Step 2

(c) Step 3 (d) Step 4

(e) Step 5

Figure 6.10. Shortest path steps.Procedure for computing the shortest paths in the sample graph.



Example – Step I – BFS

6.2. Vertex betweenness centrality 71




(e) Step 5



Stack S = {1} Stack S = {1,2,3,4}







(e) Step 5



Stack S = {1,2,3,4,5} Stack S = {1,2,3,4,5,6,7}







(e) Step 5



Stack S = {1,2,3,4,5,6,7,8}


Example – Step II – Centrality Update




(e) Step 5

Figure 6.11. Betweenness centrality updates.Steps for computing the betweenness centrality updates in the sample graph.

Once the shortest paths have been computed, the centrality updates can be-gin. The updates are computed in reverse depth order. The update for a vertexcorresponds to the updates for all its shortest path successors, plus one for the suc-cessor itself, multiplied by the ratio of the shortest paths that the edge contributedto the successor, which is simply the number of shortest paths to the vertex itselfdivided by the number of shortest paths to its successor.

This cascading update is shown in Figure 6.11. The updates for the root andall end vertices are set to zero immediately. From this, the rest of the updates arecomputed. Figure 6.11(b) shows the first set of update computations. Because thereare six paths to the final vertex, and only three paths to each of its predecessors,each predecessor receives only half of its centrality update plus one. Figure 6.11(c)joins the updates from its two successors, leading to a total centrality update ofthree for it. Figure 6.11(d) shows how its three predecessors split this score, leadingto a centrality update of 4

3 for each of them.This completes the centrality updates induced by shortest paths originating

from vertex 1. To verify this answer, the depths of each vertex in the graph from


Stack S = {1,2,3,4,5} Stack S = {1,2,3,4,5,6,7}






(e) Step 5







Stack S = {1,2,3,4,5,6} Stack S = {1,2,3,4}






(e) Step 5







Stack S = {1}


Matrix Formulation


Algorithm 6.5. Betweenness centrality matrix formulation.

b = BetweennessCentrality(G = A : BN!N )

1 b = 02 for 1 ! r ! N3 do4 d = 05 S = 06 p = 0, p(r) = 17 f = A(r, :)8 while f "= 09 do10 d = d+ 111 p = p+ f12 S(d, :) = f13 f = fA# ¬p14 while d $ 215 do16 w = S(d, :) # (1 + u)÷ p17 w = Aw18 w = w # S(d% 1, :)# p19 u = u+w20 d = d% 121 b = b+ u

Line 8 performs the breadth-first search. After updating the search based onthe new fringe obtained for the previous level, it selects the outgoing edges of thatfringe, weighting them by the number of shortest paths that go to their parents andsumming them up. It then filters out those that go to vertices that have been seenpreviously, resulting in the new fringe for the next level. The loop terminates whenno new values are seen on the fringe.

Line 14 performs the betweenness centrality updates. It processes the verticesin reverse depth order. First, it computes the weights corresponding to those derivedfrom the children values (including filtering out edges that do not go to vertices atthe current depth). It applies the computed weights to the adjacency matrix andsums up the row values. It then applies the weights corresponding to those derivedfrom parent values (including filtering out vertices not at the previous depth) andadds in the new centrality updates. The loop updates centrality for all parents notincluding the root value.

Algorithm 6.5 has a small optimization. Lines 16–19 can be combined into asingle parenthesized linear algebraic operation. This reduction allows the w variableto be optimized out during implementation.

Time complexity

The breadth-first search performed in the loop on line 8 requires the standardO(N +M) operations because each vertex is seen at most once over the course of









Time complexity



BFS Centrality Updates


Example

0 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0

0 0 0 0 3 0 0 0 1 1 1 1 3 0 0 0

0 0 0 0 0 3 3 0 1 1 1 1 3 3 3 0

0 0 0 0 0 0 0 6 1 1 1 1 3 3 3 6

d = 1, S[1,:]

d = 2, S[2,:]

d = 3, S[3,:]

d = 4, S[4,:]

d = 1, p[]

d = 2, p[]

d = 3, p[]

d = 4, p[]

Example

0 0 0 0 0 1/2 1/2 0 Vertex 8 removed

0 0 0 0 3/2 1/2 1/2 0 Vertex 7 removed

0 0 0 0 3 1/2 1/2 0 Vertex 6 removed

0 4/3 4/3 4/3 3 1/2 1/2 0 Vertex 5 removed

Edge Betweenness Centrality

�  A measure of the number of shortest paths an edge appears in.

�  Takes O(m) space and O(nm) time. �  As in the previous algorithm, we start with a BFS and

keep track of number of shortest paths to each vertex.

The Algorithm – BFS





29 do "[v]! "[v] + ![v]![w] % (1 + "[w])






The Algorithm—Centrality Update

6.3. Edge betweenness centrality 79

Algorithm 6.7. Edge betweenness centrality.

EdgeBetweennessCentrality(G = (V,E))

1 CB [(v, w)]! 0, "(v, w) # E2 for "s # V3 do4 S ! empty stack5 P [w]! empty list, "w # V6 ![t]! 0, "t # V , ![s]! 17 d[t]! $1, "t # V , d[s]! 08 Q! empty queue9 enqueue(Q, s)10 while ¬empty(Q)11 do12 v ! dequeue(Q)13 push(S, v)14 for "w # neighbors(v)15 do16 if d[w] < 017 then18 enqueue(Q,w)19 d[w]! d[v] + 120 if d[w] = d[v] + 121 then22 ![w]! ![w] + ![v]23 append(P [w], v)24 "[v]! 0, "v # V25 while ¬empty(S)26 do27 w ! pop(S)28 for v # P [w]29 do

30 "[v]! "[v] + ![v] % ( ![w]"[w] + 1)

31 CB[(v, w)] ! CB [(v, w)] + ![v] % ( ![w]"[w] + 1)

The first step in computing the betweenness centrality updates induced bypaths originating at vertex 1 is to perform a breadth-first search from that vertex,keeping track of the number of shortest paths to each vertex as the search progresses.It is easy to see that the number of shortest paths to a vertex v at depth d is simplythe sum of the number of shortest paths to all vertices with edges going to v at depthd$ 1. This progression is identical to the one for vertex betweenness centrality andis shown in Figure 6.10.

Once the shortest paths have been computed, the centrality updates can begin.The updates are computed in reverse depth order. As the updates proceed, eachvertex also keeps track of the sum of all the updates from edges originating fromit. The update for an edge corresponds to the number of shortest paths to the







(e) Step 5 (f) Step 5

Figure 6.12. Edge centrality updates.Steps for computing the edge centrality updates in sample graph.

originating vertex times the ratio of all the updates for all the edges originatingfrom the edge’s destination vertex over the number of shortest paths to destinationvertex plus one.

This cascading update is shown in Figure 6.12. The update flow for all endvertices is set to zero immediately. From this, the rest of the updates and updateflows are computed. Figure 6.12(b) shows the first set of update computations.Because there are six paths to the final vertex, and only three paths to each ofits predecessors, each predecessor receives only half of those six paths. They eachhave a single predecessor so the flow through those predecessors is just the edge’scentrality updates. In Figure 6.12(c), each of the two edges contribute three pathsto their successor and the end vertex, making the update six for each edge. Thereis a single predecessor for those two edges, so that vertex’s update flow is twelve. InFigure 6.12(d), the update of twelve is split between each of the three shortest pathedges flowing in. A shortest path edge is then added in for the current vertex, givingeach of the edges a score of five. They each have a single predecessor, so they eachget a flow update of five as well. Finally, in Figure 6.12(e), each of the preceding



Matrix Formulation


Algorithm 6.8. Edge betweenness centrality matrix formulation.

Var Type Description

S BN!N the search, keeps track of the depth at which each vertex is seenp ZN the number of shortest paths to each vertexf ZN the fringe, the number of shortest paths to vertices at the current

depthw RN the column/row weights for the BC updatesB RN!N the BC score for each edgeU RN!N the BC update for each edgev ZN the BC flow through the verticesr Z the current root value, or starting vertex for which to compute BC

updatesd Z the current depth being examined

B = EdgeBetweennessCentrality(G = A : BN!N )

1 B = 02 for 1 ! r ! N3 do4 d = 05 S = 06 p = 0, p(r) = 17 U = 08 v = 09 f = A(r, :)10 while f "= 011 do12 d = d+ 113 p = p+ f14 S(d, :) = f15 f = fA# ¬p16 while d $ 217 do18 w = S(d, :) ÷ p# v + S(d, :)19 U = A .# w20 w = S(d% 1, :)# p21 U = w .# U22 B = B+U23 v = U +.24 d = d% 1









Time complexity



BFS Centrality Updates


Questions

complex graph algorithms - clemson universityisafro/na13/l12.pdf · 60 chapter 6. complex graph...

Documents