two numerical graph algorithms

Two matrix computations fornumerical graph problems:

PageRank and Network Alignment

David F. Gleich

Sandia National LabsLivermore, CA

IBM Almaden SeminarSan Jose, CA

January 17th, 2011

In collaboration withAndrew Gray (UBC), Chen Greif (UBC)

Tracy Lau (UBC/IBM?), Mohsen Bayati (Stanford)Ying Wang (Stanford), Margot Gerritsen (Stanford)

Amin Saberi (Stanford)

Supported by the Library of Congressand Microsoft Live Labs Fellowship

David F. Gleich (Sandia) IBM Almaden 1 / 47

Sketch of talk

É two algorithms inner-outer and belief propagationÉ two problems PageRank and network alignmentÉ big graphs for bothÉ iterative matrix computations for bothÉ multi-core parallel results inner-outer only

standard flowproblem → algorithm → theory (hopefully) → empirical resultsexcept “fun” results first

some open questions at end

David F. Gleich (Sandia) IBM Almaden 2 / 47

A PageRank algorithmInstead of the power method,

x(k+1) = αPx(k) + (1− α)v.

Use an outer iteration

(− βP)x(k+1)

= (α − β)Px(k) + (1− α)v︸ ︷︷ ︸

f

.

with the inner iteration

y(j+1) = βPy(j) + f.

It’s faster!

Web Data, α = 0.99

Nodes 105,896,555Edges 3,783,733,648

Power Method 964 its 5.15 hrs.Inner-Outer 857 its 4.45 hrs.

Network-Alignment Data, α = 0.95

Nodes 4,219,893,141Edges 91,886,357,440

Power Method 271 its 54.6 hrs.Inner-Outer 188 its 36.2 hrs.

Codes and data available.

Note Web data is uk-2006 from UNIMI’s (Univ. Milano) DSI group.

David F. Gleich (Sandia) IBM Almaden 3 / 47

Network Alignment

r

t

s

t

Square

A L B

A is about 200,000 verticesB is about 300,000 verticesL has around 5,000,000 edges5 million variable integer QP∼ 90% of optimality in minutes.

Codes and data available.

DEMO

David F. Gleich (Sandia) IBM Almaden 4 / 47

PageRankSlide 5 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

PageRank is a ...... modified Markov chain,... damped random walk on a graph,... pinball game on the reverse web, or... random surfer model.

Proposed by Brin and Page in 1998, but similar ideas fromearlier... (Sebastiano Vigna is working on tracing the history –the current history dates to 1949)

Langville and Meyer (2006) is a goodgeneral reference; Berkhin (2005) haslots of goodies; and Des Higham calledit pinball.

David F. Gleich (Sandia) PageRank IBM Almaden 6 / 47

The PageRank Random Surferimportant pages↔ highly probable to visit

1

2

3

4

5

6

↓

1/6 1/2 0 0 0 01/6 0 0 1/3 0 01/6 1/2 0 1/3 0 01/6 0 1/2 0 0 01/6 0 1/2 1/3 0 11/6 0 0 0 1 0

︸ ︷︷ ︸

P

1. follow out-edges uniformly withprobability α, and

2. randomly jump according to v withprobability 1− α, we’ll assume = 1/n.

Induces a Markov chain model�

αP+ (1− α)veT�

x(α) = x(α)

or the linear system

(− αP)x(α) = (1− α)v

But it’s just a model.

Note I’m omitting important details about dangling nodes, I’ll mention them a bit later.

David F. Gleich (Sandia) PageRank IBM Almaden 7 / 47

What is α?

Our regimeÉ α ≥ .85 otherwise

power is fast.É P only available

for mat-vecotherwise customtechniquespossible.

Author αBrin and Page (1998) 0.85Najork et al. (2007) 0.85Litvak et al. (2006) 0.5Katz (1953) 0.5Experiment (2009) 0.63 ≈

p0.85 · 0.5

Algorithms (...) ≥ 0.85

αfr

ombro

wse

rs

Raw α

dens

ity

0.0

0.5

1.0

1.5

2.0

2.5

3.0 InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )

0.0 0.2 0.4 0.6 0.8 1.0

Constantine, Flaxman, Gleich, Gunawardana, Tracking the Random Surfer, WWW2010Constantine and Gleich, Random Alpha PageRank, Internet Math.

David F. Gleich (Sandia) PageRank IBM Almaden 8 / 47

PageRank AlgorithmsSlide 9 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

PageRank formulations and theoryCodes

Strongly prefer-ential PageRank PseudoRank

Graph orWeb graph

Substochasticmatrix

Weakly prefer-ential PageRank

PageRank

Sink preferentialPageRank

Eigensystems

Linear systems

Theory

Other transformations

v teleportation vectorP̄ substochastic matrix (for algorithms)d dangling node vector (d = e− PTe)

P̄+ vdT → P Strongly preferential PageRankP̄+ dT → P Weakly preferential PageRank ( 6= v)

P PageRank stochastic matrix (for theory)(− αP)x = (1− α)v PageRank linear system

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 10 / 47

MotivationWhy another PageRank algorithm?

An ideal algorithm is1. reliable2. fast over a range of α’s

→ Use Matlab’s “\”3. efficient for big problems

→ Use a Gauss-Seidel orcustom Richardson method

4. uses only matvec products→ Use the inner-outer iteration

5. uses only 2 vectors of memory→ Use the power method simple

fancy

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 11 / 47

Simple algorithmsThe power methodFor Ax = λx, the iteration

x(k+1) = Ax(k)/

Ax(k)

computes the largesteigenpair.The PageRank Markov chaineigenvector problem is

[αP+ (1− α)veT]x = x

If eTx(0) = 1 and j ≥ 0

x(k+1) = αPx(k)+(1−α)veTx(k)︸ ︷︷ ︸

=1

The Richardson methodFor Ax = b, the iteration

x(k+1) = x(k) +ω (b−Ax(k))︸ ︷︷ ︸

residual

computes x.The PageRank linear system is

(− αP)x = (1− α)v.

For ω = 1

x(k+1) = αPx(k) + (1− α)v

and the Richardson iteration isthe power method.

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 12 / 47

Inner-Outer

Note PageRank is easier when α is smaller

Thus Solve PageRank with itself using β < α!

Outer (− βP)x(k+1) = (α − β)Px(k) + (1− α)v ≡ f(k)

Inner y(0) = x(k) y(j+1) = βPy(j) + f(k)

A new parameter? What is β? 0.5

How many inner iterations? Until a residual of 10−2

Gleich, Gray, Greif, Lau, SISC 2010.David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 13 / 47

Inner-Outer algorithm

Input: P,v, α, τ, (β = 0.5, η = 10−2)Output: x1: x← v2: y← Px3: while ‖αy+ (1− α)v− x‖1 ≥ τ4: f← (α − β)y+ (1− α)v5: repeat6: x← f+ βy7: y← Px8: until ‖f+ βy− x‖1 < η9: end while10: x← αy+ (1− α)v

É uses only three vectorsof memory

Convergence?É if 0 ≤ β ≤ α, with “ex-

act” iterationÉ but also (small theo-

rem) with any η!

Parameters?É β = 0.5, η = 10−2 often

faster than the powermethod(or just a titch slower)

Note Note that the inner-loop checks its condition after doing one iteration. An inexact iteration isalways at least as good as one-step of the power method.

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 14 / 47

Inner-Outer ParametersQuestion: What parameters should we pick?

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8700

800

900

1000

1100

1200

1300

1400

1500

β

Mul

tiplic

atio

ns

in−2004, α=0.99

powerη = 1e−01η = 1e−02η = 1e−03η = 1e−04η = 1e−05

10−4

10−3

10−2

10−1

700

800

900

1000

1100

1200

1300

1400

1500

η

Mul

tiplic

atio

ns

in−2004, α=0.99

powerβ = 0.10β = 0.30β = 0.50β = 0.70

α = 0.99, in-2004 graph (1.3M nodes, 16.9M edges)

Just use β = 0.5 and η = 10−2!Note Many similar plots appear in my thesis.

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 15 / 47

The CompetitionOur Requirement: only Px is available!É Quadratic Extrapolation (Kamvar, Haveliwala, et al.)É Aggregation/Disaggregation

(Langville and Meyer; Stewart)É Permutations/Strong Components

(Del Corso, Gulli, and Romani; Langville and Meyer)É Krylov methods (Gleich, Zhukov, Berkhin;

Del Corso, Gulli, and Romani)É Padé-type extrapolation (Brezinski and Redivo-Zaglia)

É Arnoldi methods (Greif and Golub)

É Gauss-Seidel (Arasu, Novak, Tomkins, and Tomlin)

David F. Gleich (Sandia) PageRank Algorithms IBM Almaden 16 / 47

Inner-outerPerformanceSlide 17 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

Datasets

name size nonzeros avg nz/rowubc-cs-2006 51,681 673,010 13.0ubc-2006 339,147 4,203,811 12.4eu-2005 862,664 19,235,140 22.3in-2004 1,382,908 16,917,053 12.2wb-edu 9,845,725 57,156,537 5.8arabic-2005 22,744,080 639,999,458 28.1sk-2005 50,636,154 1,949,412,601 38.5uk-2007 105,896,555 3,738,733,648 35.3

David F. Gleich (Sandia) Inner-outer Performance IBM Almaden 18 / 47

One example

10 20 30 40 50 60 70 8010

−7

10−6

10−5

10−4

10−3

10−2

10−1

100

Multiplication

Res

idua

l

wb−edu, α = 0.85

powerinout

5 10 15 20

10−2

100

200 400 600 800 1000 120010

−7

10−6

10−5

10−4

10−3

10−2

10−1

100

Multiplication

Res

idua

l

wb−edu, α = 0.99

powerinout

20 40

10−2

100

τ = 10−7, β = 0.5, η = 10−2;wb-edu graph (9.8M nodes, 57.M edges)

David F. Gleich (Sandia) Inner-outer Performance IBM Almaden 19 / 47

Advantage Inner-Outerα=0.99,β=0.5,η=10−2

tol. graph work (mults.) time (secs.)power in/out gain power in/out gain

10−3 ubc-cs-2006 226 141 37.6% 1.9 1.2 35.2%ubc 242 141 41.7% 13.6 8.3 38.4%in-2004 232 129 44.4% 51.1 30.4 40.5%eu-2005 149 150 -0.7% 26.9 28.3 -5.3%wb-edu 221 130 41.2% 291.2 184.6 36.6%

arabic-2005 213 139 34.7% 779.2 502.5 35.5%sk-2005 156 144 7.7% 1718.2 1595.9 7.1%uk-2007 145 125 13.8% 2802.0 2359.3 15.8%

10−5 ubc-cs-2006 574 432 24.7% 4.7 3.6 22.9%ubc 676 484 28.4% 37.7 27.8 26.2%in-2004 657 428 34.9% 144.3 97.5 32.4%eu-2005 499 476 4.6% 89.3 87.4 2.1%wb-edu 647 417 35.5% 850.6 572.0 32.8%

arabic-2005 638 466 27.0% 2333.5 1670.0 28.4%sk-2005 523 460 12.0% 5729.0 5077.1 11.4%uk-2007 531 463 12.8% 10225.8 8661.9 15.3%

10−7 ubc-cs-2006 986 815 17.3% 8.0 6.8 15.4%ubc 1121 856 23.6% 62.5 49.0 21.6%in-2004 1108 795 28.2% 243.1 179.8 26.0%eu-2005 896 814 9.2% 159.9 148.6 7.1%wb-edu 1096 777 29.1% 1442.9 1059.0 26.6%

arabic-2005 1083 843 22.2% 3958.8 3012.9 23.9%sk-2005 951 828 12.9% 10393.3 9122.9 12.2%uk-2007 964 857 11.1% 18559.2 16016.7 13.7%

David F. Gleich (Sandia) Inner-outer Performance IBM Almaden 20 / 47

Parallelizationparallel Pxxi=x[i]/degree(i); for (j in edges of i) { atomic(y[j]+=xi); }.

1 2 3 4 5 6 7 80

1

2

3

4

5

6

7

8

Number of processors

Spe

edup

rel

ativ

e to

bes

t 1 p

roce

ssor

linearpower relativeinout relative1e−3 power1e−3 inout1e−5 power1e−5 inout1e−7 power1e−7 inout

84

5

6

David F. Gleich (Sandia) Inner-outer Performance IBM Almaden 21 / 47

Network AlignmentMotivationSlide 22 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

David F. Gleich (Sandia) Network Alignment Motivation IBM Almaden 23 / 47

David F. Gleich (Sandia) Network Alignment Motivation IBM Almaden 24 / 47

Alignment and overlap: The goal

Health organizations

Psychiatric hospitals

Health

Mental health

Educational psychology

Wikipedia LCSH

b

a1

2

3

b1

b2

is better than

1

2

3

b1

b2

r

t

s

t

Square

A L BMaximize squares/overlap in 1-1 matching

Find a good mapping to investigate similarity!

David F. Gleich (Sandia) Network Alignment Motivation IBM Almaden 25 / 47

Network alignmentSlide 26 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

Integrating Matching and Overlap: A QP

Squares produce overlap → bonus for some and j →∑

j

r

t

s

t

Square

A L B

Variables, Data

= edge indicator = weight of edgesSj squares in S

e ∈ Le = (t, ) =t

Problem

mximize

∑

:e∈L +

∑

,j∈Sj

subject to is a matching↔

mximizex

wTx+ 12x

TSx

subject to Ax ≤ e ∈ {0,1}

David F. Gleich (Sandia) Network alignment IBM Almaden 27 / 47

An example with overlap

1 1′

2 2′

3 3′

4 4′

5 5′

6

1.0

0.6

0.6

0.50.30.3

0.1 0.10.4

0.5

0.90.9

(2,2′)(2,1′)(2,3′)(2,4′)(1,2′)(1,1′)(3,2′)(3,3′)(4,2′)(4,4′)(5,5′)(6,1′)︸ ︷︷ ︸

edge order

0 0 0 0 0 1 0 1 0 1 1 10 0 0 0 1 0 1 0 1 0 0 00 0 0 0 1 0 1 0 1 0 0 00 0 0 0 1 0 1 0 1 0 0 00 1 1 1 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 0 00 1 1 1 0 0 0 0 0 0 0 01 0 0 0 0 0 0 0 0 0 0 00 1 1 1 0 0 0 0 0 0 0 01 0 0 0 0 0 0 0 0 0 0 01 0 0 0 0 0 0 0 0 0 0 01 0 0 0 1 0 0 0 0 0 0 0

︸ ︷︷ ︸

S

,

0.60.90.30.10.90.60.30.50.10.40.51.0

︸ ︷︷ ︸

w

,

A =

1 1 1 1 0 0 0 0 0 0 0 00 0 0 0 1 1 0 0 0 0 0 00 0 0 0 0 0 1 1 0 0 0 00 0 0 0 0 0 0 0 1 1 0 00 0 0 0 0 0 0 0 0 0 1 00 0 0 0 0 0 0 0 0 0 0 11 0 0 0 1 0 1 0 1 0 0 00 1 0 0 0 1 0 0 0 0 0 10 0 1 0 0 0 0 1 0 0 0 00 0 0 1 0 0 0 0 0 1 0 00 0 0 0 0 0 0 0 0 0 1 0

David F. Gleich (Sandia) Network alignment IBM Almaden 28 / 47

Network alignment

NETWORK ALIGNMENT

mximize αwTx+ β2x

TSx

subject to Ax ≤ e, ∈ {0,1}

History

É QUADRATIC ASS IGNMENT

É MAXIMUM COMMON SUBGRAPH

É PATTERN RECOGNIT ION

É ONTOLOGY MATCHING

É B IOINFORMATICS

Sparse problemsSparse L often ignored (afew exceptions).Our paper tackles thatcase explicitly.We do large problems,too.

Conte el al. Thirty years of graph matching, 2004.; Melnik et al. Similarity flooding, 2004; Blondel et al. SIREV 2004;Singh et al. RECOMB 2007; Klau, BMC Bioinformatics 10:S59, 2009.

David F. Gleich (Sandia) Network alignment IBM Almaden 29 / 47

Network AlignmentAlgorithmsSlide 30 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

Algorithms

1. LP Convert to LP, relax, solve (Skipped)2. TIGHTLP Improve the LP (Skipped)

3. I SORANK Use a PageRank heuristic (Singh et al. 2007)

4. BP Max-product belief propagation for the LP5. TIGHTBP BP for the TIGHTLP (skipped)6. MR Sub-gradient descent on TIGHTLP (Klau 2009;

skipped)

Note Not discussed: early heuristic: Flannick et al. Genome Research 16:1169–1181, 2006; anindependent BP algorithm: Bradde et al. arXiv:0905.1893, 2009

Singh et al. RECOMB2007; Klau, 2009David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 31 / 47

IsoRankmximize αwTx+ (β/2)xTSxsubject to 0 ≤ Ax ≤ e, ∈ 0,1

Solve PageRank on S and w!

1. Normalize S to stochastic P2. Normalize w to stochastic v3. Compute power iterations and round at each4. Output best solution

É Need to evaluate a range of PageRank αÉ Designed for complete bipartite L

Singh et al. RECOMB2007; Ninove Ph.D. Thesis Louvain, 2008David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 32 / 47

Inner-outer for this problem?

Only on the cores of the two graphs.Dataset Size Non-Zeros

LCSH-2 59,849 227,464WC-3 70,509 403,960

Product Graph 4,219,893,141 91,886,357,440

α = 0.95, w from text similarity

Inner-Outer 188 mat-vec 36.2 hoursPower 271 mat-vec 54.6 hours

Caveat: I’m ignoring all the details ofactually using this technique.

David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 33 / 47

Belief propagation: Our algorithm

Summary

É Construct a probabilitymodel where the mostlikely state is the solution!

É Locally update informationÉ Like a generalized dynamic

program

É It works

É Most likely, it won’tconverge

History

É BP used for computingmarginal probabilities andmaximum aposteroriprobability

É Wildly successful at solvingsatisfiability problems

É Convergent algorithm formax-weight matching

Bayati et al. 2005;David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 34 / 47

variables functionsÉ max-product of function nodesÉ variables have state 0 or 1É function nodes compute a

productÉ messages are the belief (local

objective) about a node for astate

j

iM→j{ = s} =∏

j′∈{N()\j}

Mj′→{ = s}

variable tells function j what it thinksabout being in state s. This is just theproduct of what all the other functions tell about being in state s.

j

i Mj→{ = s} =mximmy:all possible choicesfor variables ′∈N(j)

ƒj(y)

∏

′∈{N(j)\}

M′→j{′ = y′}

function j tells variable what it thinksabout being in state s. This means that wehave to locally maxamize ƒj among allpossible choices. Note y = s always (toocumbersome to include in notation.)

David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 35 / 47

NetAlign factor graph: Loopy BP

1

2

1′

2′

3′

A B 11′

12′

22′

23′

11′22′

ƒ1

ƒ2

g′1g′2g′3

h11′22′

Variables Functions

Note It’s pretty hairy to put all the stuff I should put here on a single slide. Most of it is in the paper.The rest is just “turning the crank” with standard tricks in BP algorithms.

David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 36 / 47

Get tropical

In the max-plus sense.

David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 37 / 47

Belief propagation: A viewA :m× n

A =�

ArAc

�

x : n× 1

A� x ≡

mxj 1,jjmxj 2,jj

...mxj m,jj

bond,b z≡min(b,mx(, z))

=

z < z ≤ z ≤ bb z > b

NETALIGNBP ALGORITHM

y(0) = 0,z(0) = 0,S(0) = 0, β̃ = β/2while t = 1, . . . dod = bond0,β̃(S

(t−1)T + β̃S) · e

y(t) = αw− bond0,∞[(ATr Ar − )� z(t−1)] + d

z(t) = αw− bond0,∞[(ATcAc − )� y(t−1)] + d

S(t) = (Y(t) + Z(t) − αW−D) · S− bond0,β̃(S(t−1)T + β̃S)

end while

Note α = 1, β = 2, γ = 0.99 damping, max-weight matching rounding gives 15,214 overlap, 56,361weight in 10 mins.

David F. Gleich (Sandia) Network Alignment Algorithms IBM Almaden 38 / 47

ResultsSlide 39 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

Synthetic experiments: BP does well!

0 5 10 15 200

0.2

0.4

0.6

0.8

1

roun

ded

obje

ctiv

e va

lues

expected degree of noise in L (p ⋅ n)

MR−upperMRBPBPSCIsoRank

0 5 10 15 200

0.2

0.4

0.6

0.8

1

roun

ded

obje

ctiv

e va

lues

expected degree of noise in L (p ⋅ n)

0 5 10 15 200

0.2

0.4

0.6

0.8

1

frac

tion

corr

ect

expected degree of noise in L (p ⋅ n)

MRBPBPSCIsoRank

0 5 10 15 200

0.2

0.4

0.6

0.8

1

frac

tion

corr

ect

expected degree of noise in L (p ⋅ n)

David F. Gleich (Sandia) Results IBM Almaden 40 / 47

Biological data: A close tie

0 100 200 300 400 500 600 700 0

100

200

300

376

400

Weight

Ove

rlap

max weight671.551

overlap upper bound381

BPSCBPIsoRankMR

0 100 200 300 400 500 600 700 0

100

200

300

376

400

Weight

Ove

rlap

max weight671.551

overlap upper bound381

0 500 1000 1500 2000 2500 0

200

400

600

800

1000

1076

1200

WeightO

verla

p

max weight2733

overlap upper bound1087

BPSCBPIsoRankMR

0 500 1000 1500 2000 2500 0

200

400

600

800

1000

1076

1200

WeightO

verla

p

max weight2733

overlap upper bound1087

Problem |VA| |EA| |VB| |EB| |EL|

dmela-scere 9459 25636 5696 31261 34582Mus M.-Homo S. 3247 2793 9695 32890 15810

David F. Gleich (Sandia) Results IBM Almaden 41 / 47

Real dataset

0 10000 20000 30000 40000 50000 60000 70000 0

5000

10000

15000

16836

20000

Weight

Ove

rlap

max weight60119.8

overlap upper bound17608

BPSCBPIsoRankMR

0 10000 20000 30000 40000 50000 60000 70000 0

5000

10000

15000

16836

20000

Weight

Ove

rlap

max weight60119.8

overlap upper bound17608

Problem |VA| |EA| |VB| |EB| |EL|

lcsh2wiki 297,266 248,230 205,948 382,353 4,971,629

David F. Gleich (Sandia) Results IBM Almaden 42 / 47

Matching results: A little too hot!LCSH WC

Science fiction television series Science fiction television programsTuring test Turing test

Machine learning Machine learningHot tubs Hot dog

David F. Gleich (Sandia) Results IBM Almaden 43 / 47

Foreign subject headingsÉ The US uses LCSH for subj. headings (342k verts, 258k edges).

É France uses Rameau for subj. headings (155k verts, 156k edges).

É Generate L by automatic translation and text matching.

É Used Google’s automatic translation service(translate.google.com).

É Produces 22,195,304 possible links based on text.

cardinality overlap correctManual 54,259 39,749MWM 125,609 17,134 29,133 50.54%NetAlignBP 121,316 46,534 32,467 56.32%NetAlignMR 119,120 45,977 25,086 43.52%Upper 50,753

Note NetAlignBP with α = 1, β = 2, γ = 0.99 for 100 iterations; NetAlignMR with α = 0, β = 1 for 1000iterations.

David F. Gleich (Sandia) Results IBM Almaden 44 / 47

ConclusionSlide 45 of 47

PageRank

PageRank Algorithms

Inner-outer Performance

Network AlignmentMotivation

Network alignment

Network AlignmentAlgorithms

Results

Conclusion

Philosophy

Why matrix computations?É Simple, iterative methodsÉ “Easy” to codeÉ “Easy” to parallelizeÉ “Often” apply to graph problems

David F. Gleich (Sandia) Conclusion IBM Almaden 46 / 47

Summary and Future ideasInner-outer iterations forPageRank

É Robust analysisÉ Good for general graphsÉ Can combine with other

techniquesÉ Works for Gauss-SeidelÉ Works for non-stationary

iterations

Future workÉ Gauss-Seidel performance?É OPEN Asymptotic

performance of inner-outer?É Dynamic β and η?

BP algorithms for networkalignment

É Fast and scalableÉ Good results on biology PPI

networksÉ Reasonable results with

Rameau to LCSH

Future workÉ No vertex label information

for matches?É Are “overlap” scores

significant?É Are LCSH and Wikipedia

really similar?É OPEN An approx. algorithm?

David F. Gleich (Sandia) Conclusion IBM Almaden 47 / 47

PAPER 1 stanford.edu/~dgleich/publications/2009/

gleich-2009-inner-outer.html

SIAM J. Scientific ComputingGoogle “inner outer gleich”

CODE stanford.edu/~dgleich/publications/2009/innout

Google “innout gleich”

PAPER 2 arxiv.org/abs/0907.3338

ICDM 2009Google “network alignment gleich”

CODE stanford.edu/~dgleich/publications/2009/netalign

Google “netalign gleich”