All-Pairs shortest paths via fast matrix multiplication
Uri ZwickTel Aviv University
Otwarte wykłady dla doktorantów informatykiInstytut Informatyki
Uniwersytetu Warszawskiego
Marzec 9-10, 2007
1. Algebraic matrix multiplicationa. Strassen’s algorithm
b. Rectangular matrix multiplication
2. Boolean matrix multiplicationa. Simple reduction to integer matrix multiplication
b. Computing the transitive closure of a graph.
3. Min-Plus matrix multiplicationa. Equivalence to the APSP problem
b. Expensive reduction to algebraic products
c. Fredman’s trick
Outline
4. APSP in undirected graphsa. An O(n2.38) algorithm for unweighted graphs (Seidel)
b. An O(Mn2.38) algorithm for weighted graphs (Shoshan-Zwick)
5. APSP in directed graphs1. An O(M0.68n2.58) algorithm (Zwick)
2. An O(Mn2.38) preprocessing / O(n) query answering algorithm (Yuster-Zwick)
3. An O(n2.38log M) (1+ε)-approximation algorithm
6. Summary and open problems
Short introduction toFast matrix multiplication
Algebraic Matrix Multiplication
=( )i jA a ( )i jB b ( )i jC ci
j
Can be computed naively in O(n3) time.
Matrix multiplication algorithms
ComplexityAuthors
n3—
n2.81Strassen (1969)
n2.38Coppersmith, Winograd (1990)
Conjecture/Open problem: n2+o(1) ???
Multiplying 22 matrices
8 multiplications4 additions
T(n) = 8 T(n/2) + O(n2)
T(n) = O(nlog8/log2)=O(n3)
C11 = A11B11 + A12B21
C12 = A11B12 + A12B22
C21 = A21B11 + A22B21
C22 = A21B12 + A22B22
Strassen’s 22 algorithm
11 11 11 12 21
12 11 12 12 22
21 21 11 22 21
22 21 12 22 22
C A B A B
C A B A B
C A B A B
C A B A B
1 11 22 11 22
2 21 22 11
3 11 12 22
4 22 21 11
5 11 12 22
6 21 11 11 12
7 12 22 21 22
( )( )
( )
( )
( )
( )
( )( )
( )( )
M A A B B
M A A B
M A B B
M A B B
M A A B
M A A B B
M A A B B
11 1 4 5 7
12 3 5
21 2 4
22 1 2 3 6
C M M M M
C M M
C M M
C M M M M
7 multiplications18 additions/subtractions
Subtraction!
Strassen’s nn algorithm
View each n n matrix as a 2 2 matrix whose elements are n/2 n/2 matrices.
Apply the 22 algorithm recursively.
T(n) = 7 T(n/2) + O(n2)
T(n) = O(nlog7/log2)=O(n2.81)
Works over any ring!
Matrix multiplication algorithms
The O(n2.81) bound of Strassen was improved by Pan, Bini-Capovani-Lotti-
Romani, Schönhage and finally by Coppersmith and Winograd to O(n2.38).
The algorithms are much more complicated…
We let 2 ≤ < 2.38 be the exponent of matrix multiplication.
Many believe that =2+o(1).
Rectangular Matrix multiplication
[Coppersmith ’97]: n1.85p0.54+n2+o(1)
For p ≤ n0.29, complexity = n2+o(1) !!!
=n
p
p
n
n
n
Naïve complexity: n2p
BOOLEAN MATRIX MULTIPLICATION
andTRANSIVE CLOSURE
Boolean Matrix Multiplication
=( )i jA a ( )i jB b ( )i jC ci
j
Can be computed naively in O(n3) time.
Algebraic Product
O(n2.38) algebraic
operations
Boolean Product
or () has no inverse!
?But, we can work over the integers
(modulo n+1)!
O(n2.38) operations on
O(log n) bit words
Transitive Closure
Let G=(V,E) be a directed graph.
The transitive closure G*=(V,E*) is the graph in which (u,v)E* iff there is a path from u to v.
Can be easily computed in O(mn) time.
Can also be computed in O(n) time.
Adjacency matrix of a directed graph
1
32
4
6
5
Exercise 0: If A is the adjacency matrix of a graph, then (Ak)ij=1 iff there is a path of length k from i to j.
Transitive Closure using matrix multiplication
Let G=(V,E) be a directed graph.
The transitive closure G*=(V,E*) is the graph in which (u,v)E* iff there is a path from u to v.
If A is the adjacency matrix of G, then (AI)n1 is the adjacency matrix of G*.
The matrix (AI)n1 can be computed by log n squaring operations in O(nlog n) time.
It can also be computed in O(n) time.
(ABD*C)*EBD*
D*CED*GBD*
AB
CD
EF
GH
X =
X* = =
TC(n) ≤ 2 TC(n/2) + 6 BMM(n/2) + O(n2)
A D
C
B
Exercise 1: Give O(n) algorithms for findning, in a directed graph,
a) a triangle
b) a simple quadrangle
c) a simple cycle of length k.
Hints:
1. In an acyclic graph all paths are simple.
2. In c) running time may be exponential in k.
3. Randomization makes solution much easier.
MIN-PLUS MATRIX MULTIPLICATION
An interesting special caseof the APSP problem
A B
17
23
Min-Plus product
2
510
20
30
20
Min-Plus Products
125
703
48
528
5
731
571
252
1036
Solving APSP by repeated squaring
D W for i 1 to log2n do D D*D
If W is an n by n matrix containing the edge weightsof a graph. Then Wn is the distance matrix.
Thus: APSP(n) MPP(n) log n
Actually: APSP(n) = O(MPP(n))
By induction, Wk gives the distances realized by paths that use at most k edges.
(ABD*C)*EBD*
D*CED*GBD*
AB
CD
EF
GH
X =
X* = =
APSP(n) ≤ 2 APSP(n/2) + 6 MPP(n/2) + O(n2)
A D
C
B
Algebraic Product
ij ik kjk
C A B
c a b
O(n2.38)
Min-Plus Product
min operation has no inverse!?
Using matrix multiplicationto compute min-plus products
11 12 11 12 11 12
21 22 21 22 21 22
c c a a b b
c c a a b b
min{ }ij ik kjkc a b
11 12 11 12
21 22 21 22
11 12
21 22
' '
' '
a a b b
a a b b
c c
c c
x x x xx x x x
Using matrix multiplicationto compute min-plus products
11 12 11 12
21 22 21 22
11 12
21 22
' '
' '
a a b b
a a b b
c c
c c
x x x xx x x x
n
polynomial products
M operations per
polynomial product
=Mn
operations per max-plus product
Assume: 0 ≤ aij , bij ≤ M
The min-plus product of two n n matrices can be deduced after only O(n2.5) additions and comparisons.
Fredman’s trick
Breaking a square product into several rectangular products
A2A1
B1
B2
* min *i ii
A B A B
MPP(n) ≤ (n/m) (MPP(n,m,n) + n2)
m
n
Fredman’s trick
A Bn
m
n
m
Naïve calculation requires n2m operations
air+brj ≤ ais+bsj
air - ais ≤ bsj - brj
Fredman observed that the result can be inferred after performing only O(nm2) operations
Fredman’s trick (cont.)
air+brj ≤ ais+bsj air - ais ≤ bsj - brj
• Generate all the differences air - ais and bsj - brj .
• Sort them using O(nm2) comparisons. (Non-trivial!)
• Merge the two sorted lists using O(nm2) comparisons.
The ordering of the elements in the sorted list
determines the result of the min-plus product !!!
Decision Tree Complexity
a17-a19 ≤ b92-b72
c11=a17+b71
c12=a14+b42
...
c11=a13+b31
c12=a15+b52
...
yes no
…
2.5n
…
c11=a18+b81
c12=a16+b62
...
c11=a12+b21
c12=a13+b32
...
All-Pairs Shortest Pathsin directed graphs with “real” edge weights
Running timeAuthorsn3[Floyd ’62] [Warshall ’62]
n3 (log log n / log n)1/3[Fredman ’76]n3 (log log n / log n)1/2[Takaoka ’92]
n3 / (log n)1/2[Dobosiewicz ’90]
n3 (log log n / log n)5/7[Han ’04]
n3 log log n / log n[Takaoka ’04]
n3 (log log n)1/2 / log n[Zwick ’04]
n3 / log n[Chan ’05]
n3 (log log n / log n)5/4[Han ’06]
n3 (log log n)3 / (log n)2[Chan ’07]
UNWEIGHTEDUNDIRECTED
SHORTEST PATHS
4. APSP in undirected graphs
a. An O(n2.38) algorithm for unweighted graphs (Seidel)
b. An O(Mn2.38) algorithm for weighted graphs (Shoshan-Zwick)
5. APSP in directed graphs1. An O(M0.68n2.58) algorithm (Zwick)
2. An O(Mn2.38) preprocessing / O(n) query answering algorithm (Yuster-Zwick)
3. An O(n2.38log M) (1+ε)-approximation algorithm
6. Summary and open problems
Directed versus undirected graphs
xy
z
δ(x,z) δ(x,y) + δ(y,z)
xy
z
δ(x,z) δ(x,y) + δ(y,z)
δ(x,z) ≥ δ(x,y) – δ(y,z)
δ(x,y) δ(x,z) + δ(z,y)Triangle inequality
Inverse triangle inequality
Distances in G and its square G2
Let G=(V,E). Then G2=(V,E2), where (u,v)E2 if and only if (u,v)E or there
exists wV such that (u,w),(w,v)E
Let δ (u,v) be the distance from u to v in G.Let δ2(u,v) be the distance from u to v in G2.
δ(u,v)=5 δ2(u,v)=3
Distances in G and its square G2 (cont.)
Lemma: δ2(u,v)=δ(u,v)/2 , for every u,vV.
Thus: δ(u,v) = 2δ2(u,v) or δ(u,v) = 2δ2(u,v)1
δ2(u,v) ≤δ(u,v)/2
δ(u,v) ≤2δ2(u,v)
Even distances
Lemma: If δ(u,v)=2δ2(u,v) then for every neighbor w of v we have δ2(u,w) ≥ δ2(u,v).
Let A be the adjacency matrix of the G.Let C be the distance matrix of G2
Odd distances
Lemma: If δ(u,v)=2δ2(u,v)–1 then for every neighbor w of v we have δ2(u,w) δ2(u,v) and
for at least one neighbor δ2(u,w) < δ2(u,v).
Let A be the adjacency matrix of the G.Let C be the distance matrix of G2
Exercise 2: Prove the lemma.
Even distances
Lemma: If δ(u,v)=2δ2(u,v) then for every neighbor w of v we have δ2(u,w) ≥ δ2(u,v).
Let A be the adjacency matrix of the G.Let C be the distance matrix of G2
Seidel’s algorithm
Algorithm APD(A)if A=J then return J–Ielse C←APD(A2) X←CA , deg←Ae–1 dij←2cij– [xij< cijdegj] return Dend
1. If A is an all one matrix, then all distances are 1.
2. Compute A2, the adjacency matrix of the squared graph.
3. Find, recursively, the distances in the squared graph.
4. Decide, using one integer matrix multiplication, for every two vertices u,v, whether their distance is twice the distance in the square, or twice minus 1.
Complexity: O(nlog n)
Assume that A has 1’s on the diagonal.
Boolean matrix multiplicaion
Integer matrix multiplicaion
Exercise 3: (*) Obtain a version of Seidel’s algorithm that uses only Boolean matrix multiplications.
Hint: Look at distances also modulo 3.
Distances vs. Shortest Paths
We described an algorithm for computing all distances.
How do we get a representation of the shortest paths?
We need witnesses for the Boolean matrix multiplication.
Witnesses for Boolean Matrix Multiplication
Can be computed naively in O(n3) time.
A matrix W is a matrix of witnesses iff
Can also be computed in O(nlog n) time.
Exercise 4:
a) Obtain a deterministic O(n)-time algorithm for finding unique witnesses.
b) Let 1 ≤ d ≤ n be an integer. Obtain a randomized O(n)-time algorithm for finding witnesses for all positions that have between d and 2d witnesses.
c) Obtain an O(nlog n)-time algorithm for finding all witnesses.
Hint: In b) use sampling.
Running timeAuthors
Mn[Shoshan-Zwick ’99]
All-Pairs Shortest Pathsin graphs with small integer weights
Undirected graphs. Edge weights in {0,1,…M}
Improves results of [Alon-Galil-Margalit ’91] [Seidel ’95]
DIRECTEDSHORTEST PATHS
Exercise 5: Obtain an O(nlog n) time algorithm for
computing the diameter of an unweighted directed graph.
Using matrix multiplicationto compute min-plus products
11 12 11 12 11 12
21 22 21 22 21 22
c c a a b b
c c a a b b
min{ }ij ik kjkc a b
11 12 11 12
21 22 21 22
11 12
21 22
' '
' '
a a b b
a a b b
c c
c c
x x x xx x x x
Using matrix multiplicationto compute min-plus products
11 12 11 12
21 22 21 22
11 12
21 22
' '
' '
a a b b
a a b b
c c
c c
x x x xx x x x
n
polynomial products
M operations per
polynomial product
=Mn
operations per max-plus product
Assume: 0 ≤ aij , bij ≤ M
Trying to implement the repeated squaring algorithm
Consider an easy case: all weights are 1.
D W for i 1 to log2n do D D*D
After the i-th iteration, the finite elements in D are in the range {1,…,2i}.
The cost of the min-plus product is 2i n
The cost of the last product is n+1 !!!
Sampled Repeated Squaring (Z ’98)
D Wfor i 1 to log3/2n do{
s (3/2)i+1
B rand( V , (9n ln n)/s )D min{ D , D[V,B]*D[B,V] }
}
Choose a subset of V of size (9n ln n)/s
Select the columns of D whose
indices are in B
Select the rowsof D whose
indices are in B
With high probability, all distances are correct!
The is also a slightly more complicated deterministic algorithm
Sampled Distance Products (Z ’98)
n
n
n
|B|
In the i-th iteration, the set B is of size n ln n / s, where s = (3/2)i+1
The matrices get smaller and smaller
but the elements get
larger and larger
Sampled Repeated Squaring - Correctness
D Wfor i 1 to log3/2n do{
s (3/2)i+1
B rand(V,(9 ln n)/s)D min{ D , D[V,B]*D[B,V] }
}
Invariant: After the i-th iteration, distances that are attained using at most (3/2)i
edges are correct.
Consider a shortest path that uses at most (3/2)i+1 edges
12
32
i
12
32
i 12
32
iat most at most
Let s = (3/2)i+13
/39ln(1 )
sn
ns
Failure probability :
Rectangular Matrix multiplication
[Coppersmith ’97]: n1.85p0.54+n2+o(1)
For p ≤ n0.29, complexity = n2+o(1) !!!
=n
p
p
n
n
n
Naïve complexity: n2p
Complexity of APSP algorithm
The i-th iteration:
n
n ln n / s
n n ln n / s
s=(3/2)i+1
The elements are of absolute value
at most Ms
0.54 31.85min{ , }
n nMs n
s s
0.68 2.58M n
Open problem:Can APSP in directed graphs
be solved in O(n) time?
Related result: [Yuster-Zwick’04]A directed graphs can be processed in O(n)
time so that any distance query can be answered in O(n) time.
Corollary:SSSP in directed graphs in O(n) time.
The corollary obtained using a different technique by Sankowski (2004)
The preprocessing algorithm (YZ ’05)
D W ; B Vfor i 1 to log3/2n do{
s (3/2)i+1
B rand(B,(9n ln n)/s)D[V,B] min{D[V,B] , D[V,B]*D[B,B] }D[B,V] min{D[B,V] , D[B,B]*D[B,V] }
}
The APSP algorithm
D Wfor i 1 to log3/2n do{
s (3/2)i+1
B rand(V,(9nln n)/s)
}
D min{ D , D[V,B]*D[B,V] }
Twice Sampled Distance Products
n
n
n
|B|
n
|B|
|B|
|B||B|
n
The query answering algorithm
δ(u,v) D[{u},V]*D[V,{v}]
u
v
Query time: O(n)
The preprocessing algorithm: Correctness
Invariant: After the i-th iteration, if u Bi or vBi and there is a shortest path from u to v that uses at
most (3/2)i edges, then D(u,v)=δ(u,v).
Let Bi be the i-th sample. B1 B2 B3 …
Consider a shortest path that uses at most (3/2)i+1 edges
12
32
i
12
32
i 12
32
iat most at most
The query answering algorithm: Correctness
Suppose that the shortest path from u to v uses between (3/2)i and (3/2)i+1 edges
12
32
i
12
32
i 12
32
iat most at most
u v
1. Algebraic matrix multiplicationa. Strassen’s algorithm
b. Rectangular matrix multiplication
2. Min-Plus matrix multiplicationa. Equivalence to the APSP problem
b. Expensive reduction to algebraic products
c. Fredman’s trick
3. APSP in undirected graphsa. An O(n2.38) anlgorithm for unweighted graphs (Seidel)
b. An O(Mn2.38) algorithm for weighted graphs (Shoshan-Zwick)
4. APSP in directed graphs1. An O(M0.68n2.58) algorithm (Zwick)
2. An O(Mn2.38) preprocessing / O(n) query answering alg. (Yuster-Z)
3. An O(n2.38logM) (1+ε)-approximation algorithm5. Summary and open problems
Approximate min-plus products
Obvious idea: scaling
SCALE(A,M,R):/ , if 0
, otherwise
ij ijij
Ra M a Ma
APX-MPP(A,B,M,R) : A’←SCALE(A,M,R)B’←SCALE(B,M,R)return MPP(A’,B’)
Complexity is Rn2.38, instead of Mn2.38, but small values can be
greatly distorted.
Addaptive Scaling
APX-MPP(A,B,M,R) :
C’←∞for r←log2R to log2M do A’←SCALE(A,2r,R) B’←SCALE(B,2r,R) C’←min{C’,MPP(A’,B’)}end
Complexity is Rn2.38 logMStretch at most 1+4/R
1. Algebraic matrix multiplicationa. Strassen’s algorithm
b. Rectangular matrix multiplication
2. Min-Plus matrix multiplicationa. Equivalence to the APSP problem
b. Expensive reduction to algebraic products
c. Fredman’s trick
3. APSP in undirected graphsa. An O(n2.38) anlgorithm for unweighted graphs (Seidel)
b. An O(Mn2.38) algorithm for weighted graphs (Shoshan-Zwick)
4. APSP in directed graphs1. An O(M0.68n2.58) algorithm (Zwick)
2. An O(Mn2.38) preprocessing / O(n) query answering alg. (Yuster-Z)
3. An O(n2.38logM) (1+ε)-approximation algorithm
5. Summary and open problems
Running timeAuthors
Mn2.38[Shoshan-Zwick ’99]
All-Pairs Shortest Pathsin graphs with small integer weights
Undirected graphs. Edge weights in {0,1,…M}
Improves results of [Alon-Galil-Margalit ’91] [Seidel ’95]
All-Pairs Shortest Pathsin graphs with small integer weights
Running timeAuthors
M0.68 n2.58[Zwick ’98]
Directed graphs. Edge weights in {−M,…,0,…M}
Improves results of [Alon-Galil-Margalit ’91] [Takaoka ’98]
Answering distance queries
Preprocessing time
Querytime
Authors
Mn2.38n[Yuster-Zwick ’05]
Directed graphs. Edge weights in {−M,…,0,…M}
In particular, any Mn1.38 distances can be computed in Mn2.38 time.
For dense enough graphs with small enough edge weights, this improves on Goldberg’s SSSP algorithm.
Mn2.38 vs. mn0.5log M
Running timeAuthors
(n2.38 log M)/ε[Zwick ’98]
Approximate All-Pairs Shortest Pathsin graphs with non-negative integer weights
Directed graphs. Edge weights in {0,1,…M}
(1+ε)-approximate distances
Open problems
• An O(n2.38) algorithm for the directed unweighted APSP problem?
• An O(n3-ε) algorithm for the APSP problem with edge weights in {1,2,…,n}?
• An O(n2.5-ε) algorithm for the SSSP problem with edge weights in {0,±1, ±2,…, ±n}?