![Page 1: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/1.jpg)
Recent Advances in Two-Dimensional Sparse Matrix Partitioning
SIAM PP10 2/26/2010
Michael Wolf, Erik Boman, Cédric Chevalier
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000."
![Page 2: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/2.jpg)
2
Sparse Matrix Partitioning Motivation
• Sparse matrix-vector multiplication (SpMV) is common kernel in many numerical computations - Iterative methods for solving linear systems - PageRank computation - …
• Need to make parallel SpMV kernel as fast as possible
![Page 3: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/3.jpg)
3
Parallel Sparse Matrix-Vector Multiplication
• Partition matrix nonzeros • Partition vectors
12431421
15000400
61800000
09120000
00710060
05001300
00008190
00000312
00070041
y1y2y3y4y5y6y7y8
![Page 4: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/4.jpg)
4
Objective
• Ideally we minimize total run-time • Settle for easier objective
– Work balanced – Minimize total communication volume
• Can partition matrices in different ways – 1D – 2D
• Can model problem in different ways – Graph – Bipartite graph – Hypergraph
![Page 5: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/5.jpg)
5
Parallel Matrix-Vector Multiplication
• Alternative way of visualizing partitioning
12431421
15000400
61800000
09120000
00710060
05001300
00008190
00000312
00070041
y1y2y3y4y5y6y7y8
![Page 6: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/6.jpg)
6
Parallel SpMV Communication
• sent to remote processes that have nonzeros in column
• Partial inner-products sent to process that owns vector element
![Page 7: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/7.jpg)
7
1D Partitioning
• Each process assigned nonzeros for set of columns
1D Column
• Each process assigned nonzeros for set of rows
1D Row
![Page 8: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/8.jpg)
8
When 1D Partitioning is Inadequate
n=12 nnz=34 (18,16) volume = 9
“Arrowhead” matrix
• For any 1D bisection of nxn arrowhead matrix: – nnz = 3n-2 – Volume ≈ (3/4)n
![Page 9: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/9.jpg)
9
When 1D Partitioning is Inadequate
n=12 nnz=34 (16,18) volume = 2
“Arrowhead” matrix
• 2D partitioning • O(k) volume partitioning possible
![Page 10: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/10.jpg)
10
2D Partitioning
• More flexibility in partitioning • No particular part for given row or column • More general sets of nonzeros assigned parts • Several methods of 2D partitioning
– Fine-grain hypergraph – Coarse-grain hypergraph – Mondriaan – Nested dissection symmetric partitioning method
![Page 11: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/11.jpg)
11
Fine-Grain (FG) Hypergraph Model
• Catalyurek and Aykanat (2001)
• Nonzeros represented by vertices in hypergraph
![Page 12: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/12.jpg)
12
Fine-Grain Hypergraph Model
• Rows represented by hyperedges
• Hyperedge - set of one or more vertices
![Page 13: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/13.jpg)
13
Fine-Grain Hypergraph Model
• Columns represented by hyperedges
![Page 14: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/14.jpg)
14
Fine-Grain Hypergraph Model
• 2n hyperedges
![Page 15: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/15.jpg)
15
Fine-Grain Hypergraph Model
• Partition vertices into k equal sets
• For k=2 – Volume = number of
hyperedges cut • Minimum volume
partitioning when optimally solved
• Larger NP-hard problem than 1D k=2, volume = cut = 2
![Page 16: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/16.jpg)
16
2D Coarse-Grain Partitioning
• Catalyurek and Aykanat (2001) • Two stages:
– 1D hypergraph partitioning – 1D multi-constraint hypergraph partitioning
(ensures load balance) • Bound on number of messages
![Page 17: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/17.jpg)
17
Mondriaan Partitioning
• Vastenhouw and Bisseling (2005) • Recursive bisection hypergraph partitioning • Each level: 1D row or column partitioning
![Page 18: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/18.jpg)
18
Nested Dissection Partitioning - Bisection
• Suppose A is structurally symmetric • Let G(V,E) be graph of A • Find small, balanced separator S
– Yields vertex partitioning V = (V0,V1,S) • Partition the edges such that
– E0 = {edges incident to a vertex in V0} – E1 = {edges incident to a vertex in V1}
![Page 19: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/19.jpg)
19
Nested Dissection Partitioning - Bisection
• Vertices in S and corresponding edges – Can be assigned to either part – Can use flexibility to maintain balance
• Communication Volume = 2*|S| – Regardless of S partitioning – |S| in each phase
![Page 20: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/20.jpg)
20
Nested Dissection (ND) Partitioning Method
• Recursive bisection to partition into >2 parts • Use nested dissection!
![Page 21: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/21.jpg)
21
Numerical Experiments
• Structurally symmetric matrices • k = 4, 16, 64 parts using
– 1D hypergraph partitioning – Fine-grain hypergraph partitioning (2D)
• Good quality partitions but slow – Nested dissection partitioning (2D)
• Hypergraph partitioning for all methods – Zoltan (Sandia) with PaToH (Catalyurek, serial) – Allows “fair” comparison between methods
• Vertex separators derived from edge separators – MatchBox (Purdue: Pothen, et al.)
![Page 22: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/22.jpg)
22
Communication Volume - Symmetric Matrices finan512
bcsstk30
wor
ds
cage10 w
ords
bcsstk32
wor
ds
wor
ds
Test matrices from Rob Bisseling (Utrecht)
![Page 23: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/23.jpg)
23
Runtimes of Partitioning Methods cage10
bcsstk32
seco
nds
finan512
seco
nds
seco
nds
bcsstk30
seco
nds
![Page 24: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/24.jpg)
24
Communication Volume: 1D is Inadequate
wor
ds
• c-73: nonlinear optimization (Schenk) - UF sparse matrix collection - n=169,422 nnz=1,279,274
![Page 25: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/25.jpg)
25
Communication Volume: 1D is Inadequate
wor
ds
• asic680ks: Xyce circuit simulation (Sandia) - n=682,712 nnz=2,329,176
![Page 26: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/26.jpg)
26
Parallel Sparse Matrix Partitioning Software
• Developing HPC software for sparse matrix partitioning – 1D – 2D
• Idea is to implement sparse matrix partitioning algorithms in parallel – Efficient/fast – Simple to use
• Leverage existing software – Graph/hypergraph partitioners – Linear algebra packages
• Trilinos framework
![Page 27: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/27.jpg)
27
Trilinos and Isorropia
• Trilinos – Framework for solving large-scale scientific problems – Focus on packages (independent pieces of software that are combined to solve these problems)
– Epetra: parallel linear algebra package • Isorropia
– Trilinos package for combinatorial scientific computing – Partitioning, coloring, ordering algorithms applied to
Epetra matrices – Utilizes many algorithms in Zoltan – “Zoltan for sparse matrices”
![Page 28: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/28.jpg)
28
Isorropia: Sparse Matrix Partitioning Methods
• Parallel partitioning methods • Currently exist
– 1D linear/block, cyclic, random (New!) – 1D hypergraph – 1D graph – 2D fine-grain hypergraph (New!)
• Planned – 2D linear/block, cyclic, random – 2D RCB partitioning (of nonzeros) – 2D nested dissection – Vector partitioning (for 2D matrix partitioning)
![Page 29: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/29.jpg)
29
Isorropia: Partitioning Example 1
• Simple partitioning of rowmatrix – 1D row hypergraph partitioning – Balancing number of nonzeros – Load imbalance tolerance of 1.03
using I s o r r o p i a : : Epetra : : P a r t i t i o n e r ;
ParameterList params ;params . s e t ( "PARTITIONING_METHOD" , "HYPERGRAPH" ) ;params . s e t ( "BALANCE�OBJECTIVE" , "NONZEROS" ) ;params . s e t ( "IMBALANCE�TOL" , " 1 .03 " ) ;
// rowmatrix i s an Epetra_RowMatrixPa r t i t i o n e r p a r t i t i o n e r ( rowmatrix , params , false ) ;p a r t i t i o n e r . p a r t i t i o n ( ) ;
using I s o r r o p i a : : Epetra : : P a r t i t i o n e r ;
ParameterList params ;params . s e t ( "PARTITIONING_METHOD" , "HYPERGRAPH" ) ;params . s e t ( "BALANCE�OBJECTIVE" , "NONZEROS" ) ;params . s e t ( "IMBALANCE�TOL" , " 1 .03 " ) ;
// rowmatrix i s an Epetra_RowMatrixPa r t i t i o n e r p a r t i t i o n e r ( rowmatrix , params , false ) ;p a r t i t i o n e r . p a r t i t i o n ( ) ;
![Page 30: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/30.jpg)
30
Isorropia: Partitioning Example 2
• 2D partitioning of rowmatrix – 2D fine-grain hypergraph partitioning – Balancing number of nonzeros (implicit) – Load imbalance tolerance of 1.03
using I s o r r o p i a : : Epetra : : Part i t i oner2D ;
ParameterList params ;params . s e t ( "PARTITIONING_METHOD" , "HGRAPH2D_FINEGRAIN" ) ;params . s e t ( "IMBALANCE�TOL" , " 1 .03 " ) ;
// rowmatrix i s an Epetra_RowMatrixPart i t i oner2D pa r t i t i o n e r ( rowmatrix , params , false ) ;p a r t i t i o n e r . p a r t i t i o n ( ) ;
using I s o r r o p i a : : Epetra : : Part i t i oner2D ;
ParameterList params ;params . s e t ( "PARTITIONING_METHOD" , "HGRAPH2D_FINEGRAIN" ) ;params . s e t ( "IMBALANCE�TOL" , " 1 .03 " ) ;
// rowmatrix i s an Epetra_RowMatrixPart i t i oner2D pa r t i t i o n e r ( rowmatrix , params , false ) ;p a r t i t i o n e r . p a r t i t i o n ( ) ;
![Page 31: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/31.jpg)
31
Isorropia: Redistributing Matrix Data
• After partitioning matrix – Build Redistributor from new partition – Redistribute data based on new partition – Obtain new matrix
pa r t i t i o n e r�>pa r t i t i o n ( ) ;
// Set up Red i s t r i b u t o r based on p a r t i t i o nI s o r r o p i a : : Epetra : : Red i s t r i bu to r rd ( p a r t i t i o n e r ) ;
// Red i s t r i b u t e datanewmatrix = rd . r e d i s t r i b u t e (⇥ rowmatrix , true ) ;
![Page 32: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/32.jpg)
32
Isorropia: Redistributing Matrix Data
• Shortcut – Combines partitioning/redistibution of data
using I s o r r o p i a : : Epetra : : createBalancedCopy ;
ParameterList params ;params . s e t ( "IMBALANCE�TOL" , " 1 .03 " ) ;params . s e t ( "BALANCE�OBJECTIVE" , "NONZEROS" ) ;params . s e t ( "PARTITIONING_METHOD" , "HYPERGRAPH" ) ;
// crsmatr ix and newmatrix are Epetra_CrsMatrixnewmatrix = createBalancedCopy (� crsmatr ix , params ) ;
![Page 33: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/33.jpg)
33
Isorropia: Preliminary results
• Isorropia and Epetra can be used to study matrix partitioning – Easy to experiment with different matrix partitionings – Can see impact of partitionings on different Epetra
parallel linear algebra kernels • Numerical experiments
– Runtime of SpMV for different matrix partitionings – 3 different methods: 1D linear, 1D hypergraph, 2D fine-grain
– Parallel implementations of partitioning methods – Test problems: bcsstk30, bcsstk32, c-73, asic680ks
![Page 34: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/34.jpg)
34
Isorropia: Preliminary results
• Platforms – NERSC Franklin (Cray XT4, Opteron 2.3 GHz quad core)
– SNL Odin cluster (dual 2.2GHz Opteron, Myrinet)
NERSC Franklin
![Page 35: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/35.jpg)
35
Isorropia: SpMV Timings (Franklin)
c-73
Run
time
asic680ks
Run
time
bcsstk32 R
untim
e bcsstk30
Run
time
![Page 36: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/36.jpg)
36
SpMV Timings (Franklin, normalized)
c-73
Run
time
asic680ks
Run
time
bcsstk32 R
untim
e bcsstk30
Run
time
![Page 37: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/37.jpg)
37
Isorropia: SpMV Timings (Odin)
c-73
Run
time
asic680ks
Run
time
bcsstk32 R
untim
e bcsstk30
Run
time
![Page 38: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/38.jpg)
38
Summary
• Motivation for and overview of 2D partitioning • New 2D matrix partitioning algorithm • ND matrix partitioning algorithm
– ND used in new context – Good trade off between communication volume and
partitioning time • Communication volume (comparable to fine-grain) • Partitioning time (comparable to 1D)
• Presented simple framework for sparse matrix partitioning for Trilinos/Epetra applications – First production code that supports parallel 2D
sparse matrix partitioning
![Page 39: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/39.jpg)
39
Summary of Isorropia Work
• Mixed results for SpMV runtimes – Decrease not proportional to decrease in
communication volume – Results for bcsstk30 and bcsstk32 not significantly
better than linear • 2D FG worse than 1D hypergraph
– Improvement over linear for asic680k and c-73 • 2D FG significantly better than 1D hypergraph for some k
• 2D partitioning can be effective for some matrices • Improvements needed to make 2D methods viable
– Room for improvement (e.g., PHG for FG) • 2D fine-grain partitioning in next Trilinos
release
![Page 40: Recent Advances in Two-Dimensional Sparse Matrix Partitioningmmwolf/presentations/Conferences/wolfPP10.pdf · Recent Advances in Two-Dimensional Sparse Matrix Partitioning SIAM PP10](https://reader033.vdocuments.site/reader033/viewer/2022050308/5f7012d80e3d8a24906eb014/html5/thumbnails/40.jpg)
40
Selection of Related Papers/Info 2D Partitioning:
U. Catalyurek and C. Aykanat, “A fine-grain hypergraph model for 2d decomposition of sparse matrices,” In Proc. IPDPS 8th Int’l Workshop on Solving Irregularly Structured Problems in Parallel (Irregular 2001), April 2001.
U. Catalyurek, C. Aykanat, and B. Ucar. On two-dimensional sparse matrix partitioning: Models, methods, and a recipe. To appear in SIAM Journal on Scientific Computing.
B. Vastenhouw and R. H. Bisseling. A two-dimensional data distribution method for parallel sparse matrix-vector multiplication. SIAM Review, 47(1):67–95, 2005.
Nested Dissection Partitioning: E.G. Boman and M.M. Wolf, “A Nested Dissection Approach to Sparse Matrix Partitioning for Parallel Computations,” SANDIA Technical Report 2008-5482J. (Submitted for publication)
M. Wolf, E. Boman, and C. Chevalier, “Improved Parallel Data Partitioning by Nested Dissection with Applications to Information Retrieval,” SANDIA Technical Report 2008-7908J.
Trilinos/Isorropia: http://trilinos.sandia.gov http://trilinos.sandia.gov/packages/isorropia/