automatic performance tuning sparse matrix kernels james demmel demmel/cs267_spr05
Post on 21-Dec-2015
215 views
TRANSCRIPT
![Page 1: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/1.jpg)
Automatic Performance TuningSparse Matrix Kernels
James Demmel
www.cs.berkeley.edu/~demmel/cs267_Spr05
![Page 2: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/2.jpg)
Berkeley Benchmarking and OPtimization (BeBOP)
• Prof. Katherine Yelick• Rich Vuduc
– Many results in this talk are from Vuduc’s PhD thesis, www.cs.berkeley.edu/~richie
• Rajesh Nishtala, Mark Hoemmen, Hormozd Gahvari
• Eun-Jim Im, many other earlier contributors• bebop.cs.berkeley.edu
![Page 3: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/3.jpg)
Outline
• Motivation for Automatic Performance Tuning• Recent results for sparse matrix kernels• OSKI = Optimized Sparse Kernel Interface• Future Work
![Page 4: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/4.jpg)
Motivation for Automatic Performance Tuning• Writing high performance software is hard
– Make programming easier while getting high speed
• Ideal: program in your favorite high level language (Matlab, Python, PETSc…) and get a high fraction of peak performance
• Reality: Best algorithm (and its implementation) can depend strongly on the problem, computer architecture, compiler,…– Best choice can depend on knowing a lot of applied
mathematics and computer science
• How much of this can we teach?• How much of this can we automate?
![Page 5: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/5.jpg)
Examples of Automatic Performance Tuning (1)
• Dense BLAS– Sequential– PHiPAC (UCB), then ATLAS (UTK)– Now in Matlab, many other releases– math-atlas.sourceforge.net/
• Fast Fourier Transform (FFT) & variations– Sequential and Parallel– FFTW (MIT)– Widely used– www.fftw.org
• Digital Signal Processing– SPIRAL: www.spiral.net (CMU)
• MPI Collectives (UCB, UTK)• More projects, conferences, government reports, …
![Page 6: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/6.jpg)
Examples of Automatic Performance Tuning (2)
• What do dense BLAS, FFTs, signal processing, MPI reductions have in common?– Can do the tuning off-line: once per architecture,
algorithm– Can take as much time as necessary (hours, a week…)– At run-time, algorithm choice may depend only on few
parameters• Matrix dimension, size of FFT, etc.
• Can’t always do off-line tuning– Algorithm and implementation may strongly depend on
data only known at run-time– Ex: Sparse matrix nonzero pattern determines both best
data structure and implementation of Sparse-matrix-vector-multiplication (SpMV)
– BEBOP project addresses this
![Page 7: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/7.jpg)
Tuning Dense BLAS —PHiPAC
![Page 8: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/8.jpg)
Tuning Dense BLAS– ATLAS
Extends applicability of PHIPAC; Incorporated in Matlab (with rest of LAPACK)
![Page 9: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/9.jpg)
Tuning Register Tile Sizes (Dense Matrix Multiply)
333 MHz Sun Ultra 2i
2-D slice of 3-D space; implementations color-coded by performance in Mflop/s
16 registers, but 2-by-3 tile size fastest
Needle in a haystack
![Page 10: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/10.jpg)
![Page 11: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/11.jpg)
A Sparse Matrix You Encounter Every Day
![Page 12: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/12.jpg)
Matrix-vector multiply kernel: y(i) y(i) + A(i,j)*x(j)
for each row i
for k=ptr[i] to ptr[i+1] do
y[i] = y[i] + val[k]*x[ind[k]]
SpMV with Compressed Sparse Row (CSR) Storage
Matrix-vector multiply kernel: y(i) y(i) + A(i,j)*x(j)
for each row i
for k=ptr[i] to ptr[i+1] do
y[i] = y[i] + val[k]*x[ind[k]]
![Page 13: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/13.jpg)
Motivation for Automatic Performance Tuning of SpMV
• Historical trends– Sparse matrix-vector multiply (SpMV): 10% of peak or
less– 2x faster than CSR with “hand-tuning”– Tuning becoming more difficult over time
• Performance depends on machine, kernel, matrix– Matrix known at run-time– Best data structure + implementation can be surprising
• Our approach: empirical modeling and search– Up to 4x speedups and 31% of peak for SpMV– Many optimization techniques for SpMV– Several other kernels: triangular solve, ATA*x, Ak*x– Release OSKI Library, integrate into PETSc
![Page 14: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/14.jpg)
SpMV Historical Trends: Fraction of Peak
![Page 15: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/15.jpg)
Example: The Difficulty of Tuning
• n = 21216• nnz = 1.5 M• kernel: SpMV
• Source: NASA structural analysis problem
• 8x8 dense substructure
![Page 16: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/16.jpg)
Taking advantage of block structure in SpMV
• Bottleneck is time to get matrix from memory– Only 2 flops for each nonzero in matrix
• Don’t store each nonzero with index, instead store each nonzero r-by-c block with index– Storage drops by up to 2x, if rc >> 1, all 32-bit
quantities– Time to fetch matrix from memory decreases
• Change both data structure and algorithm– Need to pick r and c– Need to change algorithm accordingly
• In example, is r=c=8 best choice?– Minimizes storage, so looks like a good idea…
![Page 17: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/17.jpg)
Speedups on Itanium 2: The Need for Search
Reference
Best: 4x2
Mflop/s
Mflop/s
![Page 18: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/18.jpg)
Register Profile: Itanium 2
190 Mflop/s
1190 Mflop/s
![Page 19: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/19.jpg)
SpMV Performance (Matrix #2): Generation 2
Ultra 2i - 9% Ultra 3 - 5%
Pentium III-M - 15%Pentium III - 19%
63 Mflop/s
35 Mflop/s
109 Mflop/s
53 Mflop/s
96 Mflop/s
42 Mflop/s
120 Mflop/s
58 Mflop/s
![Page 20: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/20.jpg)
Register Profiles: Sun and Intel x86
Ultra 2i - 11% Ultra 3 - 5%
Pentium III-M - 15%Pentium III - 21%
72 Mflop/s
35 Mflop/s
90 Mflop/s
50 Mflop/s
108 Mflop/s
42 Mflop/s
122 Mflop/s
58 Mflop/s
![Page 21: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/21.jpg)
SpMV Performance (Matrix #2): Generation 1
Power3 - 13% Power4 - 14%
Itanium 2 - 31%Itanium 1 - 7%
195 Mflop/s
100 Mflop/s
703 Mflop/s
469 Mflop/s
225 Mflop/s
103 Mflop/s
1.1 Gflop/s
276 Mflop/s
![Page 22: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/22.jpg)
Register Profiles: IBM and Intel IA-64
Power3 - 17% Power4 - 16%
Itanium 2 - 33%Itanium 1 - 8%
252 Mflop/s
122 Mflop/s
820 Mflop/s
459 Mflop/s
247 Mflop/s
107 Mflop/s
1.2 Gflop/s
190 Mflop/s
![Page 23: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/23.jpg)
Example: The Difficulty of Tuning
• n = 21216• nnz = 1.5 M• kernel: SpMV
• Source: NASA structural analysis problem
![Page 24: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/24.jpg)
Zoom in to top corner
• More complicated non-zero structure in general
![Page 25: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/25.jpg)
3x3 blocks look natural, but…
• More complicated non-zero structure in general
• Example: 3x3 blocking– Logical grid of 3x3 cells
• But would lead to lots of “fill-in”
![Page 26: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/26.jpg)
Extra Work Can Improve Efficiency!
• More complicated non-zero structure in general
• Example: 3x3 blocking– Logical grid of 3x3 cells– Fill-in explicit zeros– Unroll 3x3 block multiplies– “Fill ratio” = 1.5
• On Pentium III: 1.5x speedup!– Actual mflop rate 1.52 =
2.25 higher
![Page 27: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/27.jpg)
Automatic Register Block Size Selection
• Selecting the r x c block size– Off-line benchmark
• Precompute Mflops(r,c) using dense A for each r x c• Once per machine/architecture
– Run-time “search”• Sample A to estimate Fill(r,c) for each r x c
– Run-time heuristic model• Choose r, c to maximize Mflops(r,c) / Fill(r,c)
![Page 28: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/28.jpg)
Accurate and Efficient Adaptive Fill Estimation
• Idea: Sample matrix– Fraction of matrix to sample: s [0,1]– Cost ~ O(s * nnz)– Control cost by controlling s
• Search at run-time: the constant matters!
• Control s automatically by computing statistical confidence intervals– Idea: Monitor variance
• Cost of tuning– Lower bound: convert matrix in 5 to 40 unblocked
SpMVs– Heuristic: 1 to 11 SpMVs
![Page 29: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/29.jpg)
Accuracy of the Tuning Heuristics (1/4)
NOTE: “Fair” flops used (ops on explicit zeros not counted as “work”)See p. 375 of Vuduc’s thesis for matrices
![Page 30: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/30.jpg)
Accuracy of the Tuning Heuristics (2/4)
![Page 31: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/31.jpg)
Accuracy of the Tuning Heuristics (3/4)
![Page 32: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/32.jpg)
Accuracy of the Tuning Heuristics (3/4)DGEMV
![Page 33: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/33.jpg)
Evaluating algorithms and machines for SpMV
• Metrics– Speedups– Mflop/s (“fair” flops)– Fraction of peak
• Questions– Speedups are good, but what is “the best?”
• Independent of instruction scheduling, selection• Can SpMV be further improved or not?
– What machines are “good” for SpMV?
![Page 34: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/34.jpg)
Upper Bounds on Performance for blocked SpMV
• P = (flops) / (time)– Flops = 2 * nnz(A)
• Lower bound on time: Two main assumptions– 1. Count memory ops only (streaming)– 2. Count only compulsory, capacity misses: ignore conflicts
• Account for line sizes• Account for matrix size and nnz
• Charge minimum access “latency” i at Li cache & mem
– e.g., Saavedra-Barrera and PMaC MAPS benchmarks
1mem11
1memmem
Misses)(Misses)(Loads
HitsHitsTime
iiii
iii
![Page 35: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/35.jpg)
Example: L2 Misses on Itanium 2
Misses measured using PAPI [Browne ’00]
![Page 36: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/36.jpg)
Example: Bounds on Itanium 2
![Page 37: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/37.jpg)
Example: Bounds on Itanium 2
![Page 38: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/38.jpg)
Example: Bounds on Itanium 2
![Page 39: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/39.jpg)
Fraction of Upper Bound Across Platforms
![Page 40: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/40.jpg)
Achieved Performance and Machine Balance
• Machine balance [Callahan ’88; McCalpin ’95]– Balance = Peak Flop Rate / Bandwidth (flops /
double)– Lower is better (I.e. can hope to get higher fraction
of peak flop rate)
• Ideal balance for mat-vec: 2 flops / double– For SpMV, even less
![Page 41: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/41.jpg)
![Page 42: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/42.jpg)
Where Does the Time Go?
• Most time assigned to memory• Caches “disappear” when line sizes are equal
– Strictly increasing line sizes
1
memmem HitsHitsTimei
ii
![Page 43: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/43.jpg)
Execution Time Breakdown: Matrix 40
![Page 44: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/44.jpg)
Execution Time Breakdown (PAPI): Matrix 40
![Page 45: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/45.jpg)
Speedups with Increasing Line Size
![Page 46: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/46.jpg)
Summary: Performance Upper Bounds
• What is the best we can do for SpMV?– Limits to low-level tuning of blocked implementations– Refinements?
• What machines are good for SpMV?– Partial answer: balance characterization
• Architectural consequences?– Help to have strictly increasing line sizes
![Page 47: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/47.jpg)
Summary of Other Performance Optimizations
• Optimizations for SpMV– Register blocking (RB): up to 4x over CSR– Variable block splitting: 2.1x over CSR, 1.8x over RB– Diagonals: 2x over CSR– Reordering to create dense structure + splitting: 2x
over CSR– Symmetry: 2.8x over CSR, 2.6x over RB– Cache blocking: 2.8x over CSR– Multiple vectors (SpMM): 7x over CSR– And combinations…
• Sparse triangular solve– Hybrid sparse/dense data structure: 1.8x over CSR
• Higher-level kernels– AAT*x, ATA*x: 4x over CSR, 1.8x over RB– A*x: 2x over CSR, 1.5x over RB
![Page 48: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/48.jpg)
Example: Sparse Triangular Factor
• Raefsky4 (structural problem) + SuperLU + colmmd
• N=19779, nnz=12.6 M
Dense trailing triangle: dim=2268, 20% of total nz
Can be as high as 90+%!1.8x over CSR
![Page 49: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/49.jpg)
Cache Optimizations for AAT*x
• Cache-level: Interleave multiplication by A, AT
• Register-level: aiT to be rc block row, or diag row
n
i
Tii
Tn
T
nT xaax
a
a
aaxAA1
1
1 )(
dot product“axpy”
• Algorithmic-level transformations for A2*x, A3*x, …
… …
![Page 50: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/50.jpg)
Example: Combining Optimizations
• Register blocking, symmetry, multiple (k) vectors– Three low-level tuning parameters: r, c, v
v
kX
Y A
cr
+=
*
![Page 51: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/51.jpg)
Example: Combining Optimizations
• Register blocking, symmetry, and multiple vectors [Ben Lee @ UCB]– Symmetric, blocked, 1 vector
• Up to 2.6x over nonsymmetric, blocked, 1 vector
– Symmetric, blocked, k vectors• Up to 2.1x over nonsymmetric, blocked, k vecs.• Up to 7.3x over nonsymmetric, nonblocked, 1, vector
– Symmetric Storage: 64.7% savings
![Page 52: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/52.jpg)
Potential Impact on Applications: T3P
• Application: accelerator design [Ko] • 80% of time spent in SpMV• Relevant optimization techniques
– Symmetric storage– Register blocking
• On Single Processor Itanium 2– 1.68x speedup
• 532 Mflops, or 15% of 3.6 GFlop peak– 4.4x speedup with multiple (8) vectors
• 1380 Mflops, or 38% of peak
![Page 53: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/53.jpg)
Potential Impact on Applications: Omega3P
• Application: accelerator cavity design [Ko]• Relevant optimization techniques
– Symmetric storage– Register blocking– Reordering
• Reverse Cuthill-McKee ordering to reduce bandwidth• Traveling Salesman Problem-based ordering to create
blocks– Nodes = columns of A– Weights(u, v) = no. of nz u, v have in common– Tour = ordering of columns– Choose maximum weight tour– See [Pinar & Heath ’97]
• 2.1x speedup on Power 4, but SPMV not dominant
![Page 54: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/54.jpg)
Source: Accelerator Cavity Design Problem (Ko via Husbands)
![Page 55: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/55.jpg)
100x100 Submatrix Along Diagonal
![Page 56: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/56.jpg)
Post-RCM Reordering
![Page 57: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/57.jpg)
Before: Green + RedAfter: Green + Blue
“Microscopic” Effect of RCM Reordering
![Page 58: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/58.jpg)
“Microscopic” Effect of Combined RCM+TSP Reordering
Before: Green + RedAfter: Green + Blue
![Page 59: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/59.jpg)
(Omega3P)
![Page 60: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/60.jpg)
Optimized Sparse Kernel Interface - OSKI
• Provides sparse kernels automatically tuned for user’s matrix & machine– BLAS-style functionality: SpMV, Ax & ATy, TrSV– Hides complexity of run-time tuning– Includes new, faster locality-aware kernels: ATAx, Akx
• Faster than standard implementations– Up to 4x faster matvec, 1.8x trisolve, 4x ATA*x
• For “advanced” users & solver library writers– Available as stand-alone library (end of Oct ’04)– Available as PETSc extension (Dec ’04)
![Page 61: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/61.jpg)
How the OSKI Tunes (Overview)
Benchmarkdata
1. Build forTargetArch.
2. Benchmark
Heuristicmodels
1. EvaluateModels
Generatedcode
variants
2. SelectData Struct.
& Code
Library Install-Time (offline) Application Run-Time
To user:Matrix handlefor kernelcalls
Workloadfrom program
monitoring
Extensibility: Advanced users may write & dynamically add “Code variants” and “Heuristic models” to system.
HistoryMatrix
![Page 62: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/62.jpg)
How the OSKI Tunes (Overview)
• At library build/install-time– Pre-generate and compile code variants into dynamic
libraries– Collect benchmark data
• Measures and records speed of possible sparse data structure and code variants on target architecture
– Installation process uses standard, portable GNU AutoTools• At run-time
– Library “tunes” using heuristic models• Models analyze user’s matrix & benchmark data to choose
optimized data structure and code– Non-trivial tuning cost: up to ~40 mat-vecs
• Library limits the time it spends tuning based on estimated workload
– provided by user or inferred by library• User may reduce cost by saving tuning results for application
on future runs with same or similar matrix
![Page 63: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/63.jpg)
Optimizations in the Initial OSKI Release
• Fully automatic heuristics for– Sparse matrix-vector multiply
• Register-level blocking• Register-level blocking + symmetry + multiple vectors• Cache-level blocking
– Sparse triangular solve with register-level blocking and “switch-to-dense” optimization
– Sparse ATA*x with register-level blocking• User may select other optimizations manually
– Diagonal storage optimizations, reordering, splitting; tiled matrix powers kernel (Ak*x)
– All available in dynamic libraries– Accessible via high-level embedded script language
• “Plug-in” extensibility– Very advanced users may write their own heuristics, create new
data structures/code variants and dynamically add them to the system
![Page 64: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/64.jpg)
Statistical Models for Automatic Tuning
• Idea 1: Statistical criterion for stopping a search– A general search model
• Generate implementation• Measure performance• Repeat
– Stop when probability of being within of optimal falls below threshold
• Can estimate distribution on-line
• Idea 2: Statistical performance models– Problem: Choose 1 among m implementations at run-
time– Sample performance off-line, build statistical model
![Page 65: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/65.jpg)
Example: Select a Matmul Implementation
![Page 66: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/66.jpg)
Example: Support Vector Classification
![Page 67: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/67.jpg)
A Sparse Matrix You Encounter Every Day
Who am I?
I am aBig Repository
Of usefulAnd uselessFacts alike.
Who am I?
(Hint: Not your e-mail inbox.)
![Page 68: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/68.jpg)
What about the Google Matrix?
• Google approach– Approx. once a month: rank all pages using connectivity
structure• Find dominant eigenvector of a matrix
– At query-time: return list of pages ordered by rank• Matrix: A = G + (1-)(1/n)uuT
– Markov model: Surfer follows link with probability , jumps to a random page with probability 1-
– G is n x n connectivity matrix [n billions]• gij is non-zero if page i links to page j• Normalized so each column sums to 1• Very sparse: about 7—8 non-zeros per row (power law dist.)
– u is a vector of all 1 values– Steady-state probability xi of landing on page i is solution to x
= Ax
• Approximate x by power method: x = Akx0
– In practice, k 25
![Page 69: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/69.jpg)
Extra Slides
![Page 70: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/70.jpg)
Current Work
• Public software release• Impact on library designs: Sparse BLAS, Trilinos, PETSc,
…• Integration in large-scale applications
– DOE: Accelerator design; plasma physics– Geophysical simulation based on Block Lanczos (ATA*X; LBL)
• Systematic heuristics for data structure selection?• Evaluation of emerging architectures
– Revisiting vector micros
• Other sparse kernels– Matrix triple products, Ak*x
• Parallelism• Sparse benchmarks (with UTK) [Gahvari & Hoemmen]• Automatic tuning of MPI collective ops [Nishtala, et al.]
![Page 71: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/71.jpg)
Summary of High-Level Themes
• “Kernel-centric” optimization– Vs. basic block, trace, path optimization, for instance– Aggressive use of domain-specific knowledge
• Performance bounds modeling– Evaluating software quality– Architectural characterizations and consequences
• Empirical search– Hybrid on-line/run-time models
• Statistical performance models– Exploit information from sampling, measuring
![Page 72: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/72.jpg)
Related Work
• My bibliography: 337 entries so far• Sample area 1: Code generation
– Generative & generic programming– Sparse compilers– Domain-specific generators
• Sample area 2: Empirical search-based tuning– Kernel-centric
• linear algebra, signal processing, sorting, MPI, …
– Compiler-centric• profiling + FDO, iterative compilation, superoptimizers,
self-tuning compilers, continuous program optimization
![Page 73: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/73.jpg)
Future Directions (A Bag of Flaky Ideas)
• Composable code generators and search spaces• New application domains
– PageRank: multilevel block algorithms for topic-sensitive search?
• New kernels: cryptokernels– rich mathematical structure germane to performance; lots
of hardware
• New tuning environments– Parallel, Grid, “whole systems”
• Statistical models of application performance– Statistical learning of concise parametric models from
traces for architectural evaluation– Compiler/automatic derivation of parametric models
![Page 74: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/74.jpg)
Possible Future Work
• Different Architectures– New FP instruction sets (SSE2)– SMP / multicore platforms– Vector architectures
• Different Kernels• Higher Level Algorithms
– Parallel versions of kenels, with optimized communication– Block algorithms (eg Lanczos)– XBLAS (extra precision)
• Produce Benchmarks– Augment HPCC Benchmark
• Make it possible to combine optimizations easily for any kernel
• Related tuning activities (LAPACK & ScaLAPACK)
![Page 75: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/75.jpg)
Review of Tuning by Illustration
(Extra Slides)
![Page 76: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/76.jpg)
Splitting for Variable Blocks and Diagonals
• Decompose A = A1 + A2 + … At
– Detect “canonical” structures (sampling)– Split
– Tune each Ai
– Improve performance and save storage
• New data structures– Unaligned block CSR
• Relax alignment in rows & columns
– Row-segmented diagonals
![Page 77: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/77.jpg)
Example: Variable Block Row (Matrix #12)
2.1x over CSR1.8x over RB
![Page 78: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/78.jpg)
Example: Row-Segmented Diagonals
2x over CSR
![Page 79: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/79.jpg)
Mixed Diagonal and Block Structure
![Page 80: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/80.jpg)
Summary
• Automated block size selection– Empirical modeling and search– Register blocking for SpMV, triangular solve, ATA*x
• Not fully automated– Given a matrix, select splittings and transformations
• Lots of combinatorial problems– TSP reordering to create dense blocks (Pinar ’97;
Moon, et al. ’04)
![Page 81: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/81.jpg)
Extra Slides
![Page 82: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/82.jpg)
A Sparse Matrix You Encounter Every Day
Who am I?
I am aBig Repository
Of usefulAnd uselessFacts alike.
Who am I?
(Hint: Not your e-mail inbox.)
![Page 83: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/83.jpg)
Problem Context
• Sparse kernels abound– Models of buildings, cars, bridges, economies, …– Google PageRank algorithm
• Historical trends– Sparse matrix-vector multiply (SpMV): 10% of peak– 2x faster with “hand-tuning”– Tuning becoming more difficult over time– Promise of automatic tuning: PHiPAC/ATLAS, FFTW, …
• Challenges to high-performance– Not dense linear algebra!
• Complex data structures: indirect, irregular memory access
• Performance depends strongly on run-time inputs
![Page 84: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/84.jpg)
Key Questions, Ideas, Conclusions
• How to tune basic sparse kernels automatically?– Empirical modeling and search
• Up to 4x speedups for SpMV• 1.8x for triangular solve• 4x for ATA*x; 2x for A2*x• 7x for multiple vectors
• What are the fundamental limits on performance?– Kernel-, machine-, and matrix-specific upper bounds
• Achieve 75% or more for SpMV, limiting low-level tuning• Consequences for architecture?
• General techniques for empirical search-based tuning?– Statistical models of performance
![Page 85: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/85.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques• Upper bounds on performance• Statistical models of performance
![Page 86: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/86.jpg)
Matrix-vector multiply kernel: y(i) y(i) + A(i,j)*x(j)Matrix-vector multiply kernel: y(i) y(i) + A(i,j)*x(j)
for each row i
for k=ptr[i] to ptr[i+1] do
y[i] = y[i] + val[k]*x[ind[k]]
Compressed Sparse Row (CSR) Storage
Matrix-vector multiply kernel: y(i) y(i) + A(i,j)*x(j)
for each row i
for k=ptr[i] to ptr[i+1] do
y[i] = y[i] + val[k]*x[ind[k]]
![Page 87: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/87.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques• Upper bounds on performance• Statistical models of performance
![Page 88: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/88.jpg)
Historical Trends in SpMV Performance
• The Data– Uniprocessor SpMV performance since 1987– “Untuned” and “Tuned” implementations– Cache-based superscalar micros; some vectors– LINPACK
![Page 89: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/89.jpg)
SpMV Historical Trends: Mflop/s
![Page 90: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/90.jpg)
Example: The Difficulty of Tuning
• n = 21216• nnz = 1.5 M• kernel: SpMV
• Source: NASA structural analysis problem
![Page 91: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/91.jpg)
Still More Surprises
• More complicated non-zero structure in general
![Page 92: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/92.jpg)
Still More Surprises
• More complicated non-zero structure in general
• Example: 3x3 blocking– Logical grid of 3x3 cells
![Page 93: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/93.jpg)
Historical Trends: Mixed News
• Observations– Good news: Moore’s law like behavior– Bad news: “Untuned” is 10% peak or less,
worsening– Good news: “Tuned” roughly 2x better today, and
improving– Bad news: Tuning is complex
– (Not really news: SpMV is not LINPACK)
• Questions– Application: Automatic tuning?– Architect: What machines are good for SpMV?
![Page 94: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/94.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques
– SpMV [SC’02; IJHPCA ’04b]– Sparse triangular solve (SpTS) [ICS/POHLL ’02]– ATA*x [ICCS/WoPLA ’03]
• Upper bounds on performance• Statistical models of performance
![Page 95: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/95.jpg)
SPARSITY: Framework for Tuning SpMV
• SPARSITY: Automatic tuning for SpMV [Im & Yelick ’99]– General approach
• Identify and generate implementation space• Search space using empirical models & experiments
– Prototype library and heuristic for choosing register block size
• Also: cache-level blocking, multiple vectors
• What’s new?– New block size selection heuristic
• Within 10% of optimal — replaces previous version
– Expanded implementation space• Variable block splitting, diagonals, combinations
– New kernels: sparse triangular solve, ATA*x, A*x
![Page 96: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/96.jpg)
Automatic Register Block Size Selection
• Selecting the r x c block size– Off-line benchmark: characterize the machine
• Precompute Mflops(r,c) using dense matrix for each r x c• Once per machine/architecture
– Run-time “search”: characterize the matrix• Sample A to estimate Fill(r,c) for each r x c
– Run-time heuristic model• Choose r, c to maximize Mflops(r,c) / Fill(r,c)
• Run-time costs– Up to ~40 SpMVs (empirical worst case)
![Page 97: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/97.jpg)
Accuracy of the Tuning Heuristics (1/4)
NOTE: “Fair” flops used (ops on explicit zeros not counted as “work”)
DGEMV
![Page 98: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/98.jpg)
Accuracy of the Tuning Heuristics (2/4)DGEMV
![Page 99: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/99.jpg)
Accuracy of the Tuning Heuristics (3/4)DGEMV
![Page 100: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/100.jpg)
Accuracy of the Tuning Heuristics (4/4)DGEMV
![Page 101: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/101.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques• Upper bounds on performance
– SC’02
• Statistical models of performance
![Page 102: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/102.jpg)
Motivation for Upper Bounds Model
• Questions– Speedups are good, but what is the speed limit?
• Independent of instruction scheduling, selection
– What machines are “good” for SpMV?
![Page 103: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/103.jpg)
Upper Bounds on Performance: Blocked SpMV
• P = (flops) / (time)– Flops = 2 * nnz(A)
• Lower bound on time: Two main assumptions– 1. Count memory ops only (streaming)– 2. Count only compulsory, capacity misses: ignore conflicts
• Account for line sizes• Account for matrix size and nnz
• Charge min access “latency” i at Li cache & mem
– e.g., Saavedra-Barrera and PMaC MAPS benchmarks
1mem11
1memmem
Misses)(Misses)(Loads
HitsHitsTime
iiii
iii
![Page 104: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/104.jpg)
Example: Bounds on Itanium 2
![Page 105: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/105.jpg)
Example: Bounds on Itanium 2
![Page 106: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/106.jpg)
Example: Bounds on Itanium 2
![Page 107: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/107.jpg)
Fraction of Upper Bound Across Platforms
![Page 108: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/108.jpg)
Achieved Performance and Machine Balance
• Machine balance [Callahan ’88; McCalpin ’95]– Balance = Peak Flop Rate / Bandwidth (flops /
double)
• Ideal balance for mat-vec: 2 flops / double– For SpMV, even less
• SpMV ~ streaming– 1 / (avg load time to stream 1 array) ~ (bandwidth)– “Sustained” balance = peak flops / model bandwidth
i
iii Misses)(Misses)(LoadsTime mem11
![Page 109: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/109.jpg)
![Page 110: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/110.jpg)
Where Does the Time Go?
• Most time assigned to memory• Caches “disappear” when line sizes are equal
– Strictly increasing line sizes
1
memmem HitsHitsTimei
ii
![Page 111: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/111.jpg)
Execution Time Breakdown: Matrix 40
![Page 112: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/112.jpg)
Speedups with Increasing Line Size
![Page 113: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/113.jpg)
Summary: Performance Upper Bounds
• What is the best we can do for SpMV?– Limits to low-level tuning of blocked implementations– Refinements?
• What machines are good for SpMV?– Partial answer: balance characterization
• Architectural consequences?– Example: Strictly increasing line sizes
![Page 114: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/114.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques• Upper bounds on performance• Tuning other sparse kernels• Statistical models of performance
– FDO ’00; IJHPCA ’04a
![Page 115: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/115.jpg)
Statistical Models for Automatic Tuning
• Idea 1: Statistical criterion for stopping a search– A general search model
• Generate implementation• Measure performance• Repeat
– Stop when probability of being within of optimal falls below threshold
• Can estimate distribution on-line
• Idea 2: Statistical performance models– Problem: Choose 1 among m implementations at run-
time– Sample performance off-line, build statistical model
![Page 116: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/116.jpg)
Example: Select a Matmul Implementation
![Page 117: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/117.jpg)
Example: Support Vector Classification
![Page 118: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/118.jpg)
Road Map
• Sparse matrix-vector multiply (SpMV) in a nutshell• Historical trends and the need for search• Automatic tuning techniques• Upper bounds on performance• Tuning other sparse kernels• Statistical models of performance• Summary and Future Work
![Page 119: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/119.jpg)
Summary of High-Level Themes
• “Kernel-centric” optimization– Vs. basic block, trace, path optimization, for instance– Aggressive use of domain-specific knowledge
• Performance bounds modeling– Evaluating software quality– Architectural characterizations and consequences
• Empirical search– Hybrid on-line/run-time models
• Statistical performance models– Exploit information from sampling, measuring
![Page 120: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/120.jpg)
Related Work
• My bibliography: 337 entries so far• Sample area 1: Code generation
– Generative & generic programming– Sparse compilers– Domain-specific generators
• Sample area 2: Empirical search-based tuning– Kernel-centric
• linear algebra, signal processing, sorting, MPI, …
– Compiler-centric• profiling + FDO, iterative compilation, superoptimizers,
self-tuning compilers, continuous program optimization
![Page 121: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/121.jpg)
Future Directions (A Bag of Flaky Ideas)
• Composable code generators and search spaces• New application domains
– PageRank: multilevel block algorithms for topic-sensitive search?
• New kernels: cryptokernels– rich mathematical structure germane to performance; lots
of hardware
• New tuning environments– Parallel, Grid, “whole systems”
• Statistical models of application performance– Statistical learning of concise parametric models from
traces for architectural evaluation– Compiler/automatic derivation of parametric models
![Page 122: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/122.jpg)
Acknowledgements
• Super-advisors: Jim and Kathy• Undergraduate R.A.s: Attila, Ben, Jen, Jin,
Michael, Rajesh, Shoaib, Sriram, Tuyet-Linh• See pages xvi—xvii of dissertation.
![Page 123: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/123.jpg)
TSP-based Reordering: Before
(Pinar ’97;Moon, et al ‘04)
![Page 124: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/124.jpg)
TSP-based Reordering: After
(Pinar ’97;Moon, et al ‘04)
Up to 2xspeedupsover CSR
![Page 125: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/125.jpg)
Example: L2 Misses on Itanium 2
Misses measured using PAPI [Browne ’00]
![Page 126: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/126.jpg)
Example: Distribution of Blocked Non-Zeros
![Page 127: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/127.jpg)
Register Profile: Itanium 2
190 Mflop/s
1190 Mflop/s
![Page 128: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/128.jpg)
Register Profiles: Sun and Intel x86
Ultra 2i - 11% Ultra 3 - 5%
Pentium III-M - 15%Pentium III - 21%
72 Mflop/s
35 Mflop/s
90 Mflop/s
50 Mflop/s
108 Mflop/s
42 Mflop/s
122 Mflop/s
58 Mflop/s
![Page 129: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/129.jpg)
Register Profiles: IBM and Intel IA-64
Power3 - 17% Power4 - 16%
Itanium 2 - 33%Itanium 1 - 8%
252 Mflop/s
122 Mflop/s
820 Mflop/s
459 Mflop/s
247 Mflop/s
107 Mflop/s
1.2 Gflop/s
190 Mflop/s
![Page 130: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/130.jpg)
Accurate and Efficient Adaptive Fill Estimation
• Idea: Sample matrix– Fraction of matrix to sample: s [0,1]– Cost ~ O(s * nnz)– Control cost by controlling s
• Search at run-time: the constant matters!
• Control s automatically by computing statistical confidence intervals– Idea: Monitor variance
• Cost of tuning– Lower bound: convert matrix in 5 to 40 unblocked
SpMVs– Heuristic: 1 to 11 SpMVs
![Page 131: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/131.jpg)
Sparse/Dense Partitioning for SpTS
• Partition L into sparse (L1,L2) and dense LD:
2
1
2
1
2
1
b
b
x
x
LL
L
D
• Perform SpTS in three steps:
22
1222
111
ˆ)3(
ˆ)2(
)1(
bxL
xLbb
bxL
D
• Sparsity optimizations for (1)—(2); DTRSV for (3)• Tuning parameters: block size, size of dense triangle
![Page 132: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/132.jpg)
SpTS Performance: Power3
![Page 133: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/133.jpg)
![Page 134: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/134.jpg)
Summary of SpTS and AAT*x Results
• SpTS — Similar to SpMV– 1.8x speedups; limited benefit from low-level tuning
• AATx, ATAx– Cache interleaving only: up to 1.6x speedups– Reg + cache: up to 4x speedups
• 1.8x speedup over register only
– Similar heuristic; same accuracy (~ 10% optimal)– Further from upper bounds: 60—80%
• Opportunity for better low-level tuning a la PHiPAC/ATLAS
• Matrix triple products? Ak*x?– Preliminary work
![Page 135: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/135.jpg)
Register Blocking: Speedup
![Page 136: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/136.jpg)
Register Blocking: Performance
![Page 137: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/137.jpg)
Register Blocking: Fraction of Peak
![Page 138: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/138.jpg)
Example: Confidence Interval Estimation
![Page 139: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/139.jpg)
Costs of Tuning
![Page 140: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/140.jpg)
Splitting + UBCSR: Pentium III
![Page 141: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/141.jpg)
Splitting + UBCSR: Power4
![Page 142: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/142.jpg)
Splitting+UBCSR Storage: Power4
![Page 143: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/143.jpg)
![Page 144: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/144.jpg)
Example: Variable Block Row (Matrix #13)
![Page 145: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/145.jpg)
Dense Tuning is Hard, Too
• Even dense matrix multiply can be notoriously difficult to tune
![Page 146: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/146.jpg)
Dense matrix multiply: surprising performance as register tile size varies.
![Page 147: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/147.jpg)
![Page 148: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/148.jpg)
Preliminary Results (Matrix Set 2): Itanium 2
Web/IR
Dense FEM FEM (var) Bio LPEcon Stat
![Page 149: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/149.jpg)
Multiple Vector Performance
![Page 150: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/150.jpg)
![Page 151: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/151.jpg)
What about the Google Matrix?
• Google approach– Approx. once a month: rank all pages using connectivity
structure• Find dominant eigenvector of a matrix
– At query-time: return list of pages ordered by rank• Matrix: A = G + (1-)(1/n)uuT
– Markov model: Surfer follows link with probability , jumps to a random page with probability 1-
– G is n x n connectivity matrix [n 3 billion]• gij is non-zero if page i links to page j• Normalized so each column sums to 1• Very sparse: about 7—8 non-zeros per row (power law dist.)
– u is a vector of all 1 values– Steady-state probability xi of landing on page i is solution to x
= Ax
• Approximate x by power method: x = Akx0
– In practice, k 25
![Page 152: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/152.jpg)
![Page 153: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/153.jpg)
MAPS Benchmark Example: Power4
![Page 154: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/154.jpg)
MAPS Benchmark Example: Itanium 2
![Page 155: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/155.jpg)
Saavedra-Barrera Example: Ultra 2i
![Page 156: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/156.jpg)
![Page 157: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/157.jpg)
Summary of Results: Pentium III
![Page 158: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/158.jpg)
Summary of Results: Pentium III (3/3)
![Page 159: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/159.jpg)
Execution Time Breakdown (PAPI): Matrix 40
![Page 160: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/160.jpg)
Preliminary Results (Matrix Set 1): Itanium 2
LPFEM FEM (var) AssortedDense
![Page 161: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/161.jpg)
Tuning Sparse Triangular Solve (SpTS)
• Compute x=L-1*b where L sparse lower triangular, x & b dense
• L from sparse LU has rich dense substructure– Dense trailing triangle can account for 20—90% of
matrix non-zeros
• SpTS optimizations– Split into sparse trapezoid and dense trailing triangle– Use tuned dense BLAS (DTRSV) on dense triangle– Use Sparsity register blocking on sparse part
• Tuning parameters– Size of dense trailing triangle– Register block size
![Page 162: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/162.jpg)
Sparse Kernels and Optimizations
• Kernels– Sparse matrix-vector multiply (SpMV): y=A*x– Sparse triangular solve (SpTS): x=T-1*b– y=AAT*x, y=ATA*x– Powers (y=Ak*x), sparse triple-product (R*A*RT), …
• Optimization techniques (implementation space)– Register blocking– Cache blocking– Multiple dense vectors (x)– A has special structure (e.g., symmetric, banded, …)– Hybrid data structures (e.g., splitting, switch-to-
dense, …)– Matrix reordering
• How and when do we search?– Off-line: Benchmark implementations– Run-time: Estimate matrix properties, evaluate
performance models based on benchmark data
![Page 163: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/163.jpg)
Cache Blocked SpMV on LSI Matrix: Ultra 2i
A10k x 255k3.7M non-zeros
Baseline:16 Mflop/s
Best block size& performance:16k x 64k28 Mflop/s
![Page 164: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/164.jpg)
Cache Blocking on LSI Matrix: Pentium 4
A10k x 255k3.7M non-zeros
Baseline:44 Mflop/s
Best block size& performance:16k x 16k210 Mflop/s
![Page 165: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/165.jpg)
Cache Blocked SpMV on LSI Matrix: Itanium
A10k x 255k3.7M non-zeros
Baseline:25 Mflop/s
Best block size& performance:16k x 32k72 Mflop/s
![Page 166: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/166.jpg)
Cache Blocked SpMV on LSI Matrix: Itanium 2
A10k x 255k3.7M non-zeros
Baseline:170 Mflop/s
Best block size& performance:16k x 65k275 Mflop/s
![Page 167: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/167.jpg)
Inter-Iteration Sparse Tiling (1/3)
• [Strout, et al., ‘01]• Let A be 6x6 tridiagonal• Consider y=A2x
– t=Ax, y=At
• Nodes: vector elements• Edges: matrix elements
aij
y1
y2
y3
y4
y5
t1
t2
t3
t4
t5
x1
x2
x3
x4
x5
![Page 168: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/168.jpg)
Inter-Iteration Sparse Tiling (2/3)
• [Strout, et al., ‘01]• Let A be 6x6 tridiagonal• Consider y=A2x
– t=Ax, y=At
• Nodes: vector elements• Edges: matrix elements
aij
• Orange = everything needed to compute y1
– Reuse a11, a12
y1
y2
y3
y4
y5
t1
t2
t3
t4
t5
x1
x2
x3
x4
x5
![Page 169: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/169.jpg)
Inter-Iteration Sparse Tiling (3/3)
• [Strout, et al., ‘01]• Let A be 6x6 tridiagonal• Consider y=A2x
– t=Ax, y=At
• Nodes: vector elements• Edges: matrix elements
aij
• Orange = everything needed to compute y1
– Reuse a11, a12
• Grey = y2, y3
– Reuse a23, a33, a43
y1
y2
y3
y4
y5
t1
t2
t3
t4
t5
x1
x2
x3
x4
x5
![Page 170: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/170.jpg)
Inter-Iteration Sparse Tiling: Issues
• Tile sizes (colored regions) grow with no. of iterations and increasing out-degree– G likely to have a few
nodes with high out-degree (e.g., Yahoo)
• Mathematical tricks to limit tile size?– Judicious dropping of
edges [Ng’01]
y1
y2
y3
y4
y5
t1
t2
t3
t4
t5
x1
x2
x3
x4
x5
![Page 171: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/171.jpg)
Summary and Questions
• Need to understand matrix structure and machine– BeBOP: suite of techniques to deal with different sparse
structures and architectures• Google matrix problem
– Established techniques within an iteration– Ideas for inter-iteration optimizations– Mathematical structure of problem may help
• Questions– Structure of G?– What are the computational bottlenecks?– Enabling future computations?
• E.g., topic-sensitive PageRank multiple vector version [Haveliwala ’02]
– See www.cs.berkeley.edu/~richie/bebop/intel/google for more info, including more complete Itanium 2 results.
![Page 172: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/172.jpg)
Exploiting Matrix Structure
• Symmetry (numerical or structural)– Reuse matrix entries– Can combine with register blocking, multiple vectors,
…
• Matrix splitting– Split the matrix, e.g., into r x c and 1 x 1– No fill overhead
• Large matrices with random structure– E.g., Latent Semantic Indexing (LSI) matrices– Technique: cache blocking
• Store matrix as 2i x 2j sparse submatrices• Effective when x vector is large• Currently, search to find fastest size
![Page 173: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/173.jpg)
Symmetric SpMV Performance: Pentium 4
![Page 174: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/174.jpg)
SpMV with Split Matrices: Ultra 2i
![Page 175: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/175.jpg)
Cache Blocking on Random Matrices: Itanium
Speedup on four bandedrandom matrices.
![Page 176: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/176.jpg)
Sparse Kernels and Optimizations
• Kernels– Sparse matrix-vector multiply (SpMV): y=A*x– Sparse triangular solve (SpTS): x=T-1*b– y=AAT*x, y=ATA*x– Powers (y=Ak*x), sparse triple-product (R*A*RT), …
• Optimization techniques (implementation space)– Register blocking– Cache blocking– Multiple dense vectors (x)– A has special structure (e.g., symmetric, banded, …)– Hybrid data structures (e.g., splitting, switch-to-dense, …)– Matrix reordering
• How and when do we search?– Off-line: Benchmark implementations– Run-time: Estimate matrix properties, evaluate
performance models based on benchmark data
![Page 177: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/177.jpg)
Register Blocked SpMV: Pentium III
![Page 178: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/178.jpg)
Register Blocked SpMV: Ultra 2i
![Page 179: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/179.jpg)
Register Blocked SpMV: Power3
![Page 180: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/180.jpg)
Register Blocked SpMV: Itanium
![Page 181: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/181.jpg)
Possible Optimization Techniques
• Within an iteration, i.e., computing (G+uuT)*x once– Cache block G*x
• On linear programming matrices and matrices with random structure (e.g., LSI), 1.5—4x speedups
• Best block size is matrix and machine dependent
– Reordering and/or splitting of G to separate dense structure (rows, columns, blocks)
• Between iterations, e.g., (G+uuT)2x– (G+uuT)2x = G2x + (Gu)uTx + u(uTG)x + u(uTu)uTx
• Compute Gu, uTG, uTu once for all iterations• G2x: Inter-iteration tiling to read G only once
![Page 182: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/182.jpg)
Multiple Vector Performance: Itanium
![Page 183: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/183.jpg)
Sparse Kernels and Optimizations
• Kernels– Sparse matrix-vector multiply (SpMV): y=A*x– Sparse triangular solve (SpTS): x=T-1*b– y=AAT*x, y=ATA*x– Powers (y=Ak*x), sparse triple-product (R*A*RT), …
• Optimization techniques (implementation space)– Register blocking– Cache blocking– Multiple dense vectors (x)– A has special structure (e.g., symmetric, banded, …)– Hybrid data structures (e.g., splitting, switch-to-
dense, …)– Matrix reordering
• How and when do we search?– Off-line: Benchmark implementations– Run-time: Estimate matrix properties, evaluate
performance models based on benchmark data
![Page 184: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/184.jpg)
SpTS Performance: Itanium
(See POHLL ’02 workshop paper, at ICS ’02.)
![Page 185: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/185.jpg)
Sparse Kernels and Optimizations
• Kernels– Sparse matrix-vector multiply (SpMV): y=A*x– Sparse triangular solve (SpTS): x=T-1*b– y=AAT*x, y=ATA*x– Powers (y=Ak*x), sparse triple-product (R*A*RT), …
• Optimization techniques (implementation space)– Register blocking– Cache blocking– Multiple dense vectors (x)– A has special structure (e.g., symmetric, banded, …)– Hybrid data structures (e.g., splitting, switch-to-dense, …)– Matrix reordering
• How and when do we search?– Off-line: Benchmark implementations– Run-time: Estimate matrix properties, evaluate
performance models based on benchmark data
![Page 186: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/186.jpg)
Optimizing AAT*x
• Kernel: y=AAT*x, where A is sparse, x & y dense– Arises in linear programming, computation of SVD– Conventional implementation: compute z=AT*x, y=A*z
• Elements of A can be reused:
n
k
Tkk
Tn
T
n xaax
a
a
aay1
1
1 )(
• When ak represent blocks of columns, can apply register blocking.
![Page 187: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/187.jpg)
Optimized AAT*x Performance: Pentium III
![Page 188: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/188.jpg)
Current Directions
• Applying new optimizations– Other split data structures (variable block, diagonal,
…)– Matrix reordering to create block structure– Structural symmetry
• New kernels (triple product RART, powers Ak, …)• Tuning parameter selection• Building an automatically tuned sparse matrix
library– Extending the Sparse BLAS– Leverage existing sparse compilers as code
generation infrastructure– More thoughts on this topic tomorrow
![Page 189: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/189.jpg)
Related Work
• Automatic performance tuning systems– PHiPAC [Bilmes, et al., ’97], ATLAS [Whaley & Dongarra
’98]– FFTW [Frigo & Johnson ’98], SPIRAL [Pueschel, et al.,
’00], UHFFT [Mirkovic and Johnsson ’00]– MPI collective operations [Vadhiyar & Dongarra ’01]
• Code generation– FLAME [Gunnels & van de Geijn, ’01]– Sparse compilers: [Bik ’99], Bernoulli [Pingali, et al., ’97]– Generic programming: Blitz++ [Veldhuizen ’98], MTL
[Siek & Lumsdaine ’98], GMCL [Czarnecki, et al. ’98], …
• Sparse performance modeling– [Temam & Jalby ’92], [White & Saddayappan ’97],
[Navarro, et al., ’96], [Heras, et al., ’99], [Fraguela, et al., ’99], …
![Page 190: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/190.jpg)
More Related Work
• Compiler analysis, models– CROPS [Carter, Ferrante, et al.]; Serial sparse tiling
[Strout ’01]– TUNE [Chatterjee, et al.]– Iterative compilation [O’Boyle, et al., ’98]– Broadway compiler [Guyer & Lin, ’99]– [Brewer ’95], ADAPT [Voss ’00]
• Sparse BLAS interfaces– BLAST Forum (Chapter 3)– NIST Sparse BLAS [Remington & Pozo ’94];
SparseLib++– SPARSKIT [Saad ’94]– Parallel Sparse BLAS [Fillipone, et al. ’96]
![Page 191: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/191.jpg)
Context: Creating High-Performance Libraries
• Application performance dominated by a few computational kernels
• Today: Kernels hand-tuned by vendor or user• Performance tuning challenges
– Performance is a complicated function of kernel, architecture, compiler, and workload
– Tedious and time-consuming
• Successful automated approaches– Dense linear algebra: ATLAS/PHiPAC– Signal processing: FFTW/SPIRAL/UHFFT
![Page 192: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/192.jpg)
Cache Blocked SpMV on LSI Matrix: Itanium
![Page 193: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/193.jpg)
Sustainable Memory Bandwidth
![Page 194: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/194.jpg)
Multiple Vector Performance: Pentium 4
![Page 195: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/195.jpg)
Multiple Vector Performance: Itanium
![Page 196: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/196.jpg)
Multiple Vector Performance: Pentium 4
![Page 197: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/197.jpg)
Optimized AAT*x Performance: Ultra 2i
![Page 198: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/198.jpg)
Optimized AAT*x Performance: Pentium 4
![Page 199: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/199.jpg)
Tuning Pays Off—PHiPAC
![Page 200: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/200.jpg)
Tuning pays off – ATLAS
Extends applicability of PHIPAC; Incorporated in Matlab (with rest of LAPACK)
![Page 201: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/201.jpg)
Register Tile Sizes (Dense Matrix Multiply)
333 MHz Sun Ultra 2i
2-D slice of 3-D space; implementations color-coded by performance in Mflop/s
16 registers, but 2-by-3 tile size fastest
![Page 202: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/202.jpg)
High Precision GEMV (XBLAS)
![Page 203: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/203.jpg)
High Precision Algorithms (XBLAS)• Double-double (High precision word represented as pair of doubles)
– Many variations on these algorithms; we currently use Bailey’s• Exploiting Extra-wide Registers
– Suppose s(1) , … , s(n) have f-bit fractions, SUM has F>f bit fraction– Consider following algorithm for S = i=1,n s(i)
• Sort so that |s(1)| |s(2)| … |s(n)|• SUM = 0, for i = 1 to n SUM = SUM + s(i), end for, sum = SUM
– Theorem (D., Hida) Suppose F<2f (less than double precision)• If n 2F-f + 1, then error 1.5 ulps• If n = 2F-f + 2, then error 22f-F ulps (can be 1)• If n 2F-f + 3, then error can be arbitrary (S 0 but sum = 0 )
– Examples• s(i) double (f=53), SUM double extended (F=64)
– accurate if n 211 + 1 = 2049• Dot product of single precision x(i) and y(i)
– s(i) = x(i)*y(i) (f=2*24=48), SUM double extended (F=64) – accurate if n 216 + 1 = 65537
![Page 204: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/204.jpg)
More Extra Slides
![Page 205: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/205.jpg)
Opteron (1.4GHz, 2.8GFlop peak)
Beat ATLAS DGEMV’s 365 Mflops
Nonsymmetric peak = 504 MFlops Symmetric peak = 612 MFlops
![Page 206: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/206.jpg)
Awards• Best Paper, Intern. Conf. Parallel Processing, 2004
– “Performance models for evaluation and automatic performance tuning of symmetric sparse matrix-vector multiply”
• Best Student Paper, Intern. Conf. Supercomputing, Workshop on Performance Optimization via High-Level Languages and Libraries, 2003– Best Student Presentation too, to Richard Vuduc– “Automatic performance tuning and analysis of sparse triangular solve”
• Finalist, Best Student Paper, Supercomputing 2002– To Richard Vuduc– “Performance Optimization and Bounds for Sparse Matrix-vector Multiply”
• Best Presentation Prize, MICRO-33: 3rd ACM Workshop on Feedback-Directed Dynamic Optimization, 2000– To Richard Vuduc– “Statistical Modeling of Feedback Data in an Automatic Tuning System”
![Page 207: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/207.jpg)
Accuracy of the Tuning Heuristics (4/4)
![Page 208: Automatic Performance Tuning Sparse Matrix Kernels James Demmel demmel/cs267_Spr05](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5f5503460f94a3f235/html5/thumbnails/208.jpg)
Can Match DGEMV PerformanceDGEMV