algebraic multigrid in lattice qcd computations · the continuum limit scaling problem in qcd...

Algebraic Multigrid in LatticeQCD computations

Karsten Kahl

Bergische Universitat Wuppertal

October 4, 2011

Outline

The Multigrid Principle

Algebraic MultigridTheory of AMG

Algebraic Multigrid for QCDSmoothers for QCDInterpolation for QCD

Bootstrap Algebraic MultigridLeast Squares InterpolationBootstrap Setup

Numerical Experiments

Outlook

Algebraic Multigrid in Lattice QCD computations, Karsten Kahl 2/26

The Multigrid PrincipleAlgebraic Multigrid

Algebraic Multigrid for QCDBootstrap Algebraic Multigrid

Numerical ExperimentsOutlook

The continuum limit scaling problem — Simple Example

Partial differential equation

∆ψ(x) = ϕ, x ∈ (0, 1)2

ψ(x) = 0, x ∈ ∂(0, 1)2

Discretization by finite differences

Au = f, A ∈ RN2×N2

, u, f ∈ RN2

I smallest eigenvalue λmin ∼ h2 = N−2

I largest eigenvalue λmax = const

Continuum limit: condition number κh→0−→∞





Multigrid in a Nutshell

Fewer

First Coarse Grid

Finest Grid

SmoothThe Multigrid V−cycle

Restriction

Prolongation

Dofs





Theory of AMG

Components of Algebraic Multigrid

Given: Linear system of equationsAu = f

Wanted: Hierarchy of systems A0 = AA`u` = f`, ` = 0, . . . , L

Needed: Coarse spaces (coarse variables)Cn0 ⊃ Cn1 ⊃ . . . ⊃ CnL

Smoother

S` : Cn` −→ Cn`

High modes

Interpolation

P ``+1 : Cn`+1 −→ Cn`

Low modes

Complementarity of Smoother and InterpolationAlgebraic Multigrid in Lattice QCD computations, Karsten Kahl 5/26




Theory of AMG

Coarse variables in Algebraic Multigrid

Given: Variables u1, . . . , unWanted: Coarse variables v1, . . . , vnc with nc � n

vi =∑j∈Ni

αijuj

Aggregation

Nl ∩Nk = ∅ for all k, l

C − F splitting

αij = δij =⇒ P =

(WI

)Algebraic Multigrid in Lattice QCD computations, Karsten Kahl 6/26




Theory of AMG

Generic Multigrid Algorithm

MG`(A`, b`)

if ` < L thenu` = 0for i = 1, . . . , ν1 dou` = S`(A`, u`, b`) “Pre-smooothing”

end foru`+1 = MG(A`+1,

(P ``+1

)H(b` −A`u`))

u` = u` + P ``+1u`+1 “Coarse-grid correction”for i = 1, . . . , ν2 dou` = S`(A`, u`, b`) “Post-smoothing”

end forelseu`+1 = A−1

`+1

(P ``+1

)H(b` −A`u`)

end ifAlgebraic Multigrid in Lattice QCD computations, Karsten Kahl 7/26




Theory of AMG

Two-grid convergence strong approximation property

I Strong approximation property

‖(I − PA−1c PHA)e‖2A ≤ K‖Ae‖22

I Smoothing property

‖Se‖2A ≤ ‖e‖2A − α1〈Ae,Ae〉2‖Se‖2A ≤ ‖e‖2A − α2〈ASe,Ae〉2

I Two-grid convergence

‖S(I − PA−1

c PHA)Se‖2A ≤

1− α1K

1 + α2K

‖e‖2A

I Assumptions fulfilled on all levels ⇒ Multigrid convergence





Theory of AMG

Two-grid convergence weak approximation property

I Weak approximation property

‖ (I − PR) e‖2 ≤K

‖A‖2‖e‖A

I Convergent hermitian positive definite smoother S

λmin(I − SA) ≥ 0 and λmin(S) ≥ 1

c2S‖A‖22

I Two-grid convergence

‖S(I − PA−1

c PHA)Se‖2A ≤

(1− 1

c0

)‖e‖A

with c0 = (2 +KcS)2





Smoothers for QCDInterpolation for QCD

The continuum limit scaling problem in QCD

Wilson discretization of the Dirac equation

D =∑µ

γµ ⊗ dcµ + a · dµµ + 1 ·m

Continuum limit scaling problems:

I condition number

κ ∼ (a ·m)−1 m,a→0−→ ∞

I eigenvalues K below threshold (e.g. 100 MeV)

K ∼ V = N3sNt

Scalability in a,m and V needed (possible in optimal complexity?)(Algebraic) Multigrid






Choices for AMG in QCD Computations

Application of AMG to linear system D of Wilson fermions

Du = f,D ∈ CNdnsnc , with D† = Γ5DΓ5,

where D is non-hermitian pos. real and Γ5D hermitian indefinite

Define R`, P`, S` such that

I the Γ5-’ambiguity’ is hidden to the AMG algorithm

I the Γ5-structure is preserved in the multigrid hierarchy

D†c = Γ5DcΓ5






Domain Decomposition smoother S`

Decomposition of L

L2 L3

L1 L4

I easily parallelizable

I implementation available

I local Dirichlet boundaries

I no theory available

Canonical restrictions

Ri : L −→ Li

Block inverses

Bi := RTi (RiDR

Ti )−1Ri

RB Schwarz

for color = 1, 2 dor = b−Dzz = z + (

∑i with Li⊂L|color

Bi)rend for

[1] Hermann Schwarz 1870[2] Martin Luscher 2003






Polynomial smoother S`

Polynomial best-approximationsFor E ⊂ C \ 0 containing high modesfind polynomial q s.t.

minq∈Πn

maxz∈E|z−1 − q(z)|

I minimizer q∗ known in shortrecursion for ellipsoidal E

I easily parallelizable

I theory available

I estimate of ‖A‖ required

0 2 4 6 8−4

−3

−2

−1

0

1

2

3

4

|z−1 − q∗(z)|Algebraic Multigrid in Lattice QCD computations, Karsten Kahl 13/26





Structure-preserving restriction and prolongation

Preserve γ5 symmetry ↔ symmetry of spectrum to real axis

Pγ5 = γ5P

Dc = P †DP=⇒ D†c = γ5Dγ5

For γ5 =

(I−I

)choose prolongation P =

(P1 00 P2

)⇒ γ5-structure transferred to coarse-grid in Dc






0th and 1st order interpolation...

0th order 1st order

I aggregation

I easy to implement

I piece-wise constant

Aggregation interpolation

I C/F splitting

I next-neighbor interpolation

I linear interpolation

C/F interpolation

Bootstrap Algebraic Multigrid






... for Wilson fermions

In 4D with next neighbor coupling

kA

I next neighbor coupling oncoarse grid

I operator complexity

1 +k

|A|+

k2

|A|2. . .

I additional diagonal coupling(preserved on coarser grids)

I operator complexity

1 +81

9

1

16

(1 +

1

16· · ·





Least Squares InterpolationBootstrap Setup

Bootstrap AMG

Bootstrap Algebraic Multigrid Framework

“Continous updating components of the MG hierarchy using practical

tools and measures built from the evolving MG solver”

I Compatible Relaxation and Algebraic DistancesI Adaptive computation of nested subspacesI Quality measurement of grid hierarchies

I Least Squares interpolationI Adaptive definition of interpolationI Quality and complexity control of the multigrid method

I Bootstrap setup techniquesI Multigrid discovery of algebraically smooth errorI Multigrid quality control mechanisms






Least Squares Interpolation

Given: test vectors u(1), . . . , u(k) ∈ Cn representing low modesWanted: interpolation P accurate for test vectors u(s)

Least Squares Interpolation

LCi(pi) =

k∑s=1

ωs(u(s)i −

∑j∈Ci

(pi)ju(s)j )2 → min

pi

I interpolatory points Ci ⊂ C for i ∈ FI in neighborhood of i

I weights ωs ∼ ‖u(s)‖‖Au(s)‖−1 ∈ R+

I test vectors (→ bootstrap)

∈ Fi

pij2

pij4

pij1

pij3

∈ C






Least Squares Interpolation – Choice of Weights ωs

Idea: Weights ωs bias Least Squares fit u(s) to low modes

⇒ Intuitive choice ωs ∼ ‖u(s)‖‖Au(s)‖−1

Problem: Local errors in test vectors weighted by global weight

Remedies:

I For u(s) with large ωs and |(Au(s)

)i|

u(s)i = u

(s)i −

1

aii

(Au(s)

)i

Modified Least Squares Fit

LCi(pi) =k∑s=1

ωs(u(s)i −

1

aii

(Au(s)

)i−∑j∈Ci

(pi)ju(s)j )2 → min

pi






Bootstrap Setup - Multigrid Eigensolver

Assumption: No a priori information on low modes available!

I smoother action known, initial test vectors

u(s) = Sηu(s), u(s) random

I observation (P` = P 01 · · ·P

`−1` , A` = PH` A0P`, T` = PH` P`)

〈v`, v`〉A`〈v`, v`〉T`

=〈P`v`, P`v`〉A〈P`v`, P`v`〉2

Bootstrap Idea

Eigenpairs(v`, λ`) of (A`, T`)

−→Eigenpairs

(P`v`, λ`) of A+ interpolation error






Bootstrap Setup – Cycling Strategies

Relax on Au = 0, u ∈ U and Av = λTv, v ∈ V

Relax on Au = 0, u ∈ U

Compute V , s.t., Av = λTv, v ∈ V

Relax on Av = λTv, v ∈ V





Numerical Experiments — Settings

I GMRESI restart 32I tolerance 10−8

I Bootstrap AMGI V 2-cycle setupI polynomial smoother of degree 4I at most 16 points used in prolongation for each pointI 24(+24) test vectorsI coarsest grid 24 or 34

I PlatformI single processorI development platform: MATLAB





Numerical Experiments

GMRES(32) GMRES(32) CGN+ BootAMG

V 84 124 164 84 124 164 84 124 164

setup (time) 22.3 110 428 — — — — — —solve (time) 5.77 68 222 17.3 532 2359 23.8 407 885#iterations 17 31 33 502 2106 2536 347 944 699

total (time) 28.1 178 650 17.3 532 2536 23.8 363 885

164 κ-study

−2 −1 0 1 2 3 4 5

x 10−3

0

500

1000

1500

2000

2500

3000

3500

4000

#ite

ratio

ns

bare subtracted mass

AMG GMRES(32)

GMRES(32)

κ

κc

84, 124 Lattice

unknown parameters

(λmin) = 10−5

3-level BootAMG

164 Lattice

β = 5.8 (0.05 fm)

κ = .15462 (400 MeV)

κc = .1548

4-level BootAMG





Results: 48× 243

FGMRES+ αaMG(8) CGN

setup timings 45.22s -solve iterations 9 4476

timings 6.97s 121.82stotal timings 52.19s 121.82s

Configuration:

I 48× 243 Clover Wilson-Dirac

I parameters:I pion mass: mπ = 630 MeVI csw = 1.90952

I generated with parameters from[Del Debbio et al. 2007]with public code

Settings:

I tolerance: 10−10

I restart: 25

I block-size: 34

I local solver: 3 iterations GMRES

I coarse solver: 12 iterations GMRES

I levels: 2

I coarsening: 34 × 6→ 20

I 512 cores (Juropa @ JSC)





Results: 64× 323

FGMRES+ αaMG(8) CGN

setup timings 24.85s -solve iterations 17 9181

timings 5.05s 119.04stotal timings 29.90s 119.04s

Configuration:

I 64× 323 Wilson-Dirac

I 2HEX smeared tree level improvedClover

I parameters:I pion mass: mπ = 300 MeVI csw = 1.0

I BMW collaboration[arXiv:1011.2403][arXiv:1011.2711]

Settings:

I tolerance: 10−10

I restart: 25

I block-size: 24

I local solver: 3 iterations GMRES

I coarse solver: 12 iterations GMRES

I levels: 2

I coarsening: 44 × 6→ 20

I 4096 cores (Juropa @ JSC)





Summary and Outlook

Summary

I development stage of bootstrap AMG for QCD

I setup done once per configuration (and mass)

I high potential for parallelism in setup and solver

I implementation of 0th order interpolation (almost) available

Outlook

I optimize setup and its numerous parameters

I develop theoretical foundation

I implementation of 1st order approach

I application to physical configurations (w.r.t. β, κ, V )


algebraic multigrid in lattice qcd computations · the continuum limit scaling problem in qcd...

Documents