Parallel High-Order Geometric Multigrid Methodson Adaptive Meshes for Highly Heterogeneous

Nonlinear Stokes Flow Simulations of Earth’s Mantle

Johann Rudi∗, Hari Sundar†, Tobin Isaac∗, Georg Stadler∗, Michael Gurnis‡, and Omar Ghattas∗§¶∗Institute for Computational Engineering and Sciences, The University of Texas at Austin, USA

†School of Computing, The University of Utah, USA‡Seismological Laboratory, California Institute of Technology, USA

§Department of Mechanical Engineering, The University of Texas at Austin, USA¶Jackson School of Geosciences, The University of Texas at Austin, USA

POSTER SUMMARY

We address the problem of modeling global-scale,convection-driven flow in Earth’s mantle with associated platemotions. Modeling global mantle flow is a challenging problemdue to several inherent difficulties: (1) the severe nonlinearityof the rheology of the mantle, (2) the orders-of-magnitudeviscosity contrast and sharp viscosity gradients, and (3) thewidely-varying spatial scales. We present scalable numericalmethods and their efficient parallel implementation for solv-ing global mantle flow problems that represent a significantimprovement over our previous work (e.g., see [1]) in keyareas as the discretization as well as the linear and nonlinearsolver. In particular, we develop parallel scalable geometricmultigrid methods for preconditioning the linearized Stokessystems that arise upon discretization with high-order finiteelements on adaptive meshes. We provide numerical resultsbased on real Earth data that are likely the most realisticglobal scale mantle flow simulations conducted to date, andwe demonstrate scalability results on up to 16,384 cores.

Mantle flow is governed by nonlinear incompressibleStokes equations with shear-thinning rheology,{

−∇ ·[µ(T,u) (∇u+∇u>)

]+∇p = f(T ),

∇ · u = 0,

with temperature T , velocity u, pressure p, and effectiveviscosity µ. The viscosity depends exponentially on the tem-perature (via an Arrhenius relationship), on a power of thesecond invariant of the strain rate tensor, and incorporatesplastic yielding. The right-hand side forcing, f , is derivedfrom the Boussinesq approximation and depends on the tem-perature. We employ an inexact Newton-Krylov method tosolve the discretized nonlinear equations, with the advantagesof superlinear (or even quadratic) convergence, mesh sizeindependence, and avoidance of oversolving in early iterations[2]. Most of the solver time is spent in the inner Krylov loopand therefore it is crucial to find effective linear precondition-ers. Our discretization employs high-order finite elements onhighly adaptively refined hexahedral meshes, which allow us toresolve plate boundaries down to a few hundred meters whilestill capturing global-scale behavior. We obtain the discreteStokes system, [

A B>

B 0

] [up

]=

[f0

].

Multigrid hierarchy

GMGhigh-orderlinear

0 0

1 1

2 2

AMG2 2

3 3

4

direct solve2048 4096 8192 16384

0

0.5

1

1.5

1

1.2

0.97 0.931

1.241.15

1.34

number of cores

Weak scaling of setup & solve (relative to 2048 cores)

T/(N/P ) T/(N/P )/G

Figure 1. Left: Diagram of geometric-algebraic multigrid hierarchy. Ini-tially the adaptive mesh is coarsened geometrically (GMG) and repartitioneduniformly among all cores at each level. High-order operators are usedfor smoothing and residual computation. When the problem becomes smallenough, algebraic multigrid (AMG) is used on a small subset of cores andfor a linear finite element discretization. Right: Weak scaling of geometric-algebraic multigrid (solution of Au = f ). Normalized scaling of algorithmand implementation is T/(N/P ) and implementation only is T/(N/P )/G,where T is runtime, N is degrees of freedom, P is number of cores, and Gis number of iterations. The largest problem has 5.4B degrees of freedom.

Krylov methods are advantageous because they permitmatrix-free Jacobian applications. Within the matrix-free applyroutines, we exploit the tensor product structure of the velocityfinite element shape functions to increase the floating pointto memory operations ratio. We use a GMRES linear solverwith right preconditioning based on the upper triangular blockmatrix [5], [

A B>

B 0

]︸ ︷︷ ︸Stokes operator

[A B>

0 S

]−1︸ ︷︷ ︸

preconditioner

[u′

p′

]=

[f0

].

Here, approximations of the inverse of the viscous block,A−1 ≈ A−1, and the inverse of the Schur complement,S−1 ≈ S−1 := (BA−1B>)−1, are required.

As mentioned previously, adaptive mesh refinement (AMR)is important because the locally highly-varying viscosity needsto be resolved by the mesh (down to ∼0.5 km resolution).We use the p4est parallel AMR library, which provideshexahedral meshes with a forest-of-octrees structure and spacefilling curves for fast, parallel neighbor search, partitioning,and 2:1 balancing [3], [4]. The p4est library has beendemonstrated to scale well to O(500K) processor cores.

The inverse of the viscous block, A−1, is approximated

by an efficient, collective communication-free, parallel imple-mentation of geometric multigrid for high-order finite elementson adaptive meshes. This is a generalization of the geomet-ric multigrid method in [6] for the Poisson operator usinglinear elements. We use a hybrid geometric-algebraic multi-grid approach where the mesh is coarsened and repartitionedgeometrically first, and then once the problem size is smallenough, we revert to algebraic multigrid (PETSc’s GAMG)using only a small subset of cores (figure 1, left). On thegeometric multigrid levels, matrix-free apply functions for theviscous block have the advantage of being faster and morememory efficient than using assembled matrices. By limitingalgebraic multigrid to small problems on small core counts wecan avoid its high setup cost. Moreover, the assembled matrixfor algebraic multigrid is obtained from a linear finite elementdiscretization on the high-order nodes, which is beneficial forthe sparsity of the matrix as well as multigrid convergence.

The weak scaling of this high-order geometric multigridpreconditioner when solving the system Au = f is shown inthe chart on the right of figure 1. We consider the solutionto be converged after reducing the residual by six orders ofmagnitude. The numerical experiments were carried out onTACC’s Stampede supercomputer on up to 16,384 cores withproblem sizes up to 5.4B degrees of freedom. The normalizedscalability of the numerical algorithms and the implementationis better than optimal (i.e., <1.00 on 16,384 cores) since thenumber of iterations for convergence is decreasing as we refinethe mesh and resolve the variations in the viscosity better. Thescalability of the implementation only, which is normalizedwith respect to the number of iterations, is also shown infigure 1.

For solving the full Stokes system, it remains to find an ef-fective preconditioner for the inverse of the Schur complementS−1. We use a method that is robust with respect to extremeviscosity variations. It is an improved version of BFBT / LeastSquares Commutator methods (based on [7], [8]) and is definedas

S−1 = (BD−1B>)−1(BD−1AD−1B>)(BD−1B>)−1,

where D := diag(A). Using geometric multigrid for thepressure Laplacian, (BD−1B>)−1, is problematic due to thediscontinuous modal discretization of the pressure functionspace. We take a novel approach by re-discretizing the theLaplacian operator with continuous nodal basis functions. Thiscontinuous nodal Laplacian operator is then approximately in-verted with geometric multigrid. As a result, similar scalabilityresults as demonstrated in figure 1 hold.

Finally, we present results on solver convergence forthe nonlinear global mantle flow problem using an inexactNewton-Krylov method on 4096 cores. The graph in figure 2shows the reduction of the nonlinear (Newton) residual inred and the convergence of the linear (Krylov) residual inblue. The nonlinear solver was terminated after reducing thenonlinear residual by eight orders of magnitude. The nonlineardependence of the viscosity on the strain rate creates sharperviscosity gradients. Therefore, a grid continuation method isutilized, where the mesh is refined adaptively after the first fourNewton steps (indicated by black vertical lines). This preventsstalling of the linear convergence. The problem has 642Mvelocity and pressure degrees of freedom at the converged

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

10−12

10−9

10−6

10−3

100

12

3 4 5

6

78

91011

1213

141516

1718

1920

212223

242526

27

GMRES iteration

l2 n

orm

of

||re

sid

ua

l|| /

||in

itia

l re

sid

ual||

earth_bc1_k2_L14_gmgA1_gmgBBT0_schur1_bfbt3Newton−Krylov convergence (complexity: N=6.42e+08, P=4096, T=473min, T/P/N=1.08e−08sec)

nonlinear residual

Krylov residual

Figure 2. Left: Convergence of inexact Newton-Krylov method for nonlinearglobal mantle flow problem on 4096 cores. Red triangles indicate the nonlinear(Newton) residual `2-norm and blue lines represent the linear (Krylov)convergence in `2-norm. Each Newton step is separated by a gray verticalline. The mesh is adaptively refined after the first four Newton steps (blackvertical lines) to resolve the variations in the viscosity due to the nonlinearity.Right: Surface velocities of the Earth (around South America) at solution ofnonlinear global mantle flow problem.

solution. The surface velocities of the Earth at the solutionare shown on the right of figure 2. Due to the robustness ofthe preconditioners, we can observe steady linear convergence,which is crucial to obtain sufficiently accurate Newton steps,especially at the last couple of Newton steps when the linearsystems need to be solved with higher precision.

In conclusion, we have shown that we are able to overcomethe challenges of global mantle flow problems by designingsolvers that combine two essential components: powerful nu-merical methods and their efficient scalable parallel imple-mentation. Inexact Newton-Krylov methods are employed tohandle the severe nonlinear rheology. AMR is used to resolvethe locally highly-varying viscosity, as well as between Newtonsteps as a grid continuation technique, resulting in effectivecapture of global scale behavior. Central for reducing thetime to solution is the effectiveness and robustness of thepreconditioners for A−1 and S−1, which are built on high-order multigrid methods. This combination of inexact Newton-Krylov, high-order multigrid, and AMR permits us to carry outthe most realistic global mantle flow simulations to date.

REFERENCES

[1] C. Burstedde, O. Ghattas, M. Gurnis, E. Tan, T. Tu, G. Stadler, L. C.Wilcox, and S. Zhong, Scalable adaptive mantle convection simulationon petascale supercomputers, SC08: Proceedings of the InternationalConference for High Performance Computing, Networking, Storage andAnalysis, 2008.

[2] S. C. Eisenstat and H. F. Walker, Globally convergent inexact Newtonmethods, SIAM Journal on Optimization, 4 (1994), pp. 393–422.

[3] C. Burstedde, O. Ghattas, M. Gurnis, T. Isaac, G. Stadler, T. Warburton,and L. C. Wilcox, Extreme-scale AMR, SC10: Proceedings of theInternational Conference for High Performance Computing, Networking,Storage and Analysis, 2010.

[4] C. Burstedde, L. C. Wilcox, and O. Ghattas, p4est: Scalable algorithmsfor parallel adaptive mesh refinement on forests of octrees, SIAM Journalon Scientific Computing, 33 (2011), pp. 1103-1133.

[5] M. Benzi, G. H. Golub, and J. Liesen, Numerical solution of saddle pointproblems, Acta Numerica, 14 (2005), pp. 1–137.

[6] H. Sundar, G. Biros, C. Burstedde, J. Rudi, O. Ghattas, and G. Stadler,Parallel geometric-algebraic multigrid on unstructured forests of octrees,SC12: Proceedings of the International Conference for High PerformanceComputing, Networking, Storage and Analysis, 2012.

[7] H. Elman, V. Howle, J. Shadid, R. Shuttleworth, and R. Tuminaro, Blockpreconditioners based on approximate commutators, SIAM Journal onScientific Computing, 27 (2006), pp. 1651–1668.

[8] D. A. May and L. Moresi, Preconditioned iterative methods for Stokesflow problems arising in computational geodynamics, Physics of theEarth and Planetary Interiors, 171 (2008), pp. 33–47.