An Analysis of ASPECT Mantle Convection SimulatorPerformance and Benchmark Comparisons

Eric M. Heien[1], Timo Heister [2], Wolfgang Bangerth[2], Louise H. Kellogg[1]

1) Department of Geology, University of California, Davis CA 95616 USA 2) Texas A&M University, College Station TX 77843 USA

AbstractDue to the inaccessible nature of the deep earth, simulations are a key tool in understanding long term mantle dynamics. The associated technical challenges are substantial and include highly variable viscosities, fracture and shear zone deformation, and multiple interacting length and time scales. To better model, simulate and understand the processes involved, multiple simulators have been developed over the past several decades. ASPECT (Advanced Solver for Problems in Earth’s ConvecTion) is a recently developed mantle convection simulation code with support for many advanced features including adaptive mesh refinement, high performance solvers, strang parallel scalability and easy extensibility.ASPECT is available at http://dealii.org/aspect/

Effect of Mesh Refinement

Scaling of 2D and 3D Simulations

An under resolved mesh in the simulation can cause unphysical phenomenon. To examine this, we performed simulations with the same physical parameters and initial conditions using different mesh resolutions. The figures below show the initial mesh and simulation state with a spherical temperature perturbation.

Refinement Level = 3 (7.68e2 cells, 1.07e4 degrees of freedom)

Refinement Level = 5 (1.23e4 cells, 1.62e5 degrees of freedom)

Refinement Level = 7 (1.96e5 cells, 2.57e6 degrees of freedom)

Already at t=5Myrs we see problems with low refinement. RL=3 loses the hot material plume near the surface and fails to resolve submerging flows between the upwellings. At this point, RL=5 and RL=7 are nearly indistinguishable.

At t=60Myrs, RL=3 has obvious divergence from higher resolutions and fails to model secondary rising plumes. RL=5 and RL=7 still resolve features like small uprising plumes, but RL=5 loses some small scale features such as vortices and accurate heat modeling in the plume caps.

By t=900Myrs, RL=3 is still divergent. RL=5 has lost feature symmetry with near-surface downwellings merging over time and creating further instability. RL=7 has maintained near perfect symmetry though apparent instabilities have appeared.

Adaptive Mesh Refinement

For reference, a 1Gyr simulation at RL=3 took 19 minutes on 16 cores, RL=5 took 134 minutes on 64 cores and RL=7 took 8 hours on 128 cores.

All simulations described here were run on the TACC Ranger system. This system consists of 16-way SMP compute nodes, each with 32GB of RAM and connected with an InfiniBand network.

Results of strong scaling tests for a 2D shell with 2.56e6 degrees of freedom are shown above. All aspects of computation scale well to 64 cores, or more specifically 5e4 to 1e5 degrees of freedom (DOF) per core.

The scaling for a shellmodel in 3D is similarto 2D, with strongscaling up to 1e5DOF per core. The main issue with largescale simulations is the initialization time as shown above. By 256 cores initialization requires 30 minutes. Extrapolation indicates a simulation using 12,000 cores (4.8e8 DOF at optimal scaling) would take 1 day to initialize – longer than the allowed run time on Ranger.

Fixed Mesh (Level = 7) Adaptive Mesh (Max Level 7)

Using adaptive mesh refinement coarsens the mesh where high detail is not required. The effect on simulation results is examined below.

At t=50Myrs, there is little difference between the fixed mesh and AMR mesh though the AMR mesh has 1/3 the DOFs potentially allowing for faster solution.

At t=500Myrs, there is clear divergence though the basic structure is the same.

Based on these results, it appears refinement levels of 3 or lower in ASPECT cause inaccurate modeling of mantle plumes and second order effects. AMR causes divergence of results but not significant change in plumes or similar phenomenon.

Fixed Mesh (2.57e6 DOF) AMR (7.98e5 DOF)

Fixed Mesh (2.57e6 DOF) AMR Mesh (8.81e5 DOF)