lawrence livermore national laboratory michael e. wickett r. w. anderson, n. s. elliott, b. t....

Lawrence Livermore National Laboratory

Michael E. Wickett

R. W. Anderson, N. S. Elliott, B. T. Gunney, R. D. Hornung, L. H. Howell, B. S. Pudliner, B. S. Ryujin

LLNL-PRES-494894

Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344

Structured Adaptive Mesh Refinement in a Multiblock Arbitrary–Lagrangian–

Eulerian Radiation–Hydrodynamics Code

International Conference on Numerical Methods For Multi-Material Fluid Flows

September 5 – 9, 2011

Arcachon, France

2LLNL-PRES-494894


ALE-AMR in ARESOutline AMR introduction• Definitions, mesh and algorithm choices• AMR infrastructure and scalability• Problem setup and refinement

AMR algorithmic pieces• Derefinement tangling• Refinement of multimaterial zones• Multiblock connectivity• Refinement of mesh motion state• Diffusion solver

Some results• Hydrodynamics• Radiation• MHD• Turbulent mix

3LLNL-PRES-494894


AMR definitions

Level 0

Level 1

Level 2

Patch (domain) based refinement

Levels are a collection of patches in the same index

space

Hierarchy is the collection of levels

Levels are completely nested inside the next coarser (i.e.

properly nested)

Covered coarse zones are still stored and used for computation, but they are

synchronized with the overlying coarse mesh

NO time subcycling All levels advanced with the

same timestep

4LLNL-PRES-494894


Staggered Lagrange hydro gives preference to odd refinement ratios

A 1:rd logical correspondence between both cell and nodal quantities is only possible with odd refinement ratios

This makes invertible pairs of operators simple to construct

level n+1

r = 2 r = 3level n

anisotropicrefinement is allowed

1x3

3x1

3x3

5LLNL-PRES-494894


To preserve quadrilateral zones, coarse solution drives fine solution at interfaces

Fine node positions at coarse-fine

interfaces set by interpolation from coarse positions

Leads to a particularly simple form for each part of the Lagrange step• Coarse grid solution is advanced• Fine grid solution is advanced getting incorrect values at coarse-fine interfaces• Fine grid solution at coarse-fine interfaces is interpolated from coarse

Has implications for many other parts of the method

6LLNL-PRES-494894


AMR capability built with SAMRAI: Structured Adaptive Mesh Refinement Applications Interface

SAMRAI Provides:

Transparent parallel communication (MPI)

Dynamic gridding support

Common data types (cell, nodes, …)

Inter-patch data transfer operations (copy,

coarsen, refine, time interp, …)

Solver interfaces for SAMR data (hypre,

PETSc, kinsol)

Checkpointing and restart (HDF5)

Visualization support (VisIt)

Multiblock, enhanced/reduced connectivity

User provides:

Numerical routines (serial) for individual patches

Composition of SAMRAI classes to implement desired algorithm.

7LLNL-PRES-494894


Unique SAMRAI design characteristicsenable ARES-AMR development Arbitrary (structured) mesh coordinates• ARES staggered-mesh ALE hydro unchanged• Mesh refinement is variable by coordinate direction (anisotropic refinement)

Multiblock AMR• straightforward translation between SAMRAI patch hierarchy and ARES block-

structured mesh• “enhanced” and “reduced” block connectivity

SAMRAI PatchData, communication abstractions• native ARES data operated on directly by AMR infrastructure

no rewriting of data structure for AMR, no extra copies of data for AMR

• SAMRAI handles interlevel communication (refinement, synchronization)• ARES handles intralevel communication (neighbor comm, reductions)

SAMRAI owns neither mesh nor data• invisible SAMRAI footprint for non-AMR ARES operation

8LLNL-PRES-494894


SAMRAI provides advanced AMR scalability Distributed mesh-management: processors store only local patch data and

nearest-neighbor patch geometry (No global mesh description)• Box intersection algorithm uses Recursive Binary Tree search• Load balance uses only nearest-neighbors on a processor tree• Clustering (box generation) uses a combined task- and data-parallel algorithm

Weak scaling results• problem grown by domain tiling• linear advection of wavy front• regrid every 4 timesteps• run on BG/L (LLNL uBGL)• scaled to 64K processors

Scaling work continues• optimization of new algorithms• removing global data from ARES

9LLNL-PRES-494894


We provide easy yet flexible AMR problem setup and many options for specification of refinement Problem is usually specified at coarsest level• Nodelists/Zonelists: scale automatically from

coarse level definition• level-specific syntax provided for exceptions

Direct tagging: can choose minimum/maximum refinement level over parts of problem• amr(myzonelist) = minlevel 3

Refinement criteria: global and local• tagging on values, 1st differences, or 2nd differences

of any problem field• logical combinations of criterion

Levels can be removed, both fine and coarse

refine default

criterion den 2nd_diff threshold_unrefined 0.04 threshold_refined 0.02 normalize local limit 0.1

criterion velocity_magnitude threshold_unrefined 1e-7 threshold_refined 9e-8 combine and

criterion p 2nd_diff threshold_unrefined 0.04 threshold_refined 0.02 normalize local limit 0.1 combine or

endrefine

Automatic scaling Automatic scalingLevel specific Level specific

levelop collapse level_numlevelop remove_finest num_levelslevelop remove_coarsest num_levels

10LLNL-PRES-494894


We must take care to avoid tangling on derefinement

It is possible that when regridding the hierarchy, the removal of finer mesh can lead to tangling of the mesh at the new coarse-fine interfaces.

To avoid this, we attempt to detect if such tangling will occur on regrid and prevent such zones from derefining.

Removal of shaded fine mesh

Removal of unshaded fine mesh

11LLNL-PRES-494894


Multi-material zone refinement is in development

• The capability to refine multi-material zones (“mixed zones”) is currently in development using a geometric overlay method.

• Because we currently cannot refine mixed zones, we cannot allow them to come into existence on any other than the finest level.

• Our default refinement criteria attempt to avoid this problem.• We prevent this situation by always tagging mixed zones for refinement.

If a fine zone is allowed to coarsen, on

subsequent cycles we cannot prevent it from

being refined

12LLNL-PRES-494894


Multiblock connectivity requires special AMR boundary conditions Traditional AMR algorithms treat coarse-fine boundaries by interpolating

coarse mesh values into “phony” or “ghost” zones or nodes of fine mesh Many of ALE-AMR algorithms can use this method, but some cannot• mesh relaxation: treat hierarchy as “composite” mesh• multiblock enhanced connectivity: data structures have only 1 phony at corner

0

1 2

3

4

mesh relaxation

extra step added to fill

multiple corner “phony” zones

13LLNL-PRES-494894


Refinement of ALE grid motion state

Grid motion state determines if and how a give node is moved/relaxed

Generally, refinement can be done by priority: fine nodes take the value of the highest “ranking” neighbor state

“BND” states allow relaxation along a logical line or surface• When refining near a BND

surface, only those nodes along the surface should get the BND state upon refinement.

• Between two BND states, we choose the NULL state

Green – iBNDBlue – jBNDRed – No RelaxWhite – NULL

Priority:5 – NULL4 – Relax3 – Backup then Relax2 – Backup1 – No Relax

14LLNL-PRES-494894


AMR diffusion solver

Coarse and fine levels are coupled together into a single matrix• We do not subcycle in time, so all levels participate in every solve• Covered zones may passively participate in the linear system, but they do not

affect the solution in exposed zones Pert discretization is extended to differencing between levels• The discretization only changes for zones adjacent to coarse-fine boundaries• Fine interface difference stencils are replaced with finite differences that see both

coarse and fine data• Coarse interface difference stencils are defined to match the flux of the

corresponding fine zones• System is conservative, but not quite symmetric

Sparse matrix built with HYPRE semi-structured interface• Still experimenting with solver options• Current default is AMG-preconditioned GMRES

15LLNL-PRES-494894


Multiblock Cylindrical Sedov

t = 0.07 red - single level

black - AMR

blue - exact

density

radius

t = 0.7

single-level (fine) 0.129

AMR (3-levels) 0.125

single-level (2/3 fine) 0.197

L2 Error Norm

3 levels

16LLNL-PRES-494894


3D Multiblock Sedov

t = 0.7

density

radius

t = 0.07

3 levels single-level (fine) 0.151



L2 Error Norm

red - single level

black - AMR

blue - exact

17LLNL-PRES-494894


2D Multiblock Spherical Piston

t = 0.65

density

radiussingle-level (fine) 0.948



L2 Error Norm

t = 0.1red - single level

black - AMR

blue - exact

3 levels

18LLNL-PRES-494894


3D Multiblock Spherical Piston

t = 0.65

red – single level

black – AMR

blue – 1D ref

density

radius

t = 0.1

single-level (fine) 0.684



L2 Error Norm

3 levels

19LLNL-PRES-494894


red - single level

black - AMR

2D Multiblock Balls and Jacks Advection Test

20LLNL-PRES-494894


2D Multiblock Balls and Jacks Advection Test

AL2 scaled mass ratio


time

21LLNL-PRES-494894


3D Multiblock Balls Advection Test

t = 3.5 time



22LLNL-PRES-494894


AMR can be used to increase timesteps on a fan mesh Fan mesh preserves symmetry better

for spherical problems

Coarsen mesh toward origin to lessen impact of CFL limits

• Some problems take 8-10x fewer cycles

Similar to spatial filtering methods

• Requires interpolation in polar coordinates

Has relatively large AMR overhead, but for this special case, the AMR infrastructure can be optimized

• No regridding

• Complete synchronization unnecessary

Pressure-Driven Capsule(Post-bounce, hydro only)

23LLNL-PRES-494894


Simulations of shock tube experimentson the Richtmyer-Meshkov instability Sharp Interface w/ single mode perturbation• No-Slip Walls• Upstream inflow at post-shock• Under-resolved laminar BL• Base resolution 19 zones/Width• Refined on 2nd differences of density• 4 levels with refinement ratio of 3x3 • Fully refined resolution 513/Width

Inclined interface w/ no perturbation• studied variations in angle, shock strength,

Atwood number, resolution• resolution 56 to 282 μm• 3 to 5 AMR levels• < 30% at finest at late time

Shocked Air

Pre-shocked

AirPre-

shocked SF6

Courtesy Jeff Greenough (LLNL)

24LLNL-PRES-494894


AMR radiation diffusion examples

Graziani-Leblanc crooked pipe• Refinement on 2nd difference of

radiation temperature

Planar radiating shock wave• Semi-analytic solution from

Lowrie and Edwards

~ 8x speedup

~ 5x speedup

25LLNL-PRES-494894


2D Cylindrical AMR MHD Magnetic field points in theta direction (normal to simulation plane) Magnetic diffusion operator discretized with a variation of the Pert scheme• AMR solver is as described for radiation diffusion

Hollow conducting cylinder with constant voltage drop across ends

Conductivity is function of radius

Exact time and space dependent solution by David Miller (LLNL)

Refinement on 2nd differences of magnetic field and on material interfacesCourtesy Rob Rieben (LLNL)

26LLNL-PRES-494894


AMR with subgrid turbulence mix models is a recent addition Barenblatt burst problem is an analytic problem with no hydro or sources• K-L turbulent mix model

Coupled non-linear diffusion, growth, and dissipation• AMR diffusion solver is as described for radiation diffusion

red - analytic

black - AMR

turbulent kinetic energy

turbulent length scale

27LLNL-PRES-494894


ALE-AMR Summary

AMR has become an integral part of the ARES code

• Progress has been made in AMR-izing nearly all components

• user input, control, output is growing complete and (relatively) straightforward

Solutions on a wide range of problems give good answers compared with everywhere fine solutions

Significant performance improvement is seen in many (not all!) problems

• overhead reduction continues to be worked

• SAMRAI scaling and multiblock improvements continue

Experience is being gained, but much more is needed

• choices refinement criteria, when/where can we avoid refinement?

• number of coarser levels, when to remove levels, etc.?

lawrence livermore national laboratory michael e. wickett r. w. anderson, n. s. elliott, b. t....

Documents

coarse positions

overlying coarse mesh

ryujin llnlpres

advanced fine grid solution

odd refinement ratios

level n anisotropic

france slide

timestep slide