MNRAS 000, 1–15 (0000) Preprint 19 July 2017 Compiled using MNRAS LATEX style file v3.0

A Moving Mesh Hydrodynamic Solver for ChaNGa

Philip Chang1?, James Wadsley2, and Thomas R. Quinn31 Department of Physics, University of Wisconsin-Milwaukee, 3135 North Maryland Avenue, Milwaukee, WI 53211, USA2 Department of Physics & Astronomy, McMaster University, Hamilton, ON, L8S 4M1, Canada3 Department of Astronomy, Box 351580, University of Washington, Seattle, WA 98195-1580, USA

19 July 2017

ABSTRACTWe describe the structure and implementation of a moving-mesh hydrodynamics solverin the large-scale parallel code, Charm N-body GrAvity solver (ChaNGa). Whilelargely based on the algorithm described by Springel (2010) that is implemented inAREPO, our algorithm differs a few aspects. We describe our use of the Voronoitessellation library, VORO++, to compute the Voronoi tessellation directly. We alsoincorporate some recent advances in gradient estimation and reconstruction that givesbetter accuracy in hydrodynamic solutions at minimal computational cost. We vali-date this module with a small battery of test problems against the smooth particlehydrodynamics solver included in ChaNGa. Finally, we study one example of a scien-tific problem involving the mergers of two main sequence stars and highlight the smallquantitative differences between smooth particle and moving-mesh hydrodynamics.We close with a discussion of anticipated future improvements and advancements.

Key words: methods: numerical — hydrodynamics – stars: binaries: general

1 INTRODUCTION

Numerical simulations have played a crucial role in under-standing the hydrodynamics (HD) and magnetohydrody-namics (MHD) of gas, stars, disks, galaxies, large-scale struc-ture, and other astrophysical phenomenon. In this regard,two dominant methodologies have emerged to numericallysolve the HD and MHD equations: smooth particle hydro-dynamics (SPH) and grid-based solvers.

SPH is based upon the Lagrangian view of the Eulerequations where the sampling of a fluid is determined froma finite number of particles, and fluid quantities like den-sity and pressure are determined by computing a smoothingkernel over a number of neighbors. The Lagrangian natureof SPH allows it to conserve linear and angular momentum,but comes at the expense of comparatively poor resolutionof shocks due to its smoothing nature. On the other hand,grid based methods have superior shock capturing abilitiesdue to the use of Godonov schemes, but suffers from grideffects and possible violations of Galilean invariance.

Arbitrary Lagrangian-Eulerian (ALE) schemes havebeen devised as an effort to capture the best characteris-tics of both approaches (for a review see Donea et al. 2004).Noh (1964) proposed the first 2-D ALE scheme where theinitial grid was continuously deformed as the solution pro-gressed. In 3-d, Gnedin (1995), Pen (1998) also proposedschemes based on similar ideas. These moving-mesh (MM)

? E-mail: chang65@uwm.edu

methods suffered from mesh distortion or mesh tangling. Tocorrect this, these methods remapped the fluid onto a newmesh, which is inherently a diffusive operation. As a result,these methods were not widely adopted in astrophysics withthe exception of supernova modeling (Murphy & Burrows2008).

The need to avoid grid tangling and the expensive anddiffusive grid remapping operations lead to schemes wherethe remapping happens continuously or in localized patches.Borgers & Peskin (1987) proposed constructing a Voronoidiagram on a moving field of fluid markers, and then usingthis diagram to construct finite difference operators to solvethe fluid equations. Later, Whitehurst (1995) proposed amethod by which a Delaunay tessellation was constructedon a field of moving fluid elements on which finite volumemethods are used to solve the fluid equations. These twomethods both avoid the effect of grid tangling and diffusiveglobal grid remapping, but they were also only limited to2-D, and were, perhaps, ahead of their time.

Springel (2010, hereafter S10) described a usable ALEscheme that has proven successful. Implemented into thecode, AREPO, the scheme relies on a Voronoi tessellationto generate well-defined and unique meshes for an arbitrarydistribution of points. AREPO was built on top of the Gad-get codebase (Springel 2005), a widely used massively par-allel Tree-SPH code, and has been widely applied to a num-ber of problems. The use of Voronoi tessellations allows aunique mesh to be defined, and these meshes deform contin-uously under the movement of the mesh generating points.

© 0000 The Authors

arX

iv:1

707.

0533

3v1

[as

tro-

ph.I

M]

17

Jul 2

017

2 Chang, Wadsley, & Quinn

These cells have well defined volumes and faces on which fi-nite volume methods can be applied. The implementation ofAREPO detailed in S10 has been extended to include mag-netic fields (Pakmor et al. 2011; Mocz et al. 2014, 2016),better convergence (Pakmor et al. 2016a; Mocz et al. 2015),and new physics, such as cosmic rays (Pakmor et al. 2016b;Pfrommer et al. 2017) and have been used in a number ofdifferent problems including cosmological galaxy formation(see for instance Vogelsberger et al. 2014), and stellar merg-ers (Zhu et al. 2015; Ohlmann et al. 2016).

The scheme proposed in S10 has led to a num-ber of ALE codes including TESS(Duffell & Mac-Fadyen 2011), FVMHD3D(Gaburov et al. 2012), Shadow-Fax(Vandenbroucke & De Rijcke 2016), RICH(Yalinewichet al. 2015), and DISCO(Duffell 2016). In addition, the gen-eral scheme of determining the geometry from an arbitrarycollection of points has also led to derivative methods suchas GIZMO (Hopkins 2015).

Both AREPO and GIZMO are built on top of theGadget codebase (Springel 2005), which demonstrates thatSPH codes can be modified to an ALE scheme. A similarcode, Gasoline (Wadsley et al. 2004), has been developedby James Wadsley, Joachim Stadel and Thomas Quinn andused in a number of different problems. A successor to Gaso-line that has been under heavy development over the lastdecade is called ChaNGa (Jetley et al. 2008, 2010; Menonet al. 2015). In addition to SPH, ChaNGa includes stan-dard physics modules that have been ported from Gasolineincluding metal line cooling, star formation, turbulent diffu-sion of metals and thermal energy, and supernovae feedback(Stinson et al. 2006; Shen et al. 2010). The usefulness ofChaNGa in galaxy simulations has been demonstrated in theAGORA project (Kim et al. 2014, 2016). The most up-to-date description of the algorithms in Gasoline and ChaNGais given in Wadsley et al. (2017)

ChaNGa is also unique among astrophysical codes1 inthat it uses the Charm++ language and run-time system(Kale & Krishnan 1996) for parallelization rather than a cus-tom message-passing interface design. The use of Charm++promises that ChaNGa will be much more scalable com-pared to previous astrophysical codes. The message drivenparadigm of Charm++ allows ChaNGa to be latency toler-ant and overlap communication (significant in the SPH cal-culation) with computation (significant in the gravity calcu-lation). The concept of over-decomposition allows the workto be composed into objects independently of the numberof processors, and the Charm++ adaptive run-time sys-tem distributes these objects among the physical processorsin order to balance the computational load and minimizecommunication. Using these features, ChaNGa has demon-strated strong scaling on single timestepping problems with12 billion particles to 512K cores (with 93% efficiency) andon multi-timestepping problems with 52 million particles to128K cores on Blue Waters(Menon et al. 2015). In the eraof exascale computing, such scalability becomes increasinglyimportant.

1 However, a Charm++ version of ENZO is currently un-der development by James Bordner and Michael Norman andFVMHD3D by Gaburov et al. (2012) also uses the Charm++

language.

We implement a MM hydrodynamic solver based onthe algorithm describe in S10 in the latest public releaseof ChaNGa (version 3.2). This module does not have anofficial name, but for the purposes of this paper, we willcall it MANGA rather than calling it “the MM hydrody-namics solver module for ChaNGa.” This is mainly for rea-sons of convenience in nomenclature. Our implemention ofMANGA differs from S10 in several ways. First, ratherthan generating the Delaunay tessellation and computingits dual to produce the Voronoi tessellation, we computethe Voronoi tessellation directly using the publicly availablelibrary, VORO++ (Rycroft 2009). Second, we use an im-proved gradient estimate from Steinberg et al. (2016) and(optionally) use a total variation diminishing (TVD) lim-iter proposed by Duffell & MacFadyen (2011). Finally, weperform the reconstruction of face centered values from thehalf-time step state vector and gradient estimates.

This paper is organized as follows. We outline the basicalgorithm in § 2. We then describe the steps of the algo-rithm in brief, but in sufficient detail to allow this paperto serve as a reference for future developers and users ofMANGA. In particular, we detail the differences betweenMANGA and the implementation of AREPO described inS10. We describe the construction of the Voronoi tessellationin § 3 using the VORO++ library. We then describe the hy-drodynamic algorithm in § 4 with particular emphasis on thereconstruction of face-centered quantities in § 4.1, the deter-mination of mesh-generating point velocities and timestepsin § 4.2, the use of the entropy versus energy evolution equa-tions in § 4.3, and initialization in § 4.4. We then show theperformance on MANGA compared to the SPH solver inChaNGa on a number of 3-D test problems in § 5 includingthe Sod shock-tube problem, the Sedov-Taylor point explo-sion, the Gresho-Chan vortex, the Evrard collapse problem,a stellar hydrostatic balance problem, and a stellar merger.We describe a number of improvements targeted for the fu-ture in § 6 and close with some conclusions in § 7.

2 OUTLINE OF THE ALGORITHM

The ALE algorithm that is implemented in MANGA is sum-marized as follows. Much of the algorithm follows that ofS10, but with notable differences.

(i) We determine a valid Voronoi tessellation of the mesh-generating points using the VORO++ library.

(ii) Using the volume of the Voronoi cell and integralquantities, U, the conserved and primitive variables are de-termined. The local gradients and half-time-step conservedvariables are calculated.

(iii) The half-time-step gradients are calculated and usedalong with the half-time-step conserved variables to recon-struct the half-time-step face-centered quantities. An appro-priate (optionally total variation diminishing) gradient lim-iter is applied.

(iv) An Harten-Lax-van Leer-Contact (HLLC) or Harten-Lax-van Leer (HLL) approximate Riemann solver estimatesthe numerical flux in the rest frame of the moving faces ofthe half-time-step state. This flux is transformed back to the“lab” frame to determine the changes to the state.

(v) The new state U ′ is determined from the fluxes. New

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 3

velocities for the mesh generating points are determined.The mesh generating points drift by a full time step.

We refer the reader to S10 for an excellent and detaileddiscussion. Here we will briefly describe our methodologyand highlight differences between our implementation andthat of S10.

3 DIRECT CONSTRUCTION OF THEVORONOI TESSELLATION

The key insight of S10 to allow the construction of a robustALE scheme was the use of a Voronoi tessellation to gener-ate a mesh from an arbitrary collection of points on whichthe (M)HD equations can be solved with a finite-volumescheme. The Voronoi tessellation has some important prop-erties that make it particularly amendable for the construc-tion of an ALE scheme. First, the tessellation is unique.Second, the tessellation varies smoothly and continuouslyunder the motion of the underlying mesh generation points.By smoothly and continuously, we mean that the neighborsdo not suddenly change due to perturbation in the positionof the mesh generating points, and the quantities that definethe cell, i.e., the volume of the cell and areas of the faceschange smoothly under a perturbation of the mesh gener-ating points. This allows neighbors to be well defined andchange smoothly across a time-step.

A Voronoi tessellation can be created in two ways. First,it can be created by determining a Delaunay triangulation,which in 3-D is a partition of a space by tetrahedra where thevertices of the tetrahedra are given by the mesh-generatingpoints, and no points are contained in the circumsphere ofany given tetrahedra. The Voronoi tessellation is then com-puted by computing the dual to the Delaunay tessellation.Second, it can be computed directly.

The first approach is taken by S10, who describes thegeneration and parallelization of the Delaunay tessellationfrom which the Voronoi tessellation can be computed. Thisalgorithm is implemented in AREPO, TESS, FVM3D, andRICH because the quality of S10’s description. Here, theempty circumsphere property uniquely determines the tetra-hedra partition of the space. The Voronoi tessellation is thencomputed by determining the centers of the circumspheresfor each tetrahedra. These centers are the vertices that makeup the Voronoi tessellation.

The second case is the approach described in this pa-per. Using the publicly available Voronoi tessellation libraryVoro++2(Rycroft 2009), we compute the tessellation di-rectly. We first enclose a point p about which we wishto compute its Voronoi cell with a large rectangular cell,which is the starting guess for the Voronoi cell. The ge-ometry of this large cell is irrelevant, but it must be muchlarger than the Voronoi cell that will be computed. Using arapid nearest-neighbor search algorithm, we can search forall neighbors n up to a radius rs, which we will presume isarbitrary. We order the neighbors, n, by distance, and start-ing from the nearest n, we compute the plane that bisectsthe line connecting p and n. If the plane does not partitionthe Voronoi cell, i.e., intersect any of the faces of the Voronoi

2 http://math.lbl.gov/voro++/

cell, then n does not share a common face with p. If it does,then n shares a face with p, and the partition of the Voronoicell by this plane forms a new face of the cell, whose verticeswe compute. We then update the Voronoi cell with the newface and the modifications to existing faces. We continue bi-secting the Voronoi cell, with n’s of increasing distance fromp until the distance between p and n is more than twice thatbetween p and the most distance vertex in the Voronoi cell.All points greater than this distance can no longer partitionthe Voronoi cell, which is now complete.

We have tested the Voro++ library and found thatit has very high performance. Using its native containers,Voro++ is able to compute 100K 3-D Voronoi cells per sec-ond on a single core of an Intel core i7-3770 running at3.40GHz. Moreover, because the calculations are entirely lo-cal in the neighborhood of each point, it is straightforward toparallelize provided that a sufficiently fast neighbor searchexists. We have found that the native Voro++ containersare expensive to instantiate, so we compute the Voronoitessellation on a cell-by-cell basis. This reduces the perfor-mance of the Voro++ library substantially, but we are stillable to generate 10K cell per second per core, which is ad-equate performance. Part of this poorer performance is dueto the crude criterion to test for completion of a Voronoi cellthat we use, i.e., all remaining neighbors are at least twicethe distance to the farthest vertex. The native containers inVoro++ perform additional tests, where points only on oneside of the cells is tested. S10 also described more optimizedsearch strategies. The other reason for this slowdown is thatthe domain decomposition is totally arbitrary and not opti-mized for the construction of Voronoi tessellation. Optimiza-tion of the construction of Voronoi cells is an improvementtargeted for MANGA, but its performance is sufficient fornow.

4 HYDRODYNAMICS ON A MOVINGVORONOI TESSELLATION

MANGA solves the Euler equations and evolution equationfor entropy, which written in conservative form is:

∂ρ

∂t+ ∇ · ρv = 0 (1)

∂ρv

∂t+ ∇ · ρvv + ∇P = −ρ∇Φ (2)

∂ρe∂t+ ∇ · (ρe + P) v = −ρv · ∇Φ (3)

where ρ is the density, v is the velocity, Φ is the gravitationalpotential, e = ε + v2/2 is the specific energy, ε is the internalenergy, and P(ρ, ε) is the pressure. Equations (1) - (3) canbe written in a compact form by introducing a state vectorU = (ρ, ρv, ρe):

∂U∂t+

∫∇ · FdV = S (4)

where F = (ρv, ρvv, (ρe + P)v) is the flux function, and S =(0,−ρ∇Φ,−ρv · ∇Φ) is the source function.

To solve equation (4), we adopt the same finite volumestrategy as in S10. We refer the interested reader to S10 fora more detailed discussion of the scheme. Here, we will onlybriefly describe the scheme to document the algorithm we

MNRAS 000, 1–15 (0000)

4 Chang, Wadsley, & Quinn

have implemented and to highlight the differences betweenour scheme and that of S10.

For each cell, the integral over the volume of the ith celldefines the charge of the ith cell, U i , to be

U i =

∫iUdV =UiVi, (5)

where Vi is the volume of the cell. As in S10, we then useGauss’ theorem to convert the volume integral over the di-vergence of the flux in equation (4) to a surface integral:∫

i∇ · FdV =

∫iF · ndA (6)

We now take advantage of the fact that the volumes areVoronoi cells with a finite number of neighbors to define aintegrated flux ∑

j∈neighborsFi j Ai j =

∫iF · ndA, (7)

where Fi j and Ai j are the average flux and area of the com-mon face between cells i and j. The discrete time evolutionof the charges in the system is given by:

Un+1i = Un

i + ∆t∑j

Fi j Ai j + ∆tSi, (8)

where Fi j is an estimate of the half-time-step flux between

the initial, Uni, and final states Un+1

iand is discussed in § 4.1,

and S(n+1/2)i

=∫iSdV is the time-averaged integrated source

function.We estimate the flux across each face, Fi j , using an ap-

proximate Riemann solver. As Riemann solvers for irregularcells in multidimensions do not exist, we follow the prescrip-tion of S10. We compute the 1-D fluxes across each face inthe rest frame of that face and then collectively apply themper time-step. The steps involved are:

(i) Estimate the velocity wi j of the face – following S10,the face velocities are:

wi j =(wi − w j ) · (r i j − 0.5(r j + r i))

|r j − r i |r j − r i

|r j − r i |+ wi j, (9)

where wi j = 0.5(wi + w j ) is the average velocity of the twomesh generating points and r i j is the face center betweencells i and j.

(ii) Estimate the half-time-step state vector (in the restframe of the moving face) at the face center (r i j) betweenthe neighboring i and j cells via linear reconstruction – wediscuss this reconstruction in § 4.1.

(iii) Boost the state vector from the“lab”frame to the restframe of the face center and rotate the state vector such thatthe x-axis points along the outward normal of the face, i.e.,in the direction from i to j.

(iv) Estimate the flux using an HLL or HLLC Riemannsolver implemented following Toro (2009). Here we foundthat both Riemann solvers give acceptable performance,though the HLL solver is more diffusive for problems thatinvolve large gradients integrated over long timescale, i.e.,hydrostatic balance. By default, we choose HLLC.

(v) Boost the solved flux back into the “lab” frame.

We can then use the estimated fluxes to time evolve thecharges, U i , following equation (8).

The inclusion of (self-)gravity involves incorporating the

gravitational potential into the momentum and energy equa-tions. Here our formalism takes care of the momentum equa-tion modulo a proper definition for the time average inte-grated source function. Here we follow the suggestion of S10and define∫

iρi∇ΦidV =

12

(m(n)i

∇Φ(n)i+ m(n+1)

i∇Φ(n+1)

i

), (10)

where m =∫iρdV is the integrated mass per cell. The source

for the energy equation can similarly be defined by S10 asthe “standard approach”:∫

iρi∇Φi · vdV =

12

m(n)i

v(n)i

∇Φ(n)i+

12

m(n+1)i

v(n+1)i

∇Φ(n+1)i

, (11)

but as S10 noted, a better scheme is to utilize the Green-Gauss theorem to produce an “improved approach.” As de-scribed by S10, it involves expanding the velocity into thecell velocity component and the fluid velocity component(relative to the cell) in the above integral. The componentthat involves the cell velocity can be integrated over volumeand time to give the change in the gravitational potentialenergy due to the movement of the entire cell. The compo-nent that involves the relative (to the mesh generating point)fluid velocity can be converted to a surface integral and in-tegrated over time to give the change in the gravitationalpotential energy of the cell due to advection of gravitationalpotential energy across the cell faces. The relevant equationfor the “improved approach” is∫

iρi∇Φi · vdV = mi∇Φi · wi +∫

iρ [(v − wi) · n] (r − r i) · ∇ΦidA. (12)

Here as in S10, the density in the first term on the RHScan be integrated over the volume of the cell to give thetotal mass of the cell, holding the gravitational potentialand cell velocity to be constant. The second term can beapproximated noting that term,

∫iρ [(v − wi) · n] dV , is the

mass flux across all the faces and the term (r − r i) · ∇Φiis the change in potential going from the face to the cellcenter. We approximate this second term by using the massflux (across each face) calculated by the Riemann solver andsetting (r − r i) ·∇Φi =

(r j − r i

)·∇Φi/2. Both the “standard”

and “improved” approaches are implemented in MANGA,but the “improved” approach is used by default.

The gradient of the gravitational potential, ∇Φ, i.e., thegravitational acceleration, can either be user-specified, de-rived from the solution of Poisson’s equation:

∇2Φ = 4πGρ (13)

for the case of self-gravity, or a hybrid of the two. For self-gravity, we use the tree-based solver in ChaNGa where eachtree node contains the multipole mass moments up to hex-adecapole order. The tree traversal algorithm for gravity isbased on PKDGrav as described in Stadel (2001). It is sim-ilar to the Barnes-Hut (Barnes & Hut 1986) algorithm withsome optimizations; for a few more details see Menon et al.(2015). Each leaf node of the tree contains a number (by de-fault, 12 or less) of mesh generating points or other types ofparticles. The mass distribution, ρ, is represented by a cu-bic spline softened density distribution with fixed softening

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 5

centered at each mesh generation point. This is consistentwith how ChaNGa represents the density distribution fromcollisionless particles.

4.1 Gradient Estimation and Reconstruction ofConserved Quantities

A second order accurate code in time and space demands anappropriate estimate for the state vector at the face centersof each cell at the half timestep. In MANGA, we first makean estimate for the state vector at the half timestep, use thishalf timestep state vector to compute the spatial gradient,and then use the spatial gradient to estimate the state vectorat the face centers. For our estimate of the state vector atthe half timestep, we use an estimate for the time derivativefrom equation (4)

∂U∂t

(n)= −∇ · F(n) + S(n), (14)

where we express ∇ · F(n) in terms of gradients of the state

vector: ∇U(n)i

. The half-time step state vector can then bedetermined:

U(n+1/2)i

=U(n)i+∂U(n)

i

∂t∆t2. (15)

The state vector on face of the ith and jth cell on the (n +

1/2)-th timestep, U(n+1/2)i j , is then

U(n+1/2)i j =U(n+1/2)

i+ (r i j − ci) · ∇U(n+1/2)

i, (16)

where ci is the center of mass of the ith cell.Equations (14) and (16) require an estimate for the gra-

dient of the state vector. Here, we follow the procedure ofSteinberg et al. (2016) who improved upon the prescriptionof S10 in using the Green-Gauss theorem to estimate thesegradients. The crucial difference between Steinberg et al.(2016) and S10 is that the former estimates the gradientfrom the cell centers whereas S10 estimates the gradientsfrom mesh generating points and assigns them to the cellcenters. If the mesh generating points are at the cell centers,they these methods are the same. However, this is usuallynot the case. According to Steinberg et al. (2016), this cell-centered gradient estimate improves this convergence prop-erties to be comparable to the least square gradient estimateused in later versions of AREPO (Pakmor et al. 2016a). Thegradient is then set to:

〈∇U〉S10i = αS10

i 〈∇U〉i and αS10i = min(1,ψi j ), (17)

where αS10i

is a slope limiter defined by S10 that reducesnumerical oscillations near strong gradients, i.e., shocks, andψi j is defined as

ψi j =

(Umax

j−Ui)/∆Ui j for ∆Ui j > 0

(Uminj −Ui)/∆Ui j for ∆Ui j < 0

1 otherwise,(18)

where Umaxj

and Uminj are the maximum and minimum val-

ues of the state vector among the neighbors (including itself)of the ith cell, and ∆Ui j = 〈∇U〉i ·

(c j − ci

)is the variation

computed from the gradient to the cells containing the maxi-mum and minimum values of components of the state vector.

As noted by S10, this slope limiter is not total variation di-minishing (TVD) so spurious oscillations can still occur nearstrong gradients. To control this, we follow the suggestionby Duffell & MacFadyen (2011) and (optionally) apply anadditional correction when computing the gradient betweencells i and j

〈∇U〉′i = αDMi 〈∇U〉S10

i and αDMi = min(1,ψ ′i j ), (19)

where αDMi is a slope limiter set between the ith and jth

cells and ψ ′i j is

ψi j =

{max[θ(U j −Ui)/∆Ui j ), 1.] for |∆Ui j | > 01 otherwise, (20)

where θ < 0.5 gives a TVD limiter and ∆Ui j is definedas above, but is now applied only to the jth cell and notmaximized over all the neighbors. Note that αS10, αDM, andψi j are defined per component of the state vector and thatsetting θ = 1 reduces it to the S10 slope limiter.

We note that we work with the conserved variableswhereas S10 works with the primitive variables to estimatethe face centered primitive variables, which is then trans-formed to conserved quantities. We also note that S10 com-putes the face variable using only one estimate for the gra-dients at the n-th timestep compared to the two estimatesfor the gradients that we do here. We believe that the S10method is somewhat faster, though we have believe ourmethodology to be somewhat more robust (against produc-ing unphysical values) especially in regions with large gra-dients.

As the conserved quantities evolve or are reconstructed,the resulting density and internal energy can become un-physical, i.e., not positive definite. This is not a new problemwith Eulerian codes in general and a number of fixes mustbe applied. For instance, both FLASH (Fryxell et al. 2000;Dubey et al. 2008) and RAMSES (Teyssier 2002) enforcea density and temperature (internal energy) floor on thesequantities so that they do not become too small. How andwhere these conditions are enforced produces a balance be-tween correctness and numerical robustness. For MANGA,we have opted toward greater robustness (and user sanity)by enforcing a density and temperature floor in two places:at the cell center after every timestep and on the faces ofthe cell after the reconstruction step.

4.2 Determining the Velocities and Timesteps ofMesh Generating Points

A particular feature of the scheme (as noted by S10) is itsflexibility in that it operates as a Eulerian scheme, a La-grangian scheme, or a continuous hybridization between thetwo. For instance, setting the velocities of the mesh gener-ating points to 0 (w = 0) allows the code to operate as a Eu-lerian scheme or static-mesh (SM) scheme. Likewise, settingw = v allows the code to operate as a Lagrangian scheme. Ahybrid between these two schemes would be w = ηv, whereη is arbitrary. While this capability is available, in practicewe have mainly used η = 1.

One important improvement for this scheme that waspointed out by S10 is the need to keep the Voronoi cellsas regular or “round” as possible for which S10 proposed amethod to maintain roundness. Here we briefly describe the

MNRAS 000, 1–15 (0000)

6 Chang, Wadsley, & Quinn

method in keeping with our desire that this paper serve as acomplete and definitive reference for MANGA. As discussedin S10, “roundness” is maintained when the mesh generatingpoints, r i , are close to the centers of mass of the Voronoicells, ci . To do this, we need to correct the velocity wi suchthat it drifts toward ci at some fraction of the local soundspeed χcs,i . Once it is sufficiently close, this correction canbe continuously reduced to zero. The method proposed byS10 utilizes a definition of an effective radius for these cells,Ri as

4πR3i

3= Vi, (21)

where Vi is the volume of the cell. The correction is then

∆w = χcs,i di

0 for di < 0.9ζRi

di−0.9ζRi

0.2ζRifor 0.9ζRi ≤ di ≤ 1.1ζRi

1 for di > 1.1ζRi

, (22)

where di = ci − r i and ζ is a constant that we normally setto be 0.25. For most of our runs, we have set χ = 1 as wasdone in S10.

The definition of an effective radius also allows us todetermine a timestep for each cell. Currently, we have onlyimplemented a global timestep in MANGA, so we can definethe timestep per cell as

∆ti = ηCFLRi

cs,i + |vi − wi |, (23)

where |vi − wi | is the velocity of the fluid relative to thevelocity of the mesh generating point, and ηCFL is theCourant-Friedrich-Levy coefficient and is usually set to beηCFL = 0.5 − 0.8. The global time step is then set to be theglobal minimum ∆ti or

∆tcfl = mini∆ti, (24)

which is then sent to the timestepper in ChaNGa. Here, amodification that occurs is that ChaNGa is geared towardindividual timesteps and determines the timestep to use tobe a factor of 2−n of a global time step ∆tglobal, where n = 0to 30. Hence the timestep that is used is

∆t = 2−n∆tglobal where 2−n ≤ ∆tcfl∆tglobal

≤ 2−(n−1). (25)

The individual timestepping scheme implemented inChaNGa allows for the possibility of extending MANGA toindividual timesteps which should greatly speed up prob-lems with large dynamic ranges. We will discuss this futureimprovement in § 6.

4.3 Entropy versus Energy Evolution

The numerical solution to the energy equation (3) can besubject to spurious numerical heating particularly in coldflows that are dominated by a (numerically) noisy gravita-tional field. To help remedy this issue, S10 introduced anentropy equation that is absent a source to alleviate thisspurious heating:

∂ρs∂t+ ∇ · ρsv = 0, (26)

where s is the specific entropy. The solution to equation (26)follows the same strategy as in equation (4) with the addi-tion a column of ρs to the state vector, U, ρsv to the flux

function, F, and 0 to the source function, S. The energy evo-lution equation (3) and entropy evolution equation (26) areredundant. Moreover, they are inconsistent with each other;the energy equation (3) has a source whereas the entropyequation (26) does not. This redundancy and inconsistencyis not a bug but rather a feature. In particular, the entropyequation (26) is not subject to spurious heating whereas theenergy equation (3) is. On the other hand, the energy equa-tion can capture the effects of source heating and shock heat-ing whereas the entropy equation cannot.

To allow for the capture of external and shock heating,while minimizing spurious heating, S10 switches the evolu-tion of the gas between the energy and entropy equationsin a cell depending on the existence of shocks of sufficientstrength. In particular, S10 determines the maximum Machnumber of the shocks (if any) over all the faces of the cell(during the solution of the Riemann problem). If this max-imum mach number exceeds a certain threshold, Mth = 1.1,then the energy equation is used. Otherwise the entropyequation is used. The internal energy or entropy is then resetbased of the particular equation that was integrated. Hence,the noninclusion of a source term in the entropy equation isintentional, i.e., we eliminate spurious heating due to numer-ical noise in the gravity solve. S10 demonstrated the utility ofthis switch in the Santa Barbara cluster test (their Figures44 and 45), where the use of this switch prevent artificialhigh redshift heating of the gas, albeit with a scheme thateffectively used a much larger Mth.

We have implemented this same scheme in MANGA,and we have chosen the same values as in S10 withMth = 1.1.We note that we can choose to use only the energy evo-lution equation (3) if we set M = 0. We also note thatother moving mesh codes including TESS(Duffell & Mac-Fadyen 2011), FVMHD3D(Gaburov et al. 2012), Shadow-Fax(Vandenbroucke & De Rijcke 2016), RICH(Yalinewichet al. 2015), and DISCO(Duffell 2016) do not include theentropy equation.

4.4 Initialization

In our implementation of MANGA, we have kept the in-put/output and timestepping architecture of ChaNGa sothat we can analyze the resulting output using the largelibrary of already developed analysis software such as YT(Turk et al. 2011), which is our analysis and visualizationplatform of choice. It also allows us to rapidly switch be-tween a MM (or SM) solver and an SPH solver with the in-clusion of a single configuration flag and use the exact sameinitialization files to facilitate easy comparison between thedifferent solvers.

That being said, there are some significant differencesbetween the amount of information being needed by the SPHsolver and needed by the MM solver. In SPH, the conservedquantities per particle are stored including mass, momen-tum (or velocity), and internal energy. Primitive quantitieslike density and pressure are derived by using a kernel tosmooth over a few neighbors. In the MM formalism, theseprimitive quantities are first-class quantities that are storedper cell. Hence, when we store outputs from MANGA we alsostore these primitive quantities as well. The structure of theChaNGa/Gasoline output data format allows the analysissoftware to deal with these quantities appropriately.

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 7

As a result, MANGA has two initialization modes. Itcan be initialized using SPH initialization files used forChaNGa/Gasoline or it can be initialized using MANGAinitialization files. In the former, we use the particle posi-tions from the SPH initialization files as our mesh generat-ing points, and we set the density using the SPH smoothingkernel on the particle data. For the other primitive vari-ables, velocity and internal energy, we use the quantities inthe SPH particle. We then update the conserved quantitiesof each cell and proceed with the solve. In the latter, whenMANGA initialization (or previous output) files are avail-able, we use the primitive quantities that are stored directlyto initialize the mesh-generating points. As a result, one canrestart from an output using the exact conditions as whenit was outputted.

As with all Eulerian or ALE schemes, the mesh gener-ating points must span all of space and the space has to bewell defined, i.e., compact with appropriate boundary con-ditions. This is unlike the case in SPH where space can beeffectively infinite. As a result the initialization files mustcontain additional points that span all of space, which isof fixed size with defined boundary conditions. Currently,we have only implemented periodic boundary conditions,though we may implement reflecting boundary conditions inthe future. Here, we must be explicit with these additionalpoints, which is in contrast to AREPO, where these pointsare generated automatically by filling in the empty leaf cellsof a Barnes-Hut tree. This precludes us from using the exactsame initial conditions that have previously been developedfor ChaNGa/Gasoline, but a translator will be developed inthe near future as described in § 6. For now, these additionalmesh generating points have to be specified explicitly.

5 TEST PROBLEMS

We validate MANGA with a few test problems and com-pare the results against those obtained from the SPH mod-ule using identical initial conditions. The Voro++ library isrestricted to 3-D, so all the tests here are carried out in 3-D.In the case where the problems are fundamentally 1-D or2-D, we use boxes with appropriate aspect ratios to emulatethis effect. In principle, it is possible to (trivially) modifiedthe construction of the Voronoi tessellation to exactly pro-duced 1-D or 2-D problems, but we have not chosen to doso. These test problems used in our validation are the Sodshock-tube problem, the Sedov-Taylor point explosion, theGresho-Chan vortex problem, the Evrard collapse problem,a star in hydrostatic balance, and a stellar merger problem.

For these test problems, we use SPH initialization con-ditions, which we generate as follows. We begin with a glassof 4,096 particles in a unit cube and generate a n3, wheren = 16, block of unit cubes each containing the glass. Wecan then give each particle a mass and rescale the size of theblock to be an appropriate representation of these initialconditions. After rescaling, we cut down the block to pro-duce cube of the appropriate size that can be fed to MANGAto solve either in SPH or MM mode. We note that our cur-rent initialization limits us to particle numbers of ≈ 16Mparticles, though this can be trivially increased by using alarger n. We note that while the Sedov-Taylor and Evrardproblem are scale-free, we have chosen to pick a specific (but

arbitrary in cgs) scale for the initial conditions. For the Sodshock-tube and Gresho-Chan vortex problems, these are innormalized units.

For the SPH solver in the test problems below, we haveset the number of neighbors to be 64 and have used thestandard M4 kernel(Monaghan 1992, 2005). Additionally, wehave used the Balsara switch to suppress the shear viscositywith default α and β values (α = 1 and β = 2) (Balsara1995).

5.1 Sod Shock-tube Test

We begin with a the simple Sod shock-tube which is ini-tialized with a periodic box (in normalized units) of dimen-sion (16, 1, 1) centered at (0,0,0) with 41K mesh generatingpoints. We initialize the γ = 1.4 gas on left side of the box(x < 0) with ρ = 1 and P = 1 and on the right side (x > 0)with ρ = 0.25 and P = 0.1795, which are the same parame-ters used for a number of other code test including Gasoline(Wadsley et al. 2004) and AREPO (S10). The gas is initiallystationary (v = 0).

In the top plots of Figure 1, we plot the density (down-sampled by 10) at t = 2 for the three solvers that we havein MANGA: the SPH solver, the MM solver, and the SMsolver using the HLLC solver and θ = 0.49. The slope lim-iter is TVD, but barely so, and thus the spurious oscillationsare evident in both the SM and MM cases. This can be reme-died in part by decreasing θ so that it become more diffusive,but we have found that using the HLL solver, which is morediffusive, eliminates these oscillations. This is shown in thebottom plots of Figure 1 where these oscillations are elimi-nated in the SM case and reduced significantly in the MMcase.

The MM and SM solvers are roughly comparable tothe SPH solver. At this high particle count, the discontinu-ities are equally well resolved by SPH, MM, and SM solvers.However, there is less variation in the per cell informationin the MM and SM case compared to SPH. This is seen inthe spread of density for the SPH solver in the regions wherethe density should be constant. This is due to the nature ofthe density estimator in SPH, which is subject to the locallyrandom distribution of particles.

5.2 Sedov-Taylor Explosion

In this test problem, we initialize a uniform periodic boxwith linear size 2×1012 cm that contains 1 M� of pure hydro-gen gas at an initially constant density, ρ = 2.5×10−4 g cm−3.In the central r < 8.7×1010 cm, we initialize the temperatureto be T = 109 K, while the remainder of the gas is initializedto be 104 K. We use a total of 255K mesh-generating pointsin this example. This results in a point explosion that is de-scribed by the Sedov-Taylor solution. In Figure 2, we showthe evolution of this explosion at times, t = 33, 133, 233, and333 minutes in the MM and SPH mode.

In Figure 3, we show the radial profile of the density as afunction of radius at the same times. For comparison, we alsocompute the same test problem with the same initial condi-tions using SPH and fixed grid mode and plot these radialprofiles. The resolution of the shock is slightly better in MMmode compared to SPH in that the peak density is reached.

MNRAS 000, 1–15 (0000)

8 Chang, Wadsley, & Quinn

8 6 4 2 0 2 4 6 8x

0.0

0.2

0.4

0.6

0.8

1.0

ρ

SPH

MM

8 6 4 2 0 2 4 6 8x

0.0

0.2

0.4

0.6

0.8

1.0

ρ

SPH

SM

8 6 4 2 0 2 4 6 8x

0.0

0.2

0.4

0.6

0.8

1.0

ρ

SPH

MM

8 6 4 2 0 2 4 6 8x

0.0

0.2

0.4

0.6

0.8

1.0

ρ

SPH

SM

Figure 1. Density as a function of position at t = 2 for the Sod shocktube test solved using SPH (blue crosses), MM (left plot, black

dots), and SM (right plots, black dots) using the HLLC solver (top plots) and the HLL solver (bottom plots).

But at the resolution in this simulation, the superior reso-lution of Godonov methods to resolve shocks compared toSPH is not evident. Visually, the resolution and evenness ofthe shock looks better in Figure 2, where particle effects inthe density estimator in YT is evident in the SPH case, butnearly absent in the MM case. Essentially, this is due themass of a mesh generating point being tied to the volumethat the Voronoi cell which it defines as oppose to the fixedmass of an SPH particle.

5.3 Gresho-Chan Vortex Problem

The Gresho-Chan vortex (Gresho & Chan 1990) is an in-viscid, cylindrical vortex. The initial state is steady, withcentrifugal acceleration balancing the pressure gradient. Thesimulation volume is periodic with unit dimensions in x andy and thinner in the z direction. While the test is technically2D, because we use glass initial conditions the ability to re-tain translation symmetry in z is a test of the method. Thedensity is uniform ρ = 1 and the pressure, P, as a function

of radius, r, is given by,

P(r) =

5 + 12.5 r2 (0 ≤ r < 0.2)9 + 12.5 r2 − 20 r + 4 ln 5r (0.2 ≤ r < 0.4)3 + 4 ln 2 (r ≥ 0.4)

(27)

with a matching tangential velocity function,

vφ(r) =

5r (0 ≤ r < 0.2)2 − 5r (0.2 ≤ r < 0.4)0 (r ≥ 0.4).

(28)

and vr = 0, vz = 0. We evolve the vortex to t = 2, or ∼ 1.6rotations of the peak. We run a simulation with ∼ 50 Kparticles in a flattened periodic box with unit length in xand y and a length of 0.2 in z. While the box is 3D, it givesan equivalent 2D resolution of 642.

In the main plot of Figure 4, we show the tangentialvelocity, vφ, as a function of radius, r, from the center of thevortex at t = 2, which corresponds to ∼ 1.6 vortex rotationat r = 0.2. We do not make a comparison with SPH as theSPH solver in ChaNGa v 3.2 suffers from large angular mo-mentum diffusion and is unable to solve this problem withany fidelity. More modern implementation especially thosewith artificial viscosity limiters are much better at integrat-

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 9

10505

10

y(R

¯)

SPH t= 33 min MM SPH t= 133 min MM

10 5 0 5 10x (R¯ )

10505

10

y(R

¯)

t= 233 min

10 5 0 5 10x (R¯ )

10 5 0 5 10x (R¯ )

t= 333 min

10 5 0 5 10x (R¯ )

10-6

10-5

10-4

Den

sity

( g cm3

)Figure 2. Slice plots of a point explosion that consists of a 1 M� box of pure hydrogen gas in a 2 × 1012 cm periodic box. We show theevolution of the explosion at times, t = 33, 133, 233, and 333 minutes using the MM and SPH solvers.

ing the Gresho vortex (Wadsley et al. 2017). In any case,MANGA does a reasonable job of replicating the analyticalsolution (eq.(28)) shown as the dashed line. In the inset plotof Figure 4, we plot the L1 error norm, L1, of the tangentialvelocity, vφ, as a function of the equivalent 2D resolution,

N2, for N = 32, 64, and 128. The L1 error norm, L1, is givenby:

L1 =1N pts

∑i

|vφ,i − vφ(ri)|, (29)

where Npts is the number of mesh-generating points, vφ,i andri are the fluid tangential velocity and distance from the ori-gin associated with that point, and vφ(r) is the analytic so-lution given by equation (28). The dotted line in the insetplot shows scaling of the L1 error norm with L1 ∝ N−0.88,which is similar to the scaling L1 error norm of the Greshovortex (∝ N−0.8) found using the modern SPH code Gaso-line2 (Wadsley et al. 2017). Convergence is worst than the

L1 scaling found in S10, but S10 ran a 2D test whereas ourtest is in 3D.

5.4 Evrard Collapse Problem

In this test problem, we initialize a m0 = 1000 M� spherewith a mass profile given by

m(r) = m0

(rr0

)2, (30)

where r0 = 1.7 × 1012 cm. This gives a ρ ∝ 1/r profile. Weembed this sphere in periodic box with linear size 2 × 1012

cm and remove the excess that extends beyond this domain.We use a total of 450K mesh-generating points in this simu-lation. The gas temperature is set to a value of T = 1000 K,which is small compared to the gravitational binding energy.As this gas evolves, it will initially fall toward the center,

MNRAS 000, 1–15 (0000)

10 Chang, Wadsley, & Quinn

0.0 0.2 0.4 0.6 0.8 1.0r/xmax

0

1

2

3

4

5

6

ρ/ρ

0

MM

SPH

Figure 3. Profile plots of a point explosion shown in Figure 2 at

t = 233 minutes for both the SPH and MM solvers. The analyticsolution is shown by the solid blue line.

0.0 0.1 0.2 0.3 0.4 0.5r

0.0

0.2

0.4

0.6

0.8

1.0

v φ

32 64 128

N

10-2

10-1

L1

N−0. 88

Figure 4. Tangential velocity, vφ as a function of radius, r , for

a Gresho vortex for a equivalent 2-d resolution of 642 at t = 2,which corresponds to 1.6 vortex rotation at the most rapid spin-

ning point at r = 0.2. For reasons of clarity the points have been

downsampled by a factor of 10. The analytic (stable) solution isshown by the dashed line. Inset plot: L1 error norm as a function

of effective 2D resolution, N2, for N = 32, 64, and 128. Also shown

is the scaling L1 ∝ N−0.88 by the dotted line.

produce a shock that runs outward before finally settling toa hydrostatic equilibrium.

We show the evolution of the density and radial velocityprofile in Figure 5 at two times: t = 42 and 125 mins. TheSPH and MM solvers produce essentially the same shockposition though it does appear that the shock in the MM isbetter resolved. This also appears to be the case in the lowresolution studies in S10.

5.5 Stellar Hydrostatic Balance

As another test of gravity, we investigate the ability ofMANGA to model a hydrostatic star for many dynamicaltimes. We create a star that is modeled as a purely ionizedhydrogen n = 3 polytrope that has a central temperature ofTc = 107 K and total mass M = 1 M�. We solve hydrostaticbalance in 1-D:

∂P∂r= −GM(< r)ρ

r2 , (31)

where M(< r) =∫ r

0 4πr ′2ρ(r ′)dr ′ is the mass containedwithin r. Here we adopt a polytrope equation of state,P = Pc(ρ/ρc)(1+n)/n, where ρc and Pc = ρckBTc/mp arethe central density and pressure respectively, and n is theindex. The resulting star has a radius of R ≈ 2 × 1011 cm.We then use these 1-D models to create initial conditions forMANGA using 105 mesh generating points to model thesestars. Surrounding the polytrope is a low density atmospherewith ρ = 10−6 g cm−3, which is low compared to the centraldensity of the polytrope (11 g cm−3). The total number ofmesh-generating points in this simulation (star and atmo-sphere) is 160K.

We evolved the initial conditions using the SPH andMM modules in MANGA. It is well known that these hy-drostatic models in 1-D are not perfectly hydrostatically bal-anced when mapped to a 3-D situation. Rather, these starstend to oscillate, and only with velocity and energy damp-ing do they reach a new hydrostatic equilibrium. We do notinclude the damping in this case and, instead, allow the starto oscillate freely. In Figure 6, we plot the evolved star (inSPH and MM) at t = 5, 120, 240, and 360 minutes after ini-tialization. Here we see that the behavior of the two stars arequantitatively the same, especially for the bulk of the mass.Only in the outer envelope is there is a departure betweenthese two solutions, with the MM algorithm being smoother.In any case, this demonstrates that the MM algorithm cansuccessfully maintain hydrostatic equilibrium and that thisequilibrium is similar to that of produced from an SPH code.The ability of MANGA to produce hydrostatic models thatagree with its SPH counterpart suggests that the MM algo-rithm can be applied to dynamical stellar problems.

5.6 Stellar Merger

As a test of MANGA to investigate dynamical stellar prob-lems, we consider the merger of a m2 = 0.5 and m1 = 1M�stars, i.e., n = 3 purely ionized hydrogen polytropes, that areinitially in contact with each other, i.e., we set their initialseparation at a = r1 + r2, where r1 and r2 are the size ofstar 1 and 2 respectively. As for the stellar hydrostatic test,we include a low density atmosphere with ρ = 10−6 g cm−3.The total number of mesh-generating points between the twostars and atmosphere is 285K. We also give them a relativevelocity of v =

√G(m1 + m2)/a so that they would be circular

orbits if they were point masses. We show their evolution at2 hour intervals in in Figure 7 up to 10 hours.

While the SPH and MM calculations look broadly thesame, they do differ in the details. For instance, the SPHmodels show much greater clumpiness in the disk that sur-rounds the star and a less well-defined (though still obvi-ous) spiral pattern. On the other hand, the MM models are

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 11

M

10−2

10−1

100

101

102

103

ρ[g

cm−

3]

t = 42 mins

MMSPH

t = 125 mins

1010 1011 1012

r [cm]

−2.5

−2.0

−1.5

−1.0

−0.5

0.0

0.5

v r[c

ms−

1]

×108

1010 1011 1012

r [cm]

MMSPH

Figure 5. Snapshots of the density and radial velocity profile for the Evrard collapse test at t = 42 and 125 minutes for both the SPH

and MM solvers. The two methods agree on the position of the shock though the MM code appears to better resolve the shock.

much smoother and show a long lasting spiral pattern. TheSPH models also show less disruption of the dense core ofthe companion, whereas it appears that the MM models ap-pear more well mixed. Both of these effects may be due toa spurious surface tension effects at large density gradientsthat are known to exist in the density formulation of SPH(Agertz et al. 2007). These effects can be remedied withinstandard SPH by adopting different formulations based onthe pressure or internal energy (Saitoh & Makino 2013; Hop-kins 2013) or the geometric density average used in more re-cent versions of ChaNGa or Gasoline2 (Wadsley et al. 2017),or alternately using a MM algorithm.

On the other hand, MANGA is inferior to the SPHsolver in conserving angular momentum. For instance, fromt = 5 to t = 240 minutes, MANGA suffers a 7% change inthe angular momentum while the SPH solver suffers onlya 0.1% change in the angular momentum. This is due tothe use of the cell’s geometric quantities following S10 atthe beginning of the time-step, i.e., face areas, which breakssecond-order time integration. As discussed below, Pakmoret al. (2016a) explicitly identified this problem as responsi-ble for the non-conservation of angular momentum in whitedwarf merger problems and pointed out means to remedyit. One of the future improvements to MANGA will be toremedy this problem as we discuss below.

6 DISCUSSION AND FUTURE DIRECTIONS

MANGA as currently implemented can be used to study

some problems of scientific interest, especially stellar merg-ers. However, ChaNGa and Gasoline have mainly been usedto study galaxy formation and cosmological structure for-mation and thus have additional physics implemented thatis currently missing from MANGA. In addition, ChaNGaand Gasoline have implemented a few algorithmic improve-ments in SPH including individual timesteps and asyn-chronous gravity and SPH calculations to improve perfor-mance. We plan to target a number of improvements in thenear, medium, and long term to include a few of these im-provements and physics to allow MANGA to be more widelyapplied. The improvements in order of priority (determinedby some combination of utility and simplicity) are:

(i) True Second Order Time-integration: MANGAas currently implemented is nearly second order in time ex-cept for accounting for the changes in the mesh geometryat the half-time-step. Instead, we follow the original imple-mentation detailed in S10 where we use the mesh geometryat the beginning of a timestep to compute the area of thefaces between adjacent cells, not accounting for the changesin the area as these cells move over a timestep. This wasidentified in Pakmor et al. (2016a) as the root cause for theviolation of the angular momentum conservation in whitedwarf merger problems. There are two ways to fix this. Onemethod is the follow the algorithm implemented in TESS(Duffell & MacFadyen 2011) and do a (re-)construction ofthe Voronoi tesselation at the half-time step. Another is thefollow the method of Pakmor et al. (2016a) who adoptedHeun’s method to calculate the flux from the average of

MNRAS 000, 1–15 (0000)

12 Chang, Wadsley, & Quinn

10-6

10-5

10-4

10-3

10-2

10-1

100

101

102

ρ [

g c

m−

3]

t= 5 mins

MM

SPH

t= 120 mins

1010 1011

r [cm]

10-6

10-5

10-4

10-3

10-2

10-1

100

101

102

ρ [

g c

m−

3]

t= 240 mins

1010 1011

r [cm]

t= 360 mins

Figure 6. Comparison of the 1-D profile of a radiative solar-type star (n = 3 polytrope) in hydrostatic balance between SPH and

MANGA for times in two hour intervals (0, 2, 4, 6 hours). Note that the bulk of the star agrees between the two methods with smalldeviations at low densities where the strong density gradients are poorly resolved (especially in SPH).

the fluxes at the beginning and end of the timestep. Thefirst method is straightforward to implement in MANGA,but comes at the cost of another mesh-generation step. Thesecond method avoids the additional construction, and itmay be as straightforward to implement (given the designof MANGA), but this requires further investigation.

(ii) Generalized Equations of State: Thus far, wehave implemented an adiabatic equation of state P ∝ ργ,where γ is the adiabatic index. We have also adapted theHelmholtz equation of state (Timmes & Swesty 2000) toMANGA, but this is still in testing. In the near future, weaim to complete its implementation and also use an amal-gamation of different equations of state suitable for studiesin stellar physics similar to what is done in stellar evolutioncodes such as MESA (Paxton et al. 2011, 2013, 2015).

(iii) Individual Time Steps: MANGA as currently im-plemented imposes one universal time step. We will lever-age the individual time stepping that was developed for theSPH module in ChaNGa to allow individual timesteps forMANGA. We will use the same strategy as discussed in S10and implement the individual timesteps in a pairwise fash-ion across each face which would allow the code to be farmore efficient. We anticipate speedups between 2-4 for stel-lar mergers and possibly 1-2 orders of magnitude of improve-ment for other problems with high dynamic range.

One issue is the determination of timesteps for each cellespecially in the presence of large velocity fluctuations.S10 proposed a scheme where the signal speed, the maxi-mum speed by which hydrodynamic information can prop-

agate, from each cell to every other cell is constructed in atree-based manner (for efficiency reasons) to calculate thetimestep. Alternatively, Hopkins (2015) proposed calculat-ing the signal speed over a local patch and incorporating theSaitoh & Makino (2009) criterion to smooth out timestep-ping changes over that local patch. It is likely that we willchoose this latter scheme to implement due to the greatersimplicity by which it can be incorporated into ChaNGa.

(iv) Cosmological Hydrodynamics: As currently im-plemented, we assume that the scale factor a is fixed atunity. Modifications for a time variable scale factor wouldallow cosmological integrations. We expect to include thisphysics as outlined by S10 in the near future.

(v) Physics for Isolated and Cosmological GalaxyFormation: To target isolated and cosmological galaxy for-mation, we plan to either port or implement some additionalphysics. This includes the Shen et al. (2010) prescription formetal line cooling, self-shielding (Pontzen et al. 2008), andturbulent diffusion of metals with a cosmic UV background(Haardt & Madau 1996, 2012) and a subgrid model for starformation, thermal energy feedback, and blastwave super-novae feedback (Stinson et al. 2006).

(vi) SPH Initial Condition Translator: As mentionedabove, we currently use SPH initial conditions to initializeMANGA. These initial conditions are almost the same asregular SPH initial conditions with the noted exception thatthe SPH particles fill all of a well-defined periodic space.However, this precludes the use of initial condition codesthat already have been developed for ChaNGa. Toward that

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 13

6420246

y(R

¯)

SPH t= 5 min MM SPH t= 120 min MM

6420246

y(R

¯)

t= 240 min t= 360 min

6 4 2 0 2 4 6x (R¯ )

6420246

y(R

¯)

t= 480 min

6 4 2 0 2 4 6x (R¯ )

6 4 2 0 2 4 6x (R¯ )

t= 600 min

6 4 2 0 2 4 6x (R¯ )

107

108

109

1010

1011

1012

Pro

ject

ed D

ensi

ty( g cm

2

)Figure 7. Snapshots of a merger between a 0.5 and 1 solar mass n = 3 polytropes (stars) using the SPH and MM solvers at 2 hourintervals up to 10 hours. Both solvers qualitatively produce similar results with a few notable differences. The SPH solver produces ejecta

that is clumpy compared to the ejecta produced by the MM solver. This is likely due to numerical surface tension that is prevalent in

SPH without the pressure fix. Additionally, the smaller, denser companion is less well mixed and retains a higher density core in theSPH solver compared to the MM solver, which is also likely due to the same effect.

end, we plan on developing a translator that will fill in theatmosphere for SPH initial conditions in an adaptive mannersimilar to what AREPO does (S10).

(vii) Magnetic Fields: S10’s algorithm for unstructuredMM hydrodynamics has been extend to magnetohydrody-namics by Pakmor et al. (2011), Mocz et al. (2014), andMocz et al. (2016). We plan on implementing a suitablescheme in the near future.

(viii) Improved Voronoi Tessellation Performance:The Voronoi tessellation library, VORO++, used inMANGA has very high performance, but its full performancecapacity is not yet utilized as our test of completeness of theVoronoi cells are still relatively primitive. Additional teststhat incorporate the position of the point relative to the dif-ferent vertices may help improve performance. Further study

on the effect of the domain decomposition on the construc-tion of the Voronoi tessellation may also yield additionalperformance.

Further down the road, we are interested in further de-velopments of the algorithm including higher-order meth-ods especially discontinuous Galerkin methods, and radia-tion (magneto-)hydrodynamics.

7 CONCLUSIONS

We describe the structure and implementation of a MM hy-drodynamics module, MANGA, in the large-scale parallelcode, ChaNGa. Our implementation largely follows that of

MNRAS 000, 1–15 (0000)

14 Chang, Wadsley, & Quinn

S10, with the notable exceptions of a direct Voronoi tes-sellation calculation, use of conserved rather than primitivevariable for the reconstruction, a modified method of cal-culating the gradients that follows (Steinberg et al. 2016),reconstruct of the facial states using the half-time-step esti-mate of the state and a slope limiter that includes the Duffell& MacFadyen (2011) slope limiter. We have tested the codeon a number of test problems and found that it producesconsistent results with its SPH cousin, though in certaincases it appear to avoid some of the unphysical propertiesof (traditional) SPH. We note, however, that more modernSPH solvers can avoid many of the pitfalls with traditionalSPH (Wadsley et al. 2017).

The advent of MM codes that began with the develop-ment of AREPO (S10) has spurred tremendous interest inALE codes in the last few years. They seem ideally suitedfor astrophysics with seemingly superior treatment of in-stabilities and interfaces (But also see Mocz et al. 2015)though recent work by Lecoanet et al. (2016) suggest thatthe grid based codes, such as ATHENA (Stone et al. 2008),are excellent at sufficient resolution in this respect as well.In addition, it can follow interesting regions efficiently overa large dynamical range. It is hoped that the development ofMANGA which is planned to be released publicly and willinclude a full suite of different physics will spur further de-velopment and application of these MM codes into differentareas of astrophysics.

ACKNOWLEDGMENTS

We thank Chris Rycroft for making VORO++ publiclyavailable and for helpful and detailed discussions. We thankBert Vandenbroucke, Rudiger Pakmor, Volker Springel, andElad Steinberg for useful discussions. PC thanks Paul Duf-fell for posting a tutorial on writing an Eulerian grid codewhich was extremely clear and helpful. We thank the anony-mous reviewer for comments and suggestions that improvedthis paper. PC is supported in part by the NASA ATPprogram through NASA grant NNX13AH43G, NSF grantAST-1255469, and the University of Wisconsin-Milwaukee.JW acknowledges Natural Sciences and Engineering Re-search Council of Canada for funding support. TRQ waspartially supported by NSF award ACI-1550234. Some ofthe computations were performed on the gpc supercomputerat the SciNet HPC Consortium (Loken et al. 2010). SciNet isfunded by: the Canada Foundation for Innovation under theauspices of Compute Canada; the Government of Ontario;Ontario Research Fund - Research Excellence; and the Uni-versity of Toronto. The authors also acknowledge the TexasAdvanced Computing Center (TACC) at The University ofTexas at Austin for providing HPC resources that have con-tributed to the research results reported within this paper.URL: http://www.tacc.utexas.edu

REFERENCES

Agertz O., et al., 2007, MNRAS, 380, 963

Balsara D. S., 1995, Journal of Computational Physics, 121, 357

Barnes J., Hut P., 1986, Nature, 324, 446

Borgers C., Peskin C. S., 1987, Journal of Computational Physics,

70, 397

Donea J., Huerta A., Ponthot J.-P., RodrA guez-Ferran A.,

2004, Arbitrary Lagrangian Eulerian Methods. John Wiley &

Sons, Ltd, doi:10.1002/0470091355.ecm009, http://dx.doi.

org/10.1002/0470091355.ecm009

Dubey A., et al., 2008, in Pogorelov N. V., Audit E., Zank G. P.,

eds, Astronomical Society of the Pacific Conference Series Vol.

385, Numerical Modeling of Space Plasma Flows. p. 145

Duffell P. C., 2016, ApJS, 226, 2

Duffell P. C., MacFadyen A. I., 2011, ApJS, 197, 15

Fryxell B., et al., 2000, ApJS, 131, 273

Gaburov E., Johansen A., Levin Y., 2012, ApJ, 758, 103

Gnedin N. Y., 1995, ApJS, 97, 231

Gresho P. M., Chan S. T., 1990, International Journal for Nu-merical Methods in Fluids, 11, 621

Haardt F., Madau P., 1996, ApJ, 461, 20

Haardt F., Madau P., 2012, ApJ, 746, 125

Hopkins P. F., 2013, MNRAS, 428, 2840

Hopkins P. F., 2015, MNRAS, 450, 53

Jetley P., Gioachin F., Mendes C., Kale L. V., Quinn T. R., 2008,

Proceedings of IEEE International Parallel and DistributedProcessing Symposium

Jetley P., Wesolowski F., Gioachin F., Kale L. V., Quinn T. R.,

2010, Proceedings of the 2010 ACM/IEEE International Con-

ference for High Performance Computin

Kale L. V., Krishnan S., 1996, in Wilson G. V., Lu P., eds, ,Parallel Programming using C++. MIT Press, pp 175–213

Kim J.-h., et al., 2014, ApJS, 210, 14

Kim J.-h., et al., 2016, ApJ, 833, 202

Lecoanet D., et al., 2016, MNRAS, 455, 4274

Loken C., et al., 2010, Journal of Physics: Conference Series, 256,

012026

Menon H., Wesolowski L., Zheng G., Jetley P., Kale L., QuinnT., Governato F., 2015, Computational Astrophysics and Cos-

mology, 2, 1

Mocz P., Vogelsberger M., Hernquist L., 2014, MNRAS, 442, 43

Mocz P., Vogelsberger M., Pakmor R., Genel S., Springel V.,

Hernquist L., 2015, MNRAS, 452, 3853

Mocz P., Pakmor R., Springel V., Vogelsberger M., Marinacci F.,

Hernquist L., 2016, MNRAS, 463, 477

Monaghan J. J., 1992, ARA&A, 30, 543

Monaghan J. J., 2005, Reports on Progress in Physics, 68, 1703

Murphy J. W., Burrows A., 2008, ApJS, 179, 209

Noh W. F., 1964, in Alder B., Fernbach S., Rotenberg M., eds,

Methods in Computational Physics. Academic Press, pp 117–

179

Ohlmann S. T., Ropke F. K., Pakmor R., Springel V., 2016, ApJ,

816, L9

Pakmor R., Bauer A., Springel V., 2011, MNRAS, 418, 1392

Pakmor R., Springel V., Bauer A., Mocz P., Munoz D. J.,Ohlmann S. T., Schaal K., Zhu C., 2016a, MNRAS, 455, 1134

Pakmor R., Pfrommer C., Simpson C. M., Kannan R., SpringelV., 2016b, MNRAS, 462, 2603

Paxton B., Bildsten L., Dotter A., Herwig F., Lesaffre P., Timmes

F., 2011, ApJS, 192, 3

Paxton B., et al., 2013, ApJS, 208, 4

Paxton B., et al., 2015, ApJS, 220, 15

Pen U.-L., 1998, ApJS, 115, 19

Pfrommer C., Pakmor R., Schaal K., Simpson C. M., Springel V.,2017, MNRAS, 465, 4500

Pontzen A., et al., 2008, MNRAS, 390, 1349

Rycroft C. H., 2009, Chaos, 19, 041111

Saitoh T. R., Makino J., 2009, ApJ, 697, L99

Saitoh T. R., Makino J., 2013, ApJ, 768, 44

Shen S., Wadsley J., Stinson G., 2010, MNRAS, 407, 1581

Springel V., 2005, MNRAS, 364, 1105

Springel V., 2010, MNRAS, 401, 791

Stadel J. G., 2001, PhD thesis, UNIVERSITY OF WASHING-

TON

MNRAS 000, 1–15 (0000)

A Moving Mesh Hydrodynamic Solver for ChaNGa 15

Steinberg E., Yalinewich A., Sari R., 2016, MNRAS, 459, 1596

Stinson G., Seth A., Katz N., Wadsley J., Governato F., Quinn

T., 2006, MNRAS, 373, 1074Stone J. M., Gardiner T. A., Teuben P., Hawley J. F., Simon

J. B., 2008, ApJS, 178, 137

Teyssier R., 2002, A&A, 385, 337Timmes F. X., Swesty F. D., 2000, ApJS, 126, 501

Toro E., 2009, Riemann Solvers and Numerical Methods for FluidDynamics: A Practical Introduction. Springer Berlin Heidel-

berg

Turk M. J., Smith B. D., Oishi J. S., Skory S., Skillman S. W.,Abel T., Norman M. L., 2011, The Astrophysical Journal Sup-

plement Series, 192, 9

Vandenbroucke B., De Rijcke S., 2016, Astronomy and Comput-ing, 16, 109

Vogelsberger M., et al., 2014, MNRAS, 444, 1518

Wadsley J. W., Stadel J., Quinn T., 2004, New Astron., 9, 137Wadsley J. W., Keller B. W., Quinn T. R., 2017, preprint,

(arXiv:1707.03824)

Whitehurst R., 1995, MNRAS, 277, 655Yalinewich A., Steinberg E., Sari R., 2015, ApJS, 216, 35

Zhu C., Pakmor R., van Kerkwijk M. H., Chang P., 2015, ApJ,806, L1

This paper has been typeset from a TEX/LATEX file prepared by

the author.

MNRAS 000, 1–15 (0000)