MULTISCALE MODELING OF HELIOSPHERIC PLASMAS

T.I. Gombosi, K.G. Powell, Q.F. Stout, E.S. Davidson, D.L. DeZeeuw,L.A. Fisk, C.P.T. Groth, T.J. Linde, H.G. Marshall, P.L. Roe, B. van Leer

The University of Michigan, Ann Arbor, MI 48109Invited paper, 1997 Simulation Multiconference, Atlanta, Georgia, April 6-10. Paper No: H-461.

Key words: Magnetohydrodynamics, Adaptive MeshRefinement, Upwind schemes, Parallel computing, Spaceplasmas.

Abstract

A new MHD code has been developed for massively par-allel computers using adaptive mesh refinement (AMR)and a new 8-wave Riemann solver. The code was im-plemented on a Cray T3D massively parallel computerwith 512 PEs. In the first application, which modeledthe expansion of the solar wind from the solar surface,the code achieved 13 GFLOPS.

INTRODUCTION

Heliospheric phenomena have been a subject of inter-est virtually since the inception of the space program.Indeed, with Pioneer and Voyager in the outer helio-sphere, Ulysses executing its polar passes, WIND in or-bit upstream from Earth, SOHO observing the Sun andthe inner heliosphere, the ISTP spacecraft studying theEarth’s magnetosphere, and Galileo at Jupiter, this is anera of remarkable richness in revealing the full range ofheliospheric phenomena. This wealth of data, however,will be of value only if we derive from these observations,and the subsequent interpretations, a clear understand-ing of the underlying and governing physical processes.

Even the best sets of observations are limited in spaceand time and they must be extended by sophisticatedtheoretical models into full 3D descriptions of heliosphe-ric plasmas. It is here that a unifying multiscale, 3Dmodel plays an essential role. The model must be suffi-ciently versatile to capture the complexity of the actualphysical system from high gradient shocks and discon-tinuities, to spatially limited transition layers, to largevolumes of slowly changing plasma flows.

Our long-term goal is to develop the first 3D, multi-scale model of the heliosphere extending from the base ofthe solar corona to the free-streaming interstellar medium.

In its fully developed form the model will self-consistentlydescribe the complicated interplay between various phys-ical processes controlling the structure and dynamics ofthe heliosphere, including the solar wind outflow, thegeneration, temporal and spatial evolution of transientinterplanetary structures (such as coronal mass ejectionsor corotating interaction regions), as well as the helio-sphere’s interactions with the interstellar medium (in-cluding plasma and neutral gas), and with magnetizedand non-magnetized solar system bodies.

Several years ago the federal goverment instituted thenational High Performance Computing and Communica-tions (HPCC) program to initiate advances in the stateof the art of modern computer simulations and in newuses of high-speed computer networks. NASA activelyparticipates in this federal program through several tech-nical projects. One of these projects is the Earth andSpace Sciences (ESS) project, which last year selectednine Grand Challenge Investigator Teams to develop ap-plications at least 10 times faster than today. Our inves-tigation, entitled “Multiscale Modeling of HeliosphericPlasmas” is one of the nine selected ESS Grand Chal-lenge projects. This paper briefly outlines the methodwe use and it summarizes our first numerical results.

GOVERNING EQUATIONS

The governing equations of ideal magnetohydrodynamicsdescribe the physics of a conducting fluid in which vis-cous and resistive effects are negligible. Under these as-sumptions, the governing equations for three-dimensionalflow are conservation laws for mass, three momentumcomponents, three magnetic field components, and en-ergy. Conservation of mass is the same for the plasma asit is for a fluid, conservation of momentum and energyhave eloctromagnetic terms that do not appear in thegoverning equation of classical fluid dynamics.

The dimensionless conservative form of the ideal MHDequations can be written in the following form:

∂ W∂t

+(∇ · F

)T= S (1)

1

where W and S are eight-dimensional state and sourcevectors, while F is an 8 × 3 dimensional flux diad. Allquantities denoted by the symbol tilde are normalizedwith the help of physical quantities in the undisturbedupstream region

t = a0t/Rs (2)r = r/Rs (3)ρ = ρ/ρ0 (4)u = u/a0 (5)p = p/ρ0a

20 (6)

B = B/√µ0ρ0a2

0 (7)

where t=time, r=radius vector, ρ=mass density, u=bulkflow velocity, p=pressure, and B=magnetic field vector.The subscript 0 refers to values at the solar surface, Rsis the radius of the Sun and a0 is the hydrodynamicsound speed at the solar surface. It follows from thisnormalization procedure that the normalized state andflux vectors are:

W =

ρρ uBε

(8)

F =

ρ u

ρ u u +(p+ 1

2 B2)

I− BB

uB− Bu

u(ε+ p+ 1

2 B2)−(B · u

)B

T

(9)

where ε is the normalized internal energy density:

ε =12

[ρ u2 +

2pγ − 1

+ B2

](10)

Equation (1) is a coupled system of eight partial dif-ferential equations — conservation laws for the mass,momentum, magnetic flux, and internal energy.

SOLUTION ALGORITHM

Equation (1) was solved on a solution-adaptive Carte-sian mesh using an upwind-differencing approach. Themethod is called Multiscale Adaptive Upwind Scheme forMagnetohydrodynamics (MAUS-MHD).

MAUS-MHD is based on two key ingradients that areextremely well suited to heliosphere modeling. The firstis a data structure that allows for adaptive refinement

of the mesh in regions of interest (De Zeeuw and Powell,1992), and the second is an upwind scheme (van Leer,1979) based on a new approximate Riemann solver forideal MHD. The details of the new numerical methodhave already been published in several papers (Gom-bosi et al., 1994; Powell, 1994; Powell, 1996; Gombosiet al., 1996) and a comprehensive description of the en-tire method is presently in preparation (Powell et al.,1997). The approach has been applied to the interac-tion of the solar wind with comets and non-magnetizedplanets, as well as strongly magnetized planets.

Adaptive refinement and coarsening of a mesh is avery attractive way to make optimal use of computa-tional resources. It becomes particularly attractive forproblems in which there are disparate spatial scales. Forproblems like the heliosphere, in which typical spatialscales can differ by orders of magnitude, an adaptivemesh is a virtual necessity. The primary difficulty in im-plementing an adaptive scheme is related to the way inwhich the solution data are stored; the data structuremust be much more flexible than the simple array-typestorage used in the majority of scientific computations.

We have decided to take the most flexible possibleAMR approach, in order to give us parameters whichcan be chosen to optimize parallel performance of thecode. To this end, we have chosen a block-based treestructure, which is a generalization of the cell-based treeused in our earlier work (De Zeeuw and Powell, 1992).The root of the block-based tree is a structured coarsegrid that covers the entire flow domain. The root blockcan have an arbitrary number of children blocks, whereeach child block is one level more refined than the parentblock (i.e. ∆xchild = ∆xparent/2, ∆ychild = ∆yparent/2,∆zchild = ∆zparent/2). This data structure devolves tothe cell-based tree in the limit of child blocks that are1 × 1 × 1. This data structure is more general, how-ever, allowing arbitrary block sizes. The advantages ofallowing arbitrary block sizes for the refinement are:

• Larger blocks have better surface-area to volumeratios, yielding lower communication overheads thansmaller blocks.

• Block sizes can be chosen to make load-balancingeasier, allowing the decomposition of the domainto be done at a higher level (i.e. portions of thegrid are farmed out to the processors on a block-by-block basis, rather than on a micro-managedcell-by-cell basis).

• Because the data for the cells inside each blockcan be stored in an array data structure, indi-

2

rect addressing is avoided for flow-solver loops, andgreater locality of data is obtained, leading to morepossibilities for efficient use of the cache on eachprocessor.

The mesh is generated in such a way that impor-tant geometric and flow features are resolved. The ge-ometry is resolved by recursively dividing blocks in thevicinity of the sun until a specified cell-size is obtained.A larger block-size is specified for the remainder of themesh. Flow features are resolved by obtaining a solutionon this original mesh, and then automatically refiningcells in which the flow gradients are appreciable, andcoarsening cells in which the flow gradients are negligi-ble.

The MAUS-MHD upwind scheme is a finite-volumemethod, in which the governing equations are integratedover each cell of the mesh. This way the governingequations given by equation (1) become a set of cou-pled ordinary differential equations in time, which can besolved by a suitable numerical integration procedure. InMAUS-MHD, an optimally smoothing multistage scheme(van Leer et al., 1989) is used. To carry out the multi-stage time-stepping procedure for (1), an evaluation ofthe flux tensor at the interfaces between cells of the mesh(a so called “Riemann solver”) is required.

Previous upwind-type schemes for MHD (Brio andWu, 1988; Zachary and Colella, 1992) have been basedon the one-dimensional Riemann problem obtained bynoting that, for one-dimensional problems, the ∇·B = 0condition reduces to the constraint that Bx = const.Methods based on this approach require a separate pro-cedure for updating the component of the the magneticfield normal to a cell face in multi-dimensional prob-lems. Typically, this requires the use of a projectionscheme which solves a global elliptic equation every fewtime-steps to project out the numerical divergence thataccumulates (Zachary and Colella, 1992; Tanaka 1993]).

In MAUS-MHD a new approach is taken (Powell1994; Powell, 1996). The governing equations for 3DMHD are derived without explicitly enforcing ∇ ·B = 0a priori, and a numerical procedure that is stable forthis system is derived. Since the finite-volume schemeis a collocated one, truncation-error levels of ∇ · B canbe generated by the scheme, and the algorithm must“clean” itself from these errors in a stable manner. Theresulting equations are those given in 1, with the sourceterm S (not truly a source term, since it is proportional

to derivatives of the magnetic field) defined by

S = −∇ · B

0Bu

B · u

(11)

This form of the equations resolves the degeneracy of thecharacteristic equation system and one can derive eightnon-degenerate eigenvectors and eigenvalues. A morecomplete derivation of the equations and the Riemannsolver are given in (Powell, 1996).

The Riemann solver based on the eight equationsgiven by (1) and (11), as shown in (Gombosi et al., 1994;1996; Powell 1994; Powell, 1996), has substantially bet-ter numerical properties than one based on the form inwhich the source term is not accounted for. Such a Rie-mann solver gives a consistent update for the compo-nent of the magnetic field normal to a cell face, suchthat the resulting numerical scheme treats (∇ ·B)/ρ asa passive scalar. Any (∇ · B)/ρ that is created numer-ically is passively convected, and, in the steady state,(∇ ·B)/ρ is constant along streamlines. with this treat-ment ∇ · B = 0 is satisfied to within truncation error,once it is imposed as an initial condition to the problem.The resulting scheme is stable to these truncation-levelerrors in the divergence of B, and does not require aprojection scheme.

Given the eigenvalues and right eigenvectors of equa-tions (1) and (11) (Powell 1994; Powell, 1996), the fluxthrough an interface is given by

Φ(WL,WR

)=

12

(ΦL + ΦR)− 12

8∑k=1

|λk|αkRk (12)

where the subscripts L and R refer to quantities at theleft and right sides of the interface, while λk and Rk arethe kth eigenvalue and right eigenvector (these can befound in (Gombosi et al., 1994; Powell, 1996)), αk is theinner product of the kth left eigenvector with the statedifference, WR−WL, while the quantities, ΦL and ΦR,are simply F(WL)·n and F(WR)·n, respectively (wheren is the normal vector of the interface).

The advantages of the flux-based approach describedabove are:

1. The flux function is based on the eigenvalues andeigenvectors of the Jacobians of F with respectto W, leading to an upwind-differencing schemewhich respects the physics of the problem beingsolved,

3

Diagonal (X=Y) slice through simulation. Each box represents a processor

with 163 cells. Normalized velocity with streamlines plotted.

U

1.25

1

0.75

0.5

0.25

0

Figure 1: Velocity magnitude (grayscale) and streamlines.

2. The scheme provides capturing of shocks and otherhigh-gradient regions without oscillations in theflow variables,

3. The scheme has just enough dissipation to providea nonoscillatory solution, and no more,

4. The scheme provides a physically consistent wayto implement boundary conditions that are stableand accurate: a physically consistent flux can becalculated at all boundaries by use of the approxi-mate Riemann solver.

PARALLEL IMPLEMENTATION

The MAUS-MHD code was implemented in FORTRAN90,using a domain-decomposition approach. Blocks of cells,stored as 3D arrays, were locally stored on each pro-cessor in such a way as to achieve a balanced load —each processor held blocks containing the same numberof cells (163 in this run). At each iteration, messagescontaining cell-center states (density, momentum, mag-netic field and energy) were passed between neighboring

processors. The message-passing was implemented us-ing the Edinburgh 32-bit MPI package. Special care hadto be taken at refinement interfaces, where a block ofcells at one level of refinement abuts a block that is onelevel finer or coarser than itself. The approach taken toassure stability and conservation at these interfaces isdescribed in (DeZeeuw and Powell, 1992; Powell, 1996).The difference in refinement level also requires that in-terpolation and restriction operators be applied beforepassing data between processors.

The first case run with the code was a simulation ofthe emanation of the solar wind from the sun. It mod-els the 3D expansion of the solar wind from 1 Rs to 16Rs. The inner boundary condition describes a station-ary sphere of hot plasma with an embedded dipole field.Due to the low interplanetary pressure, plasma expandsinto the interplanetary medium. The converged solutionshows that the solar wind originating from the polar re-gions is much faster than the solar wind originating fromthe equatorial region, due to the presence of the solarmagnetic field. This fact was observed with the Ulyssesspacecraft, but until now no numerical model was ableto reproduce this observation. Figure 1 shows the x = y

4

meridional plane of the solution. The block structure ofthe parallel implementation is shown together with themagnitude of the plasma velocity (gray scale) and thesolar wind flow lines (solid lines).

PARALLEL PERFORMANCE

The code used a total of 5 levels of refinement and 512blocks of 16x16x16=4096 cells for over 2 million totalcells. A Cray T-90 Hardware Performance Monitor(HPM) count gave 1.94 × 107 floats per processor pertimestep. A total of 4.02× 1013 floats were computed inthe simulation, which ran for 3088 seconds. The result-ing performance of the code was 13.0 GFLOPS.

ACKNOWLEDGEMENTS

This work was supported by the NASA HPCC ESS pro-ject under Cooperative Agreement Number CANNCCS5-146.

REFERENCES

Barth, T.J., On unstructured grids and solvers, in Com-putational Fluid Dynamics, von Karman Lect. Ser. 1990-03, edited by H. Deconinck, Rhode-St.-Genese, Belgium,1990.

Brio, M., and C.C. Wu, An upwind differencing schemefor the equations of ideal magnetohydrodynamics, J.Comput. Phys., 75, 400, 1988.

De Zeeuw, D., and K.G. Powell, An adaptively-refinedCartesian mesh solver for the Euler equations, J. Com-put. Phys., 104, 55, 1992.

Gombosi,T.I., K.G. Powell, and D.L. De Zeeuw, Axisym-metric modeling of cometary mass loading on an adap-tively refined grid: MHD results, J. Geophys. Res., 99,21,525, 1994.

Gombosi,T.I., D.L. De Zeeuw, R.M. Haberli, and K.G.Powell, Three-dimensional multiscale MHD model of co-metary plasma environments, J. Geophys. Res., 101,15,233, 1996.

Powell, K.G., An approximate Riemann solver for mag-netohydrodynamics (that works in more than one dimen-sion), ICASE Report No. 94-24, 1994.

Powell, K.G., Solution of the Euler and magnetohydro-dynamics equations on solution-adaptive Cartesian Grids,Von Karman Institute for Fluid Dynamics, Lecture Se-ries 1996-06, 1996.

Powell, K.G., D.L. De Zeeuw, T.I. Gombosi, and P.L.Roe, Multiscale Adaptive Upwind Scheme for Magne-tohydrodynamics: MAUS-MHD, to be submitted to J.Comput. Phys., 1997.

Tanaka, T., Configurations of the solar wind flow andmagnetic field around the planets with no magnetic field:Calculation by a new MHD simulation scheme, J. Geo-phys. Res., 98, 17,251, 1993.

van Leer, B., Towards the ultimate conservative differ-ence scheme, V. A second-order sequel to Godunov’smethod, J. Comput. Phys., 32, 101, 1979.

van Leer, B., C.H. Tai, and K.G. Powell, Design of opti-mally-smoothing multi-stage schemes for the Euler equa-tions, AIAA Paper 89-1933-CP, 1989.

Zachary, A., and P. Colella, A higher-order Godunovmethod for the equations of ideal magnetohydrodynam-ics, J. Comput. Phys., 99, 341, 1992.

BIOGRAPHIES

Tamas I. Gombosi is Professor of Space Science andProfessor of Aerospace Engineering. He has extensiveexperience in solar system astrophysics. His researchareas include generalized transport theory, modeling ofplanetary environments, heliospheric physics, and morerecently, multiscale MHD modeling of solar system plas-mas. He is Interdisciplinary Scientist of the Cassini/Huy-gens mission to Saturn and its moon, Titan. ProfessorGombosi is Senior Editor of the Journal of Geophysi-cal Research – Space Physics. He is Program Directorand Co-Principal Investigator of the NASA HPCC ESSComputational Grand Challlenge Investigator Team atthe University of Michigan.

Kenneth G. Powell is Associate Professor of Aero-space Engineering. His research areas include: solution-adaptive schemes, genuinely multidimensional schemesfor compressible flows, and, most recently, numericalmethods for magnetohydrodynamics. He was a NationalScience Foundation Presidential Young Investigator from1988-1994. He is Co-Principal Investigator of the NASAHPCC ESS Computational Grand Challlenge Investiga-tor Team at the University of Michigan.

5

Quentin F. Stout is Professor of Computer Science.His research interests are in parallel computing, espe-cially in the areas of scalable parallel algorithms and inovercoming inefficiencies caused by interactions amongcommunication, synchronization, and load imbalance.This work has been applied to a variety of industrialand scientific parallel computing problems, as well as tofundamental problems in areas such as sorting, graphtheory, and geometry. He is Co-Principal Investigator ofthe NASA HPCC ESS Computational Grand ChalllengeInvestigator Team at the University of Michigan.

Edward S. Davidson is Professor of Computer Sci-ence. He developed a variety of systematic methodsfor computer performance evaluation, including a hierar-chical simulation-based approach to cost-effective com-puter design, a lattice based evaluation methodologyfor evaluating the individual and joint impact of a setof possible architectural features, Markov and analyticmodels for realistic evaluation of memory hierarchies,and a methodology for evaluating branch and data de-pendence in instruction pipelines that achieves a sepa-ration between application-dependent and architecture-dependent analysis.

Daren L. DeZeeuw is Assistant Research Scientistat the Space Physics Research Laboratory. His inter-ests involve the development and implementation of al-gorithms to solve the multidimensional Euler and MHDequations using octree-based unstructured Cartesiangrids with adaptive refinement and multigrid conver-gence acceleration. He is one of the primary developersof the code described in this paper.

Lennard A. Fisk is Professor and Chair, Depart-ment of Atmospheric, Oceanic and Space Sciences. Hisarea of expertize is heliospheric physics and the prop-agation of energetic particles in the solar system. Be-tween 1987 and 1993 he served as NASA’s Associate Ad-ministrator for Space Science and Applications. In thisposition he was responsible for the planning and direc-tion of all NASA programs concerned with space scienceand for the institutional management of the GoddardSpace Flight Center in Greenbelt, Maryland and the JetPropulsion Laboratory in Pasadena, California.

Clinton P.T. Groth is Assistant Research Scien-tist at the Space Physics Research Laboratory. His re-search interests involve generalized transport theory andthe development of higher-order moment closures for thesolution of the Boltzmann equation via advanced numer-ical methods with applications to both rarefied gaseousand anisotropic plasma flows. This research has led to a

new hierarchy of moment closures with many desirablemathematical features that appears to offer improvedmodeling of transitional rarefied flows.

Timur J. Linde is graduate student in the Depart-ment of Aerospace Engineering. He is developing a newthree-dimensional model of the interaction of the solarwind wind with the local interstellar medium using anoctree-based MHD code with adaptive mesh refinement.

Hal G. Marshall is Associate Research Scientist atthe Center for Parallel Computing. His interests focus onscientific computing using massively parallel computers.He is one of the primary developers of the code describedin this paper.

Philip L. Roe is Professor of Aerospace Engineer-ing. In 1976, he was asked to undertake research intonumerical methods for solving the compressible flow (Eu-ler) equations, with instructions to “find something new.”The outcome of this work became known as the “Roesolver”, a formula for the interface flux in a finite-volumecode, that is now a standard feature in numerous com-mercial and industrial codes. Later, he initiated an effortto link the algorithms for multidimensional partial dif-ferential equations even more closely with the appropri-ate physics, a project that has attracted an internationalteam of collaborators. This effort is now essentially com-plete for two-dimensional gasdynamics. Not only are theresults superior to those of the earlier approach, but thealgorithm, as a byproduct, minimizes message passing.Most of the tools are in place to make the extensions tothree dimensions and more complex equations.

Bram van Leer is Professor of Aerospace Engineer-ing. In the 1970s he became dissatisfied with the avail-able finite-difference methods for shocked flows, whichwere either too diffusive or oscillatory, and designed agrand plan to bring CFD to a uniformly higher level.This he did singlehandedly at Leiden Observatory duringthe years 1972-77, as reported in the 5-part article series“Towards the ultimate conservative differences scheme.”Starting from Godunov’s method (1959) he formulatedhigher-order, yet non-oscillatory, upwind-differencingmethods for the Euler equations; such methods becamethe industry’s standard in the 1980’s and are calledMUSCL-type, after the first code of its kind. His cur-rent research interest lies in convergence accelerationof steady-flow computations, in particular, the devel-opment of truly effective multigrid methods for high-Reynolds-number flows, using local preconditioning toremove the stiffness of the governing equations.

6