High Fidelity, Moment-Based Methods for Particle Transport:The confluence of PDEs, Optimization, and HPC

Ryan McClarrenTexas A&M University

Dept. of Nuclear Engineering

Workshop on Numerical PDEs Monash University, 15-19 February 2016

17 Feb 2016

Acknowledgements

I Collaborators:

I Vincent Laboure and Weixiong Zheng (TAMU PhD. Students)

I Cory Hauck (Oak Ridge National Lab)

I Martin Frank (RWTH Aachen)

I Paul Laiu, Andre Tits, and Dianne O’Leary (U. Maryland)

I Sponsors:

I US National Science Foundation, Division of Mathematical Sciences

The equations that describe particle transport start off as anintegro-differential equation

I We are interested in the phase-space density of particles, N, that travel instraight-lines between collisions. The equation that describes this is thelinear-Boltzmann equation:

(∂t + vΩ · ∇+ vσt(x , t))N(x ,Ω, v , t) =∫S2

dΩ′∫ ∞

0

dv ′ v ′σs(x , t,Ω′ → Ω, v ′ → v)N(x ,Ω, v , t) + Q(x ,Ω, v , t)

I Ω ∈ S2 is the direction of the particle’s flight (angular variable), v is theparticle speed.

I The interaction probabilities (cross-sections) are the total cross-section σt

which is the average number of collisions a particle undergoes with thematerial medium per unit distance travelled, and

I The double-differential scattering cross-section, σs(x , t,Ω′ → Ω, v ′ → v)

is the mean number of particles that scatter to direction Ω and speed vper particle traveling in the differential phase space element.

Applications

This equation very accurately describes the behavior of a variety of transportprocesses

I Neutrons in a nuclear reactor, oil well, imaging

I X-rays in high energy density situations: inertial confinement fusion,astrophysical radiating shocks

I Atmospheric radiative transfer

I Neutrinos in core-collapse supernovae

I Electron/ion transport in radiotherapy, space weather, electronics

Radical Simplifications

I For this talk we will make the assumption that the discretization in speed(energy) is a solved problem, and we only need to consider a single speedequation.

I Additionally, we will assume that the scattering is isotropic.

I Both of these are simplifications for real systems.

Fission Source

Resonance

Thermal

0.0001

0.01

1

100

0.01 meV 1 meV 1 eV 1 keV 1 MeV 10 MeVIncident Neutron Energy (eV)

Mic

rosc

opic

Rad

iativ

e C

aptu

re C

ross

-Sec

tion

(bar

ns)

Uranium-238 Radiative Capture Cross-Section

Simplified Equations

I After these simplifications we can write the resulting equation as(v−1∂t + Ω · ∇+ σt(x , t)

)ψ(x ,Ω, t) =

σs(x , t)

4πφ(x , t) + Q(x ,Ω, t),

where ψ = vN and

φ(x , t) =

∫S2

dΩψ(x ,Ω, t) = 〈ψ〉.

I We will also assume that v = 1. This is the same as scaling the timevariable.

I Initial condition: ψ(x ,Ω, 0) = F (x ,Ω), and boundary conditions are inflowconditions:

ψ(x ,Ω, t) = Γ(x ,Ω, t) for x ∈ ∂V , n · Ω < 0.

Numerical Challenges

I Phase space complexityI Need for thousands of unknowns per spatial degree of freedom

I Multiscale phenomenonI In problems where the scattering is large, the transport equation

asymptotically limits to a diffusion equation for the particlesI Need numerical methods that preserve this fact when the mesh does not

resolve the collision length scales.

I Coupling to other physics (fluid flow, etc.)

Discrete Ordinates (Sn) method

I The discrete ordinates method is a collocation method in angle that solvesthe transport equation along a particular directions (Ωj) and uses aquadrature rule, wj ,Ωj to estimate the collision terms. (Chandrasekhar)

I Leads to a simple, triangular system of discrete equations for eachdirection when the backward Euler method is used in time and a simpleiteration strategy is used

(Ωj · ∇+ σ∗t )ψ`+1j (x , tn+1) =

σs(x , t)

4π

∑j′

wj′ψ`j′(x , t

n+1) + Q∗j ,

σ∗t = σt + ∆t−1 and Q∗j = Q + ψj(x , tn).

I As a result when, σs/σt is small this iteration convergences quickly,otherwise need to include the solution of a diffusion equation in theiteration.

I This is the best understood method for deterministic particle transport.

Monte Carlo

I Rather than discretize phase space directly we sample particles and advectthem based on stochastic collision processes.

I Can be very accurate and operate on general domains in space and energy.

I Slow convergence N−1/2 typically limits applicability.

I For steady-state problems it is considered the gold standard, if you canafford the simulation.

Spherical Harmonic Functions

I Decompose the angle Ω into components

Ω = (Ω1,Ω2,Ω3)T = (sinϑ cos(ϕ), sinϑ sin(ϕ), cosϑ)T

I The normalized, complex spherical harmonic of degree ` and order k are

Y k` (Ω) =

√2`+ 1

4π

(`− k)!

(`+ k)!e ikϕPk

` (cosϑ) ,

where Pk` is an associated Legendre function.

I For convenience, we use normalized, real-valued spherical harmonics mk`

and for each degree `. For given N > 0, set

m` = (m−`` ,m−`+1` , . . . , ,m`−1

` ,m``)

T and m = (mT0 ,m

T1 , . . . ,m

TN )T

I The components of m form an orthonormal basis for the polynomial space

PN =

N∑`=0

∑k=−`

ck`mk` : ck` ∈ R for 0 ≤ ` ≤ N, |k| ≤ `

. (1)

Spherical Harmonics (PN ) Equations

I Spectral approximation in Ω

ψ ≈ ψPN ≡ mTuPN

where uPN = uPN (t, x) solves the PN equations∂tuPN + A · ∇xuPN + σauPN + σsGuPN = s , (t, x) ∈ (0,∞)× R3

uPN (0, x) = 〈mψ0(x , ·)〉 , x ∈ R3

withI s := 〈mS〉I A · ∇x ≡

∑3i=1 Ai∂xi and each Ai = 〈ΩimmT 〉 is symmetric

I G ≥ 0 is diagonal

I Angle brackets denote integration over S2: 〈·〉 :=∫S2 (·)dΩ

Properties of the PN Equations

I Good Stuff

I Fast convergence for smooth solutions

I Preserve rotational invariance of the transport operator

I Harmonics are eigenfunctions of the scattering operator

I Bad Stuff

I Gibbs phenomena near wave fronts

I Negative values for the concentration 〈ψ〉 in multi-D

I May be ill-posed in steady-state (Ai can have zero eigenvalues)

I Challenging boundary conditions

The Line Source Problem: All Methods have issues1

(a) analytic (b) Monte-Carlo

(c) S6 (d) P1 (e) P5

1T. A. Brunner. “Forms of Approximate Radiation Transport”, Tech. Rep. SAND2002-1778

Sandia National Laboratories, Jul 2002.

The issue is the closure

I The standard PN closure simply truncates the expansion for l > N.

I The Gibbs oscillations are a result.

I The negative densities are problematic for coupled simulations: what doesa negative absorption rate density mean?

I Other methods have been proposed to alleviate this issueI The MN methods use the ansatz

ψ ≈ epT c

to close the system.I Solve an optimization problem to assure that the ansatz is positive.

I Idea: Apply filters to the expansion to damp oscillations.

Filter functions

Filtering is commonly used to handle spatial gradients in linear and nonlinearadvection. We use it here for the angular approximation.

DefinitionA filter of order α is a real-valued function f ∈ Cα(R+) that satisfies

(i) f (0) = 1 , (ii) f (a)(0) = 0, for a = 1, . . . , α− 1 , (iii) f (α)(0) 6= 0.

I Several variations in the definition, but (i) and (ii) are standard.

I Define f`,N := f ( `N+1

).

I To implement, apply the filter uPN → uFPN where

[uFPN ]` = f`,N [uPN ]`

after each step of a time integration routine.2

2RGM, C.D Hauck, J. Comput. Phys., 229 (2010), pp. 5597–5614

Back to the Linesource

1.5 1.0 0.5 0.0 0.5 1.0 1.51.5

1.0

0.5

0.0

0.5

1.0

1.5

1.8

1.2

0.6

0.0

0.6

1.2

1.8

2.4

3.0

(a) P11

0.0 0.2 0.4 0.6 0.8 1.00.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7x axisExact Solution

(b) P11-Lineout

1.5 1.0 0.5 0.0 0.5 1.0 1.51.5

1.0

0.5

0.0

0.5

1.0

1.5

0.00

0.06

0.12

0.18

0.24

0.30

0.36

0.42

0.48

0.54

0.60

(c) FP11

0.0 0.2 0.4 0.6 0.8 1.00.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7x axisExact Solution

(d) FP11-Lineout

Filtered Spherical Harmonic Equations3

I The filtering procedure can be generalized to look like an anisotropicscattering operator.

I Let ψ ≈ ψFPN ≡ mTuPN , where uFPN satisfies

∂tuFPN + A · ∇xuFPN + σauFPN + σsGuFPN + σfGfuFPN = s

I The matrix Gf ≥ 0 is diagonal with components

(Gf)(`,k),(`,k) = − log f`,N

It can be interpreted as anisotropic scattering or angular diffusion.

I The issue of choosing the filter strength remains.

3D. Radice, E. Abdikamalov, L. Rezzola, and C. Ott, A new spherical harmonics scheme for multi-dimensionalradiation transport I: static matter configurations, J. Comput. Phys., 242 (2013), pp. 648–669

Convergence of the Filtered Expansion

I Frank, Hauck and Kuepper give theorems for the convergence of filteredPN .

I The error in the expansion

EN = ‖ψFPN − ψ‖L2

converges at a rate that is the smaller of the order of convergence of theunfiltered method, k, or the filter order, α:

EN = O(N−min(k,α)).

Test problem: Crooked Pipe

I This is a standard high energy density radiative transfer test problem.

I Blue region is optically thin (little interaction between radiation andmaterial)

I Red regions are optically thick (strong collisions between radiation andmaterial)

I Radiation source at left entrance.

Unfiltered calculation

Material temperature (in keV) att = 3.5 ns (from top to bottom)left: P1, P3, P5

right: P7, P39

The white regions show where the radiation density is negative.

Uniformly filtered calculation

Material temperature (in keV) att = 3.5 ns (from top to bottom)left: P1, P3, P5

right: P7, P39

Filter strength value

With the Lanczos filter and σf = 5000 m−1:

σf f (` = 1,N0 = 7) ≈ 13 m−1 ∼ σt = 20 m−1 (2)

Choosing the location of the filter

Figure: Material temperature T (in keV) at t = 3.5 ns for unfiltered P3.

Choosing the location of the filter

I σf on the order of the cross-section (same units)

I activated where negativity arises and in an upstream region of acomparable size.

Local filter

Figure: Value of σf (in cm−1) for the locally filtered calculations.

Locally filtered calculation

Material temperature (in keV) att = 3.5 ns (from top to bottom)left: P1, P3, P5

right: P7, P39

T along y = 0 at t = 3.5 ns

T along x = 2.75 cm at t = 3.5 ns

0 0.5 1 1.5 2

y (cm)

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Mate

rial te

mpera

ture

(keV

)P1

P3

P5

P7

P39

0 0.5 1 1.5 2

y (cm)

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Mate

rial te

mpera

ture

(keV

)

P1

P3

P5

P7

P39

0 0.5 1 1.5 2

y (cm)

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Mate

rial te

mpera

ture

(keV

)

P1

P3

P5

P7

P39

Figure: Clockwise from top left: Unfiltered, Uniformly filtered, L2 error, LocallyFiltered

Computational Cost

I The solution for a given time step involves the solution of a linear systemof size Nx × NΩ.

I We have not been able to develop a good preconditioner yet.I The system has a non-trivial nullspace when there are no collisions.

I Increasing the filter strength does improve the convergence.

I Number of GMRES iterations for first timestep2 Results

2.1 Empirical values

100

102

104

106

108

1010

0

200

400

600

800

1000

1200

1400

P1

P3

P5

P7

P9

P11

P13

P15

P17

(a) Uniform filtering

100

102

104

106

108

1010

0

200

400

600

800

1000

1200

1400

P1

P3

P5

P7

P9

P11

P13

P15

P17

(b) Local filtering

Figure 1: Iteration count for the first time step as a function of N and the filter strength f (in cm1), usingthe Lanczos filter. As a reference, the value of f for this test problem was in practice chosen to be 50 cm1

(vertical line).

f (cm1) P1 P3 P5 P7 P9 P11 P13 P15 P17

0 520 837 962 1041 1095 1201 1288 1338 13850.1 518 832 956 1031 1088 1191 1279 1327 13770.5 512 809 932 1001 1061 1157 1241 1288 13301 505 785 903 967 1034 1123 1198 1243 12885 457 663 760 825 873 936 1002 1038 107210 406 564 655 731 780 825 877 902 93750 246 367 432 482 528 564 606 630 663100 190 291 356 411 451 493 530 559 598500 130 192 238 277 321 352 387 414 4381000 120 172 206 236 264 292 324 348 37010000 108 135 147 160 169 180 186 197 205105 91 112 119 120 127 134 140 145 147106 62 79 92 101 108 112 114 115 118107 53 56 61 67 73 78 83 87 92108 53 53 53 54 54 56 57 59 61109 53 53 53 53 53 53 53 53 531010 53 53 53 53 53 53 53 53 53

Table 1: Number of linear iterations for the first time step and uniform filtering. The practical value forthe Crooked Pipe is f = 50 cm1.

2

Enforcing Positivity

I For nonlinear problems, positivity may be a strict requirement.

I Our goal is to modify the FPN equations to enforce positivity.

Positive, Filtered Spherical Harmonics (FP+N )

We implement the filter in the context of a kinetic scheme:

1. Given a kinetic distribution ψ, compute

EFPN[ψ] = mTuFPN

[ψ] = mTFuPN[ψ]

where F is a filtering matrix and uPN[ψ] = 〈mψ〉.[4]

2. Find uFP+N

[ψ], which solves

minimizeu∈Rn

1

2

∫S2

∣∣∣EFPN[ψ]−mTu

∣∣∣2 dΩ

subject to

∫S2

mTu dΩ =

∫S2

mTuFPNdΩ

(mTu)(Ωq) ≥ 0 , ∀ Ωq ∈ Q

where Q is a quadrature set.

3. Advance the kinetic equation with initial condition

EFP+N

[ψ] = mTuFP+N

[ψ]

4〈·〉 is integration over [−1, 1] or S2.

Convergence results

Convergence of the Positive Filtered Expansion

I The error in the expansion

EN = ‖ψFP+N− ψ‖L2

converges at a rate that is the smaller of the order of convergence of theunfiltered method, k, or the filter order, α:

EN = O(N−min(k,α)).

I Same as the convergence for standard FPN .

moment order N

101

102

‖f−

E‖L2

10-10

100

No filter

PN : Spectral

UDN : Spectral

P+N : Spectral

moment order N

101

102

‖f−

E‖L2

10-4

10-2

100

102

Lanczos (2nd order)

FPN : 1.99

UDN : 1.99

FP+N : 1.99

moment order N

101

102

‖f−

E‖L2

10-4

10-2

100

102

Spline (4th order)

FPN : 3.98

UDN : 3.98

FP+N : 3.98

moment order N

101

102

‖f−

E‖L2

10-4

10-2

100

102

Exponential (6th order)

FPN : 5.96

UDN : 5.96

FP+N : 5.96

Figure: Smooth function on [−1, 1] (f ∈ C∞([−1, 1]).

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

No filter

PN : 0.49

UDN : 0.08

P+N : 0.51

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Lanczos (2nd order)

FPN : 0.49

UDN : 0.09

FP+N : 0.51

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Spline (4th order)

FPN : 0.49

UDN : 0.07

FP+N : 0.50

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Exponential (6th order)

FPN : 0.49

UDN : 0.10

FP+N : 0.49

Figure: Step function on [−1, 1] (f ∈ Hq([−1, 1]), ∀q < 0.5).

moment order N

101

102

‖f−

E‖L2

10-10

10-5

100

No filter

PN : 3.49

Z-SN : 3.09

P+N : 3.52

moment order N

101

102

‖f−

E‖L2

10-8

10-6

10-4

10-2

Lanczos (2nd order)

FPN : 1.99

Z-SN : 2.03

FP+N : 1.99

moment order N

101

102

‖f−

E‖L2

10-10

10-5

100

Spline (4th order)

FPN : 3.47

Z-SN : 3.02

FP+N : 3.53

moment order N

101

102

‖f−

E‖L2

10-10

10-5

100

Exponential (6th order)

FPN : 3.47

Z-SN : 3.04

FP+N : 3.42

Figure: Sobolev function on [−1, 1] (f ∈ Hq([−1, 1]), ∀q < 3.5.

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

No filter

PN : 0.53

UDN : -0.04

P+N : 0.51

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Lanczos (2nd order)

FPN : 0.52

UDN : -0.02

FP+N : 0.51

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Spline (4th order)

FPN : 0.52

UDN : -0.05

FP+N : 0.51

moment order N

101

102

‖f−

E‖L2

10-2

10-1

100

Exponential (6th order)

FPN : 0.44

UDN : 0.05

FP+N : 0.45

Figure: Singular function on [−1, 1] (f ∈ L2([−1, 1])).

Linesource results

−1.5 −1 −0.5 0 0.5 1 1.5

−1.5

−1

−0.5

0

0.5

1

1.5 −0.1

0

0.1

0.2

0.3

0.4

0.5

Figure: Exact solution

-1.5 -1 -0.5 0 0.5 1 1.5

-1.5

-1

-0.5

0

0.5

1

1.5 -0.1

0

0.1

0.2

0.3

0.4

0.5

Figure: P11

-1.5 -1 -0.5 0 0.5 1 1.5

-1.5

-1

-0.5

0

0.5

1

1.5 -0.1

0

0.1

0.2

0.3

0.4

0.5

Figure: FP11

-1.5 -1 -0.5 0 0.5 1 1.5

-1.5

-1

-0.5

0

0.5

1

1.5 -0.1

0

0.1

0.2

0.3

0.4

0.5

Figure: UD11

-1.5 -1 -0.5 0 0.5 1 1.5

-1.5

-1

-0.5

0

0.5

1

1.5 -0.1

0

0.1

0.2

0.3

0.4

0.5

Figure: FP+11

0 0.2 0.4 0.6 0.8 1 1.2−0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure: Exact solution

0 0.2 0.4 0.6 0.8 1 1.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure: P11

0 0.2 0.4 0.6 0.8 1 1.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure: FP11

0 0.2 0.4 0.6 0.8 1 1.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure: UD11

0 0.2 0.4 0.6 0.8 1 1.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Figure: FP+11

Line Source EfficiencyWhich is more efficient: a better, more expensive limiter or a cheaper limiterwith more moments? The optimization is completely local, though so it shouldscale.

Computation Time (s)10

010

110

210

310

410

5

SpatialL2errorE

FPN,E

Z−SN

10-5

10-4

10-3

3

5

7

9

11

13

1517

19

3

5

79

11

13

1517

19

25

31

41

49

59

69

UDN

FP+N

Figure: Serial Efficiency Comparison, based on L2 error inconcentration.

Conclusions

I Problems of particle transport are hard and there is no perfect method.

I Moment-based methods have some positive properties, but the alsodrawbacks.

I Spectral Convergence is possible, but leads to issues with positivity andoscillations.

I Filtering and optimization-based closures are promising, but still work todo.