SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 1Vay - APS-DPP 2011

Novel Simulation Methods in the Particle-In-Cell Framework Warp

J.-L. Vay*, C.G.R. GeddesLawrence Berkeley National Laboratory, CA, USA

D.P. Grote, A. FriedmanLawrence Livermore National Laboratory, CA, USA

53rd Annual Meeting of the APS Division of Plasma PhysicsSalt Lake City, Utah, USA – November 14-18, 2011

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory*jlvay@lbl.gov

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

Markers

2Vay - APS-DPP 2011

Warp is a versatile parallel 3D Particle-In-Cell framework developed by the Heavy Ion Fusion Science Virtual National Laboratory

Standard PIC

Non-standard PIC

Laboratory frame Moving window Lorentz Boosted frameExamples: Beam generation

Neutralization in plasma

Example: Beam transport

Example: Laser plasma acceleration

Steady flow

Example: Fast injector design

Quasi-static

Example: electron cloud studies2-D slab of electrons

3-D beam

s

SPS - CERN

p+ bunches

e- clouds

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

But short wavelength instability observed at front of plasma for large (≥g 100)

3Vay - APS-DPP 2011

Conjectured that instability related to numerical dispersion, i.e. a kind of numerical Cerenkov.

Warp 2D simulation 10 GeV LPA (ne=1017cc, =130)

Longitudinal electric fieldlaser

plasma

Modeling of 10 GeV laser plasma accelerator stage is challenging

1 Vay, PRL 2007

llaser ≈ lwake ≈ Lacceleratellaser << lwake << Laccelerate

Predicted speedup: >10,000 for 10 GeV stage; > 1,000,000 for 1 TeV stage.

Boosted frame g = gwakeLab frameCalculation in boosted frame at ≈wake minimizes scale differences1

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

An electromagnetic solver based on Non-Standard Finite-Difference (NSFD) was implemented in Warp

NSFD: weighted average of quantities transverse to FD ().

NSFD=FD if =0

Yee Cole-Karkkainen (CK)

Yee/CK allows for perfect dispersion along 3D/principal axes.

x=y=z)

Cole1 and Karkkainen2 have applied NSFD to source free Maxwell equations

Warp3: switched FD/NSFD to B/E.

=> FD on source terms, i.e. standard exact current deposition schemes still valid.

*

FD

NSFD

a

b

bb

b

g

g

g

g-a

-b

-b -

b

-b

-g

-g

-g

-g

Dx

NSFD

FD

FD

NSFD

3J.-L. Vay, et al., J. Comput. Phys. 230 (2011) 5908.

x=y=z)

Vay - APS-DPP 2011 4

1J. B. Cole, IEEE Trans. Microw. Theory Tech. 45 (1997), J. B. Cole, IEEE Trans. Antennas Prop. 50 (2002).2M. Karkkainen et al., Proc. ICAP, Chamonix, France (2006).

NSFD offers tunability of numerical dispersion

perfect dispersion 2D diagonal isotropic

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

Example: Reflection of circular pulse

using 5 cells PML

with quadratic progression

and standard coefficients

or improved coefficients2

5

Perfectly Matched Layer1,2 (PML) implemented with NSFD solver - for absorption of outgoing waves -

Same high efficiency as with Yee.1JP Berenger, J. Comput. Phys. 127 (1996) 3632J.-L. Vay, J. Comput. Phys. 183 (2002) 367

NSFDNSFD

FDFD

Vay - APS-DPP 2011

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

After testing: instability mostly insensitive to numerical dispersion… …but very sensitive to time step!

6Vay - APS-DPP 2011

Sharp decrease of instability level at ct=z/√2

Tunable NSFD solver allows ct=z/√2 time step for (near) cubic cells• ct=z/√2 time step restricted to “pancake” cells in 3D using Yee FDTD solver

Use of special time step was helpful but not sufficient for large g boost

Pow

er s

pect

rum

(a.

u.)

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 7Vay - APS-DPP 2011

Digital filtering of current density and/or fields -- commonly used for improving stability and accuracy

Multiple pass of bilinear filter + compensation routinely used

100% absorption at Nyquist freq.Bilinear (B)Bilinear (B) + compensation (C)

1/2 1/41/4

Bilinearfilter

Wideband filtering difficult in parallel (footprint limited by size of local domains) or expensive

Example: wideband filters using N repetitions of bilinear filter

1×B + C

4×B + C

20×B + C

50×B + C

80×B + C

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 8

“Strided” bilinear filters enable efficient and versatile filtering1

4×BC stride 1 (G1)4×BC stride 2 (G2)4×BC stride 3 (G3)4×BC stride 4 (G4)

Bilinear filterwith stride 2

1/21/4 1/4

Using a stride N shifts the 100% absorption frequency to Fnyquist/N

Combination of filters with strides allows for more efficient filtering:• G1G2 20*B+C; speedup ×2• G1G2G3 50*B+C; speedup ×3.5• G1G2G4 80*B+C; speedup ×5.5

G1 × G2G1 × G2 × G3G1 × G2 × G420×B+C50×B+C80×B+C

Vay - APS-DPP 2011

1J.-L. Vay, et al., J. Comput. Phys. 230 (2011) 5908.

Nice, but is wideband filtering possible without altering the physics?

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 9

Tim

eT

ime

Laser field

Laser field

Lab frame

Wake frame

Hyperbolic rotation from Lorentz Transformation converts laser…

…spatial oscillations into

time beating

Vay - APS-DPP 2011

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 10Vay - APS-DPP 2011

Lab frame Frame of wake (=130)

spectrum spectrum

Spectrum very different in boosted and lab frames

Time history of laser spectrum (relative to laser l0 in vacuum)

Dephasing time

Content concentrated around l0

0 0

Content concentrated at much larger l

More filtering possible without altering physics*.*J.-L. Vay, et al., PoP Lett. 18 (2011).

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 11Vay - APS-DPP 2011

Controlling the numerical instability with tunable EM solver & filtering

Laser injection

Particle injection

Diagnostics

led to over 1 million x speedup2

+ new laser/particle injection and diagnostics through planes1:

2J.-L. Vay, et al., PoP Lett. 18 (2011) & PoP (in press).1J.-L. Vay, et al., J. Comput. Phys. 230 (2011).

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 12Vay – APS-DPP 2011

A. Friedman, D. P. Grote, and I. Haber, Phys. Fluids B 4, 2203 (1992) – code description, warped coordinates

J.-L. Vay et al., Phys. Plasmas 11 (2004) – mesh refinement

D.P. Grote et al., AIP Conf. Proc. 749, 55 (2005) – updated Warp description

R. Cohen et. al., Phys. Plasmas 12, 056708 (2005) – “Drift-Lorentz” particle pusher

J.-L. Vay, Phys. Plasmas 15, 056701 (2008) – ultra-relativistic pusher

R. Cohen et al. Nucl. Instr. & Methods 608, 53 (2009) – direct implicit Drift-Lorentz

For questions on Warp, email to

• DPGrote@lbl.gov

• Afriedman@lbl.gov

• JLVay@lbl.gov

Warp contains a lot more not described here, see

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory

BACKUP

13Vay - APS-DPP 2011

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National Laboratory 14

as well as Friedman damping algorithm- for noise control -

B push modified to

with where is damping parameter.

Yee-Friedman (YF) Cole-Karkkainen-Friedman (CKF)

Dispersion degrades with higher values of

Damping more potent on axis and more isotropic for CKF than YF.

Vay - APS-DPP 2011

SciDAC-IICompass

The Heavy Ion Fusion Science Virtual National LaboratoryVay - APS-DPP 2011

BELLA Project: state-of-the-art PWfacility for laser accelerator science

BELLA Laser

Control RoomGowning Room

Final focus< 100 cm

e- beam

~10 GeV Laser

Plasma

15