Plasma Simulation with ParallelKinetic Particle Codes

P. Gibbon, R. Speck, B. Berberich, A. Karmakar,L. Arnold, M. Masek

published in

NIC Symposium 2010G. Munster, D. Wolf, M. Kremer (Editors)

Forschungszentrum Julich GmbH,Julich Supercomputing Centre (JSC),John von Neumann Institute for Computing (NIC),Schriften des Forschungszentrums Julich, IAS Series, Vol. 3,ISBN 978-3-89336-606-4, pp. 383-390.

c© 2010 by Forschungszentrum JulichPermission to make digital or hard copies of portions of this work forpersonal or classroom use is granted provided that the copies are notmade or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwiserequires prior specific permission by the publisher mentioned above.

http://hdl.handle.net/2128/3697

Plasma Simulation with Parallel Kinetic Particle Codes

Paul Gibbon1, Robert Speck1, Benjamin Berberich1, Anupam Karmakar 1,2,Lukas Arnold 1, and Martin Ma sek3

1 Institute for Advanced Simulation, Julich Supercomputing CentreForschungszentrum Julich, 52425 Julich, Germany

E-mail: {p.gibbon, r.speck, b.berberich, a.karmakar, l.arnold}@fz-juelich.de

2 ExtreMe Matter Institute EMMI, GSI Helmholtzzentrum fur SchwerionenforschungPlanckstraße 1, 64291 Darmstadt, Germany

3 Institute of Physics, Academy of Sciences of the Czech Republic, Prague, Czech RepublicE-mail: masekm@fzu.cz

Recent work on plasma simulation with kinetic particle codes at JSC is described. Specialattention is given to developments in mesh-free plasma modelling using parallel tree codes,covering algorithmic and performance issues, simulationsof ion acceleration by high intensitylasers, and new applications of this technique to vortex-fluids and edge-plasma modelling intokamaks.

1 Introduction

Kinetic simulation methods are an indispensible tool for plasma modelling over largeranges of density and temperature. These techniques, whichtreat the plasma as a 6-dimensional (3 space, 3 momenta) phase-fluid, are more complex and expensive than hy-drodynamic or magnetohydrodynamic models. In the project JZAM04, several comple-mentary kinetic codes are being applied to model a variety ofplasma physical problems.These include BOPS, a versatile, 1-dimensional particle-in-cell (PIC) code capable of mod-elling laser-solid interaction experiments; PSC, an open-source 3D PIC code from LMUMunich; ILLUMINATION, a similar 3D code (with different wave-solvers) developed byQueen’s University Belfast; and PEPC, a parallel, mesh-free tree code developed by theauthors. The development and performance optimisation work with these codes forms amajor activity of the recently established Plasma Physics Simulation Laboratory at JSC.

In the present article we review recent applications and development work involvingthese codes, which have played a central role in two separate3rd-party projects, and alsoform the basis of several Diploma and Ph. D. theses.

2 Hard x-Ray Sources with Thin Foils

Recent experiments by the U. Duisburg-Essen group to optimize hard x-ray line emissionfrom thin coating targets were analysed with the help of the 1D PIC code BOPS. Hardx-rays are generated when a short-pulse, high-intensity laser is used to ionize a solid tar-get, generating high-velocity electrons which proceed to create inner-shell holes in nearbyatoms. The resulting hard x-ray pulse can be used for real-time imaging of rapid struc-tural changes in solids or large molecules. It is found that the x-ray pulse efficiency for

383

poor-contrast laser systems in which a large preplasma is suspected can be enhanced byusing a near-normal incidence geometry even at high laser intensities. It can be shown thatindependently of the laser contrast ratio, i.e. for a steep density gradient or even in the pres-ence of an extended density shelf, it is possible to control and optimise the Kα emissionfrom a femtosecond laser plasma by adjusting the prepulse delay and/or angle of incidenceappropriately. Laser systems with poor contrast can be better utilised by using target con-figurations with small incidence angles to optimise (resonant) absorption. The use of largeincidence angles – such as the standard 45-degree geometry –may be contributing to poorKα conversion efficiencies recorded in many recent experiments. A high-contrast systemcan be optimised either by controlled prepulse or by going tolarger incidence angles, ex-ploiting the tendency for the density profile to be remouldedby the main pulse, allowingbetter absorption1, 2.

3 Heating and Ion Acceleration in Nanostructured Foils

Nanostructure surfaces are especially promising as highlyabsorbing targets for high-peak-power sub-picosecond laser-matter interaction. Efficienthot electron, fast ion, and ther-monuclear neutron production with moderate laser intensity have already been reported,but theoretical investigations on the use of porous targetsfor these purposes are still scarce.In a recent study using PEPC, a new phenomenon of Coulombimplosionhas been identi-fied3. The implosion effect is caused by hot electrons circulating inside the shells, drawingions inwards, where they eventually meet in the centre – Figure 1.

a) b)

Figure 1. Coulomb implosion of thin shells initially arranged in a bcc foam matrix with ionized electrons confinedto the inner shell surfaces – a). Acceleration of ions off theinner surfaces leads to uniform convergence andpeaked ion temperatures (hot spots) at the shell centres – b).

Under the same irradiation conditions, a single shell simply blows apart, and does notexhibit the symmetric collapse observed in the foam latticesimulation here. In this case,some hot electrons circulate inside the shell, but most are dragged outside, leading to a netforce on the ions directed radially outwards. (A fully stripped ion shell will, by Gauss’ Law,have zero electric field inside). This implies that the laserheating is strongly modified by

384

the regular lattice structure, which in turn radically alters the ion dynamics. These findingshave recently been corroborated by 2D PIC simulations in which the electron heating andlaser propagation is treated self-consistently with a fully electromagnetic field solver –Figure 2.

a)

b)

Figure 2. 2D PIC simulation of foam array irradiated by high intensity pulse (incident from the bottom) withsimilar parameters to those in the tree-code simulations. a) Electron density at a few fs, b) ion density after100 fs.

Should this nano-implosion phenomenon scale to higher, relativistic intensities, itmight also have potential as a compact neutron source. A recent comparison betweenatomic clusters and aerogels4 suggests that the latter are capable of yielding 10 times asmany neutrons for the same laser energy because of the higherkinetic energy impartedto deuterons contained within the aerogel lattice. The present study indicates that densityenhancements created by heatingregular porous lattice structures should result in evenhigher neutron yields.

385

4 Mass Limited Ion Acceleration

One of the hot topics of laser-matter interactions is ion acceleration. Current experimentshave made dramatic progress in producing multi-MeV beams oflight ions and protonswhich may eventually be used in tumor therapy. Before this goal can be realised how-ever, a number of challenges have to be overcome regarding the beam quality. Simulationsare essential here for exploring novel target configurations in order to provide experimen-tal guidance. Microstructured targets have been proposed to reduce the ion beam energybandwidth and emittance. Our simulations show that a pure proton microdot target doesnot by itself result in a quasimonoenergetic proton beam: infact, such a beam can only beproduced with a very lightly doped target, in qualitative agreement with one-dimensionaltheory5 – Figure 3. The simulations suggest that beam quality in current experiments6

a) b)

Figure 3. Laser acceleration of proton microdot (central feature). Energy spread of the protons can be reducedby decreasing the relative proton density – here 50% (a) and 5% (b) respectively.

could be dramatically improved by choosing microdot compositions with a 510 times lowerproton fraction. Further investigations have further quantified these findings, setting lowerlimits on the useful proton fraction, beyond which an energyfiltering scheme becomesmore effective7.

5 Mesh-Free Modelling of Tokamak Edge Physics

The group “plasma edge simulations for fusion plasmas” in IEF-4 (Prof. Reiter) developsand applies 2- and 3-dimensional computer simulation codesfor interpretative andpredictive studies of physics close to the plasma containerwall (vacuum chamber).This domain is characterised by a complex interaction of plasma-chemical and turbulentprocesses. The models contain both empirical and first-principles based modules. Thelong term goal of model development is step by step replacement of thead hocby ab initiomodels, aided by increasing HPC resources based on the parallel computing paradigm.The self-consistent electrical field, obtained from the position of charged test particles, or,

386

in fluid approximations, of charged fluid parcels, is one suchcomponent.The mesh-free method offered by the tree code PEPC-B developed at JSC is a promisingcandidate either to provide self-consistent fields in tandem with existing modelling tools,or as a stand-alone (radiation-free) code for fusion plasmasimulations. Currently this isbeing jointly developed by JSC and IEF-4 to model plasma edgephenomena such as thepotentially catastrophic edge-localised modes (ELMs).

Figure 4. Example of a triangular mesh fitted in the vessel of the MAST-tokamak.

In a first step interfaces have been developed to provide PEPC-B with fusion relevantmagnetic fields. For this purpose equilibrium fields similarto those used in the transportcode EIRENE8 have been introduced. To define the external magnetic field PEPC-B as-signs vector data according to a 2d triangular mesh (5), assuming toroidal symmetry in thetokamak. This feature in principle enables PEPC-B to run full tokamak core simulations.Next, the field data provided from external sources have to besmoothed and adjusted toguaranteediv( ~B) = 0.A further addition is the inclusion of a collision module based on the Monte-Carlo modelof Takizuka and Abe9. This permits collisions between the injected tracer particles and theplasma backround to be taken into account. An example application is the modelling of im-purities such asC+ ions occurring in fusion plasmas after sputtering or gas puff scenariosin a realistic plasma environment. A current priority is to simulate gas puff experimentsrecently performed on TEXTOR and to see how these findings scale to larger machinessuch as ITER.

6 Darwin Model

A strategic goal with this project is to extend the fast summation algorithm deployed inPEPC to include self-generated magnetic fields. This “Darwin” or magnetoinductive ap-proach has been pursued within the particle-in-cell paradigm for some time10. A mesh-freeDarwin model would open up a large range of new applications of mesh-free plasma sim-ulation, not least in the tokamak modelling scenarios described above, and also in particlebeam transport in dense plasmas. The model incorporates slowly varying magnetic fields

387

by neglecting the transversal part of displacement currentin Amperes law. This transformsthe hyperbolic equation system into a set of elliptic equations, which can again by solvedby a fast summation algorithm.

φi(ri) =∑

j 6=i

qj|ri − rj |

(1)

Eli(ri) =

∑ qjrijr3ij

(2)

Ai(ri) =1

2c

∑ qjvjrij

+1

2c

∑ (qjvj · rij)rijr3ij

(3)

Bi(ri) = ∇×Ai =1

c

∑ qjvj×rijr3ij

(4)

Eti(ri) = −1

c

∂Ai

∂t= − 1

2c2

∑ qj vjrij

− 1

2c2

∑ (qj vj · rij)rijr3ij

(5)

On the other hand, numerical difficulties arising from this approximation have alreadybeen reported by previous authors who implemented the Darwin model within PIC orVlasov codes. The standard scheme known from the fully electromagnetic codes usedfor the calculation of time derivative of the vector potential causes a violent numerical in-stability destroying the whole run in a few time-steps. One of the possible solutions to thisproblem is to express the quantities in terms of Hamiltoniangeneralized variables, whichavoids the time derivative of the vector potential in the theequation of motion. Our magne-toinductive model employs a multipole expansion of the Darwin field equation, modified toaccount for finite-sized particles and evaluated within thePEPC tree algorithm framework.First tests of this model have been made on charged particle beam evolution in vacuum.

7 Tree Code Scalability/Performance

The PEPC code is written in a generic fashion without the usage of external libraries. Thisresults in excellent portability. In the PRACE benchmarking framework PEPC was run onfour different computer architectures, namely: IBM Blue Gene/P (jugene), IBM Power6(huygens), Cray XT5 (louhi), Intel Nehalem (juropa). As a test case we used5 · 107

particles and a cubic, homogeneous initial distribution. The scaling behaviour is shown inFigure 5, where the time needed for one inner loop step is shown as a function of the peakperformance of the used partition. As shown, PEPC is able to utilize the given partitionperformance independently of the architecture. Only the Intel Nehalem architecture showsa significantly better performance. PEPC’s overall scalingbehaviour is also impressivefor a tree code, although the current version is only able to use up to approximately 8192cores. But with these the code is capable of simulating more than108 particles.

388

Three different areas have been identified as challenges forfurther development of thecode: The inevitable need for global communication in Barnes-Hut tree codes, a strongmemory dependency of the number of cores and a reliable load balancing scheme. Theseproblems have to be well analysed in order to make resonable use of PEPC on larger par-titions. While the second topic is a question of data structure and organisation the twoothers focus on communication patterns and their impact on the performance. It is clearthat although the tree code is an intrinsically adaptive algorithm which performs very welleven with very inhomogeneuous particle distributions, itsperformance relies strongly on asophisticated load balancing scheme, which is closely linked to the domain decompositionmethod. But even with a wisely chosen load balancing approach the need for non-localcommunication is an inevitable consequence of the ellipticnature of the Poisson equation.On the other hand, recent analyses of the communcation pattern reveal that nearly everycommunication instance is actually required at some point in the force calculation. Forcovering the span of necessary non-local communication anddemanding memory require-ments a physical separation of communication and computation process may provide away to scale beyond104 cores.

Figure 5. Performance of PEPC on various HPC architectures,normalised to the theoretical peak performanceof a given multi-core partition.

8 Concluding Remarks

To summarize, we have reported on various activities concerning kinetic plasma modellingat JSC. Most of this work involves application and further development of the mesh-free

389

code PEPC, but recently we have also started using fully electromagnetic PIC codes tomodel laser-plasma interaction processes. Future work will include rigorous comparisonsbetween the PIC and mesh-free approaches, in order to more clearly identify the advantagesof the latter. The mesh-free Darwin or magnetoinductive approach offers completely newpossibilities in many areas of plama physics from magnetic fusion to space physics. Amajor challenge for the usage of the mesh-free technique on contemporary HPC systemsis to improve its parallel scalability. Efforts are underway to improve the current limit of8192 cores on BG/P to at least 64k cores.

Acknowledgements

This work was supported by the Alliance Program of the Helmholtz Association(HA216/EMMI). Simulations were performed with computing resources granted by theVSR of the Research Centre Julich unter project JZAM04.

References

1. P. Gibbon, M. Masek, U. Teubner, W. Lu, M. Nicoul, U. Shymanovich, A. Tarase-vitch, P. Zhou, K. Sokolowski-Tinten, and D. von der Linde,Modelling and optimi-sation of fs laser-produced Kα sources, Appl. Phys. A, 2009.

2. W. Lu, M. Nicoul, U. Shymanovich, A. Tarasevitsch, P. Zhou, K. Sokolowski-Tinten,D. von der Linde, M. Masek, P. Gibbon, and U. Teubner,Optimized Kα x-ray flashesfrom femtosecond-laser irradiated foils, Phys. Rev. E,80, 026404, 2009.

3. P. Gibbon and O. N. Rosmej,Stability of nanostructure targets irradiated by highintensity laser pulses, Plasma Phys. Contr. Fus.,49, 1873–1883, 2007.

4. V. P. Krainov and M. B. Smirnov,Nuclear fusion induced by a super-intense ultra-short laser pulse in a deuterated glass aerogel, JETP,93, 485–490, 2001.

5. A. P. L Robinson and P. Gibbon,Production of proton beams with narrow-band en-ergy spectra from laser-irradiated ultrathin foils, Phys. Rev. E,75, no. 1, 015401,2007.

6. S. M. Pfotenhauer, O. Jackel, A. Sachtleben, J. Polz, W. Ziegler, H.-P. Schlenvoigt,K.-U. Amthor, M. C. Kaluza, K. W. D. Ledingham, R. Sauerbrey,P. Gibbon, A. P. L.Robinson, and H. Schwoerer,Spectral shaping of laser generated proton beams, NewJ. Phys.,10, 033034, 2008.

7. A. P. L. Robinson, P. Gibbon, M. Zepf, S. Kar, R. G. Evans, and C. Bellei,Relativis-tically correct hole-boring and ion acceleration by circularly polarized laser pulses,Plasma Phys. and Cont. Fus.,51, 024004, 2009.

8. D. Reiter, M. Baelsmans, and P. Borner,The Eirene and B2-Eirene Codes, FusionScience and Technology,47, 172, 2005.

9. T. Takizuka and H. Abe,A binary collision model for plasma simulation with a parti-cle code, J. Comp. Phys., 1977.

10. D. W. Hewett,Elimination of electromagnetic radiation in plasma simulation: theDarwin or magnetoinductive approximation, Space. Sci. Rev.,42, 29–40, 1985.

390