Wakefield Computations at Extreme-scale for Ultra-short

Bunches using Parallel hp-Refinement

Lie-Quan Lee, Cho Ng, Arno Candel, Liling Xiao,

Greg Schussman, Lixin Ge, Zenghai Li,

Andreas Kabel, Vineet Rawat, and Kwok Ko

SLAC National Accelerator Laboratory

SciDAC 2010 Conference, Chattanooga, TN, July 11-15, 2010

Overview

* Parallel computing for accelerator modeling

* Finite-element time-domain analysis

* Moving window technique with finite-element method

– Wakefield calculations with p-refinement– Wakefield calculations with online hp-refinement

* Benchmarking and Results

– PEP-X undulator taper structure– PEP-X wiggler taper structure – Cornel ERL vacuum chamber– ILC/ProjectX Coupler

* Particle accelerators are billion-dollar class facilities– Proposed International Linear Collider: 6.75 billion dollars

– Large Hadron Collider:

6.5 billion dollars

* Advanced computing enables virtual prototyping– Cost savings from design optimization through computing can be

significant

Advanced Computing for Accelerators

30 km

27 km

* ComPASS Accelerator Project (2007-2012)– TOPS/LBL (Linear solvers and eigen-solvers)

• E Ng, X Li, I Yamazaki– ITAPS/RPI, LLNL, Sandia (Parallel curvilinear meshing

and adaptation)• Q Lu, M Shephard, L Diachin

– Ultra-scale Visualization Institute/Sandia, UC Davis• K Mooreland, K Ma

– ITAPS/CSCAPES/Sandia (Load balancing)• K Devine, E Boman

o Collaborations on algorithmic aspects

SciDAC for Accelerator Modeling

Parallel Finite Element Code Suite ACE3P

Visualization: ParaView – Meshes, Fields and Particles

Over more than a decade, SLAC has developed the conformal, higher-order, C++/MPI-based parallel finite-element suite of electromagnetic codes, under the supports of AST SciDAC1 and ComPASS SciDAC2 projects

ACE3P Modules – Accelerator physics application

Frequency Domain: Omega3P – Eigensolver (nonlinear, damping) S3P – S-Parameter

Time Domain: T3P – Wakefields and TransientsParticle Tracking: Track3P – Multipacting and Dark CurrentEM Particle-in-cell Pic3P – RF gun simulation

Aiming for the Virtual Prototyping of accelerator structures

INCITE Program

* INCITE Award: Petascale Computing for Tera-eV Accelerators (2008-2010)

- ORNL (Jaguar / Lens / Hpss)- Kitware (for ParaView visualization and analysis

software)- Center for Scalable Application Development

Software (CScADS)

o Provide major computing resources and supports for efficiently using them in tackling challenging accelerator modeling problems

SLAC’s INCITE Allocation Usage at NCCS

Average Job size:10335

Average Job size:24835

Average Job size:2005

Average Job size:1834

Average Job size:6349

Average Job size:911

2008 total4.5 million

2009 total8 million

• ~60% of total time is used with 10% to 50% of the total resources (22k cores to 120k cores)

Number of Cores

Allo

catio

n T

ime

Use

d

2010 Allocation Usage (up to 07/2010)

2010 total 12 million

1200

3000

3600

4800

6000

9000

1200

0

2400

0

3000

0

6000

0

1200

000E+00

1E+06

2E+06

3E+06

4E+06

Chart Title

Cavity Shape - Ideal in silver vs deformed in gold

Solving CEBAF BBU Using Shape Uncertainty Quantification Method

Method of Solution - Using the measured cavity parameters as inputs, the deformed cavity shape was recovered by solving the inverse problem through an optimization method. The calculations showed that the cavity was 8 mm shorter than designed, which was subsequently confirmed by measurements. The result explains why the troublesome modes have high Qs because in the deformed cavity, the fields shift away from the HOM coupler where they can be damped. This shows that quality control in cavity fabrication can play an important role in accelerator performance. .

SciDAC Success as a Collaboration between Accelerator Simulation, Computational Science and Experiment – Beam Breakup (BBU) instabilities at well below the designed beam current were observed in the CEBAF12 GeV upgrade of the Jefferson Lab (TJNAF) in which Higher Order Modes (HOM) with exceptionally high quality factor (Q) were measured. Using the shape uncertainty quantification tool developed under SciDAC, the problem was found to be a deformation of the cavity shape due to fabrication errors. This discovery was achieved as a team effort between SLAC, TOPS, and JLab which underscores the importance of the SciDAC multidisciplinary approach in tacking challenging applications.

Field profiles in deformed cavity

HOM coupler

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.E+07

1.E+08

1.E+09

2030 2080 2130 2180F (MHz)

Qex

t

Deformed

ideal

cav5-meas

High Q modes

Finite Element Discretization

* Curvilinear tetrahedral elements: High fidelity modeling

* High-order hierarchical vector basis functions (H(curl))– Provide tangential continuity required by physics – Easily set the boundary conditions– Significantly reduce phase error

0.5 mm gap

200 mm

Entities # of bases

Edge p

Face p(p-1)

Volume p(p-1)(p-2)/2

Total 6E+4F+V

Shape functions

Time-Domain Analysis

Second-order vector wave equation:Compute transient and wakefield effects of beams inside cavities

Denote:

Finite-Element Time-Domain Analysis

H(Curl)-conforming Element (discretized in space):Ni

ODE in matrix and vector form:

Newmark- Scheme for Time Stepping

* Unconditionally stable* when > 0.25* A linear system Ax=b needs to be solved for each time step* Matrix in the linear system is symmetric positive definite* Conjugate gradient + block Jacobi / incomplete Cholesky

*Gedney & Navsariwala, An unconditionally stable finite element time-domain solution of the vector wave equation, IEEE microwave and guided wave letters, vol. 5, pp. 332-334, 1995

Wakefields Calculations

* At given time, electric field and magnetic flux can be calculated:

* Wake Potential:

Coupler region

Cavity for International Linear Collider

Parallel Finite-Element Time-Domain Code: T3P

* Ongoing work:- Preconditioners and multiple

right hand sides- Fine-tune mesh partitioning

Meshgeneration

Analysis & viz

CAD model

Netcdf mesh file Input parameters

Partition mesh

Assemble matrices

Solver/time stepping

Postprocess: E/B

Strong Scaling Study

131072

256 million elementsp=2NDOFs=1.6 billion

3D PEP-X Undulator Taper

* Need to compute short-range wakefield due to 100 mm smooth transition from 50mm×6mm elliptical pipes to 75mm×25mm

* Beam bunch length is 0.5 mm in a 300 mm long structure

Computer Model of Undulator Taper

50 mm×6 mm

75 mm×25 mm

Moving Window with p-Refinement

* Use a window to limit the computational domain– Fields from the left of the window will not catch up as beam

moves with speed of light– Fields on the right of the window should be zero

* Use finite element basis function order p to implement window– p is nonzero for elements inside the window– p = 0 for any elements outside of the window

* Move the window when the beam is close to the right boundary of the window– Prepare the mesh, repartition the mesh, assemble matrices,

transfer solution, …

p=2 p=0p=0

fb

d

Solution Transfer when Window Moves

* Scatter xn to each element– Keep values in the overlapping region – Drop values in the left of the window– Set zeros in the right of the overlapping region– Redistribute to different processes according to partitioning

* Gather values to form new xn

Benchmarking with 4mm Bunch

* run on XT5 with 5 nodes (60 cores) – Case 1: the whole structure/mesh with 2 mm size : 21.5 minutes– Case 2: moving window with p refinement (same mesh): 11 minutes

* Identical wakefields within beam (40mm) and its tailing zone(10mm)

* Significantly reducing computational resources with moving window

* Additional Benefits:* Smaller cores count* Faster execution time* Solve larger problems

Wake potential

Movie of Wakefields in PEP-X Undulator Taper

Wakefields of Undulator Taper with 0.5mm Bunch

* 60 mm trailing area with 0.5mm beam bunch size* 0.25mm mesh size, 19 million quadratic tetrahedral elements* p=2 inside the window, p=0 outside* 15 windows (90mm)* Maximal NDOFs is 54.9M only, Maximal Nelems 8.6 million

Reconstructing wake of 3mm bunch

21

PEP-X Wiggler Taper

400 mm (10*taper_height)

45mm x 15 mmR48mm

Model: 400 mm transition between the rectangular beam pipe (45mm × 15mm) and circular beam pipe (radius 48mm).

Calculate: wakefields with 0.5mm bunch size with 60mm tail.

Wiggler mesh

50mm x 6 mm

75mm x 25 mm

100mm (10*taper_height)

Wiggler Taper

Undulator Taper

Challenges in Wakefields Calculation

* Number of mesh elements would be roughly 4 to 5 times larger than that of undulator taper

* Number of time steps is about 3 to 4 times larger* Meshing problem: cannot easily generate such a large

mesh (~100 million curvilinear elements)– Mesh generator is serial (parallel mesh generation is under

development at ITAPS)– Need a computer with lot of memory (128GB)

* Inefficient use of computer resources– Mesh in the new window need to be read in from the file– Or store in the memory in parallel

Can we do online mesh refinement in the window region?o Prepare mesh, repartition, assemble matrices, transfer solution, …

Mesh Refinement: Edge Splitting

* There are many different ways of splitting a tetrahedron* Divide all 6 edges: better suited for resolving short beam

bunch* Generate 8 smaller tetrahedral elements

Splitting linear tetrahedron Splitting quadratic tetrahedron

Online Parallel Mesh Refinement

* Midpoints of 6 edges are new vertices– Need a coordinated way for new vertices on process boundary

* Similar to number the DOFs (edge element) using 1st order basis function in our electromagnetic simulation

* Regard each new vertices as a DOF* Run our parallel partitioning and

numbering routines to get unique IDs cross different processors

* Form the dense mesh by local edge splitting of each element owned by the processor

* ~500 lines of additional code only

Solution Transfer When Mesh Changed

A Projection Method for Discretized Electromagnetic Fields on Unstructured Meshes, Lie-Quan Lee, CSE09, Miami, Florida, March 2-6, 2009; Tech. pub. SLAC-PUB-13333

xn xn-1 f n f n-1

p=0 p=2 p=0

26

Solution Transfer between Meshes

* Vectors xn, xn-1, f n, f n-1 need to be transferred onto the new mesh

* For xn and xn-1, a projection method is used:

* Scalable parallel projection is a challenge* Vectors f n and f n-1

– Recalculation on the new mesh

Wakefield Benchmarking

* 20mm bunch size: window size 0.5 m (b=0.1m,f=0.2m) on XT5 with 5nodes (60 cores)– Case 1: 10 mm mesh size with hp-refinement: 36.5 minutes– Case 2: 5 mm mesh size with p-refinement: 23 minutes

* Identical results for s < 0.3m.

* Longer time in Case 1 is due to overhead associated with mesh refinement

* Using less memory

Wakefields of 1mm Bunch in PEP-X Wiggler Taper

* 11 windows (170mm) with 60mm tailing area* Maximal NDOFs: 74.7 million* Maximal Nelems: 11.8 million

Reconstructing wake of 3mm bunch

Wakefields in Cornell ERL Vacuum Chamber

* Energy Recovery Linac (ERL) Aluminum Vacuum Chamber* Moving window with online mesh refinement; 18000 cores

with ~5 hours on Jaguarpf at ORNL* Not possible without algorithmic advance and INCITE-scale

resources for calculating wakefields of such a short bunch

Computer model of an ERL vacuum chamber device

For 0.6mm bunch length, loss factor is 0.413 V/pC.

Movie of Wakefields in ERL Vacuum Chamber

Preliminary Results of Wakefields in ILC Coupler

* Without Windows– Would be 256 million elements

with 1.6 billion DOFs– 100k cores with 6s per step

* With Moving Windows– ~30 million elements are used

inside each window– 12k cores with 6s per step (p=2)– 12k cores with 0.4s per step (p=1)

s = 0.3 mm

Project X shares the same cavity design

Summary

* Calculating wakefields due to ultra-short bunches requires efficient computational methods – Moving window with p-refinement

– Moving window with hp-refinement

– Parallel online mesh refinement

– Parallel solution transfer through projection

* We are making major impact on accelerator design through extreme-scale computing

* Only possible through – SciDAC supports for algorithmic advancement, and

– INCITE allocation for large computing resources