parallel mesh partitioning at slacmmwolf/presentations/cs591mh/cs591m… · 17/09/2003 · 11...
TRANSCRIPT
![Page 1: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/1.jpg)
Parallel Mesh Partitioning at SLAC
Michael M. Wolf
![Page 2: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/2.jpg)
2
Stanford Linear Accelerator Center
![Page 3: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/3.jpg)
3
Stanford Linear Accelerator Center
• DOE laboratory managed by Stanford University
• Established 1962• Located at Stanford University
in Menlo Park, CA• “Mission is to design, construct and operate
state-of-the-art electron accelerators and related experimental facilities for use in high-energy physics and synchrotron radiation research.”
• 3 kilometers long (e-gun to start of rings)• 3 Nobel Prize Winners• Home of the first U.S. Website
![Page 4: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/4.jpg)
4
Stanford Linear Accelerator Center
Courtesy Stanford Linear Accelerator Center
![Page 5: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/5.jpg)
5
Stanford Linear Accelerator Center
Courtesy Stanford Linear Accelerator Center
![Page 6: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/6.jpg)
6
Stanford e+e- Linear Collider (SLC)
![Page 7: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/7.jpg)
7
Advanced Computations Department
![Page 8: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/8.jpg)
8
Computational Mathematics
Y. Liu, I. Malik, W. Mi, J. Scoville, K. Shah, Y. Sun (Stanford)
ACD Organization/Collaborators
Accelerator Modeling
V. Ivanov, A. Kabel,K. Ko, Z. Li, C. Ng,L. Stingelin (PSI)
Computing Technologies
N. Folwell, L. Ge, A. Guetz, R. Lee, M. Wolf
Stanford (SCCM)
G. Golub, O. Livne,
Sandia
P. Knupp, T. Tautges,L. Freitag, K. Devine
LBNL (SCG)
E. Ng, P. Husbands, S. Li, A. Pinar
LLNL (CASC)
D. Brown, K. Chand, B. Henshaw
RPI (SCOREC)
M. Shephard, Y. Luo
UCD (VGRG)
K. Ma, G. Schussman
Collaborators
Advanced Computations Department
![Page 9: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/9.jpg)
9
• Accurate modeling essential for modern accelerator design.
• Uncertainty in design greatly increases cost• Accurate computer models reduce design costs
and design cycle• Need for accurate cavity design tools • ACD develops these simulation codes
• E&M field, resonant frequency, particle tracking simulations
• Conformal meshes (Unstructured grid)• Parallel processing
• Codes: Omega3P, Tau3P, Track3P, S3P, Phi3P
ACD
![Page 10: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/10.jpg)
10
Challenges in E&M Modeling of Accelerators
Ø Accurate geometry is important due to tight tolerance – needs unstructured grid to conform to curved surfaces
Ø Large, complex electromagnetic structures – large matrices after discretization (100’s of millions
of DOF), needs parallel computing for both problem storage and reduction of simulation time
Ø Small beam size ~ delta function excitation in time & space
> Time domain – needs to resolve beam size leading to huge number of grid points, long run time & numerical stability issues
> Frequency domain – wide, dense spectrum to solve for thousands of modes to high accuracy
![Page 11: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/11.jpg)
11
Motivation for New EM Capability
Modeling RDDS Cell with standard accelerator code MAFIA using Structured Grid on Desktops demonstrates the need for MORE ACCURATE EM codes
-100
-90
-80
-70
-60
-50
-40
-30
-20
-10
0
0 200 400 600 800 1000 1200 1400
number of mesh points (103)
DF
(=F-
1142
4) (M
Hz)
1.2 hrs
3.4 hrs
5.4 hrs
7.0 hrs
0.01 % ~ 1 MHz
![Page 12: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/12.jpg)
12
Typical ACD problems
![Page 13: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/13.jpg)
13
Heating in PEP-II Interaction Region
e+ e-
Center beam pipe Right crotchLeft crotch
2.65 m 2.65 m
Beam heating in the beamlinecomplex near the IR limited the PEP-II from operating at high currents. Omega3P analysis helped in redesigning the IR for the upgrade.
FULL-SCALE OMEGA3P MODEL FROM CROTCH TO CROTCH
![Page 14: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/14.jpg)
14
Next Linear Collider (NLC)
Control of Long Range Wakefield crucial to multi-bunch operation
SLC NLC
Center of Mass 100 GeV 500 GeV
Bunches per pulse 1 95
Operating Frequency (S) 2.856 GHz (X) 11.424 GHz
Number of Cavities 80,000 2 million
Post-Tuning yes No
![Page 15: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/15.jpg)
15
Ø Needs accelerating frequency calculated to 0.01% accuracy to maintain structure efficiency
Ø Optimized design could save $100 million in machine cost
The NLC Accelerating Structure
Cell optimized to increase shunt impedance (~14%) & minimize surface gradients
Cell to cell variation of order microns to suppress short range wakes by detuning
Manifold damping to suppresslong range wakes
206-Cell Round Damped Detuned Structure RDDS
11 cavity dimensions
![Page 16: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/16.jpg)
16
NLC Structure Design Requirements
NLC 206-Cell RDDS Structure
RDDS Section
Ø RDDS Cell : Design to 0.01% accuracy in accelerating frequency, Ø RDDS Section : Model damping/detuning of dipole wakefields.
RDDS Cell
![Page 17: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/17.jpg)
17
Particle Tracking in 5 Cell RDDS (Tau3P/Track3P)
![Page 18: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/18.jpg)
18
Cyclotron COMET (Omega3P)
„Dee“: RF electrode „Liner“: outer shell of RF cavity
Electric field in acceleration gapProton trajectories
Magnetic Field
First ever detailed analysis of an entire cyclotron structure - L. Stingelin, PSI
Fig 1. The cyclotron COMET
Superconductingcoils
RF DeeLiner
![Page 19: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/19.jpg)
19
Where we want to go
![Page 20: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/20.jpg)
20
• NLC X-band structure showing damage in the structure cells after high power test
• Theoretical understanding of underlying processes lacking so realistic simulation is needed
End-to-end NLC Structure Simulation
(J. Wang, C. Adolphsen – SLAC )
![Page 21: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/21.jpg)
21
End-to-end NLC Klystron Simulation
Field and particle data estimated to be TB size
![Page 22: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/22.jpg)
22
Tau3PParallel Time-Domain 3D Field Solver
![Page 23: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/23.jpg)
23
Parallel Time-Domain Field Solver – Tau3P
Follows evolution of E and H fields on primary/dual meshes (hexahedral, tetrahedral and pentahedralelements) using leap-frog scheme in time (DSI scheme)
Tau3P MAFIA
∫∫∫∫∫
∫∫∫
•+•∂∂=•
•∂∂−=•
*** dAjdA
tDdsH
dAtBdsE
![Page 24: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/24.jpg)
24
Discrete Surface Integral Method
• Electric fields on primary grid• Magnetic fields on dual grid• Primary and dual grids non-orthogonal• Dual grid constructed by joining centers of primary cells• Electric and magnetic fields advanced in time using the leapfrog algorithm• Reduces to conventional finite difference time domain method (FDTD) for non-orthogonal grids• Conforms to complicated geometry by appropriate choice of element types
Ref: N. K. Madsen, J. Comp. Phys., 119, 34 (1995)
![Page 25: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/25.jpg)
25
Tau3P Applications
![Page 26: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/26.jpg)
26
Time Domain Design & Analysis (Tau3P)
Output Coupler loading on HOM modes at the RDDS output end
Dipole mode spectrum
Matching NLC Input Coupler w/ Inline Taper
Dipole Excitation
ReflectedIncidentTransmitted
R T
![Page 27: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/27.jpg)
27
Wakefield Calculation (Tau3P)
• Response of a 23-cell X-Band Standing Wave Structure w/Input Coupler & Tapered Cells to a transit beam in Tau3P.
• Direct wakefield simulation of exact structure to verify approximate results from circuit model.
a: 4.663-4.875 mm, b:10.796-10.879 mm, t: 2.541-2.684 mm
![Page 28: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/28.jpg)
28
Steady-state Surface Electric field amplitude
Determining Peak Fields (Tau3P)
Rise time = 10,15, 20 ns
Electric field vs timeDrive pulse
§ When and where Peak Fields occur during the pulse? § Transient fields 20% higher than steady-state value
due to dispersive effects
![Page 29: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/29.jpg)
29
15-Cell H90VG5 Model
Rise time=10ns
Overshoots for different rise times Fields at different cell disks as a function of time
• Peak field appears near the middle of the structure• About 25% overshoot in peak field due to the narrower bandwidth
![Page 30: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/30.jpg)
30
Electric Fields (Tau3P)
![Page 31: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/31.jpg)
31
Tau3P Matrices
![Page 32: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/32.jpg)
32
Discrete Surface Intergral Formulation
• The DSI formulation yields:– efield += α*AH*hfield– hfield += β*AE*efield
• efield, hfield are vectors of field projections along edges/dual-edges
• AH,AE are matrices• α, β are constants proportional to dt
∫∫∫∫∫
∫∫∫
•+•∂∂=•
•∂∂−=•
*** dAjdA
tDdsH
dAtBdsE
![Page 33: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/33.jpg)
33
Tau3P Implementation
Example of Distributed Mesh
Typical AE Distributed Matrix
![Page 34: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/34.jpg)
34
Tau3P Matrix Properties
• Very Sparse Matrices– 4-20 nonzeros per row
• 2 Coupled Matrices (AH,AE)• Nonsymmetric (Rectangular)
Typical Distributed Matrix
![Page 35: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/35.jpg)
35
Tau3P Performance Problems
![Page 36: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/36.jpg)
36
Load Balancing Issues in Tau3P
• Load balancing problem inTau3P Modeling of NLC Input Coupler.– Unstructured meshes lead to matrices for
which nonzero entries are not evenly distributed.– Complicates work assignment and load balancing
in a parallel setting.– Tau3P originally used ParMETIS
to partition the domain to minimizecommunication.
NZ Distribution over 14 cpu’s Parallel Speedup
![Page 37: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/37.jpg)
37
Parallel Performance of Tau3P
2.02.8
3.9 4.45.7
6.8
9.2
Parallel Speedup
• 257K hexahedrons• 11.4 million non-zeroes
SLAC PC Cluster
![Page 38: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/38.jpg)
38
Communication in Tau3P (ParMetis Partitioning)
![Page 39: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/39.jpg)
39
Communication in Tau3P (ParMetis Partitioning)
![Page 40: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/40.jpg)
40
Improved Mesh Partitioning Schemes
![Page 41: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/41.jpg)
41
Luxury in Tau3P Mesh Partitioning
• Long simulation times– Tens of thousands of CPU hours
• Long time spent in time stepping– Millions of time steps
• Problem initialization short• Static (not dynamic) mesh partitioning• Willing to pay HIGH price upfront for
increased performance of solver
![Page 42: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/42.jpg)
42
Zoltan Overview
• Developed at Sandia National Laboratory (NM)• Collection of Data Management Services for Parallel,
Unstructured, Adaptive, and Dynamic Applications • Supports Several Load Balancing Methods:
– Graph Partitioning Algorithms• ParMETIS• Jostle
– Geometric Partitioning Algorithms (1D/2D/3D)• Recursive Coordinate Bisection (RCB)• Recursive Inertial Bisection (RIB)• Hilbert Space-Filing Curve (HSFC)• Octree Partitioning (various traversal schemes including HSFC)• Refinement Tree Based Partitioning (mesh refinement)
• Supports Dynamic Load Balancing/Data Migration
![Page 43: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/43.jpg)
43
Zoltan Partitioning Methods
![Page 44: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/44.jpg)
44
ParMETIS (Graph)
![Page 45: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/45.jpg)
45
Recursive Coordinate Bisection (Geometric)
![Page 46: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/46.jpg)
46
Recursive Inertial Bisection (Geometric)
![Page 47: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/47.jpg)
47
Hilbert Space Filling Curve (Geometric)
![Page 48: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/48.jpg)
48
Hilbert Space Filling Curve (Geometric)
![Page 49: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/49.jpg)
49
Tau3P Partitioning Results
![Page 50: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/50.jpg)
50
RDDS (5 cell w/ couplers) ParMETIS Partition
![Page 51: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/51.jpg)
51
RDDS (5 cell w/ couplers) RCB-1D(z) Partition
![Page 52: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/52.jpg)
52
RDDS (5 cell w/ couplers) RCB-3D Partition
![Page 53: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/53.jpg)
53
RDDS (5 cell w/ couplers) RIB-3D Partition
![Page 54: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/54.jpg)
54
RDDS (5 cell w/ couplers) HSFC-3D Partition
![Page 55: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/55.jpg)
55
5 Cell RDDS (8 processors) Partitioning
9038 2030 32 5387.3 sHSFC-3D
7927 1570 18 3282.4 sRIB-3D
11961 1965 26 5343.0 sRCB-3D
14363 3128 14 2218.5 sRCB-1D (z)
2909 585 14 3288.5 sParMETIS
Sum Bound. Objs
Max Bound. Objs
Sum Adj. Procs
Max Adj. Procs
Tau3P Runtime
2.0 ns runtimeIBM SP3 (NERSC)
![Page 56: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/56.jpg)
56
5 Cell RDDS (32 processors) Partitioning
26684 1279 202 10 272.2 sHSFC-3D
20156 808 162 8 266.8 sRIB-3D
24321 1404 208 10 373.2 sRCB-3D
63510 2683 66 3 67.7 sRCB-1D (z)
16405 731 134 8 165.5 sParMETIS
Sum Bound. Objs
Max Bound. Objs
Sum Adj. Procs
Max Adj. Procs
Tau3P Runtime
2.0 ns runtimeIBM SP3 (NERSC)
![Page 57: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/57.jpg)
57
H60VG3 (“real” structure)
55 cells (w/ coupler)1,122,445 elements
![Page 58: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/58.jpg)
58
H60VG3 RDDS Partitioning (w/o port grouping)
1.0 ns runtimeIBM SP3 (NERSC)
4337.9/51292.3 s292.1 s107.7/51214512
8315.0/102499.0 s327.5 s96.0/1024161024
2241.4/256129.2 s 360.0 s87.3/25611256
2117.6/128265.1 s643.0 s48.9/12810128
249.7/64627.2 s736.6 s42.7/64464
225.2/321236.6 s1455.0 s21.6/32432
212.7/162458.5 s2703.3 s11.6/1631628/83898.6 s3930.7 s8/82 8
RCB-1D Max Adj.
Procs
RCB-1D Speedup
RCB-1D Runtime
ParMETISRuntime
ParMETISSpeedup
ParMETISMax Adj.
Procs
# of Procs
![Page 59: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/59.jpg)
59
RCB Scalability Leveling Off
![Page 60: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/60.jpg)
60
RCB Scalability Leveling Off
![Page 61: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/61.jpg)
61
Coupler Port Grouping Complication
![Page 62: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/62.jpg)
62
Coupler Port Grouping Complication
![Page 63: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/63.jpg)
63
H60VG3 RDDS Partitioning (w/ coupler port grouping)
1.0 ns runtimeIBM SP3 (NERSC)
2157.6/512531.1 s438.1 s70.4/256 12512
1159.3/256516.5 s442.9 s 69.7/256 9256
651.1/128599.0 s649.6 s 47.5/128 7128
334.4/64889.3 s995.1 s31.0/64 764
216.8/321820.7 s2158.3 s14.3/32 332
212.7/162405.4 s4257.0 s7.2/16 316
28/8 3826.5 s3856.2 s8/8 2 8
RCB-1D Max Adj.
Procs
RCB-1D Speedup
RCB-1D Runtime
ParMETISRuntime
ParMETISSpeedup
ParMETISMax Adj.
Procs
# of Procs
![Page 64: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/64.jpg)
64
Constrained Mesh Partitioning
![Page 65: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/65.jpg)
65
RDDS Coupler Cell Constrained Partition (16 procs)
7 RIB-3D 5 RIB-2D-yz
14 RIB-2D-xz 6 RIB-2D-xy 8 RCB-3D 6 RCB-2D-yz 14 RCB-2D-xz 5 RCB-2D-xy
14 RCB-1D-z 8 ParMETIS
14 HSFC-3D Max Adj. ProcsMethod
![Page 66: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/66.jpg)
66
RDDS Coupler Cell Constrained Partition (32 procs)
12 RIB-3D 6 RIB-2D-yz
29 RIB-2D-xz 7 RIB-2D-xy 11 RCB-3D 7 RCB-2D-yz
29 RCB-2D-xz 7 RCB-2D-xy
29 RCB-1D-z 14 ParMETIS17 HSFC-3D
Max Adj. ProcsMethod
![Page 67: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/67.jpg)
67
Future Work I Would Have Done
• Stitching Multiple Partitions Together• Competition• Onion Partition growing• Dynamic Partitioning for Track3P
![Page 68: Parallel Mesh Partitioning at SLACmmwolf/presentations/CS591MH/CS591M… · 17/09/2003 · 11 Motivation for New EM Capability Modeling RDDS Cell with standard accelerator code MAFIA](https://reader034.vdocuments.site/reader034/viewer/2022042923/5f717c50da8aa80099623256/html5/thumbnails/68.jpg)
68
Acknowledgements
• SNL– Karen Devine, et al.
• LBNL– Ali Pinar
• SLAC (ACD)– Adam Guetz, Cho Ng, Lixin Ge, Greg
Schussman, Kwok Ko
• Work supported by U.S. DOE ASCR & HENP Divisions under contract DE-AC0376SF00515