nachos numerical modeling and high performance computing ... · arbitrary high order discontinuous...

NACHOSNumerical modeling and high performAnce computing

for evolution problems in Complex domains andHeterogeneOuS media

General presentation of scientific activities

Stephane Lanteri

INRIA, nachos project-team2004 Route des Lucioles, BP 93, 06902 Sophia Antipolis Cedex, France

Evaluation of Theme Modeling, simulation and numerical analysisMarch 17-19, 2009

S. Lanteri (INRIA, nachos project-team) March 18th, 2008 1 / 34

Content

1 NACHOS project-team

2 Scientific objectives

3 Main achievements

4 Dissemination and visibility

5 Positioning and collaborations

6 Objectives for the next period


Outline



3 Main achievements





NACHOS project-team

History

Created in July 2007 (activities started in July 2006)

Follow-up of the CAIMAN project-team

Common project-team with J.A. Dieudonne Mathematics LaboratoryUMR CNRS 6621, University of Nice-Sophia Antipolis

Composition (permanent staff)

INRIA

Montserrat Argente [TR, project-team assistant]Loula Fezoui [DR2]Stephane Lanteri [DR2], scientific leader

JAD Laboratory, UNSA

Victorita Dolean [Assistant Professor]Francesca Rapetti [Assistant Professor, HDR]

Ecole des Ponts ParisTech, CERMICS

Nathalie Glinsky [CR]


Outline



3 Main achievements





Scientific objectives

A lot of challenging problems of computational physicsare modeled by linear systems of PDEs with variable coefficients

Sources of difficulties

Variability of the coefficients

Heterogeneity in space of the propagation media

Poor knowledge of media characteristics

Computational domain

Irregularly shaped objects

Geometrical details or singularities

Dynamicity of the physical phenomena

Most real problems are unsteady with multiple time scales

Media characteristics can vary in time


Scientific objectivesLinear systems of PDEs with variable coefficients

Time domain problems

x ∈ Ω ⊂ IRd , t ∈ IR+ :∂U

∂t+

d∑i=1

Ai (x)∂U

∂xi= S(x, t)

Frequency domain problems

x ∈ Ω ⊂ IRd , ω ∈ IR+ : iωU +d∑

i=1

Ai (x)∂U

∂xi= S(x, ω)

The matrices Ai (x) characterize the media

Could be Ai (x, t) or Ai (x, ω) as well


Scientific objectivesPhysical domains and target applications

Computational electromagnetics

System of Maxwell equations

Interaction of electromagnetic fields with biological tissues

Human exposure to electromagnetic fields from wireless systemsIn collaboration with Orange Labs, Issy-les-Moulineaux center

Interaction of charged particles with electromagnetic fields

Electrical vulnerability of complex devicesIn collaboration with CEA DAM, CESTA center in Bordeaux

Computational geoseismics

System of elastodynamic equations

Interaction of seismic fields with geological media

Seismic risk assessmentIn collaboration with LGIT Laboratory in Grenoble andGeosciences Azur Laboratory in Sophia Antipolis


Scientific objectives: applications

Interaction of electromagnetic fields with biological tissues


Scientific objectives: applications



Scientific objectives: research directions

Arbitrary high order Discontinuous Galerkin (DG) methods

Formulation and analysis of DG methods on simplicial meshes

High order polynomial interpolation

Non-conformity (h-, p- and hp-adaptivity)

Numerical treatment of complex material models

Hybrid explicit-implicit time integration

Strategies for grid-induced stiffness in time domain problems

Domain Decomposition (DD) algorithms

Optimized Schwarz algorithms for wave propagation models

Hybrid iterative-direct parallel solvers for frequency domain problems

High performance computing

Algorithmics for modern parallel computing platforms


Outline



3 Main achievements





Main achievementsArbitrary high order Discontinuous Galerkin (DG) methods

Discontinuous Galerkin Time Domain (DGTD-Pp) methods

For the systems of Maxwell and elastodynamic equations

Heterogenous, isotropic, propagation media

High order nodal (Lagrange) polynomial interpolation (2D and 3D)

High order explicit leap-frog (LF) time stepping (2D and 3D)

Non-dissipative methods (centered fluxes)

Non-conforming formulations in 2D (both in h and p)

hp a priori convergence analysis

Discontinuous Galerkin Frequency Domain (DGFD-Pp) methods

For the system of Maxwell equations

Heterogenous, isotropic, propagation media

Arbitrary high order nodal (Lagrange) polynomial interpolation (2D)

Centered or upwind fluxes



Eigenmode in a PEC cavity (2D case)

Non-conforming mesh: 152 triangles (128 triangles in the refined region)

DGTD-P(p1,p2) method: p1 in the fine region and p2 in the coarse region

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

+ PhD thesis of Hassan Fahs (December 2008)




Non-conforming mesh: 152 triangles (128 triangles in the refined region)

Comparison between LF2/LF4 based DGTD-P(p1,p2) method

LF2 based DGTD-P(p1,p2) method LF4 based DGTD-P(p1,p2) method

1e-06

1e-05

1e-04

0.001

0.01

0.1

0 10 20 30 40 50 60 70 80 90

L2 e

rror

time

DGTD-P(3,2)

DGTD-P(4,3)

DGTD-P(5,4)

LF2 scheme 1e-06

1e-05

1e-04

0.001

0.01

0.1

0 10 20 30 40 50 60 70 80 90

L2 e

rror

time

DGTD-P(3,2)

DGTD-P(4,3)

DGTD-P(5,4)

LF4 scheme




LF2 based DGTD-P(p1,p2) method LF4 based DGTD-P(p1,p2) method

100

101

102

10−7

10−6

10−5

10−4

10−3

10−2

10−1

100

(DOF)1/2

L2 e

rro

r

DGTD−P(1,0), LF2DGTD−P(2,1), LF2DGTD−P(3,2), LF2DGTD−P(4,3), LF2DGTD−P(5,4), LF2DGTD−P(6,5), LF2

100

101

102

10−7

10−6

10−5

10−4

10−3

10−2

10−1

100

(DOF)1/2

L2 e

rro

r

DGTD−P(1,0), LF4DGTD−P(2,1), LF4DGTD−P(3,2), LF4DGTD−P(4,3), LF4DGTD−P(5,4), LF4DGTD−P(6,5), LF4

(p1, p2) (1,0) (2,1) (3,2) (4,3) (5,4) (6,5)LF2 1.30 2.23 2.08 2.27 2.13 2.17

LF4 1.05 2.20 3.01 4.21 4.50 4.48Asymptotic h-convergence orders


Main achievements

Hybrid explicit-implicit time integration

For the system of Maxwell equations (2D and 3D)

For overcoming grid-induces stiffness of fully explicit DGTD-Pp methods

Hybrid leap-frog/Crank-Nicolson partitioned scheme+ Originally proposed by Piperno, ESAIM: M2NA, 2006

Stability analysis based on energetic considerations

Computer implementation aspects

Partitioning of the mesh elements in explicit and implicit subsetsSparse direct solver for the linear system associated to the implicit elements

+ PhD thesis of Adrien Catella (December 2008)


Main achievements

Hybrid explicit-implicit DGTD-Pp method

Scattering of plane wave (F=200 MHz, λ = 1.5 m) by an aircraft

# vertices=360,495 and # elements=2,024,924

Edges length: Lm=9.166 10−3 m (≈ λ/163 m) and LM=6.831 10−1 m (≈ λ/2.2 m)


Main achievements



Geometric criterion: C(τi ) = 4minj∈Vi

sViVj

PiPj

Vi/Pi : volume/perimeter of τi

Vi : set of neighboring elements of τi

0.001

0.01

0.1

1

0 500000 1e+06 1.5e+06 2e+06 2.5e+06 0.001

0.01

0.1

0 500 1000 1500 2000

Distribution of the geometric criterion C


Main achievements



Cmax |Se | |Si |0.0125 2,024,320 604 (0.03 %)0.0175 2,022,464 2,460 (0.12 %)0.02 2,018,543 6,381 (0.31 %)

Definition of the subsets of explicit and implicit elements

Cmax RAM (LU) Time (LU) Time (total)

0.0125 m 12 MB 0.3 sec 6 h 39 mn0.0175 m 48 MB 1.5 sec 4 h 44 mn0.02 m 117 MB 4.2 sec 4 h 08 mn

Hybrid explicit-implicit DGTD-P1 method (Intel Xeon/2.33 GHz workstation)

Fully explicit DGTD-P1 method: 25 h 3 mn


Main achievements

Domain Decomposition (DD) algorithms

Solvers for the algebraic systems associated to DGFD-Pp methods

Schwarz algorithms for the frequency domain Maxwell equations

Formulation and analysis in the continuous case

Natural (low order) and optimized (high order) interface conditions

Design of discrete variants in the DG framework

Link with efficient algebraic sparse linear system solvers

Subdomain solversPreconditioners for interface systems

Preliminary contributions for low order DGFD-Pp methods+ PhD thesis of Hugo Fol (December 2006)

Extension to high order DGFD-Pp methods+ PhD thesis of Mohamed El Bouajaji (ongoing)


Main achievements

Hybrid iterative-direct Schwarz based solverInterface condition: Dirichlet on incoming characteristic variables

Substructuring technique at the discrete level

Full GMRES solver for the interface systemOptimized sparse direct subdomain solvers (MUMPS or PaStiX)

Scattering of a plane wave (F=300 MHz) by a missile geometry

# vertices=39,660 and # elements=205,485 (# d.o.f=4,931,640)


Main achievements

Scattering of a plane wave (F=300 MHz) by a missile geometry

# vertices=39,660 and # elements=205,485 (# d.o.f=4,931,640)

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

2 4 6 8 10 12 14

Nor

mal

ized

res

idua

l (lo

g sc

ale)

GMRES iteration

32 subdomains64 subdomains

Ns # iter GMRES RAM (LU) min/max Time (LU) Time (total)

32 11 1370 MB/1868 MB 240 sec 722 sec64 13 529 MB/ 682 MB 55 sec (4.3) 356 sec (2.0)

BULL Novascale 3045 cluster (Intel Itanium 2/1.6 GHz, InfiniBand)

CCRT (Centre de Calcul Recherche et Technologie)


Main achievements

High performance computingParallelization strategies for unstructured mesh based DG methods

Homogeneous CPU, DM or hybrid DM/SM, architecturesMessage passing programming modelMesh partitioning issues for static and dynamic computations(e.g. Vlasov/Maxwell, DG-PIC, coupled solver)

Human exposure to an electromagnetic wave radiated from localized sources

DGTD-P1 method DGTD-P2 method


Main achievements

High performance computingHuman exposure to an electromagnetic wave radiated from localized sources

DGTD-P1 method DGTD-P2 method

Method # d.o.f # CPUs CPU (min/max) Total time

DGTD-P1 21,342,084 512 1998 sec/2079 sec 2080 secDGTD-P2 53,355,210 512 7884 sec/7901 sec 7903 sec

BULL Novascale 3045 cluster (Intel Itanium 2/1.6 GHz, InfiniBand)

CCRT (Centre de Calcul Recherche et Technologie)


Outline



3 Main achievements





Dissemination and visibility

Publications in journalsApplied mathematics, scientific computing: 7

Computational physics: 9

High performance computing: 1

Other dissemination activitiesCo-organization of 2 CEA-EDF-INRIA schools

High performance scientific computing: algorithms, software tools andapplicationsNovember 2006, INRIA Paris - RocquencourtRobust methods and algorithms for solving large algebraic systems onmodern high performance computing systemsApril 2009, INRIA Sophia Antipolis-Mediterranee

Organization of a MS: High order methods for the solution of wavepropagation PDE models with applications to electromagnetics andgeoseismics at Waves 2009, June 2009, Pau

Co-organization of a MS: Toward robust hybrid parallel sparse solvers forlarge scale applications at SIAM CSE09, March 2009, Miami


Outline



3 Main achievements





Positioning and collaborations

Positioning within INRIAOngoing collaborations

SCALAPPLIX (Bordeaux - Sud-Ouest) on hybrid iterative-direct parallelsolvers for sparse linear systems (PhyLeaS associate team)CALVI (Nancy - Grand Est) on high order methods for the solutionof the Vlasov-Maxwell equations (ANR HOUPIC project)OASIS (Sophia Antipolis - Mediterranee) and PARIS (Rennes - BretagneAtlantique) on the use of modern distributed computing techniques forlarge-scale parallel simulations (ANR DiscoGrid project)

Foreseen collaboration with MAGIQUE3D (Bordeaux - Sud-Ouest) ondiscontinuous Galerkin methods for time domain geoseismics

Some common concerns with POEMS (Paris - Rocquencourt) ondiscontinuous Galerkin methods for time domain electromagnetics

Regular users of the unstructured mesh generation tools developed by GAMMA(Paris - Rocquencourt) and GEOMETRICA (Sophia Antipolis - Mediterranee)



Methodological aspects (external collaborators)

Sarah Delcourte (Claude Bernard University - Lyon 1)

DG methods for the time domain elastodynamic equations

Ronan Perrussel (Ampere Laboratory, UMR 5005, Ecole Centrale de Lyon)

DG and DD methods for the frequency domain Maxwell equations

Martin Gander (Mathematics Department, University of Geneva)

DD methods for the time domain and frequency domain Maxwell equations

Application/physical aspects

LEAT (Laboratoire d’Electronique, Antennes et Telecommunications)UMR 6071, Sophia Antipolis

Numerical modeling for time domain and frequency domain electromagneticsUltra-wideband microwave imaging (ANR MAXWELL project)

Geosciences Azur Laboratory, UMR 6526, Sophia Antipolis and LGIT(Laboratoire de Geophysique Interne et Tectonophysique), UMR 5559, Grenoble

Numerical modeling for time domain geoseismicsSeismic risk assessment (ANR QSHA project)



PhyLeaS associate teamSince January 2008

Study of parallel hybrid iterative-direct sparse linear solvers

Partners

Yousef SaadDepartment of Computer Science and EngineeringUniversity of Minnesota, USA

Matthias BollhoeferInstitute of Computational MathematicsTU Brunswick, Germany

Luc GiraudParallel Algorithms and Optimization Group, ENSEEIHT, Toulouse

SCALAPPLIX project-team, INRIA Futurs Bordeaux - Sud-Ouest

NACHOS project-team, INRIA Sophia Antipolis - Mediterranee


Outline



3 Main achievements





Objectives for the next period

High order DGTD and DGFD methodsTheoretical and numerical issues towards hp-adaptivity

Numerical treatment of complex propagation media models

Time integration schemes for DGTD methodsArbitrary high order explicit time schemes

Hybrid explicit-implicit strategies for grid-induced stiffness

DD methods for wave propagation problemsOptimized interface conditions for Schwarz algorithms

DG based discrete variants of optimized Schwarz algorithms

Efficient algebraic subdomain solvers (PhyLeaS associate team)

Software developmentsObject oriented framework in Fortran 2003

Algorithmics for hybrid CPU/GPU parallel architectures


Objectives for the next period

ApplicationsInteraction of electromagnetic fields with living tissues

Biological (thermal) effects of EM waves from wireless systems

In collaboration with Orange Labs, Issy-les-Moulineaux center

Medical applications (design of implantable micro-antenna forwireless monitoring systems)

In collaboration with Orange Labs, La Turbie center andthe LEAT Laboratory, Sophia Antipolis

Interaction of charged particles with EM fields

In collaboration with CEA DAM, CESTA center in Bordeaux


Seismic risk assessment

In collaboration with LGIT Laboratory in Grenoble andGeosciences Azur Laboratory in Sophia Antipolis

Seismic exploration

Foreseen collaboration with TOTAL(Depth Imaging Partnership INRIA strategic action)


nachos numerical modeling and high performance computing ... · arbitrary high order discontinuous...

Documents