Thomas Gerhold Institute of Aerodynamics and Flow Technology German Aerospace Center (DLR)

DLR CFD Solver TAU & Flucs for external Aerodynamic

Industrial Use of EsDs – ETP4HPC Workshop 22 June 2017 Frankfurt

DLR TAU-Code

t = T

Parallel datamanagement

Dataextraction

Adaptation

(Re-)PartitioningDeformation

Primary grid

Solution

MPICPU-j

CPU-k

CPU-i

Transitionprediction

Preprocessor

Dual grid

Solver

direct (in memory) data access between modules

t = T + ∆t

python interface forscript basedwork flow

t = T

Parallel datamanagement

Dataextraction

AdaptationAdaptation

(Re-)PartitioningDeformationDeformation

Primary gridPrimary grid

SolutionSolution

MPICPU-j

CPU-k

CPU-i

TransitionpredictionTransitionprediction

Preprocessor

Dual grid

Solver

Preprocessor

Dual gridDual grid

SolverSolver

direct (in memory) data access between modules

t = T + ∆t

python interface forscript basedwork flow

python interface forscript basedwork flow

Unstructured 2nd Order Finite Volume compressible CFD solver Used in European aircraft industry, research organization and academia Since more than 15 years > 100 daily/frequently users today

Used e.g. for A380 & A350 wing design

Pre-Flight prediction

1% L/Dmax

Mach

Pre-Flight prediction

1% L/Dmax

MachSource: Airbus

Numerical Simulation of Aircraft Aerodynamics Status Cruise

CFD-based scaling

W/T-based scaling A380 Performance

CFD & wind tunnel based performance prediction

are comparable

flight test

Europe’s Vision for Aviation

Maintaining Global Leadership & Serving Society’s Needs

ACARE 2020 / Flightpath 2050

Goals (relative to typical aircraft in 2000)

CO2 emissions reduced by 75% NOx emissions reduced by 90% 65% reduction in perceived

aircraft noise

-ACARE: Advisory Council for Aeronautics Research in Europe

Consequence Heavy demands on future product performance Step changes in aircraft technology required New design principles mandatory

Future aircraft Design may be driven by unconventional layouts Flight characteristics may be dominated by

non-linear effects

Numerical Simulation Key Enabler for Future Aircraft Design

High-fidelity methods indispensible for design & assessment of step changing aircraft

Reliable insight to new aircraft technologies Comprehensive sensitivity analysis with

risk & uncertainty management Best overall aircraft performance through integrated

aerodynamics / structures / systems design Consistent and harmonized aerodynamic and

aero-elastic data across flight envelope

Further improvement of simulation capability necessary

Numerical Simulation for Aircraft Design Current Status

Computational Fluid Dynamics (CFD) has significantly evolved over the last 30 years

Mature tool for configurations at their design point in flight envelope

Complementary to wind tunnel testing and flight tests Key tool for aeronautical research and

aircraft technology development Total potential not yet fully exploited:

full flight envelope, all relevant disciplines, multidisciplinary optimization

Numerical Simulation for Aircraft Design Vision: Digital Aircraft

Challenge: consider algorithmic and parallel efficiency and trade off

Collaboration of experts in application domain and parallel computing necessary to extract – or even generate – data parallelism in algorithms

+ Related research and prototypic Implementations done in the recent years

algorithmic efficiency

parallel efficiency

HPC for Aircraft Design

Next Generation Solver Flucs Activity started mid of 2012, supported by several prototype codes

DLR CFD Code Strategy

Flucs – what is new New code design – started from scratch C++ 11 / Python

Finite Volume & HO Discontinous Galerkin & zonal coupling

Focus on algorithmic efficiency: use of strong implicit solvers Focus on HPC

GASPI / MPI + OpenMP (2 level parallelization)

Sub-partitions per node/socket are assigned to threads, thread- synchronization avoided wherever possible

Implementation supports overlap of communication & computation

Flucs – what is new

Airbus accepted Flucs as a basis for their next generation CFD capability further development needed EIS planned ~2020/21

HPC for Aircraft Design Status TAU

Ivy Bridge 2x12 cores

95%

78%

68%

89%

65% 48%

31%

parallel efficiency (vs 24 cores)

80%

50%

63%

73%

scalability limit

~6.5K points/core

TAU: pure MPI Parallelization (typical grid sizes between 20 -100 Mpoints)

4W multigrid vs. SG gain ~factor 5-10

HPC for Aircraft Design Status TAU TAU: pure MPI Parallelization, effect of interconnect (RK 31 mio points)

Aries Benchmark by M. Pütz, Cray Europe

FDR fat tree interconnect

Cray Aries interconnect

-6/29/2017

Exa2ct Slide 15

0

20

40

60

80

100

0 5000 10000 15000 20000 25000

linear

comm_free

mpi_bsync_se_gs

mpi_early_recv_se_gs

mpi_async_se_gs

gaspi_bsync_gs

gaspi_async_gs

mpi_fence_bsync_se_gs

mpi_fence_async_se_gs

mpi_pscw_bsync_se_gs

mpi_pscw_async_se_gs

F6 Full Model 31 mio mesh points, 3 V Multigrid • 1345 mesh points / thread @level 1 • 180 mesh points / thread @level 2 • 24 mesh points / thread @level 3

Strong Scaling

TAU CFD-Proxy Xeon Phi EXA2CT Christian Simmendinger

Flucs Parallel Efficiency – Strong Scaling Scenario 1.9 Mio Elements @ C²A²S²E Cluster

40

80 200 80%

(each with two 12-core IVB EP)

TAU @ 2 mio. points

20

Note: single grid results: no multi grid technique yet implemented

Flucs

Summary / Outlook Flucs

Flucs will become TAU successor for industrial applications

Still important functionality to be implemented: strong implicit solvers, multigrid method, further turbulence models, etc. – This is ongoing.

Increased scalability demonstrated vs. legacy pure MPI parallelization

Further work for improved HPC performance planned, e.g. SIMD usage EIS planned for 2020/21

We have interest to test Flucs on EsDs

Possibly NDA for Flucs solver necessary

Flucs is a python extension – use of dynamic libraries needed