HPC for CAE

HPC for CAE with ANSYS

Peter Tiefenthaler & Christof Gebhardt

CADFEM GmbH

Herbert Güttler

MicroConsult GmbH

HPC for CAE

- 1 -

High Performance Computing (HPC) at ANSYS:

The need for speed

Usage scenarios

From workstations to cloud solutions

The software side

Applications

Licensing

HPC for CAE

- 2 -

HPC Defined

High Performance Computing (HPC) at ANSYS:

An ongoing effort designed to remove

computing limitations from engineers who

use computer aided engineering (CAE) in all phases

of design, analysis, and testing.

It is a hardware & software initiative!

HPC for CAE

- 3 -

Need for Speed

Larger FE Models

Assemblies instead of single parts and defeaturing

Lots of computations!!

Mesh fidelity continues to increase

More equations to solve

More computations!!

More complex physics being analyzed

Nonlinear, Dynamics, Multiphysics

More computations!!

Parametric studies, Optimization

More computations!!

HPC for CAE

- 4 -

HPC Revolution

Recent advancements have revolutionized the

computational speed available on workstations

- Multicore processors

- Large amounts of RAM

- SSDs replacing conventional HDs

- (Multi-) GPU support

HPC for CAE

- 5 -

Workstation

Local High End Computing power

Local High End 3D Graphics

Up to 16 (24) Cores and 512 GB Memory

Benefit

All kind of sequential simulation processing

Desktop Workstation

Z1 All-in-One Workstation

Mobile Workstation

HPC for CAE

- 6 -

Compute Server

Remote High End Computing power

No 3D Graphics

Scalable in cores, memory, disks

Redundant components

Benefit

All kind of sequential and simultaneous

simulation processing

Highly scalable in the number of cores

per job

Highly scalable in the number of

simultaneous jobs large DoE‘s

HPC for CAE

- 7 -

Usage Scenario Workstation + Compute Server

Workstation + Compute-Server

Small Jobs locally on Workstation

Pre/Postprocessing locally on Workstation

Big computing jobs centrally on compute server

Load sharing useful

Simulation data

HPC for CAE

- 8 -

Usage Scenario ‚Virtual Cluster‘

HPC for CAE

- 9 -

The bandwidth bottleneck:

Workstation + Compute-Server

The amount of data sent to the

‚number cruncher‘ is usually small

Performance benefits from using

a remote server are often canibalized by

times spent for data transfer back to the

workstation prior to postprocessing.

Can be TBs of data across nodes

for cluster setup

Results 1…100GB Model 0,1..1 GB

GB..TBs

HPC for CAE

- 10 -

Blade Workstation

Workstation inside a rack (cluster system)

3D Graphics inside the rack

Redundant components

16 Graphics Workstations in C7000 enclosure

HPC for CAE

- 11 -

Usage Scenario Blade Workstations

Blade Workstations + Compute Server +

Terminals

Small Jobs + Pre/Post on Blade

Workstation

Big computing jobs on compute server

Blade Workstation graphics access by thin

clients (terminals)

Exclusive usage per user

Compression of 3D visualization data in

software (HP RGS, Citrix HDX 3D) or

hardware

Benefit

High bandwidth connection from blade

workstation to compute server fast

postprocessing

Simulation data

3D visualization data

HPC for CAE

- 12 -

Virtual Workstations

Multiple virtualized workstation on a

single physical set of hardware

Software program behaves like

hardware

Existing solutions for 3D-applications:

VMWARE, NICE DCV, Citrix

HPC for CAE

- 13 -

Usage Scenario Virtual Workstations + Compute

Server Virtual Workstations + Compute

Server + Terminals

Small Jobs + Pre/Post on Virtual

Workstation

Big computing jobs on compute server

Blade Workstation graphics access by

thin clients (terminals)

Non-Exclusive usage GPU server

Compression of visualization data in

software

Benefit

High bandwidth connection from blade

workstation to compute server fast

postprocessing

Flexible allocation of virtual

workstations cost effective

„workstation“ usage by multiple users

Simulation data

3D visualization data

0010101001110

010101001011

010100100100

1000111010110

1011110101011

101010100101

HPC for CAE

- 14 -

Terminals & Cloud

Benefit

High bandwidth connection from blade

workstation to compute server fast

postprocessing

Flexible allocation of virtual

workstations cost effective

„workstation“ usage by multiple users

Flexible scaling of hardware resources

better scaling and availability by

external hardware sharing

Simulation data

3D visualization data

HPC for CAE

- 15 -

Parallel Processing – Hardware

2 Types of memory systems

- Shared memory parallel (SMP) <= single box, workstation/server

- Distributed memory parallel (DMP) <= multiple boxes, cluster

Workstation Cluster

HPC for CAE

- 16 -

Parallel Processing - Software

2 Types of Parallel Processing

ANSYS (SMP)

- Shared memory parallel (-np > 1)

• Can only be used on single machine

Distributed ANSYS or DANSYS (DMP)

- Distributed memory parallel (-dis –np > 1)

• Can be used on single machine or cluster

• Requires 3rd party software (MPI)

Microsoft MPI on Windows HPC clusters

Platform MPI (HP MPI) on Windows workstations/servers

Platform MPI (HP MPI) on Linux workstations/servers

Intel MPI on Windows/Linux workstations/servers

HPC for CAE

- 17 -

Parallel Processing - Hardware

GPU acceleration (-acc)

- First available in v13.0 SMP using NVIDIA GPUs

- At v14.0 available for DMP

- V14.5 supports multiple GPUs

- Can be used on single machine or cluster

HPC for CAE

- 18 -

Distributed ANSYS Design Requirements

No limitation in simulation capability

Must support all features

Continually working to add more functionality with each release

Reproducible and consistent results

Same answers achieved using 1 core or 100 cores

Same quality checks and testing are done as with SMP version

Uses the same code base as SMP version of ANSYS

Support all major platforms

Most widely used processors, operating systems, and interconnects

Supports same platforms that SMP version supports

Uses latest versions of MPI software which support the latest interconnects

HPC for CAE

- 19 -

Distributed ANSYS – Domain Decomposition

• Distributed steps (‐dis ‐np N)

– At start of first load step, decompose

FEA model into N domains.

– Each domain represents a single

process and is solved on a different

core

– Solving generates a lot of data traffic!!

– Lots of communication required to

achieve solution

– All domains should be equal in size

– Each process writes its own sets of

files (file0*, file1*, file2*,…, file[N‐1]*)

– Results are automatically combined at

end of solution

HPC for CAE

- 20 -

Distributed ANSYS – Domain Decomposition

domain 0

interprocess

communication processor 1

processor 0 (host)

processor n-1

domain

decomposition

…

elem

assemble

solve

domain 1 domain n-1

elem

assemble

solve

elem

assemble

solve

elem output elem output elem output

…

combining results

HPC for CAE

- 21 -

HPC for CAE

- 22 -

HPC for CAE

- 23 -

HPC for CAE

- 24 -

HPC for CAE

- 25 -

HPC for CAE

- 26 -

HPC for CAE

- 27 -

HPC for CAE

- 28 -

HPC for CAE

- 29 -

HPC for CAE

- 30 -

HPC for CAE

- 31 -

HPC for CAE

- 32 -

HPC for CAE

- 33 -

HPC for CAE

- 34 -

HPC for CAE

- 35 -

ANSYS Remote Solve Manager

HPC for CAE

- 36 -

ANSYS Remote Solve Manager

The RSM Client machine submits jobs to a Solve Manager running

on a remote machine.

The remote machine also acts as Compute Server.

HPC for CAE

- 37 -

ANSYS Remote Solve Manager

The RSM Client machine submits jobs to a Solve Manager running

on a remote machine.

The Solve Manager then assigns the job to remote Compute Servers

(Compute Cluster)

HPC for CAE

- 38 -

ANSYS Remote Solve Manager

HPC for CAE

- 39 -

ANSYS Remote Solve Manager

HPC for CAE

- 40 -

ANSYS Remote Solve Manager

HPC for CAE

- 41 -

ANSYS Remote Solve Manager

HPC for CAE

- 42 -

ANSYS Remote Solve Manager

HPC for CAE

- 43 -

Load Sharing

Using more than 1 node for a single job with the RSM requires

using a load sharing system

Currently ANSYS 14.5 supports the following Load Sharing Systems:

Microsoft Windows HPC Server 2008 R2

LSF (Load Sharing Facility) from Platform Computing

PBS Pro from Altair

PBS also available as freeware with strongly reduced functionality

HPC for CAE

- 44 -

Load Sharing

Load Sharing means to manage many different tasks (analysis jobs),

which are allocated to workstations, compute servers and compute

clusters.

Target: Every existing jobs

shall be distributed to the

available hardware resources

via intelligent management to

get optimal performance

HPC for CAE

- 45 -

Different Licensing Models

HPC Single Licenses

HPC Packs

HPC Workgroup Solution

HPC Enterprise Solution

HPC Parametric Pack Licensing

HPC for CAE

- 46 -

HPC Single Licenses

– Flexibility for customers using less than 8 parallel per solver seat and less than 128 parallel overall.

– 2 Cores for free in ANSYS basic license are added to the single HPC licenses available.

For example if customer buys 2 single HPC licenses, the ANSYS run can be started with 4 Cores parallel

HPC for CAE

- 47 -

HPC Packs

Release

4.5

HPC for CAE

- 48 -

HPC Workgroup Solution

HPC for CAE

- 49 -

HPC Enterprise Solution

HPC for CAE

- 50 -

HPC Parametric Pack Licensing

HPC for CAE

- 51 -

Conclusion

HPC offers many benefits for CAE

Applications ranging from workstations to cloud solutions

The framework (workbench / RSM) is established for many years

ANSYS supports a wide range of usage scenarios