parallel computing of fluid structure interactions...

PARALLEL COMPUTING PARALLEL COMPUTING OF OF

FLUID STRUCTURE INTERACTIONS PROBLEMS FLUID STRUCTURE INTERACTIONS PROBLEMS

Amit S. BaddiNovember 28, 2005

Advisor: Prof. Hasan U. Akay

Department of Mechanical EngineeringComputational Fluid Dynamics Laboratory

Indiana University Purdue University Indianapolis

MS Thesis Defenseovember 28, 2005

OutlineOutline

Introduction to Fluid-Structure Interactions i.e. Aeroelasticity

Algorithmic details of Loosely coupled approach for Aeroelastic flutter predictions

Results Validation and comparisons

Preconditioning formulation

Large scale computations on grid computing environments

Conclusions

Acknowledgements


Fluid Structure Interaction i.e. Fluid Structure Interaction i.e. AeroelasticiyAeroelasticiy

“Aeroelasticity is the phenomenon which exhibits appreciable reciprocal interactions (static or dynamic) between aerodynamic forces and the deformations induced in the structure of a flying vehicle, its control mechanisms, or its propulsion system.”Bisplinghoff (1975)

Aerodynamics

Flight Mechanics

Static Aeroelasticity

Dynamics Elasticity

Structural

Dynamics

Dynamic Aeroelasticity


Aeroelastic Flutter visualAeroelastic Flutter visual


Work Done in This ThesisWork Done in This Thesis

Mesh based parallel code coupling search and interpolation program was developed coupling the fluid and structure codes.

Multiple Programs Multiple Data (MPMD) paradigm of parallel computing was implemented.

Non-conservative local time stepping was replaced by constant time stepping resulting in improved correlation with the results available in the literature.

Termination criteria in case of mesh movement algorithm was modified resulting in reduced CPU time.

The present density based flow solver was modified to solve for the low speed flows by preconditioning the time derivative term.

Grid computing study was carried out on Teragrid in which multi-site execution was demonstrated.


FSIEDA FSIEDA

Algorithm DetailsAlgorithm Details


Calculate dynamic pressure forces on structure

Obtain dynamic response of structure using Newmark-Beta algorithm & mode-superposition

Extract nodal displacements on surface mesh

Solve the flow field using Implicit Euler equations

Extract nodal pressures on the surface mesh

Impose the displacements on CFD surface mesh

Deform CFD volume grid using Mesh movement algorithm

Interpolate pressures between non-similar surface meshes from fluid to structure domain

Interpolate displacements between non-similar surface meshes from structure to fluid domain

CSD Solver (SAPEDA)

Search & Interpolation

(SINEDA)

CFD Solver (FAPEDA)

Loosely coupled approach for simulating Aeroelastic flutterLoosely coupled approach for simulating Aeroelastic flutter


Computational Fluid Dynamics Solver Computational Fluid Dynamics Solver -- FAPEDAFAPEDA

Unsteady Euler solverUnstructured tetrahedral gridsArbitrary Lagrangian Eulerian (ALE) formulation Cell centered based Finite Volume spatial discretizationBackward Euler Implicit time integrationsLocal time stepping for Steady state solutions and Time accurate time stepping for Unsteady solutions Moving mesh for solution of moving boundary problems such as Fluid-Structure Interactions, etc.MPI for parallel communication


Computational Fluid Dynamics Solver Computational Fluid Dynamics Solver -- FAPEDA (Cont.)FAPEDA (Cont.)

Governing Equations for FAPEDAThe integral form of the Arbitrary Lagrangian-Eulerian (ALE) formulation of the 3D

time-dependent inviscid fluid-flow equations:

∫∫∫ ∫∫Ω Ω∂

=⋅+∂∂ 0ˆdSnFQdVt

r

∫∫ ∫∫Ω Ω∂

=⋅+ 0ˆ dSnFdVQr

TEwvuQ ρρρρ=where, is the vector of conserved flow variables

( )[ ]

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

+⋅−

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

+

==⋅

n

z

y

x

n

annn

Pnwu

pewvu

FnF

0

ˆˆ vvv

ρρρρ

is the convective flux vector with ALE formulation



The cell-centered finite volume formulation is employed.

The Backward Euler Implicit time integration scheme is used.

The discretized form is expressed as:

[ ]t

VQRQAn

nnn

∆∆

−−=∆

dSnQFR

QRI

tVA

nn

n

nn

∫∫ ⋅=

∂∂

+∆

=

Ω∂

+

rr)(

][][

1

∫∫Ω∂

++

⋅−=∆− dSnQFt

VQVQ nnnn

ˆ)( 11 v

where,

Rearranging,



Mesh-Movement Algorithm

The mechanism of this method is that any two neighboring nodes in the mesh are connected by a spring and the spring stiffness is inversely proportional to the distance of the two nodes.

Predictor corrector scheme is used implementing Jacobi iterative procedure.

2/1222 ])()()[( −−+−+−= ijijijm zzyyxxk

∑∑

∑∑

∑∑ ∆

=∆∆

=∆∆

=∆ +++

m

mni

m

mni

m

mni k

zkz

kyk

yk

xkx 111 ,,

Stiffness K

Displacements


Computational Fluid Dynamics Solver omputational Fluid Dynamics Solver -- FAPEDA (Cont.)FAPEDA (Cont.)

Mesh-Movement Algorithm (continued)

Termination criteria for Jacobi iterative procedure is modified, reducing the CPU time by over 75%.

Tolerancexxerror nj

nj ≤∆−∆= ∑ +

nodes internal2

1 ||||

TolerancexxMaxerror nj

nj ≤∆−∆= +

21

nodes internal||||

Previous termination criteria for Jacobi iterative procedure

Modified termination criteria for Jacobi iterative procedure


Computational Fluid Dynamics Solver Computational Fluid Dynamics Solver -- FAPEDA (Cont.)FAPEDA (Cont.)Boundary Conditions

The characteristic boundary conditions are applied using Riemann invariants on farfield boundaries.

For the moving boundaries, the velocity and wall acceleration must be taken into account

( )[ ]w

cw

annp

nWVnVVrr

rrrrrr

⋅−=∂∂

⋅−⋅−=

ρ











CSD Solver (SAPEDA)


(SINEDA)

CFD Solver (FAPEDA)



Finite element based dynamic structural solver

Quadrilateral shell elements used

Mode super-positioning is used to obtain the dynamic response of the structure

Newmark family of time integration used

Computational Structural Dynamics Solver Computational Structural Dynamics Solver –– SAPEDASAPEDA


Computational Structural Dynamics Solver Computational Structural Dynamics Solver –– SAPEDASAPEDA

The finite element discrete aeroelasticity element equation for a structural system can be written as

[ ] [ ] [ ] )()()()()()()( eeeeeee RqKqCqM =++ &&&

[M], [C] and [K] are system mass, damping and stiffness matrices

For static analysis, equation can be rewritten as

[ ] RqK =

For dynamic analysis, equation can be rewritten as

[ ] [ ] [ ] )( tRqKqCqM =++ &&&


Computational Structural Dynamics Solver Computational Structural Dynamics Solver –– SAPEDA (Cont.)SAPEDA (Cont.)

Mode superposition method

][][ 2 φωφ MK =

1. Get the generalized eigenvalue solution

3. Get the generalized displacement solution

*2 2iiiiiii FXXX =++ ωωξ &&&

2. Use first n modes to simulate structural response

)(][)(],......,,[)( 21 tXAtXtq n == φφφ


Computational Structural Dynamics Solver Computational Structural Dynamics Solver –– SAP4 (Cont.)SAP4 (Cont.)A Newmark-family of time integration scheme is used to obtain the solutionat the (n+1)th time step:

*]])[21(][)21(][1[

]])[221(][)21(][2[][

1***2

***2

1*

FXKCt

Mt

XKCt

Mt

XM

n

nn

+−+−∆−

+∆

−

++−−∆−

−∆

=

−

+

αβα

αβα

Always1/41/2The constant acceleration method

1/61/2The linear acceleration method

Always13/2The backward difference method

Always4/53/2Galerkin method

Stable Condition

βα

i

tω

32≤∆

Initial Condition:

For Flutter Analysis

Either or

00

00

XdtdX

XX

t

t

&=

=

=

=

0

0

0

0

=

≠

X

X& 0

0

0

0

≠

=

X

X&











CSD Solver (SAPEDA)


(SINEDA)

CFD Solver (FAPEDA)



unction of SINEDA: Couple Nonunction of SINEDA: Couple Non--similar meshessimilar meshes

Mesh in coupling region for Simulation Code 1

(FAPEDA)

Mesh in coupling region for Simulation Code 2

(SAPEDA)

Alternate Digital Tree (ADT) Geometric Search Algorithm

&

Bilinear Interpolation Algorithm

Element Type: Triangle

Usually dense meshes

Partitioned into blocks

Unstructured mesh

Element Type: Quadrilateral

Usually Coarse meshes

Unpartitioned

Structured mesh

SINEDAX

Y

Z

Y

Z


Search & InterpolationSearch & Interpolation-- SINEDASINEDA

Alternating Digital Tree (ADT) based geometric search algorithm

Special Geometric selection criteria incorporating minimum distance and barycentric coordinates

Bilinear interpolations for exchange of variables across non-similar meshes

MPMD paradigm of parallel computing

Based on Message Passing Interface (MPI)


earch & Interpolationearch & Interpolation-- SINEDA (continued)SINEDA (continued)

Alternating Digital Tree (ADT) generation

A

B

C

Right SonNodeLeft

Son

0A0

A

B

A

Right SonNodeLeft

Son

0B0

2A0

Right SonNodeLeft

Son

0C0

3B0

0A0A B

C

AB

A



D

A

B

CE

D

A

B

C

Right SonNodeLeft

Son

0D0

0C0

3B0

2A4

Right SonNodeLeft

Son

0E0

0D0

0C0

3B5

2A4AD

B

C

E

AD

B

C

Alternating Digital Tree (ADT) generation


X

Y

Z

X

Y

ZX

Y

Z

X

Y

Z

Alternating Digital Tree (ADT) Alternating Digital Tree (ADT) –– SINEDA (SINEDA (contndcontnd.).)

MESHMesh is stored in a form of

Alternating Digital Tree

CFD Mesh

CSD Mesh

CFD Mesh ElementBinary Tree

CSD Mesh Element Binary Tree



Alternating Digital Tree (ADT) search

A

B

D

E

F

G

H

a

b

C

X1

X2

E

C D G H

B F X1

X2

X1

A



Remote node need not lie exactly on the same plane of the local element.

Geometric selection criteria




Generalized model of a Point and Triangle in 3D space is used.

X

Y

Z

P1

P2

P3

P0

P’0

NP

V1

V2

V3

Remote node is projected onto the element plane.




ToleranceA

Adistdisterror

L

diffi ≤

++=

22 21

X

Z

P1

P2

P3

P0

P’0

NP

V1

V2

V3dist1

dist2

P”0

Y

X



ToleranceA

Adistdisterror

L

diffi ≤

++=

22 21

Inclusion of difference of Area term in error criteria is necessary.

ToleranceA

distdistA

dist

ELEL

≤⎥⎦

⎤⎢⎣

⎡ +≈⎥

⎦

⎤⎢⎣

⎡

2

22

1

2 211In case if,



But, difference of area is not the sufficient condition.

Inclusion of dist1 and dist2 in error criteria is also necessary.

ToleranceA

Adistdisterror

L

diffi ≤

++=

22 21


MPMD Paradigm of Parallel Computing MPMD Paradigm of Parallel Computing –– SINEDA (SINEDA (contndcontnd.).)

2

CODE1_WORLD

3(4)

0(1)

1(2)

2(3)

SINEDA_WORLD

0(0)

MPI_WORLD

MPI INTER-COMMUNICATORMPI INTRA-COMMUNICATOR

2

CODE2_WORLD

3(8)

0(5)

1(6)

2(7)

Local Process Rank(Global Process Rank)


AGARD 445.6 Wing: Pressure Distribution interpolated using SINEDAGARD 445.6 Wing: Pressure Distribution interpolated using SINEDA A

Pressure2.726752.658672.590582.52252.454412.386332.318242.250162.182072.113982.04591.977811.909731.841641.77356


AGARD 445.6: Displacements interpolated using SINEDA AGARD 445.6: Displacements interpolated using SINEDA

DISP0.00240.00220.00200.00190.00170.00160.00140.00130.00110.00090.00080.00060.00050.00030.0002


FSIEDA TestcaseFSIEDA Testcase

AGARD445.6 WingAGARD445.6 Wing


TEST CASETEST CASEAGARD Wing 445.6

1.833 ft

1.208 ft

45O

5.2 ft

X

Y

Z

Tetrahedral CFD Mesh168604 Nodes

1M cells

Quadrilateral CSD Mesh840 Nodes

800 Elements


AGARD 445.6 WingAGARD 445.6 Wing

Grid convergence study based on steady state results

-0.4

-0.2

0

0.2

0.4

0.6

0 0.2 0.4 0.6 0.8 1

X/C

Cp

2.5M

1M

260K

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 0.2 0.4 0.6 0.8 1

X/CC

p 2.5M

1M

260K

Coefficient of pressure distribution at M = 0.499 at root chord

Coefficient of pressure distribution at M = 1.141 at root chord


TEST CASES (Cont.)TEST CASES (Cont.)Flutter Analysis

Stable

Neutral

Unstable

µωαbUV f /∞=

Dynamic instability where-by the system extracts energy from the free stream flow producing a divergent response

The computed flutter characteristics are presented in terms of velocity index Vfwhich is defined as:


Flutter Analysis (contd.)lutter Analysis (contd.)

-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0.025

0 0.05 0.1 0.15

TIME (s)

GEN

ERAL

IZED

DI

SPLA

CEM

ENTS

MODE1MODE2MODE3MODE4

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0 0.05 0.1 0.15 0.2 0.25

TIME (s)

GEN

ERA

LIZE

D

DIS

PLA

CEM

ENTS

MODE1

MODE2

MODE3

MODE4

AGARD 445.6 wing M∞ = 0.901 and Vf = 0.35

AF = 0.875

Stable response


AF = 1.12

Unstable response

(2nd Peak Gen. Disp.)Mode1

(1st Peak Gen. Disp.)Mode1Amplification Factor (AF) =


inear Interpolation of Amplification Factor:inear Interpolation of Amplification Factor:

Amplification Factor(AF) Vs Dynamic Pressure (DP)

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.3 1.4 1.5 1.6 1.7 1.8 1.9

DP

AF AF

AF is linearly interpolated with Dynamic pressure to obtain Flutter boundary.

Vf = 0.35,DP = 1.396,AF =0.875

DP = 1.615,AF =1

Vf = 0.38

Vf = 0.4,DP = 1.823,

AF =1.12

M∞ = 0.901


Aeroelastic Flutter Boundary at Aeroelastic Flutter Boundary at M∞ = 0.901= 0.901

-0.02

-0.01

0

0.01

0.02

0 0.05 0.1 0.15 0.2 0.25TIME (s)

GA

NER

ALI

ZED

DIS

PLA

CEM

ENTS

(in)

MODE1MODE2MODE3MODE4


AF = 1.0

Neutral response (LCO)


lutter Stability Boundary For AGARD 445.6 Winglutter Stability Boundary For AGARD 445.6 Wing

0

0.2

0.4

0.6

0.8

0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2

Mach number M

Velo

city

Inde

x V

FSIEDAEXPERIMENTALBATINA-Euler[27]BATINA-NS[27]FSIEDA (He)[6]


reconditioning reconditioning

Introduction

Time-marching schemes in the density based solvers provide good stability and convergence characteristics when solving compressible flows at transonic and supersonic Mach numbers.

At low speeds, however, system stiffness resulting from dissimilar particle and acoustic velocities resulting in large condition number causes convergence rates to deteriorate.

Convergence can be made independent of Mach number by altering the acoustic speeds of the system such that all eigenvalues become of the same order and the condition number is made to approach unity.


reconditioning Formulation reconditioning Formulation

Governing equations

TTwvupQ =

∫∫∫∫∫Ω∂Ω

=⋅+∀∂∂ 0ˆ dSnFdWt

v

∫∫ =⋅+∫∫∫∂∂

ΓΩ∂

0ˆ dSnFdVQt

v

Integral form of 3D Unsteady ALE based Euler equation of Fluid flow

Preconditioned form of the 3D Unsteady ALE based Euler equation of Fluid flow

where, = Preconditioning matrixΓ= Preconditioned flow variables vector


reconditioning Formulationreconditioning Formulation

Dual time stepping

To maintain the time accuracy of the equations in case of transient solution dual time stepping is utilized and preconditioning is applied to pseudotime.

∫∫∫∫∫∫∫∫Ω∂Ω

⋅−=∂∂

Γ+∀∂∂ dSnFdVQdWt

ˆv

τ

TewvuW ρρρρ=where, = Conserved flow variable vector

The discretized form is:

[ ] nn

mn

mmmmm

Wt

Wt

RQQW

WR

QW

t ∆∀

+∆∀

−−=∆⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡∂∂

⎥⎦

⎤⎢⎣

⎡∂∂

+Γ∆∀

+⎥⎦

⎤⎢⎣

⎡∂∂

∆∀ +++ 111

τ

1

00,+→

=∴→∆∞→nm

m

WWLHSQmAs


Application of FSIEDA Application of FSIEDA on on

TeraGrid (TG)TeraGrid (TG)


NSF TeraGrid Extensible Terascale facilityNSF TeraGrid Extensible Terascale facility ((www.teragrid.orgwww.teragrid.org))

Computation sites

Log-in site (alsoNOC)

TeraGrid sites used in this project:


2

CFD_WORLD

3(4)

0(1)

1(2)

2(3)

SINEDA_WORLD0(0)

MPI_WORLD

CSD_WORLD

Local Process Rank(Global Process Rank)

0(5)

NCSA SDSC

MultisiteMultisite execution of FSIEDA on TGexecution of FSIEDA on TG


ConclusionsConclusions

A loosely coupled procedure is developed for prediction of Aeroelastic flutter boundary providing an effective alternative over strongly coupled approach.

ADT based efficient Geometric search and Interpolation program SINEDA is developed for obtaining mesh-based code coupling on non-similar meshes.

Flutter analysis was implemented by choosing initial perturbation of the structural system and examining whether the initial perturbation will decay, grow or maintain neutral conditions to determine the flutter conditions. The results obtained show good correlation with previous works of Batina and experimental results.

The accuracy improvement was obtained especially in the supersonic regime by implementing the constant time stepping over the local time stepping.

The modified termination criteria reduced the CPU time by over 75%.

Preconditioning formulation was developed by modifying the current density based transient Eulersolver with ALE formulation to solve for the low speed i.e. incompressible flows.

Explored the potential application of TeraGrid for solving large scale problems.

Multi-site execution was studied for solving potentially large scale problems utilizing multiple clusters on TeraGrid.


AcknowledgementsAcknowledgements

Prof. Hasan Akay

Prof. Akin Ecer, Prof. Hsu and Dr. Erdal Oktay

Dr. Erdal Yilmaz and Resat Payli

All the Faculty members and staff of the Mechanical Engineering Department

TeraGrid (www.teragrid.org) access to this project was supported by National Science Foundation (NSF) grant, TG-CTS050003T, under the following programs:

Partnerships for Advanced Computational InfrastructureDistributed Terascale Facility (DTF)Terascale Extensions: Enhancements to the Extensible Terascale Facility

parallel computing of fluid structure interactions...

Documents