FronTier and Applications to Convection Dominated Problems

Xiaolin Li, Wurigen Bo, James GlimmAnd the FronTier team

Department of Applied Math and StatisticsSUNY at Stony Brook

2

Outline of the talk1. Benchmark examples of FronTier2. Convection Dominated Problems3. Other problems4. Coupling FronTier with other packages5. Challenges to FronTier

3

A quotation from Albert Einstein

1. Stony Brook, AMS Department, galaxy cluster (over 500 processors)

2. Stony Brook, CEAS, Seawulf cluster

3. New York Blue: 103.22 teraflops

Major ComputingResources:

4

如果我们选择了最能为人类的福利而劳动的职业，我们就不会为它的重负所压倒，因为这是为全人类所做的牺牲，那时我们所感到将不是一点点自私而可怜的欢乐，我们的快乐属于千万人，我们的工作并不显赫一时，但将永远存在，而面对我们的骨灰，高尚的人们将会洒下热泪

卡尔.马克思

5

The Discrete Mesh Representation in Front Tracking

Volume fillingrectangular mesh(Eulerian Coord.)

(N-1) dimensionalLagrangian mesh(interface)

A 3D InterfaceA 2D Representation

Y

X

(i,j)

6

FronTierA software package based on

the front tracking method

7

Front Tracking: Scientific applications to interfacial physics, the methodology and software

1. 2006-2009, over 36,000 new lines of code, removal of hundreds of run-time bugs, set up benchmarks.

2. Collaborations with ANL, BNL, LLNL, ORNL, PNNL, Oxford.3. Over 20 publications and many conference presentations.4. Implemented iMesh interface, support SciDAC applications.5. Interoperability with Hypre and PETSc, coupling with parabolic and

elliptic solvers, implicit and Crank-Nicolson and incompressible solver.

Front tracking was first introducedBy Richtmyer in 1950’s. It was Used by Moretti for computationIn aerodynamics problems.

Extension to 2D and 3D startedBy J. Glimm and O. McBryanIn 1980’s at Courant Institute.

8

Interface Topological Changes

Algorithms to handle topological changesGrid free tracking (GF)Grid based tracking (GB)Locally grid based tracking (LGB)

Tangled interface

GF

LGBGB

9

Robust Locally Grid Based (LGB) Untangle

AdvantageLocal, it is suitable for large scale computing.Robust, It generates topologically valid surface mesh.

A robust algorithm to reconnect a grid based surface mesh with a grid free surface mesh

10

Interface Topological ChangesGrid based tracking is robust but too diffusive.Challenge: Robustness of the algorithm is crucial for large scale computing.

Grid based tracking Grid free tracking

11

Zalesak’s slotted disk (2D)

12

Interface Topological ChangesGrid based tracking is robust but too diffusive.Challenge: Robustness of the algorithm is crucial for large scale computing.

Grid based tracking Grid free tracking

13

Resolution Test

FronTier can provideResolution of surfaceDown to 1/200 of meshspacing

14

模拟维模拟模拟晶体结晶过程Resolution independent of mesh

15

Front tracking reversal test of interface in the deformation velocity field

16

2D Reversal Test

17

Front tracking reversal test of interface in 3D deformation velocity field

646464 ××

128128128 ××

0.0=t 0.3=t5.1=t

18

VOF by Kothe (2005)

0.3=t5.1=t 5.1=t 0.3=t

19

模拟维模拟模拟晶体结晶过程Deformation of 3D surface mesh

20

2D Topological Bifurcation

FronTier is equippedWith robust and Efficient bifurcationFunctions for meshesRepresenting thePropagating front

21

模拟维模拟模拟晶体结晶过程Topological merging of 3D surface mesh

22

ExamplesInterface bifurcation and merging are commonly observed in multiphase flow

mesh bifurcation in a curvature dependent surface propagation

mesh merging in a droplet collision simulation

23

εκ−= 0vvn

模拟维模拟模拟晶体结晶过程Geometry dependent interface velocity

24

Convection dominated PDE

• Hyperbolic equation• Hyperbolic-elliptic (NS equation)• Parabolic equation• Hyperbolic-parabolic (convection-diffusion)

25

FronTier Applications to the study of Rayleigh-Taylor Instability

(Coupling with hyperbolic Euler’s equation for gas dynamics)

26

Incompressible Rayleigh-Taylor instability

27

28

FronTier application: chaotic mixing

29

FronTier application: chaotic mixing

Chaotic mixing is not only important to ICF, but also a test of large scale FronTierapplication to petascale computing. We have implemented a load balanced parallel algorithm and ran up to 1024 processors on New York Blue. Collaboration with B. Cheng, John Grove, and D. Sharp at LANL.

30

3D Turbulent Mixing

31

FronTier Applications to the study of Richtmyer-Meshkov Instability

32

Inertial Confinement Fusion (ICF)

33

Simulation of Spherical RM

34

FronTier Applications to BiodieselJet Simulation

3D simulation for primary jet breakup

36

模拟三维内燃机喷嘴Ask not what the earth can do for us, ask what we can do for the earth

American consumes about 200 billion gallons per year,a 10% saving will be 20 billion gallon amounts to morethan 40 billion dollars, not to mention the benefit to theenvironment.

37

Burning biofuel in an engine

1. Understanding the difference between conventional fuel and biofuel: a) Ethanol used to blend with gasoline for automobile; b) Biodiesel blend with distillate petroleum fuels such as diesel, kerosene and heating oils for use in diesel engines, boilers possibly turbines.

2. How to burn cleanly and uniformly: The prerequisite for an optimized combustion is an optimized fuel injection spray.

3. What will affect the formation of spray: The physical properties of the fuel, the geometry of orifice and the application of pressure, the dynamics of the fluids through the nozzle.

38

Surface mesh evolution in jet simulation

39

Verification: Kelvin-Helmholtz Instability

0.03130.02050.0094

80160320

FT/GFMMesh per mode

The relative errors of the growth rate

Comparison with the dispersion relation dispersion relation

40

Verification: Rayleigh Instability

0.28530.17020.0672

0.13960.06070.0321

51020

FT/GFM (3D)

FT/GFM (2D)

Number of cells on radius

The relative errors of the growth rate

Comparison with the dispersion relation dispersion relation

41

Parallel Performance of FT

157.1159.81638416×16×64256×256×1024

157.1158.2819216×16×32256×256×512

157.1157.5409616×16×16256×256×256

157.1157.1204816×16×8256×256×128

Ideal(s)Time to solution(s)nCoresPartitionGrid

Performance of LGBJet simulation300-3million TrianglesBluegene/L 4096 cores

Weak scalingRayleigh InstabilityBluegene/L

42

High Pressure Diesel Injection

Importance: The fuel injection process plays an important role in the combustion and emissions performance of diesel engines.Experimental challenge: droplets in the spray from the breakup obscure the liquid core.Numerical challenge:

large range of spatial and temporal scaleslarge density ratio between fuel and airtopological changes of the jet surface

Figure source: Baumgarten, Atomization and Sprays, 16, (2006)

43

Nozzle Flow Simulation

Cavitation ModelA cavitation bubble is modeled as an interface between vapor and fuel.A cavitation bubble is dynamically inserted in the center of a rarefaction wave of critical strength.

Simulation setup

2D simulation 3D simulation Nozzle radius (R): 0.1mmGrid: 40/RFuel density: 0.66 g/cm3

Reynolds number: 30,000

Injector geometry: Macphee et.al. , Science, 295, (2002)

reservoir

nozzle

44

Nozzle Flow Simulation

As the flow reaches a statistically steady state, mean velocity, turbulence length scale and integral turbulence length scale are studies at the nozzle outlet

Calculated turbulence properties at the nozzle outletIntegral turbulence length scale (Λ) = 0.38RTurbulence intensity (Ti) = 0.025

Vorticity

45

Surface color is rendered by penetration velocity.

Simulation is performed with 4096 cores in BG/L at BNL.

Calculation time is about 12 days.

liquid film

Intact jet core

ligament

droplet

46*A. Lefebvre, Atomization and Sprays, 1989

Droplet Diameter Distribution

Droplet diameter is approximated from surface area d=(A/π)1/2.Sauter mean diameter (SMD) = Σd3/Σd2.The probability density function of droplet diameter distribution can be fitted by a log-normal distribution, which is consistent with experimental observations for high pressure injectors*.

time evolution of droplet diameter distribution

droplets from the simulation

47

1.23

1.22

Simulation

MMD/SMDMMD (μm)

SMD (μm)

1.224.525.520.22320.030R

1.223.724.519.52660.0250.38R

Correlation*Correlation*Simulation

Number of droplets from simulations

TiΛ

Droplet Diameter Distribution

* P.K. Wu, Atomization and Sprays, 3, (1993) MMD=Mass Medium Diameter

Λ=0.38R Ti=0.025 Λ=R Ti=0.030

Two simulations are performed with different turbulence inflow conditions.

69.05/3

3/22

'0

0 7604.01−

⎟⎠⎞

⎜⎝⎛ Λ

=⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ Λ

⎟⎟⎠

⎞⎜⎜⎝

⎛+

Λ dWe

SMDvUSMD

l

g

ρρ

An experimentally validated correlation*

48

3D Simulation of a Real Fuel Injection

All parameters are from an experiment performed by Parker*

nozzle radius (R) 0.1mmgrid 20/Rfuel density 0.66 g/cm3

gas density 0.0165 g/cm3

fluid viscosity 0.013 Poisesurface tension 24 mN/m2

Reynolds number 20,300Weber number 2.2×106

Ohnesorge number 0.073Density ratio 40

* P. Parker, Atomization and Sprays, 8, (1998)

49

FronTierand incompressible fluid

Navier-Stokes solver(Hyperbolic-Elliptic)

50

Incompressible fluid solver

51

Incompressible Rayleigh-Taylor instability on Atwood number

(From left: 0.82,0.33,0.14)

52

Incompressible Rayleigh-Taylor instability on Reynold number

(from left: 14,140,1400)

53

Incompressible code in 3D

54

FronTieron Fluid-structure interaction

55

Fluid-Rigid body interaction

Example: shock-rigid body (with biased center of mass) interaction

56

Fluid-Rigid body interaction

57

Wind power generator platform

58

FronTier and Parachute Simulation

1. Motivation2. Structure Dynamics (SD)3. Computational Fluid Dynamics (CFD)4. Couple Simulations

59

Three Stages of Parachute Dynamics

1. Deployment2. Inflation3. Terminal Descent

60

The Inflation stage of parachute

We pick up this phase for simulation because

1. Canopy geometry changes dramatically2. Canopy experiences largest force and is most stressed3. Parachute undergoes biggest deceleration4. A phase of “to be, or not to be”.

Mathematical and computational challenges

1. Process is highly nonlinear2. Requires robust geometry handling3. Needs accurate SD and CFD coupling

61

Previous Studies

K. R. Stein, R. J. Benney andT. E. Tezduyar, Journal of Aircraft, 2001.1. Unstructured grid, FEM2. Static mesh (133,097 nodes)3. Terminal descent

M. Accorsi, J. Leonard, R. BenneyAnd Keith Stein, AIAA Journal, 1999.1. Inflation study,2. SD simulation3. Fluid effects are approximated

Youngsam Kim and Charlie Peskim,Computer and Fluids, 20091. Penalty immersed boundary method2. Full 3D SD-CFD simulation

62

What is missing in FronTier

1. Incompressible solver: the speed of parachute is much less than sound speed (except for parachute of re-entry vehicle).

2. Mono component surface representation and functions: parachute surface is a fully immersed non-manifold surface.

3. Constrained surface propagation in fluid velocityfield: inflexible fabric or slightly flexible elasticsurface.

4. Collision handling: different from topologicalbifurcation.

63

The strategy of FronTier Parachute Simulation

1. Computational fabric modeling• Inflexible model• Linear spring model• Post-buckling response

2. Surface tension modeling

3. Collision and friction4. PDE coupling with immersed boundary (canopy)

• Penalty immersed boundary method (Y. Kim)• Immersed interface method (Z. Li)

dstsXxtstx )),((),(),( −= ∫ δFf

64

The Linear Spring Model

⎪⎩

⎪⎨⎧

<

≥−=

Lx

LxLxkE

ij

ijijs

0

)(21 2

Using the energy function:

⎪⎩

⎪⎨

⎧

<

≥−=

∂∂

−=∝Lx

Lxxx

Lxk

xEfa

ij

ijij

ijijs

iiti

0

)(

The acceleration and force on vortex:

65

Constrained and unconstrained curve propagation

66

Test simulations of flow around canopy

67

FronTier Applications to solute precipitation and crystal formation

68

Mathematical Model

s

esn

ess

ess

CCkdtdnv

dtCCkdn

CCkn

C

CDtC

ρ

ρ)(

)(

)(

−==

−=

−=∂∂

Δ=∂∂

The equation forsolute diffusion:

The interface propagation:

69

模拟维模拟模拟晶体结晶过程One Dimensional Solute Precipitation

70

Numerical discretization

1,2

)()2(

1)(2

)()2()(

11

1111

−=−+

=

+⎟⎟⎠

⎞⎜⎜⎝

⎛ −−

−+=

++

++++

drh

xChxCS

Sh

CCkh

xChxCxC

sn

sn

s

esn

sn

sn

sn

χχ

ρ

A semi-implicit front state solver

With correction of curvature

71

模拟维模拟模拟晶体结晶过程

0.320=aD

Two Dimensional Solute Precipitation

0Cs =ρ

72

模拟维模拟模拟晶体结晶过程Two Dimensional Solute Precipitation

640=aD

73

模拟维模拟模拟晶体结晶过程Two Dimensional Solute Precipitation

74

模拟维模拟模拟晶体结晶过程Three Dimensional Solute Precipitation

75

Precipitation with flow

CDCvtC

Δ=∇⋅+∂∂ )(

The equation for solute diffusion:

Where v is solved from the Navier-Stokes equation

76

Convection diffusion model

77

FronTierApplications to Other Problems

78

Pricing of American Option

( )

1

,)0,(,0)0,(

,0)(21

2

222

=∂

∂

−=

>−=≤=

−=

=−+∂∂

−∂∂

−∂∂

SC

ESC

ESESSCESSC

tT

CDSCS

SCSC

f

f

τ

γσγτ

The Black-ScholeEquation:

The interface Condition at all time:

Initial Condition:

79

Exercise point condition

1

1

11

1

,

,,

)(

+

−

+−

+

→

i

ii

fiii

nf

SSS

SSSS

S τAssume at

We need to find:

1+nτ

From B-S PDE:

From degenerated B-S (ODE):

80

One Dimensional American Option

81

Simulation of Cell Migration

82

Interoperation of FronTierwith other software

libraries

83

1. PETSc: TOPS supported linear solver library2. HDF4: To convert data into movies with color map3. GD: To produce interface movies4. VisIt: Visualization package by LLNL5. Weblinks:

http://sitsec.ams.sunysb.edu/trac/wiki/FronTierhttp://www.mcs.anl.gov/petsc/petsc-as/ http://www.hdfgroup.org/products/hdf4/http://www.boutell.com/gd/ https://wci.llnl.gov/codes/visit/about.html

84

Challenges to the front tracking method

85

Parallel load balancing

Like AMR, FronTier has encountered greatObstacle in load balancing and parallel scaling. One important development is adaptive partition load balancing.

86

FronTier and AMR

87

Space-time cells

Conservative tracking

88

FronTier for Dummies

1. Download and test runhg clone http://sitsec.ams.sunysb.edu/hg FronTiercd FronTierbuild.sh [–n –d –g]makecd example or application directoriesmakerun

89

Implementation

2. Start up/* Initialize basic computational data */

FrontInitStandardIO(argc,argv,&f_basic);FrontReadRegularDomainInfo(in_name,&f_basic);FrontStartUp(&front,&f_basic);FrontInitDebug(in_name);FrontInitVelo(&front,&velo_func_pack);FrontReadTimeControlInfo(in_name,&front);Problem specific initializations

90

Implementation

4. Time loopFrontProp(front);FrontSetTimeStep(front);Interior solver time step;for (;;){

FrontProp(front);Interior solver;FrontSetTimeStep(front);Interior solver time step;Output or exit;

}

91

User Query Functions

Int Dimension(interface);Int NumOfPoints(curve);Int NumOfPoints(surface);Int NumOfPoints(intfc);Int NumOfCurves(intfc);Int NumOfSurfaces(intfc);Int NumOfTris(surface);Int NumOfTris(intfc);

ArrayOfPoints(curve, coords);ArrayOfPoints(intfc, coords);ArrayOfCurves(intfc, curves);ArrayOfCurves(intfc, coords, vertex_indices);ArrayOfSurfaces(intfc, surfaces);ArrayOfTri(surface, tris);ArrayOfTri(surface, coords, vertex_indices);ArrayOfTri(intfc, tris);

92

Thank you