unstructured grid structured grid amr based on...

Copyright © Takayuki Aoki / Global Scientific Information and Computing Center, Tokyo Institute of Technology

GP GPUGP GPU

AMR based on Space-filling Curvefor Stencil Applications

AMR based on Space-filling Curvefor Stencil Applications

1

Takayuki Aoki

Global Scientific Information and Computing CenterTokyo Institute of Technology


GP GPUGP GPUMotivationMotivation

2

Prof. Nakahashi

Prof. Yoshimura

Unstructured Grid Structured Grid

Coalesced Memory Access High accuracy Long stencil

High Performance


GP GPUGP GPUFlows with sharp interfacesFlows with sharp interfaces

■ Navier-Stokes solver：Fractional Step■ Time integration：3rd TVD Runge-Kutta■ Advection term：5th WENO■ Diffusion term：4th FD■ Poisson：MG-BiCGstab■ Surface tension：CSF model■ Surface capture：CLSVOF(THINC + Level-Set)

Particle Methodex. SPH

Mesh Method (Surface Capture)

Low accuracy< 106-7 particles

High accuracy > 108-9 mesh points

not splash

Numerical noise and unphysical oscillation

Gas Liquid Two-phase Flows

A drop on the dry floor

6

Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology

GP GPUGP GPUIndustrial Appl. Steering OilIndustrial Appl. Steering Oil

7 Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology

GP GPUGP GPUDevelopment New MaterialsDevelopment New Materials

Material Microstructure

Dendritic Growth

Mechanical Structure

Improvement of fuel efficiency by reducing the weight of transportation

Developing lightweight strengthening material by controlling microstructure

Low-carbon society


GP GPUGP GPUInterface between Solid and LiquidInterface between Solid and Liquid

Phase-field

0

1

Phase A

diffusive interfacewith finite thickness

Phase B

Mesh Adaption


GP GPUGP GPU

11

AMR（Adaptive Mesh Refinement）AMR（Adaptive Mesh Refinement）

Octrees and Space Filling Curves


GP GPUGP GPU

FDM & FVMFDM & FVM

12

FDM(Finite Difference Method)

Node Center AMR

FVM(Finite Volume Method)

Cell Center AMR


GP GPUGP GPU

13

Compressible Fluid SimulationCompressible Fluid Simulation

Heavy fluid lays on light fluid and unstable.

Euler equation:

0

yxtFEQ

evu

Q

pueuuv

puu

2

E

pvevpv

uvv

2F

IDO-CF Scheme512 x 512

Y. Imai, T. Aoki and K. Takizawa, J. Comp. Phys., Vol. 227, Issue 4, 2263-2285 (2008)


GP GPUGP GPU

IDO-CF schemefor Compressible CFD

IDO-CF schemefor Compressible CFD

・Combination of FDM and FVM・Multi-Moment interpolation・Higher-order accuracy・Less dispersive and dissipative

Y. Imai, T. Aoki and K. Takizawa, J. Comp. Phys., Vol. 227, Issue 4, 2263‐2285 (2008)


GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.

PV XI

YI XYI

51 flop

83 flop

83 flop

55 flop


GP GPUGP GPU

16

Directional Splitting (1)Directional Splitting (1)Euler equation:

0

yv

xu

t

02

yuv

xpu

tu

02

ypv

xuv

tv

0

ypvev

xpueu

te

0

xu

t0

2

xpu

tu

0

xuv

tv 0

xpueu

te

0

yv

t0

yuv

tu

02

ypv

tv 0

ypvev

te


GP GPUGP GPUDirectional Splitting (2)Directional Splitting (2)

Available for explicit time integration of hyperbolic equations:

0

)(

,0

)(

,0

)(

wpepww

vwuww

z

ewvu

t

vpewv

pvvuvv

y

ewvu

t

upewuvu

puuu

x

ewvu

t

1***

nn

ewvu

ewvu

ewvu

ewvu x-directional

integrationy-directionalintegration

z-directionalintegration



PV XI

YI XYI

0 fuuff xxt xufuff ii

tx

)()( 1

0 fufuf yxx

yt

y

xufuff i

yi

y

txy

)()( 1



PV XI

YI XYI

0 fvvff yyt

yvfvf

f jjt

y

)()( 1

yvfvf

f jx

jx

txy

)()( 1

0 fvfvf xyy

xt

x



Direct Method Directional Splitting Method



Direct Method Directional Splitting Method


GP GPUGP GPUConservative Semi-LagrangianConservative Semi-LagrangianNumerical fluxes are determined by integrating interpolation function (Lagrangian Polynomials) 0

xu

t

if

if

tt

tudt

if 1f i

ix 1ixpx

xx

x

Xi

i

idxxFf )(2/1

z = ( 1.0/16.0*u[j‐2] ‐ 9.0f/16.0*u[j‐1] ‐ 9.0/16.0*u[j+1] + 1.0/16.0*u[j+2] )*dt/dx;zz = z*z; zzz = zz*z;

fn[j] = f[j] – 1.0/6.0*(zzz ‐ z)*f[j+2] + (1.0/3.0*zzz + 0.5*zz + 5.0/6.0*z)*f[j+1]+ (1.0/6.0*zzz ‐ 0.5*zz + 1.0/3.0*z)*f[j] + 1.0/6.0*(zzz ‐ z)*f[j‐1];


GP GPUGP GPUCPU, MIC, GPU PerformancesCPU, MIC, GPU Performances

53.6 59.5 57.1

457

866958

1218

0

200

400

600

800

1,000

1,200

1,400

Perf

orm

ance

[GFl

ops]

Xeon E5‐2600

Xeon Phi3110P

Xeon Phi5110P

TeslaM2050

TeslaK20C

TeslaK20X

GeForceGTX Titan

Shared Memory Use in the x-directional kernel. Super function unit Loop unrolling Variable reuse in the y- and z- loops Reduction of branch diverges

Kepler GPU Tuning


GP GPUGP GPU

1 1 1 1 1

1 1 132

Some leaves extinct by coarseningf

De-fragmentationDe-fragmentation

Defragmentation by re-numbering

1 1 1 1 1

new leaves are generated by refinement.

1 1 132 1 1

GPU memory pool


GP GPUGP GPU

Space-Filling CurveSpace-Filling Curve

Hilbert CurveHilbert Curve Morton Curve


GP GPUGP GPU

Configuration 0 Configuration 1 Configuration 2 Configuration 3

Configuration 4 Configuration 5 Configuration 6 Configuration 7

Base of Hilbert CurveBase of Hilbert Curve


GP GPUGP GPULeaf RefinementLeaf Refinement

Configuration 0 Configuration 2


GP GPUGP GPULeaf RefinementLeaf Refinement

Configuration 5 Configuration 7


GP GPUGP GPU

Hilbert CurveHilbert Curve


GP GPUGP GPU

Morton CurveMorton Curve


GP GPUGP GPU

Domain DecompositionDomain Decomposition


GP GPUGP GPU

Domain DecompositionDomain Decomposition

Level 0 Primitives Level 1 Refinements

3D Mesh Refinement


GP GPUGP GPU

: Level-Set function(distance function)

: Heaviside function

The Level-Set methods (LSM) use the signed distancefunction to capture the interface. The interface isrepresented by the zero-level set (zero-contour).

Re-initialization for Level-Set function

Fig. Takehiro Himeno, et. Al., JSME, 65-635,B(1999),pp2333-2340

Level-Set method (LSM)Level-Set method (LSM)

Advantage : Curvature calculation, Interface boundaryDrawback : Volume conservation


GP GPUGP GPU

Anti-diffusive Interface CaptureAnti-diffusive Interface Capture

・VOF(volume of fluid) type interface capturing method・Flux from tangent of hyperbola function・Semi-Lagrangian time integration

[ Xiao, etal, Int. J. Numer. Meth. Fluid. 48(2005)1023 ]

・1D implementation can be applied to 2D & 3D → Simple

・Finite Volume like usage＊ THINC is the method how to compute flux

→ 3 krenel (x, y, z) can be fused to 1 kernel. Merit in memory R/W

THINC (tangent of hyperbola for interface capturing) Scheme


GP GPUGP GPUThinc WLICThinc WLIC

WLIC – splitting the interface intospatial directions

Weighting factors depending on the normalDirections.

: nomal vector to the interface

Calculation of the normal vector


GP GPUGP GPU

39

Advection of VOFAdvection of VOFNumerical Diffusion of Advection Computationsfor VOF (Volume of Fluid) depending on mesh resolutions.

128×128 256×256 ???×???

Thinc WLIC Scheme for Anti-diffusion:


GP GPUGP GPUOctree-based GPU-AMROctree-based GPU-AMR

GPUleaf

Octreedata structure

Hilbert Space-filling Curve

Interface Adaptation

高解像度が必要な界面に動的に細かい格子を集め、計算領域全域を細かくした場合の数%の格子点数で効率的に計算する。

leaf memory is managed on CPU


GP GPUGP GPURayleigh-Taylor InstabilityRayleigh-Taylor Instability

41

Density Profile Level Set Function


GP GPUGP GPURayleigh-Taylor InstabilityRayleigh-Taylor Instability

42


GP GPUGP GPUOctree-based GPU-AMROctree-based GPU-AMR

Interface Adaptation


GP GPUGP GPU

Thank youThank you

44

unstructured grid structured grid amr based on...

Documents