organizers: bertrand rollin and jason hackl agenda · :4248-4267. cmt/artvisc.f. subroutine...

58
CCMT Agenda CMT-nek: Anatomy of the Beast Speaker: Jason Hackl | 1:30 pm – 2:15 pm Lagrangian Particles in CMT-nek Speaker: David Zwick | 2:15 pm – 2:45 pm Break Running CMT-nek Speakers: Goran Marjanovic and Brad Durant | 3:00 pm – 3:45 pm Post-Processing and visualization in CMT-nek Speakers: David Zwick and Brad Durant | 3:45 pm – 4:15 pm Lesson Learnt from CCMT’s Simulations Speakers: Fred Ouellet and Yash Mehta | 4:15 pm – 4:45 pm A Boot Camp on CMT-nek November 29, 2017 – CCMT’s seminar room Organizers: Bertrand Rollin and Jason Hackl

Upload: others

Post on 10-Sep-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Agenda

• CMT-nek: Anatomy of the BeastSpeaker: Jason Hackl | 1:30 pm – 2:15 pm

• Lagrangian Particles in CMT-nekSpeaker: David Zwick | 2:15 pm – 2:45 pm

Break

• Running CMT-nekSpeakers: Goran Marjanovic and Brad Durant | 3:00 pm – 3:45 pm

• Post-Processing and visualization in CMT-nekSpeakers: David Zwick and Brad Durant | 3:45 pm – 4:15 pm

• Lesson Learnt from CCMT’s SimulationsSpeakers: Fred Ouellet and Yash Mehta | 4:15 pm – 4:45 pm

A Boot Camp on CMT-nekNovember 29, 2017 – CCMT’s seminar room

Organizers: Bertrand Rollin and Jason Hackl

Page 2: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Participates

Page 3: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

CMT-NekDevelopers’ learning cliff

November 29, 2017

Jason Hackl

Page 4: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT4

CMT-nek developers’ learning cliff• Discretization of conservation laws

– Time marching

– Semi-discretized right-hand-side• Volume Terms: convective and diffusive

• Surface Terms: convective, then diffusive

• Where things happen– nek5000 time loop and cmt-nek execution branch

– compute_rhs_and_dt

• Governing equations– Euler equations of gas dynamics

– Artificial diffusive stress tensor

– Entropy viscosity method

• What gets done: some details

Page 5: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT5

CMT-nek: conservation laws discretizedflux = convective + diffusive

Particle source terms RDavid ZwickEquate coefficients of vT , left multiply by B-1

dU/dt = res1 = volume + surface + source

res1 = Iconv + Isfc + IKU + IGU + IGTU

volume terms done 1 element at a timesurface terms done for all faces at oncenek5000 time loop and cmt-nek execution branchcompute_rhs_and_dt fills res1 (CMTDATA) with RHS terms

BM1

Page 6: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT6

Time loop TVDRK3• core/drive1.f

• nek__multi_advance(kstep,msteps)• do i=1,msteps

call nek_advance• call cmt_nek_advance• call userchk

do stage=1,nstageif (stage.eq.1) call copy(res3(1,1,1,1,1),U(1,1,1,1,1),n)call compute_rhs_and_dtdo e=1,nelt

do eq=1,toteqdo i=1,nxyz1

c multiply u with bm1 as res has been multiplied by bm1 in compute_rhsu(i,1,1,eq,e) = bm1(i,1,1,e)*tcoef(1,stage)*res3(i,1,1,eq,e)

> +bm1(i,1,1,e)*tcoef(2,stage)*u(i,1,1,eq,e)+> - tcoef(3,stage)*res1(i,1,1,e,eq)

cmt/drive_cmt.f• set_tstep_coef sets tcoef

Page 7: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT7

drive1_cmt.f compute_rhs_and_dtvolume surface

convective

diffusive

call compute_primitive_varsif(stage.eq.1) thencall setdtcmtcall set_tstep_coefendif

call entropy_viscositycall compute_transport_props

call fluxes_full_field Iconvdo eq=1,toteq

call surface_integral_full(res1(1,1,1,1,eq),flux(ieq))enddo

call imqqtucall igtu_cmt(flux(iwm),flux(iuj),graduf) IGTU

res1 = Iconv + Isfc + IKU + IGU + IGTU

Page 8: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT8

drive1_cmt.f compute_rhs_and_dtvolume surface

convective

diffusive

do e=1,neltcall cmtusrf(e)call compute_gradients(e)do eq=1,toteq

call convective_cmt(e,eq) Iconvcall viscous_cmt(e,eq) IKUcall compute_forcing(e,eq)

enddoenddo

call igu_cmt(flux(iwp),graduf,flux(iwm)) IGUdo eq=1,toteq

call surface_integral_full(res1(1,1,1,1,eq),flux(ieq))enddo

res1 = Iconv + Isfc + IKU + IGU + IGTU

Page 9: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT9

drive1_cmt.f compute_rhs_and_dtvolume surface

convective

diffusive

ONE ELEMENT AT A TIME

All elements in sequence

Page 10: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT10

Thermodynamic stateTemperaturePressureTotal energy

CMT-nek Governing EquationsVolume fractions of

gas…

…and particles

Convective fluxes of

MassHc=(U2 ,U3 ,U4 )T

Momentum

Energy

Conserved variablesto primitive variables

Upwind numerical flux H*c

• Function of primitive variables (p+,-, u+,-,…)• Riemann solver

(Advection Upstream Splitting (AUSM))

Liou (1996) ”AUSM+” J. Comp. Phys.,129:362-382

Boundary conditions from Dirichlet U+ → AUSM+• Riemann invariants for inflow and outflow (extrapolated supersonic)• Reflect about wall-normal rst lines for walls (geometry already high-order)

No particles here; 𝜙𝜙𝑔𝑔 = 1Zwick L2.00012

Page 11: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT11

Artificial viscosity

Diffusive flux of…Mass

Momentum

Energy

X

1Guermond & Popov (2014) “Viscous regularization of Euler eqn and entropy principles” J. Appl. Math. 74:284-305

Artificial stress tensor1 (NOT Navier-Stokes) with AV that tracks the entropy residualNOT weighted by volume fraction 𝜙𝜙𝑔𝑔

AGRADUadu_evm

Page 12: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT12

Entropy viscosity method

due to entropy viscosity

Rs approx by finite diff between RK3 stages’ s

C0 by average Rs at face points. Smooth via

1Guermond & Popov (2014) “Viscous regularization of Euler eqn and entropy principles” J. Appl. Math. 74:284-305Guermond, Pasquetti & Popov (2011) J. Comp. Phys. 230:4248-4267

cmt/artvisc.fsubroutine entropy_viscositycall compute_entropy(tlag)call entropy_residual(tlag)call wavevisc(t(1,1,1,1,3))call resvisc(res2) ! overwrite res2call evmsmoothcall dsavg(res2) DONE IN USERVP

Page 13: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT13

Deville, Fischer and Mund (2002) Higher-Order Methods for Incomp. Flows CambridgeLottes & Fischer (2005) J. Sci. Comp. 24(1):45-78 http://nek5000.mcs.anl.gov , http://nek5000.github.io/NekDoc/

U- from element U+ from neighbor

U+ from neighbor

Boundary conditions weakly imposed through H*• Dirichlet: prescribe U+

• Neumann: prescribe flux directly

Discontinuous Galerkin SEM

Average

Jump

From the element’s point of view• Restriction operator E gets nodal values on my faces

• Gather-scatter operator QQT

adds them to my neighbor’svalues and gives us both the result

• All interelement communication through QQT

• I get U+ via QQT too

Page 14: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT14

DGSEM: diffusive

Baumann & Oden (1999) Comput. Methods Appl. Math. Engrg. 175:311-341

Quasi-linear flux Jacobian APrimal formNOT weighted by volume fraction 𝜙𝜙𝑔𝑔

Dolejší, V. (2004) Intl. J. Numer. Meth. Fluids 45:1083-1106Hartmann & Houston (2008) J. Comp. Phys. 227:9670-9685

Weighted residual, integrate by parts → primal form → DGSEM ops

flux = convective + diffusive

Hackl, Shringarpure, Koneru, Delchini & Balachandar (2017) in preparation for Computers & Fluids

imqqtu = [I-QQT]U

Page 15: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT15

CMT-nek geography: what lives where?Description Dimensions Var Include,common

Quad weight x J (lx1,ly1,lz1,lelt) BM1 nek/MASS, /MASS/

d/dr matrix (lxd,lxd) D,dt,… /dgradl/

Interpolation matrix (lxd,lx1) Jgl,jgt /dgradl/

Mesh Jacobian (lx1,ly1,lz1,lelt) JACM1,jacmi nek/GEOM, /GISO1/

Grid metric (lxd^3,ldim^2,lelt) rx nek/GEOM, /GISOD/

Flux at grid points (lxd^3,ldim) totalh nek/cmt/CMTDATA

Indices of face pts subroutine facind iface_flux nek/cmt/DG

Face interp matrix (lxd,lx1) jgl,jglt /dgrad/

Face quadrature (lxd,lxd) wghtf nek/cmt/DG, /facewz/

Jacobian on face (lx1,lx1) Jaco_c /SCRNS/

Numerical flux huge lots /CMTSURFLX/

Conserved variable (lx1,ly1,lz1,toteq,lelt)

u nek/cmt/CMTDATAU

J

Page 16: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT16

Spectral elements

Reference element

Page 17: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Do you have any questions?

Page 18: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Lagrangian Particles in CMT-nek

D. Zwick

Page 19: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT19

Background• Fluid solved in Eulerian reference frame

• Particles solved in Lagrangian reference frame

• Up-to-date particle code that will always work with particle example cases:– Source code:

• https://github.com/dpzwick/Nek5000/tree/particles_new

– Particle examples:• https://github.com/dpzwick/NekExamples

Source synced with UFCCMT

~ monthly

Page 20: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT20

Capabilities and Options• One-way coupling:

– Interpolation

– Integration

– Force models

– Heat transfer models

– Boundary conditions

– Restarting

– Injection

– Variable particle sizes

• Two-way coupling:– Projection filter type

– Projection filter width

– Super particle loading

– Number of ghost particles

• Four-way coupling:– Normal collision spring

coeffecient

– Normal collision damper

Page 21: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT21

Particle Files and Options

• Source code:– cmtparticles.f (Nek5000/core/cmt/)– CMTPART (Nek5000/core/cmt/)

• User code:– cmtparticles.usrp (case_directory/)– particles.inp (case_directory/)

Development, force models, driver routines, etc.

Common blocks, arrays, parameters, etc.

Body forces, initial particle distribution, pre-simulation, etc.

Input file, options, etc.

Page 22: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT22

Main Datastructures

• rpart(i,j) and ipart(i,j)– Real (r) and integer (i) properties for the jth particle– Index i is for properties (see common block /ptpointers/ )

• ptw(i,j,k,e,l)– Feedback two-way coupling from particles to fluid– (i,j,k)th grid point of eth element for lth feedback field

• rpart(jx,i) is x location of ith particle• ipart(je0,i) is local element number

of ith particle

Page 23: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT23

Lagrangian Phase

Position:

Momentum:

Energy:

• For a single particle:V

x

y

z

X

u,Tf,p,ρf

Hydrodynamic force on particle

Collisional force

Body force

Hydrodynamic heat transfer

Tp

Mp

Page 24: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT24

Note:

Hydrodynamic Models

Quasi-steady:

Added-mass:

Undisturbed:

Quasi-steady:

Forcing

Heat Transfer

Page 25: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT25

Input file: particles.inp

• Demarcated by sections:

• Note that values given in following slides provide a good starting point to run a case (taken from example cases)

- - - - PARTICLE PHYSICAL PROPERTIES - - - - - - - - - - - - - - - -- - - - PARTICLE FORCE MODELS - - - - - - - - - - - - - - - - - - - - - -- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -- - - - BOUNDARY TREATMENT - - - - - - - - - - - - - - - - - - - - - - - -- - - - RESTARTING - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - DEM - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Page 26: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT26

- - - - PARTICLE PHYSICAL PROPERTIES - - - - - - - - - - - - - - - -

1000 ! total computational particles119E-6 119E-6 ! Dp , [m]293. ! Tp0 , [K]2500. ! rho_p, [kg/m^3]840.0 ! C_p,p, [J/kg K]

Total number of particles

Particle diameter (two numbers)• If same, monodisperse• If different, polydisperse (set either uniform/gaussian in cmtparticles.usrp file)

Initial particle temperature

Particle density

Particle specific heat at constant pressure

Page 27: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT27

- - - - PARTICLE FORCE MODELS - - - - - - - - - - - - - - - - - - - - - -

2 ! quasi-steady force0 ! undisturbed force0 ! added-mass force0 ! quasi-steady heat transfer0 ! undisturbed heat transfer

Quasi-steady force (cmtparticles.f: usr_particles_f_qs)• 0: off• 1: on with Reynolds #, Mach #, and Volume fraction corrections• 2: on with Reynolds # and Volume fraction corrections

Undistrubed force(cmtparticles.f: usr_particles_f_un)• 0: off• 1: on

Added-mass force (cmtparticles.f: usr_particles_f_iu)• 0: off• 1: on with Mach # and Volume fraction corrections

Quasi-steady heat transfer (cmtparticles.f: usr_particles_q_qs)• 0: off• 1: on with Reynolds # and

Prandlt # corrections

Undisturbed heat transfer (cmtparticles.f: usr_particles_q_uu)• Scaffolding, but no model now

Page 28: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT28

- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 ! time integration method4 ! coupling method1 ! interpolation method1 ! particle io method0 ! particle injection rate step0 ! particle delay time step1 ! random number seed

Time integration (cmtparticles.f: rk3_integrate)• 1: RK3

Coupling method (cmtparticles.f)• 1: One-way (particles move but no feedback force or collisions) • 2: Two-way (one-way with feedback force to fluid, spread_props_grid)• 4: Four-way (two-way with collisions between particles, usr_particles_f_col)

Interpolation (cmtparticles.f: interp_props_part_location)• 1: Reduced barycentric Lagrange• 2: Barycentric Lagrange• 3: Tri-linear

Page 29: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT29

- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 ! time integration method4 ! coupling method1 ! interpolation method1 ! particle io method0 ! particle injection rate step0 ! particle delay time step1 ! random number seed

Particle I/O (cmtparticles.f: usr_particles_io)• < 0 : Output timers• abs() = 1: Output Lagrangian values• abs() = 2: Output Planar averages (needs tuning when used)• abs() = 3: Output 2 and 3• Note: two-way field files are output automatically

Inject total particles every this many steps

Delay particles until after this time step

Random number seed (change if different random numbers are needed between consecutive runs)

Page 30: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT30

- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -

2 ! projection method0 ! super particle pre-iterations0.1 ! mean initial particle volume fraction0.6 ! filter width/grid spacing0.3 ! smallest element width a particle will see1E-2 ! gaussian decay percent

Projection method• 1: Top-hat by counting number of particles in element• 2: Gaussian filter

Super Particle parameters

Filter width over element spacing• Any number from 0 to 1• If < 0.5, set assume rectangular domain (next slide) to 1• If > 0.5, set assume rectangular domain (next slide) to 2Smallest element width• Used if you want a small filter width• If larger than element sizes, automatically reset to smallest element widthTo what percent is Gaussian clipped at

Page 31: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT31

- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -

2 ! assume rectangular domain-1 1 ! xL_rect, xR_rect-1 -1 ! yL_rect, yR_rect-1 -1 ! zL_rect, zR_rect

Assume rectangular domain• 1: At most 7 ghost particles created, filter width/element spacing < 0.5• 2: At most 26 ghost particles created, filter width/element spacing > 0.5

X Boundary Conditions for Particles

Y Boundary Conditions for Particles

Z Boundary Conditions for Particles

• 0 : Periodic (need both left and right)• 1 : Outflow (particles leave domain)• -1: Wall (four-way coupled particles

will collide with wall, < four-way coupled will be treated as outflow)

Page 32: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT32

- - - - RESTARTING - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

5000 ! particle restart output step0 ! particle restart read last step

Every this number of steps, restart files will be output

If different than zero, particles will read this number rpartxyz and rpartdatafile to restart

- - - - DEM - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

10000 ! particle injection rate pre-simulation DEM step1E1 ! ksp0.85 ! coeffecient of restitution

Not really used anymore

Spring coefficient for collisions

Restitution coefficient for collisions

Page 33: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT33

Adding Particles to Fluid Case

• Make sure particle git code is checked out• Edit .rea file

– Param 1: initial fluid density (interpolated after)– Param 2: fluid dynamic viscosity

• Edit SIZE file– Make sure llpart is large enough to fit at most

llpart particles per processor (stay much below this in practice)

Page 34: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT34

Adding Particles to Fluid Case

Edit .usr file• Add forcing to userf()• Add to top of file:

– include "cmtparticles.usrp"

subroutine userf (ix,iy,iz,eg)include 'SIZE'include 'TOTAL'include 'NEKUSE'include 'CMTPART'include 'CMTDATA’integer e,ege = gllel(eg)

! particle forcingif (two_way .ge.2) then

if (istep .gt. time_delay) thenffx = ptw(ix,iy,iz,e,1)ffy = ptw(ix,iy,iz,e,2)ffz = ptw(ix,iy,iz,e,3)qvol= ptw(ix,iy,iz,e,5) + rhs_fluidp(ix,iy,iz,e,4) elseffx = 0.0ffy = 0.0ffz = 0.0endif

elseffx = 0.0ffy = 0.0ffz = 0.0

endif

returnend

Page 35: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT35

Adding Particles to Fluid Case• Move and edit cmtparticles.usrp file

– Set initial placement of particles (place_particles)

• Move and edit particles.inp file into directory

• Make and run as usual

Page 36: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT36

Output Files• Given input settings, output files are created:

– If > one-way coupling, files with ptw prefix are created

• Run visnek on ptwcasename in directory to get ptwcasename.nek5000 file, which is field file that can be opend in VisIt/ ParaView (certain versions)

– If Lagrangian output is set, partdata and partxyzfiles are output

• Run ./partview in directory to convert binary files to npartxyz ASCII files to be read by VisIt/ ParaView

– If restart files output, rpartdata and rpartxyz files created

Page 37: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT37

Visualization Demonstration (VisIt)• Eulerian Data

– Pseudocolor– Volume rendering– Light Sources– Clipping (box)– Three-slice– Bounding box– Annotations

• Lagrangian Data– Points– Sphere geometry– Bounding box

Sync time datahttps://wci.llnl.gov/simulation/computer-codes/visit/manuals

Page 38: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Do you have any questions?

Page 39: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Running CMT-nek

Goran Marjanovic and Bradford Durant

11/29/2017

Page 40: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT40

BEFORE WE BEGINYOU NEED, ON YOUR UNIX-LIKE SYSTEM, git make (e.g. GNU buildutils) Fortran 77 and ANSI C compilers (e.g. GCC & gfortran, Intel Fortran & C) MPI and wrappers (e.g. mpich, openmpi)

Page 41: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Getting Started

1. cd ~2. git clone https://github.com/UFCCMT/Nek5000.git -b master3. git clone https://github.com/UFCCMT/NekExamples.git -b master4. cd Nek5000/tools5. “./maketools all” or “./maketools genbox” and “./maketools genmap”

1. Change compiler options2. export PATH=“$HOME/Nek5000/bin:$PATH”

6. cd $HOME/Nek5000/short_tests/CMT/inviscid_vortex/7. cp $HOME/Nek5000/bin/makenek .

1. Open makenek, change source_root, PPLIST=“CMTNEK”

8. ./makenek pvort

Page 42: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT42

Running

SESSION.NAMEcase/full/path/to/case/dir/with/trailing/slash/

<case>.rea

<case>.map

SESSION.NAME

nek5000 *0.f0*

> logfile

Solution files (big)

Lots of stdout(redirect it)

Invoking nek5000– “./nek5000 >logfile” for serial runs– “nek5000” is executable arg to mpirun, mpiexec

e.g. “mpirun –np 128 ./nek5000 >logfile”

Every CMT-nek case needs these things1. <case>.rea file mesh, BC, runtime settings2. <case>.map file element connectivity graph3. <case>.usr file Fortran 77 source code4. SIZE file parameters, dimensions5. makenek script build script6. SESSION.NAME makes running easier

Page 43: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT43

CMT-nek geography

An element’s worth ofStorage isnx1 x ny1 x nz1 doubles

CMT-nek computes a d-dimensional gas flow field ina domain covered by a mesh of NEL elements• Each element is hexahedral, determined by 2d vertices

and face curvature specification.• Each element has nx1d grid points inside of iton np MPI tasks

Whole domain in.rea filenek uses the term “global” to refer to to the whole domain

.rea file

CMT-nek is a SIMD Fortran 77 and C code using MPI-1• Procedures act on data

for nelt=NEL/np elements

nelt of these stores 1 MPI task’s worth of data (for a given variable, or grid points, etc.)

SIZE file

Page 44: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

SIZE fileSIZEldim = 2 or 3 lx1 – controls polynomial order of approximation (N = lx1 – 1)lxd – polynomial order of over-integration/dealiasing for convective terms. In general, lxd = 3*(lx1-1)/2lx2 = lx1 for CMT-neklelg - upper bound on elementslpmax - max number of processorslpmin - min number of processorsldimt - max number of auxillary fields (temperature, scalars, etc)

memory ~ E(N+1)^3 points x 400 (wds/pt) x 8 (bytes/wd)

Page 45: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

usr fileCASENAME.USR

Fortran subroutines that allow direct access to all runtime variables

1. uservp – variable properties (see Jason’s talk for artificial viscosity)

2. usrdat – called right after geometry is loaded and before GLL points are

distributed. Elements, BCs, can be modified here

3. usrdat2 – called after GLL points are distributed and allows for only

affine transformations of geometry

4. userf – forcing term in momentum equation

5. userq – source term in energy/passive scalar equation

6. userbc, useric – boundary and initial conditions

7. userchk – called once per processor after each time step

Page 46: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT46

Running CMT-nek

*Unless NEL> lelg, the usr file expects different boundary conditions, np>lp,or you are migrating from a 2D case to a 3D case or vice-versa

I changed… Do I need to…

quantity Which lives in

recompile? re-rungenmap?

lx1,ly1,lz1 SIZE Y N

lxd,lyd,lzd SIZE Y N

ldim SIZE, .rea Y Y

np Arg to mpirun N* N*

NSTEPS .rea file N NDT .rea file N NIOSTEP .rea file N NUser file .usr file Y NThe entire mesh

.rea file N* Y

nek5000 has some unusual constraints on the compiled executable; it may not be obvious what changes are needed for different cases

Given NEL< lelt x np, the same executable that ran on 4 tasks can run on 500,000 MPI tasks.NSTEPS=#stepsDT=physical sim time stepIOSTEP=write *0.f0* files # step

Don’t mess with the usr file yet.Oddly, within limits*, you can change the whole rea file without recompiling.

Page 47: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Data layout in Nek1. Typical loop structure

2. Nek5000 structure

3. Simple low level loop routine

Page 48: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Routines of Interestcore/math.f

Page 49: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Functions of Interest

Page 50: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Variables of Interest

Page 51: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

Other Variables of Interest

Page 52: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Do you have any questions?

Page 53: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT Lessons Learnt from CCMT Simulations

Yash Mehta, Frederick Ouellet and Rahul Koneru

CMT-Nek Workshop, Fall 2017

Page 54: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT| 54

Lesson Learnt from CCMT’s Simulations Grid Generation

Currently we use 2-3 tools to generate unstructured tetrahedral grids for fully resolved simulations

We had to write scripts to automate grid generation

Rocflu’s “original” distribution has some scripts to convert grid from one form to the other and generate some simple grids (box)

Pre-Processing

Partitioning and Initialization routines run outside the solver routine

Depending on the size of the simulations these routines can take lot of computational time

At times we had issues that these routines take longer than the walltime on super-computers at the labs

Elements per core

Rocflu can’t run if there are too few elements per core

Page 55: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT| 55

Lesson Learnt from CCMT’s Simulations Post Processing

User can select solution dump frequency in Rocflu

Flow field data is written by each core for each selected time

Post processing routine merges data from all the cores into one file for each time

Higher the number of cores, longer it takes for this routine to run

Safe Restart

Rocflu can be restarted from the time for which solution files are available

There is flag which forces solution dump “x” minutes before the wall-time so that code can be restarted from that time

This flag took 6 months to debug because of issues with LLNL super-computer

Compiling the code

We had issues with compliers on some super-computers

Page 56: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT| 56

Lesson Learnt from CCMT’s Simulations

Non-EoS Lessons:

• Due to extreme initial conditions for blast problems, we must run at least the first thousand iterations at a small, fixed time step

• For reference, this value in Rocflu is 3.0e-12• This may have to be reduced in CMT-nek

• When dealing with a sharp change in particle volume fraction (𝜙𝜙𝑝𝑝 > 25%), it is sometimes necessary to smoothen the curtain edges for stability

• Upgrading to AUSM+up may be needed, it is an all-speed formulation

Page 57: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT| 57

Lesson Learnt from CCMT’s SimulationsEoS Lessons:

• Implementing the pure JWL equations should be simple

• Handling mixtures is non-trivial:

• Another conserved variable/gov. equation must be used to track each additional species

𝐷𝐷(𝜙𝜙𝑔𝑔𝜌𝜌𝑌𝑌𝑖𝑖)𝐷𝐷𝐷𝐷

= 0 (for the Euler equations)

• Then a mixture EoS algorithm is needed

• One equation model is probably the easiest to implement• This is simply a linear fit that decays the JWL eqns. into the ideal gas eqns.

• Iterative/surrogate solvers will need a little more work• There is a good chance that the iterative solver will mess with the scaling of

the code in its current form

• Will also need to artificially clip the mass fractions if they dip below zero slightly

Page 58: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))

CCMT

CCMT

Do you have any questions?