experiences with tapenade and cfd - euroad workshop... · 2017-02-17 · ornl is managed by...

ORNL is managed by UT-Battelle

for the US Department of Energy

Experiences with Applying Tapenade to a Modern 3D CFD Code

Charlotte Kotas†

Jacob Barhen†

Bob Wilson†

Mike Traweek‡

EuroADFebruary 2017

† ORNL

‡ Office of Naval Research

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

2 Experiences with Tapenade and CFD

Outline

• Motivation

– Model validation using uncertainty and sensitivity analysis

• Application

– Computational fluid dynamics code, REX

• Applying Tapenade to REX

– Work in progress


Motivation: Model Validation

• Experimental data contain uncertainties due to measurement bias, precision, and random factors

• Computational model predictions contain uncertainties due to parameters, simplifications in physical equations, and numerical techniques

Given these uncertainties, we want to combine experimental data with computational model responses to determine a best estimate for the independent variables, model parameters, and experimental responses

And use these best estimates to reduce the uncertainties

For details see:

D. G. Cacuci, Sensitivity and Uncertainty Analysis, Chapman & Hall (2003)

J. Barhen et al, “Consistent uncertainty reduction in modeling nonlinear systems”, SIAM Journal on Scientific Computing, 26 (2), 653-665 (2004).

and references therein.


Notation• Let 𝒙 be input variables, 𝜽 be

model parameters, 𝒎(𝒙, 𝜽) be the computational model response,𝒆the be experimentally measured response, 𝒓 be the true system response, 𝜺 be experimental errors, and 𝜹 𝒙, 𝜽 be model errors

– 𝒓 = 𝒆 − 𝜺 = 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽

• Let 𝒂 be a vector collecting 𝒙 and 𝜽 such that 𝒎 𝒙, 𝜽 = 𝒎(𝒂)

• Let 𝑪𝒛 be a covariance matrix computed from standard deviations in the elements of vector 𝒛

• Let ^ denote the “best estimate” of a quantity

Model

𝒙

𝜽

𝒎(𝒙, 𝜽)


Sensitivity Analysis and Uncertainty Quantification (1)

• The sensitivities of the system describe the relative importance of individual input values in overall system responses.

• Mathematically, the sensitivity matrix (𝑺) is the derivative of the desired responses 𝑹 = 𝑹(𝒎, 𝒂). We assume that the computation model computes 𝑹 directly, thus,

– 𝑺𝒏𝒊 =𝒅𝑹𝒏

𝒅𝒂𝒊=

𝝏𝑹𝒏

𝝏𝒂𝒊+

𝝏𝑹𝒏

𝝏𝒎𝝂

𝝏𝒎𝝂

𝝏𝒂𝒊=

𝝏𝒎𝒏

𝝏𝒂𝒊

– To first order, 𝒎 = 𝒎0 + 𝑺 𝒂−𝒂𝟎

• The sensitivity matrix is used to propagate uncertainty from the inputs to the model responses (to the first order):

– 𝑪𝒎 = 𝑺𝑪𝒂𝑺𝑻, where 𝑺 is evaluated at 𝒂

– If independent variables, 𝑪𝒂 is diagonal and this becomes the more

familiar 𝜎𝑚2 =

𝜕𝑚

𝜕𝑥

2𝜎𝑥

2 +𝜕𝑚

𝜕𝑦

2

𝜎𝑦2 + ⋯

Tapenade computes



• We can combine the model and experimental responses to estimate the true response and reduce uncertainty.

𝜺 = 𝒆 − 𝒓 = 𝒆 − 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽

≅ 𝒆 − 𝒎 𝒂 − 𝑺 ෝ𝒂−𝒂

• We create a generalized loss function 𝑄 based on the experimental error and the model constraints:

𝑸 =ො𝒓 − 𝒆ෝ𝒂−𝒂

⋮

𝑇 𝑪𝒆 𝑪𝒑 …

𝑪𝒑𝑇 𝑪𝒂 …

… … …

−1ො𝒓 − 𝒆ෝ𝒂−𝒂

⋮

• ℒ can be minimized for the best estimate of ො𝒓 and ෝ𝒂, and used to compute new covariance (uncertainty) estimates

ℒ = 𝑸 + 𝜆 𝒎 𝒂 − 𝒆 + 𝑺 ෝ𝒂−𝒂



• Reduction of uncertainties:

𝑺 = ቚ𝝏𝒎

𝝏𝒂 𝒂

𝑺ෝ𝒂 = ቤ𝝏𝒎

𝝏𝒂ෝ𝒂

𝑹 = 𝑪𝒆 − 𝑺𝑪𝒑𝑻 −

𝑪𝒑𝑺𝑻 + 𝑺𝑪𝒂𝑺𝑻

𝑪ෝ𝒂 = 𝑪𝒂 − 𝑪𝒑𝑻 − 𝑪𝒂𝑺𝑻 𝑹 −𝟏 𝑪𝒑 − 𝑺𝑪𝒂

𝑪ො𝒓 = 𝑪𝒆 − 𝑪𝒆𝑻 − 𝑪𝒑𝑺𝑻 𝑹 −𝟏 𝑪𝒆 − 𝑺𝑪𝒑

𝑪 ෝ𝒎 = 𝑺ෝ𝒂𝑪ෝ𝒂𝑺ෝ𝒂𝑻

NOGA=Nonlinear Optimization Generalized Adjustments

𝑪𝒂

Ce

Model

+

NOGA

Experiment ො𝒓 ෝ𝒎

𝑪 ෝ𝒎 𝑪ො𝒓

𝑪ෝ𝒂

ෝ𝒂𝒂

Ce

𝑪𝒂

Model

Cm

Experiment

e

m𝒂


Tapenade and Uncertainty

• Let Ԧ𝑎 denote the parameters and 𝜎𝑎,𝑖2

denote the uncertainty of the 𝑖 th

parameter. (Assume that the parameters are independent.)

• Let 𝑚 denote the unknowns computed by Rex (e.g. U, force coefficients)

• The uncertainty of a given element of 𝑚 is computed from the derivatives provided by Tapenade through:

𝜎𝑚,𝑖2

=𝜕𝑚𝑖

𝜕 Ԧ𝑎

𝜎𝑎,12

⋯ 0

⋮ ⋱ ⋮

0 ⋯ 𝜎𝑎,𝑛2

𝜕𝑚𝑖

𝜕 Ԧ𝑎

𝑇

Note that 𝜕𝑚𝑖

𝜕𝑎is a row vector; Tapenade is used to compute

𝜕𝑚𝑖

𝜕𝑎𝑗.

• This definition can be expanded to the full covariance matrix of uncertainties as:

𝑪𝒎 = 𝑆𝑪𝒂𝑆𝑇

Where the elements of the Jacobian matrix, 𝑆, are given by:

𝑆𝑖𝑗 =𝜕𝑚𝑖

𝜕𝑎𝑗

• Tapenade computes one row/column of the Jacobian for each pass in

forward/reverse mode


Application: REX

• Unsteady, RANS/DES Solver (IIHR: University of Iowa)

– 25-year history of on-going ONR support

– Verified & validated for naval hydrodynamics: resistance/propulsion, seakeeping, maneuvering, design optimization

– Modeling

• Two-equation turbulence and subgrid-scale models

• Free-surface capturing for air-water interface

• Density stratification and multiphase model for bubbly

– Numerical

• Pressure-based, PISO approach for incompressibility constraint

• Upwind convection schemes from second- to fifth-order

• Second-order implicit time differencing

– Capabilities

• Dynamic overset: grid generation and bodies w/relative motion

• 6DOF multi-body calculations with fully-appended UUV/subs/ships with rotating props, rudders etc.

• HPC parallel simulations using Open-MPI


CFD: Methodology (1)

• Governing Equations: Reynolds Averaged Navier-Stokes (RANS)

• Mass conservation

• Momentum conservation

¶r

¶t+Ñ× ru( ) = 0

¶

¶truéë ùû +Ñ× ruÄuéë ùû = -Ñp̂ +Ñ×

2

Reeff

Ñ2u

æ

èç

ö

ø÷

1

Reeff

=1

Re+ n

t

Re =r

0U

0L

0

m

nt=

k

w

p̂ = p +2

3rk +

2

3

1

Reeff

Ñ ×u

u, p̂( ); velocity, pressure

k,w( ); turbulent kinetic energy, dissipation

r0,m; density,viscosity

U0, L

0; characteristic velocity, length


CFD: Methodology (2)

• Fully turbulent, k-ω, two-equation turbulence model

– Turbulent kinetic energy, k

– Specific turbulent dissipation rate, ω


Applying Tapenade to REX

• TAPENADE is an Automatic Differentiation program that operates by source transformation on C and FORTRAN 77/90/95 codes. It was developed at INRIA Sophia-Antipolis, and is currently maintained by Laurent Hascoët and Valerie Pascual.

• As REX is written in (mostly) FORTRAN 95, Tapenade can be applied with minor changes to the source code to bring the code to F95 standards or simplify things for Tapenade.

– “flush(11)” replaced with “call flush(11)”

– Tapenade option –linelength to allow lines longer than F95 standard

– Adding some intermediate variables manually where Tapenade needed help seeing array size. For example, MAXVAL(ABS(A(:))) becomes result1=ABS(A(:)); MAXVAL(result1)

– Replacing DOT_PRODUCT(a(:),b(:)) with SUM(a(:)*b(:))


Tapenade “features”

• REX is complex. There are a few places where Tapenade gets confused.

– Wants to initialize at the top level a couple of local array variables

– Leaves in “#” in a couple of places, as in EXTERNAL #FUNCTION_NAME#

– Calls for nonexistent module PACKINGD. (Module PACKING exists, but not PACKINGD.)

– Puts in module variable that is not within local scope in initializing array.

• Original: A(:,:)=0.0, where A is brought into scope through “USE MYMODULE, ONLY: A”

• Adds: Ad(NX,NY)=0.0, but NX and NY are not in scope. “USE MYMODULE, ONLY: NX,NY” added by hand.


External library: PETSc

• The PETSc library is designed for parallel, scalable solutions of PDEs (https://www.mcs.anl.gov/petsc/index.html)

– Distributed matrix and vector data structures

– Linear and non-linear solvers

– Written in C with interfaces for C, C++, Fortran, and Python

– Built on the MPI library

• REX uses PETSc to store distributed matrices for use in PETSc’s linear solver routines.


Using PETSc (1)

• Tapenade is designed to generate calls to derivative versions of unknown functions if it thinks they are necessary

• For PETSc, setting the value of a vector looks like:

– call VecSetValue(rhsp,nijk,pval,INSERT_VALUES,ierr)

– Which roughly translates to rhsp(nijk)=pval

• But to Tapenade, the arguments look like (integer, integer, real, integer, integer)

– Tapenade generates

call VecSetValue_D(rhsp,nijk,pval,pvald,INSERT_VALUES,ierr)

– Tapenade doesn’t see the rhsp vector data structure as differentiable

• These library calls need to be managed by hand.

– A new vector for rhspd was created to handle the differentials of rhsp

– Replace the Tapenade generated call with calls to set rhsp and rhspd

call VecSetValue(rhsp,nijk,pval,INSERT_VALUES,ierr)

call VecSetValue(rhspd,nijk,pvald,INSERT_VALUES,ierr)


Using PETSc (2)

• PETSc’s linear solvers can be handled similarly to the example on Tapenade’s FAQ (https://www-sop.inria.fr/tropics/tapenade/faq.html#linearsolvers) for hand coding the SOLVE_D and SOLVE_B routines

– For linear solvers, hand coding is typically better than letting Tapenade compute the derivatives even when the solver source code is available.

• For forward mode, SOLVE becomes SOLVE_D which solves for 𝑦 and ሶ𝑦– 𝑦 = 𝐴−1𝑏 (as before)

– ሶ𝑦 = 𝐴−1 ሶ𝑏 − ሶ𝐴𝑦 , where ሶ𝑏 and ሶ𝐴 are inputs

https://www-sop.inria.fr/tropics/tapenade/faq.html#linearsolvers


External library: MPI

• The message passing interface (MPI) library allows different compute nodes to communicate, sharing partial results and status messages.

• Automatic Differentiation adds data dependencies to the original program that must be accounted for in the communication patterns. Plain MPI libraries are not sufficient to handle these dependencies without additional hand coding.

• The Adjoinable MPI library (http://www.mcs.anl.gov/~utke/AdjoinableMPI/AdjoinableMPIDox/index.html) was written to alleviate pressure on code developers by providing an interface between AD tools and a modified MPI library suitable for computing adjoints.

– Used with AD tools like Tapenade, Adol-C, etc

http://www.mcs.anl.gov/~utke/AdjoinableMPI/AdjoinableMPIDox/index.html


Using AMPI (1)

• From a user’s perspective, switching from MPI to AMPI involves code changes

– Changing MPI headers to AMPI headers (#include "mpif.h"⇒#include "ampi/ampif.h“)

– Changing function calls to “MPI_” to “AMPI_” version

• Use “NT” versions outside of AD regions

– Add arguments to AMPI_Recv and AMPI_Send type function to help map communications

• AMPI_TO_RECV, AMPI_TO_IRECV_WAIT, AMPI_FROM_SEND, AMPI_FROM_ISEND_WAIT

• The AMPI website suggests modifying MPI data types (e.g. MPI_REAL ⇒MPI_AREAL), but this does not appear to be necessary for Tapenade.

• Add AMPI include directory to compile step and link with the AMPI libraries


Using AMPI (2)

• Initially, several issues between Tapenade and AMPI were apparent.

• Missing bits in the AMPI library

– AMPI library ampiPlainC missing functions AMPI_Bcast and AMPI_Reduce

– AMPI library ampiCommon missing functions for TLS_AMPI_SCATTER, TLS_AMPI_BCAST, TLS_AMPI_REDUCE, TLS_AMPI_ALLREDUCE

• Tapenade not generating correct code

– Tapenade attempts to add mpif.h and ampi/ampif.h, resulting in duplicate definitions

– AMPI_BARRIER calls become calls to AMPI_BARRIER_D rather than TLS_AMPI_BARRIER

– Tapenade generated calls to TLS_AMPI_ALLGATHER, TLS_AMPI_REDUCE, TLS_AMPI_ALLREDUCE, TLS_AMPI_BCAST, TLS_AMPI_SCATTER have wrong format

– Tapenade wants to create derivatives for MPI_INTEGER data types


Using AMPI (3)

• Laurent Hascoet has been working with us to be sure that the AMPI library implements the missing functions

• Laurent Hascoet has been working to improve the Tapenade code as well

– We greatly appreciate his help!

• In addition, AMPI_Waitall is not implemented in the FORTRAN version of the AMPI library, as is noted on the AMPI website.

– As a workaround, we can loop over AMPI_WAIT calls

• Currently, the webpage user manual does not provide an example where both the “PlainC” and “Common” libraries are used. It may be necessary to link both libraries for code that contains undifferentiated routines.

– Both libraries have some common functions like AMPI_COMM_SIZE, so I’m not sure this was the original design


Validating code

• Tapenade has some features to help validate its output against finite difference estimates.

– Command line options “debugTGT” and “debugADJ”

– Requires support code from Tapenade’s “ADFirstAidKit”

• ADFirstAidKit has both a C and Fortran version of “debugAD”, which are used differently but both require simultaneous two simultaneous runs of the code

– Fortran requires two executables with small changes between them

> ./test1 | ./test2

– C requires environment variables to be set so that the code can determine which phase is currently executing

> export DBAD_PHASE=1 && ./test > out.1.txt &

export DBAD_PHASE=2 && ./test

• Output from C and Fortran version not the same


Questions?

Dr. Jacob BarhenCESAR Director and ORNL Fellow

Phone: 865-574-7131

E-mail: [email protected]

1 Bethel Valley Road

Bldg 5600, MS 6015

Oak Ridge, TN 37831-6015

USA

Dr. Charlotte KotasORNL/CESAR member of technical staff

Phone: 865-241-6597

E-mail: [email protected]

Center for Engineering Science Advanced Research (CESAR)Computer Science and Mathematics Division

Oak Ridge National Laboratory

experiences with tapenade and cfd - euroad workshop... · 2017-02-17 · ornl is managed by...

Documents