effective discrete adjoint model for openfoam -

STCE

An (more) effective Discrete Adjoint Model for OpenFOAM

Markus Towara, Uwe NaumannSoftware and Tools for Computational Engineering ScienceRWTH Aachen University

EuroAD 2013, Oxford, 10. December 2013

STCECFD @ STCE

I A. Sen (aboutFLOW Project, discrete OpenFOAM)

I A. Daztouri (aboutFLOW Project, discrete ACE+)

I M. Foerster (fortissiFOAM Project, OpenFOAM in the cloud)

I me (discrete OpenFOAM)

,

STCE RWTH-Aachen, Discrete OpenFOAM, EuroAD 2013 2/ 35

STCEOutline

OpenFOAM Introduction

Discrete OpenFOAM

Improvements

Verification and Results

Work in Progress - PISO Foam

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEOpenFOAM

I Open Field Operation and Manipulation

I Open-Source (GPLv3) CFD solver

I developed by OpenCFD Ltd., currently at version 2.2.x

I > 1 Mio. Lines of Code

I includes tools for meshing, pre-, post-processing

I rising adoption in industry and academia due to lack of licence costs → well suited for parallelarchitectures

,


STCECFD Basics

I Navier Stokes equations for incompressible steady flow:

v · ∇v = ν∇2v− 1

ρ∇p momentum conservation

∇ · v = 0 mass conservation

I Or in three Dimensions:

vx∂vx

∂x+ vy

∂vx

∂y+ vz

∂vx

∂z= −

∂p′

∂x+ ν

∂2vx∂x2

+∂2vx

∂y2+∂2vx

∂z2

vx∂vy

∂x+ vy

∂vy

∂y+ vz

∂vy

∂z= −

∂p′

∂y+ ν

∂2vy∂x2

+∂2vy

∂y2+∂2vy

∂z2

vx∂vz

∂x+ vy

∂vz

∂y+ vz

∂vz

∂z= −

∂p′

∂z+ ν

∂2vz∂x2

+∂2vz

∂y2+∂2vz

∂z2

∂u

∂x+∂v

∂y+∂w

∂z= 0

,


STCECFD Basics

(Decoupled) Solution of the partial differential equations (SIMPLE-Algorithm):

I discretize / linearize momentum conservation equations

I solve momentum equations for velocity v, assume pressure p as known

I obtained velocity field fulfills momentum equation but not mass conservation equation as thepressure field was guessed and not correct

I discretize mass conservation equation

I use mass conservation equation to correct pressure field

I use new pressure field to correct velocity field

I Loop...

,


STCETopology Optimization

no optimization added ”material” reconstruction

Add penalty term α to Navier-Stokes equation1 :

(v · ∇) v = ν∇2v−∇p− αv1C. Othmer: A continuous adjoint formulation for the computation of topological and surface sensitivities of ducted

flows. Intern. J. f. Num. Meth. in Fluids. p. 861–877, 2008.,


STCEHow to find appropriate α

I Define Cost Function J, e.g. total pressure loss between inlet and outlet:

J =

∫Γ

p+1

2v2n dΓ

I Calculate sensitivity of the Cost function w.r.t. parameters αi∂J

∂αi=???

I Calculate updated porosity field αn+1, e.g.:

αn+1i = αn

i − λ ·∂Jn

∂αni

, while insuring 0 < αi < αmax

I Loop until α converged...

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEMotivation for ADin Context of Shape Optimization

I AD promises:I greater flexibility w.r.t new objectivesI easy adaption to new solver types / generationsI calculated derivatives are exact w.r.t the used discretizationI higher order derivatives available

I Problems:I memory requirementsI how to retain parallelism?

,


STCEHow to implement

n � m, so Adjoint Mode is method of choice

I source code transformation

I operator overloadingI due to complexity of OpenFOAM code (templates, massively object oriented) the only suitable

optionI operator overloading tool dco/c++I every floating point value in OpenFOAM is replaced with custom adjoint datatype dco::a1s::typeI needed intermediate values are stored in a structure called tape

,


STCEHow to implement

I put dco into src/OpenFOAM

I include dco.hpp

I replace doubles with active datatype from dco

I OpenFOAM has own typedef for scalar floating point values→ just one substitution

I in theory we now just need to recompile OpenFOAM and are ready to go

,


STCEOpenFOAM a1s mode

in src/OpenFOAM/primitives/Scalar/doubleScalar/doubleScalar.h: replace:

namespace Foam{

typedef double doubleScalar;...

}

with:

#include ”dco.hpp”namespace Foam{

typedef dco::a1s::type doubleScalar;...

}

,


STCEOpenFOAM t1s mode

in src/OpenFOAM/primitives/Scalar/doubleScalar/doubleScalar.h: replace:

namespace Foam{

typedef double doubleScalar;...

}

with:

#include ”dco.hpp”namespace Foam{

typedef dco::t1s::type doubleScalar;...

}

,


STCEin Reality

I some minor changes have to be made in the OpenFOAM code:I unions don’t support active datatypesI some casts are missing and have to be done by handI some functions (pow,max,min) have to be adapted

,


STCEusage of discrete OpenFOAM - t1s

Black-Box tangent-linear Version of simpleFoam, calculates ∂J∂αi

:

double sens = 0;dco::t1s::set(alpha[i],1,1);

for (runTime++; !runTime.end(); runTime++){... // solve for U,p}// Sum pressure over inlet faces scaled with face areadoubleScalar J = gSum( p.boundaryField()∗patch.magSf() );

dco::t1s::get(J,sens,1);

Need to do this N-times to get full sensitivity field!

,


STCEusage of discrete OpenFOAM - a1s

Black-Box adjoint Version of simpleFoam, calculates gradient of J :

dco::a1s::static tape tape(tapeSize);double∗ sens = new double[alpha.size()];

for(int i=0; i<alpha.size(); i++)tape.register variable(alpha[i]);

for (runTime++; !runTime.end(); runTime++){... // solve for U,p}// Sum pressure over inlet faces scaled with face areadoubleScalar J = gSum( p.boundaryField()∗patch.magSf() );dco::a1s::set(J,1,−1);tape.interpret adjoint();

for(int i = 0; i<alpha.size(); i++)dco::a1s::get(alpha[i],sens[i],−1);

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEFirst Results

I Black-Box approach severely bound by memory bandwith! (Need to allocate around 20 GB)I By doing Checkpointing we can actually get faster...

,


STCEHow to improve?

I Knowledge and Profiling reveals that most of calculation time and tape memory is spent in(iterative) linear solvers

I Analytical insight allows us to treat the adjoints of linear solvers analytically

Lemma

For a Linear Equation Sytem Ax = b we can calculate the adjoints for A and b by: 2

b = A−T x→ AT b = x

A = −b · xT

I This gives us an additional Linear Equation System which we have to solve during the gatheringof the adjoints

2M. Giles, Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation, Advancesin Automatic Differentiation 2008

,


STCETreatment of Linear Solvers

Black-Box Continuous Linear Solver

I when we encounter an linear solver during the augmented forward run we can stop taping

I when we encounter the gap in the tape during the interpretation we have to fill in the gap byhand

I have to remember A and x!

,


STCEApplication to OpenFOAM

I Abstract Base Class for all Linear Solvers exists → can tackle all implemented solvers at once

I Discretization matrices are stored in a coordinate sparse matrix format → transposing matrixstraightforward

I take care not to mess up symmetry of matrices

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEResults

I we are seeing a nice improvement in both runtime and memory usage

,


STCEVerification

I We calculate a reference solution with solvers GAMG for U and smoothSolver for p, both withtolerance 1e− 10

I After that we calculate solutions with different tolerances for the forward and adjoint linearsolvers and compare to the reference solution

I 11k Cells in mesh

Velocity Sensitivity

,


STCEVerification

Runtime Tape-Memory

,


STCEError

,


STCERuntime

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEWork in Progress - Unsteady Adjoints

Unsteady PISO solver with unsteady cost function (potentially depends on every timestep)

,



,


STCESummary and Outlook

I Fully discrete mode uses loads of memory

I Treatment of linear solvers greatly reduces this

I Its also faster

I Application to unsteady flow, heat transfer, turbulence in progress

,


STCE

Thank you!Questions?

,


effective discrete adjoint model for openfoam -

Documents