ESSCC

Numerical Modelling and Model Coupling In Python

Lutz GrossESSCC @ The University of Queensland

l.gross@uq.edu.au

Sponsored by

ESSCC

Contributors• Ken Steube• John Ngui• Hans Muhlhaus• Matt Davies• Dion Weatherley• Alina Hale • Louise Kettle• Derick Howcraft• Peter Hornby• Laurent Bourgouin• Steven Gordon• Sebastian Langer• and others

ESSCC

Outline

Motivation: Geosciencesesys & Co.Example: Temperature DiffusionQuality ManagementModel coupling framework Summary

ESSCC

Motivation: Geosciences

ESSCC

A Simple Model of the Earth Mantel

( )( )

0)(0

,,,,

,

3,,,,

=−+

=−

−=−+⋅−

iiiit

ii

iijjiij

TTvTv

Tpvv δωη

velocity vi, pressure p, temperature T

ESSCC

An Advanced Model

jijti nvn ,,:ransitiondirector t −=

0

0))(())((

,

,,3,,,,

=−

=Λ−++−+−

kk

jlkijklSiijijji

v

vTpvv ηηδωη

)2)(21( lnknjniniklnjnkjlninilknjnljkninijkl −+++=Λ δδδδ

0)()()()( ,,2

,,,,,, =−++++− jkjiNikkijjtpjj TnnvvTvTcT καηρκ

⎟⎟⎠

⎞⎜⎜⎝

⎛−

= refTTa

Ae11

:Viscosity η

ESSCC

Convection Modelling

ESSCC

00

0

2

1

,

><

⎩⎨⎧

=

=+∂∂

φφ

ηη

η

φφ

for

vt ii

Φ>0

Level Set Method

Φ=0

Φ<0

ESSCC

ESSCC

ESSCC

Crustal Dynamics

( )ijjiijkkij

jijtti

uuu

u

,,,

,, 0)(

+⋅+⋅=

=−⋅

μδλσ

σρ

Wave propagation:

displacement ui, stress σij

ESSCC

Seismic Wave Amplification in Basins

ESSCC

Off-shore Tsunami Model

ESSCC

Numerical methods

Numerical Modeling

Partial Differential Equation (PDE)– coupled– time-depended– non-linear

finite elements (FEM)

abstraction

implem

entation

Mantle Convection

Volcanoes

Seismic Waves

Earthquakes Reactive transport

Plate Subduction

Tsunami

spectral methodsfinite volume

ESSCC

Physical Data PDE coefficientsproperties: ρ, λ, μ Earthquake catalog

ESSCC

Geometrical Data PDE domain

fault data base

Topography &bathymetry

geological map

ESSCC

esys & Co.

ESSCC

Concepts

triangulation

domain

geological map

Geometry

C/C++, finley

compilerarrayFEMNumerics

python, escript

scriptspatial/ temporal function

PDEMathematics

Gridsphere, metaQuokker

problem solving environment

observable,catalog

ModelApplication

ToolsUser Environment

VariablesEnginesLevel

ESSCC

Python? What is this?

• Programming language – Interactive like MATLAB– Object oriented like C++– Open source

• Tons of useful modules– numarray -> matrix algebra (discontinued) – SciPy -> MATLAB functionality

ESSCC

Interfaces

Scope: GridSphere: metaQuokker

Mathematics: Python: escript

Numerics: C/C++: finley

XML

lib.so

ESSCC

Tasks & Implementation

define geometry

set PDE coefficients• external data• non-linearity• other PDEs

solve PDE

present result

FEM mesh

Data on • nodes• elements

system of linearequations

visualization

gmsh

escript

finley

vtk

Modeller

ESSCCC++/C/Fortran/MPI/OpenMP

esys Architecture

esys

libfinley.so

escript linearPDEs

vtk libescript.so

pyvisi

interactive & script

Model

EsysXML

pycad

gmsh

GIS/GML

OpenGL netcdf lapack MKL Trilinos tetgen

ESSCC

Example: Temperature Diffusion

ESSCC

Equations

QTTc jjtp =− ,,, )(κρ

Ω∂= on 0, jjnTκΩ

(5,5)

(0,0)

x0

x1

nj

ESSCC

Use Backward Euler Scheme

0

)(

and )(

,)(

)1()()(,,

)(

)1()(

,)()(

=

+=+−

−→=

−

−

jin

ij

npnnpji

nij

nn

tnn

nT

Tdtc

QTdtc

T

dtTTTtTT

κδ

ρρκδ

Linear PDE to be solved in each time step.

ESSCC

Ingredients

• Define domain: esys.pycad• Define PDE coefficients & initial value =

spatial functions: esys.escript• Solve linear PDE in each time step: esys.escript.LinearPDE

• Show the results: esys.pyvisi

ESSCC

How to define the geometry

p2

p0 p1

p3

l01

l13

l23

l02 cl0

s0from esys.pycad import *p0=Point(0,0)p1=Point(1,0)p2=Point(0,1)p3=Point(1,1)l01=Line(p0,p1)l23=Line(p2,p3)l02=Line(p0,p2)l13=Line(p1,p3)cl0=CurveLoop(l01,l13, ¥

-l23,-l02)s0=PlaneSurface(cl0)

ESSCC

How to create a Domain ?

from esys.pycad.gmsh import Designfrom esys.finley import MakeDomainfrom esys.escript import Domain

d=Design(dim=2,element_size=0.1,order=1)

s0=PlaneSurface()d.addItem(s0)dom=MakeDomain(d)assert isinstance(dom, Domain)

ESSCC

What is a Domain object ?

A Domain class objects specifies– a geometry – a PDE solver package (here: finley)– a spatial discretization method (here:

finite elements (FEM) )

from esys.finley import ReadMeshfrom esys.escript import Domaindom=ReadMesh(“mymsh.fly”)assert isinstance(dom, Domain)

ESSCC

Spatial FunctionsFunctions of spatial coordinates in represented by esys.escript.Data objects with– a Domain attached to it– a rank and shape

• Local values are numarry compatible

1

2

1

0

1

f.getRank()

(3,3)f3=grad(dom.getX()**2)f3(x)=grad(x2)

f4=dom.getNormal()

f2=grad(length(dom.getX())

f1=sin(trace(dom.getX()))

f0=10*dom.getX()

python

(3,)

(3,)

()

(3,)

f.getShape()1

f4(x)=n

f2(x)=grad(||x||):

f1(x)=sin(trace(x))

f0(x)=10x:

expression

)1 assumes 3-dimensional domain

ESSCC

SmoothnessSmoothness defined by FunctionSpace attribute

FunctionOnBoundary(dom)

Function(dom)

Function(dom)

ContinuousFunction(dom)

ContinuousFunction(dom)

f.getFunctionSpace()==

On boundary elements

On elements

On elements

On nodes

On nodes

in FEM

f3=grad(x**2)

f4=dom.getNormal()

f2=grad(length(dom.getX())

f1=sin(trace(dom.getX())

f0=10*dom.getX()

python

ESSCC

FunctionSpace Interpolationf0=Scalar(2.,ContinuousFunction(dom))f1=Tensor(4.,FunctionOnBoundary(dom))# interpolation by userg0=f0.interpolate(Function(dom))# interpolation by contextg1=f0*f1assert g1.getFunctionSpace(dom) == ¥

FunctionOnBoundary(dom)# not everything works:try:f1.interpolate(ContinuousFunction(dom))

except e:print “cannot interpolate.”

ESSCC

Linear PDE for each time step

0

)(

,)(

)1()()(,,

)(

=

+=+− −

jin

ij

npnnpji

nij

nT

Tdtc

QTdtc

T

κδ

ρρκδ

ESSCC

LinearPDE class

0 where

on ,,

>=

Ω∂+=++

+−=+++−

iii

kllkikilikliljiklij

k,lk,likii,jkij,liklii,jklij

qru

ynXudnuBnuA

Y)(XuDuC)uBu(A

system of 2nd order, linear PDE on Domain Ω :

ESSCC

For the Diffusion Problem

44 344 21Y

npnn

D

pji

n

A

ij Tdtc

QTdtc

Tij

)1()()(,,

)( )( −+=+−ρρ

κδ

YXDuuCuBuA iiiiiijij +−=+++− ,,,,, )()(PDE singlefor class LinearPDE

ESSCC

Time Integration

from esys.escript.linearPDEs import LinearPDEassert isinstance(dom,Domain)rhocp=Scalar(9.23,Function(dom))k=1.mypde=LinearPDE(dom)mypde.setValue(A=k*kronecker(dom),¥

D=rhocp/dt)T=Scalar(0.,Solution(dom))t=0.while t<t_end :

mypde.setValue(Y=Q+rhocp/dt*T)T=mypde.getSolution()t+=dt

interpolation!

ESSCC

Piecewise Constant ?

top: rhocp=9.23

buttom: rhocp=25.6

p0 p1

p2 p3

p4 p5

from esys.pycad import *…cl0=CurveLoop(l01,l15,-l45,-l04)cl1=CurveLoop(l45,l53,-l23,-l24)s0=PlaneSurface(cl0)s1=PlaneSurface(cl1)d=Design(dim=2, order=1,

element_size=0.1)d.addItem(PropertySet(‘buttom’,s0))d.addItem(PropertySet(‘top’,s1))dom=MakeDomain(d)rhocp=Scalar(0.,Function(dom))insertTaggedValues(rhocp, top=9.23,

buttom=25.6)

ESSCC

Visualization: pyvisi

Scene()

Contours() Text()

DataCollector()

Camera()

ESSCC

On-line Visualization

from esys.pycad import *mypde=LinearPDE(dom)s=Scene(renderer=“vtk_online”) # or =“vtk_offline”d=DataCollector()T=Scalar(0.,Solution(dom))d.setData(T)Contour(scene=s, data_collector=d)T,n=0., 0while t<t_end :

T=mypde.getSolution()d.setData(T)s.render(“frame.%s.jpg”%n)t+=dt; n+=1

ESSCC

Software Quality Management

ESSCC

Challenges• Support for various platforms/compilers

– PC Linux, SGI Altix, MS Window• Various Parallelization Paradigms

– Sequential, OpenMP, MPI• Code base is still under development.• Integration of various components

– boost python, netCDF, intel’s MKL, Trilinos, …• Has to serve various production codes

– Convection, plate subduction, volcanoes, seismic wave propagation, reactive transport, …

ESSCC

Regression Testing

• Use the mathematical layer in python• Within python unittest framework:

– tests for features and functionality – tests for validity of result against reference:

Lsup(result-reference)<TOL*Lsup(reference)

– tests for PDE assemblage by inserting analytic solution

• >30000 tests run once a week

ESSCC

Testing PDE interface

• choose value for A[i,j]: constant or x0

• choose solution with approximation order:u=1.23+0.94*x0+12.34*x1

• calculate Y and y symbolically.• Test:

– Lsup(pde.getRightHandSide()) > 0

– Lsup(pde.getOperator()*u-pde.getRightHandSide())< TOL * Lsup(pde.getRightHandSide())

• In 3D: 18 tests!

ESSCC

Model Coupling Framework

ESSCC

Coupling Models

• Issues:– keep codes independent when coupling models– reuse of models

• Approach:– Implement models as subclasses of python class esys.escript.modelframe.Model

– Material parameter are object attributes– Make use of context sensitivity of escript.Data

objects (-> interpolation across domains)

ESSCC

Example: Rayleigh-Taylor Instability

( )( )0,

3,,,,

=−

−=−+⋅−

ii

iijjiij

v

gpvv ρδη0, =+

∂∂

iivt

φφ

0201

0201

><

><

⋅+⋅=⋅+⋅=φρφρρφηφηη

η, ρ Φ

vclass IncompressibleFlow

class MaterialTable

class LevelSet

ESSCC

Linking Model Parameters

from esys.escript.modelframe import Linkflow=IncompressibleFlow()mtab=MaterialTable()ls=LevelSet()flow.eta=Link(mtab,”eta”)flow.rho=Link(mtab,”rho”)mtab.characteristic=Link(ls,”phi”)ls.v=Link(flow,”v”)

Attribute v of ls uses current value of attribute v of flow.

ESSCC

Execution Schedulei. Initialization for all modelsii. While no end reached:

i. Get time step size which safe for all models= min(safe time step size of individual model)

ii. Pre-processing time step for all modelsiii. While changes too large:

i. Update material coefficients η, ρ (MaterialTable)ii. Update velocity and pressure (IncompressibleFlow)iii. Update level set function (LevelSet)

iv. Post-processing of time step for all modelsiii. Finalize all models

ESSCC

Run a Simulation

from esys.escript.modelframe import Simulationflow=IncompressibleFlow()mtab=MaterialTable()ls=LevelSet()… set links …s=Simulation([mtab, flow, ls])s.run()

Order is critical !

ESSCC

The LevelSet Model

• Use Taylor-GalerkinScheme

• Watch Courant Condition:

( )44 344 2144 344 21

iiX

in

jji

Y

nii

nn

Dvvdtvdt

,

,)(

,

2)(

,)()1(

21

−

+ +⋅+=⋅ φφφφ

domain in the everywhere iivv

dxdt θ≤

ESSCC

LevelSet Classclass LevelSet(Model):

def doInitialization(self): # overwrite Model methodself.__pde=LinearPDE(self.domain)self.__pde.setValue(D=1.)self.__phiold=self.phi

def doStepPreprocessing(self): # overwrite Model methodself.phi=self.__phiold

def doStep(self,dt): # overwrites Model methodg=dot(self.v,grad(self.phi))self.__pde.setValue(Y=self._phiold+dt*g,

X=-dt**2/2*self.v*g)self.phi=mypde.getSolution()

def doStepPostprocessing(self): # overwrite Model methodself.__phiold=self.phi

def getSaveTimeSafeSize(self,dt): # overwrite Model methodreturn inf(self.domain.getSize()/length(self.v))

ESSCC

EsysXML Files

• save simulation to XML s=Simulation([…]) s.writeXML(“sim.xml”)

• rebuild simulations from XML– modify parameters outside python

• automated building of GUIs for simulations – See metaQuokker in GridSphere

• automated checkpointing and restart – under construction

ESSCC

Remark• Other examples for Model class objects

– Visualization components– Instruments (e.g. seismograph)– Boundary Conditions– Domain definition

• Does not provide distributed workflow control• Framework under construction

– Check pointing/restart– testing framework for models– stronger parameter typing

ESSCC

Summary• esys & Co. provides an easy-to-use

environment for scientist to implement mathematical models in python.

• abstraction from numerics & implementation.• Open design • provides a mechanism to deliver and couple

models.• maintaining an high level of software quality