strongly scalable parallel simulations of high-resolution ... · modeling and simulations in...

Strongly Scalable Parallel Simulations of High-resolution Modelsin Computational Cardiology

Christoph Augustin, Gernot Plank

in coop. G. Haase, M. Liebmann, O. Steinbach, G. Holzapfel, A. Neic, A. Prassl, T. Fastl, T. Eriksson, A. Crozier

Medical University of Graz

SFB Mathematical Optimization andApplications in Biomedical Science

Modeling and Simulations in Biomechanics, September 15th, 2014

C. Augustin Modeling and Simulations in Biomechanics

1 / 23

Outline

Electromechanical Modeling

Parallel Strategies

Configuration

Numerical Examples

Open Tasks and Perspectives


1 / 23

Outline


Parallel Strategies

Configuration

Numerical Examples



2 / 23

Mechanics of Cardiovascular Tissues

matrixmuscle fiber

collagen fiber

n0sheet-normal

axis

s0

f0

sheet axis

fiber axis

Find the displacement u such that

− Div FS(u, x) = 0 for x ∈ Ω ,

u(x) = uD(x) for x ∈ ΓD ,

FS(u, x)n(x) = tN(x) for x ∈ ΓN.

S = Sp(u, x) + Sa(Vm, η, u, x)

• F = I + Grad u the deformation gradient

• Sp the passive 2nd Piola-Kirchhoff stress tensor1,2

• Sa the active 2nd Piola-Kirchhoff stress tensor2,3

• uD the prescribed displacement

• tN the prescribed traction

• n the normal vector

• Vm the transmembrane voltage and

• η state variables

1 Holzapfel and Ogden 2009. Philos. Trans. R. Soc. Lond. Ser. A, pp. 3445–3475.2 Eriksson et al. 2013. Mathematics and Mechanics of Solids, pp. 592–606.3 Smith et al. 2004. Acta Numer., pp. 371–431.


3 / 23

Passive Stress Model

Constitutive equation using the free-energy function Ψ

Sp = 2∂Ψ(C)

∂C, Ψ(C) = U(F) + Ψiso(C) + Ψaniso(C), Ψ (locally) convex

nearly incompressible: penalty with κ

e.g., U(J) =κ

2(J − 1)2

, J = det F

isotropic components: ground matrix, elastin

e.g., Ψiso(C) =c

2(J−2/3 tr(C) − 3)

anisotropic components4,5: fibers, sheets

Ψaniso(C) =a

2b

exp[b(J−2/3If − 1)2] − 1

invariant If = Ff 0 · Ff 0: stretch in fiber direction

Loading

Unloading

0 10 20 30 40 50 60

100

200

300

400

500

600

Stress S (F/A), kPa

Sti

ffnes

sd

S/

dλ

,kP

a

4 Fung 1967. American Journal of Physiology, pp. 1532–1544.5 Eriksson, Gasser, Holzapfel, Ogden, 2000–2014


4 / 23

Electical Activation in the Myocardium

The Bidomain equations6,7 describe the spread of cardiac electrical activity

Figure : Bidomain representation ofcardiac tissue in 2D

find (Vm, φe, η) such that

∇ · (σi + σe)C−1

∇φe = −∇ · σiC−1

∇Vm,

∇ · σiC−1

∇Vm = −∇ · σiC−1

∇φe + β Im,

Im = Cm

∂Vm

∂t+ Iion(Vm, η, u),

∂η

∂t= f (Vm, η)

• Vm = φi − φe the transmembrane voltage

• φi, φe intra- and extracellular potential

• η a vector of state variables

• σi, σe conductivity tensors

• Im(Vm, η) transmembrane current flow

• Ii, Ie, Iion current densities

• β surface to volume ratio of cardiac cells

Simplification: in our experiments we replace C−1

by the identity matrix

6 Tung 1978. PhD thesis,7 Vigmond et al. 2007. Prog. Biophys. Mol. Biol., pp. 3–18.


5 / 23

Active Stress Models

Relaxed Contracted

Actin

Myosin+ ATP

+ Ca2+

(+)

(+)

(+)

(+)

(-)

(-)

Z disk

• Active stress is generated by electricalactivation in the myocardium

Sa = Sa(Vm, η, u) I−sf (f 0 ⊗ f 0)

with f 0 the myocyte fiber orientation

→ s = 12

mathematical, s = 1 mechanical choice

• Cell models to compute scalar-values stressterm Sa

→ Weakly coupled electromechanicse.g. NPStress9: Sa = ε(Vm)(kSa

Vm − Sa)

→ Strongly Coupled Electromechanicse.g. Rice10: h(Sa, Sa, Vm, η, λ, λ) = 0

8 Ambrosi and Pezzuto 2012. J. Elast., pp. 199–212.9 Nash and Panfilov 2004. Progress in Biophysics and Molecular Biology, pp. 501 –522.

10 Rice et al. 2008. Biophysical Journal, pp. 2368 –2390.


6 / 23

Outline


Parallel Strategies

Configuration

Numerical Examples



7 / 23

Motivation

High-resolution geometries

• meshes with up to N = O(109) degrees of freedom

• direct solvers: solving time O(N2), very high memory consumption

• iterative solvers: solving time O(N), lower memory consumption

→ we require strongly scalable parallel algorithms using iterative solvers


8 / 23

Why do we need HR geometries?

Easy to motivate in electrics:

• very small features influence wavepropagation

⇒ resolutions below 100 µm would beworthwhile

• i.e. cellular level, O(109) cells

Harder to do so in mechanics:• use same mesh as in electics

no data mapping or mesh coarsening needed

• some phenomena of interest involve small-scale features infarcts and ischemic regions multiple tissue layers and thin structures as papillary muscles, heart strings, valves

• parallel framework is available improves computational time for smaller meshes as well


9 / 23

Global Problem

Ω

FEM + Newton yields linearized system:

K′(uk)∆u = F − K(uk)

uk+1 = u

k +∆u.

Decomposition:p

∑

i=1

A⊤i K

′i (u

ki )Ai∆ui

solve with algebraic multigrid method (AMG11,12)


10 / 23

Algebraic Multigrid Method

Ω3

Ω5

Ω6

Ω7

Ω4

Ω2Ω1

FEM + Newton yields linearized system:

K′(uk)∆u = F − K(uk)

uk+1 = u

k +∆u.

Decomposition:p

∑

i=1

A⊤i K

′i (u

ki )Ai∆ui

solve with algebraic multigrid method (AMG11,12)

restriction prol

onga

tion

pre-smoothing

post-smoothing

fine grid

base level

11 Plank et al. 2007. Biomedical Engineering, IEEE Transactions on, pp. 585–596.12 Neic et al. 2012. IEEE Trans. Biomed. Eng., pp. 2281–2290.


10 / 23

Finite Element Tearing and Interconnecting

Ω1

Ω2

Ω3

Ω4

Ω5

Ω6

Ω7

FETI: Finite element tearing and interconnectingTearing:

K′1,k

. . .

K′p,k

∆u1

...∆up

= −

K 1,k

...K p,k

generally ∆ui 6= ∆uj on the interface ΓC

13 Farhat and Roux 1991. Int. J. Numer. Methods Engrg., pp. 1205–1227.14 Klawonn and Rheinbach 2010. ZAMM Z. Angew. Math. Mech., pp. 5–32.15 Augustin, Holzapfel, and Steinbach 2014. Int. J. Numer. Meth. Engrg., pp. 290–312.


11 / 23

Finite Element Tearing and Interconnecting

Ω1

Ω2

Ω3

Ω4

Ω5

Ω6

Ω7

FETI: Finite element tearing and interconnectingInterconnecting: with Lagrange multipliers λ andboolean matrices Bi

K′1,k B

⊤1

. . ....

K′p,k B

⊤p

B1 · · · Bp 0

∆u1

...∆up

λ

= −

K 1,k

...K p,k

0

K†

i,k a generalized inverse this yields

P⊤

p∑

i=1

Bi K†i,kB

⊤i λ = P

⊤

p∑

i=1

BiK†i,k f i .

13 Farhat and Roux 1991. Int. J. Numer. Methods Engrg., pp. 1205–1227.14 Klawonn and Rheinbach 2010. ZAMM Z. Angew. Math. Mech., pp. 5–32.15 Augustin, Holzapfel, and Steinbach 2014. Int. J. Numer. Meth. Engrg., pp. 290–312.


11 / 23

Software

• FETI: C. Pechstein, C. A.

• Cardiac electromechanics: CARP: G. Plank, E. Vigmond, C. A. PETSc: Krylov Solver Package with BoomerAMG (hypre)

→ general AMG for various problems Parallel Toolbox (PT): M. Liebmann, A. Neic, G. Haase

→ special AMG for electrics and mechanics

• Tarantula, ParaView, ParMetis, PaStiX, MUMPS, FEAP

Start electrics Solver mechanics

TIME++ S = Sp + Sa(Vm, η, u, f )

u


12 / 23

Outline


Parallel Strategies

Configuration

Numerical Examples



13 / 23

Geometry - MR Images

16 from the Visible Heart Lab (www.vhlab.umn.edu/)


14 / 23

www.vhlab.umn.edu/

Geometry - Smoothing

17 with K. Bredis and M. Holler (KFU Graz)


15 / 23

Myocardium Model

• Tetrahedral or hybrid meshes18

• Parameters19, fiber directions20

• Decomposition with ParMETIS

• Computations on SuperMUC Leibniz Rechenzentrum Munich Nr 12 in Top500 List - June 2014 147 456 cores

18 Prassl et al. 2009. IEEE Trans. Biomed. Engineering, pp. 1318–1330.19 Eriksson et al. 2013. Mathematics and Mechanics of Solids, pp. 592–606.20 Bayer et al. 2012. Ann. Biomed. Eng., pp. 2243–2254.


16 / 23

Circulatory System - Pressure BC

• A to B: loading

• B to C: isovolumetric contraction

• C to D: ventricular ejection

• D to E: isovolumetric relaxation

• E to B: refilling

C

B

A

E

D

70 140

0

3

16

LV volume (ml)

LV p

ressure

(kPA

)

Challenges:

• MRI images usually taken at point B→ unloading of the geometry needed

• in isovolumetric phases the cavityvolumes have to stay constant

• pressure volume realtions in ejectionand filling phase


17 / 23

Fixed Location Boundary Conditions

• heart covered by double layered membrane(pericard)

• space between layers is filled with fluid

• attached to diaphragm and pleura

• fix base (not physiological)

• contact problem21

• use bath from bid. model⇒ soft elastic material⇒ apply D-BC to bath

21 Fritz et al. 2013. Biomech Model Mechanobiol,


18 / 23

Fixed Location Boundary Conditions

• heart covered by double layered membrane(pericard)

• space between layers is filled with fluid

• attached to diaphragm and pleura

• fix vessel in- and outlets

21 Fritz et al. 2013. Biomech Model Mechanobiol,


18 / 23

Outline


Parallel Strategies

Configuration

Numerical Examples



19 / 23

10

100

1000

10000

64 128 256 512 1024 2048 4096

Number of processes [-]

AssemblingPC Setup

SolverMonodomain

Total

Scaling for Electromechanics - AMG

• 3 686 631 nodes

• 11 059 893 DOF

• 20 524 957 tets

• 100 timesteps

• ≈ 5 NS each


20 / 23

GPU


21 / 23

Outline


Parallel Strategies

Configuration

Numerical Examples



22 / 23


Ongoing work• finish GPU assembling for mechanics• TH-elements, dynamic model, adaptive timestepping, . . .• block system solvers and preconditioning• projections between fine and coarse mesh• fitting parameters to experiments (Cardioproof project)

Wish list• contact problems, hemodynamics and FSI


23 / 23

Ambrosi, D and S Pezzuto (2012). “Active stress vs. active strain in mechanobiology: constitutive issues”. In: J. Elast.

107.2, pp. 199–212.

Augustin, CM, GA Holzapfel, and O Steinbach (2014). “Classical and All-floating FETI Methods for the Simulation of

Arterial Tissues”. In: Int. J. Numer. Meth. Engrg. 99.4, pp. 290–312.

Augustin, CM and G Plank (2013). “Simulating the mechanics of myoardial tissue using strongly scalable parallel

algorithms”. In: Biomed Tech (Berl) i.

Augustin, CM and O Steinbach (2013). “FETI Methods for the Simulation of Biological Tissues”. In: Domain

Decomposition Methods in Science and Engineering XX, Lecture Notes in Computational Science and Engineering91, pp. 503–510.

Bayer, JD et al. (2012). “A novel rule-based algorithm for assigning myocardial fiber orientation to computational heart

models”. In: Ann. Biomed. Eng. 40(10), pp. 2243–2254.

Bols, J et al. (2011). “A computational method to assess initial stresses and unloaded configuration of patient-specific

blood vessels”. In: 5th International conference on Advanced COmputational Methods in ENgineering (ACOMEN2011). Université de Liège.

Braess, Dietrich and Regina Sarazin (1997). “An efficient smoother for the Stokes problem”. In: Applied Numerical

Mathematics 23.1, pp. 3–19.

Eriksson, TSE et al. (2013). “Influence of myocardial fiber/sheet orientations on left ventricular mechanical

contraction”. In: Mathematics and Mechanics of Solids 18, pp. 592–606.

Farhat, C and FX Roux (1991). “A method of finite element tearing and interconnecting and its parallel solution

algorithm”. In: Int. J. Numer. Methods Engrg. 32, pp. 1205–1227.


23 / 23

Fritz, T et al. (2013). “Simulation of the contraction of the ventricles in a human heart model including atria and

pericardium : Finite element analysis of a frictionless contact problem.” In: Biomech Model Mechanobiol.

Fung, YC (1967). “Elasticity of soft tissues in simple elongation”. In: American Journal of Physiology 213,

pp. 1532–1544.

Gurev, V et al. (2011). “Models of cardiac electromechanics based on individual hearts imaging data”. In: Biomechanics

and modeling in mechanobiology 10.3, pp. 295–306.

Holzapfel, GA, TC Gasser, and RW Ogden (2000). “A new constitutive framework for arterial wall mechanics and a

comperative study of material models”. In: J. Elasticity 61, pp. 1–48.

Holzapfel, GA and RW Ogden (2009). “Constitutive modelling of passive myocardium: a structurally based framework

for material characterization”. In: Philos. Trans. R. Soc. Lond. Ser. A 367.1902, pp. 3445–3475.

Kerckhoffs, RCP et al. (2007). “Coupling of a 3D finite element model of cardiac ventricular mechanics to lumped

systems models of the systemic and pulmonic circulation”. In: Annals of biomedical engineering 35.1, pp. 1–18.

Klawonn, A and O Rheinbach (2010). “Highly scalable parallel domain decomposition methods with an application to

biomechanics”. In: ZAMM Z. Angew. Math. Mech. 90.1, pp. 5–32.

Nash, MP and AV Panfilov (2004). “Electromechanical model of excitable tissue to study reentrant cardiac

arrhythmias”. In: Progress in Biophysics and Molecular Biology 85, pp. 501 –522.

Neic, A et al. (2012). “Accelerating cardiac bidomain simulations using graphics processing units”. In: IEEE Trans.

Biomed. Eng. 59.8, pp. 2281–2290.

Pathmanathan, P and JP Whiteley (2009). “A numerical method for cardiac mechanoelectric simulations”. In: Ann.

Biomed. Eng. 37.5, pp. 860–73.


23 / 23

Plank, G, AJ Prassl, and CM Augustin (2013). “Computational Challenges in Building Multi-Scale and Multi-Physics

Models of Cardiac Electro-Mechanics.” In: Biomed Tech (Berl).

Plank, G et al. (2007). “Algebraic multigrid preconditioner for the cardiac bidomain model”. In: Biomedical

Engineering, IEEE Transactions on 54.4, pp. 585–596.

Prassl, AJ et al. (2009). “Automatically Generated, Anatomically Accurate Meshes for Cardiac Electrophysiology

Problems.” In: IEEE Trans. Biomed. Engineering 56.5, pp. 1318–1330.

Rice, JJ et al. (2008). “Approximate Model of Cooperative Activation and Crossbridge Cycling in Cardiac Muscle Using

Ordinary Differential Equations"”. In: Biophysical Journal 95.5, pp. 2368 –2390.

Rumpel, T and K Schweizerhof (2003). “Volume-dependent pressure loading and its influence on the stability of

structures”. In: International Journal for Numerical Methods in Engineering 56.2, pp. 211–238.

Smith, NP et al. (2004). “Multiscale computational modelling of the heart”. In: Acta Numer. 13, pp. 371–431.

Tung, L (1978). “A bi-domain model for describing ischemic myocardial D-C potentials”. PhD thesis. MIT, Cambridge,

MA.

Vanka, SP (1986). “Block-implicit multigrid solution of Navier-Stokes equations in primitive variables”. In: Journal of

Computational Physics 65.1, pp. 138–158.

Vigmond, EJ et al. (2007). “Solvers for the cardiac bidomain equations.” In: Prog. Biophys. Mol. Biol. 96.1-3, pp. 3–18.

Westerhof, N, GIJS Elzinga, and P Sipkema (1971). “An artificial arterial system for pumping hearts.” In: Journal of

applied physiology 31.5, pp. 776–781.


23 / 23

Westerhof, N, J-W Lankhaar, and BE Westerhof (2009). “The arterial windkessel”. In: Medical & biological engineering

& computing 47.2, pp. 131–141.


23 / 23

strongly scalable parallel simulations of high-resolution ... · modeling and simulations in...

Documents