High Performance Parallel Computing with

iMODFLOW-MetaSWAP

Jarno Verkaik (Deltares)

Paul van Walsum (WUR/Alterra)

and

Joseph Hughes (USGS)

Edwin Sutanudjaja (UU)

Raju Ram (TUD)

6 December, 2017

Contents

• Problem statement and solution

• Context

• Parallelization method

• Benchmarks

• Practical usage with iMOD

SURFSARA-Cartesius: the Dutch supercomputer

Problem statement and solution

Problem statement:

• In order to support decision makers in solving hydrological problems, detailed high

resolution models are often needed.

• These models typically consist of a large number of computational cells and have

large memory requirements and long run times.

Solution:

• An efficient technique for obtaining realistic run times and memory requirements is

parallel computing, where the problem is divided over multiple processor cores.

6 december 2017

Problem statement and solution

Problem statement:

• In order to support decision makers in solving hydrological problems, detailed high

resolution models are often needed.

• These models typically consist of a large number of computational cells and have

large memory requirements and long run times.

Solution:

• An efficient technique for obtaining realistic run times and memory requirements is

parallel computing, where the problem is divided over multiple processor cores.

Development of the new Parallel Krylov Solver (PKS) package together with

the USGS, Utrecht University, WUR/Alterra and Delft University of Technology

6 december 2017

Context

• 2010: start with parallelization of MT3DMS

• 2012: first applied to the National Nutrient Model (H2O, vol18)

• 2016: start parallelization for the Netherlands Hydrological Model (NHM)

entirely funded by Deltares and Alterra from own investments

• 2017 July: release of iMOD 4.0 with PKS as main new functionality

• 2017 other: PKS added to iMOD-SEAWAT fresh-salt

Measured speedup for Sand Engine Model: ~40

• 2017 other: start of project with USGS to incorporate PKS in core MODFLOW

6 december 2017

Netherlands Hydrological Model (NHM)

Components:

• MODFLOW: 3D Groundwater flow

• MetaSWAP: 1D Soil Vegetation

Atmosphere Transfer (“SVAT”)

• TRANSOL : 1 D model of salinity

• MOZART: Surface water

Coupling:

• ID-based N:1 coupling tables

• Seamless MODFLOW-MetaSWAP

coupling with groundwater level as

shared state variable

6 december 2017

Parallelization method (1)

6 december 2017

The Parallel Krylov Solver* couples subdomains at

a tight matrix level using domain decomposition:

• The model grid is decomposed into

overlapping subdomains

• Each subdomain is assigned to 1 processor

• The processors exchange data through

Message Passing Interface (MPI)

Parallelization method (2)

6 december 2017

• Solving the groundwater flow equation

requires solving the linear system for heads ℎ:

𝐴ℎ = 𝑞 e.g. for two subdomains

𝐴11 𝐴12

𝐴21 𝐴22

ℎ1

ℎ2=

𝑞1

𝑞2

• Additive Schwarz iteration (block Jacobi):

A11h(k)

1=q1 - A12h(k-1)

2

A22h(k)

2=q2 - A21h(k-1)

1

• PKS uses the block Jacobi preconditioner M in CG

• Parallel solution = serial solution

PKS supports for iMODFLOW-MetaSWAP:

• Uniform blocks

• Load weighted blocks (Recursive Coordinate Bisection algorithm)

Parallelization method (3)

6 december 2017

Example “uniform”

128 subdomains Example “RCB”

128 subdomains

Benchmarks: NHM (1)

6 december 2017

• iMODFLOW-MetaSWAP + surface water

• Maximum measured speedup ~5 on

NHM Deltares server (Windows)

• Maximum theoretical speedup is

limited by surface water (< 1/0.06 16.7)

• Exactly the same heads are computed

with PKS as for the serial case

Amdahl’s law

Benchmarks: NHM (2)

6 december 2017

• iMODFLOW-MetaSWAP only

• SURFSARA-Cartesius Dutch National supercomputer (Linux)

• Maximum measured speedup ~24.

Further reading:

Deltares R&D Highlights 2016 https://www.deltares.nl/app/uploads/2015/02/RD-Highlights-2016_lowres.pdf

Page. 64

Benchmarks: California model

6 december 2017

* Simulated on the SURFSARA-Cartesius Dutch National Super Computer # On a single code estimated to consume 12 hours

MODFLOW only;

128 core (245 Gb RAM)*;

50 x 50 meter;

23,897 x 27,974 ≈ 335 million active nodes

2,6 Gb file size;

16:34 minutes#

Further reading:

Vermeulen et. al., Large scale high resolution modeling,

MODFLOW and More conference 2017, Golden, USA.

Practical usage with iMOD (1)

Easy-to-use in three steps:

1. Install MPI software (MPICH):

www.mpich.org/static/downloads/1.4.1p1/mpich2-1.4.1p1-win-x86-64.msi

2. Modify your run-file*, Dataset 5 (Solver Configuration)

3. Start your parallel job. E.g. starting from the DOS prompt using 4 cores:

mpiexec -localonly 4 iMODFLOW.exe imodflow.run

6 december 2017

Enable PKS with a “minus”

Same options as PCG

Partitioning method, flag for merging IDF output

* The new iMOD project file (.PRJ) is not yet supported for PKS.

6 december 2017

Practical usage with iMOD (2)

Try it yourself with Tutorial 6: Model Simulation!

Practical usage with iMOD (3)

Future to-dos for PKS:

• Support more packages: MNW, SFR, LAK, and iPEST

• Update iMOD Project Manager

• Improve line segment input (HFB, ISG) which may slow down speed up

• Improve output:

• Update iMOD batch for PKS for e.g. IDF merging

• Merge BDA-files for MetaSWAP output

• Improve CSV-file naming

• Support MicroSoft MPI for improved scheduling

• Coarse-grid RCB support

• …

6 december 2017

Practical usage with iMOD (4)

Take notice:

• Overall speed up strongly depends on hardware

• To get maximum speed-up for a hi-res model tuning is necessary (e.g. load

balancing)

• When considering new iMOD functionality, always check the impact on PKS

6 december 2017

!!! Thank you for your attention !!!

6 december 2017