parallel multigrid for electrokinetic simulation in

Parallel Multigrid for Electrokinetic Simulation in Particle-Fluid Flows

Dominik Bartuschat,Madrid, SpainJuly 4, 2012

International Conference on High Performance Computing & Simulation (HPCS) 2012

Mittwoch, 4. Juli 2012

D.Bartuschat, D.Ritter, U.Rüde

Chair for System Simulation, FAU Erlangen-Nürnberg


Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])

Outline

● Motivation● The waLBerla Simulation Framework● Charged Particles in Fluid Flow● Multigrid Solver● Results

3Mittwoch, 4. Juli 2012

mailto:[email protected]



Motivation

4

Simulating agglomeration of charged particles in (micro-)fluid flow on charged plane.

● Medical applications:● Deposition of charged aerosol particles in

respiratory tract (e.g. drug delivery).● Optimization of Lab-on-a-Chip systems:● Trapping cells and viruses.● Separation of different cells.

© Kang and Li „Electrokinetic motion of particles and cells in microchannels“ Microfluidics and Nanofluidics

● Industrial applications:● Filtering particulates from exhaust gases.● Charged particle deposition in cooling

systems of fuel cells.




The waLBerla Simulation Framework



waLBerla● Widely applicable Lattice Boltzmann framework.● Suited for various flow applications.● Large-scale, MPI-based parallelization.● Dynamic application switches for heterogeneous architectures and

optimization.





waLBerla Concepts

7

Block concept:● Domain partitioned into cartesian grid of blocks.● Blocks can be assigned to different processes.● Blocks contain:

● cell data, e.g. electric potential.● global information e.g. MPI rank, location.

Communication concept:● Simple communication mechanism on uniform grids, utilizing MPI.● Ghost layers to exchange cell data with neighboring blocks.

Sweep concept:● Sweeps are work steps of a time-loop, performed on block-parallel level.● Example: MG sweep, contains sub-sweeps (restriction, prolongation, smoothing).




Charged Particles in Fluid Flow


● Discrete lattice Boltzmann equation (single relaxation time).● Domain discretized in cubes (cells).● Discrete velocities ci and associated distribution functions fi per cell.


Lattice Boltzmann Method

9

D3Q19 model

fi (x + �ci∆t, t +∆t)− fi (x , t) = −1

τ(fi − f eqi ).

⟶





Stream-Collide

● Stream step

10

The equation is solved in two steps:

● Collide step

fi (x + �ci∆t, t +∆t) = f̃i (x , t +∆t)

f̃i (x , t +∆t) = fi (x , t)−1

τ(fi − f eqi )





Fluid-Particle Interaction - Coupling waLBerla and pe

● Particles are mapped onto lattice Boltzmann grid.● Each lattice node with cell center inside object is treated as moving boundary.● Hydrodynamic forces of fluid on particle computed by momentum exchange method*.

11

* D.Yu, R. Mei, L.-S. Luo, W.Shyy „Viscous flow computations with the method of lattice Boltzmann equation“





Poisson Equation and Coulomb Force on Particles

● Electric potential described by Poisson equation with particle charge density on RHS:

● Discretized by finite differences, solved with multigrid (MG) solver.● Electrostatic force on particle:

● Force density added to pe for charged particle cells:

12

−∆Φ(x) =ρparticles (x)

�r �0

−→f (x̄b) = −ρ(x̄b)

Q�

i=1,i �=C

titC

Φ(x̄b −−→ci ) ·−→ci

−→Fq = −qparticle ·∇Φ(x)

ci: discrete lattice velocity ti: direction dependent weighting factor





Charged Particles Algorithm

13

foreach time step, do// solve Poisson equation with particle charge density

set RHS of Poisson equation

while residual too high doapply multigrid solver to Poisson equation

calculate and MPI Reduce residual norm

// solve lattice Boltzmann equation considering particle velocities

beginstream PDFs (stream step)

calculate macroscopic variables

relax towards equilibrium PDF (collide step)

MPI-communicate PDFs to ghost layers

end// couple potential solver and LBM with pebegin

calculate and add hydrodynamic force on particles

calculate and add electrostatic force on particles

pe moves particles depending on forces

end




Multigrid Solver


● Based on● Smoothing principle:

High-frequency error elimination by iterative solvers (e.g. GS).

● Coarse grid principle: Restriction to coarser grid transforms low-frequency error components to relative higher-frequency ones.

● Smoothing applied on coarse grids.● Prolongation of obtained correction

terms to finer grid.

● Iterative method for efficient solution of sparse linear systems.


Multigrid

15

● Applied recursively, v(νpre, νpost)-cycle.




● All operations implemented as compact stencil operations.

● Design goals:● Efficient and robust black-box solver.● Handling complex boundary conditions on coarse levels.● Naturally extensible to jumping coefficients.

➡ Method of choice: Galerkin coarsening.

● Finite difference stencils stored for each unknown.

● Averaging restriction, nearest-neighbor prolongation.● Preserves D3Q7 stencil on coarse grids.● Violates constant convergence rate.● Workaround for Poisson problem: Overrelaxing prolongation*.


Cell-Centered Multigrid - Implementation

16

* Bramble et. al „The analysis of multigrid algorithms for cell centered finite difference methods“, Adv. Comput. Math




Results



Charged Particles in Fluid Flow

18

Agglomeration of charged particles on charged plane in water flow.

Channel: 2.56 x 5.76 x 2.56 mmDx=10µm, Dt=4⋅10-5s, τ=1.7

Particle radius: 60µmParticle charge: 8000e

Inflow velocity: 1mm/sOther walls: No-slip BCs

Potential: Bottom -100V, Top 0V Other walls: homogen. Neumann BCs

•Computed on 144 cores (12 nodes)•71.600 time steps•643 unknowns per core•6 MG levels


http://www10.informatik.uni-erlangen.de/permalink/igahque9ie

http://www10.informatik.uni-erlangen.de/permalink/igahque9ie




19

Scaling Setup and Environment Weak scaling:

● Costant size per core:● 1283 cells.● 9.4% moving obstacle cells.

● Size doubled (y-dimension).● MG (Residal L2-Norm ≤ 10-9):

● v(2,2) with 7 levels.● 100 RBGS coarse-grid iterations.

● 3x2x2 cores per node:

Strong scaling:● Constant total size:

● 384x128x256 cells.● 9.4% moving obstacle cells.

● # nodes doubled (all dimensions).● MG (Residual L2-Norm ≤ 10-9):

● v(2,2) with 6 levels.● 180 RBGS coarse-grid iterations.

● 3x2x2 cores per node.

Executed on RRZE‘s LiMa cluster:● 500 compute nodes, each:

● 2 Xeon 5650 "Westmere-EP" chips @2.66 GHz (2 x 6 cores),

● 24 GB DDR3 RAM.● Infiniband interconnect (40 Gbit/s).




! " # $ % & ' ( ) !* !! !"*

"

$

&

(

!*

!"

+,-./0-1230/,,4

56

785

9:2;3<0,=0>,<3?

@433A:4


20

● LBM scales very well.● MG speedup:

● Saturation for more than 6 cores,● memory bandwidth limitation.

Single Node PerformanceWeak scaling (first time-step, size doubled here in different dimensions)

Highest performance with 12 cores➡ used for further scaling experiments.Total run-time

6.7sLBM0.67s

MG (8 v-cycles)6.4s





21

LBM Scaling Results

! " # $ % &! &" &# &$ &%!

&!!

"!!

'!!

#!!

(!!

$!!

)*+,-./+0123

456723-./+0123

89*+0-./+0123

:;<=*6-7>-279*.

?@ABC4

! " # $ % &! &" &# &$ &%!

&!

"!

'!

#!

(!

$!

)*+,-./+0123

456723-./+0123

89*+0-./+0123

:;<=*6-7>-279*.

?ABC4

MFLUPS: Mega fluid lattice site updates per second● Nearly ideal weak scaling (communication 2.2% ➞ 3.4%).● Very good strong scaling (communication 2.5% ➞ 6.9%).





22

MG Scaling Results

! " # $ % &! &" &# &$ &%!

&!!

"!!

'!!

#!!

(!!

$!!

)*+,-./+0123

456723-./+0123

89*+0-./+0123

:;<=*6-7>-279*.

?@ABC4

! " # $ % &! &" &# &$ &%!

&!

"!

'!

#!

(!

$!

)*+,-./+0123

456723-./+0123

89*+0-./+0123

:;<=*6-7>-279*.

?ABC4

! " # $ % &! &" &# &$ &%!

&!!

"!!

'!!

#!!

(!!

$!!

)*+,-./+0123

456723-./+0123

89*+0-./+0123

:;<=*6-7>-279*.

?ABC4-D72*-EF/G/0*H

8 v-cycles, in first time-step (later 3) MLUPS: Mega lattice site updates per second● Very good weak scaling (communication 4.4% ➞ 7.0%).● Good strong scaling (communication 4.6% ➞ 17.6%).➡ Limited by many small messages on coarsest grid.





Summary

● Parallel multiphysics algorithm for charged particles in fluid flow.● Cell-centered multigrid with variable stencils for Poisson problem.● Performance results of main components (preliminary).

● Performance evaluation on single (many-core) node● Scaling experiments on several nodes.

● Possible MG improvements:● Splitting of red and black variables (of RBGS).● Implementation of more efficient coarse-grid solver (e.g. Conjugate Gradients).● Higher order prolongation scheme, conserving seven-point stencils.




Thank you for your attention!


parallel multigrid for electrokinetic simulation in

Documents