Coupled Lattice Boltzmann and Molecular Dynamicssimulations on massively parallel computers

Jens Harting1,2, Stefan Frijters1, Florian Janoschek1,2, and Florian Günther1

1 Department of Applied Physics,Eindhoven University of Technology, Den Dolech 2, 5600MB Eindhoven, The Netherlands

E-mail: {f.a.r.janoschek, s.c.j.frijters, f.s.guenther}@tue.nl

2 Institute for Computational Physics,University of Stuttgart, Pfaffenwaldring 27, 70569 Stuttgart, Germany

E-mail: jens@icp.uni-stuttgart.de

Complex colloidal fluids play an important role in many branches of industry. Examples includeparticle-stabilized emulsions in cosmetics, food and oil industries. Their understanding requiresa study which resolves their microscopic structure, while still attaining sufficiently large lengthscales. The lattice Boltzmann method, owing to its high degree of locality, allows for suchstudies, when parallelized on modern supercomputers. However, exploiting these possibilitieson hundreds of thousands of cores is a non-trivial task. We report on our experiences whenemploying large fractions of the IBM Blue Gene/P system at the Jülich Supercomputing Centrefor our simulations and summarize recent results on particle-stabilized emulsions.

1 Introduction

Stabilizing emulsions by employing colloidal particles is a very attractive tool in the food,cosmetics and medical industries. The underlying microscopic processes of this stabi-lization can be explained by assuming an oil-water mixture. Without any additives, theseliquids will phase separate, but the mixture can be stabilized by adding small particles. Ifthese particles diffuse to the interface they will adsorb there, reducing interfacial free en-ergy and stabilizing the mixture. The particles in these mixtures block Ostwald ripening,which is one of the main processes leading to drop coarsening in emulsions. Thus, blockingthis process allows for long-term stabilization of these emulsions. The stabilized drops canbe used to produce new materials with complex hierarchical structure. Interestingly, manyof the properties of such systems can not be explained with theories derived for surfactant-stabilized systems. This is due to the larger size of the particles compared to the surfactantmolecules, and their lack of amphiphilic properties. New theoretical models have beendeveloped (and have been underlined experimentally), incorporating specific features ofparticle-stabilized systems that have no direct analogue in surfactant systems. Quantita-tively, however, the description of these systems still leaves to be desired. A promisingmethod to understand the dynamic properties of particle-stabilized multiphase flows canbe found in computer simulations. However, a suitable simulation algorithm must be ableto deal not only with simple fluid dynamics, but it is also required to simulate several fluidspecies and particle-particle and particle-fluid interactions. Some recent approaches tryingto solve these problems utilize the lattice Boltzmann (LB) method for the description ofthe solvents1. The LB method can be seen as an alternative to conventional Navier-Stokessolvers and is well-established in the literature. A number of multiphase and multicompo-nent models that are comparably straightforward to implement exist, making it attractive

1

for our current application. In addition, boundary conditions have been developed to sim-ulate suspended finite-size particles in flow. Microscopic studies such as these allow for adirect link to macroscopic experimental data and will lead to a sustantial improvement ofour understanding of particle-laden multiphase flows. These simulations remain computa-tionally very challenging because typical particles need to be at least 10 LB length unitsin diameter and typical droplet diameters should be larger by an order of magnitude. Toreduce finite-size effects and to acquire a statistically relevant number of droplets, the sidelength of the volume of interest easily reaches 1000-2000 LB length units—touching thelimit of what is currently possible on high-end supercomputers. Another constraint is givenby the very small time step in the simulations. The hydrodynamic interactions between theparticles need to be resolved on the same scale as the particle motion, pushing our timeresolution down to the nanosecond scale. To still be able to reach steady states, typicalsimulation runs require millions of LB time steps. This means that 0.5 to 1 rack months onthe Blue Gene/P system “JUGENE” in Jülich might be required for a single simulation.

2 Simulation method and implementation

We use the lattice Boltzmann (LB) method as the basis for our simulations1. In oursimulations, we effect the fluid-fluid interactions through the Shan-Chen multicomponentmodel2, 3, while we use a variant of Ladd’s method to couple the particles to our fluid4, 5.Particle-particle interactions and particle dynamics are modeled by a molecular dynamicsalgorithm. The code has recently been expanded to allow for spherical particles as well asellipsoidal ones. For a more detailed description of our simulation methods the reader isreferred to6, 7. The development of our simulation code LB3D started already in 1999 asa parallel LB solver. As the scientific focus at that time was on the behaviour of complexfluid mixtures under shear, the ability to model up to three fluid species, one of them withamphiphilic properties, using Shan and Chen’s aforementioned approach was one the firstfeatures implemented. Already in 2004, LB3D was awarded a gold star rating by the Edin-burgh Parallel Computing Centre for scaling almost linearly to 1024 processors and it alsowon several further prizes as part of the TeraGyroid project8. Further applications of thecode encompass flow in porous media9 and fluid interactions with rough and hydropho-bic surfaces10–13. In the meantime, the application was ported to most supercomputingplatforms available. A parallel molecular dynamics (MD) code was integrated into LB3Din 2008. After considerable changes and extensions both parts of the code now use thesame 3-dimensional spatial decomposition scheme, the Message Passing Interface (MPI)for communication, and modern Fortran 95 language elements. The simultaneous avail-ability of Lagrangian particles in the same code as the Eulerian LB fluid opened up newapplications for LB3D, such as tracking the velocity field in micromixers using masslesstracer particles14, a coarse-grained model for red blood cells15, or colloidal particles withvariable wettability as described in this report6. Due to the strong locality of the LB equa-tion and because the relevant interactions in colloidal systems are short-range, an efficientparallelization of such systems can be expected. In the past, LB3D indeed showed verygood scalability. On a machine such as JUGENE, the IBM Blue Gene/P system at JülichSupercomputing Centre (JSC), with 294912 cores, however, it is crucial to minimize com-munication costs and serial parts of a code even further in order to maintain high efficiencywhen running on large portions of the machine. Initially, LB3D showed only low effi-

2

1

2

4

8

16

32

64

128

256

1 2 4 8 16 32 64 128 256

speedup n

orm

aliz

ed to 1

024 c

ore

s

number of cores [1024]

beforeafterideal

(a)

1

2

4

8

16

32

64

128

256

1 2 4 8 16 32 64 128 256

speedup n

orm

aliz

ed to 1

024 c

ore

s

number of cores [1024]

beforeafterideal

(b)

Figure 1. Strong scaling of LB3D on the Blue Gene/P before and after our optimizations. (a) relates to a systemwith only one fluid component so the effect of matching or mismatching topologies of network and domaindecomposition can be examined better. (b) refers to a system with two fluid species and suspended particles asthey are of interest in this paper. The absolute execution times for small core counts did not change significantly(taken from Ref.7).

ciency there in strong scaling beyond 65536 cores. At the Jülich Blue Gene/P ExtremeScaling Workshop 2011 we could relate this to a mismatch of the network topology of thedomain decomposition in the code and the network actually employed for point-to-pointcommunication. The Blue Gene/P provides direct links only between direct neighbors ina three-dimensional torus, so a mismatch can cause severe performance losses. We inves-tigated this issue for a system of 10242×2048 lattice sites carrying only one fluid speciesand no particles. Limiting ourselves to this simpler system compared to the systems of in-terest later on has the benefit that the reduced ratio of computation to communication costsmakes possible bottlenecks more visible. We find that if we allow MPI_Cart_create() toreorder process ranks and manually choose a domain decomposition based on the knownhardware topology, efficiency can be brought close to ideal for this system. A comparisonof the speedup before and after this optimization is shown in Fig. 1(a). Systems containingcolloids and two fluid species were known to slowly degrade in parallel efficiency when thenumber of cores was increased. This is demonstrated in Fig. 1(b) for a benchmark systemof 10242× 2048 lattice nodes and 4112895 uniformly distributed particles with a radiusof 5 lattice units (resulting in a volume concentration of about 20%). Since the degradationwas not visible for a pure LB system (Fig. 1(a)), it initially was attributed to load imbal-ances and communication overhead related to the suspended particles. During the ScalingWorkshop we identified a non-parallelized loop over all particles in one of the subroutinesimplementing the coupling of the colloidal particles and the two fluids as the actual reason.Due to the low computational cost per iteration compared to the overall coupling costs forcolloids and fluids, at smaller numbers of particles or CPU cores this part of the code wasnot recognized as a possible bottleneck. A complete parallelization of the respective partsof the code produced a nearly ideal speedup up to 262144 cores also for this system. Both

3

speedup curves are depicted in Fig. 1(b). Actual production runs require checkpointing andthe output of physical observables to disk, which is not accounted for in the benchmarksabove. Pickering systems require output at most once per 100 to 1000 LB time steps, so theperformance of the output routines is of inferior importance. Using parallel HDF5 outputallows us to store fluid density fields of 4.6GB size for a system of 10242× 1152 latticesites and 454508 particles within on average 29s when using the whole system (294912cores). This corresponds to the time required to simulate 100 LB steps, so even at max-imum core count possible and maximum I/O frequency not more than 50% of the timeis spent on storing output. Particle configurations are typically smaller by two orders ofmagnitude than a density field. Therefore, this information is still written serially by theroot process. Since strongly asymmetric point-to-point communication patterns are likelyto produce buffer overflows and fail at high core counts, collective MPI operations are ap-plied for data accumulation. However, the intuitive choice of MPI_Gatherv() is not opti-mal since on Blue Gene/P, the specially optimized implementation of MPI_Allgatherv()proved to be significantly faster at the cost of requiring one receive buffer per task. ForMPI_Allgatherv() we measure an absolute time required to write one complete particleconfiguration of approximately 10s and a speedup of 77 on 131072 cores compared toMPI_Gatherv(). For even larger systems, however, MPI_Allgatherv() imposes a lowerlimit regarding the maximum number of particles, as receive buffers able to store the wholeparticle configuration need to be allocated four times per node (once per core) instead ofonly once on the node running the root process.

Figure 2. Droplets subjected to shear—but stabilized by particles—deform and might even break for high shearrates.

3 Simulation results

We apply our method to study the deformation of a liquid droplet under steady shear.Here, the simulation setup is a typical Couette flow where the upper and lower boundary

4

are moved by Lees-Edwards boundary conditions. A spherical droplet with radius R isinitially placed at the centre of the system, which can be characterized by its capillarynumber Ca≡ ηγ̇R

σ. The surface tension σ can be related to the Shan-Chen parameter gcc′ , η

is the dynamic viscosity and γ̇ is the shear rate. The deformation of the droplet is measuredby fitting an ellipse to a two-dimensional projection of the droplet, where L and B arethe length and breadth of the ellipse, respectively: D ≡ L−B

L+B . As shown by Taylor andreconfirmed in our simulations, for two fluids of equal density and equal viscosity oneobtains D = 35

32 Ca for D� 116, 17. While there have been numerous analytical studies,experiments and simulations on the deformation of simple liquid droplets within the lastcentury, the case of particle covered droplets has not been covered in such detail and itis still not fully clear how the stability and deformability changes when particles haveadsorbed to the liquid-liquid interface. In our simulations we can study these effects indetail by varying the droplet size, the number of particles, the particle size, their contactangle, as well as the shear rate and interfacial tension. Fig. 2 shows the deformation of aparticle-covered droplet for various moderate capillary numbers. As can be observed fromthe plot, the deformation shows an almost parabolic behaviour and a non-uniform particledistribution can be observed at the interface (see inset figures). For even larger shear rates,the droplets break up. The capillary numbers have been computed based on the radius ofthe large original droplet and if one would rescale it with the radius of the small dropletsoccurring after breakup, the curves would be shifted towards smaller Ca.

Figure 3. Snapshots of a 3D Bijel (a) and a Pickering emulsion (b): particles (green) self-assemble at the interfacebetween two immiscible fluids (shaded in red and blue) and stabilize a bicontinuous interface (a) or droplets (b).

We studied in detail how particles travel towards the fluid-fluid interface and getjammed there generating fluid-bicontinuous gels (so-called “Bijels”18) as shown in Fig. 3a.Further, it was demonstrated that by modifying the lyophobic/lyophilic properties of thesolved particles, the particles travel towards the fluid-fluid interface and accumulate at thesurface of a droplet thus forming a “Pickering emulsion” (see Fig. 3b). Large scale param-eter studies as well as investigations of the time dependent domain or droplet growth in Bi-jels and Pickering emulsions were performed. All simulations start from a random mixtureof fluids and particles, modelling a temperature quench. At the beginning of the simulation,

5

(a) (b)

Figure 4. Two snapshots of emulsions stabilized by particles are shown for an aspect ratio m = 2, a volumeconcentration of C≈ 0.2 and a contact angle θ = 90. (a) Pickering emulsion for a fluid radio of 5 : 2. (b) Bijel fora fluid radio of 1 : 17.

small scale droplets nucleate due to ballistic motion of particles and fluids. After the nu-cleation regime, the droplets grow due to spinodal decomposition. At some point the phaseseparation comes to a halt and droplets can only continue to grow due to coalescence (Ost-wald ripening), which is limited due to the particles that were captured at the fluid-fluidinterface. Ostwald ripening is one of the main processes leading to drop coarsening inemulsions and foams; hence stopping it allows long-term stabilization. Time-dependentmeasurements like these are very hard or even impossible to perform experimentally, butthe processes involved determine the final properties of the emulsion. Therefore, we expectto be able to contribute to a better understanding of how the final product depends on fluidcomposition, fluid properties, particle concentration, particle size, or the contact angle thefluids form with the particle surface. We have studied in detail the phase behaviour ofour system and determined a phase diagram demonstrating for which values of the particleconcentration, fluid decomposition or contact angle either a Bijel or a Pickering emulsionis formed6. It is our current main focus for these phases to understand the rheologicalproperties of Bijels and Pickering emulsions in detail. For this, further large-scale shearedsimulations will be performed in order to explore the complex non-Newtonian propertiesof these systems. Collecting the data from a number of such large-scale runs will allowus to quantitatively compare the size distribution of the stabilized droplets as well as therheological properties of such systems with experimental data.

We also studied emulsions stabilized by non-spherical, anisotropic colloidal particles:ellipsoids with an aspect ratio of m = 2. Also for these ellipsoids, we observe two differentphases: the Pickering emulsion (see Fig. 4(a)) and the Bijel (see Fig. 4(b)). Fig. 5 showsthe transition from a Bijel to a Pickering emulsion for these ellipsoids with an aspect ratioof m = 2 and a particle volume concentration of C≈ 0.2. The two control parameters usedfor the study of the phase transition are the fluid ratio and the contact angle θ. The squaresdenote the configurations which lead to a Bijel whereas the circles denote a resulting Pick-ering emulsion. If the amount of the two fluids present in the simulation is equal or not too

6

90 110 130

Flu

id r

atio

Contact angle θ[°]

1:1

4:3

9:5

5:2

Figure 5. Phase diagram demonstrating the transition from a Bijel to a Pickering emulsion. The contact angleθ and the density ratio between the two fluids are varied. The squares denote the configurations which lead to aBijel whereas the circles denote a Pickering emulsion7.

different (e.g. a ratio of 4 : 3) we find a Bijel for all considered contact angles. However, ifthe fluid ratio is increased we find a Pickering emulsion. For intermediate fluid ratios theobtained phase depends on the chosen contact angle. For example for a ratio of 9 : 5 weget a Bijel for a contact angle of 90◦ and a Pickering emulsion for all higher values of θ.

4 Conclusion and outlook

Suspensions of colloids in multiphase flows can exhibit many surprising effects, which areof potential interest to both fundamental science and industrial applications.

Our investigation of the effect of colloids on the deformability of droplets has shownthat when the fluid-fluid interface is almost saturated with colloids, the deformability of adroplet at a constant capillary number increases dramatically, and the breakup behaviourof those droplets at higher capillary number is also changed quantitatively. The resultsobtained for the particle-covered droplets will be compared to surfactant-covered dropletsin similar systems, making a quantitative link between the two.

On slightly larger scale, we investigated the behaviour of the particle-stabilized Pick-ering emulsions and Bijels, and found that the arrival at either phase after a temperaturequench depends on many properties of the system, such as fluid ratio, and size and con-tact angle of the colloids. Our simulation code has been extended to include ellipsoidalcolloids, which will introduce another parameter that can influence the formation of a par-ticular phase. In the future, Janus particles will also be considered, having different wettingproperties on different parts of the surface of the particle.

Acknowledgments

Besides support from JSC and IBM experts present at the Blue Gene/P Extreme Scalingworkshop 2011 we acknowledge computing resources that were granted via GSC grantshss08 and hss10 and by PRACE (project pra036).

7

References

1. S. Succi, The Lattice Boltzmann Equation for Fluid Dynamics and Beyond, NumericalMathematics and Scientific Computation. Oxford University Press, Oxford, 2001.

2. X. Shan and H. Chen, Lattice Boltzmann model for simulating flows with multiplephases and components, Phys. Rev. E, 47, no. 3, 1815–1819, Mar 1993.

3. H. Chen, B. M. Boghosian, P. V. Coveney, and M. Nekovee, A ternary lattice Boltz-mann model for amphiphilic fluids, Proc. R. Soc. Lond. A, 456, 2043, 2000.

4. C.K. Aidun, Y. Lu, and E.-J. Ding, Direct analysis of particulate suspensions withinertia using the discrete Boltzmann equation, J. Fluid Mech., 373, 287, 1998.

5. A.J.C. Ladd and R. Verberg, Lattice-Boltzmann simulations of particle-fluid suspen-sions, J. Stat. Phys., 104, no. 5/6, 1191–1251, 2001.

6. F. Jansen and J. Harting, From bijels to Pickering emulsions: A lattice Boltzmannstudy, Phys. Rev. E, 83, no. 4, 046707, Apr 2011.

7. F. Günther, F. Janoschek, S. Frijters, and J. Harting, “Complex colloidal fluids: en-abling a lattice Boltzmann solver for massively parallel simulations on the BlueGene/P”, submitted for publication, 2011.

8. J. Chin, P. V. Coveney, and J. Harting, The TeraGyroid project – collaborative steeringand visualization in an HPC grid for modelling complex fluids, in: Proceedings of theUK e-Science All Hands Meeting, 2004.

9. A. Narváez, T. Zauner, F. Raischel, R. Hilfer, and J. Harting, Quantitative analysisof numerical estimates for the permeability of porous media from lattice-Boltzmannsimulations, J. Stat. Mech: Theor. Exp., 2010, P211026, 2010.

10. J. Harting, C. Kunert, and H.J. Herrmann, Lattice Boltzmann simulations of apparentslip in hydrophobic microchannels, Europhys. Lett., 75, 328–334, 2006.

11. C. Kunert and J. Harting, Roughness induced apparent boundary slip in microchannelflows, Phys. Rev. Lett., 99, 176001, 2007.

12. J. Hyväluoma and J. Harting, Slip flow over structured surfaces with entrapped mi-crobubbles, Phys. Rev. Lett., 100, 246001, 2008.

13. C. Kunert, J. Harting, and O.I. Vinogradova, Random-roughness hydrodynamicboundary conditions, Phys. Rev. Lett., 105, 016001, 2010.

14. A. Sarkar, A. Narváez, and J. Harting, “Quantification of the degree of mixing inchaotic micromixers using finite time lyapunov exponents”, submitted for publication;arXiv:1012.5549, 2011.

15. F. Janoschek, F. Toschi, and J. Harting, Simplified particulate model for coarse-grained hemodynamics simulations, Phys. Rev. E, 82, 056710, 2010.

16. G.I. Taylor, The Viscosity of a Fluid Containing Small Drops of Another Fluid, Proc.R. Soc. Lond. A, 138, no. 834, 41–48, 1932.

17. G.I. Taylor, The Formation of Emulsions in Definable Fields of Flow, Proc. R. Soc.Lond. A, 146, no. 858, 501–523, 1934.

18. K. Stratford, R. Adhikari, I. Pagonabarraga, J.-C. Desplat, and M.E. Cates, ColloidalJamming at Interfaces: A Route to Fluid-Bicontinuous Gels, Science, 309, no. 4,2198–2201, 2005.

8