Methods Note/

Distributed Parallel Computing in StochasticModeling of Groundwater Systemsby Yanhui Dong1, Guomin Li2, and Haizhen Xu1

AbstractStochastic modeling is a rapidly evolving, popular approach to the study of the uncertainty and heterogeneity of

groundwater systems. However, the use of Monte Carlo-type simulations to solve practical groundwater problemsoften encounters computational bottlenecks that hinder the acquisition of meaningful results. To improve thecomputational efficiency, a system that combines stochastic model generation with MODFLOW-related programsand distributed parallel processing is investigated. The distributed computing framework, called the Java ParallelProcessing Framework, is integrated into the system to allow the batch processing of stochastic models indistributed and parallel systems. As an example, the system is applied to the stochastic delineation of well capturezones in the Pinggu Basin in Beijing. Through the use of 50 processing threads on a cluster with 10 multicorenodes, the execution times of 500 realizations are reduced to 3% compared with those of a serial execution.Through this application, the system demonstrates its potential in solving difficult computational problems inpractical stochastic modeling.

IntroductionThe uncertainty and heterogeneity of groundwater

system modeling have been gaining increasing inter-est. With the development of relevant theories andapproaches, stochastic modeling has become a popu-lar method of studying uncertainties and heterogeneities.Stochastic modeling has been widely used in a number ofstudies, including, but not limited to, parameter estima-tions (Yeh and Simunek 2002; Franssen and Kinzelbach2008), well capture zone delineations (Vassolo et al. 1998;

1Key Laboratory of Engineering Geomechanics, Institute ofGeology and Geophysics, Chinese Academy of Sciences, P.O. BOX9825, Beijing, China; lemondyh@mail.iggcas.ac.cn; hzxu-snower@mail.iggcas.ac.cn

2Corresponding author: Key Laboratory of EngineeringGeomechanics, Institute of Geology and Geophysics, ChineseAcademy of Sciences, P.O. BOX 9825, Beijing, China; guominli@mail.iggcas.ac.cn

Received February 2012, accepted June 2012.© 2012, The Author(s)Ground Water © 2012, National Ground Water Association.doi: 10.1111/j.1745-6584.2012.00967.x

van Leeuwen et al. 1999; Feyen et al. 2004), and in theinterpretation of breakthrough curves (Lemke et al. 2004;Zhang et al. 2007).

Despite the growing research initiatives in the fieldof stochastic modeling, the use of Monte Carlo-typestochastic modeling techniques to solve problems inheterogeneous aquifers remains limited, particularly inpractical groundwater problems. The excessive compu-tational demands in solving a large number of stochasticrealizations may be the crucial reasons for this limitation(Li et al. 2003; Neuman 2004).

The current article presents a distributed parallelcomputing method for stochastic modeling with MOD-FLOW (Harbaugh et al. 2000) and related programs.Through application in the stochastic delineation of wellcapture zones, the distributed parallel processing frame-work employed demonstrates effectiveness and efficiencyin solving time-consuming practical stochastic model-ing problems. Given the broad use of MODFLOW, thisefficient framework may be applied to more relevantmodeling applications.

NGWA.org GROUND WATER 1

Methods—The Java Parallel ProcessingFramework

Java Parallel Processing Framework (JPPF) is anopen-source distributed computing solution first releasedin 2005. JPPF is easy to set-up and use, and its task-oriented design is very powerful yet user-friendly. Thesefeatures allow developers to focus on their core compe-tence, which is software development, instead of wastingtime on the complexities of parallel and distributed pro-cessing. As a 100% Java framework, JPPF will run onany system that supports Java, including Mac OS, Win-dows, Linux, and z/OS, as well as on any hardware, froma simple laptop up to a mainframe computer. The JPPFframework can run Java programs, as well as any externalexecutable program, shell script, or command.

The interconnection architecture of JPPF is basedon the master-worker model. As illustrated in Figure 1,the model consists of servers (i.e., master), clients, andnodes (i.e., workers). Servers play a central role inthe framework, receiving work from the clients anddispatching these to the nodes. Clients are entry pointsinto the framework and enable developers to submit theirwork. Nodes are responsible for the actual work execution.Each node can process multiple tasks concurrently using apool of threads to make use of multicore processors. JPPFcan connect multiple servers together, thereby allowingservers to delegate some of their workload to otherservers. JPPF is equipped with additional connectivityoptions for clients and nodes, which provide a failovermechanism as well as load-balancing capabilities.

The work is distributed and classified into two units:the job and the task. The task is defined as the smallest unitof work that can be handled in the framework, whereasthe job is a logical grouping of tasks submitted together.By specifying a number of behavioral characteristics,developers can decide how the job’s work is distributedin the framework. Figure 2 shows the most common flowof a job’s execution.

Load-balancing algorithms enable the use of resourcesthat each node possesses by adjusting the number of taskssent to each node. The distribution of tasks to the nodes

Figure 1. Architecture and topology of JPPF (fromwww.jppf.org).

Figure 2. Flow chart of JPPF (from www.jppf.org).

is computed by load-balancing algorithms. The task dis-tribution is actually the main factor that determines theobserved performance of JPPF. Four predefined load-balancing algorithms can be used in JPPF. One simplealgorithm is the MANUAL algorithm, wherein each nodereceives a fixed number of tasks.

JPPF also provides a graphic tool for managementand monitoring. This tool displays the status, performance,job behavior, and framework components. Through thistool, the framework can be remotely administrated byremoving and rebooting the nodes.

Additional information on JPPF can be found athttp://www.jppf.org.

ApplicationMODFLOW and related programs such as MT3DMS

(Zheng and Wang 1999) and MODPATH (Pollock 1988)have been widely used in the stochastic modeling ofgroundwater problems. From an operational point ofview, the simulation of all the realizations is highlysuitable for parallel computing because each model can beindependently treated and simultaneously run on differentprocessors. Mainly written in FORTRAN language, theseprograms have high portability, which allows theirexecution even in a heterogeneous, parallel environment.

JPPF takes advantage of Java thread applications.For an application using threads, every thread can likelybe executed on a node in the JPPF environment. Asidefrom executing native Java applications, JPPF can alsoexecute any kind of nonnative Java program through apredefined task type, the CommandLineTask. Thus, thecodes of MODFLOW-related programs do not requiremodifications. Through its job/task and its load-balancingmechanism, JPPF can adapt existing MODFLOW-related

2 Y. Dong et al. GROUND WATER NGWA.org

Figure 3. System overview.

programs in a distributed environment with minimaleffort.

The use of a stochastic delineation of well capturezones is presented as an example to illustrate the mecha-nism by which the distributed parallel computing approachoperates. In the application, conditional Monte Carlo tech-niques are employed to evaluate the effect of hydraulicconductivity uncertainties in a groundwater system. Theprocedure can be briefly summarized as follows:

Step 1. A number of realizations for the hydraulicconductivity field are generated using the sequentialGaussian simulation (SGSIM) method (Deutsch andJournel 1998).

Step 2. Each field realization is used to generate a setof input files for MODFLOW and MODPATH.

Step 3. MODFLOW is used to solve all models,thereby resulting in a set of head distributions. Subse-quently, MODPATH is used as a particle-tracking methodfor each head field.

Step 4. Statistical processing for each realization isperformed, which results in a probability distribution ofthe capture zone.

Except for Step 1, the other steps can be processedin parallel on the computing nodes. Figure 3 shows theoverview of the entire system.

The stochastic delineation method was applied toa well field located in the Pinggu Basin in Beijing,China. A description of the hydrogeological conditionsand a deterministic delineation of the well capture werepreviously presented (Xu et al. 2011). The groundwatermodel, including an area of approximately 459 km2, wasdiscretized into 130 rows, 190 columns, and 10 layers.The well (WEL) package was used to simulate thepumping wells in the well field, whereas the areal rechargeand discharge method (Dong et al. 2012) was employedto process the 1209 agricultural wells. According toover 200 borehole data, 500 realizations of the hydraulicconductivity field were generated using the SGSIMmethod. The transient flow model, containing 74 stressperiods, was solved using the preconditioned conjugategradient (PCG) solver of MODFLOW (Hill 1990).

PerformanceTiming experiments were performed on a cluster with

1 master node and 10 computing nodes. Each computingnode was equipped with two Intel Xeon E5335 quad-coreprocessors and an 8-GB RAM. The quad-core processoris a single computing component with four independentactual processors (called “cores”), which are the units that

NGWA.org Y. Dong et al. GROUND WATER 3

read and execute program instructions. One JPPF serverwas deployed on the master node, and one JPPF nodewas deployed on each computing node. A JPPF node canconcurrently process up to eight tasks (one task per thread)because eight cores are available to each computing node.The number of processing threads in the configure file canvary to obtain the best performance. Given the architectureof Xeon E5335, the concurrent processing of five threadscan achieve a better performance based on benchmarktests.

The speedup, which is defined as the ratio of the serialexecution time to the parallel execution time, was usedto assess the performance of parallel computing and toillustrate the reduction in the running time. On the otherhand, the efficiency, which is defined as the ratio of thespeedup to the number of processors (in this case, thenumber of processing threads), was also used to measurethe fraction of the time used by the processors to performthe actual computations.

The serial execution of MODFLOW and MODPATHfor one realization took 5 to 6 min using the PCG solverwith the modified incomplete Cholesky preconditioner(MICCG) on one single node. This duration means thatthe entire stochastic delineation process would possiblytake 2 d.

The benefit of parallel processing is clearly shownin Figure 4, which reports the speedup and efficiencyusing different numbers of cores. The speedup goes upwith increasing number of cores. Moreover, the maxi-mum speedup of 30.81 was achieved using 50 cores. Theefficiency goes down from 1 to 0.62 as the number ofcores increased because the communication and synchro-nization due to the JPPF task serialization/deserializationand transportation can affect the performance to a greaterextent when the number of processing threads is increased.

In profiling the execution of the whole program, theMODFLOW simulation time accounts for over 99% ofthe running time. This distributed parallel processing ofmodels for the application has achieved remarkably goodperformance. The 2-d execution time was reduced toapproximately 5 h using 10 cores, and to less than 2 husing 50 cores.

For large MODFLOW models with massive grids ofover 10,000,000, the performance of the JPPF applicationcan be further enhanced by combining the application withthe parallel PCG solver (Dong and Li 2009). The memory

Figure 4. Speedup and efficiency versus number of cores.

Figure 5. Parallel processing times using the combinedmethod: (A) using the original serial POLCG solver for eachtask, and (B) using the parallel POLCG solver with fourthreads for each task. Ten computing nodes were used, andtwo tasks were executed on each node.

requirement of large models may be approximately 1 GB,or even 10 GB, depending on the grid size. One or twoprocessing threads can usually be used on a JPPF nodebecause of the limitation in RAM size. In this case, theuse of a parallel PCG solver ensures that more cores canbe used because it does not require any additional RAMwhen using more cores and it is typically 1.40 to 5.31times faster than the serial PCG solver.

The combined approach exploits two levels ofparallelisms in the software (coarse-grained parallelism inthe MODFLOW calculation, and fine-grained parallelismin the PCG solver) and in the hardware (computingnodes with multiple cores). Thus, the application cantake advantage of the total computational resources ofthe cluster. Given that the parallel PCG solver with thepolynomial preconditioner (POLCG) can achieve betterspeedup than the MICCG for large models (Dong and Li2009), the parallel POLCG was combined with JPPF tosolve the previous problem. Although parallel POLCG canreduce long execution time compared with serial POLCG,the execution time of the parallel POLCG may be longerthan that of the serial MICCG for a small model. However,POLCG can still be used to estimate the performance ofthe combined parallel method compared with that of theregular JPPF method.

Figure 5 presents the possible improvements madeusing the combined approach. The application wasperformed using 10 computing nodes. Given that eachcomputing node had eight cores, two tasks were executedon each node, while four threads were used by the parallelPOLCG solver to process each task. As a benefit from theparallel PCG solver, the total execution time was reducedto 46% compared with the use of the serial PCG solver.

ConclusionsThe Monte Carlo-type stochastic modeling with

MODFLOW-related programs can be processed in parallelusing a large number of computing cores through integra-tion with the distributed computing framework JPPF. Theapplication of stochastic delineation of well capture zonesin the Pinggu Basin shows the high performance of JPPF

4 Y. Dong et al. GROUND WATER NGWA.org

in reducing the execution time from days to a few hours.The parallel PCG solver of MODFLOW can also be inte-grated to provide a two-level parallel processing systemthat fully utilizes the multiprocessors.

The best feature of JPPF is its ease of use.JPPF effectively allows the utilization of existing codes,processes, and algorithms in a distributed environmentwith minimal effort and time. The distributed parallelprocessing approach described in this study is applicablenot only to MODFLOW-related programs, but also tomost groundwater modeling codes. The computationalresources are not limited to supercomputers or clusters,and all types of desktop PCs and office laptops can beintegrated to provide a distributed parallel processingnetwork. A specified mode can be used to enablea computing node to work only when the machineis idle.

AcknowledgmentsWe are grateful for the review comments and

suggestions from Dr. Frank W. Schwartz, and Dr. SteffenMehl. We would also like to thank the anonymousreviewer who made significant improvements to themanuscript. This research was supported by the NationalProgram on Key Basic Research Project of China (GrantNo. 1192010CB428801).

ReferencesDeutsch, C.V., and A.G. Journel. 1998. GSLIB: Geostatistical

Software Library and User’s Guide, 2nd ed. New York:Oxford University Press.

Dong, Y.H., and G. Li. 2009. A Parallel PCG Solver forMODFLOW. Ground Water 47, no. 6: 845–850.

Dong, Y.H., G. Li, and H.Z. Xu. 2012. An areal recharge anddischarge simulating method for MODFLOW. Computers& Geosciences 42: 203–205.

Feyen, L., A.M. Dessalegn, F. De Smedt, S. Gebremeskel, andO. Batelaan. 2004. Application of a Bayesian approach tostochastic delineation of capture zones. Ground Water 42,no. 4: 542–551.

Franssen, H. J. H., and W. Kinzelbach. 2008. Real-time ground-water flow modeling with the Ensemble Kalman Filter:

Joint estimation of states and parameters and the filterinbreeding problem. Water Resources Research 44,W09408. DOI: 10.1029/2007WR006505.

Harbaugh, A.W., E.R. Banta, M.C. Hill, and M.G. McDonald.2000. MODFLOW-2000, the U.S. Geological Surveymodular ground-water model—User guide to modulariza-tion concepts and the ground-water flow process. USGSOpen-File Report 00-92. Reston, Virginia: U.S. GeologicalSurvey.

Hill, M.C. 1990. Preconditioned conjugate-gradient 2 (PCG2), acomputer program for solving ground-water flow equations.U.S. Geological Survey Water-Resources InvestigationsReport 90-4048. Reston, Virginia: USGS.

Lemke, L.D., W.A. Barrack, L.M. Abriola, and P. Goovaerts.2004. Matching solute breakthrough with deterministicand stochastic aquifer models. Ground Water 42, no. 6:920–934.

Li, S.G., D. McLaughlin, and H.S. Liao. 2003. A computation-ally practical method for stochastic groundwater modeling.Advances in Water Resources 26, no. 11: 1137–1148.

Neuman, S.P. 2004. Stochastic groundwater models in practice.Stochastic Environmental Research and Risk Assessment18, no. 4: 268–270.

Pollock, D.W. 1988. Semianalytical computation of path linesfor finite-difference models. Ground Water 26, no. 6:743–750.

van Leeuwen, M., C.B.M. te Stroet, A.P. Butler, and J.A.Tompkins. 1999. Stochastic determination of the Wierden(Netherlands) capture zones. Ground Water 37, no. 1:8–17.

Vassolo, S., W. Kinzelbach, and W. Schafer. 1998. Determi-nation of well head protection zone by stochastic inversemodeling. Journal of Hydrology 206, no. 3/4: 268–280.

Xu H., G. Li, S. Zhang, Y.H. Dong, M. Li. 2011. Comparativestudy of numerical delineation for wellhead protection areaunder steady-state and transient state. 5th InternationalConference on Bioinformatics and Biomedical Engineering(iCBBE2011), vol. 3, 2566–2569.

Yeh, T.J., and J. Simunek. 2002. Stochastic fusion of informationfor characterizing and monitoring the Vadose Zone. VadoseZone Journal 1, no. 2: 207–221.

Zhang, Y., D.A. Benson, and B. Baeumer. 2007. Predictingthe tails of breakthrough curves in regional-scale alluvialsystems. Ground Water 45, no. 4: 473–484.

Zheng, C., and P.P. Wang. 1999. MT3DMS—A Modular Three-Dimensional Multispecies Transport Model for Simulationof Advection, Dispersion, and Chemical Reactions of Con-taminants in Ground-Water Systems. Documentation andUser’s Guide. Jacksonville, Florida: U.S. Army Corps ofEngineers.

NGWA.org Y. Dong et al. GROUND WATER 5