discrete modeling & simulation - mn.uio.no · discrete modeling & simulation contents ......

p. 4 The Evolution of

Computing: AlphaGo

p. 75 The Power to

Create Chaos

p. 84 Faraday’s Tablet

cise.aip.org

www.computer.org/cise/

Vol. 18, No. 4 | July/August 2016

Discrete Modeling & Simulation

Contents | Zoom in | Zoom out Search Issue | Next PageFor navigation instructions please click here


READ YOUR FAVORITE PUBLICATIONS YOUR WAY

SEARCH, ANNOTATE, UNDERLINE, VIEW VIDEOS, CHANGE TEXT SIZE, DEFINE

Now, your IEEE Computer Society technical publications aren’t just the most informative and state-of-the-art

most exciting, interactive, and customizable to your reading preferences.

The new myCS format for all IEEE Computer Society digital publications is:

• Mobile friendly.

• Customizable.

• Adaptive.

• Personal.

Just go to www.computer.org/mycs-info

Login to mycs.computer.org

qqM

Mq

qM

MqM

THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

qqM

Mq

qM

MqM



http://mycs.computer.org

http://www.computer.org/mycs-info

http://www.computer.org/cise


http://www.qmags.com


p. 4 The Evolution of

Computing: AlphaGo

p. 75 The Power to

Create Chaos

p. 84 Faraday’s Tablet

cise.aip.org

www.computer.org/cise/

Vol. 18, No. 4 | July/August 2016

Discrete Modeling & Simulation



http://www.ieee.org/index.html

http://cise.aip.org

http://www.computer.org/mycs-info

http://www.computer.org/cise/

Kirk BornePrincipal Data

Scientist,Booz Allen Hamilton

Satyam PriyadarshyChief Data Scientist,

Halliburton

Bill FranksChief Analytics

Experience the Newest and Most Advanced Thinking in Big Data Analytics

Rock Star SpeakersBig Data: Big Hype or Big Imperative?BOTH.

Business departments know the promise of

www.computer.org/bda

03 November 2016 | Austin, TX

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



http://www.computer.org/bda





EDITORIAL BOARD MEMBERSJoan Adler, Technion-IIT, [email protected]

Francis J. Alexander, Los Alamos National Laboratory, [email protected] Beichl, Nat’l Inst. of Standards and Technology, [email protected]

Bruce Boghosian, Tufts Univ., [email protected] Bungartz, Institut für Informatik, [email protected]

Norman Chonacky, Yale Univ. (EIC Emeritus), [email protected] DiPierro, DePaul Univ., [email protected]

Jack Dongarra, Univ. of Tennessee, [email protected] Eigenmann, Purdue Univ., [email protected]

William J. Feiereisen, Intel Corporation, [email protected] Fox, Indiana Univ., [email protected]

K. Scott Hemmert, Sandia National Laboratories, [email protected] P. Landau, Univ. of Georgia, [email protected]

Konstantin Läufer, Loyola Univ. Chicago, [email protected] D. Myers, University of Michigan, [email protected]

Manish Parashar, Rutgers Univ., [email protected] Rundle, Univ. of California, Davis, [email protected]

Robin Selinger, Kent State Univ., [email protected] L. Sterling, Indiana Univ., [email protected]

John West, University of Texas, Austin, [email protected]

DEPARTMENT EDITORSBooks: Stephen P. Weppner,

Eckerd College, [email protected] Prescriptions: Ernst Mucke, Identity Solutions, ernst.mucke@gmail.

com, and Francis Sullivan, [email protected], IDA/Center for Computing Sciences

Computer Simulations: Barry I. Schneider, NIST, [email protected], andGabriel A. Wainer, Carleton University, [email protected]

Education: Rubin H. Landau, Oregon State Univ., [email protected], and Scott Lathrop, University of Illinois, [email protected]

Leadership Computing: James J. Hack, ORNL, [email protected],and Michael E. Papka, ANL, [email protected]

Novel Architectures: Volodymyr Kindratenko, University of Illinois, [email protected],

and Pedro Trancoso, Univ. of Cyprus, [email protected] Programming: Konrad Hinsen, CNRS Orléans,

[email protected] Matthew Turk, NCSA, [email protected]

Software Engineering Track: Jeffrey Carver, University of Alabama, [email protected], and Damian Rouson, Sourcery Institute,

[email protected] Last Word: Charles Day, [email protected]

Visualization Corner: Joao Comba, UFRGS, [email protected], and Daniel Weiskopf, Univ. Stuttgart, [email protected]

Your Homework Assignment: Nargess Memarsadeghi, NASA Goddard SpaceFlight Center, [email protected]

IEEE Antennas & Propagation Society Liaison:Don Wilton, Univ. of Houston, [email protected]

IEEE Signal Processing Society Liaison:Mrityunjoy Chakraborty, Indian Institute of Technology, [email protected]

CS MAGAZINE OPERATIONS COMMITTEEForrest Shull (chair), Brian Blake, Maria Ebling, Lieven Eeckhout,

Miguel Encarnacao, Nathan Ensmenger, Sumi Helal, San Murugesan, Ahmad-Reza Sadeghi, Yong Rui, Diomidis Spinellis, George K. Thiruvathukal, Mazin Yousif, Daniel Zeng

CS PUBLICATIONS BOARDDavid S. Ebert (VP for Publications), Alfredo Benso, Irena Bojanova, Greg Byrd,

Min Chen, Robert Dupuis, Niklas Elmqvist, Davide Falessi, William Ribarsky, Forrest Shull, Melanie Tory

EDITORIAL OFFICEPublications Coordinator: [email protected]

COMPUTING IN SCIENCE & ENGINEERINGc/o IEEE Computer Society10662 Los Vaqueros Circle, Los Alamitos, CA 90720 USAPhone +1 714 821 8380; Fax +1 714 821 4010Websites: www.computer.org/cise or http://cise.aip.org/

EDITOR IN CHIEFGeorge K. Thiruvathukal, Loyola Univ. Chicago, [email protected]

ASSOCIATE EDITORS IN CHIEFJeffrey Carver, University of Alabama, [email protected] X. Chen, George Mason Univ., [email protected]

Judith Bayard Cushing, The Evergreen State College, [email protected] Gottlieb, Indiana Univ., [email protected]

Douglass E. Post, Carnegie Mellon Univ., [email protected]

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_________

___________

____________

______________________

________

____________________

___________________________

__________________________

_________________________

_________________________

__________________________

_________________________

_________________________

_________________________

____________________

______________________

________

____________________

_________________________

________________

_____________________

_________________________

__________________

_________________

__________________

__

__

_______________

________

______________

___________________

__________

____________

STAFFEditorial Product Lead: Cathy Martin, [email protected] Management: Jenny StoutOperations Manager: Monette VelascoSenior Advertising Coordinator: Marian AndersonDirector of Membership: Eric BerkowitzDirector, Products & Services: Evan ButterfieldSenior Manager, Editorial Services: Robin BaldwinManager, Editorial Services: Brian BrannonSenior Business Development Manager: Sandra Brown

AMERICAN INSTITUTE OF PHYSICS STAFFMarketing Director, Magazines: Jeff Bebee, [email protected] Liaison: Charles Day, [email protected]

mailto:[email protected]














































http://cise.aip.org/













DISCRETE MODELING AND SIMULATION

8 Guest Editors’ Introduction

Georgios Ch. Sirakoulis, Jarosław Wa̧s, and Gabriel A. Wainer

Discrete Modeling and Simulation

11 Eliciting Characteristics of H5N1 in High-Risk Regions Using Phylogeography

and Phylodynamic Simulations

Neil Giridharan and Dhananjai M. Rao

To design vaccines and mitigate epidemics, evolutionary characteristics of avian influenza viruses are typically studied through longitudinal surveillance and serological assays. However, such in vivo analysis is often reactive and limited due to the complexities and costs of long-term, multinational surveillance. A novel in silico approach combines two different agent-based simulation methods to help inform vaccine design.

25 A Scalable Modeling and Simulation Environment for Chemical Gas

Emergencies

Moon Gi Seok, Tag Gon Kim, Changbeom Choi, and Daejin Park

A scalable, hybrid agent-based simulation model incorporates an interactive computational fluid dynamics gas flow model to reflect human movement under chemical gas exposure. The proposed environment provides a simulator-level interface to integrate a continuous and discrete-event simulator by resolving data/event interaction and time synchronization among heterogeneous simulation models.

34 Agent-Based Creation and Simulation of Artificial Social Networks and the

Analysis of Their Properties

Marek Zachara and Cezary Piskor-Ignatowicz

With the correct model, a wide range of social networks can be created and their properties investigated. Such networks can be used to test and validate various hypotheses. An agent-based model allows for the building of networks that have properties similar to those observed in the real world.

42 A Conceptual Modeling and Simulation Framework for System Design

Eric Coatanéa, Ric Roca, Hossein Mokhtarian, Faisal Mokammel,

and Kimmo Ikkala

The dimensional analysis conceptual modeling (DACM) framework is a conceptual modeling mechanism for lifecycle systems engineering. Originally developed for military projects, the DACM framework is now available for other applications, too. This powerful approach handles the specifying, discovering, validating, and reusing of building blocks as well as system behavior analysis in early development stages.

HIGH-PERFORMANCE COMPUTING

53 Speedup of Micromagnetic Simulations with C++ AMP on Graphics

Processing Units

Ru Zhu

STATEMENT OF PURPOSE

Computing in Science & Engineering(CiSE) aims to support and promote the

emerging discipline of computational science and engineering and to foster the use of computers and computational techniques

in scientific research and education. Every issue contains broad-interest theme articles,

departments, news reports, and editorial comment. Collateral materials such as

source code are made available electronically over the Internet. The intended audience comprises physical scientists, engineers, mathematicians, and others who would

benefit from computational methodologies. All articles and technical notes in CiSE

are peer-reviewed.

Cover illustration: Andrew Bakerwww.debutart.com/illustration/

andrew-baker

For more information on these and other computing topics, please visit the IEEE Computer Society Digital Library at www.computer.org/csdl.

July/August 2016, Vol. 18, No. 4

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_______

_______

http://www.debutart.com/illustration/andrew-baker

http://www.computer.org/csdl

http://www.sfiprogram.org





COLUMNS

4 From the Editors

Jim X. Chen

The Evolution of Computing: AlphaGo

84 The Last Word

Charles Day

Faraday’s Tablet

DEPARTMENTS

60 Software Engineering Track

Jeff Gostick, Mahmoudreza Aghighi,

James Hinebaugh, Tom Tranter,

Michael A. Hoeh, Harold Day,

Brennan Spellacy, Mostafa Sharqawy,

Aimy Bazylak, Alan Burns,

Werner Lehnert, and Andreas Putz

OpenPNM: A Pore Network Modeling Package

Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in Computing in Science & Engineering does not necessarily constitute endorsement by IEEE, the IEEE Computer Society, or the AIP. All submissions are subject to editing for style, clarity, and length. IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html. Circulation: Computing in Science & Engineering (ISSN 1521-9615) is published bimonthly by the AIP and the IEEE Computer Society. IEEE Headquarters, Three Park Ave., 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Cir., Los Alamitos, CA 90720, phone +1 714 821 8380; IEEE Computer Society Headquarters, 2001 L St., Ste. 700, Washington, D.C., 20036; AIP Circulation and Fulfillment Department, 1NO1, 2 Huntington Quadrangle, Melville, NY, 11747-4502. Subscribe to Computing in Science & Engineering by visiting www.computer.org/cise. Reuse Rights and

Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy. An accepted manuscript is a version that has been revised by the author to incorporate review suggestions, but not the published version with copy-editing, proofreading and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or [email protected]. Copyright © 2016 IEEE. All rights reserved. Abstracting and Library

Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Dr., Danvers, MA 01923. Postmaster: Send undelivered copies and address changes to Computing in Science & Engineering,445 Hoes Ln., Piscataway, NJ 08855. Periodicals postage paid at New York, NY, and at additional mailing offices. Canadian GST #125634188. Canada Post Corporation (Canadian distribution) publications mail agreement number 40013885. Return undeliverable Canadian addresses to PO Box 122, Niagara Falls, ON L2E 6S8 Canada. Printed in the USA.

75 Scientific Programming

Konrad Hinsen

The Power to Create Chaos

80 Leadership Computing

Eric Gedenk

Illuminating the Universe’s Ignition

RESOURCES

79 AIP Membership InformationInsidebackCover

IEEE Computer Society Information

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____________________________

____________

_____

http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html


http://www.ieee.org/publications_standards/publications/rights/paperversionpolicy.html







4 Computing in Science & Engineering 1521-9615/16/$33.00 © 2016 IEEE Copublished by the IEEE CS and the AIP July/August 2016

FROM THE EDITORS

by Jim X. ChenGeorge Mason University

The Evolution of Computing: AlphaGo

Google’s AlphaGo program made headlines recently when it played a game of Go with a top human player and won. We’re conditioned to like chal-lenges until we face a formidable opponent, but what happens when we face a computer?

I learned to play Go when I was in elementary school. It’s the simplest board game concept, yet it’s the most difficult to play. I quickly gave up because it can take hours to finish just one game, and it takes seemingly forever to improve your skills. It’s worth learning the game, but I wouldn’t recommend it to all youngsters.

Figure 1 shows the Go board—on the 19 × 19 array, two players (black and white) take turns placing a stone of their own color on an empty grid point. Adjacent stones of the same color along the vertical and horizontal grid lines form an area. When an area is surrounded by an opponent’s stones, it’s “alive” if it has at least one empty point inside the area; otherwise, the stones in the area are captured and removed from the board. However, if the surrounded area has only one empty point left, it’s captured because an opponent can place a stone in that point and solidly surround it. Therefore, an area is only really alive if it has two separate empty points inside—for example, as Figure 2 shows, if black

Figure 1. A Go board. On the 19 × 19 array, two players take turns placing a stone of their own color on an empty grid point.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







July/August 2016 5

places a piece at X on a border or corner, then its area is alive. If white places a piece at X, the black stones will eventually be removed. The players count points in their own live areas to decide who won or lost. That’s it—simple! But rather frustratingly, it’s difficult to see where to place a stone to gain a territorial advantage.

When I started my MS in computer science back in 1983, I looked at the game again in an artificial intelligence course. Roughly speaking, the person who starts a match has 19 × 19 possible empty points to place his or her stone, leaving 19 × 19 – 1 empty points for his or her opponent’s next move. This goes on to searching (19 × 19)! = 361! possible combinations for the best results. Compared to other games, the Go board has many more options—for example, a chess board has only (8 × 8)! = 64! possible combinations. Furthermore, a chess piece’s move is confined to its next step, as the king

Figure 3. Microprocessor data over time. The number of transistors, integer performance, CPU frequency, power consumption, and number of cores has shifted—in some cases, dramatically. Original data collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten Dotted line extrapolations by C. Moore.

107

106

105

104

103

102

101

100

1975 1980 1985 1990 1995 2000 2005 2010 2015

No. cores

Typical power (Watts)

Frequency (MHz)

Single-threadperformance(SpecINT)

Transistors (thousands)

Figure 2. A critical point. If black places a piece at X, its area is alive. If white places a piece at X, black stones will eventually be removed.

X X

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







6 July/August 2016

FROM THE EDITORS

can only move one spot away from its current position, significantly reducing its number of combinations.

So where does this leave humans versus computers? The huge number of options in Go (361!) is still beyond computing power; ordinary human players can see just a few moves ahead. In fact, let’s assume a human player can see three steps ahead—if so, we have at most 361 × 360 × 359 = 46,655,640 situations to consider, which is a manageable number for computers. Good human players can memorize successful gambits played in other games and maybe see a few moves further ahead, but machines can remember all existing games, although it takes a tremendous effort to pull that information together. AlphaGo was initially trained to mimic human play by attempting to match the moves of expert players from recorded games (its database holds 30 million moves). Once it reached a certain degree of proficiency, it was trained further by playing multiple games against other instances of itself, using reinforcement learning to improve its play. This accomplishment is all the more amazing because there’s no easy way to evaluate a position in Go—the number of stones on the board is a weak indicator of a position’s strength, and a territorial advantage is difficult to calculate. A professional human player can make relatively easier judgments compared to a machine due to instincts that algorithms can’t capture, but AlphaGo might have broken this barrier.

But kudos to AlphaGo aside, the real story here is the evolution of computing. In 1983, CPU speed or frequency was 25 MHz. Assuming 1,000 CPU cycles correspond to making a decision on a Go board, it would have taken 31 minutes to make a decision about 361 × 360 × 359 = 46,655,640 possible moves back then. Today, a single CPU’s frequency is above 4 GHz, as shown in Figure 3, making it 160 times faster on the same algorithm—in this case, taking less than 12 seconds to make that same decision.1 In Figure 3, you can see that although CPU frequency stalls at its current level, the number of cores and transistors continues to double every other year or so. If we design

Figure 4. GFLOPS in GPU versus CPU over time. Here, GPUs are measured in GFLOPS instead

of number of cores, which integrates speed instead of just the number of parallel processing

units.

5,750

Nvidia GPU single precision

Nvidia GPU double precision

Intel CPU double precision

Intel CPU single precisionTh

eore

tica

l GFL

OP

S

5,500

5,250

5,000

4,750

4,500

4,250

4,000

3,750

3,500

3,250

3,000

2,750

2,500

2,250

2,000

1,750

1,500

1,250

1,000

750

500

250

0

Apr-01 Sep-02 Jan-04 May-05 Oct-06 Feb-08 Jul-09 Nov-10 Apr-12 Aug-13 Dec-14

GeForce 780 Ti

GeForce GTX 680

GeForce GTX 580

GeForce GTX 480

GeForce GTX 280

GeForce 8800 GTX

GeForce 7800 GTXGeForce 6800 Ultra

GeForce FX 5800

Pentium 4

Woodcrest

Bloomfield

Harpertown

Westmere

Sandy BridgeIvy Bridge

Tesla K40Tesla K20X

Tesla M2090

Tesla C2050Tesla C1060

GeForce GTX TITAN

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







July/August 2016 7

better computing algorithms, we could double the computing speed at its current level every other year or so.

In computing, a better measurement of computing power and performance is FLOPS (floating-point operations per second). Most microprocessors today can carry out 4 FLOPS per clock cycle, thus a single-core 2.5-GHz processor has a theoretical performance of 10 billion FLOPS, or 10 GFLOPS. Today, the most stunning recent hardware advancement is the GPU, which is akin to having thousands of tiny CPU cores for parallel processing. Figure 4 shows the evolution of the GPU compared to the CPU. Here, GPUs are measured in GFLOPS instead of number of cores, which integrates speed instead of just the number of parallel processing units (for example, the Nvidia GeForce GTX TITAN Z has 5,760 cores at 8,122 GFLOPS). If Go moves could be decided in parallel, we could end up with 0.01 second for making a move. Visualize that for a moment: from 31 minutes to 0.01 second in 30 some years.

A lphaGo is probably most powerful at its database, which is the weakest aspect of human capacity com-

pared to computers, not in its evaluation. It’ll be interest-ing to see what would happen if we removed the database. Regardless, with its advancement in computing and stor-age power, AlphaGo beats the most diligent and deeply intelligent human brains. The ramifications on the next evolutionary computing step remain to be seen.

Reference

1. W. Harrod, “A Journey to Exascale Computing,” Proc. High Performance Computing, Networking, Storage and Analysis, 2012, pp. 1702–1730.

Jim X. Chen is a professor of computer science at George Ma-son University. He’s also an associate editor in chief of CiSE.Chen’s research is in graphics, virtual reality, and medical im-aging; he has a PhD in computer science from the University of Central Florida. Contact him at [email protected].

Erratum

In “Lesbian, Gay, Bisexual, Transgender, and Queer Students’ Sense of Belonging in Com-

puting: An Intersectional Approach,” which appeared in Computing in Science & Engineer-ing, vol. 18, no. 3, 2016, pp. 24; doi: 10.1109/MCSE.2016.45, the originally published

version of the article contains an error in the author bio section. The correct text should read

“Heather Wright is a research associate for the Computing Research Association’s Center for

Evaluating the Research Pipeline (CERP). Her research interests are centered on her passion

for promoting social justice among all groups of individuals—especially those who are under-

represented in computer science and other closely related fields. Wright received a BS in

sociology from Radford University, with minors in technical & business writing and women’s

studies. Contact her at [email protected].”

Computing in Science & Engineering regrets this error.

DEADLINE FOR 2017 AWARD NOMINATIONS

DUE: 15 OCTOBER 2016

In 1982, on the occasion of its thirtieth anniversary, the IEEE Computer Society established the Computer Entrepreneur Award to recognize and honor the technical managers and entrepreneurial leaders who are responsible for the growth of some segment of the

must have taken place over

generally and openly visible.

All members of the profession are invited to nominate a colleague who they consider most eligible to be considered for this award. Awarded to individuals whose entrepreneurial leadership is responsible for the growth of some segment of the computer industry.

COMPUTER ENTREPRENEUR AWARD

AWARD SITE: https://www.computer.org/web/awards/entrepreneur

www.computer.org/awards

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



__________

____________________________

__________



https://www.computer.org/web/awards/entrepreneur

http://www.computer.org/awards






GUEST EDITORS’ INTRODUCTION

Discrete Modeling and Simulation

Georgios Ch. Sirakoulis | Democritus University of ThraceJarosław Wa̧s | AGH University of Science and Technology Gabriel A. Wainer | Carleton University

During the past few years, the modeling community has given much attention to advancing the performance of its models and simulation tools, aiming to help tackle modeling difficulties and resulting computational burden in almost every field of application. Nevertheless, it has become apparent that, in several physical systems, the solution to the underlying modeling problems lies

in the nature of the systems themselves. Thus, advancing the modeling community’s abilities through com-putational tools that can capture a system’s essential features—where global behavior arises from the col-lective effect of simple or discrete components that interact locally—is a rather essential and timely task.

Problems to Solve

Although several mathematical tools and advanced solutions do exist and support the tasks of modeling and simulation in many different aspects, computer scientists and their colleagues are still interested in alternatives that can couple the principles of smaller computational burden and complexity with much wider applicability. Consequently, discrete models with nontrivial behavior have successfully proven to be an appealing computational tool for researchers who want to generate and manage complexity using very simple rules of dynamical transitions.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







www.computer.org/cise 9

This special issue deals with the application of discrete modeling and simulation tools to problems from different fields, including physics, engineer-ing, environment science, social science, and life sciences. Many of the authors highlighted here examine the computing abilities and principles of discrete models with a focus on their expressive dy-namics, their emergent computation, and their in-herent parallelism, making them suitable for high-performance computing. Such models can also successfully tackle the computational bottleneck in terms of the complexity inherent in a variety of numerical simulations.

In This Issue

We received a wide variety of interesting articles for this special issue that represent the unique aspects and novelties of discrete modeling and simulation tools and their applications in several interesting computational science domains.

In the first article, “Eliciting Characteristics of H5N1 in High-Risk Regions Using Phylo-geography and Phylodynamic Simulations,” Neil Giridharan and Dhananjai Rao propose a com-putational approach that couples different agent-based modeling methods to simulate ecological characteristics of avian influenza virus (AIV) epidemics, via the development of prophylactic strategies and vaccines. The introduced agent-based simulation methods include epidemiologi-cal simulation and phylogeographic annotations to identify high-risk countries, and phylodynamic simulations to elicit AIVs’ evolutionary character-istics in the aforementioned high-risk countries. The resulting analysis can be used to guide time frames for redesigning vaccines for different geo-graphic regions, prevent and contain outbreaks, guide and focus sampling and surveillance efforts of migratory waterfowl in areas with high anti-genic drift, and potentially guide surveillance ef-forts to assess risk of novel strains emerging at the human-animal interface.

In the second article, entitled, “A Scal-able Modeling and Simulation Environment for Chemical Gas Emergencies,” Moon Gi Seok and his colleagues propose a scalable and hybrid agent-based simulation (ABS) model that incorporates an interactive computational fluid dynamics gas flow model for designing countermeasures in the event of an uncontrolled chemical gas emergency. The proposed simulation environment’s require-ments are twofold: resolve hybrid simulation be-tween discrete-event and continuous simulations,

and support the combination of various simula-tors in various domains. The proposed ABS mod-el’s scalability is provided via PDEVS, a parallel discrete-event system specification model of hu-man behavior. The authors implemented a distrib-uted, real-time simulation of a chemical gas emer-gency for parallel execution of a GPU-based game engine and constructive simulators by utilizing multiple GPUs.

In the third article, “Agent-Based Creation and Simulation of Artificial Social Networks and the Analysis of Their Properties,” Marek Zachara and Cezary Piskor-Ignatowicz propose a lively method for artificially creating a social network, utilizing simulation and a multi-agent environ-ment that enables the design and development of networks with properties similar to those ob-served in the real world. In such a manner, sta-tistical parameters—the likelihood of the rela-tionship distribution to a power-law distribution and the scaling factor of the relationship distri-bution—have been selected to enhance the resul-tant network’s resemblance to natural networks. To achieve a network with real-life properties, the agents need not only to have an opportunity to meet each other and form relations, but they must also be allowed to voluntarily choose the other agents they meet, based on shared interests. The number of these voluntary meetings seems to be one of the primary factors influencing the resul-tant networks’ parameters.

In the last article, “A Conceptual Modeling and Simulation Framework for System Design,” Eric Coatanéa and his colleagues present the di-mensional analysis conceptual modeling (DACM) framework, which intends to offer a solid scientific approach to modeling and simulating early design problems. Although initially developed for military use, DACM has been used in several engineering domains, from redesigning a torpedo system to modeling a laser system and its interaction with metal powders. Concepts on how DACM can be enhanced and utilized as a specification tool are becoming apparent through a set of design struc-ture and design mapping matrices, forming a “fin-gerprint” of the design problem and the conceptual design solutions.

Clearly, the computational scientists who must pro-ceed with modeling and simulations must over-

come the constraints caused by their selected method-ologies and tools. In some cases, they might even have

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








GUEST EDITORS’ INTRODUCTION

10 July/August 2016

Selected articles and columns from IEEE Computer Society publications are also available for free at

http://ComputingNow.computer.org.

to develop their own tools and methods. Th is special issue should help CiSE readers understand the poten-tial of discrete modeling and simulation and to design and develop their own models and tools.

Georgios Ch. Sirakoulis is an associate professor in the Department of Electrical and Computer Engineering at Democritus University of Th race; he’s also visiting professor in University of West England. His research interest include cellular automata theory and applica-tions, modeling and simulation, emergent circuits and systems, complex systems, bioinspired computation/biocomputation, modern electronic models, and green and unconventional computing. Sirakoulis received a PhD in electrical and computer engineering from Dem-ocritus University of Th race. Contact him at [email protected].

Jarosław Wa̧s is an assistant professor in the Department of Applied Computer Science at AGH University of Sci-ence and Technology. His research areas include agent-based modeling, cellular automata, numerical meth-ods, multi-agent systems, complex systems, granular

fl ow, and crowd dynamics. Wąs holds a PhD in com-puter science and habilitation from AGH University of Science and Technology. Contact him at [email protected].

Gabriel A. Wainer is a professor in the Department of Systems and Computer Engineering at Carleton Uni-versity, Canada. His research interests include discrete-event simulation, cellular models, parallel and distribut-ed simulation, and real-time simulation. He is a Fellow of the Society of Modeling and Simulation International (SCS). Wainer received a PhD in software engineering from Université D’Aix-Marseille III. Contact him at [email protected].

Stay relevant with the IEEE Computer Society

More at www.computer.org/webinars

What’s the Buzz?

covered in an IEEE Computer Society

more by industry’s leading experts:

Hidden “Gotcha’s” Most IT Experts Miss in Hybrid Cloud Achieving Application Delivery Velocity with a 482% ROIPreparing for the Next Generation of Enterprise Applications

Driven DevelopmentFour Strategies for Responsive Design

Learn something new. Check out a Computer Society webinar today!

What’s the Buzz?

covered in an IEEE Computer Society

more by industry’s leading experts:

Hidden “Gotcha’s” Most IT Experts Miss in Hybrid CloudAchieving Application DeliveryVelocity with a 482% ROIPreparing for the Next Generation of Enterprise Applications

Driven DevelopmentFour Strategies for ResponsiveDesign

Learn something new. Check out aComputer Society webinar today!

Keeping YOU at the

Center of TechnologyIEEE Computer Society

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



______

_____

______________

_______

____

http://ComputingNow.computer.org




http://www.computer.org/webinars





July/August 2016 Copublished by the IEEE CS and the AIP 1521-9615/16/$33.00 © 2016 IEEE Computing in Science & Engineering 11

DISCRETE SIMULATION AND MODELING

Eliciting Characteristics of H5N1 in

High-Risk Regions Using Phylogeography

and Phylodynamic Simulations

Neil Giridharan | William Mason High School, Mason, Ohio

Dhananjai M. Rao | Miami University, Oxford, Ohio

To design vaccines and mitigate epidemics, evolutionary characteristics of avian influenza viruses are typically studied through longitudinal surveillance and serological assays. However, such in vivo analysis is often reactive and limited due to the complexities and costs of long-term, multinational surveillance. A novel in silico approach combines two different agent-based simulation methods to help inform vaccine design.

Avian influenza is caused by several serotypes of the influenza A viruses (including the dominant H5N1 strain) that are endemic in migratory waterfowl, the natural intercontinental vectors of these viruses.1 The avian influenza virus (AIV) transmits to poultry via contaminated water and feed, causing widespread mortality and resulting in severe economic losses, including the 2015

US epidemic. AIVs cause recurring epidemics because they undergo continuous change in the haemmag-glutinin (HA) surface protein. Changes to HA cause antigenic drift, which enables new strains to escape host immunity, ultimately causing new infections. Infected hosts shed viruses with changes to HA, giving rise to more diverse strains. As Figure 1 illustrates, the cycle continues with the establishment of new viral lineages; strain prevalence varies both temporally and spatially throughout the world.

Vaccination is the most prevalent prophylactic method for containing avian influenza epidemics.1 Vac-cines are designed from several ancestors of the prevalent viral strains circulating in the host population.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








12 July/August 2016

These prevalent strains are identified in vivo through surveillance, sampling, sequencing, and phylogenet-ic analysis. As of December 2014, the World Health Organization (WHO) recommends 27 different H5N1 vaccine candidate strains for different parts of the world to immunize against prevalent strains from 14 different clades in the WHO H5N1 no-menclature phylogram.1

However, continuous antigenic drift in viruses requires candidate strains and vaccines to be up-dated every six to eight months,2 posing significant national and international challenges to surveil-lance, sequencing, and analysis efforts. Moreover, the in vivo vaccine candidate strain selection method is reactive and typically lags an outbreak by 10 to 18 months, degrading vaccine efficacy and impeding effective containment of emergent epi-demics. Furthermore, in vivo sampling and analy-sis doesn’t provide a comprehensive understanding of the ecological factors influencing evolutionary processes and epidemic progressions, thereby hin-dering design and administration of prophylaxis or containment strategies.

These aforementioned shortcomings of the traditional in vivo approach can be alleviated by enhancing them using in silico or computational methods, thereby informing vaccine design for rap-idly evolving lineages to improve vaccine efficacy, using antigenic drifts and environmental factors to determine frequency of vaccine redesign in different geographic regions, identifying high-risk regions for antigenic shift and increased surveillance, and identifying influential ecological and epidemio-logical factors to mitigate their impacts and contain epidemics. Specifically, our investigation proposes a novel multidisciplinary approach that combines two different agent-based simulation methods,

namely, epidemiological simulation and phylogeo-graphic annotations to identify high-risk countries, and phylodynamic simulations to elicit AIVs evolu-tionary characteristics in high-risk countries.

Background

Computational epidemiology integrates computer science and classical surveillance-based epidemiol-ogy to enable comprehensive understanding of dis-eases, epidemic forecasting, and administration of prophylactic strategies. It’s fundamentally based on modeling and simulation (M&S) with agent-based, discrete modeling methods being widely used. An agent models epidemic progression in a single in-dividual or a collection of individuals along with pertinent ecological processes and interactions. The de facto standard for modeling epidemics is the compartmental model, in which the population is subdivided into independent subsets or compart-ments based on their epidemiological states, such as susceptible (S), exposed (E), infective (I), and recov-ered (R), or SEIR. Disease progression is modeled via probabilistic transitions between the compartments.

SEARUMS, the M&S environment used in this study, utilizes agent-based modeling and par-allel discrete event simulation (PDES) to enable rapid epidemiological analysis.3,4 The agents and migratory flyways are generated by using surveil-lance data available in GIS format. The agents in the SEARUMS model the migratory life cycle of a flock of waterfowl of the same species and imple-ment the SEIR compartmental model to character-ize epidemic progression. Agents interact with each other when flocks logically overlap to propagate epidemics during and after migration. Aggregation of many birds into a single agent has been adopted to reduce computational resources and simulation runtime, which can be significant for larger mod-els even when PDES techniques are utilized.4 The waterfowl model used in this study has been gen-erated, verified, and validated as discussed in our earlier publications3,5 and supplements. This inves-tigation utilizes a validated model from our earlier investigations5 to identify highly connected coun-tries involved in the spread of H5N1 and uses phy-lodynamics to elicit evolutionary characteristics of H5N1 viruses in these high-risk countries.

Phylodynamics is a special case of computa-tional epidemiology that combines phylogenetics with epidemiological modeling and simulation to simultaneously analyze the phenotypic and geno-typic evolution (see Figure 2).6 Specifically, agents model both epidemic progression as well as abstract

Figure 1. Overview of the genetic and antigenic diversity in avian influenza

viruses (AIVs) that arise due to repeated epidemics. Prevalent strains

are identified in vivo through surveillance, sampling, sequencing, and

phylogenetic analysis.

Infected hosts shed viruses (untilhost gains immunity to currentstrain) with genetic changes

Antigenically different virusescause new infection in hosts

Viruses

Example phylogram with 3clades/lineages created from

in vivo sampling and sequencing

Avianhosts

Viru

ses

wit

h su

ffici

ent

gene

tic

chan

ges

beco

me

anti

geni

cally

dif

fere

nt a

ndev

ade

host

’s im

mun

ity

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








changes occurring in viral genotypes, which are sampled during simulation to construct a phyloge-netic tree or phylogram (see Figure 2). Ecological parameters in the simulation are set such that the in vivo and in silico phylograms are similar, thereby validating the phylodynamic model. The calibrated model provides information about unobservable or unknown in vivo ecological parameters. The mod-el is also used to analyze the effect of influencing ecological parameters via vaccination or other pro-phylactic strategies via varying parameter settings. Extending the simulation time to logically simulate into the future provides forecasts about anticipated evolutionary changes to assist planning.

As summarized by Erik Volz and colleagues,6

Bryan Grenfell and coauthors6 postulated the the-ory and general approach for phylodynamic model-ing, whereas our research focuses on its application to AIVs. Neil Ferguson and colleagues6 as well as Katia Koelle and coauthors6 discuss the use of phy-lodynamic simulations and sensitivity analysis to identify the role of ecological factors in the spread of influenza A infections, particularly in humans. Trevor Bedford and colleagues7 use a phylodynamic model of influenza A evolution to show that few an-tigenic dimensions are sufficient to account for the paradoxically limited diversity of H3N2 human influenza strains, whereas Benjamin Roche and coauthors8 discuss extensive validation of their individual-based phylodynamic model. In a recent article,9 they present the use of phylodynamic anal-yses to assess the impact of antigenic, epidemiologi-cal, and ecological factors to explain higher genetic diversity and weaker immune escape in avian influ-enza viruses when compared to human strains.

Unlike earlier investigations6–9 that focus on or involve human influenza strains, our research focuses on avian influenza, specifically H5N1. This investigation distinguishes itself from prior work, including our own,3,5 by combining phylodynam-ics with results from phylogeography. This research utilizes a validated model of migratory waterfowl to identify key countries with many direct infection pathways for H5N1 to other countries. Because highly connected countries influence global diversi-ty of viruses, they’re deemed high-risk regions; this study uses phylodynamic simulation to quantify the ecological characteristics underlying the evolution of H5N1 strains in these high-risk countries. The phylodynamic modeling and simulation extends validated models proposed by Bedford and col-leagues7 by explicitly modeling multiple high-risk waterfowl species native to each country. The results

from phylogeographic and phylodynamic analyses are then used to infer epidemiological, ecological, and temporogeospatial characteristics to inform vaccine design and prophylaxis in each country.

Methodology

The proposed method for identifying high risk countries and eliciting H5N1 evolutionary char-acteristics in them consists of two distinct phases (see Figure 3).

Phase 1: Identifying High-Risk Countries

The high-risk countries used in this study have been identified via epidemiological simulations of migratory waterfowl conducted via SEARUMS,3

an ecological and epidemiological modeling and analysis system developed in Java.5 SEARUMS in-cludes tools for generating epidemiological models of migratory birds from GIS data obtained from the Global Register of Migratory Species (GROMS) database.10,11 The process of generating the model from GROMS GIS data is discussed in our earlier publication.11 Figure 4a illustrates the generated model for one waterfowl species, but SEARUMS has been used to generate a model that includes all 22 high-risk waterfowl species involved in the global dispersion of H5N1 as shown in Figure 4b.12 Each agent in the model, shown as a circle in Figure 4, represents a flock of collocated birds that migrate as a unit. Lifecycle activities, migratory behaviors, in-teragent interactions, and epidemic progressions are accomplished via the exchange of discrete events.

Generation of infection graph. The epidemic spread is simulated for a period of five years by introducing an initial infection seeded in Guangdong, China, cor-responding to the H5N1 nomenclature phylogram.1

Figure 2. Overview of using phylodynamic simulation to infer ecological

properties. agents model both epidemic progression as well as abstract

changes occurring in viral genotypes, which are sampled during simulation to

construct a phylogenetic tree or phylogram.

In silico In vivo

Simulation parameters are tunedsuch that in vivo and in silicophylograms are biologically

similar. The simulation parametersare used to infer the unobservable

ecological properties.

Simulation parametersettings

Unknown ecologicalparameters

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









14 July/August 2016

SEARUMS tracks and annotates infections occurring between pairs of agents during simulations to gener-ate an infection graph at the end of the simulation.

The resulting directed acyclic graph (DAG) shows lo-cations where the flocks originate as nodes with edges, indicating direct infection between pairs of locations. The stochastic nature of simulations requires 25 runs; edges that occur in the majority of simulations are retained as the dominant infection pathways. Each simulation run requires about 5.5 hours when using eight threads on an Intel Core i7-3770K CPU at 3.5 GHz and about 4 Gbytes of RAM.

Phylogeographic annotation of infection graph. The phylogeographic method for validation discussed in our earlier publication5 has been adapted to annotate edges in the infection graph using clades constituting the complete H5N1 nomenclature phylogram from World Health Organization (WHO)/World Orga-nization for Animal Health (OIE)/Food and Agri-culture Organization of the United Nations (FAO).1

Each clade contains in vivo viral samples with less than 1.5 percent difference in nucleotides from vari-ous countries. An edge is assigned to a clade if the clade contains strains from the two countries con-nected by an edge. The annotation is based on the inference that genetically similar viruses from two different countries require a direct vector and pathway to enable their dispersion. For example, the infection pathway between 30°52'12''N, 28°22'14.5194''Eand 36°41'4.9194''N, 36°41'4.9194''E is geocoded to Matruh, Egypt, and Mugla, Turkey, respec-tively, with annotation to clade #2.2.1, which con-tains H5N1 HA sequences from Egypt and Turkey with a less than 1.5 percent difference: A/duck/Egypt/08355S-NLQP/2008 and A/chicken/Turkey/Ipsala563/2008. Figure 4c shows the re-sulting annotated infection graph.

Identification of high-risk countries. Phylogeographi-cally annotated edges are strong evidence supporting infection pathways and are used to assess influence of countries. The diagram in Figure 5 summarizes the number of annotated edges (from Figure 4c) between pairs of countries. The countries involved

Figure 3. Overview of the key steps in the proposed methodology that combines epidemiological, phylogeographic, and phylodynamic

analysis (high-resolution image in supplementary materials at http://pc2lab.cec.miamioh.edu/documents/cise16_suppl.pdf).

Validated model ofhigh-risk waterfowl

generated usingGROMS database

SearumsInfection

graph

Aggregate resultsfrom manysimulations

EPIFLU

Viral strains

Phylogeographicannotaion

High-riskcountries

WHO/FAO/OIEPROCEDURE

In vivo phylogram

PER-COUNTRYPHYLOGRAMS

Ana

lysi

s of

each

hig

h-ris

kco

untr

y

Phylogramcomparator

In silico phylogram

Calibration

In silico

Per-countryparameters

Phylodynamicsimulation

Results &analyses

Phase 2: phylodynamic simulation & analysisPhase 1: epidemic simulation & phylogeographic analysis

Figure 4. Epidemiological model of migratory waterfowl and resulting

infection pathways: (a) single waterfowl species, (b) model with 22 high-risk

species, and (c) annotated infection graph. (See http://pc2lab.cec.miamioh.

edu/documents/cise16_suppl.pdf for videos, high-resolution images, and

additional images of full phylogenetic trees.)

(a)

(b)

(c)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



___________________________________

___________________

http://pc2lab.cec.miamioh.edu/documents/cise16_suppl.pdf

http://pc2lab.cec.miamioh.edu/documents/cise16_suppl.pdf






in the highest number of intercountry infections are deemed “high risk” because outbreaks can re-sult in maximum dispersion of novel viral strains. The high-risk countries are CHN (China, 14), TUR (Turkey, 7), VNM (Vietnam, 6), RUS (Russia, 6), and NGA (Nigeria, 4), where the value in parenthe-ses is the number of other countries to which viruses are dispersed. Most of these countries had two or just one other directly connected country. China and Russia weren’t used for further analysis due to lack of sufficient in vivo samples and low geospatial resolution of annotation when compared to the size and geographic diversity reported elsewhere. Conse-quently, the phylodynamic analysis to elicit H5N1 characteristics are focused on the other three coun-tries, namely, Turkey, Vietnam, and Nigeria.

Phase 2: Phylodynamic Simulation

The second phase commences with construction of the reference in vivo phylograms shown in Figure 6 for Turkey, Vietnam, and Nigeria. The in vivo phylo-grams are used to compare with the in silico phylo-grams and validate the phylodynamic simulation and the parameters settings. The phylogenetic trees for the three countries were generated using the same procedure used by WHO/OIE/FAO.1 Full-length HA segments (more than 1,600 nucleotides) were obtained from the GISAID EpiFlu database13 and were collected between 2005 and 2009. The HA se-quences were aligned using MUSCLE14 and the phy-logenetic trees were constructed with PAUP*15 using neighbor-joining and the same standard GTR Imodel as WHO/OIE/FAO.1,5 The newick form of the phylogram generated by PAUP* was used to cat-egorize leaves into clades such that percentage pair-wise nucleotide distances between and within clades are more than 1.5 percent and less than 1.5 percent, respectively, concordant with WHO/OIE/FAO clade definition criteria.1 The clades in the trees are shown in different colors (colors aren’t significant) in Figure 6, with Turkey, Vietnam, and Nigeria having 2, 26, and 4 clades, respectively—that is, Turkey has the lowest genetic diversity of H5N1 strains while Vietnam has the highest diversity among the three countries. The depth of the branches between each pair of sequences is proportional to the number of nucleotide differences between them.

Phylodynamic model and assumptions. The phylody-namic model and simulator used in this study have been developed in Java by enhancing the Antigen7

simulator. Antigen has been developed for phy-lodynamic analysis of human influenza, with the

following features added to extend it for avian in-fluenza analysis: simulation of multiple species with different birth and death rates; births occurring only during specific brooding seasons rather than throughout the year; genetic and antigenic prop-erties of viruses modeled independently; antigenic distances between simulated HA strains estimated by using a cross-immunity approach; phylogenetic trees constructed based on genetic differences rath-er than difference in emergence times; and infec-tion rates and infective periods accounting for sea-sonal variations in the countries.

Unlike SEARUMS, the phylodynamic simu-lator uses an individual-based approach. The epidemiological properties of each individual is characterized using the standard SIS (susceptible

infected susceptible) compartmental model summarized in Figure 7. The SIS model is used due to the endemic nature of H5N1 in waterfowl, resulting in very low mortality rates. The model doesn’t include human interactions because infec-tions in humans are very sporadic, and human-hu-man transmission is unsustained.5,9 Consequently, humans don’t play a role in antigenic diversity in viruses9 and aren’t included in our model.

Hosts in each species are added during their re-spective brooding season to model births (at rate μb)and removed throughout the year to model deaths (at rate μd). Average lifespan of different species (see column LS in Table 1) has been used to deter-mine birth and death rates so as to maintain bird

Figure 5. Infection pathways between pairs of countries.

High-risk countries are in red.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









16 July/August 2016

populations at the end of each year. Infection spreads from an infective host to a susceptible host based on the contact rate and seasonal sinusoidal modulation with amplitude Ω as shown in Figure 7. Seasonal modulation9 has been used to account for influence of temperature on viral durability on

surfaces where birds come in contact with each other.

The viral phenotypes are modeled using ab-stract 2D vectors similar to the approach used by Bedford and colleagues.7 Mutations in the viruses shed by hosts are uniformly distributed through-out the strain at the rate of . Mutations essentially change the coordinates of the viruses in the ab-stract 2D space. This enables Euclidean distances to be mapped to antigenic change using the meth-od proposed by Julia Gog and colleagues, which is used by other investigators as well.7,9 The antigenic parameters proposed by Roche and colleagues9 for H5N1 have been used in the simulations so that an antigenically different1 strain that isn’t present in the host’s immune history causes new infections. Host immune history drives the viral selection pres-sure, causing specific serotypes to establish in the waterfowl population. The viruses are periodically sampled during simulation (similar to the in vivo sampling process) to yield the in silico phylogenetic tree at the end of the simulation. The phylogenetic

Figure 6. Reference in vivo phylogenetic trees generated using the procedure recommended by the WHO/OIE/FAO H5N1 evolution group.1 The

clades in the trees are shown in different colors (colors aren’t significant), with (a) Turkey, (b) Vietnam, and (c) Nigeria having 2, 26, and 4 clades,

respectively—that is, Turkey has the lowest genetic diversity of H5N1 strains while Vietnam has the highest diversity among the three countries.

(a) (b) (c)

Figure 7. Overview of conceptual SIS (susceptible

infected susceptible) model. The model doesn’t

include human interactions because infections

in humans are very sporadic, and human-human

transmission is unsustained.

S I

Infection from antigenicallydifferent strain

μbbirth (during brooding)

Birth

Death

ν(recovered from infection)

μb

μd

Deathμd

β (1 + Ω.2πt )

Mutationin strains

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








tree construction uses genetic distances between viruses along with common ancestor information to produce the phylogram. The simulation uses Gillespie’s stochastic simulation algorithm (SSA) along with Tau-Leap optimization and a time step of 0.1 days8 to ensure accurate simulations.

The design of the phylodynamic model in-volves the following domain-specific assumptions and limitations:

■ It doesn’t explicitly model the influence of sea-sonal migration on contact rates ( ) and uses as an aggregate parameter.

■ Birth (μb) and death (μd) are set such that the overall population of each species (and conse-quently their relative fractions; see Table 1) is maintained in each year of simulation. How-ever, over 15-year periods, the relative fraction of bird species can vary, which isn’t modeled.

■ The role of temperature and environmental up-take is only implicitly modeled as modulation on contact rate ( ) and therefore their influ-ence can’t be separately assessed.

Phylodynamic model validation. We used the same validation procedure proposed by Roche and col-leagues8 to validate our multispecies phylodynamic model by comparing the results against the exten-sively validated phylodynamic model from Bed-ford and colleagues7 (referred to as the “reference model”). For validation, we created a multispecies model, but with all the species having identical pa-rameter settings to compare against the original

single species model. The chart in Figure 8a shows a comparison of the 95 percent confidence interval (CI) from 250 replications of the stochastic simula-tion over a period of the years. The red line in the chart shows the percentage difference between the average number of infective individuals. Figure 8b compares antigenic diversity in the reference model versus the multispecies models. Antigenic diversity directly determines the number of clades in the resulting phylogram. Note that the 95 percent CI of the reference and multispecies models are very close and overlap each other in Figure 8.

The box plots in Figure 9 show the results from statistical comparison of key epidemiological and antigenic attributes, namely, peak infection, peak infection day, and antigenic diversity. The p-values from Kolmogorov-Smirnov (KS) two-sample tests conducted on the reference and multispecies model were greater than 0.05, establishing that the vali-dated reference and proposed multispecies models are statistically indistinguishable. Similar com-parisons were conducted for different settings of parameters to ensure that the epidemiological and antigenic results from the reference and proposed multispecies models were statistically the same, thereby establishing the validity of salient aspects of the multispecies model.

Next, the influence of varying brooding peri-ods and seasonal modulation was verified using a metamorphic validation approach.16 Specifically, the brooding period for one species was modified to be 10 days earlier when compared to the reference model. Results from 250 stochastic simulations

Table 1. Population distribution (N = 20,000 for all three countries) and brooding period of high-risk

waterfowl species identified using GIS data from the GROMS database.

Waterfowl species Life span (years) Brood time

Population fraction

Vietnam Turkey Nigeria

Eurasian wigeon (A. Penelope) 2.02 Feb.–Apr. 0.75 0.74 0.15

Common teal (A. Crecca) 2.5 Dec.–Feb. 0.02 0.02

Northern pintail (A. Acuta) 3 Feb.–July 0.05 0.04 0.06

Tufted duck (A. Fuligula) 3.5 Feb.–Apr. 0.02 0.02

Northern lapwing (V. Vanellus) 3.5 Apr.–July 0.15

Black godwit (L. Limosa) 18 Feb.–Apr. 0.15 0 0.13

Ruff (P. Pugnax) 4.4 Mar.–Jun. 0.62

Common gull (L. Canus) 10 Mar.–Apr. 0.05 0.03

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









18 July/August 2016

were used to verify that the peak infection day correctly changed on an average by 5.9 ± 4.41 days as per expectation. Similarly, the influence of 0.1 sinusoidal modulation on contact rates was veri-fied to introduce the expected 10 percent sinusoi-dal change in infection patterns as shown in Fig-ure 10, thereby establishing the proposed model’s validity.

Per-country models. The three independent phylo-dynamic models for Turkey, Vietnam, and Nige-ria involve different subsets of high-risk waterfowl species found in those countries (see Table 1). The waterfowl species in the different countries and their relative populations were extracted from the GROMS GIS dataset.10 The brooding seasons and average life spans of the waterfowl have been ob-tained from various data sources aggregated in the animal diversity database maintained by the Uni-versity of Michigan.17 The average infective dura-

tion, that is, the time during which an infected bird can spread infection to other birds, is five days.9

The ecological parameters in the per-country phylodynamic model whose values aren’t known and can vary between countries include

■ Contact rate , the average number of birds that can get infected each day by another infective host, has been estimated at 2.847 contacts/day18

on average in small lakes. This value is used as the reference to determine contact rate via calibration.

■ Mutation rate is the average number of mu-tations per day in the HA segment of viruses shed by infective hosts. The diversity in ecosys-tems, climatic conditions, and bird species give raise to different mutation rates for the viruses in different countries.

Several investigations19 have analyzed the evo-lutionary rate of the HA segment of H5N1 viruses and have consistently reported the following per-site/year ( ) mutation rates: Turkey (17.8 to 2.06), Nigeria (1.62 to 4.05), and Vietnam (15.81 to 18.65). The per-site/year values are converted to ,the daily mutation rate using the relationship = 1,800 ÷ 365, where 1,800 is the average number of nucleotides in HA.

The actual values for and for each coun-try are determined via calibration, which requires a systematic search of the parameter space based on their estimated values. Note that additional combi-nations of and (beyond estimated ranges) could yield in silico phylograms with the same number of clades and properties as in vivo phylograms. How-ever, such combinations of and that are beyond the estimated range of values can’t be substantiated with observed data and aren’t biologically meaning-ful. The seasonal temperature in Turkey and Viet-nam, for example, swings considerably (from 16 °C to 29 °C), which influences environmental viral du-rability and consequently the spread of epidemics. Accordingly, for these two countries, the contact rate is scaled by a seasonal sinusoidal modulation of 1 + 0.1 cos(2πt ÷ 365) (see Figure 7), where simu-lation time t is day of year (0 < t < 365); the value of Ω = 0.1 is proposed by Roche and colleagues,9 but sinusoidal modulation isn’t used for Nigeria as the temperature is very uniform throughout the year (with less than a 2 °C change).

Model calibration. Calibration is an empirical pro-cess through which selected parameters in the mod-el are fine-tuned such that the model characterizes

Figure 8. Comparison of infective population and antigenic diversity

in the reference model7 versus multispecies models from 250

simulations: (a) infective population and (b) antigenic diversity. The

red line in the chart shows the percentage difference between the

average number of infective individuals.

0 200 400 600 800 1,000 1,200 1,400

Infe

cted

pop

ulat

ion

Logical simulation time (days)

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

–1

–0.5

0

0.5

1

The plots for the reference and multispecies model are similar and overlap each other

% Δ mean

Multispecies

Reference

(a)

0 200 400 600 800 1,000 1,200 1,400

Infe

cted

pop

ulat

ion


0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

5.0 4.5

–1

–0.5

0

0.5

1

% Δ mean

Multispecies

Reference

(b)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








observed data with sufficient accuracy. Calibrat-ing the two parameters, namely, contact rate and mutation rate , for a given country is ac-complished by comparing the in silico phylogram generated via simulation to the corresponding in vivo phylogram (see Figure 6). The range of values for the two parameters has been explored using the following initial estimates: 2.847 contacts/day for and different ranges of mutation rates listed earlier.18

The in silico phylograms don’t have taxa (iden-tifiers such as A/duck/Egypt) similar to the in vivo phylogram. Consequently, comparison of the two phylograms is accomplished by using the fol-lowing standard phylogram topology (or shape) metrics20 in decreasing order of importance: num-ber of clades (primary determinant of antigenic diversity), interclade distance (estimates evolution-ary distance between clades), average number of child nodes (leaf nodes have zero children, which reflects speciation), and average depth (number of intermediate nodes to the root, which reflects establishment/extinction rates of strains). Note that both in vivo and in silico clades contain se-quences that have a less than 1.5 percent differ-ence between each other. Consequently, intraclade distances aren’t a useful distinguishing factor and aren’t included in the set of metrics. The ETE tool-kit21 has been used to develop Python scripts to analyze the phylograms and generate comparative metrics. Because the parameter space is reasonably constrained, calibration has been performed by exhaustively searching the solution space in small increments (to avoid potential pitfalls with heuris-tic searches).

The simulations for calibration and analysis were conducted for a period of 18 years, with the first 15 serving as “burn-in” time. The burn-in period accounts for the difference between the putative root of the WHO/OIE/FAO reference tree (A/turkey/England/5092/1991) versus the actual viral isolates in the nomenclature phyl-ogram that start from 2006 for the three coun-tries. The in silico viral sampling commences only after the burn-in period and continues until end of simulation. The simulations for Turkey, Vietnam, and Nigeria were conducted for three, four, and three years, respectively, correspond-ing to the range of years used in the WHO/OIE/FAO reference phylogram. Each simulation re-quires 4 Gbytes of RAM and approximately 522 ± 42 seconds of runtime on an Intel Xeon X5550 CPU at 2.67 GHz.

Results and Discussions

The phylogeographic and phylodynamic analysis methods we just discussed were utilized to analyze the evolution of H5N1 in three high-risk countries, namely, Turkey, Vietnam, and Nigeria. The charts in Figure 11 show success rates, that is, the fraction of stochastic simulations that yield the same number

Figure 9. Statistical comparison of key epidemiological and

antigenic characteristics of the reference model (green) and

the multispecies model (orange): (a) peak infection and (b)

antigenic diversity.

Multispecies (M-S) modelReference model (Ref)

Year 1

D = 0.0837,p-value = 0.3434

2,000

2,500

3,000

3,500

Ref M-S

Year 2

D = 0.0478,p-value = 0.9365

2,500

3,000

3,500

4,000

Ref M-S

Year 3

D = 0.0876,p-value = 0.2899

2,500

3,000

3,500

4,000

4,500

Ref M-S

(a)

Year 1D = 0.0717,

p-value = 0.5387

Ref M-S0.5

1.0

1.5

Year 2D = 0.0478,

p-value = 0.9365

Ref M-S2.0

2.5

3.0

Year 3D = 0.0558,

p-value = 0.8298

Ref M-S3.0

3.5

4.0

4.5

(b)

Figure 10. Comparison between reference and multispecies models with

0.1 sinusoidal modulation (Ω = 0.1) on contact rate ( ).

0 200 400 600 800 1,000 1,200 1,400

Infe

cted

pop

ulat

ion


Infected population comparisonreference vs. 10% modulation

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

Zoom-in of 95% CI

-75

0

75

0 400 800 1,200

% Δ mean

Multispecies

Reference %Δ

Mea

n

–1

–0.5

0

0.5

1

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









20 July/August 2016

of clades (the primary metric) between in silico and in vivo phylograms. The values plotted are from 50 stochastic runs for each pair of parameter settings, for a total of 7,500 simulation replications per coun-try. Table 2 tabulates the other metrics correspond-ing to parameter values for contact rate and mu-tation rate that yield the same number of clades. The entries in the table have been ordered with the best configuration for each country being listed first.

The data in Figure 11a shows that different pairs of values for and can explain the diver-sity of viruses in Turkey and Nigeria after 15 years of burn-in time. However, the secondary metric

in Table 2, namely, interclade distance, provides a strong arbitration to identify the best ecologi-cal settings for these two countries. Note that, for Vietnam, there is a much higher viral diversity, so only one set of parameter settings repeatably yields the observed viral diversity.

Analysis of Overfitting

To estimate issues with overfitting, we looked at the impact of changing calibrated parameter settings by 10 percent on the mean number of clades. Statistical analysis using F-test (for variance) and T-test (for means) used results from 20 independent simulations for each parameter settings. In all cases, the variance was statistically similar, with F-tests yielding p-value 0.05 0.05. The results from the T-tests also showed that in almost all cases, a ± 10 percent change in parameters didn’t result in a sta-tistically significant (p-value 0.05 > 0.05) change in the mean number of clades from the simulation. The overall statistically significant consistency in mean and variance suggests that the model and param-eter settings haven’t been overfitted to the observed number of clades for Turkey, Nigeria, and Vietnam.

Assessment of Critical Parameters

We used the GSA method22 to rank and identify critical parameters in the model. The GSA method utilizes success/failure rates from simulations con-ducted using a range of values for a given param-eter to yield a dm,n statistic. This statistic is sensitive to differences in both central tendency and any dif-ference in the distribution functions. The magni-tude of the dm,n statistic indicates that parameter’s importance in a model (larger values indicate that the parameter has a higher influence).

Table 3 shows the dm,n statistic measured by varying the parameter by ± 50 percent (in steps of 10 percent) around its calibrated settings and from 20 simulation runs for each setting. The param-eters in Table 3 are ordered from most influential to least. Overall, contact rate is the most critical parameter that influences genetic diversity as it plays a critical role in the spread of epidemics. However, the range for is small (from 2.0 to 3.0 in Table 2, a 50 percent change), consistent with the expectation that behaviors and interactions of wild waterfowl species don’t significantly vary between the coun-tries. The next influential parameter is mutation rate

, which introduces antigenic variations enabling viral lineages to become endemic in the population. Collectively, as indicated by Table 3, lower contact and mutation rates reduce the evolutionary rate and

Figure 11. Comparison of success rates (fraction of in silico phylograms that yield same number of clades as in vivo phylograms) for different parameter settings for and : (a) Turkey, (b) Vietnam, and (c) Nigeria. The

highest success rates are highlighted in green.

1.5

2.0

2.5

3.0

3.5

0.0088

0.0090.0092

0.0094

0.0096

0.0098

0.010.0102

Con

tact

rat

e (β

)

Mutation rate (ψ )

0.0

0.2

0.4

0.6

0.8

1.0

(a)

Mutation rate (ψ )(b)

(c)

1.5

2.0

2.5

3.0

3.5

0.0780.08

0.0820.084

0.0860.088

0.090.092

0.0

0.2

0.4

0.6

0.8

1.0

Con

tact

rat

e (β

)

1.5

2.0

2.5

3.0

3.5

0.0080.01

0.0120.014

0.0160.018

0.020.022

0.0

0.2

0.4

0.6

0.8

1.0

Mutation rate (ψ )

Con

tact

rat

e (β

)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








genetic diversity of viruses, resulting in fewer distinct clades in the phylogram.

The diversity in bird species, which influ-ences brooding season, birth (μb), and death (μd), ranks third, indicating it’s a key parameter. In other words, diversity of bird species and differ-ences in their life cycle is an important ecological characteristics to be modeled. Seasonal modula-tion plays some role in explaining the antigenic diversity and can’t be completely ignored. In con-trast, the net number of individuals in the model (N ) and the number of initially infected birds (Io) don’t have significant influence on the results from simulations.

Phylogram Comparisons

Figure 12 shows an in silico phylogenetic tree for the three countries using the best parameter set-tings tabulated in Table 2. Comparing to the in vivo phylograms in Figure 6, the in silico phylo-grams also exhibit similar and consistent structure with the same number of clades. As summarized by the column titled “No. child nodes” in Table 2, the degree of nodes in the in vivo and in silico phylograms were almost identical, establishing that the evolutionary characteristics are consistently modeled.

The depth of nodes is a measure of long-term lineage establishment and overall genetic diver-sity among the strains in each country. Figure 13 shows a comparison of the fraction of nodes at dif-ferent depths, with nodes closer to the root having a lower depth. The column titled “Node depth” in Table 2 summarizes the average node depths. The data in Figure 13 shows that the evolutionary shape of the in vivo and in silico phylograms are similar. Combined with the intercluster distances, consistency in node depths establishes that the two

phylograms show similar evolutionary characteris-tics. Collectively, the phylodynamic analysis high-lights that the in vivo and in silico phylograms are quantitatively, structurally, and visually accurate, thereby establishing validity of the calibrated pa-rameter settings.

Inferences

The parameter settings used to generate the in sili-co phylogram now provide additional information about the evolution of H5N1 in the three countries, information that can’t be directly obtained from in vivo data. The charts in Figure 11 along with data from Tables 2 and 3 show that even modest chang-es in raw contact rates can have noticeable impacts on H5N1 evolution and immune escape. Reduc-tion of contact rates is typically accomplished by culling infected birds, particularly livestock. The inferences from the proposed method support the current practice of large-scale culling, similar to

Table 2. Metrics used for comparing in vivo and in silico phylograms for contract rate ( ) and mutation

rate ( ) values that yield the same number of clades.

Country Interclade distances No. child nodes (%) Node depth (%)

Turkey 2.1 0.0093 20 1.92 10

2.0 0.0091 21 0.31 16

2.0 0.0096 23 0.30 16

Vietnam 3.0 0.083 32 0.18 7

Nigeria 2.5 0.013 10 0.18 21

2.2 0.014 16 0.23 22

Table 3. Ranking of parameters from generalized

sensitivity analysis (GSA).22

Parameter

Score (dm,n statistic)

Turkey Nigeria Vietnam

Contact rate ( ) 0.526 0.447 0.216

Mutation rate ( ) 0.446 0.385 0.354

Bird species 0.191 0.138 0.343

Seasonal modulation (Ω)

0.174 N/A 0.123

Population (N) 0.156 0.133 0.094

Initial infection (Io) 0.109 0.091 0.0878

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









22 July/August 2016

the containment efforts pursued during the 2015 H5N1 outbreak in the American Midwest. Fur-thermore, the data also supports CDC/FAO efforts

to encourage isolation of water and feed of domes-tic birds, which further reduces contact rates.

Influencing the mutation rate ( ) of viruses, particularly in livestock, is indirectly accom-plished through vaccination. Vaccinated birds are immune to viruses in several clades, thereby pre-venting epidemics and consequently improving further diversification in those lineages. The charts in Figure 11 along with data in Table 3 suggest that extensive vaccination can decrease viral diver-sification rates in the different countries. However, unlike contact rates, a more extensive vaccination effort is necessary to influence viral diversifica-tion rates. The data also suggests that economic constrains, particularly in poorer countries, drive culling as primary containment strategy over vac-cination. The mutation rates for H5N1 in Vietnam (0.083) is significantly higher when compared to Turkey (0.0093) and Nigeria (0.013). The data suggests that vi-ral strains are experiencing a more rapid evolution and have the potential to give rise to novel strains through reassortments in Viet-nam. Consequently, Vietnam requires greater em-phasis on surveillance and containment efforts to mitigate emerging epidemics.

The implications of studying AIVs extends to human and swine influenza viruses because

studies have established that all currently known influenza A viruses originated in aquatic birds. Although this study focused on three high-risk countries, the methods and analysis can be read-ily extended to all 12 distinct global geographi-cal regions included in the WHO vaccine design and recommendation process.2 The analysis can be used to guide time frames for redesigning vaccines for different geographic regions, thereby extend-ing applicability of vaccines without compromis-ing their efficacy. Similarly, the results can also be used to inform prophylaxis of poultry and livestock to prevent and contain emergent outbreaks from novel viral strains such as H5N3. Furthermore, the models can be used to guide and focus sampling and surveillance efforts of migratory waterfowl in areas with high antigenic drift. Moreover, the as-sumptions in the model, particularly the influence of migration and impact of environmental uptake, can be relaxed to provide a more comprehensive model. The proposed methodology can guide surveillance efforts to assess risk of novel strains emerging through reassortments at the human-animal interface.

Figure 12. Sample in silico phylogenetic trees generated using the

best parameter settings shown in Table 2 identified via calibration.

The clades in the tree are shown in different colors.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








References

1. WHO/OIE/FAO H5N1 Evolution Working Group, “Continued Evolution of Highly Pathogenic Avian Influenza A (H5N1): Updated Nomenclature,” Influenza and Other Respiratory Viruses, vol. 6, no. 1, 2012, pp. 1–5.

2. “Candidate Vaccine Viruses and Potency Testing Reagents for Influenza A (H5N1),” World Health Org., 2014; www.who.int/influenza/vaccines/virus/candidates_reagents/a_h5n1.

3. D.M. Rao, A. Chernyakhovsky, and V. Rao, “Mod-eling and Analysis of Global Epidemiology of Avian Influenza,” Environmental Modelling & Software,vol. 24, no. 1, 2009, pp. 124–134.

4. D.M. Rao, “Accelerating Parallel Agent-Based Epidemiological Simulations,” Proc. ACM SIGSIM PADS Conf., 2014, pp. 127–138.

5. D.M. Rao, “Enhancing Epidemiological Analysis of Intercontinental Dispersion of H5N1 Viral Strains by Migratory Waterfowl Using Phylogeography,” BMC Proc., vol. 8, no. 6, 2014, p. S1.

6. E.M. Volz, K. Koelle, and T. Bedford, “Viral Phylo-dynamics,” PLoS Computational Biology, vol. 9, no. 3, 2013, p. e1002947.

7. T. Bedford, A. Rambaut, and M. Pascual, “Canalization of the Evolutionary Trajectory of the Human Influenza Virus,” BMC Biology, vol. 10, no. 1, 2012, pp. 1– 12.

8. B. Roche, J.M. Drake, and P. Rohani, “An Agent-Based Model to Study the Epidemiological and Evolutionary Dynamics of Influenza Viruses,” BMC Bioinformatics, vol. 12, no. 1, 2011, pp. 1–10.

9. B. Roche et al., “Adaptive Evolution and Environ-mental Durability Jointly Structure Phylodynamic Patterns in Avian Influenza Viruses,” PLoS Biology,vol. 12, no. 8, 2014, p. e1001931.

10. “Global Register of Migratory Species (GROMS): Summarising Knowledge about Migratory Species for Conservation,” 2013; www.groms.de.

11. D.M. Rao and A. Chernyakhovsky, “Automatic Gen-eration of Global Agent-Based Model of Migratory Wa-terfowl for Epidemiological Analysis,” Proc. 27th Euro-pean Simulation and Modelling Conf., 2013, pp. 21– 28.

12. W. Hagemeijer and T. Mundkur, “Migratory Fly-ways in Europe, Africa, and Asia and the Spread of HPAI H5N1,” Proc. Int’ l Scientific Conf. Avian Influenza and Wild Birds, 2006; www.fao.org/avi-anf lu/conferences/rome_avian/documents/hage-meijer-mundkur.pdf.

13. “Action Stations: The Time for Sitting on Flu Data Is Over,” Nature, vol. 441, no. 7097, 2006, p. 1028.

14. R.C. Edgar, “MUSCLE: Multiple Sequence Alignment with High Accuracy and High Throughput,” Nucleic Acids Research, vol. 32, no. 5, 2004, pp. 1792–1797.

15. D.L. Swofford, “PAUP*: Phylogenetic Analysis Using Parsimony (and Other Methods) 4.0 Beta,” 2003; http://paup.csit.fsu.edu.

16. A. Ramanathan, C.A. Steed, and L.L. Pullum, “Verification of Compartmental Epidemiological Models Using Metamorphic Testing, Model Check-ing and Visual Analytics,” Proc. ASE/IEEE Int’ l Conf. BioMedical Computing, 2012, pp. 68–73.

17. P. Myers et al., “Animal Diversity Database,” 2015; http://animaldiversity.org.

18. P. Rohani et al., “Environmental Transmission of Low Pathogenicity Avian Influenza Viruses and Its Impli-cations for Pathogen Invasion,” Proc. Nat’ l Academy of Sciences, vol. 106, no. 25, 2009, pp. 10365–10369.

19. G. Cattolia et al., “Evidence for Differing Evolution-ary Dynamics of A/H5N1 Viruses among Countries Applying or Not Applying Avian Influenza Vaccina-tion in Poultry,” Vaccine, vol. 29, no. 11, 2011, pp. 9368– 9375.

20. P. Puigbo, Y.I. Wolf, and E.V. Koonin, “Genome-Wide Comparative Analysis Of Phylogenetic Trees:

Figure 13. Comparison of node depths (other depths with less than 1

percent of nodes not shown for brevity). Combined with the intercluster

distances, consistency in node depths establishes that the two phylograms

show similar evolutionary characteristics.

0

15

30

45

60

1 2 3 4 5 6 7 8 9 10

Nod

es (

%)

Node depths

Turkey

0

15

30

45

60

1 2 3 4 5 6 7 8 9 10

Nod

es (

%)

Node depths

Nigeria

0

15

30

45

60

1 2 3 4 5 6 7 8 9 10

Nod

es (

%)

Node depths

Vietnam

Nearleaf

Nearroot

Nearleaf

Nearroot

Nearleaf

Nearroot

In silico In vivo

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_________________

______________________________

____________


http://www.who.int/influenza/vaccines/virus/candidates_reagents/a_h5n1

http://www.groms.de

http://www.fao.org/avianflu/conferences/rome_avian/documents/hagemeijer-mundkur.pdf

http://paup.csit.fsu.edu

http://animaldiversity.org






24 July/August 2016

The Prokaryotic Forest of Life,” Methods in Molecu-lar Biology, vol. 856, 2013, pp. 1792–1797.

21. J. Huerta-Cepas, J. Dopazo, and T. Gabaldn, “ETE: A Python Environment for Tree Explora-tion,” BMC Bioinformatics, vol. 11, no. 24, 2010; http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-24.

22. B. Guven and A. Howard, “Identifying the Critical Parameters of a Cyanobacterial Growth and Move-ment Model by Using Generalised Sensitivity Anal-ysis,” Ecological Modelling, vol. 207, no. 1, 2007, pp. 11–21.

Neil Giridharan is a student at William Mason High School in Mason, Ohio. His research interests include computational biology, disease forecasting, and viral phylodynamics.

Dhananjai M. Rao is an assistant professor in the Depart-ment of Computer Science and Software Engineering (CSE), Miami University, Oxford, Ohio. His research interests include computational epidemiology, disease forecasting, and parallel simulation. Rao has a PhD in computer science and computer engineering from the University of Cincinnati. Contact him at [email protected].



Take the CS Library wherever you go!

IEEE Computer Society magazines and Transactions are now available to subscribers in the portable ePub format.

Just download the articles from the IEEE Computer Society Digital Library, and you can read them on any device that supports ePub. For more information, including a list of compatible devices, visit

www.computer.org/epub

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____________________

_________

_____

http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-24



http://www.computer.org/epub







A Scalable Modeling and Simulation

Environment for Chemical Gas Emergencies

Moon Gi Seok and Tag Gon Kim | Korea Advanced Institute of Science and Technology

Changbeom Choi | Handong Global University

Daejin Park | Kyungpook National University

A scalable, hybrid agent-based simulation model incorporates an interactive computational fluid dynamics gas flow model to reflect human movement under chemical gas exposure. The proposed environment provides a simulator-level interface to integrate a continuous and discrete-event simulator by resolving data/event interaction and time synchronization among heterogeneous simulation models.

One of the more urgent problems in emergency situations is the uncontrolled release of chemical gas via accidental spillage or intentional attack. In such chemical emergencies, developing proper countermeasures is important in both managing the disaster and reducing casualties. Counter-measures can include planning evacuation processes, identifying the roles of emergency control

centers, preparing geographical features, and installing chemical gas sensors. But before determining specific countermeasures to chemical gas exposure, system analysts should

estimate the damages caused by the gas’s gradual propagation from the initial leak source and then antici-pate the intended effects of applying various countermeasures. Generally, these two steps can be handled through a unified modeling and simulation (M&S) framework. The M&S method of designing counter-measures considers human movement and chemical gas flow as atomic components modeled under emer-gency situations. M&S for the movement of human individuals is a well-known topic discussed in the agent-based M&S domain; M&S for describing gas flow dynamics is handled in the computational fluid

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








26 July/August 2016

dynamics (CFD) domain.1,2 Lately, a few studies in-tegrate human agent-based simulation (ABS) mod-els and the heterogeneous CFD model.3,4 However, there are no considerations for simulation models that can implement the simulation or interact with other simulation models, such as for an emergency assistance process or emergency evacuation plans.

Because countermeasure development is inter-disciplinary, the M&S environment should support scalability. To reflect human planning and to reuse conventional movement, we propose a scalable hy-brid human ABS model to model the patterns of individuals under continuous chemical gas exposure (see Figure 1); we also use an interactive CFD simula-tor to exchange gas data with other domain models. The proposed simulation environment’s requirements are twofold: resolve hybrid simulation between dis-crete event and continuous simulations, and support the combination of various simulators in various domains. The simulation environment provides an interface (called an adapter) for a digital event simu-lator and a continuous-time simulator to support

interoperation via a runtime infrastructure (RTI)5

based on the presimulation concept,6 letting mul-tiple discrete-event and continuous models interact with each other seamlessly. We’ve used the proposed simulation framework as a virtual training environ-ment, integrating a 3D-mapped game engine to give a realistic experience in designing countermeasures for trainees and trainers facing chemical gas exposure.

A Gas Emergency Model

for Evaluating Countermeasures

To model human agent behavior under chemical gas exposure, we needed to meet several requirements:

■ extend conventional human movement in the ABS model7–9 to reflect individual evacuation procedures,

■ assess continuous chemical gas exposure, and ■ reconfigure the human ABS model, changing

the evaluation of evacuation processes (which are affected by the agent’s decision-making procedures) and incorporating them with sur-rounding emergency assistance systems to evaluate countermeasures.

The human agent model reconfiguration in partic-ular helps analysts evaluate various types of emer-gency procedures.

To make these individual procedures composable, our proposed scalable agent model uses a multilevel behavioral model. It also includes a low-level physical chemical damage assessment model to calculate the amount of accumulated chemical compounds.

To model human behavior, we employ a paral-lel discrete-event system specification (PDEVS).10

PDEVS provides a formal specification for describ-ing a discrete-event model, with input/output events, states, and state-transition functions triggered as in-ternal events at a user-specified desired time and as external events from outside the model.

PDEVS also supports hierarchical and modu-lar modeling features that are appropriate for scal-able construction. These features enable us to build the behavioral part of a human agent component in the coupling of two PDEVS models, which are described in multiple levels. In the broad view, the analyst can easily build a group of human agents by coupling each human agent or interactive hu-man group with additional PDEVS emergency as-sistance models, as Figure 2a shows.

The emergency procedures consist of multiple subprocesses, with the transition between separated subprocesses following internal or external conditions.

Figure 1. Scalable modeling and simulation (M&S) environment for a chemical gas emergency. The hierarchical interoperation of discrete-event simulation for human movement and continuous simulation for dynamic gas flow work well together (CFD: computational fluid dynamics).

Evacuationprocedures

model

Gassensormodel

Gas

Sensor

Chemical gas emergency

InteractiveCFD

modelHumanagents

Emergencycontrol

center model

Scalable and hybrid simulation environment

Game engine

Simulation

Training andexperience

Effectiveness ofcountermeasures

Replaceable Extension

…

Participation

Emergency countermeasures

[Gas emergency system model]

Emer

genc

yco

ntro

lce

nter

Environmentalparameters

StandaloneCFD

model

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








Using the PDEVS formalism, the set of evacuation subprocesses {P1, P2, P3, …, Pn} and conditions for transition can easily map into process-execution states, state-transition functions, and time-triggered internal or external events, as Figure 2b shows. During a certain process’s execution, the high-level model can decide where to change the path or desti-nation. Events related to human movement fall into the physical movement model, such as desired veloc-ity, final position, congestion awareness, and escape point arrival.

The conventional ABS models for human movement7–9 are discrete-event or discrete-time models in the simulation perspective and they move based on each human agent’s individual deci-sions as exchange interactions (events) with neigh-bors and the environment model. These movement models can be described using state (Ssense to sense and update neighbors’ information and Srule to move based on defined rules) and state-transition functions. If the agent’s movement information (which affects neighbors’ movement or the cur-rent executed emergency process) changes, it sends movement events to neighbors and human models. During agent moves in the positioned grid space, which is shared with the CFD gas flow models, the model sends positioned grid events to analog-to-event (AE) converters in the damage assessment

model. The dynamics of damage assessment work as follows:

ddt

Acc t x y t c( ) ( , , ) ,avg= ×

where Acc(t) is the total amount of accumulated chemical gas, (x, y, t) is the gas density of the partitioned grid (x, y) at t, and cavg is the average amount of inhaled gas per second

The event-to-analog (EA) converter generates an overdose event to other PDEVS behavioral models when Acc(t) crosses the threshold of maximum inhalation. When the PDEVS models receive overdose events, other models fall into dead states if the agent hasn’t safely exited.

Fluid dynamics scientists and engineers have proposed various methods of numerical analysis and simulation techniques for gas flow models.2,11

CFD models produce high-fidelity simulation re-sults that consider multiple parameters such as wind direction, wind velocity, geometry, and vis-cosity. Among these parameters, geographical features and wind direction can affect evacuation plans, including the policies that determine a de-sired exit point or escape path.

Conventional gas flow CFD simulators work in standalone mode, with no concerns about in-teraction with other external models. To allow

Figure 2. Overall structure and detailed modeling of hybrid human agents. The multilevel model integrates human movement, evacuation procedures, and a gas damage situation through model structures and submodels of human agents (DEVS: discrete-event system specification).

Humanagent

Humanagent

Humanagent

Human crowd

DEVS evacuationprocess model

Gas damageassessment

model

DEVSmovement

model

Dead!Overdose

ExitWalk-ing

SP1

! x1

SP2

SP3SPn

…

!x2

[Emergency processes]

T

IGA

AE converter

Density (Neighbor/own) movement event

Positioned gridOverdose

! x3

Dead

?y1

Movement requirement/movement result

Ssense Saction

!Overdose

Exit

Positioned gridOverdose

? Escaped

?Escaped

! Neighbor info

IGA: inhaled gas amount

Information affecting process transition

maximum inhalation

Shared grids withgas flow model

?Positioned grid

Transitionalstate

Terminablestate

ExternaltransitionInternal

transition

Input event

Output event

!

?

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









28 July/August 2016

interoperation with other models, our proposed interactive CFD simulator is designed with the fol-lowing specifications:

<G, Y, S, X, , , Δt, ta, , >,

where G is the set of the grid vector, such as the 2D r s grid array, G = {<i, j>, 0 ≤ i < r, 0 ≤ j < s}; Y is the gas density set of probing grids; S is the set of grid states, S = Sparam (the set of physical parameter) Svar (the state of physical variables); X is the input set, X = Xprobe (the grid vector set for density probing) Xgas (the updating state set of target grids); is the global transition function, SG

Δt SG; Δt is the varying time base; ta is the time advance function for stability, SG Δt Δt;

is the input function, SG Xgas SG; and is the output function, SG Xprobe Y.

Fluid engineers developed conventional CFD simulators by considering geographical features and governing equations, as shown in Figure 3. The CFD simulator reflects geometric information, includ-ing grid types, in G and Sparam. During simulation,

a conventional CFD simulator updates the Svar it-erative using based on the defined governing equations, boundary conditions, and numerical schemes.

The extended CFD simulator supports X and Yof the exchanged data and and of the functions for handling X, Y. Using and , Algorithm 1 describes the extended CFD simulator’s overall simulation procedures.

A Scalable Simulation Environment

For simulator-level scalability, our proposed simu-lation environment uses an RTI, a middleware implementation that follows the IEEE standard in-teroperation architecture (high-level architecture; HLA)12 and enables simulators to freely join and interoperate.

Figure 4 illustrates the proposed simulation environment’s overall architecture. The PDEVS simulator computes the PDEVS agent behavioral models and any additional event-based emergency assistance models. The continuous-time models are computed using their own simulators due to the potential differences in numerical schemes. For successful PDEVS and CFD simulator inter-operation to occur, the following issues should be resolved:

■ resolve different time resolution and simula-tion-time advancement between event-based simulators and continuous-time simulators;

■ handle high-resolution continuous data incoming from continuous simulator; and

■ perform parallel computation capabilities for overall simulation speedup.

To resolve the different time-advance mecha-nisms, we use the presimulation-based HLA in-teroperation method,6 which helps continuous simulators find the minimum next event time (MNET) between the continuous-time and dis-crete-event simulators. Using this method, we developed a simulation environment to provide two different adapters (loadable libraries) for each discrete-event and continuous-time simulator. Af-ter loading each library, the simulators can execute presimulation-based interoperation with each other through the RTI with minimal effort. The simu-lation environment has been used in various case studies of hybrid simulation.6,13

The presimulation-based HLA interoperation technique runs under the condition that no interac-tion occurs between continuous-time simulators, but

Figure 3. Conventional CFD simulator. Fluid engineers developed conventional CFD simulators by considering geographical features and governing equations.

(based on fluiddynamic equation

and boundarycondition)

Solid

Solid

Freespace

Gas flow

Inlet Outlet

Discrete grids

Standalone CFD simulator

Numerical solver

Algorithm 1. CFD simulation.

1: initialize S, tc = 0

2: while tc< tend do

3: SG ← χ(SG, Xgas)

4: Δt ← getdt (SG , Δt)

5: SG ←Δ (SG,Δt)

6: if SG is not converged then7: Goto 38: end if9: Y ← ∧ (SG, Xprobe)

10: tc ← tc+ Δt11: end while

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








in our case study, a continuous-time human damage assessment simulator and an almost continuous-time (high-resolution discrete-time) gas sensor simulator require gas density data from the continuous-time CFD simulator. To follow the presimulation tech-nique, our proposed environment provides an extended adapter to integrate multiple coupled continuous-time simulators into a single continuous simulator through an adapter. Figure 5 shows this extension. The adapter coordinates the execution of continuous simulators and delivers continuous coupled data among the simula-tors. In other words, the adapter is extended to support the role of high-performance middleware based on in-terprocess communication.

Under the adapter’s execution coordination, each continuous simulator runs sequentially, so as not to receive piecewise linear segments of gas density data from the past. During iterative execu-tions of continuous simulators, if AE converters in the continuous models detect events before the PDEVS model’s MNET, all continuous simula-tors will be synchronized at that time and then execute the presimulation-based HLA interop-eration protocols. The proposed approach enables the exchange of event-based data abstracted and converted from high-resolution continuous data through the RTI.

The presimulation-based HLA interoperation leads to sequential execution among multiple simu-lators. Using the simulation environment to evalu-ate various gas emergency scenarios and to serve as a virtual training environment requires simulation

speedup. Even though the simulators run sequen-tially, we can achieve speedup through parallel simulation of each simulation model in both the PDEVS and continuous simulators.

Case Study

Using the proposed scalable hybrid M&S environ-ment, we evaluated the effectiveness of countermea-sures designed for specific chemical gas emergency systems by comparing simulation results in terms of survivability.

The target scenarios determine the evacua-tion procedure of pedestrians in a shopping zone after a chemical explosion. We adopted the basic concept of a general agent-based approach1 in sim-ulating human pedestrian behavior under the trig-gered circumstance and then implemented gas flow propagation using Roe’s scheme,2 as derived from a compressible gas CFD model.

The extended CFD simulator from the Roe’s scheme-based simulator11 initializes chemical gas condition parameters such as density, pressure, and stable grid states. The low-level movement models of human agents initialize agent states including initial position and walking speed. The low-level damage assessment models of human agents initialize the assumed maximum inhala-tion (0.05 g).

The first experiment is about the human agents’ evacuation to a single explicit exit point, with vari-ous levels of gas sensor performance. Each agent’s individual evacuation procedures are as follows:

Figure 4. The simulation environment’s architecture: high-level-architecture (HLA)/runtime infrastructure (RTI)-based interoperation of parallel discrete-event human agent simulator and continuous-time interactive CFD simulator.

Sensorsimulator

Replaceable

HLA-compliantgame engine

orpredeveloped

simulatorParallel DEVS simulator

Movementmodel

Human agentbehavioral model

Adapter for presimulationAdapter

Controlcenter

Evacuationprocess model

InteractiveCFD

simulator

Humandamage

assessmentsimulator Sensor

simulator

Event-driven emergencyassistance models

Continuoussimulators for

emergencyassistance

HLA / RTI

A/E convertersMultisolvercoordination

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









30 July/August 2016

■ Human agents walk around randomly while shopping.

■ If walking agents receive an alarm event near a gas sensor or a yelling event from other escap-ing human agents, then these agents run to a single explicit exit point.

As Figure 6 shows, high gas-sensing performance results in high survivability. These simulation results can be used to choose gas sensor performance requirements.

The second experiment evaluates the effectiveness of countermeasures in evacuation to a single explicit exit point with or without direction from an emer-gency control center. This control center directs the crowd to escape by sharing hidden exit point infor-mation after the alarm is activated. It then monitors crowd evacuation and gas flow. If some humans are

Figure 5. Simulator extension. The adapter is extended to support the role of high-performance middleware based on interprocess communication.

Shared memory

Continuoussimulator #1

Input extrapolation

Receive all stops Call A/E converters

Send TSync to adapter and choose MNETbetween received MNETs

EventsMNETs

A/Econverters

[Interactions][Timing information]

[Event is generatedor TSync == MNET]

TSync←TSync+δ,(TSync

≤MNET) send Go(TSync)

TGrant TReq

Tsync

Continuoussimulator #2 …

Go(TSync),inputs

Go(TSync),inputs

Stopoutputs

Stopoutputs

Tsync

Figure 6. Gas sensor performance and the implications for survivability.

Sensing gas density (kg/m2)

Simulation iterations: 20

0.01 0.05 0.1 0.5

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Sur

viva

bilit

y (%

)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








positioned in a region with chemical gas exposure, the control center orders them to escape through other exit points. The sensing density of gas sensors is fixed at 0.05 kg/m2 in this experiment. Each agent’s indi-vidual evacuation procedures are as follows:

■ Human agents walk around randomly while shopping.

■ If walking agents receive an alarm event near a gas sensor or from the control center, the agents run to an explicit exit point (E1) or a hidden exit point (E2) after calculating the dis-tance from their current position.

■ If the escaping agents receive information about a specific exit point from the control center, the agents run to that specific exit.

We assume that the effectiveness of the de-scribed countermeasures is defined as the ratio of the

number of survivors among all human agents. As Figure 7 shows, human agents in the scenario escape to two different exit positions. The waiting crowds around E1 could be exposed to incoming gas, so the control center orders them to escape to E2. After re-ceiving the order, a group of waiting agents moves to E2. The control center increases survivability signifi-cantly, from 0.55 to 0.83 total agents surviving.

To prove our framework’s scalability, we inte-grated a custom-implemented 3D-mapped game engine14 with our simulation environment via RTI. If other simulators (called constructive simulators) support real-time simulation, the implemented simu-lation environment can be used as a virtual training environment for evaluating evacuation procedures and processes under an emergency situation.

To support real-time simulation, each construc-tive simulator must provide parallel simulation ca-pability. Specifically, the CFD simulator causes a

Figure 7. Simulation experiment. Gas flow is simulated according to wind direction, and human agents, assisted by the control center, follow an evacuation process.

1

1

2

3

2

4

5

4

53

Gasdensity

Waiting agents around E1 moveto E2 to avoid coming gas afterreceiving center’s notifications

Agents wait to exit1 agent in a cell2 agents in a cell3 agents in a cell4 agents in a cell

Agents escape to well-known exit point (E1)

5 agents in a cellAlarmSolid

Evacuation with control center

Evacuation without control center

E1

E2

E1 E1

E2 E2

Emergency center notifiesagents about

hidden exit point (E2)

Wind

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









32 July/August 2016

performance bottleneck due to the large number of grids, so the simulation computation of the grids must utilize multiple cores on the GPU.15,16 The PDEVS simulator also uses a GPU-based simulation architec-ture17 to compute human behavior in parallel, using the multiple cores on the GPU to increase speedup.

Figure 8 shows the execution time reduction cre-ated by GPU-based parallel execution of sequential procedures for pedestrian movement simulations in

a cell space. Each cell-space model is allocated into a single core in an array of multiprocessing units. After the cell-space simulation results are updated, the HLA-based interoperation time shares the previ-ously updated cell-space status for synchronization with the pedestrian agent simulator.

In the virtual training environment, human agents are nonplayer characters (NPCs) in the game client. The trainee can intentionally invoke a chemical explosion to transfer the exposure event to the CFD simulator, as Figure 9a shows. With the simulation of gas flow, constructive simulators model the on/off event for gas sensors and dead events for NPCs and players, and the results are then reflected in the game engine (Figure 9b).

Future work will require additional repeated sim-ulations to validate our proposed framework-

based countermeasures’ statistical effectiveness against real cases. The proposed simulator must be integrated at the simulator-kernel level with addi-tional hardware resources such as GPU-based ac-celeration to enable real-time interoperation.

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) and funded by the Ministry of Education (2014R1A6A3A04059410 and 2015R1A2A2A01007186).

Figure 8. GPU-based acceleration for real-time simulation of parallel discrete-event system specification human behavior. Each cell-space model is allocated into a single core in an array of multiprocessing units.

Sequential cell spacesimulation time

…… … …

Sequential pedestrianagent simulation time

Physical execution time(sequential simulation

from tcur to tnev)

Physical execution time(GPU-based simulation)

HLA-based interoperation timetcur : current simulation time tnev :next event time

Multiprocessor

GPU

Multiprocessor

GPU

Singleprocessor

Multiprocessor

GPU

…

Physical time of (tnev – tcur)

Reduced

Figure 9. Virtual training environment: (a) intentional chemical gas explosion and (b) simulation data reflected back in the game engine. Human agents are nonplayer characters in the game client.

Simulation environment Simulation environment

Chemical gas explosionevent

<Game engine>

Agent position, dead eventsensor on/off event

<Game engine>

Sensor on

Agent dead

User control agent

(b)(a)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








References

1. B. Zhang, W. Chan, and S.V. Ukkusuri, “On the Modelling of Transportation Evacuation: An Agent-Based Discrete-Event Hybrid-Space Approach,” J. Simulation, vol. 8, no. 4, 2014, pp. 259–270.

2. P. Roe, “Characteristic-Based Schemes for the Euler Equations,” Annual Rev. Fluid Mechanics, vol. 18, no. 1, 1986, pp. 337–365.

3. J.M. Epstein, R. Pankajakshan, and R.A. Hammond, “Combining Computational Fluid Dynamics and Agent-Based Modeling: A New Approach to Evacuation Planning,” PloS One, vol. 6, no. 5, 2011, p. e20139.

4. J. Shi, A. Ren, and C. Chen, “Agent-Based Evacuation Model of Large Public Buildings under Fire Conditions,” Automation in Construction, vol. 18, no. 3, 2009, pp. 338–347.

5. IEEE Std. 1516-2000, 1516.1-2000, 1516.2-2000 for Modeling and Simulation (M&S) High Level Architecture (HLA)-, IEEE, 2000.

6. C. Sung and T.G. Kim, “Framework for Simulation of Hybrid Systems: Interoperation of Discrete Event and Continuous Simulators Using HLA/RTI,” Proc. Principles of Advanced and Distributed Simulation,2011, pp. 1–8.

7. M. Batty, “Agent-Based Pedestrian Modelling,” Advanced Spatial Analysis: The CASA Book of GIS,vol. 81, ESRI Press, 2003, pp. 81–106.

8. A. Willis et al., “Human Movement Behaviour in Urban Spaces: Implications for the Design and Modelling of Effective Pedestrian Environments,” Environment and Planning B, vol. 31, no. 6, 2004, pp. 805–828.

9. J. Kerridge, J. Hine, and M. Wigan, “Agent-Based Modelling of Pedestrian Movements: The Questions That Need to Be Asked and Answered,” Environment and Planning B, vol. 28, no. 3, 2001, pp. 327–342.

10. B.P. Zeigler, T.G. Kim, and H. Praehofer, Theory of Modeling and Simulation, Academic Press, 2000.

11. J. Blazek, Computational Fluid Dynamics: Principles and Applications, Elsevier, 2005.

12. IEEE Std. 1516-2010 for Modeling and Simulation (M&S) High Level Architecture (HLA), Framework and Rules, (revision of IEEE Std. 1516-2000), IEEE, 2010.

13. M.G. Seok et al., “Framework for Simulation of the Verilog/Spice Mixed Model: Interoperation of Verilog and Spice Simulators Using HLA/RTI for Model Reusability,” Proc. Int’ l Conf. Very Large Scale Integration, 2014, pp. 1–6.

14. C. Choi et al., “Military Serious Game Federation Development and Execution Process Based on Interoperation between Game Application and Constructive Simulators,” Int’ l J. Simulation and Process Modelling, vol. 10, no. 2, 2015, pp. 103–116.

15. M. de la Asunción et al., “An MPI-CUDA Implementation of an Improved Roe Method for Two-Layer Shallow Water Systems,” J. Parallel and Distributed Computing, vol. 72, no. 9, 2012, pp. 1065–1072.

16. A.S. Antoniou et al., “Acceleration of a Finite-Difference WENO Scheme for Large-Scale Simulations on Many-Core Architectures,” Proc. 48th AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition, 2010, pp. 4–7.

17. M.G. Seok and T.G. Kim, “Parallel Discrete Event Simulation for DEVS Cellular Models Using a GPU,” Proc. Symp. High Performance Computing,2012, article 11.

Moon Gi Seok is working toward a PhD at the Korea Advanced Institute of Science and Technology. His re-search interests include HLA/RTI-based heterogeneous simulation interoperation, mixed-signal system valida-tion, and high-performance simulation methodology. Contact him at [email protected].

Tag Gon Kim is a professor in the Department of Elec-trical Engineering and Computer Science at the Korea Advanced Institute of Science and Technology. His research interests include methodological aspects of systems modeling simulation analysis of computer and communication networks, and development of simula-tion environments. Kim received a PhD in computer engineering with a specialization in systems modeling simulation from the University of Arizona. Contact him at [email protected].

Changbeom Choi is an assistant professor in the School of Creative Convergence Education at Handong Global University. His research interests include discrete-event systems modeling and simulation, verification, validation, and accreditation (VV&A), defense modeling and simu-lation, and agent-based simulation. Choi received a PhD in electrical engineering with a specialization in systems modeling and simulation from the Korea Advanced Insti-tute of Science and Technology. Contact him at [email protected].

Daejin Park (corresponding author) is a professor in the School of Electronics Engineering at Kyungpook National University. His research interests include large-scale discrete-event IoT device platforms, hard-ware-software hybrid simulation framework, and simu-lation acceleration for IoT sensing applications. Park received a PhD in electrical engineering from the Korea Advanced Institute of Science and Technology. Contact him at [email protected].

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_____________

________

_____

__________

____________












Agent-Based Creation and Simulation of

Artificial Social Networks and the Analysis

of Their Properties

Marek Zachara and Cezary Piskor-Ignatowicz | AGH University of Science and Technology

With the correct model, a wide range of social networks can be created and their properties investigated. Such networks can be used to test and validate various hypotheses. An agent-based model allows for the building of networks that have properties similar to those observed in the real world.

Social networks are complex systems consisting primarily of people, organizations, and groups and the relations among them.1 These relations can be expressed in many ways, such as commu-nication, cooperation, and dependency, and they can change in time and span from short, casual meetings to lifelong bonds. For several years, researchers have analyzed real social networks,

trying to build models that allow for the creation of artificial networks that resemble real ones. Artificial networks offer several advantages, letting us simulate specific processes such as the spread of infections2

or the reach of marketing efforts.3 However, an artificial network’s properties can have a profound impact on such simulations—for example, using random Erdős-Rényi networks4 to model an infection leads to the conclusion that such an infection will die off if the percentage of infected individuals is below a cer-tain threshold, whereas similar simulations performed on a scale-free network show that the infection can persist and spread even with very few infected nodes.5

To build a network with the desired properties, we first need to identify them. One of the most impor-tant features of real-world social networks is the heavy-tailed distribution of multiple relationships among peers.6 The specific distribution type hasn’t yet been reliably determined, but two of them are consid-ered to be the most common: power law7 and log-normal.8 An ideal scale-free network has a power-law

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








distribution, but it’s difficult to distinguish between both distributions in real networks.9

Another important parameter of social net-works is the clustering coefficient. This coeffi-cient, originating from graph theory, is used in sociology to evaluate group-building processes and group bonds. It has gradually become one of the primary parameters that describe a social net-work’s structure.10,11 For Erdős-Rényi networks,4

it can be calculated as C = (<k>)/N, where <k> is a graph’s average node degree and N is the number of its nodes.

As opposed to other work (see the sidebar), one of the key features of our model is its use of in-telligent agents to construct and evolve a network: each agent is different and has its own individual “personality,” an approach adopted from our previ-ous work,12 where it was used to simulate customer behavior. Each agent can form relations with other agents based on its own individual preferences. Yet, the resultant network demonstrates the desired properties in terms of the heavy-tailed relationship distribution and network clustering.

Our model allows for an unrestricted number of agent types, with the agents’ properties changing over time. The simulated network is dynamic, mean-ing that new connections are formed and destroyed over the course of the simulation as a result of indi-vidual agents’ actions, yet the whole network retains stable properties. The model is flexible and has several parameters that can be used to create a wide range of networks. The model’s usability is confirmed by its ability to reproduce networks with properties similar to those observed in the real world and reported in the literature.13,14

One of the primary factors used in our model is the concept of interest areas, which denote com-mon interests among agents, similar to humans’ interests.15 In recent years, the concept of trust between agents has been explored16 as a comple-mentary method of building and simulating so-cial networks.17 However, we haven’t used this concept in our model yet because trust is useful only if agents base their decisions on other agents’ actions. We do consider incorporating trust into the model in the future to analyze other effects in

Related Work in Building Artificial Social Networks

When discussing agent-based models used for the construc-

tion and simulation of social networks, it’s important to

point out that the definition of an agent as used by most authors

in the field of social networks doesn’t match the usual meaning

of an agent in the field of artificial intelligence and computer sci-

ence. The latter agent is an autonomous entity, usually gathering

and processing selected data, equipped with the internal decision

process. In the social network-related literature, an agent is most

often just a passive object, like a physical particle, with no internal

data processing.

A thorough summary of various models, their classifications,

and popularity can be found in a recent survey,1 but a short review

will also be presented in the main text. The most prominent class-

es include random Erdos-Rényi networks, together with regular

lattices based on cellular automata approach and scale-free net-

works built with a preferential attachment model.2 These models

generally use “dumb” agents, which don’t perform any processing

or decisions.

We’ve also reviewed other models3,4 that are based on local

collisions between agents in a manner similar to gas particles,

in which the network of relations is obtained. Although this ap-

proach allows for relatively large-scale simulations, it also uses the

“dumb” model of agents. Finally, some models are based on intel-

ligent/autonomous agents with local memory and decisiveness,5,6

but they have some noticeable drawbacks—the simulated popula-

tions are quite small, and there’s no information as to whether the

models can produce a network with a sufficiently similar heavy-

tailed distribution of the relations. They’re also based on rather

arbitrary assumptions.

References

1. F. Amblard et al., “Which Models Are Used in Social Simulation

to Generate Social Networks?: A Review of 17 Years of Publi-

cations in JASSS,” Proc. Winter Simulation Conf., 2015, pp.

4021–4032.

2. A.-L. Barabasi and R. Albert, “Emergence of Scaling in Random

Networks,” Science, vol. 286, no. 5439, 1999, pp. 509–512.

3. M.C. González, P.G. Lind, and H.J. Herrmann, “System of Mobile

Agents to Model Social Networks,” Physical Rev. Letters, vol.

96, no. 8, 2006; http://journals.aps.org/prl/abstract/10.1103/

PhysRevLett.96.088702.

4. R. De Caux et al., “Dynamic, Small-World Social Network Genera-

tion through Local Agent Interactions,” Complexity, vol. 19, no. 6,

2014, pp. 44–53.

5. E. zu Erbach-Schoenberg, S. Bullock, and S. Brailsford, “A Model of

Spatially Constrained Social Network Dynamics,” Social Science Com-puter Rev., vol. 32, no. 3, 2013, pp. 373–392.

6. E.M. Jin, M. Girvan, and M.E. Newman, “Structure of Growing

Social Networks,” Physical Rev. E, vol. 64, no. 4, 2001; http://

dx.doi.org/10.1103/PhysRevE.64.046132.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____

______________


http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.96.088702

http://dx.doi.org/10.1103/PhysRevE.64.046132






36 July/August 2016

social networks, such as the spread of information and beliefs.

Simulation and Primary Elements

To experiment with the development of a virtual social network, we built a dedicated simulation en-vironment after evaluating available solutions (like the Java Agent Development Framework, JADE) and finding them unsuitable for our needs. This environment allows for deployment of semiau-tonomous agents that can “live” and interact with other agents. This approach has already been suc-cessfully used for a number of social relation simu-lations.18,19 Our environment is highly scalable, utilizing parallel processing and distributed agents. On a 24-CPU machine, a simulation of 1,000 steps of 2,500 agents takes about 4 minutes; it takes roughly 17 minutes for 10,000 agents and 65 min-utes for 25,000 agents. Considering that the num-ber of cross-agent interactions increases nonlin-early with the number of agents, we consider these results to be quite good. Basic principles of agents, their interactions, and the building of multi-agent systems can be found in several books,20 so we’ll only briefly mention the key aspects related to our simulation here.

Each agent has properties that influence its be-havior—for example, an agent’s “personality” im-pacts the number and strength of the relations it can have and maintain with other agents. Initially, the environment is populated with the requested number of agents, each with its properties random-ized according to the requested range and distribu-tion. Then, during the simulation cycle, each agent is called on and has its routine executed. Each cycle consists of agents “meeting” other agents in the virtual environment and developing relationships with them, as described later in this section.

For convenience, the simulation parameters set before the simulation is initiated are prefixed with a capital V.

Model of Agents and Their Relations

Equation 1 defines the agent model:

A = {L, E, ∪R}, (1)

where L is agent location, ∪R is the set of relation-ships, and E is agent personality (see Equation 3). Each relation between two agents A1, A2 has a strength and an age parameters (see Equation 2):

R = {A1, A2, s, g}. (2)

Initially, there are no relationships among the agents; the relations are created and developed as described in the next section.

Each agent’s personality consists of its friendli-ness factor, which impacts its ability to create and maintain relations, and a set of interests in vari-ous possible areas. The set of interests and inter-est areas are an expansion of the initial model we presented elsewhere.12 The purpose of the interest areas is to emulate an aspect of the real-world en-vironment in which people have different interest profiles and are more likely to establish relation-ships with people whose interests match their own. As we demonstrate later, the number of interest areas has a noticeable impact on simulation results and the character of the resulting relationship dis-tribution. For the purpose of the simulations, we generate the friendliness value randomly for each agent, as well as the interests across the range of available areas:

E f w w, ,..., ,V1 in{ }= (3)

where f is the agent’s friendliness (0 to 100 per-cent), w1 is the agent’s interests in ith area, and wVin

is the number of available interest areas. The inter-ests are normalized.

Agent Behavior and the Simulation

Environment

A social network’s simulation and development are based on the concept of meetings and subsequent relations formed between them.

During each cycle, agents are involved in sev-eral voluntary and involuntary meetings. Involun-tary meetings consist of agents meeting random strangers as well as neighbors. The probabilities of these meetings and the neighborhood range are supplied as simulation parameters. Depending on the agent’s friendliness and the number of maxi-mum meetings an agent can facilitate (a simulation parameter), there’s usually a remaining number of meetings the agent can have with its “friends” who share its interests. This number is calculated as

m V f V V V V V ,f mm c ps d pn nr2π= × − × − × × × (4)

where mf is the number of “friends” met per cycle, f is the agent’s friendliness, Vmm is the maximum number of meetings, Vc is the number of agents, Vd

is the density, Vnr is the neighbourhood range, Vps is the probability of meeting a stranger and Vpn is the probability of meeting a neighbor.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








Depending on the simulation parameters, an agent will have a varying number of meetings with strangers, neighbors, and friends (see Table 1 for sample values). After establishing the number of meetings with friends that will happen each cycle, an agent builds a priority list of other agents it has relations with, based on the likelihood of interests:

ww w

w wmin( , )

max( ,..., ),l

i oi

Vi

V

11 in

in

∑==

(5)

where wl is the likelihood of two agents’ interests, wi is the agent’s interests in ith area, woi is the other agent’s interests in ith area, and Vin is the number of available interest areas.

Each agent then selects the mf number of other agents with the highest wl values and meets them dur-ing the simulation cycle. It’s also possible that a friend will bring its own friend to a meeting, providing an extra chance for the agent to meet the friend’s friend, represented by an additional simulation parameter Vpf.

If two agents meet, a relationship between them could start to develop or develop further. The proba-bility of initiating a relationship between two agents that don’t yet have one depends on their friendliness:

P(Rn) = ( f + fo)/ 2, (6)

where P(Rn) is the probability of initiating a rela-tionship, f is the agent’s friendliness, and fo is the other agent’s friendliness. If a relationship is initi-ated, it has an initial strength s equal to Vs (a simu-lation parameter) and age of 1 (see Equation 2).

Once a relationship is created, it’s subject to development. A relationship can have its strength sincreased or decreased. An increase happens when two agents meet; the amount of increase depends on

ss w

V g(100% )

,tt l

ra1

0=− ××

(7)

where st is the relationship’s strength at step t, wl is the likelihood of two agents’ interests, g is the rela-tionship’s age, and Vra is the impact of age on the relationship. A relationship’s strength increases as-ymptotically depending on the likelihood of the two agent’s interests and the relationship’s age—older re-lationships change less than newly formed ones.

However, all relationships decay over time, de-pending on the simulation parameter

st1 = st0*(1 – Vrw), (8)

where st is the relationship’s strength at step t, and Vrw is the relationship-weakening factor. Once a re-lationship’s strength reaches a predefined low value (such as 0.05), it’s considered destroyed and is re-moved from the agents’ lists.

Impact of the Model and Simulation

Parameters

The purpose of the proposed agent model and simu-lation algorithm is to emulate real-world social net-works. Such natural social networks have certain properties:

■ Estimated scaling parameter α is calculated by using a specific method9 to handle the discrete data. Specifically,

� ∑α +−

⎡

⎣⎢⎢

⎤

⎦⎥⎥=

−

min

nx

x1 ln

0.5i

i

n

1

1

.

■ xmin is the value denoting the power law’s ap-plicability to part of the distribution. Power law often applies only to values greater than some xmin for empirical data. Researchers9

have recommended using the Kolmogorov-Smirnov (KS) test21 to estimate this cutoff value.

■ The goodness-of-fit test is a standard method for assessing the hypothesis that the data ob-served follows a certain distribution. When calculating the distance between the assessed data and synthetically generated distribu-tions, a p value can be calculated to quantify

Table 1. Initial parameters and the results of the sample simulation.

Key simulation parameters Simulation results

Population 5,000 α 3.39

Simulation cycles 3,000 xmin 82

Person’s interests 15 P 0.56

Max. meetings 20 C 0.095

Relation weakening 0.02

Meeting parameters Meetings per cycle

Stranger meet chance 0.02% Strangers 1.0

Neighbor meet chance 1% Neighbors 2.79

Neighbor range 0.1 Friends 6.62

Friends’ friend meet chance 0.5% Friends’ friends 2.74

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









38 July/August 2016

the hypothesis’s plausibility. We calculated the value of p using the bootstrap method from the R package poweRlaw.22 We assume that a p > 0.10 indicates that the power law is plausible.9

■ The network average clustering coefficient is an average value of local clustering, which is the ratio of existing triangles of relationships between a specific agent and others it has rela-tionships with to all the possible triangles for the agent and its neighbors.

A Sample Scenario

Our sample scenario is based on simulation param-eter values that are reasonably near the middle of the ranges necessary to obtain results in observed real-life distributions. Figure 1 shows the results of the relation distribution. This distribution is very similar to a real-world network.13

As the figure shows, the distribution is heavy-tailed and very similar to those observed in real-world scale-free networks.9 The distribution parameters (listed under simulation results in Table 1) also con-firm that the parameters of the agents’ relationship distributions are similar to those observed in real-world networks, as α is within the range observed for such networks and the p value is very high. A p > 0.10 implies a high probability of the distribu-tion following the power law, which is typical for these networks.

Impact of Agent Personality

Two primary parameters determine an agent’s ability to create and maintain relationships: the

number of interest areas (Vin) and the maximum number of meetings an agent can service per cy-cle (Vmm). The number of interest areas determines the likelihood of two agents having matching interests (see Equation 5) and therefore subse-quently build a relationship. As Figure 2 shows, the number of interest areas (Vin) has a significant impact on both the scaling factor α and the clus-tering coefficient. With an increasing number of interest areas, it’s less likely to find agents with matching interests, therefore the network param-eters deteriorate. It’s interesting to note that the change of the scaling factor isn’t monotonic with the change of interest areas, with a local maxi-mum near the value of 10 and a local minimum near the value of 25.

The number of meetings an agent can have during a cycle primarily impacts the network’s clusterization. For values of Vmm higher than 6, the scaling factor α remains stable and the observed clustering coefficient raises monotonically from 0.01 to 0.15, with the latest value observed for the highest evaluated Vmm of 50. If the parameter’s val-ue is very low, the scaling factor reaches very high values, indicating an unnatural network structure due to the fact that agents can’t build enough rela-tionships with others.

The final parameter discussed here is the chance of meeting a friend’s friend during a meet-ing between an agent and its friend (Vpf). This pa-rameter also primarily impacts clusterization, rais-ing it monotonically to the value of 0.30 for Vpf of 0.02, the highest value we tested. Its impact on the scaling factor is similar to Vmm, that is, for very low values of Vpf, below 0.002, α rises significantly, in-dicating that it’s not possible to build a real-world-like network without second-tier relations. This also proves the validity of an approach presented elsewhere.18

Forced versus Voluntary Meetings

Interactions between agents are implemented in the form of the meetings they have with each oth-er. Several different classes of meetings can hap-pen, depending on which group the agent’s part-ner belongs to. Involuntary meetings occur when agents don’t choose to meet—for example, be-tween strangers, controlled by the parameter Vps,or between neighbors (such meetings are “forced” by the simulation environment, and their number is controlled by simulation parameters). The latter depends on two parameters, Vnr and Vpn, that con-trol the size (radius) of each agent’s neighborhood

Figure 1. Sample simulation results. The distribution of agents’ relations are very similar to a real-world network.

100

Agents’ relations

Age

nts

coun

t

10

11 22 3 4 5 10 20 30 100 200

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








and the probability of an agent meeting a neigh-bor. Voluntary meetings occur between an agent and its friends, as described in Equation 4. Because the average number of meetings is defined by the Vmm parameter, increasing the number of involun-tary meetings reduces the number of voluntary meetings an agent can have. Our simulation found that the ratio of these two types of meet-ings greatly impacts the resultant structure of the relationship network.

Increasing the number of involuntary meetings (and thus decreasing the number of meetings with friends) has a severe impact on network properties. Increasing the distant meeting chance from 0 to 0.001 increases α from approximately 3.2 to 3.7 and decreases the clustering coefficient from 0.11 to 0.05. Increasing the neighbor range Vnr from 0 to 0.25 increases α from 3.2 to 3.9 and decreases the clustering coefficient from 0.16 to 0.05. Be-cause we haven’t yet implemented anything that would emulate the interests related to local com-munities, from an agent’s point of view, there’s no major distinction between a total stranger and an unknown neighbor.

Time-Related Parameters and Simulation

Stability

So far, we’ve discussed various factors related to building relationships between agents. In real life, however, some processes (such as boredom) are responsible for both building and destroying

relationships. Only when these forces are at equi-librium can a stable system be achieved.

To facilitate the erosion of relationships (which can eventually lead to removal of a relation), we introduce two mechanisms: decay and age. Each relationship is subject to decay/weakening, as de-scribed in Equation 8, and to impact by age, as de-scribed in Equation 7. The longer a relation lasts, the smaller the increase in its strength from meet-ings. Our experiments show that both of these mechanisms decrease a network’s clustering with the increased value of the appropriate parameter, as can be expected. The α value has a minimum for relationship age impact (between 1.0 and 2.0) and rises outside this range. Increasing Vrw results in a slight decrease of α.

Without any external stimuli, we expected that a simulation will develop for several cycles and then achieve some stability, probably with some fluctuation of the parameters around the mean values. This expectation was confirmed in our simulation. After an initial few hundred cycles, all the observed parameters stabilized (clustering coefficient, average number of meetings of various types, and so on), as Figure 3 shows. This stabiliza-tion was also seen with the scaling factor α, but the experiments showed that the methods commonly employed for its calculation might not have been as robust as expected.

By employing extensive simulations and evalu-ating a large number of scenarios, we’ve also come

Figure 2. Number of agents’ interest classes and their impact on simulation results. As expected, (a) the clustering coefficient drops with increasing number of interests areas available to the agents. On the other hand, (b) the scaling factor achieves a desired low value for Vin near 25.

1210

8Sc

ale

para

met

er6

4

(b)10 20 30

Interests40 50

(a)

10

0.15

0.10

Clu

ster

ing

coef

ficie

nt0.

05

20 30Interests

40 50

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









40 July/August 2016

to the conclusion that the currently used meth-od for estimating α and the cutoff value xmin in Equation 9 might need to be adjusted.

Our experiments demonstrated that the value of α is extremely sensitive to the chosen xmin value. The quality of matching for the observed distribu-tion against synthetically generated datasets, de-noted by the p value, is very sensitive to the xmin as well. On the other hand, the KS test used to iden-tify xmin is very sensitive to even small variations in the values distribution.

Figure 4 illustrates the value of α calculated every 100 simulation steps for a 15,000-step simu-lation. It includes the values calculated using the current state-of-the-art method (black circles) and values calculated using xmin averaged over a range of datasets. As the figure shows, the latter method provides more coherent results, with lower varia-tion and fewer rejections of the distribution hy-pothesis (p < 0.10).

To assess the created network, especially its re-semblance to a natural (real-life) network, we

used certain statistical parameters. The only notice-able difference between our network and real-life networks is the clustering coefficient. We assume that a slightly larger value of this coefficient ob-served in the real-life network is due to other forms of activities, including group meetings in real life, whereas the model presented here essentially relies on one-to-one meetings. This is something that re-quires further investigation.

The experiments showed that virtually all the model elements were necessary—the network couldn’t achieve the expected properties with the investigated parameters set to zero. One of our key findings was that to achieve a network with real-life properties, the agents need to have not only an opportunity to meet each other and form relationships, but they must also be allowed to voluntarily choose the other agents they meet, based on their shared interests. The number of these voluntary meetings seems to be one of the primary factors influencing the resultant net-work’s parameters.

There certainly seem to be several possible extensions and opportunities to fine-tune the model. Some specialized uses could require the addition of new elements. It seems, however, that the current model is a satisfactory basis for spe-cific applications and future research and could provide useful insights into the creation of social networks.

References

1. P. Holme and J. Saramaki, “Temporal Net-works,” Physics Reports, vol. 519, no. 3, 2012, pp. 97–125.

2. R. Pastor-Satorras et al., “Epidemic Processes in Complex Networks,” Rev. Modern Physics, vol. 87, no. 3, 2015, pp. 925–979.

3. L. Weng, F. Menczer, and Y.-Y. Ahn, “Virality Predic-tion and Community Structure in Social Networks,”

Figure 3. Change in simulation parameters over time. The clustering coefficient is presented against the right scale (clustering coefficient values only).

1211109876543210

50 100 500Cycle

1000 3000

Clustering coefficient

0.10

0.09

0.08

0.07

0.06

0.05

0.04

0.03

Relations per agent Friends met Neighbours met Friends’ friends met

Figure 4. Scaling factor α, calculated during the course of a simulation, with xmin estimated separately for each snapshots of data (black circles) and xmin averaged over a number of steps (white circles).

3.8

3.7

3.6

3.5

3.4

Scal

ing

fact

or

3.3

3.2

3.1

3.0

2.9

2.8

2.7

2.60 2,000 4,000 6,000 8,000

Simulation cycle10,000 12,000 14,000

0.0 0.4 0.6 0.8

P-value0.2

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








Scientific Reports, vol. 3, no. 2522, 2013; www.nature.com/articles/srep02522.

4. P. Erdős and A. Rényi, “On Random Graphs I,” Publicationes Mathematicae Debrecen, vol. 6, 1959, pp. 290–297.

5. A. Vespignani, “Modelling Dynamical Processes in Complex Socio-technical Systems,” Nature Physics,vol. 8, no. 1, 2012, pp. 32–39.

6. A. Vázquez et al., “Modeling Bursts and Heavy Tails in Human Dynamics,” Physical Rev. E, vol. 73, no. 3, 2006; http://journals.aps.org/pre/abstract/10.1103/PhysRevE.73.036127.

7. L. Muchnik et al., “Origins of Power-Law De-gree Distribution in the Heterogeneity of Human Activity in Social Networks,” Scientific Reports,vol. 3, May 2013; www.nature.com/articles/srep01783.

8. N.Z. Gong et al., “Evolution of Social-Attribute Networks: Measurements, Modeling, and Impli-cations Using Google+,” Proc. ACM Conf. Internet Measurement, 2012, pp. 131–144.

9. A. Clauset, C.R. Shalizi, and M.E.J. Newman, “Power-Law Distributions in Empirical Data,” SIAM Rev., vol. 51, no. 4, 2009, pp. 661–703.

10. R. Albert and A.-L. Barabasi, “Statistical Mechanics of Complex Networks,” Rev. Modern Physics, vol. 74, no. 1, 2002, pp. 47–97.

11. M.E. Newman, “The Structure and Function of Complex Networks,” SIAM Rev., vol. 45, no. 2, 2003, pp. 167–256.

12. M. Zachara, E. Majchrzyk-Zachara, and C. Piskor-Ignatowicz, “Simulation of Customers Behaviour as a Method for Explaining Certain Market Char-acteristic,” Trans. Computational Collective Intelli-gence, vol. 9, 2013, pp. 217–236.

13. M.E.J. Newman, S. Forrest, and J. Balthrop, “Email Networks and the Spread of Computer Viruses,” Physical Rev. E, vol. 66, no. 3, 2002; http://journa ls.aps.org/pre/abstract/10.1103/PhysRevE.66.035101.

14. F. Liljeros et al., “The Web of Human Sexual Contacts,” Nature, vol. 411, no. 6840, 2001, pp. 907–908.

15. Z.-D. Zhao et al., “Emergence of Scaling in Human-Interest Dynamics,” CoRR, vol. abs/1307.7796, 2013; http://dblp.unitrier.de/db/journals/corr/corr1307.html#ZhaoYZZHL13.

16. L. Mui, M. Mohtashemi, and A. Halberstadt, “A Computational Model of Trust and Reputation,” Proc. Annual Hawaii Int’ l Conf. System Sciences,2002, pp. 2431–2439.

17. W. Sherchan, S. Nepal, and C. Paris, “A Survey of Trust in Social Networks,” ACM Computing

Surveys, vol. 45, no. 4, 2013; http://doi.acm.org/10.1145/2501654.2501661.

18. E.M. Jin, M. Girvan, and M.E. Newman, “Struc-ture of Growing Social Networks,” Physical Rev. E, vol. 64, no. 4, 2001; http://journals.aps.org/pre/abstract/10.1103/PhysRevE.64.046132.

19. J. Wąs and R. Lubaś, “Towards Realistic and Effective Agent-Based Models of Crowd Dy-namics,” Neurocomputing, vol. 146, 2014, pp. 199–209.

20. H. Lin, Architectural Design of Multi-agent Sys-tems: Technologies and Techniques, IGI Global, 2007.

21. W.H. Press et al., Numerical Recipes in C: The Art of Scientific Computing, 2nd ed., Cambridge Univ. Press, 1992.

22. C.S. Gillespie, “Fitting Heavy Tailed Distribu-tions: The poweRlaw Package,” J. Statistical Software, vol. 64, no. 2, 2015; https://arxiv.org/abs/1407.3492.

Marek Zachara is an assistant professor at the AGH University of Science and Technology (UST), Kra-kow, Poland, and an independent consultant for sev-eral companies. His research and professional interests include graphs transformations and analysis, simula-tion, behavioral modeling, and applications’ security. Zachara received a PhD in computer science from AGH UST. He’s a founding member of the Polish Ar-tificial Intelligence Society. Contact him at [email protected].

Cezary Piskor-Ignatowicz is working as a research as-sistant and pursuing a PhD at the AGH University of Science and Technology. His research interests include modeling social and economic networks, information diffusion, and dynamics of change propagation in large networks. Piskor-Ignatowicz received an MS in physics from Jagiellonian University. Contact him at [email protected].



qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_________

_________

______

______

_____________

______

____________

__________

_______________________

_____________

_____


http://journals.aps.org/pre/abstract/10.1103/PhysRevE.73.036127

http://www.nature.com/articles/srep01783


http://dblp.unitrier.de/db/journals/corr/corr1307.html#ZhaoYZZHL13

http://doi.acm.org/10.1145/2501654.2501661


https://arxiv.org/abs/1407.3492




http://www.nature.com/articles/srep02522







A Conceptual Modeling and Simulation

Framework for System Design

Eric Coatanéa | Tampere University of Technology

Ric Roca | Johns Hopkins University Applied Physics Laboratory

Hossein Mokhtarian | Tampere University of Technology

Faisal Mokammel | Aalto University

Kimmo Ikkala | Tampere University of Technology

The dimensional analysis conceptual modeling (DACM) framework is a conceptual modeling mechanism for lifecycle systems engineering. Originally developed for military projects, the DACM framework is now available for other applications, too. This powerful approach handles the specify-ing, discovering, validating, and reusing of building blocks as well as system behavior analysis in early development stages.

Computer modeling and simulation (M&S) techniques and methods are present in many areas, in-cluding system engineering, acquisition, training, analysis, experimentation, planning, and testing. But for various reasons, past research and development (R&D) for M&S systems hasn’t produced broad classes of models capable of being used and reused across multiple domains. Factors include

(but are not limited to) proprietary architectures, lack of consistent and clearly defined development stan-dards, model fidelity, and scalability issues that computer science and software engineering have yet to over-come. For M&S to mature both as an industry and academic discipline, the M&S community must develop standards and frameworks in the areas of conceptual modeling, because conceptual development stages carry implications for multiple decisions that heavily impact subsequent stages of system development and perfor-mance. M&S at the conceptual level is consequently a strategic domain poorly explored at the moment.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








According to leading researchers, the advance-ment of M&S conceptual modeling best practices, grounded in engineering and scientific formalisms, could have a significant impact on future model devel-opment, system quality, and reuse costs for stakehold-ers in both industry and academia. More research is needed to address M&S at the conceptual level—the effort we describe here attempts to answer this need.

Research Method

In his topical book summarizing years of research work and discovery associated with behavioral psychology and decision making in economics, Daniel Kahne-man analyzed the reasons behind biased and irrational decisions.1 The fundamental highlights of his research are the two modes the human brain uses, System 1 (the automatic and fast-thinking mode) and System 2 (the analytical and slow mode). System 2 consumes a lot of brain power and consequently is hard to use for a long period of time. System 1 is usually more efficient in urgent and dangerous situations, when fast actions or reactions must be taken, thus it’s usually favored by humans, even when a situation’s complexity actually requires System 2. In engineering, early development phases such as requirements elicitation, initial concept developments and strategic decision making belong to the class of activities best handled by System 2.

Cause-effect analysis is the other most com-mon mechanism that humans use for action and reaction in the physical world.2 One idea developed in this article is that well-informed causal analysis can efficiently support conceptual modeling and analysis of complex systems and facilitate the use of the reflexive mode. Our dimensional analysis con-ceptual modeling (DACM) framework thus had to fulfill four important characteristics:

■ be able to favor the brain’s slow and reflective mode,

■ use the brain’s natural tendency to classify infor-mation in the form of cause-effect relationships,

■ offer mechanisms to organize and simplify the complexity in problem representation, and

■ propose a mechanism to simulate behavior us-ing qualitative information.

Dimensional Analysis Theory

Dimensional analysis (DA) offers an approach for reducing the complexity of modeling problems to the simplest form before going into more details with any type of qualitative or quantitative model-ing or simulation.3 The DA theory has been devel-oped over the years by an active research community

that includes prominent researchers in both physics and engineering.4–6 Essentially, DA aims to deduce from the dimensions of the variables (length, mass, time, and so on) used in models certain constraints on possible relationship among variables.

For example, in the most familiar dimen-sional notation, learned in high school or college physics, force is usually represented as M × L ×T –2, where M is mass, L is length, and T is time. Newton’s law, F = m × a, where F is force, m is mass, and a is acceleration, is constrained by the dimensional homogeneity principle, which can be verified by checking the dimensions on both sides of Newton’s law.

Another widely used result in DA is Vashy-Buckingham’s ∏-theorem, which was stated and proved in 1914.4 This theorem identifies the num-ber of independent dimensionless numbers that can characterize a given physical situation, offering a way to simplify problem complexity by grouping the vari-ables into dimensionless primitives. Every law that takes the form yo = f(x1, x2, x3, …, xn) can take the alternative form,

( )∏ = ∏ ∏ ∏f , ,..., ,n0 1 2 (1)

where ∏i are the dimensionless products. This al-ternative form is the final result of DA and is the consequence of the Vashy-Buckingham theorem.

A dimensionless number is a product that takes the following form:

π = ⋅ ⋅ ⋅α α αy x x x ,k i j l mij il ml (2)

where xi are repeating variables, yi are performance variables, and αij are exponents. Equation 2 pres-ents the dimensionless form of the primitive (the reusable modeling primitives) we used to develop the framework presented in this article. Examples of these primitives are present in multiple domains of science, including the efficiency rate, the Reyn-olds number, and the Froude number.

To understand how dimensionless primitives are associated with the causal graphs used in this work, consider the causal relations among energy (En) mea-sured in Joules ( J), power (P) measured in Watts (W), and time (t) in measured seconds (s) (see Figure 1).

From the causally oriented graph presented in Figure 1 and presenting causal relationship be-tween En, P, and t, we can use Equation 2 to con-struct a dimensionless product:

π = ⋅ ⋅− −En t P .En1 1 (3)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









44 July/August 2016

In this article, we use a mathematical machin-ery that R. Bhaskar and Anil Nigam7 developed to reason about a system using the type of relationship derived from Equation 2. A dimensionless group ini-tially presented in Equation 2 can be expressed

π= ⋅ ⋅ ⋅α α α− − −y x x x ,i k j l mij il mi (4)

which can be divided by xj to form

π= ⋅ ⋅ ⋅α α α− − −y

xx

xx

xx

x.i

jk

j

j

l

j

m

j

ij il mi

(5)

From Equation 4, we can write a partial deriv-ative involving variables yi and xj, taking the form

π α∂∂=− ⋅ ⋅ ⋅

α α α− − −yx

xx

xx

xx

.i

jk ij

j

j

l

j

m

j

ij il mi

(6)

The partial derivative can be reformulated and simplified by replacing Equation 5 with Equation 6:

α∂∂=−

yx

yx

.i

jij

i

j

(7)

From Equation 7, the sign of the derivative (∂yi)/(∂xj) can be determined by simply verifying the

sign of exponent αij. This simple machinery pro-vides a powerful approach for propagating qualita-tive optimization objectives (maximize, minimize) in a causal network.

Let’s take the small example of Figure 1, in which we define the initial objective of minimiz-ing energy (En). What should be the resulting ob-jectives for power (P) and time (t)? By using Equa-tion 3, we can derive two partial derivatives:

∂∂=

EnP

EnP

1 . (8)

∂∂=

Ent

Ent

1 . (9)

From Equations 8 and 9, we can deduce that both P and t vary in the same direction as En due to the sign of the partial derivative. Consequently, if En needs to be minimized, we must also mini-mize P and t (see Figure 2). This process is called in this article backward propagation.

Fundamental Types of Variables in a

Modeling Problem

The bond graph modeling approach8,9 is a domain-independent graphical description of physical systems’ dynamic behavior. It introduces several categories of fundamental variables, including

■ energy, ■ power variables (efforts and flows), and ■ state variables (displacements and momentums).

We use a fourth category, called connecting variables,10 in our DACM framework to describe material, component-specific properties, geometric dimensions, tolerances, and so on. Table 1 summa-rizes the categories used in our work.

Design Structure Matrix and Domain

Mapping Matrix

A design structures matrix (DSM)10 is a way of rep-resenting a graph that can be used to list all of a sys-tem’s constituent parts or a process’s activities and the corresponding information exchange, interactions, and dependency patterns. DSMs compare interac-tions between elements of a similar nature, whereas the domain mapping matrix (DMM) is used to map elements belonging to different domains, for example, functions and physical subsystems.

The DSM and DMM are used systemati-cally in the software tool supporting our DACM framework to represent interconnections in the functional structures, causal graphs, and the laws

Figure 2. Backward propagation of objectives in a causal graph representing the relations among energy, time, and power.

Initial objective:To be minimized

Propagated objective:To minimize

Propagated objective:To minimize

En

P

t

Figure 1. A small causal graph showing the relations among energy (E), time (t), and power (P).

En

P

t

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








governing the system. DMMs map the functions with the variables. DMMs also map variables with elementary units of the International System of Units (SI) and variables with system laws.

The DACM Framework

We can summarize the DACM process into a few fundamental steps.

Step 1: Indicate the Model’s Objectives

The modeler starts by explicitly providing ratio-nales for the model’s aim and its borders. The func-tional modeling process is controlled in this phase by using an ontology (http://protege.stanford.edu) and a normalized set of functional terms.11

Step 2: List the Problem’s Fundamental

Variables

Table 1 defines seven categories of possible vari-ables when modeling a system. More specific terms for the variables can be defined by using a taxono-my described by Julie Hirtz and her colleagues.11

Step 3: Position System Variables

on the Functional Structure

The variables listed in Step 2 are placed on the func-tional decomposition built in Step 1, and a color is associated with each variable:

■ Exogenous variables colored in black are vari-ables imposed by the system’s external envi-ronment. There’s no degree of freedom for changing exogenous variables.

■ Independent design variables colored in green aren’t influenced by any other variable in the system and can be selected during the design process.

■ Dependent design variables colored in blue are influenced by other variables and thus more difficult to control. These variables can be se-lected during the design process.

■ Performance variables colored in red are a spe-cial class of dependent design variables that are important for evaluating overall system per-formance. Designers try to optimize them by minimizing or maximizing them or by obtain-ing a target value for them.

Step 4: Develop a Causal Ordering

of the Variables

The fourth step of the DACM process is funda-mental. During this phase, the cause-effect re-lationships between variables are defined in the

form of a causal graph. The algorithm shown in Figure 3 generates the causal graph by consider-ing multiple rules derived from a bond graph and DA metrics.12 The algorithm is associated with a mapping of functional vocabulary with bond graph organs.

Figure 4 provides a visualization of the causal algorithm generated by our DACM tool.

Step 5: Construct the Model’s

Behavioral Laws

In Step 5, the DACM software tool automatically generates the system’s governing laws, which are derived from DA theory.

Step 6: Determine the Objectives

Associated with Performance Variables

The sixth step assigns objectives to the perfor-mance variables colored in red in Step 3. Us-ing the simulation machinery defined in Step 5, those objectives are propagated backward in the causal graph generated in Step 4. The objec-tives can be qualitative (that is, maximizing or minimizing).

Step 7: Find the Contradictions

The propagation of objectives in the causal graph can generate contradictions.13 For example, the re-sulting objective of propagation can lead to vari-ables that should simultaneously answer contradic-tory objectives.14

Step 8: Determine Design Directions

and Potential Added Values

The eighth step involves generating a virtual design of an experiment and computing the impact of dif-

Table 1. Fundamental categories of variables.

Primary type of variables Secondary categories

Overall system variables Energy (En)

Efficiency rate (η)

Power variables (p) Generalized effort (effort)

Generalized flow (flow)

State variables Generalized displacement (displacement)

Generalized momentum (momentum)

Connecting variables (connecting)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_________________


http://protege.stanford.edu






46 July/August 2016

ferent variables on performance variables. These variables are then ranked according to their impact level.

Step 9: Generate Innovative Solutions

to Design Contradictions

The ninth step applies inventive design principles to remove design contradictions. In the next section, we describe a case study with a torpedo in which we at-tempted to transform exogenous variables into a design variable by increasing the studied system’s boundaries.

Two Case Studies

To evaluate the DACM framework’s capacity for M&S conceptual design solutions, we set up two case studies. Several other inventive principles have been developed in this research but aren’t presented in this article. Those principles are independent of the TRIZ approach,15,16 providing an alternative approach for generalizing inventive principles.

Case Study 1: A Torpedo

To present our framework’s innovative usage, we focused on a torpedo moving into water. Specifi-

Figure 3. The algorithm that generates the causal graph by considering multiple rules derived from a bond graph and dimensional analysis metrics.

Create a functionalstructure

Map the functionalstructure with bond graph organs

Is it a bijectivemapping?

Use the causal rules associatedwith the individual bond graph

components to definethe causality for individual

flows and efforts

Integrate more knowledge related to

-type of conservative law considered (force or flow),-type of energy transmitted, and-type of variables associated with the functions

No

Yes

Use causal rules to definethe causality among:

displacement, momentum,and connecting variables

Always add abidirectional causality for

associated flows and efforts

By the time integration happens,a flow entering in a box is causally

linked with a displacementand effort is causally linked

with a momentum

By the time derivation happens,a displacement is causally linked

with a flow leaving a box, andmomentum is causally linkedwith an effort leaving a box

Are the variables of thestructure completely

ordered?

Travel into the structurefrom left to right by

following the connections

Identify modeling problem

End

Display in form of acausal graph

Yes

No

Figure 4. Causal graph visualization. The example presents the causal graph of a pressure regulator regulating an output pressure (Pout). The variables describe the fluid properties and the internal structure of the piston and spring used in the pressure regulator’s architecture.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








cally, the designers want to increase the torpedo’s speed under water. The case study’s two initial ob-jectives in the context of this article were to extract the most important set of variables and the units re-quired to model the torpedo in water, and to define the causal relationships between those variables,

The torpedo model takes into account both the torpedo and its interaction with the water; its internal structure isn’t described in this model. Rather, we’re studying the function of moving into water. The wa-ter resists this movement and generate a drag force Fd. To generate the model of the torpedo moving into water, we need to apply Steps 1, 2, and 3 of our framework. Figure 5 summarizes those three steps of the DACM framework.

From the steps summarized in Figure 5 and and by using the causal ordering algorithm and tool presented in Step 4, we can create the causal ordering graph for the torpedo (see Figure 6).

To derive the mathematical laws, we need to know each variable’s SI (International System of Units) units. Table 2 shows the list of variables gov-erning the problem with their SI elementary units.

Qualitative objectives are defined for the perfor-mance variables presented in red in Figure 6. We want to maximize the torpedo speed and to minimize the drag force. The propagation of those objectives in the causal graph is done by using a simulation machinery.7

The computation of the associated ∏number is provided as follows17

π μ ρ= ⋅ ⋅ ⋅ ⋅− − −−

Fd V A ,Fd2

12

12

34

3

(10)

where πFd is both a dimensionless primitive and drag coefficient Cd.

A second dimensionless primitive can be formed for V:

V Fd A .V

12

12

32 2π ρ μ= ⋅ ⋅ ⋅ ⋅ − (11)

With these two equations, we’ve proven that the DACM framework can rediscover formulas re-lated to fluid dynamics. The computation leading to both equations uses DMMs (Domain Mapping Matrices) presented in Tables 3 and 4.

Figure 5. Functional representation of the interaction of a torpedo and water. The variables in green, blue, and black are listed in Table 2.

Torpedomoves into

water

Torpedo movesby acting on

the water

V

ρμ

A

μ

A

Torpedo andwater interact

together

Water resiststhe torpedomovement

Energy isdissipatedinto water

V Fd

Fd

ρ

Figure 6. Causal ordering graph for the torpedo case study. (Independent design variables are green, exogenous variables are black, and performance variables are red.) The variables listed in the causal graph are the same presented in Table 2.

Fd

A

V

ρ

μ

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









48 July/August 2016

The linear algebra transformation used to generate Equations 10 and 11 is presented in the following Equation 12 in which C is the vector representing the exponents of Equations 10 and 11. B is the matrix representing the dimensions of the performance variable and A represents the dimensions of the repeating variables.17

=− ⋅−C A B( ) .T1 (12)

Two contradictions appear in Figure 7 on the drag force Fd and on the speed V both must be maximized and minimized to fulfill the initial objectives, which contradicts the initial design objectives. Let’s select the contradiction on the

drag force Fd as the constraint to be solved. An approach based on virtual design of experiment (DOE) is proposed in this article to evaluate the impact of the different parameters influencing the drag force Fd. To conduct this DOE, orders of magnitude have to be proposed for variables V[84 m/s, 250 m/s], A [1.5 m2, 4 m2], μ [0,000653 Pa·s, 0,00179 Pa·s], and ρ [0.179 kg/m3, 1.03 kg/m3]. These ranges are extracted from a technical database, and we use the Taguchi approach18 to conduct the DOE.

Four variables influence Fd: V, A, μ and ρ. The degrees of freedom (DOF) allowing us to select the proper Taguchi table is computed by using the fol-lowing method: a DOF is allocated for each vari-able and for the model’s mean value (DOF = 1 + 1 + 1 + 1 + 1 = 5). At a minimum, an L5 orthogonal array is used to analyze variable sensitivity. We se-lect low and high quantitative levels for V, A, μ, and ρ compute the effect of the variables on Fd:

∑ ∑= −− −XiG

nG

n.X High in X Low in

1 1(13)

The parameters V, A, μ, and ρ in Equation 10 that have a significant influence on Fd are ranked by order of magnitude in Figure 8.

Table 2. List of problem variables and International System of Units (SI) elementary units.

Categories (objectives,

other variables) Variables

SI elementary units

(L = length, M = mass, T = time)

Performance variables

Speed (V ) LT –1

Drag force (Fd) MLT –2

Other variables Fluid density (ρ) ML–3

Torpedo reference surface (related to shape and length, A)

L2

Fluid dynamic viscosity (μ) ML–1T –1

Table 3. Domain mapping matrix (DMM) representing node Fd in Figure 6.*

[B] [A]

Unit/variables Drag force (Fd) Torpedo reference surface (A)

Fluid density (ρ); fluid dynamic velocity (μ)

Speed (V )

Mass (M) 1 0 2 0

Length (L) 1 2 –4 1

Time (T ) –2 0 –1 –1

*[A] represents the different variables influencing this node; [B] is the matrix representing the variables influenced by [A].

Table 4. DMM representing node V in Figure 6.

[B] [A]

Unit/variables V Fd × A ρ Fluid dynamic velocity (μ)

Mass 0 1 1 1

Length 1 3 –3 –1

Time –1 –2 0 –1

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








The fluid properties have a significant impact on Fd since the A, the reference surface of the tor-pedo, is already a parameter that has been opti-mized in the design. V can’t be considered because it’s also a final objective of the study. The central question is how to modify μ and ρ. Finding an answer to this question limits or removes the con-tradictions detected in the causal graph in Figure 7. Transforming μ and ρ into a design parameter that can be controlled implies modifying the torpedo system’s borders. The fluid in which the torpedo is moving must be integrated in the system’s bound-aries. How do we do this in practice?

The DOE helps us notice that gases such as CO2 have very low density and viscosity, μ. Thus, integrating a pressurized bottle of CO2 into the torpedo and injecting the gas in front of it is a possible solution. This is what the Russian tor-pedo Skval does. Another option is to generate bubbles of gas at the torpedo’s surface by creating very high local temperatures or local difference of pressure. These changes in torpedo design modify the causal graph in Figure 7, which create other contradictions in turn. The torpedo might, for example, become more difficult to control. This analysis process will have to be iterated to another parts of the torpedo architecture in order to solve the other contradicitons emerging from this new design.

Case Study 2: Laser Light Interacting

with Metal Powder

Our second case study focuses on the development of a simulation model to analyze the interaction between a laser source and a metal. This case study’s objective is to present in detail the method used to compute the model’s mathematical laws. Figure 9 shows the functional model and a list of variables influencing the process. Table 5 shows the variable names and units, and Figure 10 shows the causal graph associated with the process and the variables.

The example in our case study develops the mathematic machinery associated with the part of the causal graph surrounded in red in Figure 10. The melted metal rate mm is influenced by multiple variables, including the metal and the equipment properties. We used a DMM for representing the machine and material variables influencing mm(see Table 6). A short calculation shows that we have eight variables because ε is dimensionless and can be removed from this list (ε is reintegrated at the end of the process when the different dimen-

sionless primitives are combined). We have eight variables and four initial dimensions (mass, length, time, and temperature), so we can form 8 – 4 = 4 dimensionless primitives. Consequently, matrix [B] has four columns, selected by priority of dependent or independent design variables. Matrix [A] also has four columns; the last variable ε is kept outside the two matrixes because of being dimensionless already (see Table 5 and 6).

The next element to consider is the value of the determinant of [A]. A computation of det[A] shows that its value is null, meaning the rank of [A] is lower than 4. By removing mass because all values are null in [A] and by combining two col-umns, we get a new matrix [A] of rank 3, non-null (see Table 7).

We can now use Equation 12 (presented in case study 1) to determine the following dimensionless groups. Equation 14 is generated by using Table 8, Equation 15 is generated by

Figure 7. Causal graph of a torpedo with the propagation of the two given objectives in blue. The propagated objectives7 are represented in red in the figure.

min

MAX

min

min

MAX

min

min

MAX

V

ρ

μ

Fd

A

Figure 8. Order of magnitude of the different parameters influencing drag force Fd.

–250

–200

–150

Ran

ked

effe

ct o

n th

e dr

ag f

orce

(%

)

–100

–50

0

50

100

150

200

250

Effect of ρ Effectof μ

Effect of V Effect of A

Variables affecting the drag force

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









50 July/August 2016

using Table 9, Equation 16 is generated by us-ing Table 10, and Equation 17 is generated by using Table 11:

π α= ⋅ ⋅ ⋅ ⋅−m h C T( ) ,m m f p2 1 1

m(14)

π α= − ⋅ ⋅ ⋅ ⋅−−P P h C T( ) ( ) ,P P c f p

3 1 1c

(15)

π α= ⋅ ⋅ ⋅ ⋅−m h C T( ) ,m f p2 1 1 (16)

π α= ⋅ ⋅ ⋅ ⋅− − −D h C T( ) .D f p

32 1 1 (17)

Equation 1 in this article shows that the in-dividual dimensionless groups can be combined together. We can then combine them in the follow-ing form:

π π π π= −f ( , , ).m P P m Dm c(18)

According to the product theorem,3,4 it’s also possible to write Equation 18 in the following form:

π π π π= ⋅ ⋅ ⋅α β γ−K .m P P m Dm c

The constant K and exponents α, β, and γ should be defined using experiments on additive manu-facturing equipments. The experiments can also be used to adjust the model and verify the real influ-ence of the parameters considered in the model. It can be possible that some parameters have little im-pact on the performance parameters, and those ex-periments should help to detect them and remove them from the model without losing its dimension-al homogeneity.

Table 5. Variables involved in the laser melting process.

Variable Name Unit Quantity

mm Melted metal rate Kg/s MT−1

Nd Nozzle diameter m L

P Laser power Watt ML2T−3

m Powder flow rate Kg/s MT−1

Pr Powder feed pressure Pa ML−1T−2

D Laser spot diameter m L

ρ Powder density Kg/m3 ML−3

t Melting point °C t

Cp Specific heat capacity Joule/g°C L2T−2t−1

hf Heat of fusion Joule/g L2T−2

Pc Critical power Watt ML2T−3

α Thermal diffusivity m2/s L2T−1

ε Absorptivity -- Dimensionless

Figure 9. Functional model of a process using a laser to melt a metallic powder. The variables listed in the figure are listed in Table 5.

Convertenergy

Convertpowderstate

Metallic powder

Meltedmetal

Electrical

energy

Light energy

mmm

P

Nd

T

ε α

Cp hf

D

Pc

Pr

Guideand

conveypowder

ρ

ρ

Figure 10. Causal graph generated via the causal ordering software.

mm

Cp

T

hf

P–Pc m Nd

Pr

Input extrapolation

ρ

ρ

ε

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








This second case study is a very general pre-sentation of the DACM approach and is only lim-ited by the requirement of having parameters that can be measured. It’s also possible to add measur-ing units not belonging to the SI system of units

if needed. We’ve demonstrated here that it’s pos-sible to use this approach to determine complex relations between parameters. However, the rela-tions between dimensionless primitives should be validated using an experimental process.

One aspect not covered in this article is the pos-sibility of using DACM as a specification tool.

The specification framework is based on a set of DSMs and DMMs populated by numbers; these matrices form a “fingerprint” of the design problem and its conceptual solution. This “identity card”19

can be used to specify problem requirements in a compact manner and validate the produced so-lution. The DACM framework can also be used to specify the development of reusable modeling

Table 6. Initial DMM for the causal relations related to the melted metal rate mm.

Unit/variables [B] [A]

∏mm mm P – Pc m D hf Cp T α ε

Mass 1 1 1 0 0 0 0 0 0

Length 0 2 0 1 2 2 0 2 0

Time –1 –3 –1 0 –2 –2 0 –1 0

Temperature 0 0 0 0 0 –1 1 0 0

Table 7. DMM for mm with determinant of [A],

non-null.

Unit/

variables [B] [A]

∏mm mm P – Pc M D hf Cp Tα

Length 0 2 0 1 2 2 2

Time –1 –3 –1 0 –2 –2 –1

Temperature 0 0 0 0 0 –1 1

Table 8. DMM for the first dimensionless

primitive.


∏mm mm hf Cp Tα

Length 0 2 2 2

Time –1 –2 –2 –1

Temperature 0 0 –1 1

Table 11. DMM for the fourth dimensionless

primitive.


∏mm D hf Cp Tα

Length 1 2 2 2

Time 0 –2 –2 –1


Table 9. DMM for the second dimensionless

primitive.


∏mm P – Pc hf Cp Tα

Length 2 2 2 2

Time –3 –2 –2 –1


Table 10. DMM for the third dimensionless

primitive.


∏mm m hf Cp Tα

Length 0 2 2 2

Time –1 –2 –2 –1


qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









52 July/August 2016

primitives and create new models to address alter-native problems.

DACM is supported by a computer-aided tool that’s currently in its test phase in an industrial context. The development team’s future work is to create a user interface for the tool. Alternative us-ages of the DACM framework are also currently under development.

References

1. D. Kahneman, Thinking, Fast and Slow, Farrar, Straus and Giroux, 2011.

2. M. Kistler, Causation and Laws of Nature, Rout-ledge, 2006.

3. P.W. Bridgman, “Dimensional Analysis,” Encyclo-paedia Britannica, vol. 7, 1969, pp. 439–449.

4. G.I. Barenblatt, Scaling, Self-Similarity, and Inter-mediate Asymptotics, Cambridge Univ. Press, 1996.

5. J.C. Maxwell, A Treatise on Electricity and Magne-tism, 3rd ed., Dover, 1954.

6. W. Matz, Le Principe de Similitude en Génie Chi-mique (in French), Dunod, 1959.

7. R. Bhashkar and A. Nigam, “Qualitative Physics Using Dimensional Analysis,” Artificial Intelligence,vol. 45, 1990, pp. 73–111.

8. H.M. Paynter, Analysis and Design of Engineering Systems, MIT Press, 1961.

9. T. Shim, Introduction to Physical System Modelling Using Bond Graphs, University of Michigan-Dear-born, 2002.

10. D.V. Steward, “The Design Structure System: A Method for Managing the Design of Complex Sys-tems,” IEEE Trans. Eng. Management, vol. 28, no. 3, 1981, pp. 71–74.

11. J. Hirtz et al., “A Functional Basis for Engineering Design: Reconciling and Evolving Previous Efforts,” Research in Eng. Design, Springer-Verlag, 2002, pp. 65–82.

12. Q. Shen and T. Peng, “Combining Dimensional Analysis and Heuristics for Causal Ordering,” In Memory of Rob Milne: A Tribute to a Pioneering AI Scientist, Entrepreneur and Mountaineer, A. Bundy and S. Wilson, eds., IOS Press, 2006.

13. J. Ring “Discovering the Real Problematic Situa-tion: The First Aspect of Conceptual Design,” IN-COSE INSIGHT, vol. 17, no. 4, 2014, pp. 11–14.

14. J.N. Warfield, Understanding Complexity: Thought and Behavior, AJAR Publishing Co., 2002

15. G. Altshuller, Creativity as an Exact Science, Gordon & Breach, 1984.

16. S.D. Savransky, Engineering of Creativity, Introduc-tion to TRIZ Methodology of Inventive Problem Solv-ing, CRC Press, 2000.

17. T. Szyrtes and P. Rozsa, Applied Dimensional Analy-sis and Modeling, Elsevier Science & Technology, 2006

18. P. Goos and B. Jones, Optimal Design of Experi-ments: A Case Study Approach, Wiley, 2011

19. G. Sirin et al., “A Model Identity Card to Support Simulation Model Development Process in a Col-laborative Multidisciplinary Design Environment,” IEEE Systems J., vol. 9, no. 4, pp. 1151–1162

Eric Coatanéa is a full professor in the Department of Mechanical Engineering and Industrial Systems at Tampere University of Technology. His research inter-ests include system engineering, design methodologies, and manufacturing. Coatanéa received a double doc-torate in engineering design and product development from Aalto University and the University of West Brit-tany. Contact him at [email protected].

Ric Roca is a member of the professional staff at Johns Hopkins University Applied Physics Laboratory. His re-search interests include modeling and simulation proj-ects and technologies. He sponsored the development of the DACM framework during his tenure as deputy di-rector of the Joint Assessment and Enabling Capabilities Office within the Personnel & Readiness Directorate in the Office of the Undersecretary of Defense for Readi-ness, US Department of Defense. Contact him at [email protected].

Hossein Mokhtarian is a joint doctoral student in the Department of Mechanical Engineering and Industrial Systems at Tampere University of Technology and the University of Grenoble. Contact him at [email protected].

Faisal Mokammel is a doctoral student in the Depart-ment of Engineering Design and Production at Aalto University. Contact him at [email protected].

Kimmo Ikkala is a doctoral student in the Department of Mechanical Engineering and Industrial Systems at Tampere University of Technology. Contact him at [email protected].



qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____________

__________

__

________

_________

________________

____________













Micromagnetic simulations are impor-tant tools for studying magnetic dy-namics and designing new magnetic devices, and solvers running on CPUs

such as OOMMF1 and magpar2 have been wide-ly used in magnetism research. However, mi-cromagnetic simulations of complex structures require fine geometrical discretization and are time-consuming.

To tackle this challenge, researchers are ex-ploring how to apply general-purpose GPUs such as MuMax,3,4 FastMag,5,6 GPMagnet,7 and Mi-croMagnum8 to micromagnetic studies. Due to GPU units’ high computing power, these imple-mentations have achieved considerable speed-ups compared to CPU-based implementations. Moreover, general-purpose GPUs are cheap, with most costing less than US$1,000, significantly lowering the price of complex micromagnetic simulations.

However, these general-purpose GPU im-plementations are based exclusively on Nvidia’s parallel computing platform (CUDA), and their applications are limited to Nvidia GPUs. In 2012, Microsoft released its own parallel pro-gramming library—called C++ AMP—an open specification that’s a cross-hardware platform supports GPUs from all major hardware vendors and runs on Windows.9 The most recent version of C++ AMP can also run on Linux.10 Software based on C++ AMP can run on virtually all the

latest GPUs, including those from Nvidia, AMD, and Intel. The purpose of our work is to imple-ment a cross-hardware platform micromagnetic solver based on the C++ AMP framework for solving the Landau-Lifshitz-Gilbert (LLG) equa-tion, so that the dynamics of magnetic nanostruc-tures can be studied.

Principle

Consider a magnetization vector M M M M( , , )x y z

�=

in a computational cell that belongs to the com-puting region. For its saturation magnetization, M M M M .s x y z

2 2 2= + + The magnetic energy den-sity related to this vector can be written as

AMM

MM

MM

KM M

M

H M H M12

x

s

y

s

z

s

uy z

s

demag extern

2 2 2

2 2

2

0 0

� � � �

ε

μ μ( )

( )

( )

= ∇⎛

⎝⎜⎜⎜

⎞

⎠⎟⎟⎟⎟+ ∇⎛

⎝⎜⎜⎜⎜

⎞

⎠⎟⎟⎟⎟+ ∇⎛

⎝⎜⎜⎜

⎞

⎠⎟⎟⎟⎟

⎡

⎣

⎢⎢⎢

⎤

⎦

⎥⎥⎥

++

− − (1)

The right hand side of Equation 1 consists of the exchange, anisotropy, demagnetization, and Zeeman energy densities, where A is the material exchange constant, Ku is the uniaxial anisotropy constant, μ0 is the vacuum perme-ability, Hdemag is the demagnetization field, and Hextern is the external field. The anisotropy

Speedup of Micromagnetic Simulations with C++

AMP on Graphics Processing Units

Ru Zhu | Graceland University

Using a finite-difference micromagnetic solver to increase GPU performance achieved two orders of magnitude improvement compared with a typical CPU-based solver. The solver, called Grace, is based on C++ AMP and can run on GPUs from various hardware vendors, regardless of whether it runs on a dedicated or integrated graphics processor.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







54 July/August 2016


energy is assumed to be uniaxial with an easy axis in the x direction.

The magnetization vector change is caused by the effective magnetic field Heff, which is derived from the magnetic energy density:

��

� � � �

δεμ δ

=−

= + + +

HM

H H H H .

eff

exch anis demag extern

0

(2)

where �

δε δM gives the functional derivative of εwith respect to

�M . In Equation 2, the exchange

field and anisotropy field are represented by �H exch

and �H anis, respectively. According to Equations 1 and 2, the exchange

field’s x component is

μ= ∇H

AM

M2

exch xs

x,0

22 . (3)

To derive the exchange field Hexch, we need to discretize the computing region properly and consider the magnetizations of neighboring com-putational cells. The entire computing region is divided into nx × ny × nz cells, each cell with an equal volume of δxδyδz. The cells are labeled with indices

0 ≤ i ≤ nx – 1,0 ≤ j ≤ ny – 1,0 ≤ k ≤ nz – 1.

Note that i, j, and k are zero-indexed to follow the convention of C++ programming language.

According to Equation 3, the Cartesian com-ponents of the effective field can be expressed as11

H

AM

M i j k M i j k M i j kx

M i j k M i j k M i j ky

M i j k M i j k M i j kz

2 ( 1, , ) 2 ( , , ) ( 1, , )

( , 1, ) 2 ( , , ) ( , 1, )

( , , 1) 2 ( , , ) ( , , ).

exch x

s

x x x

x x x

x x x

,

02 2

2

2

μ δ

δ

δ

{

}

=

+ − + −

++ − + −

++ − + −=

(4)

Other components of �H exch can be obtained by

replacing x with y or z in Equation 4.According to Equations 1 and 2,

μ=H

KM

M2

anis xu

sx,

02 . (5)

The LLG equation in the low damping limit is12

��

� � �

γα

μ

αγα

μ

=−+

×

−+

× ×⎡⎣⎢

dMdt

M H

MM M H

1( )

(1 )( )

eff

seff

2 0

2 0 , (6)

where α is the damping constant, and γ is the gy-romagnetic ratio.

To speed up the micromagnetic simulation, we need to decrease per-step simulation time, most of which is consumed in calculating the demagnetiza-tion field. The brute-force calculation of the demag-netization field is known to be proportional to the square of the number N of computational cells.13

However, this calculation can be accelerated by tak-ing advantage of the discrete convolution theorem and the fast Fourier transform (FFT).14 With the zero-padding method (introduced later), nonperi-odic boundary conditions can be used.

Consider a rectangular magnetic sample divided into nx × ny × nz cells in 3D space. After zero-padding, the input data size increases to 2nx × 2ny × 2nz, as Fig-ure 1 shows. Because demagnetization field data after an FFT occurrence contains complex numbers, there’s an additional factor of two in FFT output size.

In the case of a finite 3D computing problem, the demagnetization field can be calculated as

∑∑∑= − − −=

−

=

−

=

−

H i j k

M l m n K l i m j n k

( , , )

( , , ) ( , , )

demag

n

n

m

n

l

n

0

1

0

1

0

1 zyx

(7)

Figure 1. A cross-sectional view of a rectangular magnetic sample after zero-padding. Because demagnetization field data after a fast Fourier transform occurrence contains complex numbers, there’s an additional factor of two in FFT output size.

2ny–1

2nx–1

ny

nx0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








or

∑∑∑= − − −

+ − − −

+ − − −

=

−

=

−

=

−

H i j k




( , , )

{ ( , , ) ( , , )

( , , ) ( , , )

( , , ) ( , , )},

demag x

x xxn

n

m

n

l

n

y xy

z xz

,

0

1

0

1

0

1 zyx

(8)

where K is the demagnetization kernel matrix, of which the formulation can be found else-where.13 Other components of Hdemag can be obtained by permutations of indices in Equa-tion 8.

By applying the discrete Fourier transform (DFT) theorem to both sides of the equation, we get

� �

� �

� �

= ⋅

+ ⋅

+ ⋅

H i j k M i j k K i j k

M i j k K i j k

M i j k K i j k

( , , ) ( , , ) ( , , )

( , , ) ( , , )

( , , ) ( , , )

demag x x xx

y xy

z xz

,

. (9)

Finally, the demagnetization field Hdemagcan be obtained by taking the inverse FFT of �Hdemag ,

�= −H FFT H( )demag x demag x,1

, . (10)

Note that Hdemag contains only real numbers because �Hdemag is the dot product of FFTs of two real arrays.

Implementation

GPUs have an intrinsically different hardware ar-chitecture from CPUs, notably to handle their large number of arithmetic logic units (ALUs), which were initially designed for graphics render-ing but are now also used for general-purpose com-puting. Because GPUs are specialized for comput-ing-intensive, highly parallel computation, they’re ideally suited for micromagnetic simulations in which large numbers of computational cells must be processed in parallel (see Figure 2).

C++ AMP uses the High Level Shading Lan-guage (HLSL), which was initially designed for Microsoft’s graphics library, DirectX.9 Compared to popular GPU programming languages such as CUDA, it’s a fully cross-hardware platform, which means that programs written in C++ AMP can be migrated to another hardware vendor without any modification. Compared with other cross-hard-ware platform GPU programming languages such as Open Computing Language (OpenCL), C++ AMP has a much simpler API, thus reducing the programming effort.

GPUs usually have their own memory, also known as graphic memory, and data I/O is very fast between its ALUs and memory (more than 100 Gbytes/s), compared to the I/O between GPU and CPU (about 10 Gbytes/s). The bottleneck to boost GPU computing performance is therefore the data transfer from CPU to GPU, and vice versa. In the micromagnetic solver presented here (which is called Grace), the only data transfer between CPU and GPU takes place when the region’s initial conditions

Figure 2. A comparison of CPU and GPU hardware architectures. The GPU has more arithmetic logic units dedicated to data processing, but the amounts of cache and RAM are limited on GPUs.

Controller

ALU ALU

ALUALU

Cache

DRAM

CPU GPU

DRAM

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








56 July/August 2016


are computed and when the final data is calculated by GPU and transferred back to the CPU for display. In this way, simulation speed can be maximized.

As mentioned earlier, the most time-consuming part of micromagnetic simulation involves calculat-ing the demagnetization field. In each time step, this calculation requires three different phases: per-form FFTs to magnetization components, create a member-wise result of the previous step (and FFTs of demagnetization tensors), and carry out an inverse FFT. In 3D space, six FFTs must be performed for each time step. The FFTs of demagnetization tensor K are carried out at the simulation’s start and aren’t computed later because K is constant.13

An FFT library based on C++ AMP has already been implemented15 and adapted to calculate the solver’s demagnetization field. At the time of this writing, the FFT library can only handle single-precision floats, so this solver is currently limited to single-precision computing. However, software should gain double-precision calculation capability with the next update of the C++ AMP FFT library.

Results

To evaluate the speedup gained by GPU, we can use micromagnetic standard problem 3 to test the solver’s performance.16 A cubic magnetic particle with exchange constant A = 1 × 10–11 J/m, Ms = 1,000 kA/m, and Hanis = 100 kA/m is divided into

grids of N × N × N, and the minimum energy state is reached by applying the LLG equation to each computational cell. As shown in Equation 5, the anisotropy field is maximized when magnetiza-tion is parallel or antiparallel with the easy axis. The relaxation process involves the magnetization dynamics under the influence of demagnetization, exchange, and uniaxial anisotropy fields.

To benchmark the solver, we tested two dif-ferent GPUs: an Nvidia GTX 650 Ti and an AMD Radeon HD 7970 GHz. Both GPUs ran on an Intel Xeon E5410 CPU with 4 Gbytes of RAM. The AMD Radeon 7970 is among the fast-est on the consumer market but still costs less than $500. The Nvidia GTX 650 Ti is a middle-end product that costs less than $100. For com-parison, we used MuMax on the GTX 650 Ti and OOMMF on the CPU, with data from the same hardware system, which contains a quad-core In-tel Xeon E5410 CPU. Dimensions with factors of 2, 5, and 10 were benchmarked to demonstrate solver performance on various problem sizes (see Table 1). As we can see, this magnetic solver can solve problems of any size, limited by the graphic memory allocable by the GPU, but we must note that the OOMMF simulation was done in dou-ble precision; the benchmark on the GPUs was in single precision only. Furthermore, OOMMF has more advanced ordinary differential equation

Table 1. Per-step simulation time (ms) needed by CPU and GPU solvers for different 3D problem sizes

with the Euler algorithm.

Size CPU Xeon E5410

Nvidia GTX 650

Ti (MuMax 3.6.2

Windows 64-bit)

Nvidia GTX 650

Ti (Grace)

AMD Radeon 7970

(Grace)

512 (83) 3.42 1.09 0.733 0.677

4,096 (163) 4.63 1.20 0.764 0.683

8,000 (203) 9.53 1.87 1.94 1.02

15,625 (253) 11.4 2.05 3.21 1.19

32,768 (323) 14.6 2.46 3.14 1.24

64,000 (403) 62.3 6.57 9.54 2.87

125,000 (503) 79.1 12.7 22.3 4.94

262,144 (643) 117 14.0 31.2 8.62

512,000 (803) 778 40.5 71.1 18.1

100,000 (1003) 933 79.2 N/A 31.4

2,097,152 (1283) 1382 103 N/A 56.9

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








(ODE) solvers than the Euler method bench-marked here, as well as a conjugate gradient solv-er, which further reduces its simulation time. As Table 1 shows, Grace outperforms MuMax for smaller input problem sizes, but was roughly two times slower for larger inputs.

Notably, at smaller problem sizes (number of cells < 10,000), the GPU solver isn’t significantly faster or slower than the CPU solver. This is owing to two factors: data I/O overhead and the GPU’s kernel launching overhead. The data transfer be-tween GPU and CPU main memory takes time. For a smaller problem size, the calculation on the GPU can be completed very quickly, so in this case, the computing power won’t be fully utilized. For larger problems, data I/O time can be negli-gible when compared to computing time. The ker-nel-launching overhead is a constant regardless of problem size, thus it’s significant when the problem size is small.

As Figures 3 and 4 show, the per-step simulation times show a staircase-like behavior with respect to input size. This is because the dominant part of mi-cromagnetic simulation is the FFT calculation done by the FFT libraries—calculation performance is optimized for power-of-two input sizes, so for other input factors, the per-step time is significantly larger. In Figure 4, we see that the greatest speedup fac-

tors were achieved at non-power-of-two numbers, in which case the GPU FFT library outperforms that of OOMMF.

For extremely large simulation sizes, in which the number of cells is on the magnitude of millions, budget GPUs are limited by graph-ics memory and can’t support the running of the program. This is reflected in the last two rows of Table 1.

To validate the simulation result, we dis-cretized the cubic particle to 10 × 10 × 10 cells and found the transition point from flower state to vortex state to be near l = 8.47 lex, where l is the cube’s edge length and μ=l A M2ex s0

2 is the intrinsic length scale. Figure 5 shows the cu-bic particle’s magnetizations before and after the transition.

We also used micromagnetic standard problem 4 to validate the calculation results.17 In this prob-lem, a rectangular film sample is divided into 500 × 125 × 3 cells, with a mesh size of 1 nm × 1 nm × 1 nm. We set the sample’s exchange constant as 1.3 × 10–11 J/m and the saturation magnetization as 8 × 105 A/m. There was no anisotropy present. We relaxed the system to the S-state by setting a large damping constant before a switching field 1 of (–24.6 mT, 4.3 mT, 0 mT). During the switching, the damping constant α is set to 0.02. The input file is as follows:

-simulation 10000 1000000 5e-7

# writeInterval timesteps dt

-rectang 1000 250 6

# nx ny nz

Figure 3. Time needed to carry out one time step on different 3D problem sizes. Note the staircase behavior due to slow FFT calculations for non-power-of-two input sizes.

10,000

1,000

100

Tim

e (m

s)

10

1

0.1

512

4,09

6

32,7

68

No. cells

262,

144

2,09

7,15

2

CPU (Intel Xenon E5410)

Nvidia GTX 650 Ti (Mumax3)

Nvidia GTX 650 Ti (Grace)AMD Radeon 7970 GHz (Grace)

Figure 4. Speedup of GPU solver on an AMD Radeon HD 7970 GHz compared to the OOMMF CPU solver. The speedup increases with problem size.

512 8,192No. cells

131,072 2,097,152

Nvidia GTX 650 Ti(Mumax3)

Nvidia GTX 650 Ti(Grace)

AMD Radeon 7970GHz (Grace)

50

45

40

35

30

25

20Spee

dup

15

10

5

0

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








58 July/August 2016


-material 0.02 1.3e-11 800 0 0 0 0 0

# alpha A (J/m) M_init.x M_init.y

M_init.z Hkx Hky Hkz

-readInitState

-externfield -19.576 3.422 0. 0 3000000 3000000

# x y z startTime decayTime stopTime

Note that the magnetization and switching field unit has been converted to kA/m.

According to Figures 6 and 7, the average mag-netization results and the magnetization distribu-tion from Grace is in good agreement with that of OOMMF.17 The result is thus reliable.

I n the future, we expect the solver to fully explore the potential of GPU computing by

utilizing the shared memory; it should also be

able to reduce the kernel launching overhead by reducing the number of kernel function calls. You can download our solver from https://github.com/cygnusc/grace.

Acknowledgments

This work is supported by the Graceland University pro-fessional development program. I acknowledge Pieter B. Visscher of the University of Alabama for helpful discus-sions of demagnetization field calculation.

References

1. M.J. Donahue and D.G. Porter, OOMMF User’s Guide, US Dept. Commerce, Nat’l Inst. Standards and Tech., 1999.

2. W. Scholz et al., “Scalable Parallel Micromagnetic Solvers for Magnetic Nanostructures,” Computa-tional Materials Science, vol. 28, no. 2, 2003, pp. 366–383.

3. A. Vansteenkiste and B. Van de Wiele, “MuMax: A New High-Performance Micromagnetic Simulation Tool,” J. Magnetism and Magnetic Materials, vol. 323, no. 21, 2011, pp. 2585–2591.

Figure 5. In micromagnetic standard problem 3, the magnetization in the cubic particle is in (a) the flower state at l = 8.47 lex and (b) the vortex state at l = 8.6 lex.

10

8

86

42

0 0 2 4 6 8 10

6

4

2

0

z

yx

10

8

8 6 4 2 0 02

4 68

10

6

4

2

0

10

z

y x

(a)

(b)

Figure 6. Average magnetization versus time during the reversal in micromagnetic standard problem 4, field 1. The OOMMF simulation results are also presented for comparison.

1

0.5

–0.5

Aver

age

mag

netiz

atio

n

–10 0.1 0.2 0.3

T (ns)0.4 0.5

0

OOMMF Mx+ OOMMF My×OOMMF Mz Grace Mx

Grace My Grace Mz

Figure 7. Magnetization distribution when the component of magnetization (Mx) first crosses zero in micromagnetic standard problem 4, field 1. The domain wall is clearly seen on the left side.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



__________

https://github.com/cygnusc/grace






4. A. Vansteenkiste et al., “The Design and Verifica-tion of MuMax3,” AIP Advances, vol. 4, no. 10, 2014, pp. 107–133.

5. R. Chang et al., “FastMag: Fast Micromagnetic Solver for Complex Magnetic Structures,” J. Applied Physics, vol. 109, no. 7, 2011, article no. 07D358.

6. S. Li, B. Livshitz, and V. Lomakin, “Graphics Pro-cessing Unit Accelerated Micromagnetic Solver,” IEEE Trans. Magnetics, vol. 46, no. 6, 2010, pp. 2373–2375.

7. L. Lopez-Diaz et al., “Micromagnetic Simulations Using Graphics Processing Units,” J. Physics D: Ap-plied Physics, vol. 45, no. 32, 2012, article no. 323001.

8. A. Drews et al., “MicroMagnum: Fast Micromagnet-ic Simulator for Computations on CPU and Graphics Processing Unit (GPU),” 2015; http://micromagnum.informatik.uni-hamburg.de.

9. K. Gregory and A. Miller, C++ AMP: Acceler-ated Massive Parallelism with Microsoft Visual C++,O’Reilly, 2012.

10. K. Lisa, “AMD Announces Heterogeneous C++ AMP Language for Developers,” AMD, Aug. 2014; www.amd.com/en-us/press-releases/Pages/developer-language-2014aug26.aspx.

11. M.J. Donahue and D.G. Porter, “Exchange Energy Formulations for 3D Micromagnetics,” Physica B: Condensed Matter, vol. 343, no. 1, 2004, pp. 177–183.

12. T.L. Gilbert, “A Phenomenological Theory of Damping in Ferromagnetic Materials,” IEEE Trans. Magnetics,vol. 40, no. 6, 2004, pp. 3443–3449.

13. Y. Nakatani, Y. Uesaka, and N. Hayashi, “Direct Solution of the Landau-Lifshitz-Gilbert Equation for Micromagnetics,” Japanese J. Applied Physics, vol. 28, no. 12R, 1989, pp. 24–85.

14. N. Hayashi, K. Saito, and Y. Nakatani, “Calculation of Demagnetizing Field Distribution Based on Fast Fourier Transform of Convolution,” Japanese J. Ap-plied Physics, vol. 35, no. 12A, 1996, pp. 6065–6073.

15. D. Moth et al., “C++ AMP FFT Library,” CodePlex, Jan. 2013; https://ampfft.codeplex.com.

16. M. Donahue et al., “μMAG Standard Problem #3,” μMAG Organization, Mar. 1998; www.ctcms.nist.gov/~rdm/spec3.html.

17. B. McMichael et al., “μMAG Standard Problem #4,” μMAG Organization, Feb. 2000; www.ctcms.nist.gov/~rdm/std4/spec4.html.

Ru Zhu is an assistant professor of physics, computer science, and information technology at Graceland Uni-versity. His research interests include micromagnetics, GPU, and high-performance computing. Zhu received a PhD in physics from the University of Alabama. He’s a member of the American Association of Physics Teach-ers. Contact him at [email protected].

The American Institute of Physics is an organization of scientific societies in the physical sciences, representing scientists, engineers, and educators. AIP offers authoritative information, services, and expertise in physics education and student programs, science communication, government relations, career services

for science and engineering professionals, statistical research in physics employment and education, industrial outreach, and the history of physics and allied fields. AIP publishes PHYSICS TODAY, the most closely followed magazine of the physical sciences community, and is also home to the Society of Physics Students and the Niels Bohr Library and Archives. AIP owns AIP Publishing LLC, a scholarly publisher in the physical and related sciences.

Board of Directors: Louis J. Lanzerotti (Chair), Robert G. W. Brown (CEO), Judith L. Flippen-Anderson (Corporate Secretary), J. Daniel Bourland, Charles Carter, Beth Cunningham, Robert Doering, Judy Dubno, Michael D. Duncan, David Ernst, Kate Kirby, Rudolf Ludeke, Kevin B. Marvel, Faith Morrison, Dian Seidel.



qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_____________

________________

_________ _______________

________________

___


http://www.amd.com/en-us/press-releases/Pages/developer-language-2014aug26.aspx

https://ampfft.codeplex.com

http://www.ctcms.nist.gov/~rdm/spec3.html

http://www.ctcms.nist.gov/~rdm/std4/spec4.html



http://micromagnum.informatik.uni-hamburg.de





SECTION TITLEEditors: Konrad Hinsen, [email protected] | Konstantin Läufer, [email protected]

SOFTWARE ENGINEERING TRACK


OpenPNM: A Pore Network Modeling Package

Jeff Gostick and Mahmoudreza Aghighi | McGill UniversityJames Hinebaugh | University of TorontoTom Tranter | University of LeedsMichael A. Hoeh | Forschungszentrum JülichHarold Day and Brennan Spellacy | McGill UniversityMostafa H. Sharqawy | University of Toronto and University of GuelphAimy Bazylak | University of TorontoAlan Burns | University of LeedsWerner Lehnert | Forschungszentrum Jülich and RWTH AachenAndreas Putz | Automotive Fuel Cell Cooperation

Pore network modeling (PNM) is a well-established and long-standing approach for simulating transport in porous materials.1–3 PNMs represent an alterna-tive to the more widely used continuum modeling

in which a porous material is treated as a volume-averaged continuum without resolving microscale features. Continu-um models are mathematically rigorous, but they have some

practical limitations. First, each phenomenon being modeled requires experimentally measured constitutive relationships that describe the media’s macroscopic transport properties, such as the permeability coefficient or effective diffusivity. These can be challenging to measure, especially for multi-phase flow conditions. Second, treating the porous medium as a volume-averaged continuum means that discrete pore-

Editors: Jeffrey Carver, [email protected], and Damian Rouson, [email protected]

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



___________________








scale phenomena and events aren’t resolved, so only the average amount of a fluid phase in each compu-tational node is known. Moreover, the distribution of phases within the continuum isn’t often well pre-dicted by such models, which rely on simple exten-sions to Darcy’s law for multiphase flow, rather than more rigorous formulations.4 Finally, it sometimes isn’t appropriate to invoke the volume-averaged ap-proximation, particularly in thin materials such as membranes and electrodes.5

PNMs solve these issues but sacrifice some mathematical rigor. Instead of solving nth-order par-tial differential equations (PDEs), the pore space is treated as a network of “pipes.” Transport inside the network is modeled using finite difference schemes to solve 1D analytical solutions of the relevant trans-port equations. Despite this simplification, PNMs successfully and efficiently predict numerous aspects of multiphase transport.6 The sizes and connectiv-ity of the pores and throats are chosen to match the known physical structure, meaning that the inter-play between the structure and flow characteristics is included. Structural properties of the porous ma-terial can be readily obtained from various imaging techniques7,8 or computer-generated structures.9–11

In the most basic case, pore and throat sizes are ad-justed arbitrarily to allow the model to reproduce known experimental properties.12 PNMs are natu-rally geared toward percolation calculations,13 so they simulate realistic fluid invasion processes with computational ease.14 This allows for truly pore-scale descriptions of the fluid distribution within the media, which has major impacts on almost all other transport processes. By simply setting pores and throats filled with one phase as closed to other phases, PNMs can predict constitutive relationships for experimentally inaccessible multiphase param-eters.15–17 Extensive reviews of pore network mod-eling and comparisons between the approaches are available in the literature.18–21

This article provides an overview of the OpenPNM package for pore network simulations. Currently, the only commercially viable software product is PoreXpert, a program arising from a research group at Plymouth University (formerly known as Pore-Cor22), and groups sometimes pub-lish overviews of their in-house code.23 To the best of our knowledge, no other projects are under way to produce an open source PNM framework, in stark contrast to the field of computational fluid dynamics, where a multitude of powerful commer-cial and open source options24,25 exist. Generally, researchers in the PNM community develop their

own code to be used internally by their research group. This requires considerable resources and investment, and existing code might not be opti-mized for speed, modularity, extensibility, or main-tainability; it’s almost never well documented for future users. These problems are all too common, which were key motivators behind OpenPNM’s development. The aim is to provide the porous media community with a general, powerful, and flexible framework for tackling all manner of PNM problems from the same code base. It is hoped this will enable the sharing of code between research-ers, provide a common baseline for comparing models, and allow us all to build on the work of others, thereby dramatically increasing the pace of research and discovery.

OpenPNM was designed with three overall ob-jectives in mind: accessibility to a wide audience, generality to as many applications as possible, and extensibility to simulate any type of physical process. The first and foremost principle is to ensure that the code is accessible, in both the conceptual and physi-cal sense. OpenPNM was coded in Python, which is a powerful, easy-to-use, and free object-oriented programming (OOP) language that’s increasingly used in engineering calculations.26,27 Great effort was expended creating detailed documentation, and the OpenPNM website (openpnm.org) has a continuously growing list of tutorials and examples. The code is hosted publicly and is completely open source and free. It’s registered with the Python pack-age index (PyPI) so it can be installed with a simple pip install openpnm command. OpenPNM re-quires no compilation of source code during instal-lation (it’s written entirely in Python, which is an interpreted scripting language). OpenPNM main-tains good computational performance by relying heavily on NumPy28 and SciPy (www.scipy.org) to perform numerical operations, both of which em-ploy precompiled C code for speed. Another critical aspect of accessibility is to ensure that the code is well tested. At the time of this writing, 85 percent of the 7,000 lines of code in the package are tested regularly via an extensive suite of unit and integra-tion tests. The second overall principle is generality, so the package is fully agnostic to network topology, shape, and dimension, allowing traditional cubic lattices and fully random networks to be treated identically. This is accomplished using approaches borrowed from graph theory such as adjacency and incidence matrices to store network topology and ar-chitecture. OpenPNM includes a large suite of tools for working with network topology, such as find-

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



________

_________


http://www.openpnm.org

http://www.scipy.org






62 July/August 2016

ing pores connected to given pores, and for label-ing pores and throats for quick access later. Finally, the package is designed to be easily customized by researchers wishing to apply their own pore-scale models, for example to calculate pore-wall surface areas in some special way. Many common pore-scale models are included with the package, but extensi-bility and customization were a primary consider-ation. Adding custom models is straightforward and requires only basic function definitions that con-tain blocks of procedural-style code, along with an understanding of how OpenPNM stores data.

This article doesn’t provide exhaustive details about each method or function; the code itself is heavily documented in that regard, so the interested reader is directed there for more details. We highly recommend the use of an integrated development environment (IDE) that supports autocomplete and provides an object inspection pane to render context-aware help files for each method (such as Spyder). Some familiarity with Python and a basic knowledge of OOP would also be helpful, such as knowing definitions of a method and class.

Data Storage

OpenPNM stores all data, such as pore diameters, in NumPy ndarrays,28 which have become the de facto standard numerical array data type in Python. They support slicing, fancy indexing, broadcasting, vec-torization, and all the typically expected array opera-tions. This approach was chosen over a more object-oriented option such as that used by NetworkX29

because these operations are very fast when vector-ized. OpenPNM also relies heavily on SciPy, which is designed to work specifically on NumPy arrays.

Pore and Throat Properties

One of OpenPNM’s main design considerations was to accommodate all networks of arbitrary di-mensionality, connectivity, shape, and so on. To ac-complish this, OpenPNM stores all pore and throat data in the most generic way possible: as lists (arrays) of either NP or NT length, corresponding to the number of pores and throats in the network, respec-tively. This means that each pore (or throat) has an index, and all properties for that pore (or throat) are stored in the array element corresponding to that in-dex. Thus, the diameter for pore 15 is stored in the ‘pore.diameter’ array in element 15, and the length of throat 32 is stored in the ‘throat.length’ array at element 32. All of the property arrays are stored in a dictionary, which is similar to a structured variable or struct in other programming languages. This

allows each property array to be accessed by its name or key, with a syntax such as net[‘pore.diameter’]or net[‘throat.length’], where net is the name of the dictionary object.

Several rules have been imposed to control data integrity. First, all names must begin with ei-ther pore or throat, which serves to identify the type of information stored in the array. Second, for the sake of consistency, only arrays of length NP or NT are allowed in the dictionary. Any scalar values written to the dictionaries are cast into full-length vectors, effectively applying the scalar value to all network locations. This simplifies subsequent nu-merical steps because all arrays are of equal and known length. The drawback of this approach is that storing data isn’t as memory-efficient as possible, but this isn’t as important on modern computers, which typically have several Gbytes of RAM. In addition, there’s no limitation on the size of other dimensions, meaning that NP × 3 or NT × 2 arrays are allowed, such as the ‘pore.coords’ array, which stores each pore’s [X,Y,Z] coordinates in an NP× 3 list.

There are also no limitations on the data type that can be stored, but OpenPNM does distin-guish between Boolean arrays and all other ar-ray types. The Boolean arrays are treated as la-bels, while all other arrays are assumed to contain numerical data describing pore or throat proper-ties. Labels enable easy retrieval of a list of im-portant pores (or throats), such as all pores on the top face of the network. Several labels are added to the Network object during the genera-tion steps, but users can apply their own labels as needed. For instance, you might perform a complex filtering to find all pores within a cer-tain distance of some location that also pos-sess a volume within some range. To avoid re-peating this query, simply apply a label to the pores. Labels are applied by adding a new array to the dictionary with the label name, contain-ing True values for the pore (or throat) loca-tions where the label applies. Thus, the array net[‘pore.top’] is set to True for every pore at the top of the Network object and False elsewhere.

Network Topology

The only topology definitions required by OpenPNM are that each throat connects exactly two pores, no more and no less, and that throats are nondirectional, meaning that transport in either direction is equal. Other general but nonessential rules are that pores can have an arbitrary number of throats (even zero, al-though pores with zero throats lead to singular matrices

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








and other problems and thus should be avoided) and two pores should be connected by no more than one throat, unless there’s some real physical reason for this, since unintentional duplicate connections impact the rate of species exchange between pores.

One of the challenges when storing networks in list-based arrays is tracking the topology of the pore and throat connections. An important prop-erty of a throat is which pores are found on either end, so that Network connectivity can be stored as a list of throat properties equivalently to oth-er physical throat properties in an NT × 2 list of [pore I, pore J] pairs. This storage scheme hap-pens to define an adjacency matrix in the sparse storage scheme known as IJV (or COO in the scipy.sparse module), a commonly used means of representing topology in graph theory. This is an NP × NP array with nonzero values, V, at locations (I, J) indicating that pores I and J are connected. It’s symmetrical since the throats are bidirectional and highly sparse because a given pore only connects with a small subset of nearby pores in the network. Figure 1 shows a simple Network topology along with its

corresponding adjacency matrix and its IJV (or COO) representation. Also shown is the incidence matrix for the same topology. Adjacency and incidence matrices are theoretically equivalent means of representing to-pology; both can be represented in IJV format, but each has different practical advantages.

OpenPNM Network objects include numerous methods for querying the topology, such as find-ing pores connected to a given throat (find_con-nected_pores) or finding the throats neighboring certain pores (find_neighbor_throats). Each of these queries is performed by inspecting the adja-cency or incidence matrices. For instance, to find all pores that are direct neighbors to pore 5 requires finding which columns on row 5 of the adjacency matrix contain nonzeros. Alternatively, to find all throats that are directly connected to pore 5, it’s easier to find all nonzero entries on row 5 of the in-cidence matrix. Both of these operations are most efficiently performed on sparse matrices stored in the list-of-lists (LIL) format, so OpenPNM stores copies of both in a private location for use in the event of such queries.

Figure 1. Schematic of random network architecture with pore (node) and throat (bond) numbers labeled. Also shown are the adjacency

matrix (bottom left) and incidence matrix (bottom middle) representation of this network and the IJV sparse representation of the

adjacency matrix (bottom right).

1

2 3

4

5

1 2

3

46

7

1 2 3 4 5

1

2

3

4

5

0 1 1 1 0

1 0 1 0 0

1 1 0 1 1

1 0 1 0 1

0 0 1 1 0

1 2 3 4 5 6 7

1

2

3

4

5

1 1 1 0 0 0 0

1 0 0 1 0 0 0

0 1 0 1 1 1 0

0 0 1 0 1 0 1

0 0 0 0 0 1 1

i j

1

2

3

4

5

6

7

1 2

1 3

1 4

2 3

3 4

3 5

4 5

5

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









64 July/August 2016

Implementation

OpenPNM has five main objects: Network, Geome-try, Physics, Phase, and Algorithm. Each of these inherits from the Core class, plus has some additional methods or functionality added for its specific role.

The Core Objects

The Core objects in OpenPNM contain data and are used to perform calculations. Each Core object is a subclassed version of Python’s dict or diction-ary, with several additional methods added that are specific to handling OpenPNM’s data. The main role of the Core class is to manage the data stored in the dictionary. This means storing and tracking label and property arrays, implementing the data integrity rules mentioned above, returning lists of pores and throats based on some combination of labels, and so forth.

Network. A Network object represents a fully self-contained topological entity, meaning that when two separate Network objects are created, they don’t interact with each other. If two separate Net-works need to exchange information, then they must be stitched together to form a single Network,using the provided topology manipulation tools. At minimum, a Network needs pore coordinates and the throat connections to define its topology.

The GenericNetwork class has a number of ad-ditional methods added for performing topological queries, such as finding the pores directly connect-ed to a given pore (find_neighbor_pores), finding the throats that connect given pairs of pores (find_connecting_throat), and many others.

The GenericNetwork class itself isn’t responsible for creating network topologies. For this, there are several subclasses of the GenericNetwork class, such as Cubic, which creates the standard lattice with speci-fied connectivity patterns (6, 8, 26, and so on), and Delaunay which uses a Delaunay tessellation to con-nect random points in space. In addition to generating topologies, it’s also possible to import networks from external sources, and several formats are supported.

Geometry. Geometry objects track and manage the physical properties and dimensions of pores and throats. OpenPNM was designed to allow net-works to include multiple regions with differing properties for modeling multilayered, stratified, or generally heterogeneous media. In these cases, multiple Geometry objects can be created and as-signed to different subsets of pores and throats. The difference between separate Geometry objects lies

in the unique set of pore-scale models that were ap-plied to calculate the geometrical properties.

Several subclassed versions of GenericGeom-etry are included in the package for convenience, including the standard Stick_and_Ball, as well as Voronoi, which is combined with the DelaunayNetwork class to model fibrous materials.30,31 Most frequently, however, users define their own custom Geometry classes, which consist of an assortment of pore-scale models with suitable parameters.

Phase. Phase objects manage the thermophysical properties of the solids, liquids, and gases that exist in the Network. Because fluids can move around during the course of a simulation, via invasion percolation for instance, a Phase object is defined everywhere in the Network, and the actual presence of a phase in a given location is tracked using an occupancy list, a number between 0 and 1 that indicates fractional filling.

Because thermophysical properties are gener-ally dependent on each other (viscosity is a func-tion of temperature), consideration was made to re-generate property values as conditions change. For example, if the temperature of a Phase changes, all temperature-dependent properties can be recal-culated by calling the regenerate method of the Phase object.

OpenPNM includes predefined Phase sub-classes for Air, Water, and Mercury. Creating new fluids requires only defining a new subclass of GenericPhase and assigning the suitable models, either from the Phase models library or custom written models, with the appropriate parameters.

Physics. The combination of pore-scale geometry and thermophysical properties are what dictate the actual transport parameters in the pore network. For instance, fluid viscosity and throat diameter are both required to calculate the hydraulic con-ductance according to the Hagen-Poiseuille model. When a Physics object is created, it must be told which Phase and Geometry objects it applies to. This allows the models attached to a Physics ob-ject to find the necessary thermophysical and geo-metrical properties.

The only subclass included with the pack-age is Standard, which contains an assortment of commonly used pore-scale physics models such as the Hagen-Poiseuille model for hydraulic conduc-tance and the Washburn equation for capillary en-try pressure. Creating custom pore-scale Physicsmodels is one of the key ways PNMs differentiate themselves, so creating and adding custom models

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








was designed to be as flexible as possible. A custom Physics class can be created by choosing or coding the necessary pore-scale models, then assigning them to a GenericPhysics object.

Algorithm. Algorithm objects also derive from the Core class, and they too store their own data, which is typically the result of some algorithm or calculation. Algorithms have numerous additional methods beyond those supplied by Core, and these methods can be quite complex, depending on the Algorithm.

The results of any calculation are stored on the Algorithm object to prevent overwriting or inter-fering with data on other objects. For instance, when a diffusion calculation determines the con-centration of a species in each pore, the resulting array is stored under ‘pore.mole_fraction’ on the Algorithm, even though mole fraction is techni-cally a Phase property. This stored data can be used in subsequent calculations, but it must be explicitly transferred to the new Algorithm object.

Object Relationships

A given simulation consists of only one Network ob-ject and one or more each of Geometry, Phase, and Physics objects. It’s useful to think about these vari-ous objects in terms of layers that stack together for a typical simulation. The Network contains all the pores and throats for a particular simulation and forms the base layer. One of the main features of OpenPNM is the ability to model heterogeneous materials, for in-stance, where different pore size distributions are to be applied to different regions. This is accomplished by defining multiple Geometry objects that each apply to a separate group of pores (or throats) as shown in Figure 2. Spatial overlap of Geometry objects is forbid-den because it creates a conflict over which Geometryobject should calculate and store information for those locations. Next, a layer of two Phase objects is added to the stack to allow multiphase simulations. Phases span all pores and throats but are immiscible and so are represented side by side. In fact, multiple Phases can exist in a given pore simultaneously. Then, a layer of four Physics objects are added between the

Figure 2. Schematic representation of Core object relationships. (a) Overlap in the vertical direction indicates that objects are associated with the

same pores (or throats), while overlap in the horizontal direction indicates with which Phases each object is associated. Phases by definition don’t

overlap with each other but span all pores and throats, Geometry objects span all Phases but only a limited set of pores and throats, and Physics

objects exist at the intersection of Phase and Geometry objects. (b) The objects form a conceptual stack with each object representing a layer,

creating the physical interpretation of overlaps shown in (a). (c) The layers that are physically near each other in the stack can exchange data (read-

only) which offers a convenient way to retrieve a complete list of data that may be dispersed across several objects (Geometry 1 and Geometry 2).

Spa

tial

ove

rlap

Spa

tial

ove

rlap

Phase overlap Phase overlap

Pore i

Pore jPore k

Pore mPore n

Pore p

Throat i

Throat j

Throat m

Throat p

Pore i

Pore jPore k

Pore mPore n

Pore p

Throat i

Throat j

Throat m

Throat p

Pore i

Pore jPore k

Pore mPore n

Pore p

Throat i

Throat j

Throat m

Throat p

Pore i

Pore jPore k

Pore mPore n

Pore p

Throat i

Throat j

Throat m

Throat p

Physics 2 Physics 4

Physics 1 Physics 3Geometry 1

Geometry 2

(a) (c)

(b)

Network

Spa

tial

ove

rlap

Physics 2 Physics 4

Geometry 1

Geometry 2eoeoPheom

hyoho etry 2

Psics 2yet y

4s 4cscPhysi s 4Pe

yse

Phase overlap

Physics 1 Physics 3Phase 1 Phase 2

Phase 1 Phase 2Network

AlgorAlgorAlgorithithithiii +hm 11+hm 11+hm 11hhh

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









66 July/August 2016

Geometry and Phase layers. Physics objects combine geometrical information about pores (or throats) with the properties of the fluids in that pore (or throat). Thus, Physics objects require information from one Geometry and one Phase object, and therefore exist at each intersection of Geometry and Phase objects as shown in Figure 2b. Put another way, a Physicsobject only applies to one phase because the thermo-physical properties of the Phases are different and change the behavior of pore-scale Physics. Moreover, a different Physics is required for each Geometrysince different geometrical properties may result in different pore-scale behavior.

Once a simulation has been set up as shown in Figure 2b, it’s ready for calculations. An arbitrary number of Algorithm objects can be created and add-ed as layers to the stack, with each Algorithm looking up the information it requires from other objects in the simulation and producing results, such as chang-ing the occupancy of the phases in various pores due to percolation, or calculating mass fractions.

One drawback of having multiple Geometry and Physics objects for different regions of the Networkis that a single list containing all property values for the entire network isn’t readily available. OpenPNM addresses this issue by allowing Network and Phaseobjects (which by definition encompass all pores and throats) to retrieve and combine data from the Ge-ometry and Physics objects, respectively. The data exchange between the various layers is indicated by the arrows in Figure 2c, conveying that associated objects are able to read data from each other. Writ-ing data between layers isn’t allowed.

Pore-Scale Models

Pore-scale models are the most important aspect of OpenPNM as they elevate the basic topological graph to the level of a pore network model by giv-ing physical meaning to the pores and throats. Mod-

els are also the main way that users can customize OpenPNM to suit their particular scientific endeav-ors. Before delving into the code’s relevant machin-ery, it’s better to first discuss the meaning of models.

The main difference between PNMs and con-tinuum models is how they treat the transport properties between two physical locations in the do-main. For a specific example, consider viscous flow. In continuum modeling, the flow rate or pressure drop between two neighboring locations is dictated by the medium’s permeability coefficient, which is typically measured in the lab on a sample of representative material. In a PNM, the two neigh-boring locations are treated as actual pores and con-nected by a throat. The flow rate between these two pores is treated as flow through a pipe, which can be described by any number of analytical solutions depending on the geometry assigned to the pipe.32

One typical approach is to use the Hagen-Poiseuille equation for single phase flow in a cylindrical tube:

π

μ=

−−

−

qR P P

L( )

8,i j i j

i j

4

(1)

where Pi and Pj are the pressures in pores i and j, Li–jand Ri–j are the length and radius of the throat (pipe) connecting pores i and j, and μ is the fluid viscos-ity. Figure 3 shows the dimensions and geometry of this pore-throat-pore conduit. If pressure loss in each half-pore is neglected for simplicity, then the total flow rate given by Equation 1 can be generalized as

Q = gi–j(Pi – Pj), (2)

where gi–j is the conduit’s hydraulic conductance. Therefore, a hydraulic conductance model

would return values of g given throat radii R and length L from the Geometry object and viscosity μfrom the Phase object. Using the vectorization capa-bilities of NumPy, this can be done in a few lines:

def hydraulic_conductance(geometry, physics):

mu = physics.interpolate_

data(physics[‘pore.viscosity’])

L = geometry[‘throat.length’]

R = geometry[‘throat.radius’]

g = 3.14159*(R**4(/(8*mu*L)

return g

A few key points are illustrated here. First, viscosity is a Phase property, yet it’s accessed via physics. This utilizes the data exchange rules outlined earlier. Sec-

Figure 3. Schematic diagram showing the definitions and

dimension of a typical pore-throat-pore conduit.

r1

r2

LT

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








ond, like most Phase properties, viscosity is defined in pores so it must be interpolated to find throat values. Third, the array g is NT long because it’s the result of element-wise operations between ‘throat.radius’ and ‘throat.length’. Finally, the pore and throat properties used weren’t passed in as hardcoded numerical values; instead, the objects containing the values were passed, and the values were retrieved “on demand.” This means that if the viscosity is changed on the Phase object, then rerunning the above code will automatically utilize the updated values without any effort on the user’s part. This is the mechanism by which changing conditions are transmitted to all other dependent properties.

The above code snippet is a simple but represen-tative example of a pore-scale model. It’s expected that users will devise their own such models of ar-bitrary complexity. To utilize any custom-made pore-scale models, a user only needs to create a file in the working directory (such as “my_models.py”), populate it with his or her own function definitions, and import the file by entering import my_models.This will provide access to all models in the file with my_models.hydraulic_conductance.

Assigning Models to Core Objects

To ensure that physical properties can be recalculated as needed, it’s necessary to save the pore-scale model and all parameters in memory. To accomplish this, every Core object has an attribute (that is, physics.models) that actually contains a nested dictionary called the ModelsDict as shown in Figure 4.

ModelsDict has an add method that performs the service of associating the model with the Core object. The arguments required by the add method are the name of the pore or throat property (propname) where the values generated by the model are to be stored (‘throat.hydraulic_conductance’), a handle to the model to be used (my_models.hydraulic_conduc-tance), and any parameters required by the specific model. The add method stores all the received param-eters in the ModelsDict using the given propname as a key. Also shown in Figure 4 is the ModelWrapperobject, which houses each model’s specific arguments.

A few nuances must be considered when dealing with models. First, models are stored in ModelsDictunder a specified propname; by default, the values pro-duced when the model is run will be stored in the par-ent Core object dictionary under the same propname.Second, the numerical values produced by a model remain in the Core object dictionary as constant values until regenerate is called to rerun each model.

Algorithms

Numerous key algorithms are included with OpenPNM, including various percolation algorithms (Drainage and InvasionPercolation), and several linear transport models such as FickianDiffusion, Stokes-Flow, OhmicConduction, and FourierConduction.

Each algorithm is slightly different, but in general, Algorithms are instantiated by passing in a Networkobject and some additional arguments. Algorithmobjects have a setup method that allows specifying

Figure 4. Schematic diagram showing how pore-scale models are associated with Core objects. When the regenerate method of the

ModelsDict is called, it calls the run method of each ModelWrapper in the order in which they’re stored. The values returned from the

model are placed into the Core object’s dictionary under the same property name as that of the stored model.

Core object

ModelsDict object

method i

method ii

attribute i

attribute ii

‘pore.prop i’

‘pore.prop ii’

‘throat.prop i’

‘throat.prop ii’

Met

hods

and

attr

ibut

es Dic

tiona

ryda

ta

attribute i

add

remove

regenerate

reorder

Met

hods

and

attr

ibut

es

‘pore.prop ii’

‘throat.prop i’

‘throat.prop ii’Dic

tiona

ryda

ta

ModelWrapper object

method iii

regenerate

method ii

attribute i

attribute ii

Met

hods

and

attr

ibut

es

‘model’

‘argument i’

‘argument ii’

‘argument iii’Dic

tiona

ryda

ta

models

‘pore.prop i’

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









68 July/August 2016

various parameters required by the algorithm. The calculation is executed by calling the run method. Finally, the results are stored on the Algorithm object but can be transferred to other Core objects (usually a Phase) for use in further calculations.

Percolation and Invasion

Performing multiphase transport calculations in the pore network is a central role of PNMs. Realistic pore-scale fluid configurations are simulated using percola-tion theory to determine how an invading phase will displace a defending phase. OpenPNM contains sev-eral algorithms for performing such calculations.

Drainage. In a porosimetry experiment,33 the vol-ume of a non-wetting fluid injected into a specimen is tracked at discrete pressure steps. The process is referred to as drainage and is mathematically simu-lated as an access-limited bond percolation prob-lem. Access limitations are important because the invading phase can only invade throats that are ac-cessible from the sample surface and subsequently those connected directly to the reservoir of the in-vading phase. Simulating porosimetry experiments is an essential part of PNMs because the results can be compared to experimental data to verify that correct pore and throat size distributions have been used when combined with other information such as the permeability and porosity of the material.

To conduct this simulation in OpenPNM, throats must first be assigned capillary entry pres-sures, indicating the pressure that must be applied to the invading phase for it to enter that throat. Re-lating throat invasion pressure to geometric throat properties is almost universally done with the Wash-burn equation, but other, more suitable options are available.34 The Drainage algorithm uses the connected_components method included in the csgraph module of scipy.sparse to perform a standard graph theory clustering operation over the Network.

Invasion percolation. Invasion percolation differs from drainage in subtle but important ways. In drainage, all accessible throats with entry pressures lower than a certain value are simultaneously invaded (along with their neighboring pores). Invasion percolation applies a similar logic on the scale of single throats by invading only the single most easily invaded ac-cessible throat and its neighboring pore on each step. In physical terms, this is equivalent to a quasi-static, rate-controlled injection experiment.35

During an invasion percolation simulation, when a pore is invaded, new throats become ac-

cessible and join the invasion front. The algorithm must choose the throat with the lowest entry pres-sure for the next invasion, requiring a continually maintained, dynamically changing, and sorted list of throat entry pressures.36 A standard graph theory algorithm for this process isn’t available in SciPy, so a basic algorithm was implemented in OpenPNM using Python’s built-in heapq module, which is based on priority queues using a heap data structure. In simple terms, this means that a list of initially accessible throats is sorted into a heap, which has the property that the smallest value is al-ways found in element 0. This allows instant access to the next throat that should be invaded. The pore attached to this throat is then invaded, and all of its throats are added to the heap, and the procedure is repeated until all pores and throats are invaded.

Resistor Network Calculations

One of the main uses of pore network models is to simulate transport phenomena through the pore space, usually in the presence of a second phase. A typical example is the diffusion of a gaseous species through a pore space that’s partially filled with a liquid. To model a domain of any useful size, it’s difficult to model such a scenario by discretizing the pore space as a finite-element mesh. First, it would take many nodes to model even a few pores. Second, the placement of liquid is a complex task, requiring the solution of high-order PDEs37 or a Lattice-Boltzmann approach,38 both of which are highly computationally intensive. An alternative to these methods is the Full Morphology approach based on image analysis,39 which can analyze a 2D or 3D pixel image of a porous material and place phases using structuring elements. However, the user is still reliant on other methods to solve the equations of fluid flow and also limited to much smaller domains.

Pore network modeling starts by recognizing that, in many applications, an approximate model of a suitably large domain is more useful than a highly rigorous model of a limited number of pores. With this in mind, each throat represents a resistor in the network through which the species of inter-est must travel to reach the neighboring pore. The resistance to diffusion or flow offered by a throat constriction is a function of its geometry, as well as the fluid properties such as viscosity or diffusion coefficient. In PNMs, this resistance is described by an appropriate pore-scale physics model, such as the Hagen-Poiseuille model given in Equation 1. This and other pore-scale physics models are cast in

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








terms of a resistance by analogy with Ohm’s law. In the case of Equation 1, pressure (P) is the driving force, μR4/8L represents the conductance to flow due to the geometrical properties of the throat, μprovides the resistance to flow caused by the flow-ing fluid’s viscosity, and Q is the volumetric flow of fluid analogous to current in Ohm’s law. This can be recast as

π

μ= − = −−

−−Q

RL

P P g X X8

( ) ( ),i ji j

i j i j i j

4

(3)

where gi–j is the conductance between pores i and j, and X is the unknown to be solved for. At steady state and in the absence of any source or sink terms, the net material flow through pore i is zero, hence

∑= −−−

g X X0 ( ),i ji j

n

i j (4)

where pore i has n neighbors. Applying this equa-tion to each pore in the network results in a system of linear equations in X that can be readily solved by any matrix inversion algorithm subject to given boundary conditions. Importantly, the conduc-tance gi–j can be set to very small values for pores blocked by another phase, making it trivial to in-corporate the impact of multiple phases on trans-port processes.

Performing transport calculations in OpenPNM begins with instantiating an Algorithm object, then specifying the necessary boundary conditions us-ing the apply_boundary_conditions method of the Algorithm. The Algorithm’s run method is then called to handle the processes of building the coef-ficient matrix, applying specified boundary condi-tions, and calling the matrix inversion routine. The main requirement from the user is to ensure that the conductance values for each throat are calculated correctly, meaning that the appropriate models for geometrical sizes, thermophysical properties, and pore-scale physics have been applied.

Application and Demonstration

The intense development of OpenPNM over the past few years has resulted in a concise and power-ful framework that can perform significant compu-tations in minimal lines of code. This final section will describe the steps required to calculate the ef-fective diffusivity of a porous material as a function of liquid water saturation, a typical application of PNMs.16 The full script is given in the sidebar.

The first step is to create a Network object, in this case with 3,125 pores on a cubic grid with a spacing of 100 μm between them:

pn = OpenPNM.Network.Cubic(shape=[25, 25,

5], spacing=0.0001)

Next, a Geometry object must be instantiated:

geo = OpenPNM.Geometry.

GenericGeometry(network=pn,

pores=pn.pores(),

throats=pn.throats())

In this step, the Geometry object is associated with the Network (pn) and assigned to all the pores and throats. The GenericGeometry class has no predefined pore-scale models, so these must be add-ed. The script in the sidebar illustrates how to add geometrical properties such as ‘pore.diameter’and ‘throat.length’ models to the geo object, as well as direct assignment of calculated values such as ‘pore.volume’. Some subclasses included with OpenPNM have predefined pore-scale geometry models; these can be used as templates for users to create one for their own specific materials.

Next, Phases are added using the predefined subclasses for Air and Water:

air = OpenPNM.Phases.Air(network=pn)

water = OpenPNM.Phases.Water(network=pn)

Only the Network object with which these Phaseobjects are to be associated is required as an argument.

This simulation will require first invading liquid water into the network, then diffusing gas through dry void space. The Phases will each require their own Physics objects:

phys_air = OpenPNM.Physics.

GenericPhysics(network=pn,

phase=air,

geometry=geo)

phys_water = OpenPNM.Physics.

GenericPhysics(network=pn,

phase=water,

geometry=geo)

Physics objects require the Network, a Phase,and a Geometry object as arguments. Both the Phys-ics objects above operate on the same pores and throats (defined by the geo object), but each apply to a different Phase. Both of the above objects are instances of the GenericPhysics class, which has no pore-scale models. The sidebar illustrates how to add ‘throat.capillary_pressure’ to the water

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









70 July/August 2016

Sample Script

The following script shows how to set up a Network, define a Geometry by adding the necessary models, create two

predefined Phase objects, define the necessary Physics objects, and finally perform sequential simulations using two

Algorithm objects:import OpenPNMimport scipy as spimport OpenPNM.Geometry.models as gmimport OpenPNM.Physics.models as pmpn = OpenPNM.Network.Cubic(shape=[10, 10, 10], spacing=0.0001)geom = OpenPNM.Geometry.GenericGeometry(network=pn, pores=pn.pores(), throats=pn.throats())geom[‘pore.seed’] = sp.rand(geom.num_pores())geom.models.add(propname=’pore.diameter’, model=gm.pore_diameter.weibull, shape=2.77, loc=6.9e-7, scale=9.8e-6)geom.models.add(propname=’throat.diameter’, model=gm.throat_misc.minpore, pore_prop=’pore.diameter’)geom.models.add(propname=’throat.length’, model=gm.throat_length.straight)geom[‘pore.area’] = 3.14159/4*geom[‘pore.diameter’]**2geom[‘pore.volume’] = 4/3*3.14159*(geom[‘pore.diameter’]/2)**3geom[‘throat.area’] = 3.14159/4*geom[‘throat.diameter’]**2geom[‘throat.volume’] = 3.14159/4*geom[‘throat.diameter’]**2* geom[‘throat.length’]air = OpenPNM.Phases.Air(network=pn)water = OpenPNM.Phases.Water(network=pn)water[‘pore.contact_angle’] = 110.0water[‘pore.surface_tension’] = 0.072phys_air = OpenPNM.Physics.GenericPhysics(network=pn, phase=air, geometry=geom)phys_water = OpenPNM.Physics.GenericPhysics(network=pn, phase=water, geometry=geom)phys_air.models.add(propname=’throat.diffusive_conductance’, model=pm.diffusive_conductance.bulk_diffusion)phys_water.models.add(propname=’throat.capillary_pressure’, model=pm.capillary_pressure.washburn)OP = OpenPNM.Algorithms.OrdinaryPercolation(network=pn, invading_phase=water)OP.run(inlets=pn.pores(‘bottom’))phys_air[‘throat.conductance’] = phys_air[‘throat.diffusive_conductance’]*(OP[‘throat.inv_Pc’] > 8000)FD = OpenPNM.Algorithms.FickianDiffusion(network=pn, phase=air)FD.set_boundary_conditions(pores=pn.pores(‘top’), bctype=’Dirichlet’, bcvalue=0.5)FD.set_boundary_conditions(pores=pn.pores(‘bottom’), bctype=’Dirichlet’, bcvalue=0.1)FD.run(conductance=’throat.conductance’)

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








phase and ‘throat.diffusive_conductance’ to the gas phase.

Finally, two Algorithm objects are required to perform the water invasion and gas diffusion simulations. For water invasion, drainage will be used:

D = OpenPNM.Algorithms.Drainage(network=pn)

D.setup(invading_phase=water,

defending_phase=air)

D.set_inlets(pn.pores(‘bottom’))

D.run()

The run method will perform the simula-tion and place arrays called ‘pore.inv_Pc’ and ‘throat.inv_Pc’ in D’s dictionary. These arrays contain the pressure at which each pore and throat was invaded, thus requiring a simple Boolean com-parison to find all locations invaded at some ap-plied pressure (for example, 8,000 Pa).Finally, gas diffusion is calculated using the provided FickianDiffusion Algorithm subclass:

FD = OpenPNM.Algorithms.

FickianDiffusion(network=pn, phase=air)

The Phase on which this Algorithm operates is required as an argument, as this gives the algorithm access to the diffusive conductance values stored on phys_air. Mole fraction boundary conditions are set on the top and bottom of the Network:

FD.set_boundary_conditions(bctype=’Dirichlet’,

bcvalue=0.5, pores=pn.pores(‘top’))

FD.set_boundary_conditions(bctype=’Dirichlet’,

bcvalue=0.1, pores=pn.pores(‘bottom’))

The transport calculation is executed by calling the run command:

FD.run()

Figure 5 shows the resulting Network, water con-figuration (blue pores), and mass fraction distribution (colored pores). Each specific transport phenomena subclass has a method for calculating the effective value of its transport property for the entire Network;for FickianDiffusion, this is calc_effective_diffu-sivity, which in this case is approximately half the bulk value and 10 percent less than the dry Network.

Open PNM aims to provide users with an easy-to-use, computationally efficient and fully

customizable framework for performing PNM cal-culations of all sorts. We offer this package to the porous media community in the hope that it will become a standard tool in the field, allowing re-searchers to share code, build on each other’s work, and compare results directly.

Acknowledgments

OpenPNM was made possible by the support of the Automotive Fuel Cell Cooperation and the Natural Science and Engineering Research Council of Can-ada (NSERC) through the Collaborative Research & Development and the Discovery Grant programs. Fi-nancial support from the NSERC Canada Research Chairs Program, NSERC Collaborative Research and

Figure 5. 3D rendering of a pore network showing water invading in from bottom (blue) and gas diffusion from top to bottom (yellow to

red). The sizes of the spheres are proportional to their diameter. Throats are drawn as thin lines to enhance visualization.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM









72 July/August 2016

Training Experience Program (CREATE) in Distrib-uted Generation for Remote Communities (DGRC), Canadian Foundation for Innovation (CFI), and Ontario Ministry of Research and Innovation Early Researcher Award are also gratefully acknowledged. European collaborators acknowledge the support and funding of the Engineering and Physical Sciences Re-search Council (EPSRC).

References

1. I. Chatzis and F.A.L. Dullien, “The Modeling of Mercury Porosimetry and the Relative Permeability of Mercury in Sandstones Using Percolation Theory,” Int’ l J. Chemical Eng., vol. 25, no. 1, 1985.

2. R.G. Larson, L.E. Scriven, and H.T. Davis, “Percolation Theory of Two Phase Flow in Porous Media,” Chemical Eng. Science, vol. 36, no. 1, 1981, pp. 57–73.

3. M.M. Dias and A.C. Payatakes, “Network Models for Two-Phase Flow in Porous Media Part 1. Immiscible Microdisplacement of Non-Wetting Fluids,” J. Fluid Mechanics, vol. 164, no. 3, 1986, pp. 305–336.

4. C.T. Miller et al., “Multiphase Flow and Transport Modeling in Heterogeneous Porous Media: Challenges and Approaches,” Advances in Water Resources, vol. 21, no. 2, 1998, pp. 77–120.

5. P.A. García-Salaberri et al., “Effective Diffusivity in Partially-Saturated Carbon-Fiber Gas Diffusion Layers: Effect of Local Saturation and Application to Macroscopic Continuum Models,” J. Power Sources, vol. 296, no. 11, 2015, pp. 440–453.

6. M.J. Blunt et al., “Pore-Scale Imaging and Modelling,” Advances in Water Resources, vol. 51, no. 1, 2013, pp. 197–216.

7. H. Dong and M.J. Blunt, “Pore-Network Extraction from Micro-Computerized-Tomography Images,” Physical Rev. E, vol. 80, no. 3, 2009, article no. 036307.

8. D. Silin et al., “Microtomography and Pore-Scale Modeling of Two-Phase Fluid Distribution,” Transport in Porous Media, vol. 86, no. 2, 2010, pp. 495–515.

9. J. Hinebaugh, Z. Fishman, and A. Bazylak, “Unstructured Pore Network Modeling with Heterogeneous PEMFC GDL Porosity Distributions,” J. Electrochemical Soc., vol. 157, no. 11, 2010, pp. B1651–B1657.

10. S.L. Bryant, D.W. Mellor, and C.A. Cade, “Physically Representative Network Models of Transport in Porous Media,” AIChE J., vol. 39, no. 3, 1993, pp. 387–396.

11. R. Thiedmann et al., “Random Geometric Graphs for Modelling the Pore Space of Fibre-Based

Materials,” J. Materials Science, vol. 46, no. 24, 2011, pp. 7745–7759.

12. M.A. Ioannidis and I. Chatzis, “Network Modelling of Pore Structure and Transport Properties of Porous Media,” Chemical Eng. Science, vol. 48, no. 5, 1993, pp. 951–972.

13. A.G. Hunt, “Basic Transport Properties in Natural Porous Media: Continuum Percolation Theory and Fractal Model,” Complexity, vol. 10, no. 3, 2005, pp. 22–37.

14. M. Rebai and M. Prat, “Scale Effect and Two-Phase Flow in a Thin Hydrophobic Porous Layer: Application to Water Transport in Gas Diffusion Layers of Proton Exchange Membrane Fuel Cells,” J. Power Sources, vol. 192, no. 2, 2009, pp. 534–543.

15. M.J. Blunt et al., “Detailed Physics, Predictive Capabilities and Macroscopic Consequences for Pore-Network Models of Multiphase Flow,” Advances in Water Resources, vol. 25, nos. 8–12, 2002, pp. 1069–1089.

16. J. Gostick et al., “Pore Network Modeling of Fibrous Gas Diffusion Layers for Polymer Electrolyte Membrane Fuel Cells,” J. Power Sources,vol. 173, no. 1, 2007, pp. 277–290.

17. P.C. Reeves and M.A. Celia, “A Functional Relationship Between Capillary Pressure, Saturation, and Interfacial Area as Revealed by a Pore-Scale Network Model,” Water Resources Research, vol. 32, no. 8, 1996, pp. 2345–2358.

18. M. Sahimi, “Flow Phenomena in Rocks: From Continuum Models to Fractals, Percolation, Cellular Automata, and Simulated Annealing,” Rev. Modern Physics, vol. 65, no. 4, 1993, pp. 1393–1534.

19. M.J. Blunt, “Flow in Porous Media —Pore-Network Models and Multiphase Flow,” Current Opinion in Colloid & Interface Science, vol. 6, no. 3, 2001, pp. 197–207.

20. V. Joekar-Niasar and S.M. Hassanizadeh, “Analysis of Fundamentals of Two-Phase Flow in Porous Media Using Dynamic Pore-Network Models: A Review,” Critical Reviews in Environmental Science & Tech., vol. 42, no. 18, 2012, pp. 1895–1976.

21. M. Prat, “Recent Advances in Pore-Scale Models for Drying of Porous Media,” Chemical Eng. J., vol. 86, nos. 1–2, 2002, pp. 153–164.

22. A. Johnson et al., “An Improved Simulation of Void Structure, Water Retention and Hydraulic Conductivity in Soil with the Pore-Cor Three-Dimensional Network,” European J. Soil Science, vol. 54, no. 3, 2003, pp. 477–490.

23. A. Raoof et al., “PoreFlow: A Complex Pore-Network Model for Simulation of Reactive Transport in Variably Saturated Porous Media,”

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








Computers & Geosciences, vol. 61, 2013, pp. 160–174.

24. M. Secanell et al., “OpenFCST: An Open-Source Mathematical Modelling Software for Polymer Electrolyte Fuel Cells,” ECS Trans., vol. 64, no. 3, 2014, pp. 655–680.

25. R. Cimrman, “SfePy: Write Your Own FE Application,” ArXiv14046391 Cs, Apr. 2014; http://arxiv.org/abs/1404.6391.

26. T.E. Oliphant, “Python for Scientific Computing,” Computing in Science & Eng., vol. 9, no. 3, 2007, pp. 10–20.

27. K.J. Millman and M. Aivazis, “Python for Scientists and Engineers,” Computing in Science & Eng., vol. 13, no. 2, 2011, pp. 9–12.

28. S. van der Walt, S.C. Colbert, and G. Varoquaux, “The NumPy Array: A Structure for Efficient Numerical Computation,” Computing in Science & Eng., vol. 13, no. 2, 2011, pp. 22–30.

29. A.A. Hagberg, D.A. Schult, and P.J. Swart, “Exploring Network Structure, Dynamics, and Function Using NetworkX,” Proc. 7th Python in Science Conf., 2008, pp. 11–15.

30. J.T. Gostick, “Random Pore Network Modeling of Fibrous PEMFC Gas Diffusion Media Using Voronoi and Delaunay Tessellations,” J. Electrochemical Soc., vol. 160, no. 8, 2013, pp. F731–F743.

31. K.E. Thompson, “Pore-Scale Modeling of Fluid Transport in Disordered Fibrous Materials,” AIChE J., vol. 48, no. 7, 2002, pp. 1369–1389.

32. V. Joekar-Niasar et al., “Network Model Investigation of Interfacial Area, Capillary Pressure and Saturation Relationships in Granular Porous Media,” Water Resources Research, vol. 46, no. 6, 2010, p. W06526.

33. H. Giesche, “Mercury Porosimetry: A General (Practical) Overview,” Particle & Particle Systems Characterization, vol. 23, no. 1, 2006, pp. 9–19.

34. W.B. Lindquist, “The Geometry of Primary Drainage,” J. Colloid Interface Science, vol. 296, no. 2, 2006, pp. 655–668.

35. V. Sygouni, C.D. Tsakiroglou, and A.C. Payatakes, “Capillary Pressure Spectrometry: Toward a New Method for the Measurement of the Fractional Wettability of Porous Media,” Physics of Fluids, vol. 18, no. 5, 2006, p. 053302.

36. A.P. Sheppard et al., “Invasion Percolation: New Algorithms and Universality Classes,” J. Physics A,vol. 32, no. 49, 1999, p. L521.

37. A.Q. Raeini, M.J. Blunt, and B. Bijeljic, “Modelling Two-Phase Flow in Porous Media at the Pore Scale Using the Volume-of-Fluid Method,” J. Computational Physics, vol. 231, no. 17, 2012, pp. 5653–5668.

38. C. Pan, M. Hilpert, and C.T. Miller, “Lattice-Boltzmann Simulation of Two-Phase Flow in Porous Media,” Water Resources Research, vol. 40, no. 1, 2004, p. W01501.

39. M. Hilpert and C.T. Miller, “Pore-Morphology-Based Simulation of Drainage in Totally Wetting Porous Media,” Advances in Water Resources, vol. 24, nos. 3–4, 2001, pp. 243–255.

Jeff Gostick, corresponding author, is an assistant pro-fessor in chemical engineering at McGill University. He received a PhD in chemical engineering from the Uni-versity of Waterloo. Gostick is licensed as P.Eng., and he’s a member of the American Institute of Chemical Engineers, the Canadian Society of Chemical Engi-neers, Interpore, and the Electrochemical Society. Con-tact him at [email protected].

Mahmoudreza Aghighi is a PhD candidate working on pore network modeling at McGill University. His research focuses on modeling transport phenomena in porous media, multiphase flows, and scientific comput-ing. Aghighi received an M.Sc. in chemical engineering from Sharif University of Technology, Tehran. Contact him at [email protected].

James Hinebaugh is a postdoctoral associate at the University of Toronto. He specializes in creating pore-scale simulation environments from high-resolution computed tomography images and is a co-founder of the OpenPNM project. Hinebaugh received a PhD in mechanical engineering from the University of Toronto. Contact him at [email protected].

Tom Tranter is a PhD candidate working on modeling and measuring multiphase flow in polymer electrolyte fuel cells at the University of Leeds. His research in-terests include renewable energy, solar energy, and renewable heating. Contact him at [email protected].

Michael Hoeh is a PhD candidate working on water electrolyzers at the Institute of Energy and Climate Re-search at Forschungszentrum Jülich. He was the first re-cruited member of OpenPNM from outside the initial founding group. His research interests include renew-able energy, energy storage and energy systems. Contact him at [email protected].

Harold Day is a graduate of McGill University in me-chanical engineering, where he also received an M.Eng. in chemical engineering. Contact him at [email protected].

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____________

_________________

____________

___

___________

___________________

_______


http://arxiv.org/abs/1404.6391












74 July/August 2016

Brennan Spellacy is a graduate of at McGill University in chemical engineering. Contact him at [email protected].

Mostafa H. Sharqawy is an assistant professor of me-chanical engineering at the University of Guelph. His research centers on design, performance evaluation, and improvement of sustainable energy and water systems. Sharqawy is a licensed P.Eng in Ontario and a member of American Society of Mechanical Engineering and International Desalination Association. Contact him at [email protected].

Aimy Bazylak is an associate professor in the Depart-ment of Mechanical and Industrial Engineering at the University of Toronto. Her research focuses on transport in porous media with an emphasis on visualization via radiography and tomography. Bazylak received a PhD in mechanical engineering from the University of Victoria. She is a licensed P.Eng and holder of a Canada Research Chair in Thermofluidics for Clean Energy. Contact her at [email protected].

Alan Burns is an associate professor in the School of Chemical and Process Engineering at the University of Leeds and a senior developer of the commercial CFD code Ansys CFX. His research interests are in numerical

algorithms, mathematical models, and applications of multiphase flows. Burns received a PhD in mathemati-cal physics from the University of Durham. Contact him at [email protected].

Werner Lehnert is a professor in the Faculty of Me-chanical Engineering at RWTH Aachen University in Germany and heads the high-temperature polymer-elec-trolyte fuel cells group at the Jülich Institute of Energy and Climate Research. Lehnert received a PhD in physi-cal chemistry from the University of Düsseldorf. Con-tact him at [email protected].

Andreas Putz is a senior research scientist and heads the modeling and simulation group at Automotive Fuel Cell Cooperation. His research focuses on the numerical simulation of transport, electrochemical, and degrada-tion processes of the membrane electrode assembly for automotive fuel cell applications. Putz received a PhD in applied mathematics from the University of British Co-lumbia. Contact him at [email protected].



qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



________________

_________________

____________________________

_____________

__________












SCIENTIFIC PROGRAMMING



The Power to Create Chaos

Konrad Hinsen | Centre de Biophysique Moléculaire in Orléans

Occasionally, I hear from readers about the articles I write for the Scientific Programming department. My recent piece on technical debt1 provoked more feedback than anything else I’ve written about here

before, so I suspect the topic resonated with many of you. Most the emails I received were about personal experiences people have had with technical debt recognized too late to be handled gracefully. But a few concentrated on a very specific point: my claim that computers exhibit chaotic behavior, which is a problem when using them in research. I’ll take these reactions as a pretext to elaborate on this point, which isn’t (in my opinion) yet sufficiently appreciated.

I call this a pretext because I won’t actually address the objections raised in much detail: they focus on the problems of defining chaos mathematically for discrete rather than continuous systems. The standard definitions of chaos refer to infinitesimal changes in initial conditions, which make no sense for computers and other discrete state systems because

the smallest possible change is one bit. Readers interested in this topic can find detailed discussions in the literature on cellular automata; Stephen Wolfram’s A New Kind of Science2 is a good entry point, especially chapters 4 and 7. However, in the context of scientific programming, which is the topic of this department, the precise mathematical definition of chaotic behavior is much less relevant than its consequences on software development and testing.

Defining Chaos

Chaos is a mathematical concept from the theory of dynamical systems, which are defined by a state space and a time evolution rule for state. A dynamical system starts in an initial state; the time evolution rule is then applied to yield subsequent states. Both the state space and the time variable can be continuous or discrete. A computer is a dynamical system with a discrete state that consists of the computer’s memory and its processor’s internal state. Time is discrete as well, the elementary time step

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








76 July/August 2016

being the execution of one processor instruction. The processor’s instruction set provides the details of the time evolution rule. A computation’s initial conditions are memory contents plus processor state when the computation is started. Execution proceeds until the program reaches its end—if it ever does. The final memory contents contain the computation’s result. Note that “memory” should be understood in a wide enough sense to include all data storage available, including hard disks, network storage, and so on. Note also that what I consider here is computation in the narrow sense of mechanically processing information. When you add multiple processes, communication between them, or external events, everything becomes more complicated.

The defining aspect of chaos is the strong sensitivity of a dynamical system’s behavior to initial conditions: small changes in these conditions can cause large changes in the system’s future behavior for which no useful bounds can be established. Chaotic behavior in nature makes the long-term evolution of many phenomena unpredictable even though they’re deterministic. An often cited example is the weather, which can be predicted for only a very short time—about a week—not because of any inherently random processes or a lack of computational power but because the initial conditions that enter into the prediction can be measured only to some finite precision.

Computers are engineered dynamical systems for which these problems don’t exist. We know a computation’s initial state precisely—we can even store it for future reuse—but changes in the initial state are of interest as a way to explore the consequences of errors. Erratic behavior of the computing hardware itself is rare enough that it can safely be ignored, except for extremely large parallel computers. But human errors in the preparation of the initial state—program and input data—are an important cause of incorrect results. That’s why it makes sense to ask whether a computation’s behavior changes if the correct initial state is modified in some way.

Saying that a computer behaves chaotically means that a computation’s result depends strongly on the initial state, to the point that a small change in this state can change the result beyond any useful predictable bound. As I already mentioned, some of the criteria of traditional chaos theory don’t apply: changes can’t be made infinitely small, as the smallest possible change is a one-bit flip, and deviations can’t become infinite because the computation’s state consists of a finite number of bits. The latter restriction applies to every physical system, of course, but the finiteness of our planet’s atmosphere hasn’t prevented scientists from applying chaos theory to weather forecasting.

Amplifying Errors

The mechanism that causes chaotic behavior in computation is the amplification of small changes by subsequent steps. The impact of a one-bit flip can be small, say, if the bit happens to

represent the least significant digit of an input number. At the other extreme, a one-bit flip in a processor instruction can crash the program, leading to no useful result at all. In between these two extremes, a one-bit flip can lead to results that differ from the correct result in arbitrary ways. The worst case isn’t a program crash or a huge difference in the final result but an incorrect result that looks credible. Such a mistake has a good chance of going unnoticed.

I found a nice, simple illustration of error amplification in computation in a lecture by Gérard Berry (www.college-de-france.fr/site/gerard-berry/inaugural-lecture-2008-01-17-18h00.htm). Suppose you live in a city with a grid-like street layout (see Figure 1). You want to explain to a friend at the other end of the city how to reach you by car. You provide a list of driving directions that tell your friend what to do at every corner: turn left (L), turn right (R), or continue straight on (S). Your driving instructions are thus a string made of the letters L, R, and S, which isn’t very different from a program written as a list of processor instructions. By following these instructions, your friend will move along the blue path from the bottom of the grid to the top.

But suppose that some mistake happens in transmitting the driving directions or that your friend takes a wrong turn. The smallest possible mistake would be the replacement of a

Figure 1. The consequences of minimal mistakes in driving

directions. A list of driving directions for every corner—turn left

(L), turn right (R), or continue straight on (S)—becomes a string

made of the letters L, R, and S. The smallest possible mistake

would be the replacement of a single letter. Two such minimally

modified instruction sequences, in red and violet, show the

resulting paths.

SRLLSRSSLRRSLRSLS

SRLLSRSRLRRSLRSLS

SRLLSRSSLSRSLRSLS

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_______

_______________________________________

_________

http://www.college-de-france.fr/site/gerard-berry/inaugural-lecture-2008-01-17-18h00.htm






single letter. Figure 1 shows two such minimally modified instruction sequences, in red and violet, together with the resulting paths. It’s clear that the places you reach by following these modified instructions aren’t at all close to the real destination or even close to each other. In fact, minimal mistakes can take you anywhere on the grid.

In this simple example, the mechanism of error amplification is easy to understand: each minimal one-letter change rotates the remaining part of the path by 90 degrees compared to the correct one. The more steps that remain to be done after the mutated instruction, the farther the arrival point will be from the intended one. This is also the general mechanism that makes a computation’s outcome in the presence of mistakes so hard to predict.

An error that grows linearly with the number of computational steps isn’t really that bad. It doesn’t yet deserve the label “chaotic behavior.” After all, our driving directions programming language provides several useful guarantees. No matter what mistake you make, a program consisting of N instructions is guaranteed to terminate after exactly Nsteps and yield a valid position on the grid (assuming the grid is big enough), which is at most 2N grid steps away from the intended destination. There’s no possibility of nontermination (driving around the grid endlessly) or a crash (hitting an invalid position). Unfortunately, we don’t get such guarantees for the programs we use in computational science. The reason why we have them for our driving directions is that the language is very limited; it can be processed by what’s known as a finite state machine in automata theory, whereas standard programming languages require a more powerful automaton—in fact, the most powerful type of automaton known today—called a Turing machine. As in other aspects of life, more power also means more responsibility, as mistakes can have more serious consequences. Turing machines give you so much power that you can easily create chaos (www.scipublish.com/journals/AM/papers/1506). If we added features such as loops or tests based on current position to our driving directions language, we would need a Turing machine to process it, possibly condemning our friends, intentionally or by mistake, to spend the rest of their lives driving around the city.

In writing scientific software using Turing-complete languages, we have exactly that power and we should be constantly watching out for the unexpected consequences of mistakes. However, that isn’t how computational scientists behave in practice. The dominant attitude is to trust computational results if they “look right,” that is, they don’t disagree with expectations and prior knowledge of the problem under study. I’ve even heard people say that they “trust their intuition” to spot potential mistakes in the results. Now that isn’t in itself an absurd idea: experienced experimentalists do recognize suspect results coming from instruments they know well. It’s certainly possible to develop an intuition for the credibility of data.

However, the crucial difference is that experimental equipment is carefully designed not to exhibit chaotic behavior, such that minor damage or production defects don’t lead to unexpected results. If you put a mite under a microscope, a defect in the instrument could lead to a blurred image but not to an image showing an extra pair of legs. If the image is sharp enough to let you count the mite’s legs, then you know you can trust that observation. But this approach doesn’t carry over to software. There’s no typical symptom of a programming mistake—anything is possible. Even the most experienced software developers can’t judge if a program is correct by inspecting the source code, or by running it on a few sample inputs, except if the program is trivially small. Moreover, in many applications of computational science, correctness of a program isn’t even something we could aim for. We can only decide if a program is correct if we have some other means of specifying what the correct result is. This often isn’t the case in a research setting, where much code is written for computing quantities that nobody has ever computed before.

Mechanized Mathematics

A common attitude that we should probably start to question in this context is the one of considering computation as mechanized mathematics. The focus of mathematics is on precise statements that can be proven right or wrong. In research, we often use computation to explore imprecise statements—scientific hypotheses—whose domains of validity aren’t known yet. A useful complementary notion to mathematical correctness is the robustness of the computational tools we use. This is well explained in an essay by Gerald Sussman that draws on his background in electrical engineering (http://groups.csail.mit.edu/mac/users/gjs/6.945/readings/robust-systems.pdf). A robust computer program produces reasonable output for reasonable input, even if the latter differs from what the program was initially designed to handle. We have similar concepts in computational science, for example, the notion of numerical stability of algorithms. But robustness criteria don’t play a major role in the design and implementation of scientific software today.

To link these abstract considerations of potential chaotic behavior to the practice of computational science, I tried to apply them to my own work in molecular simulations. Most of my work is about extracting information from simulation trajectories, which are roughly 1-Gbyte datasets—too big for inspection by eye but small enough to be processed on my laptop. I spend much of my day writing, modifying, and running Python scripts that perform various geometrical and statistical analyses on these trajectories. These scripts are rather short but rely on a collection of libraries, ranging from general and widely used ones such as NumPy (www.numpy.org) or h5py (www.h5py.org) to domain-specific ones such as Mosaic (http://github.com/mosaic-data-model/mosaic-python).

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



__________

_______ _________

____

__________

_______

_______________________________________

____


http://www.scipublish.com/journals/AM/papers/1506

http://groups.csail.mit.edu/mac/users/gjs/6.945/readings/robust-systems.pdf

http://www.numpy.org

http://www.h5py.org

http://github.com/mosaic-data-model/mosaic-python

http://www.numpy.org






78 July/August 2016

The computation that’s performed when I run one of my scripts is defined by the script, the file containing the trajectory, the Python language, and all the libraries used by the script. But that’s just what’s immediately visible. The libraries I’ve cited depend on other libraries, and the Python interpreter is written in C. A different way to present this complex assembly of software is as a set of consecutive layers that transform a general-purpose computer into a tool for performing a very specific analysis of a simulation trajectory. Each of these layers can be described in terms of the notation in which the additional information being added is expressed and the tool that processes this information:

1. the processor instruction set, executed by the computer;2. the C language, translated into processor instructions

by a C compiler;3. the Python language, executed by an interpreter written

in C;4. the Python language augmented by the NumPy library,

written in C and plain Python;5. more libraries; and6. the file format for the data files, interpreted by an analy-

sis script written in Python and augmented by various libraries.

All the layers listed here define both a part of the computation and the notation in which some other part of the computation is expressed. For example, the Python language defines the data representation and memory management aspects of the computation, in addition to defining what is and isn’t a legal Python program. My analysis script defines all the algorithms as well as the input file format for the trajectories.

For historical and practical reasons, we use different labels to refer to these notations: the first three are called programming languages, whereas the last one is a file format.The intermediate ones that simply add libraries are rarely recognized as distinct notations at all. To see that they really are, consider the small Python script:

import numpy

print(numpy.arange(5))

This isn’t a valid program in the Python language, but it’s valid in the Python-plus-NumPy language, which shows that these two languages are distinct. Libraries should thus be treated as language extensions.

All the layers but the last one are general-purpose, Turing-complete programming languages. There are, of course, significant differences between them, which is why these different layers exist. The processor instruction level isn’t very convenient for human programmers, being difficult to read and understand. Moreover, processor instructions are a high-risk notation: any sequence of bytes can be executed

as instructions, but most byte sequences won’t produce anything useful and could even damage stored data. The C language is much more convenient and provides a better level of verification, with many possible mistakes caught by the compiler. The Python language is even more convenient and helps avoid mistakes by expressing a computation in much fewer lines of code. Similarly, each library layer adds more convenience for computations made up of the kind of operations that these libraries implement, at the same time preventing mistakes by allowing the programmer to write less code for a specific computation.

However, none of these layers adds useful guarantees about the computation’s behavior. The potential of generating chaos remains present until the very last layer, which is defined by my trajectory analysis script. From the viewpoint of robustness, it would be preferable to have as much of the computation as possible expressed using notations that limit the impact of mistakes. Instead of libraries that add optional shorthand notation for some operations to a general-purpose language, we should have successive layers of languages that enforce the use of a more specialized and less dangerous notation.

This idea is sometimes advocated as the “principle of least power”: every aspect of a computation should be described in a language with just as much expressive power as strictly required for the task (www.w3.org/DesignIssues/Principles.html). The additional guarantees that a limited language can make also offer more opportunities for analyzing and optimizing a program. Many domain-specific languages (DSLs) are based on this idea.3 In contrast to a library, a DSL is both more and less than a general-purpose language, with the “less” part often including giving up Turing-completeness. A data file format is then nothing but an extreme case of a DSL in which only constant data is allowed and no algorithm can be expressed at all.

The main obstacle to such an approach is psychological. We associate the term “language” in the context of programming with something complex that takes many years to master. Most computational scientists would be happy to get away with learning only one programming language in their life. But as I explained earlier, a language can be a small variation on another one. Just as today’s libraries are extensions to general-purpose languages, we could have library-like pieces of code that remove or restrict features of languages. As an example, removing while-loops and recursive function calls would turn Python into a language in which every program is guaranteed to terminate. A quick inspection of the trajectory analysis scripts that I wrote over the past few months showed that they could all be written in such a restricted Python dialect. In fact, they already are. I rarely need all the power that Python gives me. But I can’t ask Python to verify that I didn’t use that power by mistake.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



__________

http://www.w3.org/DesignIssues/Principles.html






These observations apply with minor variations in languages, libraries, and file formats to most computations done to-

day in science and engineering. In the short history of com-puting, we can observe a general tendency toward providing more expressive power wherever some aspect of a computation is defined. In the early days, we had Fortran programs reading simply structured input files, but as soon as computers could handle larger programs, embedded scripting languages—usu-ally Turing-complete—became a desirable feature for custom-izing application software. From there, it was a small step to writing application software in a scripting language augmented by DSLs. There are even cases of languages that became ac-cidentally Turing-complete as features were added, the best-known example being C++ templates. It is, in fact, possible to do arbitrary computations, including chaotic ones, as part of the compilation of a C++ program. The downsides of too much freedom in program structure have been known for a while, leading to the widespread adoption of structured programming in the 1980s and to the growing popularity of functional pro-gramming in recent years.4 But advocates of these approaches to safer programming were always keen to point out that no expressive power is lost in adopting them. Perhaps it’s time to reevaluate the importance of computational omnipotence.

References

1. K. Hinsen, “Technical Debt in Computational Science,” Computing in Science & Eng., vol. 17, no. 6, 2015, pp. 103–103.

2. S. Wolfram, A New Kind of Science, Wolfram Media, 2002.3. K. Hinsen, “A Glimpse of the Future of Scientific

Programming,” Computing in Science & Eng., vol. 15, no. 1, 2013, pp. 84–88.

4. K. Hinsen, “The Promises of Functional Programming,” Computing in Science & Eng., vol. 11, no. 4, 2009, pp. 86–90.

Konrad Hinsen is a researcher at the Centre de Biophysique Moléculaire in Orléans (France) and at the Synchrotron Soleil in Saint Aubin (France). His research interests include protein struc-ture and dynamics and scientific computing. Hinsen has a PhD in theoretical physics from RWTH Aachen University (Germany). Contact him at [email protected].

Selected articles and columns from IEEE Computer Society pub-lications are also available for free at http://ComputingNow.

computer.org.

Advertising Personnel

Marian Anderson: Sr. Advertising CoordinatorEmail: [email protected]: +1 714 816 2139 | Fax: +1 714 821 4010

Sandy Brown: Sr. Business Development Mgr.Email [email protected]: +1 714 816 2144 | Fax: +1 714 821 4010

Advertising Sales Representatives (display)

Central, Northwest, Far East:Eric KincaidEmail: [email protected]: +1 214 673 3742Fax: +1 888 886 8599

Northeast, Midwest, Europe, Middle East:Ann & David SchisslerEmail: [email protected], [email protected]: +1 508 394 4026Fax: +1 508 394 1707

Southwest, California:Mike HughesEmail: [email protected]: +1 805 529 6790

Southeast:Heather BuonadiesEmail: [email protected]: +1 973 304 4123Fax: +1 973 585 7071

Heather BuonadiesEmail: [email protected]: +1 973 304 4123Fax: +1 973 585 7071

Advertising Sales Representatives (Jobs Board)

Heather BuonadiesEmail: [email protected]: +1 973 304 4123Fax: +1 973 585 7071

ADVERTISER INFORMATION

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_____________

_______________

________________

_____________ ________________

______________________________

______________________________

________________



http://computingnow.computer.org














SECTION TITLEEditors: Konrad Hinsen, hinsen@cnrs-o

LEADERSHIP COMPUTINGEditors: James J. Hack, [email protected] | Michael E. Papka, [email protected]


Illuminating the Universe’s Ignition

Eric Gedenk | Oak Ridge National Laboratory

Drive away from city lights on a clear night, and you can see innumerable stars. Several hundred thou-sand years after the Big Bang, though, the uni-verse looked very different—and much darker.

Minutes after the Big Bang, the universe was filled with radiation and a nearly uniform, featureless atomic gas of hy-drogen and helium so hot that every atom was ionized—or stripped of its electrons, which roamed freely. By the time 400,000 years elapsed, expansion of the universe had cooled so much that every hydrogen atom was able to capture an electron to neutralize it, and the radiation had shifted its colors to invisible wavelengths, as shown in Figure 1.

What came next is called the “cosmic dark ages,” before gravity managed to pull any of that gas in to make stars. The total density of matter, then as now, was everywhere domi-nated by dark matter, a mysterious form of matter invisible to the human eye that scientists only know exists based on its gravitational pull.

“When the first galaxies and stars formed about 100 million years after the Big Bang, starlight leaked into the intergalactic medium, where most of the hydrogen and heli-um created in the Big Bang was still waiting, not yet formed into galaxies,” said Paul Shapiro, the Frank N. Edmonds Jr. Regents Professor in Astronomy at the University of Tex-as at Austin. “As ultraviolet photons of starlight streamed across the universe, they struck these intergalactic atoms and photoionized them [meaning light served as the catalyst for ionization], releasing fast-moving electrons that heated the gas to 10,000 Kelvin.”

Shapiro noted that as this process continued, galaxies and groups of galaxies formed. Zones of hot, ionized inter-galactic gas grew up around them, filling the universe with a patchwork quilt of ionized and neutral zones, until the ion-ized zones became large enough to overlap entirely. “By the end of the first billion years, the whole universe was filled with ionized gas—reionized, actually, since atoms were born

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_______________








in an ionized state a billion years before that but had since then recombined. We call this transfor-mation the ‘epoch of reionization,’” Shapiro said. “It was a kind of ‘second coming’ of ionization—a giant backlash by the first galaxies on the universe that created them.”

A Multi-Institutional Team

Shapiro can speak confidently on this topic because he and a multi-institutional research team have been using supercomputing resources at the US De-partment of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) to simulate this phenomenon with unparalleled detail and accuracy.

The team, which includes collaborators from the Universities of Texas, Strasbourg (France), Zu-rich (Switzerland), Sussex (UK), Madrid (Spain), and Potsdam (Germany), simulated the reionization of the universe with a special focus on how it hap-pened in our own neighborhood—the Milky Way and Andromeda galaxies along with a number of smaller ones that compose the Local Group.

The team’s goal was to learn about reionization across the universe, while predicting its imprints that can be observed today on these nearby celes-tial objects. Astronomers think that when dwarf galaxies much smaller than the Milky Way were engulfed by reionization, for example, their star formation was suppressed. If the team’s newest simulation confirmed this, it would solve a major puzzle in the standard theory of cosmology. Its first results are soon to be published in the Monthly No-tices of the Royal Astronomical Society.

Shapiro noted that these simulations were the first to have all three components—large enough volume, fine enough spatial grid, and conditions just right to model the local universe—while calculating all the major physical processes involved in reioniz-ing the universe in a fully coupled way. This meant that, in addition to calculating the gravitational pull of dark matter, atomic gas, and stars on each other, researchers also had to simulate the hydrodynamics of hydrogen gas flowing within and between galax-ies and the transport of the radiation that helped si-multaneously complete the universe’s ionization and heating.

“You can’t simulate the radiation as an after-thought,” Shapiro said. “You can’t just simulate the gravitational collapse of galaxies and then ‘paint on’ the radiation and let it ionize. That isn’t good enough, because there is feedback. The radiation influences how galaxies form and release the star-light that causes reionization—so we need to do it simultaneously.”

There was a barrier to doing this, however, which even some of the fastest supercomputers couldn’t overcome previously. To solve the equa-tions describing the time history of reionization, researchers needed to compute “snapshots” of the moving gas, stars, dark matter, and radiation to track the expanding surfaces of the ionized zones (called “ionization fronts”) without blurring them.

The radiation that causes these surfaces to ex-pand travels at the speed of light—much faster than the gas, dark matter, and stars. To follow the ra-diation and ionization fronts accurately, while also tracking the slower-moving gas, stars, and dark mat-ter, Shapiro’s team programmed the ionization and radiation snapshots to run on GPUs. The team also used CPUs on the same parallel computer nodes to compute the motions of gas, dark matter, and stars, separately.

Such a multifaceted simulation needed not only a powerful supercomputer but also one that

Figure 1. In less than a billion years after the Big Bang, the first galaxies flooded the universe with enough ultraviolet starlight to ionize every atom in the intergalactic gas and heat it to 10,000 degrees Kelvin. This image from a 3D supercomputer simulation shows hot, ionized gas (bright patches) and cold, neutral gas (dark regions). The image is a snapshot of how the ionized gas (created by millions of galaxies that clustered in space along the threads of a cosmic web) might have been distributed within a piece of the expanding universe 300 million light-years across today (centered on our Milky Way galaxy), when half the volume was still occupied by neutral gas.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM








LEADERSHIP COMPUTING

82 July/August 2016

had plenty of GPUs so the team could offload its radiation and ionization calculations. Thankfully, the Oak Ridge Leadership Computing Facility’s (OLCF’s) Cray XK7 Titan supercomputer fit the description.

The OLCF, a DOE Office of Science User Facility located at ORNL, built Titan with a GPU for every one of its 18,688 compute nodes. Shapiro’s team needed 8,192 GPUs to run their simulations.

“Titan is special in having such a large num-ber of GPUs, and that is something that made our simulations a Titan-specific milestone,” Shapiro said.

Cosmic Dawn, Cosmic Datasets

To simulate such fine spatial detail over such a large volume, the researchers generated enormous data sets—over 2 Pbytes (2 million Gbytes), in fact.

The team stored 138 snapshots, one for every 10 million years of cosmic time, starting at a time about 10 million years after the Big Bang. Each snapshot created about 16 Tbytes of raw data. Be-cause saving this much data indefinitely wasn’t pos-sible, the team also created a greatly reduced data set on the order of 100 Tbytes for further analysis and a more permanent archival record.

The team’s use of the OLCF’s analysis clusters Eos and Rhea allowed the researchers to scan each “snapshot” for the positions of the N-body particles so they could identify the locations and masses of dense, quasispherical clumps of particles called halos using a “halo finder” program. Each halo is the scaffolding within which a galaxy formed, with gas, dark matter, and star particles held together by their mutual gravitational attraction. With ev-ery snapshot, the number and location of galactic halos changed.

By matching this catalogue of galactic halos lo-cated by the halo finder to the properties of the gas and stars within them in each snapshot, the team could track the evolving properties of all the galax-ies in their simulation for comparison with obser-vations of such early galaxies.

These OLCF clusters were necessary to enable a parallel computing analysis of the raw datasets that were otherwise too large to be analyzed by lesser computers.

After analyzing their Cosmic Dawn (CoDa) simulation snapshots, Shapiro’s team found that the simulated reionization of the local universe ended a bit later than observations suggest for

the universe-at-large, but the predicted proper-ties of galaxies, intergalactic gas, and radiation backgrounds that filled the universe at such early times agreed with those observed once an adjust-ment was made for this late reionization. The team also found that star formation in the dwarf galax-ies—those of total mass below a few billion times the mass of the sun—was severely suppressed by reionization, as hypothesized. The bulk of the star formation in the universe during the epoch of reionization was dominated instead by galax-ies with masses between 10 billion and 100 billion times that of the sun.

Perhaps most importantly, CoDa will now serve as a pathfinder for a follow-up simulation, CoDa II, already planned, in which parameters that control the efficiency for converting cold, dense gas into stars will, with the hindsight provided by CoDa I, be adjusted to make reionization end a bit earlier.

Despite the extreme scale of the CoDa simula-tion, Shapiro said there are still other ways to

improve. Even with the future of exascale comput-ing—computers capable of a thousand-fold speed-up over current petascale machines—he noted that completely simulating the universe in all macro- and microscopic detail would still demand more power.

However, as next-generation supercomputers add more GPUs and memory, researchers will be able to zoom in on small patches of the universe, and within these, zoom in further on individual galaxies, offering a more detailed look at star and galaxy formation. “We may be able to say, ‘I can’t follow the formation of all the stars, but maybe I can zoom in on one galaxy, then zoom in on one cloud of molecular gas within it, which might wind up forming several thousand stars,’” Shapiro said.

Acknowledgments

The Oak Ridge Leadership Computing Facility is a DOE Office of Science User Facility supported under contract DE-AC05-00OR22725.

This manuscript has been authored by UT-Battelle, LLC under contract number DE-AC05-00OR22725 with the US Department of Energy. The US gov-ernment retains and the publisher, by accepting the article for publication, acknowledges that the US gov-ernment retains a nonexclusive, paid-up, irrevocable,

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM










worldwide license to publish or reproduce the pub-lished form of this manuscript, or allow others to do so, for US government purposes. The DOE will pro-vide public access to these results of federally spon-sored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Image by Hyunbae Park, Paul Shapiro, Jun-Hwan Choi, and Anson D’Aloisio of Paul Shapiro’s Cosmology Research Group in the University of Texas Astronomy Department, using data supplied by Pierre Ocvirk (University of Strasbourg) from a simulation of hydro-dynamics, radiation, and gravity by Shapiro and his international collaboration team under DOE INCITE Program Award AST031 on the Titan supercomput-er at Oak Ridge National Laboratory, with support from NSF AST-1009799, NASA NNX11AE09G, and NASA JPL SURP Project Nbr. 1492788, with image

processed at the University of Texas at Austin’s Texas Advanced Computing Center under NSF XSEDE TG-AST0900005.

Eric Gedenk is a science writer for Oak Ridge National Laboratory. His interests include science writing, computing, media frames, and bass guitar. Gedenk received a BS in journalism and an MA in German from the University of Tennessee, Knoxville. Contact him at [email protected].

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



_________________________

_______

____________

_____________________



http://energy.gov/downloads/doe-public-access-plan


http://cybersecurity.ieee.org






THE LAST WORD

by Charles Day

Faraday’s Tablet

Idon’t believe in one staple of science fiction: time travel. The notion that a cause might precede its effect is unphysical. But I’m open to the possibility of another science fic-tion staple: technologically advanced space aliens. If such aliens exist, we might be able to reap one of the purported benefits of time travel, namely, watching the past.To see how, imagine that intelligent extraterrestrials visited Earth in 1831. Thanks

to their stealth technology, they were able to move among humans undetected. And because of their curiosity about our science, they installed a miniature video camera in the London laboratory of one of Earth’s greatest scientists of that or any other time, Michael Faraday.

Now imagine further that the camera was recently discovered and that archivists at the British Library have figured out how to view the recordings. What would they see? Faraday would be center stage, setting up, performing, and documenting the experi-ments that would lead to his discovery of electromagnetic induction.

Would Faraday’s work have been made easier and more effective if the aliens had secretly left him with a tablet computer? The aliens in my thought experiment follow a rule akin to Star Trek’s Prime Directive, something that forbids them from interfer-ing in the development of other civilizations. Faraday’s tablet wouldn’t be loaded with knowledge from the future, but it would have a camera, a virtual keyboard, and access to the Royal Society’s Philosophical Transactions and other journals.

My hunch is that even with a tablet, we would still see Faraday going about his research in the same way. The tablet might be convenient, but it wouldn’t be transformative.

A month before the iPad made its debut in April 2010, Wired’s Steven Levy gushed at its potential to change the way we use computers. “No more files and folders, physical keyboards and mouses,” he wrote. “Instead, the iPad offers a streamlined yet powerful intuitive experience that’s psychically in tune with our mobile, attention-challenged, super-connected new century.”

Two years later, Macworld published an article entitled, “How the iPad Helps Scientists Do Their Jobs.” The help in question amounted to the convenience of being able to take a camera, notebook, and reference book on field trips and to conferences.

So how could tablets benefit, if not transform, computational science? I received a hint of an answer when I bumped into Victor, a physicist I know, at a recent recep-tion. When I asked him what he was working on, he whipped out his iPad to show me slides from a talk he gave on the statistical mechanics of income distribution, which got me thinking: What if Victor wanted to do more than display his analysis on his iPad? What if he also wanted to conduct the analysis via his iPad?

The app I have in mind for tackling this challenge would interface to a program that runs on a computer elsewhere. Using the app, a scientist could reset the

program’s input parameters, and once the run finished, the app would display the results on the iPad screen. Being able to run and examine simulations, wherever and whenever you want, wouldn’t break new computational ground, but it would be fun!

Charles Day is Physics Today’s editor in chief. The views in this column are his own and not nec-essarily those of either Physics Today or its publisher, the American Institute of Physics.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM







PURPOSE: The IEEE Computer Society is the world’s largest association of computing professionals and is the leading provider of technical information in the field.MEMBERSHIP: Members receive the monthly magazine Computer, discounts, and opportunities to serve (all activities are led by volunteer members). Membership is open to all IEEE members, affiliate society members, and others interested in the computer field.COMPUTER SOCIETY WEBSITE: www.computer.orgOMBUDSMAN: Direct unresolved complaints to [email protected]: Regular and student chapters worldwide provide the opportunity to interact with colleagues, hear technical experts, and serve the local professional community.AVAILABLE INFORMATION: To check membership status, report an address change, or obtain more information on any of the following, email Customer Service at [email protected] or call +1 714 821 8380 (international) or our toll-free number, +1 800 272 6657 (US):

Membership applicationsPublications catalogDraft standards and order formsTechnical committee listTechnical committee applicationChapter start-up proceduresStudent scholarship informationVolunteer leaders/staff directoryIEEE senior member grade application (requires 10 yearspractice and significant performance in five of those 10)

PUBLICATIONS AND ACTIVITIESComputer: The flagship publication of the IEEE Computer Society, Computer, publishes peer-reviewed technical content that covers all aspects of computer science, computer engineering, technology, and applications.Periodicals: The society publishes 13 magazines, 18 transactions, and one letters. Refer to membership application or request information as noted above.Conference Proceedings & Books: Conference Publishing Services publishes more than 175 titles every year.Standards Working Groups: More than 150 groups produce IEEE standards used throughout the world.Technical Committees: TCs provide professional interaction in more than 45 technical areas and directly influence computer engineering conferences and publications.Conferences/Education: The society holds about 200 conferences each year and sponsors many educational activities, including computing science accreditation.Certifications: The society offers two software developer credentials. For more information, visit www.computer.org/certification.

NEXT BOARD MEETING

13–14 November 2016, New Brunswick, NJ, USA

EXECUTIVE COMMITTEEPresident: Roger U. FujiiPresident-Elect: Jean-Luc Gaudiot; Past President: Thomas M. Conte; Secretary: Gregory T. Byrd; Treasurer: Forrest Shull; VP, Member & Geographic Activities: Nita K. Patel; VP, Publications: David S. Ebert; VP, Professional & Educational Activities: Andy T. Chen; VP, Standards Activities: Mark Paulk; VP, Technical & Conference Activities: Hausi A. Müller; 2016 IEEE Director & Delegate Division VIII: John W. Walz; 2016 IEEE Director & Delegate Division V: Harold Javid; 2017 IEEE Director-Elect & Delegate Division V: Dejan S. Miloji i

BOARD OF GOVERNORSTerm Expriring 2016: David A. Bader, Pierre Bourque, Dennis J. Frailey, Jill I. Gostin, Atsuhiro Goto, Rob Reilly, Christina M. SchoberTerm Expiring 2017: David Lomet, Ming C. Lin, Gregory T. Byrd, Alfredo Benso, Forrest Shull, Fabrizio Lombardi, Hausi A. MüllerTerm Expiring 2018: Ann DeMarle, Fred Douglis, Vladimir Getov, Bruce M. McMillin, Cecilia Metra, Kunio Uchiyama, Stefano Zanero

EXECUTIVE STAFFExecutive Director: Angela R. BurgessDirector, Governance & Associate Executive Director: Anne Marie KellyDirector, Finance & Accounting: Sunny HwangDirector, Membership: Eric BerkowitzDirector, Products & Services: Evan M. ButterfieldDirector, Sales & Marketing: Chris Jensen

COMPUTER SOCIETY OFFICESWashington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928Phone: Fax: +1 202 728 9614Email: [email protected] Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720Phone: +1 714 821 8380Email: [email protected]

MEMBERSHIP & PUBLICATION ORDERSPhone: Fax:Asia/Pacific: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 107-0062, JapanPhone: Fax: +81 3 3408 3553Email: [email protected]

IEEE BOARD OF DIRECTORSPresident & CEO: Barry L. ShoopPresident-Elect: Karen BartlesonPast President: Howard E. MichelSecretary: Parviz FamouriTreasurer: Jerry L. HudginsDirector & President, IEEE-USA: Peter Alan EcksteinDirector & President, Standards Association: Bruce P. KraemerDirector & VP, Educational Activities: S.K. RameshDirector & VP, Membership and Geographic Activities: Wai-Choong (Lawrence) WongDirector & VP, Publication Services and Products: Sheila HemamiDirector & VP, Technical Activities: Jose M.F. MouraDirector & Delegate Division V: Harold JavidDirector & Delegate Division VIII: John W. Walz

revised 25 May 2016

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



____________

_________

________

_______

_____________

__________

___________

__________

http://www.computer.org



http://www.computer.org/certification










IEEE Computer Society 2016 Call forMAJOR AWARD NOMINATIONSHelp Recognize Computing’s Most Prestigious

contributors in the teaching and R&D computing communities. All members of the profession are invited to nominate individuals who they consider most eligible to receive international recognition of an appropriate society award.

Deadline: 15 October 2016Nomination Site: awards.computer.orgFor more information visit: www.computer.org/awards

Computer Entrepreneur AwardSterling Silver GobletVision and leadership resulting in the growth of some segment of the computer industry.

Technical Achievement Award

Contributions to computer science or computer technology.

Harry H. Goode Memorial Award

Information sciences, including seminal ideas, algorithms, computing directions, and concepts.

Hans Karlsson Award

Team leadership and achievement through collaboration in computing standards.

Richard E. Merwin Distinguished Service Award

Outstanding volunteer service to the profession at large, including service to the IEEE Computer Society.

Harlan D. Mills Award

Contributions to the practice of software engineering through the application of sound theory.

Computer Pioneer Award

Pioneering concepts and development of the

W. Wallace McDowell Award

Recent theoretical, design, educational, practical, or other tangible innovative contributions.

Taylor L. Booth Award

Contributions to computer science and engineering education.

Computer Science & Engineering Undergraduate Teaching Award

Recognizes outstanding contributions to undergraduate education.

IEEE-CS/Software Engineering Institute Watts S. Humphrey Software Process Achievement Award(Joint award by CS/SEI)

Software professionals or teams responsible for an improvement to their organization’s ability to create and evolve software-dependent systems.

qqM

Mq

qM

MqM



qqM

Mq

qM

MqM



http://awards.computer.org

http://www.computer.org/awards





discrete modeling & simulation - mn.uio.no · discrete modeling & simulation contents ......

Documents