1

Applications

SURAgrid “All Hands” Meeting, Washington DCMarch 14 – 16, 2007

BioSim

Mahantesh Halappanavar,Ashutosh Mishra, Ravindra Joshi,

Mike Sachon

2

BioSim: Bio-electric Simulator for Whole Body Tissues

Numerical simulations for electrostimulation of tissues and whole-body biomodels

Predicts spatial and time dependent currents and voltages in part or whole-body biomodels

Numerous diagnostic and therapeutic applications, e.g., neurogenesis, cancer treatment, etc.

Fast parallelized computational approach

3

Simulation Models Whole-body discretized within a cubic space

simulation volume From electrical standpoint, tissues are characterized

as conductivities and permittivities Cartesian grid of points along the three axes. Thus, at

most a total of six nearest neighbors

* Dimensions in millimeters

4

Numerical Models

Kirchhoff’s node analysis

Recast to compute matrix only once

For large models, matrix inversion is intractable

LU decomposition of the matrix

0)]/}({/}{)/[( LAVdtVdLA

)]([]||][[ tBVVM tdtt

5

Numerical Models

Voltage: User-specified time-dependent waveform

Impose boundary conditions locally

Actual data for conductivity and permittivity

Results in extremely sparse (asymmetric) matrix

Red: Total elements in the matrix

Blue: Nonzero Values

[M]

6

Why Focus on Solvers? Scaling: (Source: David Keys, NIA Nov 2006)

– “Science” phase scales as: – “Solver” phase scales as – Computation will be almost all solver after several doublings– Optimal solver saves computation cycles for physics

)(NO

)( 23

NO

)(NO

7

The Landscape of Sparse Ax=b Solvers

Pivoting

LU

GMRES, QMR, …

Cholesky

Conjugate gradient

DirectA = LU

Iterativey’ = Ay

Non-symmetric

Symmetricpositivedefinite

More Robust Less Storage

More Robust

More General

Source: John Gilbert, Sparse Matrix Days in MIT 18.337

8

LU Decomposition

Source: Florin Dobrian

9

LU Decomposition

Source: Florin Dobrian

10

Computational Complexity

100 X 100 X 10 nodes: ~75 GB of memory (8-B floating precision)

Sparse data structure: ~ 6 MB (in our case) Sparse direct solver: SuperLU-DIST

– Xiaoye S. Li and James W. Dimmel, “SuperLU-DIST: A Scalable Distributed-Memory Sparse Direct Solver for Unsymmetric Linear Systems”, ACM Trans. Mathematical Software, June 2003, Volume 29, Number 2, Pages 110-140.

Fill reducing orderings with Metis– G. Karypis and V. Kumar, “A fast and high quality multilevel

scheme for partitioning irregular graphs”, SIAM Journal on Scientific Computing, 1999, Volume 20, Number 1.

11

Performance on compute clusters

144,000-node Rat Model

Blue: Average iteration time

Cyan: Factorization time

12

Output: Visualization with MATLAB

Potential Profile at a depth of 12mm

13

Output: Visualization with MATLAB

Simulated Potential Evolution Along the Entire 51-mm Width of the Rat Model

14

Deployment on

Mileva: 4-node cluster dedicated for SURAgrid purposes

Authentication – ODU Root CA– Cross certification with SURA Bridge – Compatibility of accounts for ODU users

Authorization Initial Goals:

– Develop larger whole-body models with greater resolution – Scalability tests

15

Grid Workflow

Establish user accounts for ODU users – SURAgrid Central User Authentication and

Authorization System– Off-line/Customized (e.g., USC, LSU)

Manually launch jobs based on remote resource – SSH/GSISSH/SURAgrid Portal– PBS/LSF/SGE

Transfer files – SCP/GSISCP/SURAgrid Portal

16

Recent Efforts in grid-enabling:

Porting to 100% open source tools (GCC/GFORTRAN)

SURAgrid Sites:– Texas A&M University: Calclab– University of Virginia: Grid04 and Grid11

Experiments with MUMPS 4– Symmetric matrices and out-of-core

Acknowledgements:– Jim Jokl, Steve Losen, Steve Johnson, Brain Brooks,

Kate Barzee and Mary Fran Yafchak

17

News: (February 14, 2007)

18

19

Conclusions

Science:– Electrostimulation has variety of diagnostic and

therapeutic applications– While numerical simulations provide many advantages

over real experiments, they can be very arduous

Grid enabling:– New possibilities with grid computing– Grid-enabling an application is complex and time

consuming– Security is nontrivial

20

Future Steps

Grid-enable BioSim– Explore alternatives for grid enabling BioSim– Explore funding opportunities– Load Balancing– Establish new collaborations – Scalability experiments with large compute clusters

accessible via SURAgrid Future applications:

– Molecular and Cellular Dynamics– Computational Nano-Electronics– Tools: Gromacs, DL-POLY, NAMD

21

References and Contacts

A Mishra, R Joshi, K Schoenbach and C Clark, “A Fast Parallelized Computational Approach Based on Sparse LU Factorization for Predictions of Spatial and Time-Dependent Currents and Voltages in Full-Body Biomodels”, IEEE Trans. Plasma Science, August 2006, Volume 34, Number 4.

http://www.lions.odu.edu/~rjoshi/ Ravindra Joshi, Ashutosh Mishra, Mike Sachon,

Mahantesh Halappanavar– (rjoshi, amishra, msachon, mhalappa)@odu.edu

22

Teaching Initiative

CS775/875: Distributed Computing

Ravi Mukkamala

Professor, Department of Computer Science

23

Details:

Graduate course with ~15 students Guest lecture Followed by a homework

– Familiarize with grid computing concepts– Hands-on approach– Experiment with Globus services & commands

Acknowledgements:– Jim Jokl, Steve Losen, Steve Johnson, Brain

Brooks, Nicole Geiger, Kate Barzee and Mary Fran Yafchak

24

25

Conclusions:

Laboratory for testing the concepts Potential to attract students For SURAgrid

– Large number of short-lived certificates– Cleanup … (CRLs?/home drives/…)– Centralized account creation (Still painful )– Short term funding/internships for grad/under-grad

students?

26

THANK YOU !!!