automatic launch and tracking the computational simulations with liflow and sumatra
TRANSCRIPT
Automatic launch and tracking the computational simulations with
LiFlow and Sumatra
Evgeniy KuklinUral-PDC 2016
Living system simulation
• Significant computational recourses.
• High degree of qualification in CS.
• Numerous computational experiments on the same model.
• routine and time-consuming.
• Low reproducibility of numerical experiments.
2
Personalized heart models project
• IIP, IMM UrB RAS; UrFU.
• Biophysicists, mathematicians, software developers.
• Heart simulation software for parallel computing systems.
• Only professional computer scientists can use it.
• Almost zero reproducibility of experiments.
3
Personalized heart models project
Requirements for biophysicists & mathematicians:
• Simple graphical user interface.
• Ability to execute a series of experiments with various parameter values.
Reproducibility requirements:
• Archive data storage.
• Metadata capturing for easy replicating.
4
LiFlow
LiFlow – LIving system simulation workFLOW:
• Graphical interface for the users.
• Executing a series of experiments on parallel computing systems.
• Easy to learn and use.
5
Related work
Scientific workflow systems:• Taverna,
• Kepler,
• Triana.
Systems for reproducibility of computational experiments:
• CDE,
• Madagascar,
• Sumatra.
6
Sumatra
• Capture the information required to recreate the computational experiment environment instead of capturing the experimental context itself.
• Keep up experiment catalog with ability to search by tags or data.
• Store obtained data in the archive.
• (-) Lacks a convenient desktop user interface.
7
Workflow in LiFlow
8
Computational Package
The information required to execute a series of experiments:
• Source code (git repository).
• Initial data and parameters.
• Generator of experiment series.
Supported generators:• Parameter: initial value, final value, and increment.
• Explicit parameter values.
9
How LiFlow works
1. User provides the computational package.
2. Computational package is transferred to the supercomputer using SSH.
3. The source code is build on the supercomputer.
4. Experiment series generator (Python script) is executed and produce the input data with different parameter values.
5. Sumatra project is set up to store the environment.
6. The jobs are putted into the supercomputer resource manager (SLURM) queue.
7. After job completion Sumatra compresses and places obtained data to the archive using the NFS protocol.
8. User received a e-mail notification. 10
LiFlow architecture
11
Sumatra technical details
12
Default project LiFlow
Storing the project information in the project local directory.
Using shared database file to create the single database of all experiments of all users.Available by --store option.
Manual job launch. Sumatra direct interaction with the SLURM Workload Manager.Available by option--launch_mode=slurm-mpi.
Storing the output data in the project local directory.
Using shared folder via NFS to create the single archive for all system users.Available by --archive option.
Conslusion
LiFlow:• Convenient graphical user interface for executing the
series of experiments.
LiFlow + Sumatra:• Archiving experimental results and metadata for
reproducibility.
Approbation:• Supercomputer “Uran” of the Krasovskii Institute of
Mathematics and Mechanics.
• Computational cluster of the Ural Federal University.
13
Conslusion
The system is intended to be used by researchers in mathematical biology and biophysics without extensive knowledge in parallel computing.
14
Is suitable for other
research projects with
own simulation code.
GUI