institute of problem of chemical physics, moscow district, chernogolovka

GRID SERVICES FOR COMPUTATIONAL CHEMISTRYGRID SERVICES FOR COMPUTATIONAL CHEMISTRY AT THE RUSSIAN GRID POLYGONS: AT THE RUSSIAN GRID POLYGONS:

ANALYSIS OF THE STATE AND PERSPECTIVESANALYSIS OF THE STATE AND PERSPECTIVES

Institute of Problem of Chemical Physics, Moscow district, Chernogolovka

Varlamov DVarlamov Dmitry / mitry / Варламов ДмитрийВарламов Дмитрий

Volokhov V.M., Volokhov A.V., Pivushkov A.V., Pokatovich G.A., Prokhorov A.I.

Dubna-2012, Distributed Computing and Grid-technologies in Science and EducationDubna-2012, Distributed Computing and Grid-technologies in Science and Education

Chemistry – a science about materials, i.e. Chemistry – a science about materials, i.e. about the world in which we live… about the world in which we live…

Constants of rates of reactions

Industrial parameters of reactions

Production of new and modified materials

USERS: science, industry, business

Features of synthesis of the new and changed

materials

The new and improved production technologies of materials

Quantum-mechanical and molecular dynamics simulation

Optimization of nanomaterials and nanostructures

Quantum-chemical calculations: electronic structure of molecules, Quantum-chemical calculations: electronic structure of molecules, interaction potentials, substance structure etcinteraction potentials, substance structure etc.


Quantum-statistical level Calculation of models taking into account the immediate environment of clusters of atoms and

small molecules

Hierarchical multilarge-scale approach to modeling of objects from Hierarchical multilarge-scale approach to modeling of objects from quantum to macrolevel – all levels demand high efficience calculationsquantum to macrolevel – all levels demand high efficience calculations

Quantum-mechanical levelModeling by the ab initio methods of molecules and small clusters from 10-100 atoms (possible

phases in a material)

Kinetic levelSystem from 103 to 106 atoms , i.e. objects the size of 100-1000 nm3

a. evolution of quantum non-equilibrium systems consisting of hundreds of clustersb. methods of molecular dynamics (equations of classical mechanics) for atoms and

small molecules, considered as classical particles

Mesoscopic levelThe viscosity, thermal conductivity, coefficient of friction, the wave processes in cells, meso-and

macroscopic processes in materials, etc.

The level of continuum media and construction levelDefining the parameters of the finite element method, which implements a model of continuous media and the level of the model structures, based on the theory of mechanisms and machines,

and the theory of complex systems


Typical needs of quantum chemistry for high-Typical needs of quantum chemistry for high-efficiency calculationsefficiency calculations

1. First two applied scientific calculations for Petaflops performance represented researches on molecular modeling: calculations of electronic structure of high-temperature superconductors (cuprates) by the quantum Monte-Carlo method and studying of effect of a huge magnetoresistance in magnetic nanoparticles (“Jaguar”, Oak Ridge, USA - 1,64 Pf)

2. Resource allocation (examples): researches of properties of water in low-dimensional systems – up to 8·106 CPU(core)-hours (Argonne National Laboratory); development of chemical catalysts – up to 30·106 CPU-hours (“Jaguar”) per year (2010);

3. Typical FULL docking of protein ligand: 200 atoms x 300 000 configurations x 1000 core-hours – about 300 Pflops!!!

4. By some estimates practically most of the supercomputer centers of the USA (San Diego, Ohio, Illinois etc.) allocate to 25-30 % of capacities for needs of biochemistry and molecular modeling, quantum chemistry, nanotechnological calculations


O

Sn

Pt

Research of catalitic hydrogen disintegration on nanostructures of Pt on a crystal surface of SnO2

2( ) ?

? ( )

U H

U H H

H H

SKIF-MSU («Chebyshev»: VASP, 200 CPU, 15 hours, 10 steps of optimization, total requirement - 100-200 steps

U(H2)

U(H+H)rH-H

rH-H

EXAMPLE OF ONE OF RESOURCE-INTENSIVE APPLICATIONEXAMPLE OF ONE OF RESOURCE-INTENSIVE APPLICATIONOF QUANTUM CHEMISTRYOF QUANTUM CHEMISTRY

Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

OSnPt

II.1-7.23

2.63

2.60

Structure of the cluster Pt19 put on a surface SnO2, distances are given in Å,stability for separation of a cluster from a surface is resulted in eV

VASP complex intended for carrying out of calculations taking into account translating symmetry is used.

For each point of calculation it is necessary more than days of operation on 100 CPU, necessary quantity of points – more than hundred days


O SnH

-2.39 -3.08

Pt

-1.61 -2.56

-2.15

-2.35

Dynamics of migration of a proton on a platinum cluster


VASP complex intended for carrying out of calculations taking into account translating symmetry is used.

For each point of calculation it is necessary more than days of operation on 100 CPU, necessary quantity of points – more than hundred days

EXAMPLE OF ANOTHER OF RESOURCE-INTENSIVE EXAMPLE OF ANOTHER OF RESOURCE-INTENSIVE APPLICATIONAPPLICATION

Trajectory calculations of sections of chemical reactions

H2+O2

Σ=Sn/NH+H+O+O

H2+O+O

H2O+O

H2O2

H+H+O2

H+HO2

HO+HO

HO+O+H

Start conditions:

• 4 angles of mutual orientation• aim parameters• 2 quantum oscillatory numbers• 2 quantum rotational numbers• energy of collision (in mass center system)

Up to 1-10 millions independent trajectories – at least 2·106 CPU hours

Chemical reactions are modeled by movement of atoms on potential surfaces within the limits of classical dynamics


Typical examples of supercomputer centers Typical examples of supercomputer centers focused on quantum chemical and molecular focused on quantum chemical and molecular

dynamics calculationsdynamics calculations

1. The EPSRC (Engineering and Physical Sciences Research Council) UK National Service for Computational Chemistry Software (NSCCS, http://www.nsccs.ac.uk – >250 Tflops

2. The National Resource for Biomedical and Chemistry Supercomputing (NRBSC, http://www.nrbsc.org) - calculation of molecular systems from 20000 to 120000 atoms, provides resources to external users up to 100 000 CPU-hours per structure;

3. Chemical Computing Group (http://www.chemcomp.com), Quebec, Canada – >120 Tflops

4. Lawrence Berkeley National Laboratory (USA, http://www.lbl.gov/csd), chemical branch – > 450 Tflops;

5. Lehrstuhl für Theoretische Chemie der Technische Universität München, Germany (http://www.lrz.de/services/software/chemie) - own Linux cluster (96 Tflops) plus constantly allocated resources of Supercomputer Center;

6. Swiss National Supercomputing Centre (http://www.cscs.ch) – from tenth Tflops to subPflops allocated resources for chemical calculations


Scientific directions of operations of IPCPScientific directions of operations of IPCP

IPCP RAS represents the largest in Russia academic institute which are carrying out researches in following areas:

• the theory of elementary chemical processes;• structure of molecules and structure of solid state objects;• creation of materials with in advance set properties;• kinetics and mechanisms of complicate chemical reactions;• nanotechnologies;• creation of biologically active substances and medical

drugs;• Biotechnologies and biochemistry;• and many other scientific directions….


1. Государственный контракт № 07.514.11.4019 от 23 сентября 2011 г. с ИПХФ РАН в рамках ФЦП «Исследования и разработки по приоритетным направлениям развития научно-технологического комплекса России на 2007-2013 годы» по теме «Разработка технологии проведения высокопроизводительных расчетов в области вычислительной химии в различных распределенных средах с применением методов виртуализации приложений и ресурсов» – основной исполнитель

2. ФЦП "Развитие инфраструктуры наноиндустрии в Российской Федерации на 2008-2011 годы", проект «Создание Национальной нанотехнологической сети (ГридННС)», государственный контракт № 16.647.12.2031 от 13.05.2011 г. – соисполнитель

3. Государственный контракт в рамках ФЦП «Исследования и разработки по приоритетным направлениям развития научно-технологического комплекса России на 2007-2013 годы» на тему «Развитие пилотной зоны российской грид-системы для высокопроизводительных вычислений, в том числе в интересах федеральных ядерных центров», тематика: «Адаптация пакетов прикладных программ для работы в пилотной инфраструктуре грид-системы. Разработка методов проведения грид-вычислений с использованием динамически формируемых сред исполнения грид-заданий» – соисполнитель

УЧАСТИЕ ИПХФ В ГРИД-ПРОЕКТАХУЧАСТИЕ ИПХФ В ГРИД-ПРОЕКТАХ

Федеральные целевые программыФедеральные целевые программы


1. Программа № 13 фундаментальных исследований Президиума РАН на 2009-2012 годы «Проблемы создания национальной научной распределенной информационно-вычислительной среды на основе развития GRID технологий и современных телекоммуникационных сетей», проект «Исследование методов виртуализации вычислительных сред и приложений в области вычислительной химии. Динамическое формирование параллельных программных сред на распределенных ресурсах»;

2. Программа № 27 фундаментальных исследований Президиума РАН на 2009-2012 годы «Основы фундаментальных исследований наноматериалов», проект «Самоорганизация наноразмерных материалов и процессы их взаимодействия с адсорбируемыми соединениями: компьютерное моделирование в параллельных и распределенных GRID средах терафлопного уровня»

Программы фундаментальных исследований Президиума РАН Программы фундаментальных исследований Президиума РАН

1. Грант РФФИ № 11-07-00686-а «Разработка методов динамического формирования параллельных вычислительных сред на различных типах грид ресурсов (на примере приложений квантовой химии)»

Гранты Российского Фонда фундаментальных исследованийГранты Российского Фонда фундаментальных исследований


Historical sequence of usage of GRID technologies atHistorical sequence of usage of GRID technologies at IPCPIPCP

2004 - 2007 г. Condor2005 - 2007 г. Globus 3/4

2006 - 2010 г. LCG-2 – gLite (EGEE-RDIG)2008 - … Unicore (SKIF Polygon)

2010 - …. Globus Toolkit 4 (GridNNN)

The quantum-chemical applications adapted for usage in the distributed GRID environments:

• Gaussian 03 (parallel, with taking into account license restrictions)• GAMESS US (parallel – socket and MPI variants)• CPMD (parallel)• Dalton (parallel)• NAMD (parallel) and many other:• Author’s programs – for example, investigations of tunnel properties of

structures (non-stationary equation Schrödinger )


On the basis of IPCP GRID center some Resource Sites of several Russian grid polygons (GridNNN, Minsvyaz GRID, SKIF-Polygon etc.) with support of a number of the established applied quantum-chemical packages (>10) and the Virtual Organizations (up to 3) oriented to computational chemistry constantly work

Access to computing resources of a network of GridNNN (till 40-50 Tf and to 10000 CPU) and to the large volume of the applied quantum and chemical software (Firefly, MolPro, LaMMPS, NAMD, AbInit etc.) supported by infrastructure of GridNNN is provided


Current achievementsCurrent achievements

Virtual OrganizationsVirtual Organizations ( (VOVO) ) in computational chemistryin computational chemistry fieldfieldwithin National Nanotechnological Networkwithin National Nanotechnological Network ( (GridNNNGridNNN))

Participants of GridNNN anParticipants of GridNNN anв в mentioned above VOmentioned above VO

НИВЦ МГУ («Чебышев») НИИЯФ МГУ ОИЯИ (Дубна) РНЦ “Курчатовский Институт” Казанский НЦ РАН ПИЯФ ИПХФ РАН ИМСС УрО РАН (Пермь) СПбГУ ВЦ ДВО РАН (Хабаровск) СПИИРАНи другие ВУЗы, НИИ РАН, инновационные и коммерческие предприятия

The IPCP created and supports 3 Virtual organizations "NanoChem", "Gamess", "Gaussian" in the field of computational chemistry within infrastructure «National Nanotechnological Network» (GridNNN) for carrying out calculations with use applied quantum and chemical software (Gaussian, GAMESS, NAMD, etc.) on GRID polygon of the Russian National Nanotechnological Network (http://nanogrid.icp.ac.ru/virtual.php, http://ngrid.ru/ngrid/gridnnn/volist)

Resource site of IPCP in SKIF gridResource site of IPCP in SKIF grid polygonpolygon((category “A” sitecategory “A” site))

• ОС – Linux Ubuntu, Unicore 6.2 Middleware– Gateway (https://unicorgw.icp.ac.ru:8080); – Server container (Unicore/X);– Target System Interface (TSI);– UAS (Unicore Atomic Services);– Unicore User Database – XUUDB;– PBS/Torque;– User Interface – Command-line client

• Access through the distributed environment to the main supercomputer resources of SKIF-Polygon is provided (for example, SKIF-MSU, different Russian sites)

• Quantum-chemical applications (Gamess-US, Gaussian, NAMD, author’s multiparameter tasks) also are adapted for usage in Unicore environment


Use standard (from the point of view of the chemist and any end user) applied programs in grid environments is caused by several parameters: •demand of applied packages scientific and industrial researchers, i.e. end users (there is no sense to carry out very labor-consuming adaptation and the subsequent control of the grid-environment for low-used packages);•license restrictions (including existing for freely extended packages); for commercial applied packages of a condition of licensing can or limit extremely possibility of realization them in the grid-environment, or outright forbid it;•availability of source code package that is highly desirable for some modules packages (eg, communication, or responsible for the implementation of parallel protocols);•ability to work without an interactive user interaction and graphical display of information (which is still unlikely to realize in a grid environment).

Quantum-chemical and molecular dynamic applied packages – Quantum-chemical and molecular dynamic applied packages – features of usefeatures of use


Quantum-chemical and molecular dynamic applied packages on Quantum-chemical and molecular dynamic applied packages on space of GridNNN and other grid polygonsspace of GridNNN and other grid polygons

(all are established on a resource grid sites of IPCP)(all are established on a resource grid sites of IPCP)

Quantum Chemistry:1.Gaussian – including high-level web-interface 2.GAMESS-US – including high-level web-interface3.FireFly4.AbInit5.NAMD6.VASP7.PWScf – including high-level web-interface8.NWChem – including high-level web-interface

Molecular dynamics1.LAMMPS2.OpenMX3.GROMACS


Adaptation of the application package for use in grid environments

1.Grid Gateway settings for service of an applied package(s) and providing information on it to the monitor of resources

2.Creation of low-level interfaces for applied package(s) for solving of incoming jobs

3.Creation of high-level web-oriented user interfaces (POI, WIG)

4.In some cases – modification of source codes of applied packages


Usercertificate

Simplified scheme of grid calculations with high-level interfaces to applied Simplified scheme of grid calculations with high-level interfaces to applied software packages software packages ((middleware GridNNN, gLite, Unicoremiddleware GridNNN, gLite, Unicore))

Datafiles

Results Sp

ace

of

gri

d p

oly

go

ns

Sp

ace

of

gri

d p

oly

go

ns

Lo

w-l

evel

in

terf

aces

to

ap

pli

ed p

acka

ges

on

Res

ou

rce

Sit

esП

ПП

на

рес

урсн

ых

сай

тах

Authorisedgrid-user

Authorization and certifications

Loading, edition, storing the input data

High-level formation of job

Resource selection

Job launch

Job’s monitoring

Return of the received results

Formation of final result

Web interfaces, POI, WIG

Gat

eway

to

grid

env

ironm

ent

WWW-portalWIG(s)


Web-interfaces was created (http://nanogrid.icp.ac.ru , http://webgrid.icp.ac.ru – grid portal of IPCP)

to work with the quantum-chemical software packages GAMESS-US, Gaussian, NAMD, NWChem, PWscf etc. in virtual organizations of GridNNN

and the pilot zone of the Russian grid system for high performance computing

Friendly to the end user (researcher) Web interfaces to applied software packages considerably reduce labor input of work of the user with similar packages in the conditions of the distributed computing environments.The created interfaces allow the user to work via the Internet browser and to carry out the following actions:

•to authorize the user for start of a program complex and to carry out its certification in the Virtual Organization;

•to prepare a job (including creation and edition of initial data and configuration files) in compliance with package requirements;

•to start an applied package in grid-range infrastructure (if necessary – on any or chosen resource);

•to conduct monitoring of performance of a job (including stopping and restart);

•on termination of job – to receive results of the distributed calculations.


Example ofExample of modulemodule of of Web-Web-interface for interface for quantum-chemical applied package quantum-chemical applied package GAMESS GAMESS

((for loading and edition initial filesfor loading and edition initial files))



Module of the web interface for full-scale preparation of data and configuration files for a quantum-chemical GAMESS-US applied package in

the distributed environments


((resource selectionresource selection))



((monitoring of jobs)monitoring of jobs)


The problems of creation and The problems of creation and development of applied GRID servicesdevelopment of applied GRID services

• High economic costs of creation and operation of computing resources and infrastructure, absence of steady support of the Russian grid polygons;

• The absence (or low) interest from industry and business:– the general low culture of carrying out technological development in the majority of

scientific and industrial organizations;

– inability and unwillingness to use results of mathematical modeling and high-efficiency calculations.

• Absence of skilled professionals as from computer specialists (systems analysts, programmers), and by the users (chemists, engineers);

• Practical absence of domestic applied packages high software cost, the license restrictions, the reduced area of use, difficulty of development;

• Weak settlement of the legal and economic relations between owners of computing services (including GRID) and consumers


Suggested ways to solve problemsSuggested ways to solve problems• Stimulation of the user(s) (industrial first of all) for use of computing

GRID services:

– absence or the minimum payment of using by GRID resources at the first stages of innovative development;

– training of users as a full-time, and in a mode of interactive on-line - creation of information web resources on training to work with interfaces of applied packages and with GRID resources;

– introduction in plans of training of applied experts (engineers, chemists, etc.) additional courses of work with applied software packages and carrying out by students of obligatory calculations of real scientific and technical and engineering tasks with use of supercomputers and GRID infrastructures;

– creation friendly, with a large number of base templates, interfaces (WWW focused) to GRID resources;

– introduction by the RF state as a certification condition when developing a part of commodity products and materials of obligatory application of means of mathematical modeling and testing


Suggested ways to solve problems (continued)Suggested ways to solve problems (continued)

• Government policies and stimulation of developers of applied GRID services and resources– creation of the State (or equity), stable, constantly running computing

infrastructure (supercomputers, clusters of medium-sized, high-speed communication channels, etc.);

– compensation by the State of a part of operational costs of the supercomputer centers for support of the distributed calculations (grants or the budgetary financing);

– tax privileges for development of the domestic applied software, creation of the innovative parks focused on it;

– improvement of standard and legal and economic regulation in the field of strategic information technologies;

– in the field of the State education: (a) expanding the number of budget places in universities for professions' “system administration/programming”, “application programming” etc.; (b) Use a wide range of undergraduate and graduate students of universities to develop the necessary system and application software in the learning process; (в) conduct training of university teachers in these specialties.


Within the IPCP GRID Resource Center integrated into the Within the IPCP GRID Resource Center integrated into the National Nanotechnological Network and a pilot zone of the National Nanotechnological Network and a pilot zone of the Russian grid-system for high-efficiency calculations, access Russian grid-system for high-efficiency calculations, access to a number of quantum-chemical and molecular-dynamic to a number of quantum-chemical and molecular-dynamic applied packages for carrying out large-scale calculations on applied packages for carrying out large-scale calculations on the Russian grid-ranges with use both simplified, and the Russian grid-ranges with use both simplified, and problem-oriented high-level web interfaces is realizedproblem-oriented high-level web interfaces is realized

ConclusionsConclusions

THANKS FOR YOUR ATTENTION!!!!


institute of problem of chemical physics, moscow district, chernogolovka

Documents