Grid activities in JINRGrid activities in JINR &&

Meta-cluster of distributed Meta-cluster of distributed computing “Dubna-Grid”computing “Dubna-Grid”

Korenkov VladimirKorenkov Vladimir

JINR, DubnaJINR, Dubna

In 2006 the Joint Institute for Nuclear Research (JINR) - an international intergovernmental scientific research organization

- celebrated its 50-th anniversary.

JINR Member StatesJINR Member StatesArmeniaArmenia

AzerbaijanAzerbaijan

BelarusBelarus

BulgariaBulgaria

CubaCuba

Czech RepublicCzech Republic

GeorgiaGeorgia

KazakhstanKazakhstan

Democratic People’s Democratic People’s Republic of KoreaRepublic of Korea

MoldovaMoldova

MongoliaMongolia

PolandPoland

RomaniaRomania

RussiaRussia

Slovak RepublicSlovak Republic

UkraineUkraine

UzbekistanUzbekistan

VietnamVietnam

JINR is a large multidisciplinary scientific centre incorporating:

– basic research in frontier particle, nuclear and condensed matter physics,

– development and application of high technologies, and

– university education in the relevant fields of knowledge.

International CooperationInternational Cooperation

JINR’s partners are about 700 institutions located in 60 countries

AUSTRIABELGIUM

CERNCROATIA

DENMARKFINLANDFRANCE

GREECE

IRELAND

NETHERLANDSNORWAY

PORTUGAL

SLOVENIASPAIN

SWEDENSWITZERLAND

YUGOSLAVIAUNITED KINGDOM

BULGARIA

CZECH REPUBLIC

GERMANY

HUNGARY

ITALY

EUROPE

DUBNA

AFRICAEGYPT

SOUTH AFRICA

AUSTRALIA AND

OCEANIAAUSTRALIA

REPUBLICSOF

FORMER USSR

AMERICA

BRAZILCANADA

UNITED STATES

CHINA

ISRAELJAPAN

SOUTH KOREATURKEY

INDIA

DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA

MONGOLIA

VIETNAM

ASIA

BRAZILCANADA

CLAFCUBA

UNITED STATES

Science Bringing Nations TogetherScience Bringing Nations Together

CERN and JINR joint exhibitions:CERN and JINR joint exhibitions: 1997:1997: University of Oslo (Norway) University of Oslo (Norway) 1998:1998: UNESCO, ParisUNESCO, Paris 1999:1999: UN Office in GenevaUN Office in Geneva 2000:2000: European Parliament in BrusselsEuropean Parliament in Brussels 2001:2001: Russian State Duma, MoscowRussian State Duma, Moscow 2002:2002: Ministry of Education and Ministry of Education and

Sciences, Bucharest, RomaniaSciences, Bucharest, Romania 2003:2003: Yerevan University, ArmeniaYerevan University, Armenia

Diplomatic Academy, MoscowDiplomatic Academy, Moscow 2005:2005: University of University of Thessaloniki, GreeceThessaloniki, Greece

GenevaGeneva

BrusselsBrussels

MoscowMoscowBucharestBucharestThessalonikiThessaloniki

At present the JINR LAN comprises more 6000 computers and nodes (Gigabit Ethernet)

Local Area Network Backbone

- support and development support and development LCG/EGEE infrastructure;LCG/EGEE infrastructure;- participation in LCG middleware testing/evaluation, participation in LCG middleware testing/evaluation, - participation in Data and Service Challenges,participation in Data and Service Challenges,- CASTOR and dCache usage/development; CASTOR and dCache usage/development; - grid monitoring and accounting system development;grid monitoring and accounting system development;- EGEE-RDIG portal support and development, EGEE-RDIG portal support and development, - MCDB development; MCDB development; - HEP application; HEP application; - User & Administrator Training and Education

- support of JINR member states in the LCG/EGE activities.support of JINR member states in the LCG/EGE activities.  

JINRJINR in the LCG/EGEE Project in the LCG/EGEE Project

RDIGRDIG monitoring&accountingmonitoring&accounting http://rocmon.jinr.ru:8080http://rocmon.jinr.ru:8080

ArchitectureArchitecture

wwwwww..egeeegee--rdigrdig..ruru

The “Dubna-Grid” Project is aimed at the The “Dubna-Grid” Project is aimed at the creation of a distributed environment of meta-creation of a distributed environment of meta-computing on the basis of vacant computing computing on the basis of vacant computing resources of “office” computers. resources of “office” computers. Implementation of the project “Dubna-Grid” will allow:Implementation of the project “Dubna-Grid” will allow:

to create a unified computing environment of the city of Dubna (meta-cluster) on to create a unified computing environment of the city of Dubna (meta-cluster) on the basis of the resources of scientific and educational institutions, in particular, the basis of the resources of scientific and educational institutions, in particular, JINR subdivisions, University “Dubna”, secondary schools and other organizations JINR subdivisions, University “Dubna”, secondary schools and other organizations concerned; concerned; to create a segment of the international Grid-infrastructure, operating in frames of to create a segment of the international Grid-infrastructure, operating in frames of various Grid systems (LGG, NorduGrid and OSG). To perform research of control, various Grid systems (LGG, NorduGrid and OSG). To perform research of control, security and stability within the Grid environment.security and stability within the Grid environment. to provide execution of large-scale computing tasks of JINR and other enterprises to provide execution of large-scale computing tasks of JINR and other enterprises of Dubna scientific – industrial complex and other organizations concerned;of Dubna scientific – industrial complex and other organizations concerned; to distribute experience of creation of the city segment of the Grid-infrastructure to distribute experience of creation of the city segment of the Grid-infrastructure for creation of similar systems in other Russian cities.for creation of similar systems in other Russian cities.

Meta-cluster of distributed computing “Dubna-Grid”Meta-cluster of distributed computing “Dubna-Grid”

DUBNA-GRIDDUBNA-GRID

Main stages of realization Main stages of realization

development of technologies on realization of a development of technologies on realization of a separate administration of office computers and separate administration of office computers and nodes of the Grid infrastructure on the basis of nodes of the Grid infrastructure on the basis of software shells of virtual machinessoftware shells of virtual machines

creation of a meta-cluster prototype integrating creation of a meta-cluster prototype integrating resources of various organizationsresources of various organizations

development of mass installation technologies and development of mass installation technologies and spreading software to all accessible nodes of the city spreading software to all accessible nodes of the city infrastructureinfrastructure

installation of various Grid systems at the meta-installation of various Grid systems at the meta-cluster nodes and accordance of these resources to cluster nodes and accordance of these resources to the users of other Grid projects such as LCG, the users of other Grid projects such as LCG, NorduGrid, EGEE, OSG.NorduGrid, EGEE, OSG.

training of specialists in the field of Grid technologies. training of specialists in the field of Grid technologies. Holding seminars, giving lectures and Holding seminars, giving lectures and demonstrations.demonstrations.

City Informational-Educational Computer Network “Dubna”

Network resources of the Project

JINR Backbone

Construction of the Construction of the systemsystem

The Meta-cluster can be interpreted as a cluster, but the The Meta-cluster can be interpreted as a cluster, but the computing resources are distributed over different places. computing resources are distributed over different places. Basically meta-clusters are constructed from resources Basically meta-clusters are constructed from resources which are specially provided for it, but the “Dubna-Grid” which are specially provided for it, but the “Dubna-Grid” meta-cluster uses vacant computer resourcesmeta-cluster uses vacant computer resources of office of office computers in parallel with their main purpose. computers in parallel with their main purpose.

Dubna-Grid Meta-cluster includes a managed server, a Dubna-Grid Meta-cluster includes a managed server, a bridge and computational nodes. Computational nodes bridge and computational nodes. Computational nodes represent Virtual PCs emulated by virtual machine represent Virtual PCs emulated by virtual machine technology (VMware).technology (VMware).

By means of VMware we have two computers instead of By means of VMware we have two computers instead of one, there are no differences between real and virtual PCs, one, there are no differences between real and virtual PCs, they both have its own memory, IP address, processor and they both have its own memory, IP address, processor and etc, but physically they both use the same resources. Two etc, but physically they both use the same resources. Two machines share one processor, memory, Ethernet card and machines share one processor, memory, Ethernet card and all of PC devices. As all nodes in the cluster represent the all of PC devices. As all nodes in the cluster represent the virtual PCs, we have absolutely a homogenous virtual PCs, we have absolutely a homogenous environment for building the infrastructure. environment for building the infrastructure.

Construction of the systemConstruction of the system

In order to create the computing meta-cluster the following In order to create the computing meta-cluster the following approaches were used:approaches were used:

time-shared equipment with common access to the resources;time-shared equipment with common access to the resources; common network infrastructure for real and virtual resources common network infrastructure for real and virtual resources

and own additional network infrastructure for virtual resources and own additional network infrastructure for virtual resources virtual clustering of the equipment.virtual clustering of the equipment.

The following software tools and technologies were used:The following software tools and technologies were used: software support for virtual machines maintenance (VMware),software support for virtual machines maintenance (VMware), virtual network (VLAN),virtual network (VLAN), virtual access to the software and data (AFS),virtual access to the software and data (AFS), building and loading OS images to the Virtual PCs of the meta-building and loading OS images to the Virtual PCs of the meta-

cluster (Warewulf package).cluster (Warewulf package).

On the central server the following software is installed:On the central server the following software is installed: Scientific Linux CERN OS; Scientific Linux CERN OS; a package for support of cluster architecture Warewulf; a package for support of cluster architecture Warewulf; Ganglia monitoring system; Ganglia monitoring system; OpenAFS, OpenAFS, the batch system Torque with Maui scheduler. the batch system Torque with Maui scheduler.

Logical schemeLogical scheme of the “Dubna-Grid” meta-cluster of the “Dubna-Grid” meta-cluster

ConclusionConclusion

A distributed meta-computing environment of A distributed meta-computing environment of Dubna city based on vacant computing Dubna city based on vacant computing resources (computational, storage) of office resources (computational, storage) of office computers of scientific and educational computers of scientific and educational institutions has been created.institutions has been created.

Mass installation technologies and spreading Mass installation technologies and spreading software to all accessible nodes of the city software to all accessible nodes of the city infrastructure have been developedinfrastructure have been developed

Virtual meta-clusters have been integrated Virtual meta-clusters have been integrated with JINR batch system with JINR batch system

Monitoring system of the meta-cluster has Monitoring system of the meta-cluster has been developedbeen developed

First real tasks have been runFirst real tasks have been run, , including including simulation of ttbar simulation of ttbar 4m in ATLAS detector 4m in ATLAS detector

Grid infrastructure in University Centre of JINRGrid infrastructure in University Centre of JINR

Grid infrastructure is a Grid infrastructure is a set of virtual machines set of virtual machines (VMs) running on physical (VMs) running on physical ones (hosts)ones (hosts) Virtualisation was made Virtualisation was made using User Mode Linuxusing User Mode Linux current number of VMs is current number of VMs is 36 (6 VMs on each of 6 36 (6 VMs on each of 6 hosts)hosts) all virtual resources are all virtual resources are grouped into independent grouped into independent testbeds which in turn can testbeds which in turn can be used for different aims: be used for different aims: system administrators and system administrators and users training in grid field, users training in grid field, debugging and testing debugging and testing custom grid services in custom grid services in desirable grid environmentdesirable grid environment

● Course for system administrators using Nordugrid ARC middleware, LCG and gLite was successfully conducted on that infrastructure

The two major Grid conference in Russia (Grid2004 and Grid2006), hosted by Joint Institute for Nuclear Research in Dubna, has been hailed a success: - more than 200 participants; - about 100 reports.

http://lit.jinr.ru/grid2004/http://grid2006.jinr.ru