experiences with using unicore in production grid infrastructures deisa and d-grid
DESCRIPTION
Michael Rambadt [email protected]. Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid. Table of content. The Eurpopean DEISA project The German Grid initiative D-Grid The UNICORE Grid middleware - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/1.jpg)
Forschungszentrum Jülichin der Helmholtz-Gesellschaft
Experiences with using UNICORE in Experiences with using UNICORE in Production Grid Infrastructures DEISA Production Grid Infrastructures DEISA
and D-Gridand D-Grid
Michael Rambadt
![Page 2: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/2.jpg)
ISGC 2007 [email protected] 2 Forschungszentrum Jülich
Table of contentTable of content
The Eurpopean DEISA project The German Grid initiative D-Grid The UNICORE Grid middleware Lessons Learned from UNICORE in
production in DEISA and D-Grid Conclusions
![Page 3: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/3.jpg)
ISGC 2007 [email protected] 3 Forschungszentrum Jülich
DEISA – general aspectsDEISA – general aspects
Consortium of leading national supercomputing centers in EU
Deploy and operate an innovative, distributed, terascale Grid-empowered infrastructure to enhance and reinforce High Performance
Computing in Europe to be used by scientists and industries in
a coherent and comfortable way with production quality being stable, secure, reliable,
persistent, ...
![Page 5: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/5.jpg)
ISGC 2007 [email protected] 5 Forschungszentrum Jülich
DEISA Service ActivitiesDEISA Service Activities
SA1 – Network Operation and Support SA2 – Data Management with Global File Systems SA3 – Resource Management
Deployment and operation of global scheduling services for the European super-cluster as well as for its heterogeneous Grid extension
SA4 – Applications and User Support SA5 – Security
![Page 6: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/6.jpg)
ISGC 2007 [email protected] 6 Forschungszentrum Jülich
Table of contentTable of content
The Eurpopean DEISA project The German Grid initiative D-Grid Grid middleware UNICORE Lessons Learned from UNICORE in
production in DEISA and D-Grid Conclusions
![Page 7: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/7.jpg)
ISGC 2007 [email protected] 7 Forschungszentrum Jülich
Max Mustermann
- general aspects- general aspects
the German Grid initiative builds up and operates a sustainable Grid
infrastructure establishes methods of e-science in the German
scientific community More than 100 partners Started in 2005 with a 100 Million Euro funding from
the German ministry for Education and Research
ISGC 2007
![Page 8: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/8.jpg)
ISGC 2007 [email protected] 8 Forschungszentrum Jülich
Max Mustermann
- general aspects- general aspects
Initiative contains following projects DGI - D-Grid Integration project AstroGrid-D in astronomy C3-Grid for climate research HEP-Grid for high energy physics InGrid for engineering research MediGrid for medical research TextGrid for humanities Wisent for energy meteorology
ISGC 2007
![Page 9: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/9.jpg)
ISGC 2007 [email protected] 9 Forschungszentrum Jülich
Table of contentTable of content
The Eurpopean DEISA project The German Grid initiative D-Grid Grid middleware UNICORE Lessons Learned from UNICORE in production
in DEISA and D-Grid Conclusions
![Page 10: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/10.jpg)
ISGC 2007 [email protected] 10 Forschungszentrum Jülich
Motivation: Why ?Motivation: Why ?
Scientists have to use huge computational and storage resources
![Page 11: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/11.jpg)
ISGC 2007 [email protected] 11 Forschungszentrum Jülich
Supercomputers are managed by Resource Management Systems (RMSs) that handle the scheduling
But: There are many RMSs available Many proprietary ways of job submission
IBM Loadleveler llsubmit… Torque Resource Manager qsub…
Different job description languages
(# of nodes, memory requirements…)
Motivation: Why ?Motivation: Why ?
![Page 12: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/12.jpg)
ISGC 2007 [email protected] 12 Forschungszentrum Jülich
The UNICORE Grid
UNiform Interface to COmputing Resources Started as a German funded Project in 1997 Enhanced in many (European) projects Seamless and secure access to distributed resources
and data intuitive GUI workflow engine for
complex multi-site multi-step workflows job monitoring
No ‘learn overhead’ if a new RMS is used easy installation and configuration
![Page 13: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/13.jpg)
ISGC 2007 [email protected] 13 Forschungszentrum Jülich
20082007200620052004200320022001200019991998
EUROGRID
UNICORE Plus
GRIP
GRIDSTART
UniGrids
NextGRID
CoreGRID
EGEE-II
Chemomentum
Phosphorus
A-WARE
eDEISA
OMII-Europe
UNICORE
DGI
VIOLA
OpenMolGRID
DEISA
developmentdevelopment
![Page 14: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/14.jpg)
ISGC 2007 [email protected] 14 Forschungszentrum Jülich
Table of contentTable of content
The Eurpopean DEISA project The German Grid initiative D-Grid Grid middleware UNICORE Lessons Learned from UNICORE in
production in DEISA and D-Grid Conclusions
![Page 15: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/15.jpg)
ISGC 2007 [email protected] 15 Forschungszentrum Jülich
in productionin production
UNICORE production use on JUMP at Research Centre Juelich (IBM p690 eSeries Cluster (1312 CPUs, 8.9 TFlops)
Almost 1/3 of all jobs are UNICORE jobs
0%
10%
20%
30%
40%
50%
60%
70%
Dez05
Jan06
Feb06
Mrz06
Apr06
Mai06
Jun06
Jul06
Aug06
Sep06
Okt06
Nov06
Dez06
Jan07
Feb07
UNICORE usage on JUMP
non UNICORE usage
UNICORE usage on JUMP Non UNICORE usage
![Page 16: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/16.jpg)
ISGC 2007 [email protected] 16 Forschungszentrum Jülich
Lessons learned…Lessons learned…
Often “only” initial hurdles adapting applications
managing certificates How easy is it to access the Grid Middleware?
Automatic user management in DEISA in D-Grid
Users have to be stimulated and encouraged to use Grid technology for applications, computations, data
transfer and access to resources adapt/integrate their applications to/into Grids once convinced they likely use it further on
![Page 17: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/17.jpg)
ISGC 2007 [email protected] 17 Forschungszentrum Jülich
More lessons learned…More lessons learned…
Fulfillment of functional requirements is not enough Users want
help to overcome initial hurdles 24/7 availability of the Grid infrastructure
Monitoring tool SIMON for DEISA and D-Grid UNICORE components
24/7 availability of the Grid experts support hotline, help desk, mailing lists, …
long-term commitment for continuous development and support
workshops, hands-on training, …
![Page 18: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/18.jpg)
ISGC 2007 [email protected] 18 Forschungszentrum Jülich
Even more lessons learned…Even more lessons learned…
Operation of production environments is costly certification authority, administrative tools, integration into
site management, licenses, … Agreement on what the users want and what
the developers implement is crucial Scientist sometimes don’t like GUIs
DEISA developed DESHL as command line interface to UNICORE to address this
![Page 19: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/19.jpg)
ISGC 2007 [email protected] 19 Forschungszentrum Jülich
Table of contentTable of content
The Eurpopean DEISA project The German Grid initiative D-Grid Grid middleware UNICORE Lessons Learned from UNICORE in
production in DEISA and D-Grid Conclusions
![Page 20: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/20.jpg)
ISGC 2007 [email protected] 20 Forschungszentrum Jülich
ConclusionsConclusions
Production Grids are possible But: only interesting for scientists if
deployment of new production software offers added value
easy usage, increased effectiveness, decreased cost, … integration of legacy applications
Success of the Grid Middleware depends on successful
interaction with other components, working groups, colleagues…
Functionality is important but also support, support, support…
![Page 21: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/21.jpg)
ISGC 2007 [email protected] 21 Forschungszentrum Jülich
Dietmar Erwin (2005):
“ If Grid technology delivers 80% of the vision by 2010 it was successful ”
ConclusionsConclusions
![Page 24: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/24.jpg)
ISGC 2007 [email protected] 24 Forschungszentrum Jülich
Th
an k s…
To all partners who contributed to the UNICORE development in DEISA and D-Grid
![Page 25: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/25.jpg)
ISGC 2007 [email protected] 25 Forschungszentrum Jülich
Th
an k s…
Central Institute for Applied Mathematics
Forschungszentrum Jülich GmbH52425 Jülich, Germany
CINECAVia Magnanelli 6/3
40033 Casalecchio di Reno, Italy
Rechenzentrum GarchingMax-Planck-Institute for Plasmaphysics
85748 Garching, Germany
2
1
3
R. Breu, L. Clementi, Th. Fieseler, A. Giesler, P. Malfetti, R. Menday, J. Reetz, A. Streit, P. Wieder
2
1
1
1
3
1
1 1
2
![Page 26: Experiences with using UNICORE in Production Grid Infrastructures DEISA and D-Grid](https://reader035.vdocuments.site/reader035/viewer/2022062723/56813fe9550346895daadbad/html5/thumbnails/26.jpg)
ISGC 2007 [email protected] 26 Forschungszentrum Jülich
Getting Getting
Download UNICORE from
http://www.unicore.eu