forschungszentrum karlsruhe in der helmholtz-gemeinschaft european and german grid computing...
Post on 15-Jan-2016
216 views
TRANSCRIPT
Forschungszentrum Karlsruhein der Helmholtz-Gemeinschaft
European and German Grid Computing Projects
Marcel Kunze
Forschungszentrum Karlsruhe
DESY Seminar
December 2002
Marcel Kunze, DESY Seminar
Grid Computing Potential
1980 2005
Marcel Kunze, DESY Seminar
What is Grid Computing?
Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
Marcel Kunze, DESY Seminar
Grid Computing: Diverse Areas of Application
• Computational GridHigh Performance Computing (HPC)Perspective: Parallelization of programs, as fast as possible
• Data GridHigh Throughput Computing (HTC)Perspective : Parallelization of data, as much as possible
• Gaming GridCommunication between playersPerspective: LAN + WAN-Party, as entertaining as possible
Marcel Kunze, DESY Seminar
Grid Computing: Commercial Aspects
• Huge potential of Grid Computing in the field of information technology:– Cost reduction through more efficient/diverse use of system resources
– „Resource on demand“Added flexibility, e.g. dynamically satisfy increased computing demand in bank accounting sector at the end of an accounting period
– „Business continuity“ Proliferation of mission-critical IT-Services to ease disaster recovery
– Possibility of system consolidation: Reduction of system complexity by standardization of services
Marcel Kunze, DESY Seminar
Grid Computing: Scientific Aspects
• Enables formation of „critical mass“ of resources and expertise to allow for interdisciplinary projects in so-called “Virtual Organizations” (VO)– Tele-Collaboration
– Instrument Sharing
– Computing Resource Sharing
– Data Sharing
• Potential fields of application – Bio-Informatics and Medicine
– Astronomy
– Earth Sciences
– High Energy Physics (HEP)
Marcel Kunze, DESY Seminar
Biomedical Applications
• Data mining on genomic databases (exponential growth)
• Indexing of medical databases (Tb/hospital/year)
• Collaborative framework for large scale experiments (e.g. epidemiological studies)
• Parallel processing for
–Databases analysis–Complex 3D modelling
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
Astronomy: Past and Future of the Universe
New phase of astronomy, storing, searching and analysing Petabytes of data:
• Virtual Observatories – GAVO, NVO, AVO, AstroGrid– Store all wavelengths, need
distributed joins– NVO 500 TB/yr from 2004
• Grid Computing might help to– Master the data streams– Federate databases with different
schema– Maintain meta data (information)
and provenance data (history)
Crab Nebula viewed
At four different
wavelengths: X-ray,
optical, infrared, radio.
Marcel Kunze, DESY Seminar
Earth Observation
ESA missions:
• about 100 Gbytes of data per day (ERS 1/2)
• 500 Gbytes, for the next ENVISAT mission (launched March 1st)
EO requirements for the Grid:
• enhance the ability to access high level products
• allow reprocessing of large historical archives
• improve Earth science complex applications (data fusion, data mining, modelling …)
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
What is Grid Computing?
Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
Marcel Kunze, DESY Seminar
Elements of the Problem
• Resource sharing– Computers, storage, sensors, networks, …
– Heterogeneity of device, mechanism, policy
– Sharing conditional: negotiation, payment, …
• Coordinated problem solving– Integration of distributed resources
– Compound quality of service requirements
• Dynamic, multi-institutional virtual orgs– Dynamic overlays on classic org structures
– Map to underlying control mechanisms
Marcel Kunze, DESY Seminar
The Grid World: Current Status
• Dozens of major Grid projects in scientific & technical computing/research & education– Deployment, application, technology
• Considerable consensus on key concepts and technologies– Open source Globus Toolkit™ a de facto standard for major
protocols & services
– Far from complete or perfect, but out there, evolving rapidly, and large tool/user base
• Global Grid Forum a significant force• Industrial interest emerging rapidly
Marcel Kunze, DESY Seminar
“Web Services”
• Increasingly popular standards-based framework for accessing network applications– W3C standardization; Microsoft, IBM, Sun, others
• WSDL: Web Services Description Language– Interface Definition Language for Web services
• SOAP: Simple Object Access Protocol– XML-based RPC protocol; common WSDL target
• WS-Inspection– Conventions for locating service descriptions
• UDDI: Universal Description, Discovery & Integration – Directory for Web services
Marcel Kunze, DESY Seminar
Open Grid Services Architecture(OGSA)
• Platform independent industrial and scientific standard to construct Grid computing components
• Service orientation to virtualize resources• From Web services:
– Standard interface definition mechanisms: multiple protocol bindings, multiple implementations, local/remote transparency
• Building on Globus Toolkit:– Grid service: semantics for service interactions
– Management of transient instances (& state)
– Factory, Registry, Discovery, other services
– Reliable and secure transport
• Multiple hosting targets: J2EE, .NET, “C”, …
Marcel Kunze, DESY Seminar
The Grid Service
• A (potentially transient) Web service with specified interfaces & behaviors, including– Creation (Factory)
– Global naming (GSH) & references (GSR)
– Lifetime management
– Registration & Discovery
– Authorization
– Notification
– Concurrency
– Manageability
Marcel Kunze, DESY Seminar
Grid Projects
Marcel Kunze, DESY Seminar
GRID Projects World Wide
• EU– EDG (EU-IST) – R&D EU GRID project [ www.edg.org ]– CrossGRID QoS – Real Time apps. [ www.crossgrid.org ]– DataTAG GLUE (EU-USA) [ www.datatag.org ]– LCG The LHC Computing GRID – Deployment [ cern.ch/lcg ]– The new 16,2 B Euro EU VI Framework Prog. GEANT based GRID projects
• USA– GriPhyN iVDGL-VDTv1 PPDG ( NSF, DoE )[ www.griphyn.org ] [ www.idvgl.org ] [ www.ppdg.org ]
• Asia– ApGrid Pragma (USA-Asia)[ www.apgrid.org ]
Marcel Kunze, DESY Seminar
GRIDLAB
GRIA
EGSO
DATATAG
CROSSGRID
DATAGRID
Applications
GRIP EUROGRID
DAMIENMiddleware
& Tools
Underlying Infrastructures ScienceIndustry / business
- Links with European National efforts
- Links with US projects (GriPhyN, PPDG, iVDGL,…)
IST Grid Project SpaceA
. B
axevanidis
Marcel Kunze, DESY Seminar
The EU DataGrid Project
• 21 Partners• 9.8 M Euros EU funding over 3 years• 90% for middleware and applications
(Particle Physics, Earth Obs. and Biomedical)• Three year phased developments & demos
(2001-2003)• Spin-off:
– DataTAG (2002-2003)
– CrossGrid (2002-2004)
– GridStart (2002-2004)
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
Research and Academic Institutes•CESNET (Czech Republic)•Commissariat à l'énergie atomique (CEA) – France•Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)•Consiglio Nazionale delle Ricerche (Italy)•Helsinki Institute of Physics – Finland•Institut de Fisica d'Altes Energies (IFAE) - Spain•Istituto Trentino di Cultura (IRST) – Italy•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany•Royal Netherlands Meteorological Institute (KNMI)•Ruprecht-Karls-Universität Heidelberg - Germany•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands•Swedish Research Council - Sweden
DataGrid Partners
Industrial Partners•Datamat (Italy)•IBM-UK (UK)•CS-SI (France)
Main Partners•CERN – International (Switzerland/France)•CNRS - France•ESA/ESRIN – International (Italy)•INFN – Italy•NIKHEF – The Netherlands•PPARC - UK
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
EU DataGrid Project Objectives
• Use Grid technology to develop a sustainable computing model for effective share of computing resources and data for large scientific communities
• Specific project objectives:– Middleware for fabric & Grid management
– Large scale testbeds
– Production quality demonstrations
• Key products: – Resource broker (Compute elements, storage elements)
– Replica manager
– Virtual Organization manager
Marcel Kunze, DESY Seminar
EDG: Interaction with Sister Projects
• CrossGrid– Using the same security certs.– Testbed sites install EDG software
• Extending it for needs of intensive interactive applications
– Participating in the EDG testing activities
– Representatives in each projects architecture & management groups
• DataTAG (EDT)– EDT is deploying EDG sw to
investigate inter-operability with US projects (iVDGL, GriPhyN, PPDG)
– Results feedback into EDG software releases (e.g. GLUE compatible information providers/consumers)
• NorduGrid– Using the same security certs.– Involved in EDG architecture work
• Good ideas for gatekeeper and MDS configuration
• Helped develop GDMP and GSI extensions for Replica Catalog
• Involved in GLUE schema work• Security policy
– Middleware testing– Working in WP8 (HEP applications)
• iVDGL/GriPhyN/PPDG– US members in EDG architecture
group– Looking for common packaging and
toolkit usage solutions
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
CrossGrid: An EDG Partner
• New category of Grid enabled applications– Computing and data intensive
– Distributed
– Interactive, near real time response (a person in a loop)
– Layered
• New programming tools• Grid more user friendly, secure and efficient• Interoperability with other Grids• Implementation of standards
Marcel Kunze, DESY Seminar
CrossGrid Layered Architecture
ApplicationsAnd Supporting
Tools
ApplicationsDevelopment
Support
Grid Common Services
Grid Visualisation
Kernel
Grid Visualisation
Kernel
DataMining on
Grid
DataMining on
Grid
Interactive Distributed
Data Access
Interactive Distributed
Data Access
Globus Replica
Manager
Globus Replica
Manager
Roaming Access
Roaming Access
Grid Resource
Management
Grid Resource
Management
Grid Monitoring
Grid Monitoring
MPICH-GMPICH-G
Distributed Data
Collection
Distributed Data
Collection
User Interaction Service
User Interaction Service
DataGridReplica
Manager
DataGridReplica
Manager
Datagrid Job
Manager
Datagrid Job
Manager
GRAMGRAM GSIGSIReplica CatalogReplica Catalog GASSGASSMDSMDS GridFTPGridFTPGlobus-IOGlobus-IO
Resource Manager
Resource Manager
CPUCPU
ResourceManagerResourceManager
Resource Manager
Resource Manager
Secondary Storage
Secondary Storage
ResourceManagerResourceManager
Scientific Instruments
(Medical Scaners, Satelites, Radars)
Scientific Instruments
(Medical Scaners, Satelites, Radars)
ResourceManagerResourceManager
DetectorLocal High
Level Trigger
DetectorLocal High
Level Trigger
ResourceManagerResourceManager
VR systems(Caves,immerse desks)
VR systems(Caves,immerse desks)
ResourceManagerResourceManager
Visualization tools
Visualization tools
Optimization of Data Access
Optimization of Data Access
Tertiary StorageTertiary Storage
Local Resources
BiomedicalApplicationBiomedicalApplication
PortalPortalPerformance
AnalysisPerformance
AnalysisMPI
VerificationMPI
VerificationMetrics and BenchmarksMetrics and Benchmarks
HEP High LevelTriggerHEP High
LevelTriggerFlood
ApplicationFlood
ApplicationHEP Interactive Distributed Data
Access Application
HEP Interactive Distributed Data
Access Application
HEP DataMining on Grid
Application
HEP DataMining on Grid
Application
WeatherForecast
application
WeatherForecast
application
Marcel Kunze, DESY Seminar
Interactive Treatment Planning
The vascular geometry can be modified using a library of models
Draw interactively Computational geometry
Marcel Kunze, DESY Seminar
Biomedical Application Sample pulse flow simulation
Marcel Kunze, DESY Seminar
Through links with sister projects, there is thepotential for a uniform global scientific applications grid
Benefit of StandardizationF
abrizio G
agliardi
Marcel Kunze, DESY Seminar
Common WorldGrid Demo at IST2002
• WorldGrid Demo together with US and EU partners at IST2002• Seamless interoperation of EDG with US Middleware (via GLUE)
VO centric Ganglia monitor
Marcel Kunze, DESY Seminar
WorldGrid Lessons (+)
• Advantages of the Grid:– Possibility to execute tasks and move files over a
distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords)
– Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system
– Possibility to do worldwide production in a perfectly coordinated way, using identical software (RPMs), scripts and databases
• GLUE interfacing works to make the middleware worlds talk to each other. Better solution: Agree on open standards environment like OGSA !
Marcel Kunze, DESY Seminar
WorldGrid Lessons (-)
• EDG stability very much dependent on the quality of the basic building blocks (Globus, Condor, etc)
• Globus support needs to be strengthened (formal contract, European support center, internal Globus support team)
• Understand impact of OGSA and industrial involvement– EDG Toolkit might need to be refactored
Marcel Kunze, DESY Seminar
EDG Future Plans
• Concentrate on production quality and real applications
• Educate new users and disseminate results
• Complete the program of work till end of 2003
• Port EDG to other platforms than Linux (Solaris)
• Port EDG to Globus ToolKit version 3 (OGSA)
• Make plans to conserve momentum and assets in the EU FP6
Marcel Kunze, DESY Seminar
A Proposal to the EU
• EDG has already demonstrated the viability of Grid technology
• EU had a fast start in supporting 18 Grid projects in the last two years!
• RN Geant offers an excellent basis for a large European Grid infrastructure
• The EU FP6 program should encourage and support the deployment and production quality operation of a large international Grid infrastructure open to research and industry in Europe
Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
Enabling Grids and e-Science in Europe
• EGEE: Integrated Infrastructure Initiative to support European Research Area
• Vision: to create and deploy Grid technologies to enable the widespread uptake of e-Science applications throughout the European Research Area
• Four key objectives:– integrating Grid technological developments from across Europe;– establishing a Europe-wide Grid infrastructure for science and industry
with a focus on heterogeneity and interoperability;– enabling the creation of e-Science applications from across the scientific
and industrial spectrum; – ensuring the timely delivery of the project’s programme of work, guided
by the needs of academic and industrial partners.
Start by integration of the national Grid initiatives Fabrizio
Gagliardi
Marcel Kunze, DESY Seminar
GRID Computing Center:
Infrastructure and Services
GRID Computing Center GridKa
Competence Center:
Applications and e-Science
Marcel Kunze, DESY Seminar
Partners in the HGF24.000 employees, 2.5 Billion Euro yearly budget
Grid Computing perfectly well matches the HGF vision of “Konzertierte Forschung”
• ALFRED-WEGENER-INSTITUT FÜR POLAR- UND MEERESFORSCHUNG AWI• DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY• DEUTSCHES KREBSFORSCHUNGSZENTRUM DKFZ• DEUTSCHES ZENTRUM FÜR LUFT- UND RAUMFAHRT DLR• FORSCHUNGSZENTRUM JÜLICH FZJ• FORSCHUNGSZENTRUM KARLSRUHE FZK• GESELLSCHAFT FÜR BIOTECHNOLOGISCHE FORSCHUNG GBF• GEOFORSCHUNGSZENTRUM POTSDAM GFZ• GKSS-FORSCHUNGSZENTRUM GEESTHACHT GKSS• GSF-FORSCHUNGSZENTRUM FÜR UMWELT UND GESUNDHEIT GSF• GESELLSCHAFT FÜR SCHWERIONENFORSCHUNG GSI• HAHN-MEITNER-INSTITUT BERLIN HMI• MAX-PLANCK-INSTITUT FÜR PLASMAPHYSIK IPP• MAX-DELBRÜCK-CENTRUM FÜR MOLEKULARE MEDIZIN MDC• UFZ-UMWELTFORSCHUNGSZENTRUM LEIPZIG-HALLE UFZ
Marcel Kunze, DESY Seminar
A German Grid InitiativeD-GRID
• Initially driven by the HGF centers and the DFN
• Open to accept further partners in academia and industry
• Aim at a coordination of Grid activities
• Deployment of a new generation networking infrastructure (towards “TeraGrids”)
• Promotion of open standards for interfaces and protocols (GGF)
Stuttgart
Leipzig
Berlin
Frankfurt
Karlsruhe
Garching
Kiel
Braunschweig
Dresden
Aachen
RegensburgKaiserslautern
Augsburg
Bielefeld
Hannover
Erlangen
Heidelberg
Ilmenau
Würzburg
Magdeburg
Marburg
Göttingen
Oldenburg
Essen
St. Augustin
Rostock
Global UpstreamHamburg
10 Gbit/s2,4 Gbit/s2,4 Gbit/s622 Mbit/s
Marcel Kunze, DESY Seminar
Scientific Program
• Installation of a coordination office for Grid Computing• Foster the enhanced application of IT in the field of
scientific and engineering disciplines• Management of the relationship between the German
Grid competence centers and representation of their common interests in the Global Grid Forum
• Training and education in the field of Grid Computing and e-Science
• Deployment of national Grid resources and development of a program for academic and industrial projects
Marcel Kunze, DESY Seminar
Technical Program
• Goal: Integration of all kind of resourcese.g. sensors, computing infrastructure, people,…
• Deployment of a state-of-the-art production backbone (2003: 10 GBPS, 2006: 100 GBPS)
• Discussion of safety and trust relationship• Management of German Grid certificates• Selection of middleware components and installation procedures
(Globus, UNICORE)• Infrastructure: Find qualified manpower to offer and support the
additional services• Definition of cost model (Grid-Accounting, credit points)• Collaboration with the German HPC centers
=> D-GRID Kickoff Workshop in January 2003 (Wissenschaftszentrum Bonn)
Marcel Kunze, DESY Seminar
Where do we go from here??
Marcel Kunze, DESY Seminar
e-Science (enhanced Science)Konzertierte Forschung
• e-Science is about more than networks, GRIDs, High Performance Computing, cluster computing ...
• e-Science: „e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.“ (Dr. John Taylor, director of UK research council)
• Foster the transition of data Grid to semantic Grid– Federation of resources
– Federation of distinguished data sources
– Knowledge retrieval and collaboratories
Marcel Kunze, DESY Seminar
Knowledge Grid
Semantic Grid: Three Layer Grid Abstraction
Information Grid
Computation/Data GridDat
a to
Kno
wle
dge
Con
trol
Automation
e-Science
Marcel Kunze, DESY Seminar
Grid-enabled Applications
Prototype Grid Infrastructures
Gèant: World Class Networking
• Where do we need to get to ?– Applications to support an “e-society”
(“Cyber-Infrastructure”)
– An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”)
– A powerful and flexible network infrastructure
• Where do we need to invest ?– Applications targeted at realistic problems
in “e-science”
– Prototypes of Grid infrastructures
– Maintain and improve the GEANT network
The Future