technische universität münchen e-science in germany the d-grid initiative iatul meeting london...

Technische Universität München

E-Science in Germany

The D-Grid Initiative

IATUL Meeting London November 2008

INTERNATIONAL ASSOCIATION OF TECHNOLOGICAL UNIVERSITY LIBRARIES

TECHNISCHE UNIVERSITÄT MÜNCHENREINER KALLENBORN


• Solitary activities–UNICORE projects (expired)–DFN development programme (expired 2001)–Sporadic consortia and projects of individual persons or institutes–LHC-Grid at Tier1-Centre Karlsruhe



E-Science in Germany up to 2003 (1) Hegering et al. 2004


• Grid requirements do not stop at national boundaries• German scientists are part of international communities• German scientists depend on access to international resources (computers, data,

experiments)

• No national initiative comparable to UK, USA, NL• No national effort for a grid-infrastructure• There is no grid without standards• There is no use for each scientific community reinventing all grid aspects• Germany shouldn’t sleep away chances of the grid



E-Science in Germany up to 2003 (2) Hegering et al. 2004


• Since 2003: several workshop initiated by the Research Centre Karlsruhe and the Helmholtz Association of German Research Centres

– All relevant scientific bodies involved, including• DFN - Deutsches Forschungsnetz (Germany‘s National Research and

Education Network)• BMBF – Federal Ministry of Education and Research• Industry

– Consensus on the necessity to bundle all isolated Grid-activities in Germany – Consensus on the necessity of a national support programme



D-Grid: development Hegering et al. 2004


• Establishment of an efficient e-science framework• Bundling of activities in order to trigger synergy potential for global, distributed and

new scientific collaboration based on internet services• D-Grid Strategy Paper and Policy Statement

– discussing status and consequences of the grid technology on scientific research and recommending a long-term strategic grid research and development initiative

– resulting in an announcement of the German e-Science Initiative by the German Ministry for Research and Education (BMBF) in March 2004, together with a call for proposals in the areas of Grid Computing, e-Learning, and Knowledge Management

– Demanding (material) committment of scientific communities

June 15th, 2004: Initiation of a BMBF D-Grid support programme allocating 300 million Euro within 5 years



D-Grid: Objectives Hegering et al. 2004




E-Science: Framework


• Steering/Executive Board– The steering board is built by the project leaders of the community projects

and the work package leaders of the D-Grid integration project. The chair of the steering board is the D-Grid project coordinator (Wolfgang Gentzsch). The steering group meets regulary every month.



D-Grid: Organization D-Grid.de 2008


• Elements– D-GRID competence network and infrastructure

• integration of partners with their resources based on a high-performance backbone

• seamless access to resources by uniform interfaces– Grid-software initiative (from 2004 - 2005)

• homogenization and standardization of grid middleware• network- and application-specific service development (QoS)• e-science pilot projects and development of “Services for Scientists”

(S4S), i.e. self-supporting services in the long run• Prerequisite: Cooperation of science and economy

– Projects with partners from science and economy are deeply recommended



D-Grid: Programme (1) Schlie-Roosen et al. 2003


• D-Grid 1 (2005 – 2008)– Homogenization and qualification of the Grid-infrastructure by prototypical

applications– generic middleware development (toolkit for e-science)– IT services for scientists, designed and developed by the ‘early adopters’ of

the computer science and scientific computing communities– This global services infrastructure is being tested and used by so-called

Community Grids in the areas of highenergy physics, astrophysics, alternative energy, medicine and life sciences, earth sciences (e.g. climate), engineering sciences, and scientific libraries

– science and scientific computing communities

• D-Grid-1 consists of the DGI Integration project and the following seven Community Grid projects

– AstroGrid-D (Astronomy)– C3-Grid (Earth Sciences)– HEP Grid (High-Energy Physics)– InGrid (Engineering)– MediGrid (Medical Research)– TextGrid (Scientific Libraries)– WISENT (Knowledge Network Energy Meteorology)



D-Grid: Programme (2) Schlie-Roosen et al. 2003 / BMBF 2008


• D-Grid 2 (2008 - 2010)– stronger focus on grids with industry participation, with more than

40 companies involved– National Grid-Infrastructure on production level– establishing of e-science applications and services in

• Construction industry• Finance• Aerospace• Automotive• enterprise information• resource planning systems• geographical data• general IT services



D-Grid: Programme (2) Schlie-Roosen et al. 2003 / BMBF 2008


• D-Grid 3 (2008 - )– will extend the grid infrastructure with

• a Service Level Agreement (SLA) and • a knowledge management layer• adding several virtual competence centres• encouraging global service-oriented architectures in the industry• and using this grid infrastructure for the benefit of our whole society

April 22nd, 2008: BMBF Call for proposals



D-Grid: Programme (3) BMBF 2008


Integration Project (DGI-2) AeroGridAstroGrid-DBauVOGrid BIS-GridBiz2GridC3 GridD-MONF&L-GridFinGridGDI-GridHEP GridInGridIVOMMediGRIDPartnerGrid ProGRID SuGITextGridWISENT



D-Grid: Projects D-Grid.de 2008


• Integration Project DGI - 1 (2006 – 2007)

– establish a core Grid infrastructure in Germany on which existing community Grid projects are based

– is to be the foundation also of other community projects

• Integration Project DGI – 2 (2008 – 2010)

– put this infrastructure on a sustainable basis for long-term use

• work package 1: Support of Grid users and resource vendors• work package 2: Operation of D-Grid infrastructure• work package 3: Security• work package 4: Management of data and information• work package 5: Development projects• work package 6: Sustainability and coordination



D-Grid Projects: Integration Project (1) BMBF 2008


• Objectives– establishment and operation of a scalable Core D-Grid infrastructure by

the D-Grid integration project (DGI)– evaluation of interoperability and performance of resources– Build a grid infrastructure so everybody can join – integration of new communities

• The Core D-Grid comprises– Acquisition and integration of compute and storage resources– Implementation and provision of middleware (Globus, gLite, UNICORE)– Attachment of storage robots to the storage element of the D-Grid

software (dCache, SRM/SRB)– Admission of users to resources and applications (AA)– Operation of a virtual Grid Operating Center (GOC)

• Use of Standard Grid Services– Globus 2.4.3 (gLite) and 4.0.3, Unicore 5, dCache, SRB/iRODS, OGSA-DAI,

GridSphere, GAT, VOMS and Shibboleth



D-Grid: D-Grid Projects: Integration Project (2) Core D-Grid Infrastructure Gentzsch 2007


• AeroGrid requirements– permanent and effective Grid infrastructure for the cooperation between

industry, research centres, and universities in aerospace engineering and research

– Suitability for daily use, especially for the productive operation by the service provider after the end of the project

– Applicability to similarly organized communities in science and industry

• AeroGrid usage scenarios from the turbine simulation domain – Usage of compute resources via the AeroGrid: Today in the aerodynamic

design of engine components, CFD calculations for configurations with more than 30 blade sections and over 30 million grid points are being conducted. Usually, the necessary computing power is not available locally, so that the use of Grid technology to access remote computing resources is required

– Collaboration during the design of new engine components: In industrial design projects involving up-to-date simulation codes, sometimes support by developers of these codes is necessary. To support this, AeroGrid will provide the possibility to jointly conduct simulations in a multi-site virtual organization sharing the same data

– Cooperative further development of the DLR CFD code TRACE: The further development of the CFD software system TRACE is carried out in close cooperation between the DLR Institute of Propulsion Technology and a couple of universities. AeroGrid environment will be used to share a common code basis between all development partners



D-Grid Projects: AeroGrid D-Grid.de 2008


• AstroGrid-D - The German Astronomy Community Grid (GACG)– Joining together astronomical science institutions into a unified nationwide

research infrastructure of collaborative distributed environments using innovative grid technologies

– Creation of a grid based infrastructure for astronomical and astrophysical research, based on integration of their computational facilities, improving the efficiency and usability of their hardware resources

– Integration of distributed astronomical data archives and, in an extended time frame, of instruments and experiments into a common research infrastructure

– Support of other research institutes to enable them to share their resources, data and application software within the AstroGrid infrastructure, especially to enable small university workgroups to join the collaboration

– Strengthening the ties of the German astronomy community to the huge developments within the international activities of the astronomy community



D-Grid Projects: AstroGrid-D D-Grid.de 2008


• BauVOGrid – a Grid-based Platform for the Virtual Organisation in Construction

– controlled representation of responsibility/authorisation structures – fast configuration and management of both global VO processes and local

company-specific processes in accordance with the set up responsibility VO structure

– fast, flexible and secure access to information from different sources (documents, drawings, photographic material etc.) both from headquarters and from the construction site

– ad-hoc changes in the process flow, using semi-automatic process simulation – mobile capturing of processes and process/product data on the site, thereby

providing a basis for faster and more efficient decision making with participation of all site workers



D-Grid Projects: BauVOGrid D-Grid.de 2008


• BisGrid - Grid-based Integration and Orchestration of Business Information Systems

– developing and providing organisational and technical extensions of existing Business Information Systems based on the current state of the art in Grid technologies, EAI, and SOA

– On the organisational layer: develop new forms of inter-organisational collaboration and new business models

– On the technical layer: extend WS-BPEL-based orchestration to allow the orchestration of stateful WSRF-based Grid services

– integrate existing middleware systems that are already well-established in the commercial sector (JBOSS, UNICORE 6)

– application scenarios: evaluate the project results exemplarily for the integration of ERP, PDM, and CRM systems

– Summarizing: intent is to extend the application potential of Grid technologies to explicitly support SMEs in business information processing



D-Grid Projects: BisGrid D-Grid.de 2008


• Bis2Grid - a price based scheduler to the D-Grid community and to other related project

– provide foundations for an effective application of Grid technologies in enterprises

– design and implementation of the billing and pricing component – business-models have to be developed that are adequate for an application in the

Grid – distribute and parallelize real world applications and– conceptualize and implement economic business models at the same time

• Project partners– IBM Deutschland Entwicklung GmbH – Universität Karlsruhe (TH): Institut Informationswirtschaft und –management – Philipps-Universität Marburg: Arbeitsgruppe Praktische Informatik/Verteilte

Systeme des Fachbereiches Mathematik und Informatik – FZI Forschungszentrum Informatik Karlsruhe: Forschungsbereich Information

Process Engineering – Associated partner: BMW München



D-Grid Projects: Bis2Grid D-Grid.de 2008


• C3 Grid - Collaborative Climate Community Data and Processing Grid - Goals– Connection of distributed climate data archives– Uniform access and central metadata catalogue

• C3 Grid - Collaborative Climate Community Data and Processing Grid – Working Packages

– User Interface (API/GUI) – Grid - Information Service – Connection of local database-systems / Metadata Backend-Interface – Dataproviding and Preprocessing – Grid-Data Management – Coordidated Grid-Scheduling – Implementation and Integration of the C3-Grid – Sustainable Connection with Earth System Science – Project Management and Co-operation with the Integration Project



D-Grid Projects: C3 Grid D-Grid.de 2008


• D-MON - Horizontal Integration of Resource- and Service Monitoring in D-Grid

– is to realize a Grid wide monitoring architecture across several underlying, heterogeneous systems

– taking in consideration multiple resource providers and virtual organisations– horizontal integration of different grid monitoring services – provide communities and virtual organizations with multitenant VO-specific views

onto – and solely onto – their Grid resources – vertical integration within D-MON deals with the integration and provisioning of

an VO-specific information within the monitoring architecture



D-Grid Projects: D-MON D-Grid.de 2008


• F&L Grid – A grid infrastructure for research and education - goals– Tool to offer IT-services based on the network- and administrative infrastructure

of DFN (German Research Network) – The pilot sevice-offering will be Backup- and Archive services, later expanded to

Information Lifecycle Management (ILM)



D-Grid Projects: F&L Grid D-Grid.de 2008


• FinGrid – Financial Business Grid– FinGrid aims both to contribute to the literature by empirical analysis of the

potentials of Grid from an economic perspective and by the design and development of three Grid-based prototypes

• Prototype I is a pricing and billing component for Grid-based services – in cooperation with Deutsche Bank and IBM

• Prototype II is a Grid-based customer portfolio performance measurement and management tool – in cooperation with DataSynapse and Dresdner Bank

• Prototype III is an asset-backed security factory based on Grid architecture – in cooperation with FinanzIT and PA Consulting



D-Grid Projects: FinGrid D-Grid.de 2008


• SDI-Grid - Spatial Data Infrastructure Grid– Efficient mining and processing of spatial data for the simulation of noise

dispersion and disaster management– solutions for efficient integration and processing of geodata based on geo

information systems (GIS) and spatial data infrastructure (SDI) – Services for integration, processing and management of spatial data are to be

developed and implemented within the D-Grid infrastructure. – A »Proof of Concept« will be given using a number of representative scenarios

such as emergency routing for disaster management, flood simulation and sound propagation simulatio

– Services, components and practices developed as a part of this project will be provided to all participants in the e-Science infrastructure well beyond the end of the project's duration.

– By enlisting commercial partners for a feasibility study and product development, sustainable usage is being secured.

– Additionally, the prototypical implementation developed during the project will be a basis for successful commercial ventures.



D-Grid Projects: GDI-Grid /SDI-Grid D-Grid.de 2008


• HEP Grid – High Energy Community Grid– CPU and mass storage-intensive applications in particle physics require a more

flexible local, distributed and dynamic data management, a well adapted job scheduling as well as accounting and monitoring of data resource utilization.

– The e-Science User Support centre in Germany, the Tier-1 centre GridKa at the “Forschungszetrum Karlsruhe”, provides essential user support within the international Particle Physics Grid project “World-Wide LHC Computing Grid, WLCG“. The support tasks need to be automated to a large extent. Core elements for this are monitoring of jobs, error identification and the enabling of direct access to executing jobs for early control and steering.

– So far many grid tools are successfully applied to central production tasks, which are easy to schedule and predict. It is indispensable for the development of the e-Science infrastructure in Germany to develop and extend grid technologies for the individual scientific data analysis

• HEP Grid – Work packages– Scalable storage– Job monitoring– Distributed data analysis and use of grid resources



D-Grid Projects: HEP Grid D-Grid.de 2008


• InGrid – Innovative Grid Developments for Engineering Applications– enable engineering projects for grid-based application and efficient use of

common compute and software resources – Five typical applications (foundry technologies, metal forming technologies,

groundwater flow and transport, turbine simulation and fluid-structure interaction) are considered as showcases in order to cover the three central areas of computationally intensive engineering applications, that are

• coupled multi-scale problems, • coupled multi-discipline problems, and • distributed simulation-based optimization

• The community specific goals can be summarized as follows – Support of virtual prototyping by means of

• internet-based collaborative methods, • mapping of engineering workflows on a grid environment • integration of specific technical data sources and • building of portals for selected engineering scenarios.

– Development of a knowledge-based system for the optimization ofengineering workflows and support of decision-making processes

– Development of co-operation, business, safety and confidence modelsfor scientific engineering applications.



D-Grid Projects: InGrid D-Grid.de 2008


• IVOM – Interoperability und Integration of VO-Management Technologies in D-Grid – goals

– The implementation of a comprehensive VO-Management, including the integration into Globus Toolkit 4, gLite and UNICORE, enables a uniform management of users and their rights. It will be used for authentication and authorization in the Communities, independent of the utilized Grid middleware.

– The authentication of users in Shibboleth Grid environments is directly performed with the users' home organizations and therefore tightly coupled to these organizations' practices for accession and leaving. Here the home organizations will be able to define the technical approaches for the users’ authentication. The common PKI-based approaches are disburdened when using alternative approaches.

– Communities with existing user databases can provide a larger number of users access to Grid services by using Shibboleth. The previously required mass rollout of user certificates is no longer necessary.

– The administrative overhead of renewal or potential revocation of user certificates is no longer necessary. Moreover, the user does not need to renew his certificate on a regular basis any more.



D-Grid Projects: IVOM D-Grid.de 2008


• MediGrid – Grid computing for medicine and life sciences based on three representative scenarios

– Bioinformatics applications• Protein structure prediction (RNAi screening)• Sequence analysis• Ontology etc.

– Medical image processing applications• Processing ultrasonic images of prostate biopsies• Streaming simulation in blood vessels• Functional analysis of MRT brain images etc.

– Clinical research• Research on sleeping etc.



D-Grid Projects: MediGrid (1) D-Grid.de 2008


• MediGRID is realized as a consortium of eight partners– Module A: Coordination

• Telematics platform for Medical Research Networks (TMF e.V.) – Module B: Ontology tools

• University Leipzig – Module C: Resource fusion

• Konrad-Zuse-Institute, Berlin – Module D: Middleware

• Fraunhofer-Gesellschaft IAO / FIRST, Stuttgart – Module E: Bioinformatics

• Uniklinikum Schleswig-Holstein, Kiel – Module F: Image Processing

• Charité, University Medicine Berlin – Module G: Clinical Research

• University Marburg – Module H: eScience

• University Göttingen, Department of Medical Informatics



D-Grid Projects: MediGrid (2) D-Grid.de 2008


• PartnerGrid - Cooperative Grid-solutions for industrial applications– Design and production processes in the industry are based more and more on

division of labour. The companies concentrate on their core competences and buy parts, equipment and services from third parties, instead of covering the full value chain. With the choice between different potential partners the costs can be reduced and the best fitting partner can be selected

a software platform (PartnerGrid-infrastructure) will be developed based on Grid-technology, that supports efficiently the collaborative work in between virtual organisations

Showcase scenarios: foundry, metal-processing industry

• Work packages• Definition and execution of complex and distributed process workflows • Simple usage interface for the access and the usage of the software

infrastructure via portal technology • Transparent access to distributed data in the form of file systems and data

bases • Management of distributed, virtual organisations und their protection by the

realisation of integrated security concepts • Standard interfaces for the integration of complex, application specific

analysation modules • Interface to service providers for the execution of very compute intensive

application



D-Grid Projects: PartnerGrid D-Grid.de 2008


• ProGrid - Grid technology for collaborative product development - goals– utilize Grid technology for collaborative product development and– demonstrate grid advantages by means of selected use cases

• Application scenarios• Process optimization• Optimization in fluid mechanics• Multidisciplinary structural optimization• Connection between product data management software and computer aided

engineering systems• Collaboration platform



D-Grid Projects: ProGrid D-Grid.de 2008


• SuGi - Sustainable Grid Infrastructure– Training

• disseminate the knowledge of grid technology and to enhance its use • research experiences gained in the DGI project will be made available to

small and medium-sized computing centers of universities as well as to small and medium-sized enterprises

• Therefore, SuGI will offer own training courses; members of SuGI will attend to external courses, create video and audio recordings and provide these online to the D-Grid communities

• SuGI will examine the fitness of existing e-learning concepts and will apply the appropriate methods

– Maintenance• SuGI will develop tools to simplify mainenance tasks of installed grid

infrastructure and give service providers a chance to concentrate on their main goal, to provide grid services

• main aspects for the sustainability of the grid infrastructure will be investigated

• Legal aspects covering software license conditions will be investigated to enable service providers to offer a wide spectrum of grid services in Germany

• Likewise economic aspects such as profitability and standardization will be taken into account



D-Grid Projects: SuGi D-Grid.de 2008


• TextGrid - a Community Grid for the Humanities– creating a community grid for the collaborative editing, annotation, analysis, and

publication of specialist text resources – establishes a truly interdisciplinary platform and a virtual workbench for research – focussed on (annotated) text as a data type

• Work packages– Survey on the re-usability of existing tools

• This work package collates tools for processing, analysing, annotating, editing and publishing text data in academic research. Relevant tools are evaluated as to their applicability in the TextGrid specific context, and regarding their adaptability

– Tool development• This work package creates a grid-enabled workbench to process, analyse, annotate, edit and publish XML-encoded

textual data for academic research. This establishes the foundation of a virtual research library. The workbench is modular and extensible, and it aims to foster an active community for the collaborative creation and open access to text resources

– Grid Integration• The software modules created in the other work packages of the project are connected to the D-Grid Integration Platform

(DGI) via TextGrid-specific middleware components. This work package establishes this intermediary layer, and defines the interfaces between them. This demands a high level of communication to TextGrid members as well as the relevant people at the DGI project, particularly for conveying TextGrid-specific requirements to the DGI.

– Demonstrator development• Test material will demonstrate the efficiency of the grid-enabled tools in the areas publishing, processing, retrieval, and

linking. The results of the experiments with various test profiles at adequate levels of complexity will feed back into the TextGrid development process.

– Semantic Web plus TextGrid = Semantic TextGrid• TextGrid sets out to interconnect various dictionaries with rich markup to create overarching networks of words. These

networks serve as a core ontoloy in the application of semantic methods for processing text assets, and for interweaving texts and dictionaries.

– Project management and external communications• This work package takes care that all announced targets of the TextGrid project are achieved within the set timeframe

and within budget. This includes the formal and administrative coordination of the project. Another focus of this work package is the publication of project results on a national and international level, particularly in communication with the target community



D-Grid Projects: TextGrid D-Grid.de 2008


• WISENT - a Knowledge Network in Energy Meteorology – IT-optimised cooperation in research and development of organisations in the

area of energy meteorology – Development of electronic services for analysis and visualization of large

datasets in atmopheric research and meteorology– Development of new collaborative methods for the optimization in the use of

renewable energy resources like solar and wind energy



D-Grid Projects: WISENT D-Grid.de 2008


• eSciDoc - a flexible infrastructure for management and storage of cultural heritage data

• HyperImage – Image oriented e-science networks• Im Wissensnetz (in the knowledge net) - support interdisciplinary knowledge creation

processes by means of an eScience Semantic Desktop in the field of the rapid prototyping high-technology

• Instant-Grid - a Knoppix-based live-CD bundled with a pre-configured grid-environment based on the Globus-Toolkit

• Ontoverse - knowledge management in the life sciences network. It aims at the development of a new, internet-based application for cooperative and interdisciplinary ontology building in terms of an ontology-wiki

• StemNet – knowledge management for stem cell biology• Wikinger - WIKI Next Generation Enhanced Repository - an interdisciplinary research

project involving computer scientists, engineers and historians who will develop new questions, methods and processes of web-based knowledge generation and/or to develop them to a higher level



D-Grid Projects: Further Projects D-Grid.de 2008


• On January 1 2008, the D-Grid Company (D-Grid Development and Operation GmbH) has been founded

– The main objective of the D-Grid GmbH is to support development, operation and maintenance of a sustainable Grid Infrastructure for German research, industry and commercial users, in a professional way

• In more detail, main tasks of the company are:– Coordination of all D-Grid projects funded by the German Government– Development and implementation of Grid-Economy models for financing a

sustainable operation of the open grid infrastructure– Support of research projects to investigate new technology developments in grid

and gridrelated fields– Collaboration in national and international standards organizations– Planning of a Grid Operation Center to coordinate operation of grid resources– Consulting and training for grid users– Implementing a set of maintenance and support services (GERT – Grid Emergency

Response Team)– Organization of technical workshops and conferences in the grid field– Marketing communications to create awareness and attract new user

communities



D-Grid Projects: D-Grid Operating Company D-Grid.de 2008


• One of the cornerstones of the D-Grid initiative: Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities

• Interfacing institutional repositories (OAI)• Digitization as a prerequisite for access to information• Location- and time-independent services• Electronic Reference Services - Further development and cooperation• Growing importance of information literacy• Cooperation between library and faculty• Growing importance of metadata standards• Data archiving and preservation• Software development by libraries - we need appropriate interfaces to commercial

integrated library management systems• etc.



Grid and Libraries




Thank you!

Join IATUL!

and


• H,-G. Hegering: D-Grid Initiative – Statusberichthttp://www.rv-nrw.de/content/koop/workshops/20040601.dgrid/Hegering_DGrid_Statusbericht_ZKI_Mar04.pdf

• W. Gentzsch: 2007D-Grid and the Integration of New Communitieshttp://www.ogf.org/OGF20/materials/783/OGF20-D-grid.pdf

• F. Schlie-Roosen et al.: German Steps to e-Sciencehttp://www.pd.infn.it/einfrastructures/eirgSlides/German_View_escience.pdf



References

technische universität münchen e-science in germany the d-grid initiative iatul meeting london...

Documents