the german grid initiative a platform for e-science in germany dr. harald kornmayer

Dr. Harald Kornmayer D-GRID CGW 05, 22-Nov-2005

The german Grid initiativeA platform for e-Science

in GermanyDr. Harald Kornmayer

Institut for Scientific ComputingForschungszentrum Karlsruhe

for the whole D-Grid initiative

Dr. Harald Kornmayer D-Grid CGW 05, 22-Nov-2005

2

Agenda

• e-Science in Germany• D-Grid initiative• Community projects• D-Grid Integration project• Summary and Outlook


3

Grid projects in the past

• Many german research institutions participated in various Grid projects

• Germany was strongly involved in the development of UNICORE middleware

and much more….

• National Grid initiative were started by many countries (small large)

• But Germany was still a „white“ area on the Grid map


4

D-Grid – 2001 to 2004

• Some activities on national level: – 2001: At Forschungszentrum Karlsruhe

GridKa is launched as a large Grid computing center for LHC computingwww.gridka.de

– 2002: The German Ministry for Education and Research (BMBF) disseminated "IT-2006", a funding programme for enhancing the Information Technology in Germany www.it2006.de

– 2002-2003: The idea of a national Grid emerged from Forschungszentrum Karlsruhe – including the name “D-Grid” - and many institutions joined this initiative www.d-grid.de


5

D-Grid to e-Science

• ~300 scientists from ~100 institutions contributed• All aspects of e-Science discussed

A „white paper“ was presented to the ministry

„e-Science in Germany“• This paper became the base for the BMBF funding

programme

„e-Science“ covers three areas: - Grid computing- Knowledge management- e-Learning


6

Knowledge management & e-Learning

• Knowledge Management– IT based handling and management of information and

knowledge– Open access to scientific information– eSciDoc (FIZ Karlsruhe & Max-Planck-Society)

• 6 million Euro, 09/2004 - 08/2009

– More projects in the pipeline (Ontoverse, WISENT, …)

• e-Learning– e-Learning = Using new media in university teaching– Seeking methods for enhancing innovation potential given

by e-Learning and e-Teaching techniques within universities

• 40 million Euro, 01/2005 - 12/2007

a general platform needed D-Grid


7

Generic platform and generic Grid services

e-Science architecture

Grid Computing Knowledge management

e-Learning

Ont

over

se

Wik

inge

r

WIS

EM …

Ast

rono

my

Clim

ate

Life

scie

nce

…

Hig

h en

gery

phy

sics

Eng

inee

ring

D-Grid integration project

Pro

ject

1

Pro

ject

2

…

e-Science


8

D-Grid – goals

The D-Grid initiative should: • provide very soon a nationwide reliable, sustainable and robust

Grid infrastructure – build around the “Kern-D-Grid“ (Core-D-Grid)

• develop this “Kern-D-Grid” to an infrastructure for all e-Science applications

• demonstrate the usability of Grids for (at least) six different communities

• Contribute in the development of Grid middleware and tools by integrating the needs of the scientific communities

• (re-)use existing technical and middleware solutions from the international context (e.g. GGF, EGEE, UNICORE)

• …..

• Longterm goal: "Services for Scientists"


9

D-Grid - projectsC

om

munit

y

Com

munit

y

Gri

ds

Gri

ds

Inte

gra

tion

Inte

gra

tion

Pro

ject

Pro

ject

Generic Grid Middleware and Grid Services

Grid specificDevelopments

High Energy Physics

CGMiddle-

ware





AstrophysicsMedicine and Lifesciences

Climate Research

EngineeringApplications

CGMiddle-

ware

CGMiddle-

ware

CGMiddle-

ware

CGMiddle-

ware


Social Science

CGMiddle-

ware

….

Today: 7 projects• 6 Community projects:

• Astropyhsics, Climate research, Medicine, Engineering, HEP, social science

• D-Grid integration project (DGI)• to build a general platform for e-Science in Germany


10

AstroGrid-D

Goals:

• Creation of an nation wide collaborative environment for astronomical research institutions using Grid technology

• Setup and operating of a Grid infrastructure for the Astro-Community to optimize the usage of the existing resources

• Establishing the access to distributed astronomical data archive

• Integration of astronomical instruments and experiments in the reseach infrastructure

• Close collaboration with the international Virtual observatory


11

Workpackages: • Management• Resource integration• Meta data management• Distributed data management• Distributed data base access

and distributedd data flows• Resource management for

Grid-jobs• Job-monitoring and

interaktive job steering• User interface and Grid

application toolkit

AstroGrid-D

Technology:• Gridshpere• Workflow Engine from the Planck

satellite mission (Process Coordinator ProC)

• Cactus• …

International Projects:• Planck satellite• Lofar – radio astronomy• VO - Virtual Observatories• LIGO – Gravitational waves• Numerical relativity• ….


12

C3-Grid

C3-Grid = Collaborative Climate Community Grid

Goals:• Access to distributed climate data archives• Establishing scientfic workflows in earth science

community• Setup of a productive Grid infrastructure for climate

research in Germany• Integration of existing heterogenous systems

(computing and data)• Enable global cooperation


13

C3-Grid

Scientific Workflows: • Modelling of earth systems

(Climate, Oceans, Biosphere)• Acquisition of measuring data

(classical measurement, satellites)• Comparisons of

• models and measurements• model A and model B

• Access to Data essential!!• Meta data • distributed archives

• Long term availability of data must be guaranteed!!

Workpackages: • User interfaces• Grid Information services• Access to distributed local

databases/meta data bases• Preprocessing of data• Grid data management• Grid scheduling• C3 Grid infrastructure• Substainable integration• Project management


14

HEP-Grid

Goals:• Development of applications and components for the analysis of

data of international high energy physics and astroparticle physics experiments in a national Grid environment

• Integrating with the exisiting Tier-structure of LCG

• Development of components for data analysis using Grid resources

• Strengthen the participation in international Grid projects– Use of the gLite middleware from the EGEE project

• Three main focus: – Data management

– Tools for job monitoring and automatic support

– Distributed data analysis by end users („physisist“)


15

InGrid

• Innovative Grid developments for engineering science applications

• Goals:–Common use and community specific development of

Grid environments for engineering applications–Efficient use of common resources for:

• Modelling• Simulation• Optimisation

–Basic and application research–Strong cooperation with industry


16

InGrid

Applications: Five different engineering domains•Foundry industry

•Metal forming

•Flow physics

•Transportation

processes in ground

water

•Magneto-hydrodynamics

coupling

Methods and models:Engineering specific problems should be solved on the Grid•Knowledge based support for decision

processes

•Support for engineer specifc

workflows

• distributed simulation based

optimisation of products and processes

Grid specific developments:• Security- and Trust models in economic

scenarios

•Cooperation- and Business models

•Licensing


17

MediGrid

Goals:• Demonstrate the usability of Grid services for

medicine and life science by connecting large and high dimensional data sets

• Setup of a Grid environment for medical research

• Transfer of project results into the generic D-Grid platform

• MediGrid is open for new medical applications


18

MediGrid

Workpackages:

• Project management• Ontology tools• Resource fusion• Middleware• Enhanced Trust and

Security• Medical e-Science

framework •Legal aspects•Biometric aspectsof highly dimensional data

DGI

C Ressourcenfusion

B Ontologie

A K

oord

inie

run

g

MetadataData

MetadataData

MetadataData

Zugriffskontrolle

D Middleware

...

H e

Sci

en

ce,S

ich

erh

eit

, In

tern

ati

on

alit

ät

E

Biomed. Informatik

F

Bildver-arbeitung

G

Klinische Forschung

E

Klassische BioInformatik

Nach Machbarkeit gepipelinede Projekte

Weitere, z.B. Forst-

oder Geistes-Wissen-schaften

• Clinical research• Biomedical informatics• Image processing


19

TextGrid

Goals:• Establish a community Grid for text based

science (social science) to support– tools for edition workd– modules for text data processing – the definition of standardized interfaces to

publication software– modules to controll and monitor access to text data

and text data tools• TextGrid will be a part of Semantic Grid • Funding just approved

– From a different department in ministry


20

DGI - D-Grid Integration project

Integration of • new developments from the D-Grid communities • developments of other projects (UNICORE, EGEE,...)

to establish a generic platform for e-Science which • contains the software stack needed by the communites• provides a reliable and sustainable Grid infrastructure Kern-D-Grid

• offers Grid services for new application domains

to reach the longterm vision of

„Grid, knowledge and e-Learning services for Scientists“


21

DGI – Integration procedure

Idea: • development of new and

enhanced services in the Communtiy Grid (CG) projects

• Generalisation of services and integration into the common D-Grid platform will be done by the DGI together with the CGs

Inte

gra

tion P

roje

ctIn

tegra

tion P

roje

ct

Com

mun

ity O

ne

Com

mun

ity T

wo

Com

mun

ity T

hree

Working Groups: • DGI will coordinate different groups together with the CGs

• Grid scheduling• Data & Meta data management• VO management• AAI • Security

Working Group 1

Working Group 2

Working Group 3


22

DGI – FG1: Middleware & tools

Middleware:• Globus 4.x• gLite (LCG)• UNICORE• GAT and GridSphere

Data Management:• SRM/dCache• OGSA-DAI • Meta data schemas

VO Management:• VOMS and Shibboleth

Tasks:• Support communites

– with training– with installation packages– with support for new operating

systems

• Coordinate together with communites the development of new tools

– Prevent „reinventing the wheel“


23

DGI - Infrastructure

Kern-D-Grid- Integrate all participating

resources and providers- Operate the Kern-D-Grid

- including User Support

- Provide monitoring and accouting tools for all middleware platforms

- prepare the accounting and – later – billing

- Contribute to the development of a general resource description language

- Offer services for new communites

FZJ/ZAM IBM-Supercomputer mit 8,5 TFlopsSTK-Datenroboter mit 2,2 PByte

32 CPUs300 TByte

FZK/IWR 8 Knoten Opteron 2x2.2 GHz 8 Prozessoren einer NEC SX-5 eine p630 mit vier Prozessoreneine SX-6i für Tests2 Knoten Opteron 2x2.2 GHz für Tests

100%50%50%50%50%

LRZ SGI-Höchstleistungsrechner mit ca. 20 TFlop/s, Intel IA32 und IA 64-Cluster, IBM p690, SunFire 80

5%von allen

MPI/RZG IBM Supercomputer mit 4,5 TFlops, PC-Cluster mit 2 TFlopsDatenroboter mit 8 Pbyte

32 CPUs400 TByte

PC² Cluster mit ca. 400 Xeon 64 Bit CPUs, Hochleistungsvisualisierung und FPGAs

10%

RWTH/RZ 2 SunFire 6900 mit je 24 UltraSPARC IV 100%

TU-Dresden/ZIH

SGI O2K(56 Proz)/O3K(192 Proz) : T3E (64 Proz):PC-Cluster mit ca. 30 CPUs, ab 2005 neues System mit 1000 Prozessoren

10%20%20%2%

Uni-H/RRZN PC-Cluster mit 64 CPUs assoz.

Uni-KA PC-Pool assoz.

FHG/ITWM assoz.

PC²

RRZN

TUD

RZG

LRZ

RWTH

FZJ

FZK

FHG/ITWM

Uni-KA


24

DGI – Network & Security

Goals:• Extend the existing German network by Grid

specific elements• Evaluate alternative network transport

protocols• Build and operate an infrastructure for

authorisation and authentication (AAI)• Evaluate and/or develop Grid suitable firewall

concepts• Provide a Grid specific CERT team

– CERT = Computer Emergency Response Team)


25

DGI – Management& Sustainablity

Tasks:• Coordinate the integration

procedures within the DGI• Coordinate the

cooperation between the DGI and the community projects

• Establish connections between D-Grid and knowledge management/e-Learing

• Support new applications domains to the Grid

• Evalute methods for facilitating sustainability

Sustainablity: • Take the different roles

into account

• evaluate the legal framework concerning sustainability

SUSTAIN-ABILITY

User/virtual organisationR

esou

rce

prov

ider

/

Res

earc

h co

mpu

ting

cent

er

Funding agency


26

D-Grid – collaborations

• D-Grid will/wants collaborate with many other projects– EGEE– DEISA– Globus– other national initiatives– …

• D-Grid will contribute to standards – GGF, …

• D-Grid will disseminate the results to new user communities

• D-Grid want to start an effort together with industry– In 2006, a call is expected to encourage „e-Science

service providers“ (commerical or academic) to propose long term solutions for „e-Science services“


27

Conclusion and Outlook

• D-Grid started finally in September 2005– and successfully!!

• Grid technology will be established in (at least) six scientific communities

• All e-Science domains will work in a close collaboration

• DGI will provide a general infrastructure for e-Science

• All three e-Science domains will organize the“German e-Science Conference" (Nov. 22 - 24, 2006, Baden-Baden)

• First results of the D-Grid initiative will come soon!!

• All the work presented is funded by the BMBF

the german grid initiative a platform for e-science in germany dr. harald kornmayer

Documents