C. Loomis – Title – Date – 1

European DataGrid

Charles Loomis

Seillac

June 26, 2002

Outline

EDG Introduction & Goals

EDG Architecture

Example Job Submission

Future Developments

Further Information

C. Loomis – Title – Date – 2

European DataGrid (EDG)

European DataGrid

EU-funded, 3-year project (2001-3)

Goals:—develop grid middleware

—deploy onto working testbed

—demonstrate grid technology with working applications

Strong application component unique!

EDG Organization

WP1 Workload Mgt.

WP2 Data Mgt.

WP3 Info. & Monitoring Sys.

WP4 Fabric Mgt.

WP5 Storage Mgt.

WP6 Testbed

WP7 Networking

WP8 HEP Apps.

WP9 Biomedical Apps.

WP10

Earth Ob. Apps.

WP11

Dissemination

WP12

Project Mgt.

6 Partners; 21 Associates

C. Loomis – Title – Date – 3

Computing Grid Electrical Grid

Accessing computing resources as simple as plugging in a toaster!

Fixed resources (CPU)

Mobile resource (Data)

Semi-mobile (Tape)

Transparency (I.e. standards)

Simplifies sharing & trading of resources.—Does not imply that all resources are free.

Should be applicable from desktop to supercomputers.

Security (Owners retain control!)

System administrators control machine access.

Who can read your data? Software enforces access control.

CPU Cycles

Network Bandwidth

Disk/tape Storage

Data/Databases

Software

C. Loomis – Title – Date – 4

Transparency for Physicists

Single Login:

Access all available local & remote resources w/o a dedicated account.

Consistent User Environment:

Same working environment everywhere--same software tools; same configuration.

Brokering:

System finds “optimal” resources for task automatically.

Resource discovery, moving job to data, moving data to job.

High-Level Selection of Resources:

Ex. choose storage by size/lifetime/reliability/cost/transfer speed; not disk/tape/shared file system/location.

Security:

Does not impede authorized access to resources.

Data and computations are (reasonably) safe from tampering.

C. Loomis – Title – Date – 5

Interoperability & Interdependency

Standards:

Global Grid Forum (like W3C)

Or from dominant implementation

Plethora of Projects:

Sharing of code & expertise.

Competing for people, funding, etc.

C. Loomis – Title – Date – 6

EDG Architecture

Global Batch System:

Centralized Architecture.

Heavy infrastructure.

Other choices Peer-to-peer

Computing Element

Storage Element

Site X

Information Systems

submit

submit

publishstate

query

retrieve

retrieve

broker choosesoptimal site for job

Resource Broker

User Interface

C. Loomis – Title – Date – 7

Hidden Infrastructure

Certification Authorities

Attest to your identity.

Virtual Organizations

Maintain membership lists.Used for group authorization.

/C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr

Replica Catalog

Location & DirectoryInfo. for data.

Maintained by:

ALICEATLASBaBarCMSLHCbBiomedical Apps.Earth Observation

C. Loomis – Title – Date – 8

EDG Testbeds

Production Testbed (1.1.4)

For applications to use & stress software in “semi-production” environment.

10 sites (6 countries)

Development Testbed (1.2b5)

To facilitate testing and integration of new middleware.

4 sites (4 countries)

Site Location

Catania Catania (I)

CC-IN2P3 Lyon (F)

CERN* Geneva (CH)

CNAF* Bologna (I)

Imperial College

London (UK)

LIP** Lisbon (PT)

MSU Moscow (Russia)

NIKHEF* Amsterdam (NL)

Padova Padova (I)

RAL* Rutherford (UK)

Torino Torino (I)

C. Loomis – Title – Date – 9

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

#!/bin/sh

# Get matched physical file name.

filename = `./GetPFN.pl $EDG_WL_RB_BROKERINFO`

# Analyze this file.

count_evts $filename

Find data file and count events.

C. Loomis – Title – Date – 10

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

# Executable to run and the arguments.Executable    = "ChecksumRC.sh";Arguments     = "none";

# Define file names for standard output and error.StdOutput     = "std.out";StdError      = "std.err";

# Files to transfer as input and output.InputSandbox  = {"ChecksumRC.sh","GetPFN.pl”,”count_evts”};OutputSandbox = {"std.out","std.err"};

# Desired file and protocol.InputData = {"LF:utve_v_1_71008101_12902_0609.lv0"};DataAccessProtocol = {"file"};

# My experiment's "Data Catalog"ReplicaCatalog = "ldap://grid-vo.nikhef.nl:10389/rc=EarthObReplicaCatalog,dc=eu-datagrid,dc=org";

Describe the requirements for the job.

C. Loomis – Title – Date – 11

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

> grid-proxy-init

Your identity: /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr

Enter GRID pass phrase for this identity: *************

Creating proxy .......................................... Done

Your proxy is valid until Sun Jun 23 06:13:51 2002

Create a proxy for authorization like logging in.

C. Loomis – Title – Date – 12

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

> dg-job-submit ChecksumRC.jdl

Connecting to host testbed011.cern.ch, port 7771Logging to host testbed011.cern.ch, port 15830

************************************************************************************                                     JOB SUBMIT OUTCOME  The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is:

https://testbed011.cern.ch:7846/137.138.181.253/162422289427581?testbed011.cern.ch:7771

************************************************************************************

Submit job. Broker will automatically find appropriate place.

C. Loomis – Title – Date – 13

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

>dg-job-status <job id>

…

dg_JobId                =    https://testbed011.cern.ch:7846/137.138.181.253/…

Status               =    OutputReady

Last Update Time (UTC)  =    Sat Jun 22 16:29:30 2002

Job Destination         =    gppce05.gridpp.rl.ac.uk:2119/jobmanager-pbs-S

Status Reason           =    terminated

Job Owner               =    /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/…

Status Enter Time (UTC) =    Sat Jun 22 16:29:30 2002

Get the status of the submitted job.

C. Loomis – Title – Date – 14

Example

Prerequisites

write script

job description

Job Submission

grid-proxy-init

dg-job-submit

dg-job-status

dg-job-get-output

>dg-job-get-output <job id>

…

 Output sandbox files for the job: - https://testbed011.cern.ch:7846/137.138.181.253/… have been successfully retrieved and stored in the directory: /tmp/162422289427581

Pull the output back to home machine.

>cat /tmp/162422289427581/std.out

File: /flatfiles/eo/utve_v_1_71008101_12902_0609.lv0Contains: 1034 events

C. Loomis – Title – Date – 15

Future Developments

EDG Plans:more advanced data management (I.e. real “Storage Element”)

job splitting, checkpointing; interactive jobs

more robust, consistent security model

OGSA (Open Grid Services Architecture)Proposal by Globus/IBM based on web-services

Adds concepts like notification and soft-state registration.

Standardizes the protocol and interface descriptions for Grid services.

Cross-fertilization of ProjectsLAL is ideally situated for this. (EDG, LCG, XtremWeb, CGP2P)

C. Loomis – Title – Date – 16

Conclusions

EDG has interesting (to a physicist) software & working testbed:

Current release:—Basic features available.

—Somewhat painful: reliability, low-level mucking around

Fall release:—Expect big change in functionality, especially data management.

—Concerted effort on reliability should make testbed less frustrating.

About the time to get involved…

Long-term success:

When you work routinely in a “Grid” environment on your desktop.

Fully automatic: resource discovery, optimization, and task dispatch.

Not there yet, but going in right direction.

C. Loomis – Title – Date – 17

Getting Involved & Further Surfing

Talk to grid contact:

ATLAS (Laura Perini)

ALICE (Piergiorgio Cerello)

CMS (Andrea Sciaba)

LHCb (Eric van Herwijnin)

others (see me)

Register:

“User Registration” @ WP6 website

Warnings:

Dev. project Frequent changes.

No guarantees on availability.

Schedule:

EDG 1.2 (shortly)

EDG 2.0 (fall 2002)

Interesting Web Sites

EDG (Main Website)

http://www.eu-datagrid.org

EDG (WP6) http://marianne.in2p3.fr

CrossGrid http://www.crossgrid.org

DataTag http://www.datatag.org

GridPP http://www.gridpp.ac.uk

LCG http://lhcgrid.web.cern.ch/LHCgrid

NorduGrid http://www.nordugrid.org

Condor http://www.cs.wisc.edu/condor

Globus http://www.globus.org

GriPhyN http://www.griphyn.org

iVDGL http://www.ivdgl.org

PPDG http://www.ppdg.net

OGSA http://www.globus.org/ogsa

Global Grid Forum

http://www.gridforum.org