introduction to the current edg testbed software krak ó w, december 2002 steve fisher...

Introduction to the current EDG Testbed

SoftwareKraków, December 2002

Steve Fisher [email protected] – RAL

on behalf of

The European DataGrid Project Team

http://www.edg.org/

The EDG Testbed Software - 2

The European DataGrid

Funded by the European Union Jan 1, 2001 - Dec 31, 2003

Develop, implement and exploit a large-scale data and CPU-oriented computational GRID.

Develop middleware, in collaboration with some of the leading centres of competence in GRID technology.

Complement, and help to coordinate at a European level, several on-going national GRID projects.


The EDG Main Partners

CERN – International (Switzerland/France)

CNRS - France

ESA/ESRIN – International (Italy)

INFN - Italy

NIKHEF – The Netherlands

PPARC - UK


Research and Academic Institutes• CESNET (Czech Republic)• Commissariat à l'énergie atomique (CEA) – France• Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)• Consiglio Nazionale delle Ricerche (Italy)• Helsinki Institute of Physics – Finland• Institut de Fisica d'Altes Energies (IFAE) - Spain• Istituto Trentino di Cultura (IRST) – Italy• Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany• Royal Netherlands Meteorological Institute (KNMI)• Ruprecht-Karls-Universität Heidelberg - Germany• Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands• Swedish Research Council - Sweden

EDG Assistant Partners

Industrial Partners• Datamat (Italy)• IBM-UK (UK)• CS-SI (France)


}} ApplicationsApplications

WP1: Work Load Management System

WP2: Data Management

WP3: Information and Monitoring

WP4: Fabric Management

WP5: Storage Element

WP6: Testbed and demonstrators

WP7: Network Monitoring

WP8: High Energy Physics

WP9: Earth Observation

WP10: Biology

WP11: Dissemination

WP12: Management

EDG structure: work packages


Current EDG Testbed

CERNLyon

RAL

Manchester NIKHEF

Reference site: CERN

Testbed1 EDG sites

NorduGrid

Italy:• Bologna • Cagliari• Catania• Milano• Napoli• Padova• Parma• Pisa• Roma• Torino

NorduGrid:• Bergen• Copenhagen• Helsinki• Lund• Oslo• Stockholm• Uppsala

Karlsruhe

BarcelonaMadrid

Lisboa


Security: Authentication/Authorization

Authentication Who you are

users identified by certificates signed by a CA

Authorization What you are allowed to do

based on membership of Virtual Organizations (VO).


Certificate Request

VO

user

cert-request

grid-cert-request

once in every two-three years


Requesting a Certificate

grid-cert-request

A certificate request and private key is being created.

[...]

Using configuration from /usr/local/grid/globus/etc/globus-user-ssleay.conf

Generating a 1024 bit RSA private key

[...]

A private key and a certificate request has been generated with the subject:

/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner

[...]

Your private key is stored in .../.globus/userkey.pem

Your request is stored in .../.globus/usercert_request.pem

Please e-mail the certificate request to the CERN CA

cat .../.globus/usercert_request.pem | mail [email protected]

Your certificate will be mailed to you within two working days.


Certificate Signing

CA

user

cert-request

grid-cert-request

certificate

cert signing


Registration/Authorization

User registration in an EDG Virtual Organisation

convert your certificate: openssl pkcs12 –export –in ~/.globus/usercert.pem –inkey

~/.globus/userkey.pem –out user.p12 –name ’Joe Smith’

import your certificate in your browser

sign the usage guidelines: https://marianne.in2p3.fr/cgi-bin/datagrid/register/account.pl

ask an account from your VO administrator by email

-> You are registered in the VO server and have a user account.


Registration

VO

user

registrationcert.pkcs12

convertcertificate

Usage guidelines

Account Registration

once for the lifetime of the VO – you may

change the certificate keys!


Starting a Session

user

proxy-certgrid-proxy-init

cert.pkcs12

certificate

every 12/24 hours


Usage

You must have a valid certificate from a trusted CA!

“login”: grid-proxy-init

short lifetime certificate: 24 hours

Enter PEM pass phrase:

...........................+++++

....................................+++++

checking the proxy: grid-proxy-info -subject

/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner/CN=proxy

-> use the grid services

“logout”: grid-proxy-destroy


Configuration on the Server

CA

service

host-cert

cert signing

host-request

grid-cert-request

ca-certificate

crl

cert/crl update

crl automatically updated

periodically


Authorization Information

VO-server

service

host-cert

gridmapmkgridmap

ca-certificates

crls

automatically updated

periodically


Using a Service

user service

proxy-certgrid-proxy-init

cert.pkcs12

certificate

host/proxy certs exchanged

host-cert

gridmap

ca-certificates

crls


EDG Logical Machine Types1. User Interface (UI)

2. Resource Broker (RB)

3. Information Service (IS)

4. Computing Element (CE) Gatekeeper

(Front-end Node)

Worker Nodes (WN)

5. Storage Element (SE)

6. Replica Catalog (RC)


Information Systems overview

The aim of the Information and Monitoring Service is to deliver a flexible infrastructure that provides information on

the EU DataGrid itself grid applications

EDG info systems are based upon Globus MDS (Metacomputing Directory Service or Monitoring and Discovery Service as it is now called)

Based on OpenLDAP, a hierarchical database

The information system is currently used mainly by the middleware.

You can use it to find out what is going on


LDAP attributes

A schema describes the attributes and the types of the attributes associated with data objects

Example - some attributes of SiteInfo: siteName: RALDEV

sysAdminContact: [email protected]

userSupportContact: [email protected]

siteSecurityContact: [email protected]

dataGridVersion: 1.2

InstallationDate: 20020704142800Z


LDAP hierarchy

Lightweight Directory Assess Protocol (LDAP) offers a hierarchical view of information

The objects are arranged in a Directory Information Tree (DIT)

One or more attributes represent the Relative Distinguished Name (RDN)

An object is identified by its Distinguished name This is its RDN with the Distinguished name of its parent


RDNs and DNs

RDN

SE seId=dev02.hepgrid.clrc.ac.u

k

Protocols seProtocol=gridftp seProtocol=rfio seProtocol=file

DN Site

Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

SE seId=dev02.hepgrid.clrc.ac.uk,M

ds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

Protocols seProtocol=gridftp,

seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

seProtocol=rfio, seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

seProtocol=file, seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

supportedprotocols

SE

site


MDS GRISs & GIISs

Information providers are scripts which when invoked by the LDAP server make available the desired information

Information is cached by the server to improve performance

Within MDS the EDG information providers are invoked by a local LDAP server, the Grid Resource Information Server (GRIS)

“Aggregate directories”, Grid Information Index Servers (GIIS), are used to group resources

The GRISs use soft state registration to register with one or more GIISs

The GIIS can then act as a single point of contact for a number of resources

A GIIS may represent a site, country, virtual organization, etc.

In turn a GIIS may register with another GIIS


EDG Information Providers & the Directory Information Tree

computing element

storage elements that are close (not necessarily at the same site)

status supportedprotocols

file statistics

network information between this and other sites

storage element

site information

site


EDG GRIS/GIIS Hierarchy

Information providers publish information to a local LDAP server known as a Grid Resource Information Server (GRIS)

Each country has a GIIS to which all of the site GIISs register

There is a top level datagrid GIIS to which all of the country GIISs register

Each Site has a Grid Information Index Server (GIIS) which acts as a single point of contact for all of the sites resources. The GRISs register with their site GIISsiteA siteDsiteCsiteB

countryA countryB

datagrid

information providers





EDG Information Providers

The EDG have produced information providers: Site information

The Computing Element

The Storage Element

Network Monitoring

All of the EDG data objects are dynamic, they have a time stamp and a time to live (used by the cache mechanism) associated with them


Siteinfo

in=siteinfo,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

objectClass: SiteInfo

objectClass: DataGridTop

objectClass: DynamicObject

siteName: RALDEV

sysAdminContact: [email protected]

userSupportContact: [email protected]

siteSecurityContact: [email protected]

dataGridVersion: 1.2

installationDate: 20020704142800Z


Computing Element

ceId=dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs-M,hn=dev01.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid

objectClass: DataGridTopobjectClass: ComputingElement CEId:

dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs-M

GlobusResourceContactString:dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs:/O=Grid/O=UKHEP/CN=dev01.hepgrid.clrc.ac.uk

GRAMVersion: ?Architecture: intelOpSys: RH 6.2MinPhysicalMemory: 258MinLocalDiskSpace: 2048TotalCPUs: 1FreeCPUs: 1NumSMPs: 0MinSPUProcessors: 0MaxSPUProcessors: 0TotalJobs: 0RunningJobs: 0IdleJobs: 0

MaxTotalJobs: 1 MaxRunningJobs: 1 WorstTraversalTime: 108000 EstimatedTraversalTime: 0 Active: TRUE Priority: 20 MaxCPUTime: 108000 MaxWallClockTime: 432000AverageSI00: 300 MinSI00: 300 MaxSI00: 300 AuthorizedUser:/O=Grid/O=UKHEP/

OU=hepgrid.clrc.ac.uk/CN=Tim EvesAuthorizedUser:/O=Grid/O=UKHEP/

OU=hepgrid.clrc.ac.uk/CN=Tim Folkes RunTimeEnvironment: RALDEV AFSAvailable: FALSE OutboundIP: TRUE InboundIP: FALSE QueueName: MLRMSType: PBS LRMSVersion: OpenPBS_2.3


Queries can be posed to the current Information and Monitoring Service using LDAP search commands

An LDAP search consists of the following components

Querying the Information & Monitoring Service

$ldapsearch\ -x\ -H ldap://lxshare0225.cern.ch:2135\ -b 'Mds-Vo-name=datagrid,o=grid\ 'objectclass=ComputingElment‘\ CEId FreeCPUs \ -s base|one|sub

“simple” authenticationuniform resource identifierbase distinguished name for searchfilterattributes to be returnedscope of the search specifying just

the base object, one-level or the complete subtree


Querying the GRIS/GIIS Hierarchy

Mds-Vo-name=siteB,o=grid This will look at all the data from siteB

Mds-Vo-name =siteA

Mds-Vo-name =siteD

Mds-Vo-name =siteC

Mds-Vo-name =siteB

Mds-Vo-name =countryA

Mds-Vo-name =countryB

Mds-Vo-name =datagrid

Mds-Vo-name=countryA,o=grid This will look at all the data from

countryA

Mds-Vo-name=datagrid,o=grid This will look at all the data

Mds-Vo-name=siteB,Mds-Vo-name=countryA,o=grid

This will look at all the data from siteB

Mds-Vo-name=siteB, Mds-Vo-name=countryA,Mds-Vo-name=datagrid,o=grid

This will look at all the data from siteB


The EDG WMS

The user interacts with GRID via a Workload Management System

The Goal of WMS is the distributed scheduling and resource management in a GRID environment.

What does it allow GRID users to do?

To submit their jobs

To execute them

To get information about their status

To retrieve their output

The WMS tries to optimize the usage of resources


WMS Components

WMS is currently composed of the following parts:

1. User Interface (UI) : access point for the user to the GRID

2. Resource Broker (RB) : the broker of GRID resources, performing the match-making

3. Job Submission System (JSS) : provides a reliable submission system

4. Information Index (II) : a specialized Globus GIIS (LDAP server) used by the Resource Broker as a filter to the information service (IS) to select resources

5. Logging and Bookkeeping services (LB) : store Job Info available for users to query


WMS UI Commands dg-job-submit

submits a job

dg-job-list-matchlists resources matching a job description

dg-job-cancelcancels a given job

dg-job-statusdisplays the status of the job (submitted, waiting, ready, scheduled, running, chkpt,

done, outputready, aborted, cleared)

dg-job-get-outputreturns the job-output to the user

dg-job-get-logging-infodisplays logging information about submitted jobs

dg-job-id-infois a utility for the user to display job info in a formatted style


Example of UI Command Options

dg-job-submit –r <res_id> –n <user e-mail address> -c <config file> -o <output file> <job.jdl>

-r the job is submitted by the RB directly to the computing element identified by <res_id>

-n an e-mail message containing basic information regarding the job (status and identification) is sent to the specified <e-mail address> when the job enters one of the following status:

DONE or ABORTED

READY

RUNNING

-c the configuration file <config file> is pointed by the UI instead of the standard configuration file

-o the generated dg_jobId is written in the <output file>

dg-job-status –i <input file> (or dg_jobId)

-i the bookkeeping information about dg_jobId contained in the <input file> are displayed


Job Description Language (JDL) Mandatory for every single JDL file:

• Executable (contains the command name)

• Other attributes:• InputSandbox

• OutputSandbox

Mandatory for JDL file dealing with Data Management:• ReplicaCatalog (contains the Replica Catalog Identifier)

• DataAccessProtocol (contains the protocol or the list of protocols which the application is able to speak with for accessing InputData on a given SE)

If InputData contains at least one PFN and no LFNs, only DataAccessProtocol is mandatory.

If InputData contains at least one LFN, both ReplicaCatalog and DataAccessProtocol are mandatory.


Example JDL File

Executable = “gridTest”;

InputData = “LF:testbed0-00019”;

ReplicaCatalog = “ldap://sunlab2g.cnaf.infn.it:2010/ \ rc=WP2 INFN Test, dc=infn, dc=it”;

DataAccessProtocol = “gridftp”;

StdError = “stderr.log”;

StdOutput = “stdout.log”;

OutputSandbox = {“stderr.log”, “stdout.log”};

InputSandbox = {“home/joda/test/gridTest”};

Rank = “other.MaxCpuTime”;

Requirements = other.Architecture==“INTEL” && \ other.OpSys==“LINUX” && other.FreeCpus >=4;


A Job Submission Example

UIJDL

Logging &Book-keeping(LB)

ResourceBroker (RB)

Job SubmissionService (JSS)

StorageElement(SE)

ComputeComputeElement CE)Element CE)

Information Service (IS)

ReplicaCatalogue(RC)



UIJDL


ResourceBroker (RB)


StorageElement(SE)

ComputeComputeElement (CE)Element (CE)



Job SubmitEvent

Input Sandbox

Job Status

submitted



UIJDL


ResourceBroker (RB)


StorageElement(SE)




Job Status

submitted

waiting



UIJDL


ResourceBroker (RB)


StorageElement(SE)




Job Status

submitted

waiting

ready



UIJDL


ResourceBroker (RB)

Job SubmissionService(JSS)

StorageElement (SE)




Job Status

submitted

waiting

ready

BrokerInfo

scheduled



UIJDL


ResourceBroker (RB)


StorageElement(SE)




Job Status

submitted

waiting

ready

scheduledInput Sandbox

running



UIJDL


ResourceBroker (RB)


StorageElement(SE)




Job Status

submitted

waiting

ready

scheduled

Job Status

running



UIJDL

Logging &Book-keeping

ResourceBroker

Job SubmissionService

StorageElement

ComputeComputeElementElement

Information Service

ReplicaCatalogue

submitted

waiting

ready

scheduled

running

Job Status

done

Job Status



UIJDL

Logging &Book-keeping

ResourceBroker

Job SubmissionService

StorageElement

ComputeComputeElementElement

Information Service

ReplicaCatalogue

submitted

waiting

ready

scheduled

running

done

Job Status

Job Status

outputready

Output Sandbox



UIJDL


ResourceBroker (RB)

Job SubmissionService (JS)

StorageElement(SE)




Output Sandbox

cleared

submitted

waiting

ready

scheduled

running

done

Job Status

outputready


EDG Data Management Tools

Tools for Locating data

Copying data

Managing and replicating data

Meta Data management

On EDG Testbed you have EDG Replica Catalog

globus-url-copy (GridFTP)

EDG Replica Manager

Grid Data Mirroring Package (GDMP)


EDG Replica Catalog

Based upon the Globus LDAP Replica Catalog (will be replaced by RLS)

Stores LFN/PFN mappings and additional information (e.g. filesize): Physical File Name (PFN): host + full path & and file name

Logical File Name (LFN): logical name that may be resolved to PFNs

LFN : PFN = 1 : n

Only files on storage elements may be registered

Each VO has a specific storage dir on an SE

Example PFN: lxshare0222.cern.ch/flatfiles/SE1/iteam/file1.dat host storage dir

LFN must be full path of file starting from storage dirLFN of above PFN: file1.dat


EDG Replica Catalog

API and command line tools addLogicalFileName

getLogicalFileName

deleteLogicalFileName

getPhysicalFileName

addPhysicalFileName

deletePhysicalFileName

addLogicalFileAttribute

getLogicalFileAttribute

deleteLogicalFileAttribute

http://cmsdoc.cern.ch/cms/grid/userguide/gdmp-3-0/node85.html


globus-url-copy

Low level tool for secure copying

globus-url-copy <protocol>://<source file> \ <protocol>://<destination file>

Main Protocols: gsiftp – for secure transfer, only available on SE and CE

file – for accessing files stored on the local file system on e.g. UI, WN

globus-url-copy file://`pwd`/file1.dat \ gsiftp://lxshare0222.cern.ch/ \ flatfiles/SE1/EDGTutorial/file1.dat


The EDG Replica Manager

Extends the Globus replica manager

Client side tool

Allows replication (copy) and registering of files in RC

Keeps RC consistent with stored data.


The Replica Manager APIs

(un)registerEntry(LogicalFileName lfn,

FileName source)

Replica Catalogue operations only - no file transfer

copyFile(FileName source,

FileName destination,

String protocol)

allows for third-party transfer

transfer between: two StorageElements or ComputingElement and Storage Element Space management policies under development


copyAndRegisterFile(LogicalFileName lfn,

FileName source,


String protocol)

third-party transfer but :

files can only be registered in Replica Catalogue if destination PFN contains a valid SE

replicateFile(LogicalFileName lfn,

FileName source,


String protocol)

deleteFile(LogicalFileName lfn,

FileName source)

The Replica Manager APIs


based on CMS requirements for replicating Objectivity files for High Level Trigger studies

production prototype project for evaluating Grid technologies (especially Globus)

http://cern.ch/GDMP


Overview of Components

EDG Replica Catalogue

Site1 Site3Site2

GDMP client


Subscription Model

All the sites that subscribe to a particular site get notified whenever there is an update in its catalog.

Site 1

Site 3

Site 2

Subscriberlist

Subscriberlist

subscribe subscribe


Export / Import Catalogue

Export Catalog information about the new files

produced . is published

Import Catalog information about the files which

have been published by other sites but not yet transferred locally

As soon as the file is transferred locally, it is removed from the import catalogue.

Possible to pull the information about new files into your import catalogue.

Site 1

Site 3

exportcatalog

importcatalog

Site 2

exportcatalog

1)register, publish new files

2) transfer files2) transfer files

1) get info aboutnew files

3) delete files


Usage gdmp_ping

Ping a GDMP server and get its status

gdmp_host_subscribe first thing to be done by a site

gdmp_register_local_file Registers a file in local file catalogue but NOT in Replica Catalogue (RC)

gdmp_publish_catalogue send information of newly created files to subscribed hosts (no real data transfer) – update RC

gdmp_replicate_get - gdmp_replicate_put get/put all the files from the import catalogue – update RC

gdmp_remove_local_file Delete a local file and update RC

gdmp_get_catalogue Get remote catalogue contents – for error recovery


GDMP vs. EDG Replica Manager

GDMP Replicates sets of files

Replication between SEs

Mass storage interface

File size as logical attribute

Subscription model

Event notification

CRC file size check

Support for Objectivity

Replica Manager Replicates single files

Replication between SEs, CEs to SE.

introduction to the current edg testbed software krak ó w, december 2002 steve fisher...

Documents