overview of middleware
DESCRIPTION
Overview of middleware. Alexandre Duarte CERN IT-GD-OPS UFCG LSD 1st EELA Grid School Itacuruçá, 04-15/12/2006. Input “sandbox”. DataSets info. UI JDL. Output “sandbox”. voms-proxy-init. SE & CE info. Output “sandbox”. Expanded JDL. Job Submit Event. Job Query. - PowerPoint PPT PresentationTRANSCRIPT
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Alexandre DuarteCERN IT-GD-OPSUFCG LSD1st EELA Grid SchoolItacuruçá, 04-15/12/2006
Overview of middleware
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Job Workflow in gLite
UIJDL
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputingComputingElementElement
Information Information ServiceService
Job Status
LFCLFCCatalogCatalog
DataSets info
Author.&Authen.
Job S
ub
mit
Even
t
Job
Qu
ery
Job
Stat
us
Input “sandbox”
Input “sandbox” + Broker InfoGlobus RSL
Output “sandbox”
Output “sandbox”
Job Status
Pu
blis
h
vom
s-pr
oxy-
init
Exp
and
ed J
DL
SE & CE info
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Job Workflow in gLite
UIJDL
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputingComputingElementElement
Information Information ServiceService
Job Status
LFCLFCCatalogCatalog
DataSets info
Author.&Authen.
Job S
ub
mit
Even
t
Job
Qu
ery
Job
Stat
us
Input “sandbox”
Input “sandbox” + Broker InfoGlobus RSL
Output “sandbox”
Output “sandbox”
Job Status
Pu
blis
h
vom
s-pr
oxy-
init
Exp
and
ed J
DL
SE & CE info
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Middleware structure
• Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware
• Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory
• Foundation Grid Middleware will be deployed on the EGEE infrastructure– Must be complete and robust– Should allow interoperation
with other major grid infrastructures
– Should not assume the use of Higher-Level Grid Services
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Grid Foundation: Security
• Authentication based on X.509 PKI infrastructure– Certificate Authorities (CA) issue (long lived) certificates
identifying individuals (much like a passport) Commonly used in web browsers to authenticate to sites
– Trust between CAs and sites is established (offline)– In order to reduce vulnerability, on the Grid user identification is
done by using (short lived) proxies of their certificates
• Proxies can– Be delegated to a service such that it can act on the user’s
behalf– Include additional attributes (like VO information via the VO
Membership Service VOMS)– Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire)
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Virtual Organization Membership Service:– Extends the proxy with info on VO
membership, group, roles– Fully compatible with GSI– Each VO has a database containing
group membership, roles and capabilities informations for each user
– User contacts VOMS server requesting his authorization info
– Server sends authorization info to the client, which includes it in a proxy certificate
[sam@egris195 sam]$ voms-proxy-init -voms gilda
Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=ITACURUCA/CN=ITACURUCA13/[email protected]
Enter GRID pass phrase:
Creating temporary proxy ............................ Done
Contacting egris197.eela.ufrj.br:15001 [/C=BR/O=ICPEDU/O=UFF BrGrid CA/O=UFRJ/OU=IF/CN=host/egris197.eela.ufrj.br] "gilda" Done
Creating proxy .............................................. Done
Your proxy is valid until Tue Dec 5 10:55:10 2006
Grid Foundation: VOMSQuery
Authentication
Request
AuthDB
C=IT/O=INFN /L=CNAF/CN=Pinco Palla/CN=proxy
VOMSAC
VOMSAC
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Grid foundation: Information Systems
• BDII: Information system in gLite 3.0 (by LCG)– LDAP database that is updated by a process – More than one DBs is used separate read and write– A port forwarder is used internally to select the correct DB
• R-GMA: provides a uniform method to access and publish distributed information and monitoring data– Used for job and infrastructure monitoring in gLite 3.0
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Grid foundation: Computing Element
• gLite-CE: based on GSI enabled Condor-C– Coming from LCG-CE (based on GT2 GRAM)– Supported by Condor. More efficient.– Deployed for the first time in gLite 3.0
• CREAM: new lightweight CE– Not yet in gLite 3 release. Will need exposure to users on
dedicated system.– WSDL interface– Will support bulk submission of jobs from WMS and optimization
of input/output file transfer.– Plans are to have a CE with both Condor-C and CREAM
interfaces
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Grid foundation: Storage Element
• Storage Element– Common interface: SRMv1,migrating to SRMv2– Various implementation from LCG and other external projects
disk-based: DPM, dCache / tape-based: Castor, dCache
– Support for ACLs in DPM (in future in Castor and dCache)– Common rfio library for Castor and DPM being added
• Posix-like file access:– Grid File Access Layer (GFAL) by LCG
Support for ACL in the SRM layer (currently in DPM only) Support for SRMv2 being added
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
High Level Services: File transfer
• FTS: Reliable, scalable and customizable file transfer– Manages transfers through channels
mono-directional network pipes between two sites
– Web service interface
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
High Level Services: Workload mgmt.
• WMS helps the user accessing computing resources – Resource brokering, management of job input/output, ...
• gLite-RB: GT2 + Condor-G– To be replaced when the gLite WMS proves to be reliable
• gLite WMS: Web service (WMProxy) + Condor-G– Management of complex workflows (DAGs) and compound jobs
bulk submission and shared input sandboxes support for input files on different servers (scattered sandboxes)
– Job File Perusal: file peeking during job execution– Supports collection of information from CEMon, BDII, R-GMA
and from DLI and StorageIndex data management interfaces– Support for parallel jobs (MPI) when the home dir is not shared– Deployed for the first time in gLite 3.0
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
High Level Services: Workflows
• Direct Acyclic Graph (DAG) is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs
• A Collection is a group of jobs with no dependencies– basically a collection of JDL’s
• A Parametric job is a job having one or more attributes in the JDL that vary their values according to parameters
• Using compound jobs it is possible to have one shot submission of a (possibly very large, up to thousands) group of jobs – Submission time reduction
Single call to WMProxy server Single Authentication and Authorization process Sharing of files between jobs
– Availability of both a single Job ID to manage the group as a whole and an ID for each single job in the group
nodeEnodeC
nodeA
nodeD
nodeB
1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
www.glite.org
Questions ?