Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA

www.eu-eela.org

E-infrastructure shared between Europe and Latin America

gLite Overview

Christian Grunfeld, UNLP

8th EELA Tutorial, La Plata, 11/12-12/12, 2006

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 2

E-infrastructure shared between Europe and Latin America

Middleware

• Grid Middleware – Layer between user applications and grid resources.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 3

E-infrastructure shared between Europe and Latin America

Grid Concepts

• VOs:Individuals and/or Institutions having direct access to resources.

User Interface

User Interface

Grid services

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 4

E-infrastructure shared between Europe and Latin America

Introduction

• Grid Sistems & Applications aim is to:

– Integrate– Virtualise– Manage

Resources and services across different VOs.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 5

E-infrastructure shared between Europe and Latin America

Grid Requirements

• Heterogeneous• VO Resource Sharing• Resource Utilization• Job Execution• Data Services• Security• Scalability• Availability• Specific Requeriments

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 6

E-infrastructure shared between Europe and Latin America

Multi-VOs

• Users join VOs

• Virtual organisation contributes resources & negotiates access

• Grid middleware runs on each resource– “Storage elements”

– “Compute elements”

• Additional services (both people and grid middleware) enable the grid

• Effect:

collaboration

INTERNET

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 7

E-infrastructure shared between Europe and Latin America

Building on GSI

• Build on Grid Security Infrastructure to create services that include:– Job submission: run a job on a remote computer– Information services: So I know which computer to use– File transfer: so large data files can be transferred– Replica management: so I can have multiple versions of a file

“close” to the computers where I want to run jobs

• Production grids are (currently) based on the Globus Toolkit release 2 Globus Alliance: http://www.globus.org/

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 8

E-infrastructure shared between Europe and Latin America

gLite

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 9

E-infrastructure shared between Europe and Latin America

gLite

• gLite is the next generation middleware for grid computing.

• Born from the collaborative efforts from academic and industrial research centers as part of the EGEE Project.

• The gLite Grid services follow a Service Oriented Architecture– facilitate interoperability among Grid services – allow easier compliance with upcoming standards

• Architecture is not bound to specific implementations– services are expected to work together – services can be deployed and used independently

• The gLite service decomposition has been largely influenced by the work performed in the LCG project

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 10

E-infrastructure shared between Europe and Latin America

gLite – Service Decomposition

5 High level services

+ CLI & API

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 11

E-infrastructure shared between Europe and Latin America

gLite – Security Services

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 12

E-infrastructure shared between Europe and Latin America

gLite – Security Services Authentication

Identify entities (users, systems and services) when establishing context for message exchange (Who are you?).

Aim - Provide a Credential having a universal value that works for many purposes across many infrastructures, communities, VOs and projects.

gLite uses the PKI (X.509) infrastructure using CAs as thrusted third parties.

MyProxy (http://grid.ncsa.uiuc.edu/myproxy/)

Trust domain: The set of all EGEE CAs is our Trust Domain.

•Revocation: Identities must be revocated timely

•Credential Storage: Local or delegated credential (Services or Users)

•Privacy Preservation: Use of personal data

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 13

E-infrastructure shared between Europe and Latin America

gLite – Security Services Authorization

Allows or denies access to services, based on policies.

•Agent: The user interacts with a centralized Authorization Server•Push*: Authorization Services issue Tokens.•Pull: The resource asks to the Authorization Services.

Authorization Sources:

•Attribute Authority (AA): User <-> Set of Attibutes. (VOMS)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 14

E-infrastructure shared between Europe and Latin America

gLite – Grid Access

Two possibilities: APIs and CLI.

The use of web-services allows the automatic generation of APIs

(error prone, lack of tools).

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 15

E-infrastructure shared between Europe and Latin America

gLite – Information and Monitoring Services

Information services are vital low level component of Grids.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 16

E-infrastructure shared between Europe and Latin America

gLite – Information and Monitoring ServicesBasic info and monitoring services (RGMA)

•Information is provided by a Publish and Consume mechanism.•Appearance of a single federated database to query through the SQL.•Each VO has a VDB.

•Schema - Contains tables (GLUE)•Registry – List of available sources of information (Mediation)•Producers – Source of information (Primary, Secondary, On-demand)•Consumers – Make queries against tables (Continuous, Latest, History)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 17

E-infrastructure shared between Europe and Latin America

gLite – Information and Monitoring Services Job Monitoring, Service Discovery, Network performance Monitoring

•Job Monitoring – Java logging service, log4j, Apache/Chainsaw (for other languages).

•Service Discovery – Locates suitable services to both users and services (Library!).

•Network Performance Monitoring – Many network monitoring frameworks. Aim: perform a standard interface to those frameworks.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 18

E-infrastructure shared between Europe and Latin America

gLite – Job Management Services

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 19

E-infrastructure shared between Europe and Latin America

gLite – Job Management ServicesAccounting

•Accumulates information about the resource usage done by users or groups of users (VOs).

•Information on Grid Services/Resources needs sensors (Resource Metering, Metering Abstraction Layer, Usage Records).

•Records are collected by the Accounting System (Queries: Users, Groups, Resource)

•Grid services should register themselves with a pricing service when accounting for billing purposes.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 20

E-infrastructure shared between Europe and Latin America

gLite – Job Management ServicesComputing Element

•Service that represent the computing resource that is responsible of the job management: (submission, control, etc.)

•CEs refer to a set or a cluster of computational resources (WN) managed by LRMS, to dispatch jobs matching users requests.

•Two job submission models (accordingly to user requests and site policies):

• PUSH (jobs pushed to CE queue), • PULL (jobs coming from WMS when CE queue is empty)

•CE responsible to collect accounting information.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 21

E-infrastructure shared between Europe and Latin America

gLite – Job Management ServicesWorkload Management

•WMS set of middleware components responsible of distribution and management of jobs across Grid resources.

•Two core components of WMS:

•WM: accept and satisfy requests for job management.Matchmaking is the process of assigning the best available resource.

•L&B: keep track of job execution in term of events: (Submitted, Running, Done,...)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 22

E-infrastructure shared between Europe and Latin America

gLite – Job Management ServicesJob Provenance, Package Manager

•Job Provenance (JP) - Keeps track of submitted jobs for long periods (months, years).

•Package Manager – Helper service to automate: installing, configuring, updating and removing of software components. (RPM, dpkg/APT, Portage, …)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 23

E-infrastructure shared between Europe and Latin America

gLite – Data Services

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 24

E-infrastructure shared between Europe and Latin America

gLite – Data ServicesStorage Element

Needed Service are at least:

•Storage back-end (Drivers and Hardware)

•SRM Interface (Storage Specific)

•Transfer service (GridFTP)

•Native POSIX like file I/O API (gLite-I/O)

•Auxiliary Accounting and Logging services

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 25

E-infrastructure shared between Europe and Latin America

gLite – Data ServicesCatalogs

OS like file access metaphor.

•LFN (Logical file name)

•GUID (Grid unique identifier)

•SimLinks

•SURL (Site URL)

•TURL (Transfer URL)

Catalogs:

•Authorization Base•Metadata Base•Metadata Schema•Replica Catalog•File Catalog•File Authorization•Metadata•Combined Catalog•Storage Index

gLite - (FireMAN)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 26

E-infrastructure shared between Europe and Latin America

gLite – Helper Services

Configuration and Instrumentation Service – Query service state.

Agreement Service – Implements a communication protocol for the SLAs.

Bandwidth Allocation & Reservation service (BAR) – Controlling, Balancing and Manage Network flows.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 27

E-infrastructure shared between Europe and Latin America

Components

• Site:– Computing Element (CE)

Gateway to local computing resources (cluster de worker nodes)

– Worker Nodes (WN)– Storage Element (SE)

Gateway to local storage (disk, tape) A gridftp server, and SRM Interface, IO server

– User Interfaces (UI) User’s access point to the grid Client programs using some/all grid services.

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 28

E-infrastructure shared between Europe and Latin America

Components

• Grid – or VO- wide– Security

Virtual Organization Server (VOMS) MyProxy server (Proxy)

– Information System– Job handling

Workload Management System (WMS) Logging & Bookkeping (LB)

– Data management File catalog (FiReMan) File Transfer Service (FTS) File Placement Service (FPS)

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 29

E-infrastructure shared between Europe and Latin America

VOMS

• Virtual Organization Membership Service– Multiple VOs– Multiple roles in VO

Compatible X509 extensions Signed by VOMS server

– Web admin interface– Supports MyProxy– Resources providers grant access to VOs or roles– Sites map VO members/roles to local auth mechanism (unix

users accounts) Allows for local policy

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 30

E-infrastructure shared between Europe and Latin America

MyProxy

• MyProxy– Allows longer lived jobs / increases security

WMS renews proxy Users should not produce long lives proxies

– Allows for secure user mobility Users does not need to copy globus-keys around

Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 31

E-infrastructure shared between Europe and Latin AmericaReferences

• Globus Alliance: http://www.globus.org

• gLite middleware: http://glite.web.cern.ch/glite

• gLite 3.0 user guide: https://edms.cern.ch/file/722398//gLite-3-UserGuide.html