tecnologia dei servizi grid e cloud computing - lezione 7b 0 lezione 7b - 9 dicembre 2009 il...
TRANSCRIPT
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 1
Lezione 7b - 9 Dicembre 2009
Il materiale didattico usato in questo corso è stato mutuato da quello utilizzato da Paolo Veronesi per il corso di Griglie Computazionali per la Laurea Specialistica in Informatica tenuto nell’anno accademico 2008/09 presso l’Università degli Studi di Ferrara.
Paolo [email protected], [email protected]://www.cnaf.infn.it/~pveronesi/unife/
Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica
“Tecnologia dei Servizi “Grid e cloud computing” A.A. 2009/2010
Giorgio Pietro Maggi [email protected], http://www.ba.infn.it/~maggi
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 2
Defining the Grid
A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 3
The EGEE Project Aim of EGEE:
“to establish a seamless European Grid infrastructure for the support of the European Research Area (ERA)”
EGEE 1 April 2004 – 31 March 2007 71 partners in 27 countries, federated in regional Grids
EGEE-II 1 April 2007 – 30 April 2008 Expanded consortium
EGEE-III 1 May 2008 – 30 April 2010 Transition to sustainable model
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 4
Enabling Grid for E-sciencE project
ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…
>250 sites48 countries>50,000 CPUs>20 PetaBytes>10,000 users>150 VOs>150,000 jobs/day
Flagship Grid infrastructure project co-funded by the European Commission starting from April 2004Entering now in the 3° phase
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 5
Defining the Grid
A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 6
EGEE Infrastructure
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 7
EGEE Infrastructures Production service
Scaling up the infrastructure with resource centres around the globe Stable, well-supported infrastructure, running only well-tested and reliable
middleware
Pre-production service Run in parallel with the production service (restricted nr of sites) First deployment of new versions of the gLite middleware Test-bed for applications and other external functionality
T-Infrastructure (Training&Education) Complete suite of Grid elements
and application (Testbed, CA, VO, monitoring, support, …)
Everyone can register and use GILDA for training and testing
20 sites on 3 continents
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 8
EGEE Operations Process Geographically distributed responsibility for
operations: There is no “central” operation Regional Operation Centers
Responsible or resource centers in their region Tools are developed/hosted at different sites:
GOC DB (RAL), SAM (CERN), GStat (Taipei), CIC Portal (Lyon)
Grid operator on duty 10 teams working in weekly rotation Crucial in improving site stability and management
Operations coordination Weekly operations meetings Regular ROC managers meetings Series of EGEE Operations Workshops
Procedures described in Operations Manual Introducing new sites Site downtime scheduling Suspending a site Escalation procedures; etc.
Highlights: Distributed operation Evolving and maturing procedures Procedures being in introduced into and shared with the related infrastructure projects
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 9
Central probes (SAM)
Local probes
Network monitoring
Doubled size and usage without impact on operations
Improved reliability through multi-level monitoring
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 10
EGEE operations
Operations Coord. Centre (OCC)
- management, oversight of all operational and support activities
Regional OperationsCentres (ROC)
- providing the core of the support infrastructure, each supporting a number of resource centres within its region
Resource Centres (RC)
- providing resources
(computing, storage, network…)
- At FZK, coordination and management of user support, single point of contact for users
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 11
Monitoring Visualization
11
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 12
The EGEE support infrastructure
•RC A
•RC B
•RC C
•RC A
•RC B
•RC C•ROC C•ROC BROC N
RC A
RC B
RC C
TPM
VO TPM CVO TPM B
VO TPM A
GGUS
Central
System
Middleware
supportMiddleware
supportMiddleware
support
Deployment
supportMiddleware
supportDeployment
support
VO Support
CVO Support
BVO Support
A
Middleware
supportMiddleware
supportMiddleware
support
•ROC C•ROC BROC N
Network Support
Network Support Other GridsOther GridsOther Grids
Other GridsOther GridsOther Grids
CODCIC
Portal
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 13
Defining the Grid
A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 14
gLite Middleware Distribution
Combines components from different providers Condor and Globus (via VDT) LCG EGEE Others
Focus on providing a deployable MW distribution for EGEE production service
Middleware services + configuration tools
Follows a service oriented approach Usage of webservices where useful and possible performance-wise
Complemented by application-level servcies
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 15
Production Grid MiddlewareKey factors in EGEE Grid Middleware Development:
1. Strict software processUse industry standard software engineering methods Software configuration management, version control, defect tracking,
automatic build system, …
2. Conservative approach in what software to useAvoid “cutting-edge” software Deployment on over 200 sites cannot assume a homogenous
environment – middleware needs to work with many underlying software flavors
Avoid evolving standards Evolving standards change quickly (and sometime significantly cf.
OGSI vs. WSRF) – impossible to keep pace on > 200 sites
Long (and te
dious) path
from prototypes to
production
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 16
Certification
Functional Tests
Testbed Deployment
gLite Process
Development
Software Error Fixing
Integration
Deployment Packages
Integration Tests
Installation Guide, Release Notes, etc
Pre-Production
Scalability Tests
Pre-Production Deployment
Fail
Fail
Fail
Pass
Pass
Pass
Production Infrastructure
Problem
Directives
ExternalSoftware
Directives
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 17
gLite Software Process Technical Coordination Group (TCG)
gathers & prioritizes user requirementsfrom HEP, biomed, (industry), sites
gLite development is client-driven! Software from EGEE-JRA1 and other projects
JRA1 preview testbed (currently being set up) early exposure to users of “uncertified” components
SA3 Integration Team Ensures components are deployable and work Deployment Modules implemented high-level gLite node types
(WMS, CE, R-GMA Server, VOMS Server, FTS, etc) Build system now spun off into the ETICS project (Jan 2006)
SA3 Certification Team Merge of the JRA1 testing and SA1 certification teams Dedicated testbed; test release candidates and patches Develop test suites
SA1 Pre-Production System Scale tests by users
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 18
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 19
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM,
Upper Middleware & Tools
Lower Middleware Bonds
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 20
Defining the Grid
A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 21
EGEE Applications >270 VOs from several
scientific domains Astronomy & Astrophysics Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences
Further applications under evaluation
Applications have moved from testing to routine and daily usage
~80-95% efficiency
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 22
The Future of Grids Increasing the number of infrastructure users by increasing
awareness Dissemination and outreach Training and education
Increasing the number of applications by improving application support and middleware functionality Improved usability through high level grid middleware extensions
Increasing the grid infrastructure Incubating related projects Ensuring interoperability between projects
Protecting user investments Towards a sustainable grid infrastructure
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 23
Grid Interoperability
Incubator for new Grid efforts world-wide Infrastructure and application efforts
Leading role in building world-wide Grids through interoperation efforts Bilateral: EGEE/OSG, EGEE/NDGF,
EGEE/NAREGI, EGEE/Unicore/DEISA Multilateral: Grid Interoperability Now
(GIN)
Experiences and requirements fed back into standardization process (OGF) Many EGEE members are area directors,
WG chairs, WG members
Contacts with industry strengthened Industry Days, Industry Task Force,
Business Associates Programme
GIN
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 24
EGEE working with related infrastructure projects
GIN
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 25
Evolution
European e-Infrastructure
Testbeds Utility ServiceRoutine Usage
National
Global
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 26
Need to prepare permanent, common Grid infrastructure Ensure the long-term sustainability of the European e-Infrastructure
independent of short project funding cycles Coordinate the integration and interaction between National Grid
Infrastructures (NGIs) Operate the production Grid infrastructure on a European level for a wide
range of scientific disciplines
Must be no gap in the support of the production
grid
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 27
Summary
Grids represent a powerful new tool for science
Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago)
EGEE offers … … a mechanism for linking together people, resources and data of
many scientific community … a basic set of middleware for gridfying applications with
documentation, training and support … regular forums for linking with grid experts, other communities and
industry
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 28
gLite Middleware overview
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 29
Grid Middleware
When using a PC or workstation you Login with a username and
password (“Authentication”) Use rights given to you
(“Authorisation”) Run jobs Manage files: create them,
read/write, list directories Components are linked by a
bus Operating system One admin domain
• When using a Grid you– Login with digital credentials (“Authentication”)
– Use rights given you (“Authorisation”)
– Run jobs– Manage files: create them, read/write, list directories
• Services are linked by the Internet
• Middleware• Many admin domains
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 30
EGEE Project and gLite• Enabling Grids for E-sciencE (EGEE) is the largest multi-disciplinary grid
infrastructure in the world– Brings together more than 120 European organisations – Consists of 250 sites in 48 countries and more than 68,000 CPUs – Is available to some 8,000 users 24 hours a day, 7 days a week– Processes more than 150,000 jobs per day from different scientific domains
• gLite is the middleware powering the EGEE infrastructure and many other related projects
– Is an integrated set of components designed to enable resource sharing among different institutions
– Pulls together contributions from many other projects, including LCG and VDT– Enable users with a large set of services
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 31
The “global” grid
e-Infrastructures adopting gLite
e-Infrastructures interoperable or in pro-gress to be made interoperable with gLite
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 32
Additional Infrastructures: GILDA• EGEE provides a training infrastructure: GILDA (Grid INFN
Laboratory for Dissemination Activities)– Runs the entire gLite stack protocols– Used to demonstrate EGEE grid technology project– Supports beginner and expert training courses on gLite
• Adopted by several Grid projects worldwide
• Own Certification Authority
• Available 365 days for everyone !
• Used in the ISSGC schools series
• Since 2007 other middleware than gLite are tested on GILDA
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 33
•20 sites in 3 continents•> 11000 certificates issued, >20% renewed at least once•> 250 courses, training events, official university curricula•> 2,000,000 hits on the web site from >100 different countries •> 4.5 TB of training material downloaded from the web site
The GILDA t-Infrastructure (https://gilda.ct.infn.it)
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 34
gLite in the Grid “ecosystem”
. . .
LCG
EGEE
Used in
USA EU
NextGrid DEISAGridCC
Future grids
EDG
Globus MyProxyCondor ...
VDT
DataTAG
CrossGrid ...
OSG, …
SRM
…
interactive
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 35
The Middleware structure• Applications have access both to
Higher-level Grid Services and to Foundation Grid Middleware
• Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory
• Foundation Grid Middleware are actually developed in EGEE
– Must be complete and robust– Should allow interoperation with other major
grid infrastructures– Should not assume the use of Higher-Level
Grid Services
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 36
gLite infrastructure
Workload Management System (WMS)Data Management
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 37
Typical Job workflow
JDL
Logging &Book-keeping
ResourceBroker
Job SubmissionService
StorageElement
ComputingComputingElementElement
Information Service
Job Status
ReplicaCatalog
Job SubmitEvent
Input Sandbox
JDL
Job
Input Sandbox
Output Sandbox
Output Sandbox
User Interface
Author.Service
voms-proxy-init
GSI data acc/trans
f
glite-job-submit myjob.jdlMyjob.jdl
Executable = “gridTest”;StdError = “stderr.log”;StdOutput = “stdout.log”;InputSandbox = {“/home/joda/test/gridTest”};OutputSandbox = {“stderr.log”, “stdout.log”};InputData = “lfn:testbed0-00019”;DataAccessProtocol = “gridftp”;Requirements = other.Architecture==“INTEL” && \
other.OpSys==“LINUX”;Rank = “other.GlueHostBenchmarkSF00”;
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 38
Security System
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 39
gLite Security
• Authentication based on X.509 PKI infrastructure– Certificate Authorities (CA) issue (long lived) certificates
identifying individuals (much like a passport)– Trust between CAs and sites is established (offline)– In order to reduce vulnerability, Grid user identification
is done by (short lived) proxies of their certificates
• Proxies can– Be delegated to a service such that it can act on the
user’s behalf– Include additional attributes (like VO information via the
VO Membership Service VOMS)– Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire)
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 40
Which CA are trusted in LCG/EGEE?
http://www.eugridpma.org/
“The EUGridPMA is the international organization to coordinate the trust fabric for e-Science grid authentication in Europe. It collaborates with the regional peers APGridPMA for the Asia-Pacific and The Americas Grid PMA in the International Grid Trust Federation. The charter document defines the group's objective, scope and operation. It is the basis for the guidelines documents on the accreditation procedure, the Authentication profile for X.509 secured "classic" certification authorities and other IGTF recognised Profiles. “
In LCG/EGEE CA are installed on machine trough rpms.In LCG/EGEE CA are installed on machine trough rpms.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 41
Grid resources (A)
Grid resources (B)
Conventional grid security
Certification Authority (CA)BobCert request
User Interface (UI)
Bob´s Grid certificate
Sysadmin A :- Create user “grid1“- Map Bob´s certificate to “grid01“
Sysadmin B :- Create user “user001“- Map Bob´s certificate to “user001“
- Single sign-on- Delegation through proxy certificate
- Manual user “mapping“- No info about VOs
grid-proxy-init
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 42
gLite: VOMS
Virtual Organization Membership Service (VOMS)
EGEE/gLite enhancement for VO management
Provides information on user's relationship with Virtual Organization (VO)Membership
Group membership
Roles of user
Multiple VOUser can register to multiple VOs and create an aggregate proxy
Access ressources in every registered VO
Backward compatibilityExtra VO related information in users proxy certificate
Users proxy can still be used with non VOMS-aware services
7 Maggio 2009 – Paolo Veronesi Griglie Computazionali - Lezione 007 42
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 43
gLite: VOMS - Web interface Requires a valid certificate from a
recognized CA imported on the browser
VO user can
Query membership details
Register himself in the VONeeds a valid certificate
Track his requests VO manager can
Handle requests from users
Administer the VO Everybody can
Get information about the VO
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 44
Grid resources (A)
Grid resources (B)
gLite – Enhanced security in gLite
Certification Authority (CA)BobCert request
User Interface (UI)
Bob´s Grid certificate
VO Database
VO Service
VO Manager
VO membership request
VO
VO AccountPool
VO AccountPool
Automatic mappingfor Bob
Automatic mappingfor Bob
voms-proxy-init
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b 45
LCAS & LCMAPS• At resources level, authorization info is extracted
from the proxy and processed by LCAS and LCMAPS
• Local Centre Authorization Service (LCAS)– Checks if the user is authorized– Checks if the user is banned at the site
• Local Credential Mapping Service (LCMAPS)– Map remote credentials to local credentials (eg. different
UNIX uid/gid)– Map also VOMS group and roles (full support of FQAN)
enables privileges separations