![Page 1: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/1.jpg)
Steve Fisher, Shantenu Jha*, Andre Merzky, O Weidner, A Wilson
http://saga.cct.lsu.edu
EGEE-III INFSO-RI-222667
Providing uniform and standardised access to European grid middleware with SAGA
![Page 2: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/2.jpg)
SAGA: In a nutshell
There exists a lack of programmatic approaches that:• Provide general-purpose, basic &common grid functionality for
applications and thus hide underlying complexity, varying semantics..
• The building blocks upon which to construct “consistent” higher-levels of functionality and abstractions
• Meets the need for a Broad Spectrum of Application: • Simple scripts, Gateways, Smart Applications and
Production Grade Tooling, Workflow…
Simple, integrated, stable, uniform and high-level interface• Simple and Stable: 80:20 restricted scope and Standard• Integrated: Similar semantics & style across• Uniform: Same interface for different distributed systems
UMD: Components of gLite, ARC, UNICORE & globus
![Page 3: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/3.jpg)
SAGA: In a thousand words..
![Page 4: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/4.jpg)
gLite CREAM Overview
CREAM = Computing Resource Execution And Management
Lightweight service for job management operation at the Computing Element (CE) level
CREAM can be used via:• gLite Workload Management System• the ICE service• a generic client (Java/C++ + command line
interfaces available)• The Simple API for Grid Applications
(SAGA).
![Page 5: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/5.jpg)
gLite CREAM SAGA Adaptor
SAGA provides a prototype job adaptor to access gLite CREAM CEs
Implemented with the CREAM C++ API Currently provides the following features:
• job description (saga-jd to JDL mapping)• job control (run, submit, suspend, cancel, etc...)• listing of jobs associated with a CE• re-connection to running jobs• Encapsulates (hides) proxy delegation
![Page 6: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/6.jpg)
SAGA/CREAM Python Example
![Page 7: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/7.jpg)
SAGA/CREAM C++ Example
![Page 8: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/8.jpg)
SAGA/CREAM Getting Started
The adaptor code is available via SVN:https://svn.cct.lsu.edu/repos/saga-adaptors/glite/
Due to the limitations of the CREAM/gLite build-system the adaptor only supports platforms that are supported by gLite (Scientific Linux, etc…)
But will be release with glite in near future Please send comments and/or questions to:
![Page 9: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/9.jpg)
Movie time…
![Page 10: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/10.jpg)
SAGA BES Adaptor
SAGA BES adaptor based on bes++• C reference implementation for HPC-BP
by Chris Smith (Platform) et.al.• has been used for BES-HPCBP interop• uses gSoap• provides ’besclient’ and ’besserver’
• code is somewhat messy, as usual with gsoap based code
![Page 11: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/11.jpg)
SAGA BES Adaptor (cont.)
C++ wrapper for bes++ client lib was added simplifies adaptor code to (job::run)void job_cpi_impl::sync_run (saga::impl::void_t & ret) { if ( state_ != saga::job::New ) { SAGA_ADAPTOR_THROW ("can run only 'New' jobs", saga::IncorrectState); }
job_epr_ = bes_.run (jsdl_); jobid_ = "[" + endpoint_url_.get_string () + "]-[" + job_epr_->str + "]";
{ adaptor_data_type adata (this); // scoped lock state_ = adata->get_saga_state (bes_.get_state (job_epr_)); }}
• svn co https://svn.cct.lsu.edu/repos/saga-adaptors/ogf/trunk• license: GPL (due to gSoap)
![Page 12: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/12.jpg)
SAGA BES Adaptor (cont.)
SAGA example submit to BES#include <saga/saga.hpp>
int main () { saga::session s; saga::context c; c.set_attribute (saga::attributes::context_username, “ogf”); c.set_attribute (saga::attributes::context_userpass, “ogf”); s.add_context (c); saga::job::service js (s, ep.url); saga::job::job j = js.run_job_job (“/bin/sleep 10”); while ( j.get_state () == saga::job::Running ) { cout << "Running" << endl; ::sleep (1); } if (j.get_state () == saga::job::Done ) cout << "Done" << endl; else cout << "Failed" << endl; return j.get_state ();}
![Page 13: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/13.jpg)
SAGA BES Adaptor (cont.)
funtional:• endpoint verification• certificate and username/pass auth• SAGA job description -> JSDL• JSDL -> SAGA job description• job submission• BES job state -> SAGA state• terminate job
not yet implemented• list backend jobs• reconnect to job based on job-id
![Page 14: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/14.jpg)
SAGA BES Adaptor (cont.)
Interop Tests: bes++: besserver with SAGA backend ;-)
fully functional Unicore: https://zam1161v01.zam.kfa-juelich.de:8002/
fully functional GridSam: https://gridsam-endpoint.oerc.ox.ac.uk:18443/
demo certificate expired ARC: http://knowarc1.grid.niif.hu:50000/arex
demo CA expired
Planned Tests:• NAREGI, CREAM, Genesis-II, PBS-Pro, MS HPC Server• Please provide us with interop endpoints! :-)
![Page 15: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/15.jpg)
GANGA OVERVIEW
![Page 16: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/16.jpg)
SAGA-GANGA Integration
![Page 17: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/17.jpg)
DIANE INTEGRATION
Diane without SAGA Diane with SAGA
DIANE is an execution manager with support for pilot-jobs + worker agents(IDEAS Redux)
![Page 18: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/18.jpg)
Master
Agents scheduling
Heterogeneous resourcesallocation (Ganga + Ganga/SAGA)
Lattice-QCD Applications on heterogeneous resources
Ganga/gLite
Ganga/SAGA (to TeraGrid)
Ganga/SAGA (to *)
Payload distribution
Application-aware (and resource-aware) scheduling
Federating resources! EGEE Conference (Apr’10) (Not in this demo:
cloud resources, additional Grid infrastructures…)
![Page 19: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/19.jpg)
RENKEI Project Aims
SAGA-Engine
gLiteNAREGISRBiRODS
Adpt Adpt Adpt
C++ InterfacePython Binding
Service & Applications Svc Apps Apps
Cloud LRMSLSF/PBS/SGE/…
Middleware-independent service & application
RNSYet Another FC
service based on OGF standard
SAGA adaptors
SAGA framework
This activity is funded by MEXT as a part of RENKEI project which develops seamless linkage of resources in the Grids and the local one for e-Science.
KEK
Osaka Univ.Tsukuba
Univ.
HEPLibrary
SAGA
![Page 20: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/20.jpg)
BigJob: Infrastructure Independent Pilot-Job
![Page 21: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/21.jpg)
BigJob: Infrastructure Independent Pilot-Job (Each sub-job is a MPI-based MD)
![Page 22: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/22.jpg)
Service Discovery
This API (GFD.144) identifies URLs of services Hides the GLUE-I ↔ GLUE-II transition from the user saga-sd --sf "Site IN ('INFN-CNAF', 'RAL-LCG2')" --
sa• Return details of all services you are authorized to use at
INFN-CNAF and RAL-LCG2
gLite adapter being added to UI and WN patches this week
NAREGI just starting work on an adapter• Hope to see it at next OGF in Chicago
![Page 23: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/23.jpg)
Information System Navigator
ISN exposes the schema to allow navigation over any ER style schema
gLite adapter supports GLUE-I now. GLUE-II will be tested when data are available.
Available as experimental API in LSU SAGA implementation with CLI and gLite adapter
GUI portal to SD and ISN being tested. Will be made generally available soon.
![Page 24: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/24.jpg)
SAGA: Available Adaptors Job Adaptors
• Fork (localhost), SSH, Condor, Globus GRAM2, OMII GridSAM, Amazon EC2, Platform LSF
File Adaptors• Local FS, Globus GridFTP, Hadoop Distributed Filesystem (HDFS),
CloudStore KFS, OpenCloud Sector-Sphere
Replica Adaptors• PostgreSQL/SQLite3, Globus RLS
Advert Adaptors• PostgreSQL/SQLite3, Hadoop H-Base, Hypertable
Other Adaptors• Default RPC / Stream / SD
![Page 25: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/25.jpg)
SAGA-based Tools and ProjectsAdvantage of Standards
JSAGA from IN2P3 (Lyon)• http://grid.in2p3.fr/jsaga/index.html• gLite adaptors exist
JAVASAGA (Amsterdam)• Has a wide range of adaptors• JAVASAGA gets released by gLite (next few weeks)
NAREGI/KEK (Active)• http://www.ogf.org/OGF27/materials/1767/OGF27_SAGA_KEK.pdf
DEISA/DESHL• http://www.fz-juelich.de/nic-series/volume38/pringle.pdf )• http://deisa-jra7.forge.nesc.ac.uk/ and
http://www.ogf.org/OGF19/materials/501/SAGA-DEISA.ppt
XtreemOS• http://saga.cct.lsu.edu/index.php?option=com_content&task=view&id=95
&Itemid=174
![Page 26: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/26.jpg)
Summary..
Power of well-defined, agreed upon interfaces• Tools, Gateways and Applications
Advantages across levels (close to the middleware, upto direct application level), and scales (single EGEE/EGI site, to largest TG resources)
SAGA provides application-level interoperability across different middleware and distributed resource types
Interoperability first step towards effective utilisation of DCI• IDEAS• http://www.cct.lsu.edu/~sjha/select_publications
![Page 27: Providing uniform and standardised access to European grid middleware with SAGA](https://reader034.vdocuments.site/reader034/viewer/2022051518/568161e2550346895dd1f7ba/html5/thumbnails/27.jpg)
SAGA Team and DPA Team and the UK-EPSRC (UK EPSRC: DPA, OMII-UK , OMII-UK PAL), NSF (HPCOPS, Cybertools) and LA-BOR
People:
SAGA D&D: Hartmut Kaiser, Ole Weidner, Andre Merzky, Andre Luckow, Joohyun Kim, Lukasz Lacinski, João Abecasis, Chris Miceli, Bety Rodriguez-Milla
SAGA Users: Andre Luckow, Yaakoub el-Khamra, Kate Stamou, Cybertools (Abhinav Thota, Jeff, N. Kim), Owain Kenway
Google SoC: Michael Miceli, Saurabh Sehgal, Miklos Erdelyi
Collaborators and Contributors: Steve Fisher & Group, Sylvain Renaud (JSAGA), Go Iwai & Yoshiyuki Watase (KEK)
DPA: Dan Katz, Murray Cole, Manish Parashar, Omer Rana, Jon Weissman
Acknowledgements