r-gma: production services for information and monitoring in the grid john walk / ral

EGEE is a project funded by the European Union under contract IST-2003-508833

R-GMA: Production Services for Information

and Monitoring in the Grid

John Walk / RAL

UK e-Science AHM2004, Nottingham

www.eu-egee.org

R-GMA, UK e-Science AHM, 1 September 2004 - 2

What’s the talk about?

• Overview of R-GMA

• Practical experience with R-GMA

• Building a production-quality Web Services version of R-GMA


Who are we?

• The system (R-GMA) Grid Information and Monitoring System created in the European

DataGrid (EDG) project Now being taken forward as part of the production-quality middleware

being developed in the EGEE project

• The people (JRA1-UK) 9 Full-time at Rutherford Appleton Laboratory (CCLRC) 1 Full-time at Queen Mary, University of London Contributions from Heriot-Watt University (Edinburgh) and Trinity

College Dublin

• We can be e-mailed at: [email protected]


Relational Grid Monitoring Architecture

ProducerService

RegistryService

ConsumerService

AP

IA

PI

Mediator

Consumerapplication

Producerapplication

Publish Data

Send Query

Receive Data

Register

LocateQu

ery

Da

ta

• An implementation of the GGF Grid Monitoring Architecture (GMA).


Relational Grid Monitoring Architecture

ProducerService

RegistryService

ConsumerService

AP

IA

PI

Mediator

SchemaService

Consumerapplication

Producerapplication

Publish Tuples

Send Query

Receive Tuples

Register

LocateQu

ery

Tu

ples

SQL “CREATE TABLE”

SQL “INSERT”

SQL “SELECT”

• An implementation of the GGF Grid Monitoring Architecture (GMA)

• All data modelled as tables: a single schema gives the impression of one (loosely coupled) virtual database for VO


Secondary Producers

• Used to republish data To co-locate information to speed up queries To reduce network traffic To use different producer properties

PP

PP

PP

PP

SP

PP – Primary Producer

SP – Secondary Producer


A unified approach to information and monitoring

• Provides information on both: The Grid itself

• Mainly for the middleware packages, but the user may also query it to understand the status of the Grid

Grid applications• For users

• Flexible infrastructure All data has a timestamp enabling its use for monitoring Dynamic addition and deletion of information producers Dynamic creation and deletion of tables in schema Able to cope with failures in a distributed environment


Resilience

• Soft state registration protects services and registry from defunct producers and consumers

• Registry and schema are replicated to avoid single point of failure (and improve scalability)

• All services tolerate failure of remote service or resource by seeking an alternative, trying again later, or failing gracefully if these are not possible


Grid Application: CMS Job Monitoring

http://cmsinfo.cern.ch/


BOSS Job wrapper

PMonte Carlo

Simulation Job

BOSS Job wrapper

PMonte Carlo

Simulation Job

R-GMA in CMS

C BOSS Database

R-GMA Primary Producer

C R-GMA Consumer

P

In testing…• Target: 2000-job batch• Exceeded in 2004


Grid Middleware: Network Monitoring

http://ccwp7.in2p3.fr/


R-GMA in EDG WP7 Network Monitoring

SP

PP

PingERIPerf

UDPmonGridFTP

Log

Network Monitoring Nodes Storage Elements

Network ManagersResource BrokersReplica Managers

R-GMA Primary Producer

SP R-GMA Secondary Producer

P

TCP throughput

ICMP packet loss

Transfer times

Network metrics include…

MySQLDatabase


Enabling Grids for E-science in Europe

“EGEE will integrate current national, regional and thematic Grid efforts to create a seamless European Grid infrastructure for the support of the European Research Area” (http://www.eu-egee.org)

• R-GMA is the Information Services component of the gLite middleware being developed for EGEE

• Main efforts are: Producing production-quality software Migrating to Web Services Improving portability

• Duration of project is two years, with first release of software due after first year (March 2005)


Main achievements so far (JRA1-UK)

• Specification complete

• Design complete

• WSDL produced for user interfaces

• Prototype of new API available (Java, C, C++, Python), as a wrapper around the old one (feedback welcome)

• Registry replication completed; outline design for schema replication produced


Next steps

• Complete move to WS-I compliant Web Services

• Introduce multi-VO support

• Complete security design work

• Complete the schema replication

• Port to Win32 (EGEE secondary platform)

• Watch developments elsewhere and seek to inter-work (e.g. contribution to OGSA; collaboration with L&B work)


R-GMA Web Services

• API available for Java, C, C++ and Python

• Users may by-pass API if they wish, but API is the easiest way to use R-GMA services

• Registry and Schema also implemented as Web Services, but not accessed directly by users, except for administration

UserApplication

R-G

MA

AP

I

SO

AP

ST

UB

AX

ISS

ER

VLE

T

Primary Producer Service

Secondary Producer Service

On-demand Producer Service

Consumer Service

SOAP/HTTPMessages

SOAP/HTTPMessages

AndSOAP Faults

Tomcat Servlet Container

Created from WSDLusing gSOAP or Axis


R-GMA Security

• Many issues are EGEE-wide and are being addressed by JRA3 (Security)

• User requirements vary widely from HEP (would probably do without it) to BioMed (very stringent confidentiality requirements)

• Will allow installations to turn security off where performance is paramount

• Some R-GMA users require access to data to be controlled down to sub-table level. Proposal is to use parameterized views, e.g.

SELECT * from Job where Owner=[DN] : DN=[DN];


Further Information

• JRA1-UK web site http://hepunx.rl.ac.uk/egee/jra1-uk/

• E-mail [email protected]

Thanks to the EU and our national funding agencies for

their support of this work

r-gma: production services for information and monitoring in the grid john walk / ral

Documents