Resource Brokering on Complex Resource Brokering on Complex GridsGrids

EUROGRID and GRIP EUROGRID and GRIP

Presented by John Brooke ESNW

October 3/4UK/Japan N+N

January 17, 2003

ContentsContents

Aims of the resource broker

Functionality of the ancestral broker EUROGRID broker

Interoperability architecture (UNICORE-Globus)

Towards a resource description ontology

Relation to the GGF and OGSA

OrganizationFirewalls

Users

VirtualOrganisationBrokers

SystemBrokers

ComputeResources

Brokers as VOs

Client

Broker Servic

e

Broker Servic

e

Broker Servic

e

Broker Servic

e

Broker

Service

Broker

Service

Client

Client

Client

Client

Broker Servic

e

Resource

Resource

Resource

Resource

Resource

Resource

Broker

Service

VO Layer

Specialist Layer

Site Layer

Replication

Federated Brokering

Interoperability for BrokeringInteroperability for Brokering

We want to broker on Grids controlled by either UNICORE or Globus. In GRIP we developed two methods

1. Bifurcation, separate “sub-brokers” for a Globus or a Unicore Grid. This is achieved and is extensible to a limited extent.

2. Constructing an extendable resource broker utilising a Grid Resource Ontology to handle mappings of resource terms.

1

Ancestral EUROGRID Broker

The API allows two levels of operation:Resource Checking: Static requirements, capability and

capacity.QoS Checking: Performance vs cost. Tickets can be issued

as a “guarantee”.Protocol can be used symmetrically by Broker.

User

BrokerNJS

ExecutionNJS

ExecutionNJS

1 CheckQoS

2 CheckQoS

2 CheckQoS

3 CheckQoS_Outcome

3 CheckQoS_Outcome

4 CheckQoS_Outcome

Brokering for Workflows

BezierSGI Onyx @ Manchester

Vtk + VizServer

DiracSGI Onyx @ QMUL

LB3D with RealityGridSteering API

LaptopSHU Conference Centre

UNICOREGateway and NJS

Manchester

Fir

ew

al

l

SGI OpenGL VizServer

Simulation

Data

VizServer clientSteering GUI The Mind Electric GLUE web service hosting environment with OGSA extensionsSingle sign-on using UK e-Science digital certificates

UNICOREGateway and NJS

QMUL

Steering (XML)

Interoperable Broker – Method 1

1.1. The Network Job Supervisor (NJS) delegates the The Network Job Supervisor (NJS) delegates the Resource Check to the Broker at the Vsite.Resource Check to the Broker at the Vsite.

2.2. The UNICORE brokering track utilises the IDB The UNICORE brokering track utilises the IDB exactly as for the ancestral broker.exactly as for the ancestral broker.

3.3. The Globus track uses a translator of the QoS check The Globus track uses a translator of the QoS check object. The translation service is extendable.object. The translation service is extendable.

4.4. The results of the translation are used to drive the The results of the translation are used to drive the LDAP search and the Globus broker then utilises LDAP search and the Globus broker then utilises MDS to perform this.MDS to perform this.

UNICORE NJS 4.0 gave much greater power and UNICORE NJS 4.0 gave much greater power and flexibility in brokering for complex workflows.flexibility in brokering for complex workflows.

NJSNJS

BrokerBroker

Unicore BrokerUnicore Broker Globus BrokerGlobus Broker

IDBIDB TranslatorTranslator FilterFilter

Basic TranslatorBasic Translator MDS(GRIIS/GRIS)MDS(GRIIS/GRIS)

Delegates resource checkDelegates resource check

LookupLookupresourcesresources

Delegates translationDelegates translation Uses to drive Uses to drive LDAP searchLDAP search

PerformsPerformsLDAP searchLDAP search

DiagramDiagramOf Broker Of Broker ArchitectureArchitecture

Architecture – Method 1Architecture – Method 1

Pros and ConsPros and Cons

•A nice feature of Method 1 is that no alteration needs to be made to the client side of UNICORE, thus no alteration for application plugins or “expert” brokers

•Also no alterations need to be made to Globus.

•However the UNICORE description of Grid resources is very different from the MDS-2 description. MDS-2 does not publish software resource and user environment, Unicore does not check dynamic resource, e.g. machine loading.

•The need for resource description translation is thus highlighted.

NJSNJS

BrokerBroker

Unicore BrokerUnicore Broker Globus BrokerGlobus Broker

IDBIDBTranslatorTranslator

FilterFilterOntology engineOntology engine

Resource DiscoveryResource DiscoveryServiceService

Delegates resource checkDelegates resource check

LookupLookupresourcesresources

Delegates Delegates translationtranslation Uses to drive Uses to drive

MDS searchMDS search

HierarchicalHierarchicalGrid SearchGrid Search

DiagramDiagramOf Broker Of Broker ArchitectureArchitecture

Architecture – Method 2Architecture – Method 2

FilterFilter

Uses to Uses to Drive MDSDrive MDSSearchSearch

HierarchicalHierarchicalGrid SearchGrid Search

OtherOtherBrokersBrokers

Resource DiscoveryResource DiscoveryServiceService

OntologiesOntologies

•Defines knowledge domain and allows reasoning on this domain.

•If we can create a Grid Resource Ontology, creation of specialist translation classes from basic Grid translator becomes possible.

•IDB at sites can be created via ontology, it contains site specific information which the clients job specification cannot do.

•So brokers take client request formulated in RR space, at each site use translator to convert to RR space, offers come back with capability and QoS.

BrokerNJS

IDB

TSI/Host GT3

Gateway

Client

Broker

NJS

Host Host

Gateway

Client

R-GMANJS NJS

Host

Site-Wide BrokeringNormal EUROGRID/GRIP Brokering

Local Brokering Configurations

Persistent Virtual

Environments

Metascheduling

Service

Workflow Manager

Clients

Broker

Chargeable Schedulable GridServices

Other Brokers

Site Feedback Policy

Manager

Resource Usage

Monitor

Banking

Services

Brokering and OGSA Services

1

Relevant GGF workRelevant GGF work

•Grid Protocol Architecture-RG : Core Grid Functions

•Grid Resource Allocation Agreement Protocol-WG :advanced reservation, co-allocation

•CIM-based schema-WG : successor to LDAP

•GESA-WG looking at economic issues of scheduling

The recently-formed Semantic Grid RG is very interested in the Grid Resource Ontology idea.

1

Points for DiscussionPoints for Discussion

•What is the relationship between brokering and scheduling?

•How to deal with legacy (not Grid-aware) schedulers?

•How to relate the ontologies from the application side (Resource Requestor) to the service provision side (Resource Provider)?

•How does a broker estimate upper and lower bounds for turnaround time?

•How far does the broker trust information from the service provider. Should it monitor running workflows?

January 17, 2003 GRIP First Project Review 1

RR space

RP space

RP space

RP spaceRR space

request

request

Request referral

sync

Figure 1: Request from RR space at A mapped into resource providers at B and C, with C forwarding a request formulated in RR space to RP space at D. B and C synchronize at end of workflow before results returned to the initiator A.

AB

CD

RR and RP SpacesRR and RP Spaces