a meta-brokering framework for science...
TRANSCRIPT
J Grid ComputingDOI 10.1007/s10723-016-9378-7
AMeta-Brokering Framework for Science Gateways
Krisztian Karoczkai ·Attila Kertesz ·Peter Kacsuk
Received: 20 January 2016 / Accepted: 7 September 2016© Springer Science+Business Media Dordrecht 2016
Abstract Recently scientific communities produce agrowing number of computation-intensive applica-tions, which calls for the interoperation of distributedinfrastructures including Clouds, Grids and privateclusters. The European SHIWA and ER-flow projectshave enabled the combination of heterogeneous sci-entific workflows, and their execution in a large-scalesystem consisting of multiple Distributed ComputingInfrastructures. One of the resource management chal-lenges of these projects is called parameter study jobscheduling. A parameter study job of a workflow gen-erally has a large number of input files to be consumedby independent job instances. In this paper we proposea meta-brokering framework for science gateways tosupport the execution of such workflows. In order tocope with the high uncertainty and unpredictable loadof the utilized distributed infrastructures, we introducethe so called resource priority services. These toolsare capable of determining and dynamically updat-ing priorities of the available infrastructures to beselected for job instances. Our evaluations show thatthis approach implies an efficient distribution of job
K. Karoczkai · A. Kertesz (�) · P. KacsukLaboratory of Parallel and Distributed Systems,MTA SZTAKI, Budapest, Hungarye-mail: [email protected]
K. Karoczkaie-mail: [email protected]
P. Kacsuke-mail: [email protected]
instances among the available computing resourcesresulting in shorter makespan for parameter studyworkflows.
Keywords Meta-brokering · Interoperability ·Distributed infrastructures · Workflows
1 Introduction
User communities worldwide use many kinds ofworkflow management and execution environments,which also raises interoperation problems amongthese working groups [27]. Workflow development,testing and validation are generally time-consumingprocesses and they require specific expertise to man-age. These tasks hinder the growth of the number ofavailable workflows and slow down the production ofresearch results, so it is important to reuse them. Com-munities using similar workflow engines can benefitfrom sharing previously developed and tested appli-cations, like using the myExperiment collaborativeenvironment [1].
Besides sharing, interconnecting and combiningthese workflows can lead to further research gainsand save time. Unfortunately, workflows developedfor one workflow enactment system is most of thetime not compatible with workflows of other systems.In the past, if two user communities using differentworkflow systems tried to collaborate, they had toredesign the application from scratch to the desired
K. Karoczkai et al.
workflow execution platform. To overcome these dif-ficulties, new workflow interoperability technologiesmust be developed, which was the basic goal of theEuropean SHIWA project [2]. By using the SHIWAtechnologies, publicly available workflows can beused by different research communities [25] work-ing on different workflow systems, and are enabled tobe executed on multiple distributed computing infras-tructures. As a result, workflow communities are notlocked any more to one workflow enactment systemand its supported computing infrastructure. The maingoal of the European ER-flow project [3] was to dis-seminate the achievements of the SHIWA project anduse these achievements to build workflow user com-munities in Europe. It provided application support toresearch communities within and beyond the projectconsortium to develop, share and execute workflowswith the SHIWA Simulation Platform. Both projectssupported the execution of parameter study workflowsover a growing number of DCIs, therefore it is cru-cial to develop such resource allocation algorithmsthat are capable of efficiently distributing a large num-ber of simultaneously submitted parameter study jobsamong resources of these heterogeneous infrastruc-tures. These requirements also serve as a motivationfor this work.
Managing these heterogeneous DCIs and schedul-ing the jobs of these scientific workflows are diffi-cult problems and require sophisticated approaches totackle. Many of these workflows apply the parame-ter study construct to examine large input sets withspecific algorithms (e.g. [1]). A parameter study (PS)job of a workflow generally has high number of inputfiles to be consumed by independent job instances.The state-of-the-art approaches for executing suchparameter study jobs can be classified to two differ-ent strategies: (i) pull models try to avoid relying onoutdated information on DCI load, therefore they starta pilot job in a DCI and pull jobs to it. Concern-ing the more traditional push approach (ii), generallyall job instances are submitted simultaneously to thescheduling component of the workflow managementsystem. This may cause significant overheads in ser-vice response time and bottleneck problems. There-fore additional information needs to be taken intoaccount during workflow execution. For example, Fanet al. [10] applied an estimation of execution and waittime prediction based on Karnak prediction servicein the SEAGrid science gateway using data mining
techniques. In this paper we measure the waiting timeinstead of using job run time estimations.
To cope with these problems, in this paper wepropose a meta-brokering framework for science gate-ways [26] to enable grouped DCI selection and sub-mission for these instances. As job scheduling in theseheterogeneous distributed systems is NP-hard [12],there is a need for solutions that apply some sort ofheuristics and sophisticated approximations. Some ofthese methods are based on runtime estimates and theinaccuracy of these estimates is a perennial problemmentioned in the job scheduling literature. Even ifusers are required to provide these values, there is nosubstantial improvement in the overall average accu-racy [9]. The background load of the utilized DCIs isalso hard to estimate, and only some of them providepublic information on dynamic resource load, whichare usually outdated [9]. Our current approach tries tofollow dynamicity in resource utilization with the helpof so called resource priority services.
Therefore the main contributions of this paper are:(i) the design of meta-brokering framework for sci-ence gateways to enable automatic DCI selection forparameter study jobs of workflows, and (ii) the evalu-ation of this approach in a real-world science gatewaysolution called gUSE.
The remainder of this paper is as follows: Section 2introduces the multi-DCI selection problem, andSections 3 and 4 describe our proposed meta-brokering framework to support the efficient execu-tion of parameter study workflows, including imple-mentation details and evaluations. Section 5 presentsimprovement ideas to our current solution. Finally,Section 6 discusses related works, and the contribu-tions are summarized in Section 7.
2 Multi-DCI Selection Problem
There are many user communities who need to accessseveral DCIs in a transparent way, but they don’t wantto learn the peculiar features of these infrastructures.They want to concentrate on the execution of theirscientific application. A science gateway [28] pro-vides an interface between a scientist (or the researchcommunity) and the distributed computing infrastruc-tures (DCIs). It can be imagined as a framework thatprovides a specific set of enabling technologies aswell as frontend and backend services that together
A Meta-Brokering Framework for Science Gateways
Fig. 1 Problem of multi-DCI selection in science gateways
build a generic gateway. These frameworks are notspecialized for a certain scientific area and hencescientists from many different areas can use them.Typical examples of such enabling technologies are:web application containers, portal or web applica-tion frameworks, database management systems, andworkflow management systems. To efficiently man-age multiple, heterogeneous distributed infrastructureswithin a science gateway, a meta-brokering approachis needed to coordinate, manage and efficiently dis-tribute parameter study job instances over the avail-able DCIs. Workflows applying the parameter studyconstruct have multiple input sets to be executed with
multiple job instances of the same program binary.Hence, a parameter study job has a large numberof input files to be consumed by independent jobinstances.
Meta-brokering means a higher level brokeringapproach that schedules user jobs among variousdistributed infrastructures. This approach can alsobe regarded as a federation management solution.Current state-of-the-art works usually target one ortwo infrastructure types to form a federation. E.g.in Grid infrastructures, the InterGrid approach [18]promotes interlinking of Grid systems through peer-ing agreements to enable inter-Grid resource sharing.Regarding Cloud systems, Buyya et al. [19] suggesta Cloud federation-oriented, opportunistic and scal-able application services provisioning environmentcalled InterCloud. They envision utility oriented fed-erated IaaS systems that are able to predict applicationservice behavior for intelligent down and up-scalinginfrastructures. Some works investigated federatingGrid and Cloud systems [20], but managing multi-ple infrastructures is rarely supported by multi-DCIbrokering.
The problem for multi-DCI selection in sciencegateways is depicted in Fig. 1. The traditionalapproach for execution parameter study jobs is to sub-mit all instances simultaneously (i.e. one-by-one) tothe scheduling component of the workflow manage-ment system of a science gateway that forwards themto a predefined infrastructure set by the user. Thismay cause significant overheads in service responsetime and bottleneck problems, and it can also overloadcertain DCIs.
Our proposed default meta-brokering approach tobe applied for parameter study jobs of a workflow dis-tributes the load evenly among the connected DCIsbased on their number of available resources. Thisapproach is shown in Fig. 2. This job distribution
Fig. 2 Defaultmeta-brokering approachfor DCI selection forparameter study jobinstances of a workflow
K. Karoczkai et al.
implies a load balancing that puts more job instancesto DCIs having higher throughput, thus having moreresources to execute a certain number of jobs.
Note that this approach does not take into accountthe background load of the managed DCIs (i.e. thealready running or waiting jobs on the actual infras-tructure). Specific DCIs like Grids may have mon-itoring components that provide such informationthrough so-called Information Systems. Unfortunatelythe monitored data they publish is generally outdated,and cannot be used for exact schedules. Not to men-tion that usually the execution times of the backgroundjobs are unknown, even user estimates are inaccurate[9] for their submitted jobs. This uncertainty makesjob scheduling even harder in Grids, and of course,this is the case for multiple, heterogeneous DCIs.Some DCIs such as local clusters may provide moreaccurate load information, but for example commer-cial clouds inherently hide load information on theunderlying infrastructure.
In an earlier work we have designed a meta-brokering approach [17] suitable for these needs hav-ing five major components (shown in Fig. 3). TheMeta-Broker Core is responsible for managing theinteraction with the other components and handlinguser interactions. The MatchMaker component per-forms the scheduling of the jobs by selecting a suitablebroker. This decision making is based on aggregatedstatic and dynamic data stored by the Information Col-lector component in a local database. The InformationSystem Agent is responsible for regularly updatingstatic and dynamic information on resource avail-ability from the interconnected infrastructures. TheInvoker component forwards the jobs to the selected
broker and receives the results. Each job submitted formatchmaking is supplied with a standard descriptiondocument containing their quality of service attributes.More information on these components and the uti-lized description language can be read in [17]. Bydeveloping our meta-brokering framework we tookinto account these theoretical guidelines, and createda concrete implementation to our DCI-Bridge tool tobe detailed in the next subsection.
Various DCIs have different characteristics, whichmakes them hard to compare. Service Grids haveactive resources that continuously execute submittedjobs, while desktop Grids have volatile resources thatcan go offline any time a volunteer donor wants touse it. Clouds have virtual resources that have tobe deployed before using them, which also affectsthe load and waiting time of the jobs. What com-mon in these characteristics is that in all DCIs acertain amount of delay exists that affects the exe-cution time of a submitted job. Note that once aDCI is selected by the meta-broker for a job (orgroup of jobs), the local scheduler of the appropriateDCI will perform the actual resource selection withinthe DCI.
In order to incorporate dynamic information to ourmeta-brokering approach we try to track and esti-mate this delay. This estimation will be incorporatedto the former default job distribution approach toarrive to a more dynamic solution capable of avoid-ing overloading certain DCIs. This extended approachwill be described in detail in the next section afterintroducing our reference science gateway architec-ture and the solution for extending it with the defaultmeta-brokering approach.
Fig. 3 The architecture ofGMBS
A Meta-Brokering Framework for Science Gateways
3 Static Meta-brokering for Science Gateways
Our proposed meta-brokering solution for parameterstudy workflows enables a randomized DCI selec-tion and submission for these PS instances to betterbalance the load among them.
In order to achieve this in a science gateway, ourapproach proposes a meta-brokering component forthe workflow managers. Traditionally each job of aworkflow is assigned to a DCI. During workflow exe-cution, the workflow manager uses the given DCIto submit and execute the actual job. With the helpof a meta-brokering component, the user can assigna job to more than one DCI, and the final execu-tion environment will be determined during runtime.The role of this new component is to select oneof the candidate DCIs. The default algorithm cantake into account the number of resources of eachcandidate DCI, and distribute parameter study jobinstances evenly among them (i.e. bigger DCIs shouldget more job instances). In this way the resourcenumber represents a static weight for the actualDCI.
3.1 Implementation in gUSE
Our proposed meta-brokering solution for parameterstudy workflows is developed for a science gatewayfamily called gUSE [8]. It provides necessary software
stacks to develop science gateway frameworks andinstances for various scientific communities (offeringa simplified user interface that is highly tailored to theneeds of the given community).
DCIs are managed by the so-called DCI-Bridgecomponent in gUSE. It has been developed as an inde-pendent service for science gateways capable of sub-mitting jobs of scientific workflows to different com-puting infrastructures. It hides the details of accessingthese DCIs, and accepts standard job descriptions (inXML-based Job Submission Description Languageformat – JSDL) sent through a standard BES (i.e.Basic Execution Service) interface. DCI-Bridges canbe chained together to enable job forwarding amongthem (with the help of so-called proxy DCI-Bridges).
There are different user roles within DCI Bridge(more details in [24]). First of all, (i) the systemadministrator is responsible for the definition of themain settings of the DCI Bridge objects (as shown inFig. 4), for adding or removing a connector of a certainresource to (or from) an existing middleware group,further for the enabling or disabling and observing theflow of jobs to a certain resource. (ii) The common(power- or end-) user can browse among the resourceswhich may be selected as the targets of the submis-sions to be defined during the workflow/job configu-ration phase. The common user may not change theactual settings of the DCI Bridge therefore he or shehas a slightly modified user interface.
Fig. 4 DCI-Bridge configuration
K. Karoczkai et al.
Fig. 5 DCI-Bridgearchitecture with DCIplugins
The DCI Bridge has three main important proper-ties. Firstly, it supplies the user interface, where thereferences about the resources of job executions aredefined. Consequently only those resources are visibleduring workflow/job configuration, which have been
defined by the Administrator user on DCI Bridge.Secondly, it contains the base routing parameters toremote resources, thus the actual job submissionscan be controlled and observed via the DCI Bridge.Thirdly, which is most important for this paper, the
Fig. 6 The extended DCI-Bridge with the Broker plugin
A Meta-Brokering Framework for Science Gateways
single interface of the DCI Bridge is an ideal inser-tion point for meta-brokering. In the background, withthe call of a special subroutine-like service it performsthe eventual late binding of jobs to resources of theavailable DCIs. A DCI-Bridge separates managementoperations of a concrete infrastructure into a plugin(as shown in Fig. 5). Therefore each plugin is respon-sible for storing jobs in its waiting queue (using theFirst Come First Served strategy - FCFS), submittingjobs to its DCI, monitoring their states and handlingpossible failures. The so-called Plugin selector com-ponent is used to determine which plugin should beselected to an incoming job request, based on its JSDLcontent.
We decided to develop a new plugin for our meta-brokering solution. This BROKER plugin is not con-nected to any DCIs, its role is to determine whichDCI plugin should be contacted to manage an actualjob submission. This decision is made on the infor-mation available in the actual job description (JSDL),and on the properties of the available DCIs. The JSDLcontains the possible list of target infrastructures,and a meta-brokering algorithm changes this list to aconcrete DCI (by modifying the JSDL) after select-ing one of them. Finally the modified JSDL is sentback to the Plugin Selector. The extended DCI-Bridge
architecture is shown in Fig. 6 (Resource PriorityServices will be introduced in the next section).
By default, our proposed meta-brokering approachuses the static resource number of the available DCIsto determine the submission environment. In the DCI-Bridge we represent the throughput of DCIs by pri-ority numbers, which are initially set to their staticresource number by the system administrator of a sci-ence gateway. This number represents a weight fora DCI, and the BROKER plugin uses these weightsto perform weighted randomized DCI selection forparameter study jobs of a workflow. This means thatDCIs are selected randomly, but a DCI having twicehigher weight then another gets selected with doubledprobability.
This also means that in a science gateway using thisapproach users need to specify a list of DCIs and analgorithm name for all jobs of a workflow, if meta-brokering is needed to execute the workflow. Whensuch a workflow is submitted, the JSDL of an actualjob is sent to the DCI-Bridge, and it is given to theBROKER plugin. First it checks, if the user is allowedto utilize the named DCIs (i.e. has valid proxies toexecute jobs on it). This step could be done in thescience gateway itself, but since the DCI-Bridge hasthese capabilities, it is reasonable to perform it here.
Fig. 7 The extended DCI-Bridge with the Broker plugin
K. Karoczkai et al.
If this checking process does not find a valid proxyto a DCI listed in the JSDL of a job, it will excludeit from the potential list of DCIs to be used for theselection process. This step is exemplified in Fig. 7.In this case the user listed three DCIs, and only haveproxies (i.e. credentials) for two of them (namely toHunGrid and LPDS Cloud), therefore these two DCIswill be the candidates for executing instances of theactual parameter study job.
Once the list of potential DCIs are determined,the BROKER plugin takes their priority values andperforms a randomized selection weighted with thepriorities. Such a selection can be imagined for all jobinstances as: all the candidate DCIs are put into a hatwith occurrences given by their priority values, andone candidate is taken out of the hat randomly. Theselected DCI will be written to the JSDL of the actualjob instance, and it will be sent back to the Pluginselector, then forwarded to the named DCI plugin toperform the actual submission.
In order to use the proposed meta-brokering featurein gUSE, first the administrator of a DCI-Bridge ser-vice needs to set the priority weights for each DCI –as shown in Fig. 8. Weights are numerical values that
typically reflect the capacity of a given resource (e.g.,the number of CPUs available).
During submitting a job of a scientific workflow,the brokering service in gUSE will choose one of theactual computing resources with a distribution corre-sponding to the weights (by using weighted randomdistribution). Resources with higher weights take partwith higher priority in the resource selection process,for example, brokering composed of three computingresources (local resource, gLite, cloud) and weights:1 (local), 2 (gLite), and 3 (cloud), will submit about16 % (local), 33 % (gLite), and 50 % (cloud) ofthe submitted jobs to the related resources. In caseof static brokering, these weights are basically con-stants; the distribution of the submitted jobs over theresources does not change in time.
“0” weight means that no jobs are brokered to thatDCI (e.g., LPDS OCCI), while DCIs having a positiveinteger value are potential candidates for the job exe-cution. Once these weights have been set and saved,the jobs of a workflow can use the MetaBroker ser-vice of gUSE by selecting resource type: “broker”for the actual job. This selection possibility is shownin Fig. 9. Users should have valid certificates to all
Fig. 8 Defining DCIweights for the MetaBrokerin DCI-Bridge
A Meta-Brokering Framework for Science Gateways
Fig. 9 MetaBroker configuration in gUSE
selected DCIs, and the selected algorithm will deter-mine the actual DCI selection based on the predefinedweights.
3.2 Evaluation
In order to evaluate our proposed meta-brokeringframework for science gateways, we have designedan artificial workflow that helped us to examine theintroduced brokering capabilities of our solution. Thisworkflow is shown in Fig. 10. It contains two gener-ation phases to create parameter study job instances:the first job (Gen10) generates 10 output files whichimplies the creation of 10 instances for the next 3 jobsin both paths, and the third jobs in these paths (labeled
Gen5A and Gen5B) generates 5 output files eachto result in 5 dynamically created instances for theremaining jobs sorted to four paths. These generatorjobs are marked with red circles in Fig. 10. The finaljobs of the workflow collects the results of all parame-ter study job instances. In this way the execution of theworkflow requires running 2062 job instances, whichwould take around 3 hours with a fully parallel execu-tion (theoretically). In practice data transfers, latenciesof the applied middleware and queue waiting timesintroduce additional execution time.
Our evaluation environment consisted of a gUSE-based science gatewaywith aDCI-Bridge connected tofour DCIs: two PBS clusters located at Italy and Hun-gary: INAF-PBS (with 200 nodes) and LPDS-PBS
Fig. 10 Test workflow for evaluation
K. Karoczkai et al.
(with 20 nodes) respectively, and two private cloudslocated at different regions of Budapest, Hungary:LPDS-Cloud (with 20 VMs) and SZTAKI-Cloud(with 30 VMs). In the first evaluation phase weexecuted the test workflow separately in all DCIs.We performed the measurements three times, andcounted their average shown in Table 1. Only theINAF PBS cluster had enough resources to performa fully parallel execution (meaning no queue waitingtimes occurred, only data transfers and communi-cation processes prolonged the theoretical executiontime). Therefore in the following measurements weexclude this cluster from the evaluation, and use onlythe remaining three DCIs.
In the second evaluation phase, we used the ear-lier introduced static meta-brokering capability of theDCI-Bridge service. In this case we configured theBroker plugin with the same priorities to the avail-able three DCIs, which means that the parameter studyjob instances are distributed equally among the DCIsduring workflow execution. The result is shown inTable 2. Since we used all DCIs this time to executethe parameter study job instances, we achieved someperformance gain compared to two previous singleexecutions. Nevertheless we experienced some VMfailures in the cloud infrastructures, therefore someinstances had to be reexecuted.
In the third evaluation phase we set the priori-ties according to the actual computing capacities (i.e.resource numbers) and previous performance valuesof the DCIs. The result is shown in Table II. In thisphase we experienced additional performance gains.Though we also had some VM failures in this case,the DCIs having more computing capacities receivedmore job instances than the others (i.e. the job distribu-tion was weighted by the priorities), which resulted inadditional speedup. The fourth row of the table showsthe number of executed instances per DCIs.
In order to evaluate our proposed meta-brokeringframework in a more dynamic environment, werepeated the third phase of Section 3.2 with increased
Table 1 Results of the first evaluation phase
DCIs of the science gateway
INAF-PBSLPDS-PBSSZTAKI-CloudLPDS-Cloud
Exec. time 3h 30min 13h 22min 4h 46min 6h 1min
Table 2 Results of the second and third evaluation phase
DCIs of the science gateway
LPDS-PBS SZTAKI-Cloud LPDS-Cloud
2nd phase Priorities 1 1 1
2nd phase Exec. time 7h 26min
3rd phase Priorities 1 3 2
3rd phase Distribution 339 979 744
3rd phase Exec. time 4h 1min
background load on the strongest DCIs. Therefore wegenerated additional load with additional jobs on cer-tain VMs of the SZTAKI Cloud to simulate a realworld utilization. In this case we experienced longermakespan for the test workflow, hence the result was5h 57min. The instance distribution was also modifiedin this case: our new algorithm distributed the jobsaccording to the predefined static weights.
Figure 11 shows the total number of jobs submittedto all DCIs. This experiment proved our initial hypoth-esis that dynamic changes in the background load canreally affect our brokered execution. As a result weneed to react to these changes by modifying the staticpriorities. This modification can be performed by ourproposed Resource Priority Service (to be discussedin the next section), but we have to mention that theinitial priorities for the DCIs and the normalizationfunction for refreshing the priorities should be cho-sen carefully with possible pretesting measurementsin order to achieve additional performance gains.
Fig. 11 The applied job instance distribution in the thirdevaluation phase
A Meta-Brokering Framework for Science Gateways
4 Dynamic Meta-brokering for Science Gateways
As we introduced in Section 2, generally there is aneed to change these DCI weights dynamically toavoid overloading an infrastructure, thus to reflect thebackground load in the weights (not only the sizeof a DCI). Next we detail, how DCI weights canbe updated, and present a proof-of-concept imple-mentation with a new component called a ResourcePriority Service (RPS). These components are inde-pendent from DCI-Bridges, and can use any logic andalgorithm to determine and refresh these priorities.
4.1 Implementation in gUSE
In order to modify DCI weights based on theirdynamic background load, we propose to regularlymeasure the waiting time on these infrastructures. Ourreference implementation for a Resource Priority Ser-vice uses so-called robot or probe jobs to determinethis latency for a DCI that reflects its background load.To transform the waiting time to weights (i.e. prior-ities as we call them later), we use this latency todecrease the predefined number of available resourcesthat serves as a base for determining DCI weights.
This task can be done by using some kind ofnormalization function. In general, the higher the mea-sured waiting time is, the lower the weight the actualDCI should have. To achieve this, in our referenceimplementation of the Resource Priority Service in[11] we apply f(x) = 1-(log(x)/5) for the waitingtime in minutes for getting a weight multiplier valuebetween 0 and 1. This value will be multiplied bythe predefined resource number in order to get lowerweights for higher waiting times. This normalization
0
0
0
0
Val
ues
0.60.650.7
0.750.8
0.850.9
0.951
2 5 10 15 20 2
Waiti
25 30 35 40
ing �me (min
0 45 50 55 6
n)
60
Fig. 12 The applied function for weight normalization
function is depicted in Fig. 12, where the x axis repre-sents the measured waiting time, and the y axis showsthe multiplier values. For a concrete science gatewaythe applied normalization function may be different,since the average waiting times can differ for variousDCIs, types of workflows and on the size of the activescientific community.
An actual priority update from an RPS is stored inthe DCI-Bridge under a unique algorithm name. Thismeans that in this solution the priorities of any DCIset can be managed and updated by an RPS under aspecial algorithm name. A reference proof-of-conceptimplementation of an RPS service [11] is alreadyavailable. This version uses the previously introducedand discussed weight normalization to determine DCIpriorities. This service is implemented as a web ser-vice that can be deployed in a web server (e.g. aTomcat server that hosts gUSE components and ser-vices).
Figure 13 illustrates, how robot jobs, also calledas probe jobs, are used to refresh DCI (also calledas Middleware) weights. We have two DCIs inthis case: Middleware A and Middleware B, bothhaving one-one resource. The following steps areneeded to determine or update the weights of theseDCIs:
1. The RPS should initiate the execution of a probejob on Middleware A and B (this can also be doneseparately)
2. a) The DCI-Bridge submits a probe job to Middle-ware A b) The DCI-Bridge submits a probe job toMiddleware B
3. The appropriate DCIs execute the probe jobs4. Once a probe job is finished, the DCI-Bridge gets
notified5. The DCI-Bridge sends a notification to the RPS
of finishing the probe job (separately for the twoDCIs)
6. The RPS service calculates and sends an updatedDCI weight to the DCI-Bridge (as a reply for eachnotification)
Before deploying an RPS service, a system admin-istrator needs to configure it with the WSDL of aDCI-Bridge to be connected to it. After deploying it,the service should be further configured with the man-aged algorithm name, with the JSDLs of robot jobsspecifying their DCIs and with the initial priorities ofthese DCIs. As a prerequisite, the repository of the
K. Karoczkai et al.
Fig. 13 Load predictionwith probe jobs
gUSE portal using the appropriate DCI-Bridge mustcontain these robot jobs as single workflows.
This approach also enables that different algo-rithms (i.e. priorities) can be used by the users forjobs of a specific workflow, as shown in Fig. 14. Thisfigure depicts a workflow having generator and collec-tor jobs used to manage parameter study job instances.Between these jobs the workflow contains compute-intensive and data-intensive PS instances. By usingdifferent algorithms for these two instance categories,more efficient workflow execution could be achieved.Therefore various scientific communities can developtheir own RPS to better represent and manage certainDCIs in their science gateways.
Note that random selection is an important featurefor selecting the submission environment. It could alsobe possible to choose the DCI with the highest priorityfor all job instances, but it could result in overloadingthe actual DCI. Randomized selection avoids this andalso selects DCIs with less priority to better balancethe load among the potential infrastructures.
5 Improvements
Though the presented meta-brokering framework canprovide significant performance gains for parameterstudy workflows in science gateways (SG), our real-world experiences show that further improvementscan be useful. As we have seen in Section 4, we userobot jobs, also called as probe jobs to determine thebackground load of a DCI.
If a DCI has a small number of computationalresources or nodes, these probe jobs can waste sig-nificant execution time. Nevertheless, if a DCI hasheterogeneous nodes (one is faster than the other),the measured load on a slow or overloaded nodemay rule out a generally well performing DCI fromthe selection process. These issues can hinder theperformance gains of the meta-brokering framework,therefore gathered some improvement suggestions inthis section.
Figure 15 depicts a situation for 12 PS job instancesto be executed, where a DCI (Middleware B) has a
Fig. 14 A sampleworkflow using differentDCI priorities
A Meta-Brokering Framework for Science Gateways
Fig. 15 Load distributionfor different DCIs
higher weight (i.e. 3, meaning that it is three timesfaster) than the other (Middleware A with weight 1),but it has only one resource behind a waiting queue.In this case we experience performance loss, becauseMiddleware A could execute more jobs than assigned(because it has a higher number of resources to par-allelize job executions) according to the measuredweight by probe jobs. This situation is further detailedin Fig. 16, by exemplifying parallel job executionsover time.
The optimal job distribution in this case wouldbe to submit more jobs to Middleware A, thus theweight determination should incorporate the number
Fig. 16 A possible unequal job distribution corresponding toFig. 15
of available nodes on a DCI, i.e. the level of par-allelization achievable on a DCI. Such an optimaldistribution is shown in Fig. 17. According to thisfinding, the RPS algorithm, taking into account onlyresource speeds measured by probe jobs, can providesignificant performance gain only for DCIs with sim-ilar parallelization capabilities (i.e. similar resourcenumber). As a result, we can improve the job distribu-tion by replacing the default RPS algorithm to countthe weight value by measured speed * paralleliza-tion capability. Thus the new weight for the situationdepicted in Fig. 17 will change from (B-3, A-1) to(B-3, A-4).
Fig. 17 Equal job distribution corresponding to Fig. 17
K. Karoczkai et al.
Fig. 18 Probe jobs inwaiting queues
Nevertheless we can still find cases resulting inan inefficient distribution. Therefore we can furtherimprove the distribution by revising the weight calcu-lation with the following equations for n DCIs:
weighta=(speeda /�speed)*(parrarela /�parrarel
weightb=(speedb /�speed)*(parrarelb /�parrarel
. . .
weightn=(speedn /�speed)*(parrareln /�parrarel
The second problem we experienced is that theprobe jobs used for determining the speed of DCIresources can get stuck in waiting queues and reservefree resources causing additional waiting of user jobs
as shown in Fig. 18. In such cases a new weight calcu-lation for a group of jobs is done by taking the latestmeasured speed value of a DCI, which is definitelybetter than the one will be measured for the actu-ally waiting probe job. A good solution for this wouldbe to modify the measurement process, and track theexecution process of these probe jobs, and mark theelapsed time since submitting the probe job. We canenable this by modifying the DCI-Bridge, not towait for weight updates from the RPS, but to triggerweights for DCIs right before a meta-brokering phase.This modified version is shown in Fig. 19.
The third problem we found is the presence of thirdparty jobs in DCIs. This is one of the main reasons
Fig. 19 Triggered weightcalculation by DCI-Bridge
A Meta-Brokering Framework for Science Gateways
uncertainty is preset in distributed systems. SomeDCIs,such as Grids or clusters may allow job submissionsand executions from different portals or gateways.In this situation the science gateway we are using isunaware that there are other jobs in the system thanthe gateway submitted. This case is shown in Fig. 20.
According to the calculated weights the systemexpect that a certain number of jobs can be run in par-allel (e.g. 4 jobs in Middleware A of Fig. 20), but only2 of them can be executed after submission, since 2other jobs (marked with grey color) occupy the restof the nodes. This means that optimal job distribution(and the highest performance gain) can be achievedonly for such DCIs, for which the actual sciencegateway has full control. Typically cloud infrastruc-tures can provide this. For others the meta-brokeringalgorithms have to be developed by taking into suchuncertainty. We have also researched solutions forthese cases by applying fuzzy approaches [21].
6 Related Works
Scheduling in Grid systems, which is one of the tasksof Grid resource managers, become even more com-plicated with multi-organizational shared resources,therefore Grid scheduling is also NP-hard [12].Schwiegelshohn et al. [13] showed that the perfor-mance of Garey and Graham’s list scheduling algo-rithm is significantly worse in Grids than in generalmultiprocessor systems.
While the principle of workflows is easy to con-ceive, their execution is very complex and no generalsolution exists. In an earlier joint work by Bacsoet al. [23] we examined the resource allocation chal-lenges of parameter study jobs in distributed com-puting infrastructures, and proposed a series of joballocation models that helps refining and simplifyingthe problem complexity. We examined some specialcases that are polynomial and showed how more com-plex scenarios can be reduced to these models. Thestate-of-the-art approaches for executing parameterstudy workflows show high diversity based on theirapplication area and execution environments. Hirales-Carbajal et al. [5] present an experimental study of22 deterministic non-preemptive multiple workflowscheduling strategies in Grids. While their objec-tive is to schedule and execute the whole workflow,and minimize its makespan, we restrict ourselves to
parameter study jobs of such workflows. Oprescuet al. [6] propose a budget constraint-based resourceselection approach for cloud applications, which canschedule bags of tasks onto multiple clouds with dif-ferent CPU performance and cost, minimizing com-pletion time with maximized budget. Their schedulerlearns to estimate task completion times at run time.GridBot [7] represents an approach for execution ofbags-of-tasks on multiple Grids, clusters, and volun-teer computing Grids. It has a Workload Managercomponent that is responsible for brokering amongthese environments, which is similar to our approach,but they focus on tasks more suitable for volunteerGrids.
Casanova et al. [14] focused on the same prob-lem of scheduling parameter sweep applications onGrids, with particular attention to file transfers andnetwork performance. Also, their motivation is inalignment with most of the papers in this area. Theirapproach is modifying existing heuristics so that theyare adaptive in a dynamic heterogeneous environment.The core of the scheduling is a Gantt chart that iscreated and updated periodically and keeps track ofjob and resource assignments. Assignments are gov-erned by a heuristics called sufferage, where a task isassigned to a host if the task would suffer the mostif done otherwise. Maheswaran et al. [15] addressesthe issue of mapping independent tasks onto hetero-geneous computing systems. They apply heuristicsaiming at optimizing for throughput, i.e. increasingthe finished task per time unit ratio. J. L. Lucas-Simarro et al. [16] proposed different schedulingstrategies for optimal deployment of services acrossmultiple clouds based on various optimization crite-ria. The examined scheduling policies include bud-get, performance, load balancing and other dynamicconditions.
The execution of parameter study jobs can be clas-sified to two strategies: the push and pull models.Pull models try to avoid relying on outdated informa-tion on DCI load, therefore they start a pilot job in aDCI and pull jobs to it, i.e. feed the actual resourcewith own jobs. For example DIRAC and GWpilot [4]use this approach. Concerning the more traditionalpush approach, generally all job instances are submit-ted simultaneously to the scheduling component ofthe workflow management system. This may causesignificant overheads in service response time andbottleneck problems.
K. Karoczkai et al.
Fig. 20 Presence of thirdparty jobs
Since science gateways inherently work with vari-ous infrastructures, previous works are hard to apply.Scheduling among these diverse infrastructures needto understand resource utilization and handle vary-ing resource load. In this work we propose a solutionthat is capable of executing many parameter study jobinstances over several diverse DCIs.
7 Conclusion
Managing heterogeneous DCIs and scheduling param-eter study jobs of scientific workflows are also diffi-cult problems and require sophisticated approaches. Inthis paper we proposed a meta-brokering frameworkfor science gateways that use the push model for jobsubmission in order to support the efficient executionof parameter sweep jobs and workflows containingthis type of nodes. We apply a dynamic, weighted jobinstance distribution among the available infrastruc-tures, and rely on resource priority services to copewith the high uncertainty and unpredictable load of theutilized infrastructures. Our evaluation showed thatthis approach implies more efficient distribution of jobinstances among the available computing resourcesthan the random DCI selection algorithm resulting inshorter makespan for parameter study workflows.
Nevertheless there is still room for further improve-ments. Various science gateways and user communi-ties may have different needs, therefore own metricscould be defined for custom resource priority ser-vices. In this approach we only took into account the
measured waiting times by the priority service, whichworks well for DCIs having homogeneous resources.We encourage the development of additional method-ologies and algorithms to this framework by interestedresearch groups addressing more heterogeneous DCIsand specific workflows.
Notice that the proposed solution can be used forany science gateway that uses the OGF BES jobsubmission standard. Such gateways can easily beextended with the DCI Bridge and the resource prior-ity services. Since DCI Bridge supports every majorDCI types and the internal resource selection algo-rithm of the resource priority services can easily bechanged the joint usage of these two services provide avery flexible framework to distribute parameter sweepjobs among various infrastructures using the majorDCI types.
In order to demonstrate the practical usage ofthese principals, WS-PGRADE/gUSE, one of themost widely used science gateway frameworks, wasextended with this concept. From gUSE version 3.7.1,all the WS-PGRADE/gUSE gateways contain themeta-broker support. Meta-brokering is further inves-tigated in the VIALACTEA project where PBS clusterwith different capacity provide the underlying compu-tational infrastructure for the project.
Acknowledgments The research leading to these results hasreceived funding from the European Community’s SeventhFramework Programme FP7/2007-2013 under grant agreementno. 607380 (VIALACTEA), no. 312579 (ER-FLOW) and no.608886 (CloudSME). This paper is a significantly revised andextended version of the workshop paper [22].
A Meta-Brokering Framework for Science Gateways
References
1. Wiggins, A.: Success-Abandonment-Classification work-flow at myExperiment. Online: http://www.myexperiment.org/workflows/140.html (2012)
2. SHaring Interoperable Workflows for large-scale scien-tific simulations on Available DCIs (SHIWA) EU FP7project.Online: http://www.shiwa-workflow.eu/ (2012)
3. Building a European Research Community through Inter-operable Workflows and Data (ER-flow) Eu FP7 project.Online: http://www.erflow.eu/ (2013)
4. Rubio-Montero, A.J., Huedo, E., Castejon, F., Mayo-Garcia, R.: GWpilot: Enabling multi-level scheduling indistributed infrastructures with GridWay and pilot jobs. Fut.Gener. Comput. Syst. 45, 25–52 (2015)
5. Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R.,Gonzalez-Garcia, J.L., Roblitz, T., Ramirez-Alcaraz, J.M.:Multiple workflow scheduling strategies with user run timeestimates on a grid. Journal of Grid Computing (2012)
6. Oprescu, A., Kielmann, T.: Bag-of-tasks scheduling underbudget constraints. CloudCom, 351–359 (2010)
7. Silberstein, M., Sharov, A., Geiger, D., Schuster, A.:GridBot, execution of bags of tasks in multiple grids.In: Proceedings of the Conference on High Perfor-mance Computing Networking, Storage and Analysis(SC ’09) (2009)
8. Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann,G., Balasko, A., Karoczkai, K., Marton, I.: WS-PGRADE/gUSE generic DCI gateway framework for alarge variety of user communities. J. Grid Comput. 10(4),601–630 (2012)
9. Lee, C.B., Schwartzman, Y., Hardy, J., Snavely, A.: Areuser runtime estimates inherently inaccurate? SpringerLNCS 3277, 253–263 (2005)
10. Fan, Y., Pamidighantam, S., Smith, W.: Incorporating jobpredictions into the SEAGrid science gateway. ACM, NY,USA (2014)
11. Resource Priority Service for gUSE. Online: http://sourceforge.net/projects/priorityservice.guse.p/ (2015)
12. Ullman, J.D.: NP-complete scheduling problems. J. Com-put. Syst. Sci. 10(3), 384–393 (1975)
13. Schwiegelshohn, U., Tchernykh, A., Yahyapour, R.: Onlinescheduling in grids. 22nd IEEE International Symposiumon Parallel and Distributed Processing (IPDPS 2008), pp.1–10 (2008)
14. Casanova, H., et al.: Heuristics for scheduling parametersweep applications in grid environments. HeterogeneousComputing Workshop, 2000. (HCW 2000) Proceedings.9th. IEEE (2000)
15. Muthucumaru, M., Ali, S., Siegal, H.J., Hensgen, D.,Freund, R.F.: Dynamic matching and scheduling of aclass of independent tasks onto heterogeneous computing
systems. In: Heterogeneous Computing Workshop, 1999.(HCW’99) Proceedings, pp. 30–44. IEEE (1999)
16. Lucas-Simarro, J.L., Moreno-Vozmediano, R., Montero,R.S., Llorente, I.M.: Scheduling strategies for optimal ser-vice deployment across multiple clouds. Future Genera-tion Computer Systems, doi:10.1016/j.future.2012.01.007(2012)
17. Kertesz, A., Kacsuk, P.: GMBS: A new middleware servicefor making grids interoperable. Fut. Gener. Comput. Syst.16, 542–553 (2010)
18. Assuncao, M.D., Buyya, R., Venugopal, S.: InterGrid: Acase for internetworking islands of grids. Concurrency andComputation: Practice and Experience (CCPE) (2007)
19. Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: Utility-oriented federation of cloud computing environmentsfor scaling of application services. Lect. Notes Com-put. Sci. Algorithm. Architectures Parallel Process. 6081(2010)
20. Buyya, R., Ranjan, R.: Special section: Federated resourcemanagement in grid and cloud computing systems. Fut.Gener. Comput. Syst. 26, 1189–1191 (2010)
21. Kertesz, A., Maros, G., Dombi, J.D.: Multi-job meta-brokering in distributed computing infrastructures usingpliant logic. In: Proceedings of the 22th Euromicro Inter-national Conference on Parallel, Distributed and Network-Based Computing (PDP’14), pp. 138–145. IEEE CS, Turin,Italy (2014)
22. Karoczkai, K., Kertesz, A., Kacsuk, P.: Brokering solu-tion for science gateways using multiple distributed com-puting infrastructures. In: 7th International Workshopon Science Gateways (IWSG), pp. 28-33, 3–5 (2015).doi:10.1109/IWSG.2015.12
23. Bacso, G., Kis, T., Visegradi, A., Kertesz, A., Nemeth, Z.S.:A set of successive job allocation models in distributedcomputing infrastructures (2015). doi:10.1007/s10723-015-9347-6
24. DCI-Bridge User Manual. http://sourceforge.net/projects/dcibridge/files/DCI-Bridge-3.5.2/Documentation/DCIBridgeManualv3.5.2.pdf (2015)
25. Korkhov, V., Krefting, D., Kukla, T., et al.: Explor-ing workflow interoperability for neuroimage analysis onthe SHIWA platform. J Grid Comput. 11, 505 (2013).doi:10.1007/s10723-013-9262-7
26. Kiss, T.: Science gateways for the broader take-up of dis-tributed computing infrastructures. J Grid Comput. 10, 599(2012). doi:10.1007/s10723-012-9245-0
27. Liu, J., Pacitti, E., Valduriez, P., et al.: A survey of data-intensive scientific workflow management. J Grid Comput.13, 457 (2015). doi:10.1007/s10723-015-9329-8
28. Kacsuk, P. (ed.): Science Gateways for DistributedComputing Infrastructures: Development Framework andExploitation by Scientific User Communities, Springer. pp.301 (2014)