a scheduling framework for enterprise servicescgi.di.uoa.gr/~sekkas/papers/jss_scheduler.pdf · as...

www.elsevier.com/locate/jss

The Journal of Systems and Software 79 (2006) 259–272

A scheduling framework for enterprise services

Vassileios Tsetsos, Odysseas Sekkas, Ioannis Priggouris, Stathes Hadjiefthymiades *

Department of Informatics and Telecommunications, University of Athens, TYPA Bldg., Panepistimiopolis, Ilissia, GR-15784 Athens, Greece

Received 25 January 2005; received in revised form 28 April 2005; accepted 22 May 2005Available online 11 July 2005

Abstract

Application-level scheduling is a common process in the enterprise domain, but in order to be productive it should be time-effec-tive, interoperable and reliable. In this paper we present the design and implementation aspects of a modular scheduling solution,which aims to fulfill the above-mentioned requirements. The proposed solution constitutes an integral part of an existing serviceprovisioning platform targeted to the location based services (LBS) domain; however application to other service-oriented architec-tures is also possible. A careful design process is followed in order to identify the scheduler�s functional subsystems and the relation-ships between them. The implementation is based on the J2EE framework, thus guaranteeing independence from underlyingtechnologies and applicability in enterprise environments. The high quality and scalability of the scheduler is also validated throughan intensive performance analysis.� 2005 Elsevier Inc. All rights reserved.

Keywords: Time-critical scheduling; Enterprise applications; J2EE; Push paradigm; Middleware platform

1. Introduction

Effective task scheduling is a ‘‘hot’’ topic in the com-puting community since its early years. Effectiveness, ingeneral, is a vague notion unless we relate it to the dif-ferent types of scheduling. Roughly speaking, schedul-ing can be divided in two broad categories: low levelscheduling and application (or service) level enterprisescheduling. The term ‘‘enterprise’’ intends to point outthe large-scale nature of this category. The first categoryconsists of scheduling techniques and systems in multi-tasking Operating Systems or network elements suchas QoS-aware routers. These systems should be extre-mely fast and support priority-based scheduling. Onthe other hand, the second category includes schedulers

0164-1212/$ - see front matter � 2005 Elsevier Inc. All rights reserved.doi:10.1016/j.jss.2005.05.014

* Corresponding author. Tel.: +3021 0727 5327; fax: +3021 07275601.

E-mail addresses: [email protected] (V. Tsetsos), [email protected] (O. Sekkas), [email protected] (I. Priggouris), [email protected] (S. Hadjiefthymiades).

that should execute tasks either in real-time or sometime in the future, periodically or not. Such schedulersmay be responsible for the execution of batch jobs orfor time-specific service provisioning. Moreover, tasksmay be scheduled upon the triggering of events (e.g.,in case John Doe logs in the main server, the system in-forms the administrator by e-mail). They should be reli-able and highly scalable in order to be usable in theenterprise domain.

In this paper the architecture of an enterprise sched-uling service is presented. This service, referred asScheduler from now on, was developed as an integratedcomponent of a service provisioning platform used inthe mobile networking context. Its role is to supportthe time/event-triggered service delivery. Although themiddleware and service provision platform was focusedon mobile-oriented services (e.g., location-based andcontext-aware services) the design of the Scheduler wasbased on a much more general-purpose requirementsanalysis and can support a wide range of real-time appli-cations. A prototype implementation of the Scheduler

mailto:[email protected]

mailto:o.sekkas@ di.uoa.gr

mailto:o.sekkas@ di.uoa.gr

mailto:[email protected]

mailto:shadj@di. uoa.gr

mailto:shadj@di. uoa.gr

260 V. Tsetsos et al. / The Journal of Systems and Software 79 (2006) 259–272

was also developed and tested for its performance andefficiency. Implementation was based on Java technol-ogy, which comprises an open and widely adopted stan-dard in the enterprise domain.

The rest of the paper is structured as follows. In Sec-tion 2 we provide a brief summary of work related toservice scheduling. Section 3 presents the architectureand implementation of the Scheduler, starting fromthe requirements specification that drove the designphase and concluding with an in-depth analysis of thearchitecture and key implementation decisions. Section4 summarizes the results of the various tests carriedout in order to assess the performance and stability ofthe prototype. In Section 5 we describe in brief PoLoS,a middleware platform for location-based services,which hosted the Scheduler service in its architecture.In Section 6, we discuss some scalability and clusteringissues. Finally, future work envisaged is presented fol-lowed by some conclusions.

1 Some developers define ‘‘job’’ as a set of ‘‘tasks’’. In this paperthese terms have the same meaning and are used interchangeably.

2. Requirements analysis and related work

An enterprise scheduler, as part of a broader enter-prise solution, has to meet a lot of critical requirements.These requirements can be classified as functional (i.e.,company and end-user requirements) and technical.They also vary according to the type of services thescheduler is intended to provide. Existing solutions wereevaluated in order to identify the desired characteristics,which apply to all general purpose real-time enterpriseschedulers (e.g., large numbers of clients, high qualityof offered services, etc.).

The most important requirement is the accuracy ofthe scheduling engine. A highly accurate engine guaran-tees that the scheduled tasks will commence executionon time or, at least, with the lowest possible delay. Weshould also note, that end-users are typically interestedin low turnaround times (i.e., from task registration tilltask completion), since only at the end of task execu-tions they obtain the requested results. From the sched-uler perspective, however, we are interested only instarting delays, since turnaround times are affected byother factors beyond scheduler control (e.g., networkdelays). This delay should, in any case, be very short(e.g., in the order of milliseconds) so as not to decreasethe reliability of the scheduling service. Furthermore,the users should be able to schedule both periodic andaperiodic tasks. In the first case, the capability of deter-mining an ending time would be required. At this point,it should be noted that the ways a task can be scheduledvary from very simple to very expressive and flexible.The final decision should be taken according to therequirements of the targeted application.

The performance of the scheduler should be high,even when hundreds of tasks have to be executed con-

currently. This requirement stems directly from thelarge-scale nature of the enterprise applications and ser-vices. In general, scalability is one of the most importantfeatures that a well-designed scheduler should provide.

Another important feature of an enterprise scheduleris persistence. Scheduled jobs need to be saved in a per-sistent store (e.g., a relational, object or XML database).This will enable them to automatically be rescheduledafter a system crash, thus catering for a smooth failover.Stored information could be further exploited by theenterprise, for management purposes or for issuing sta-tistical reports.

Logging is an efficient and flexible mechanism formonitoring scheduler�s activities. As the scheduler trig-gers the execution of enterprise jobs, it has a crucial rolein the provisioning process which makes it suitable forthe maintenance of logs. These logs can be used foradministrative, charging and billing purposes. Specialprovision should be taken, though, so that the loggingactivity does not degrade the scheduler�s performance.

Apart from the general requirements, there are tech-nical requirements that need to be satisfied. Portabilityacross different operating systems, application serversand databases is the most important of them, as it en-sures independence from specific platforms and poten-tial use by a large market share.

Other requirements, such as stability and support forclustering and load-balancing, are not considered, be-cause most application servers implement such function-alities. In Section 6, we elaborate on the clustering andload-balancing issues. A user-friendly and fully func-tional graphical user interface for administration pur-poses is also desirable, but it is more considered as anexternal utility rather than part of the scheduler�s corearchitecture. Another potential requirement is the capa-bility to assign priorities to tasks. CPU and other typesof schedulers employ priority queues to separate themission-critical tasks from the less urgent ones. How-ever, this separation is not very important for a real-timescheduler where the only differentiation between itstasks is their time of execution.

The first popular job scheduler was cron, a UNIXutility. Cron executes jobs1 (mainly shell scripts) sched-uled on an hourly, daily, weekly or monthly basis andis still an integral part of any Unix/Linux platform.However, cron is not targeted to the enterprise domain,as its main role is to assist in administering UNIX sys-tems. On the other hand, many companies have pro-duced and provide commercial enterprise schedulers.Below we present briefly the most important of them,highlight their core features and point out the commonfunctionality they have.

V. Tsetsos et al. / The Journal of Systems and Software 79 (2006) 259–272 261

Flux (Sims, 2002) is a commercial job scheduler thatcan be used either as a standalone program or as a soft-ware component. A user can schedule time-driven,event-driven and file-driven tasks (triggered by the crea-tion or update of specific files). Flux features a rich anduser-friendly graphical user interface (GUI) for creating,editing and monitoring jobs. It also enables the creationof very expressive and flexible schedules, which can beexecuted even in different time zones. Finally, it includesclustering and failover capabilities. The latter is per-formed with the assistance of multiple deployed sched-uler instances.

Kronova eScheduler (Dale, 2003) is a scheduler ad-dressed solely to the enterprise domain. It is used mainlyfor scheduling batch jobs, such as weekly backups andmonthly reports. eScheduler supports many types oftasks: operating system commands, Enterprise Java-Beans (EJBs), local and remote Java objects and e-mailmessages. The main component of its scheduling engineis a session EJB and the scheduled jobs are implementedas entity EJBs.

To the best of our knowledge, the only open sourceJ2EE (Java 2 Enterprise Edition) (Cattell et al., 2000)scheduler is Pulsar (Bhattacharya, 2003). The core ofPulsar�s engine is based on the standard Java Timer class(Joy et al., 2000). The scheduler runs as a servlet and thetasks are either stateless sessions EJBs or Java classes,which can be scheduled using an XML (ExtensibleMarkup Language) file containing all the requiredparameters. Currently, there is support only for oneapplication server and, in general, this scheduler lacksthe advanced features of the aforementioned commer-cial products.

The above-mentioned solutions have much function-ality in common; indicative examples are persistence andlogging support. Furthermore, most of them come witha graphical management tool, while all of them arebased on EJBs in order to take advantage of the facili-ties provided by application server containers (see Sec-tion 3).

Concluding this section we should point out thatalthough task (job) scheduling constitutes a central pro-cess in many enterprise systems, there is limited pub-lished work on the design and performance evaluationof such software. The present paper tries to describe indetail such a design and evaluation process in order tosupplement the related bibliography and act as a guidefor future similar implementations.

3. Architecture and implementation

3.1. Technology overview

Trying to meet the requirements identified in the pre-vious sections, the Scheduler capitalized on state-of-the-

art enterprise technologies and standards, like the J2EE1.3 application framework (Cattell et al., 2000). Opensource tools and software were used for the implementa-tion of the Scheduler�s prototype and extra features(e.g., logging support).

The main advantage of the J2EE application frame-work is the built-in EJB 2.0 support. EJBs are distrib-uted components running within a container of theapplication server. They allow the separation of applica-tion logic from system-level services thus allowing thedeveloper to concentrate on the business domain issuesand not on system programming. In addition, the EJBcontainer takes care for advanced issues such as instancepooling, transactions, security and caching. Three EJBtypes are currently defined in the J2EE specification: ses-sion, entity and message-driven beans (MDBs). Sessionbeans, either stateless or stateful, represent the enter-prise processes and implement the application logic. En-tity beans perform the object-oriented mapping to anunderlying datastore table. There are two entity beantypes: those with Container Managed Persistence(CMP) and those with Bean Managed Persistence(BMP). CMP entity beans are more flexible as they aremapped to database entries through an abstraction layerthat makes them portable across different databases.Database transactions are also handled transparentlyby the container. Message-driven beans are specialEJB components that can receive JMS (Java MessageService) messages and consume these messages fromqueues or topics where they are placed by any validJMS client (Cattell et al., 2000). A message-driven beanis decoupled from the clients that address messages to it.It is the server�s responsibility to provide concurrentmessage consumption by pooling multiple message-driven bean instances (Roman et al., 2002). Thesebeans are ideal for implementing asynchronous commu-nication and binding with other message-orientedapplications.

3.2. Architecture overview

Based on the requirements identified in Section 2, thefollowing core subsystems were identified for the sched-uling service:

• the Scheduling Subsystem, which includes the client-side scheduler interface and the main registrationmechanisms;

• the Task Execution Subsystem, which is responsiblefor the accurate execution of the scheduled tasks;

• the Persistence Subsystem, which performs the persis-tence of the scheduled tasks, thus providing thedesired reliability.

The core element in the architecture is the Scheduler-Bean, which is responsible for the initialization of the

Fig. 1. Scheduler architecture.


scheduler and the scheduling/removal of tasks. Thiscomponent can be used for scheduling two differenttypes of tasks:

1. Time-triggered tasks, which can be either periodic oraperiodic (Franklin and Zdonik, 1998). These taskshave certain time constraints, which are defined upontheir registration and should be respected until theirexpiration. Such tasks are registered on the Timer

object, which is responsible for their timely execution.2. Event-triggered tasks. These are tasks requiring

immediate execution, without any time constraints.Although not strictly bound to the scheduling para-digm such tasks are supported by the scheduling ser-vice in order to provide a unified service executioninterface.

The SchedulerBean exports a well-defined interfacetowards client applications that need to perform opera-tions on the Scheduler. The SchedulerBean uses theTaskBean component in order to persist tasks in a rela-tional database. The primary key of each stored task is aunique identifier generated by the SequenceBean. Event-triggered tasks are not stored in the database, as theirexecution is not likely to be repeated in the future; in-stead they are directly forwarded for execution to theTaskHandler.

When a new task needs to be executed the Scheduler-Bean forwards it to the Task Execution Subsystem. Com-munication between the two subsystems is implementedasynchronously using the message queuing model. TheSchedulerBean places the new task in the Queue fromwhere it is instantly consumed by the TaskHandler. Ofcourse the time consumption can potentially varydepending on the load experienced by the Queue. Subse-quently the TaskHandler acts as a simple client for theservice/task to be invoked by commencing its execution.As will be shown in Section 4, the asynchronous mes-sage-oriented implementation of the Task Execution

Subsystem, which decouples the core scheduling enginefrom the execution of the tasks, has a significant effecton the overall performance of the scheduling mechanism.

Finally, the Scheduler features a flexible and reconfig-urable logging mechanism, which can log every internalactivity with little additional performance overhead. Thescheduler architecture along with the relationships be-tween the different subsystems and their components isdepicted in Fig. 1. Further elaboration on each subsys-tem and implementation details are provided below.

3.3. Detailed design and implementation

The components comprising the different subsystemsof the Scheduler are implemented as EJBs, thus allowingtheir operation in distributed environments. All subsys-tems were designed and implemented so that they can bedeployed inside any J2EE-compliant application server,either on the same machine or on remotely connectedhosts. The former approach is more efficient in termsof delay and time-response (due to the local interfacesdefined in EJB 2.0 specification), while the latter pro-vides all the advantages of a distributed and, thus,highly scalable architecture. The implementation detailsfor the Scheduler are presented in the following sections.Implementation took place on the Jboss (Stark et al.,2002) application server, which was also used for thetesting phase. The core scheduling engine was imple-mented using the standard Java Timer class (Joy et al.,2000) which is highly scalable. As it will be shown inthe section of the performance analysis later on, thisimplementation proved to be very efficient. Finally, thelightweight Log4J (Gulcu, 2003) framework, developedby the Jakarta Project (Goodwill, 2001) team, was usedfor the logging mechanism.

3.3.1. The scheduling subsystemThe core component of the subsystem is the Schedul-

erBean, which is modeled as a stateless session bean. The

Table 1The methods of the scheduler

Method Arguments Return value

Initialize – –

Schedule userID: the identity of the user that scheduled the taskinvocationParameters: information regarding the start time,stop time and execution period of the scheduled task

scheduleID: a unique ID identifying the specific task.It is also used as a primary key for the persistence mechanism

taskData: parameters related to the task execution logic

Remove scheduleID: the ID of the task to be removed –


SchedulerBean is responsible for scheduling tasks in theTimer or sending tasks directly for execution in theTaskHandler. Furthermore, it collaborates with the Per-sistence Subsystem, which stores the information of eachtask in a database in case of system failure. The Sched-

ulerBean has three basic methods: ‘‘initialize’’, ‘‘sche-dule’’ and ‘‘remove’’ (see Table 1).

Method ‘‘initialize’’. This method initializes theScheduler. Initialization includes among others therescheduling of all tasks that are stored in the database.Every task retrieved from the database is checked for theexpiration date and if it has not expired yet is resched-

Fig. 2. The sequence diagram

Fig. 3. The sequence diagram of meth

uled, or else is removed from the database. The relevantUML Sequence Diagram can be viewed in Fig. 2.

Method ‘‘schedule’’. This method is invoked when anew task needs to be scheduled. After retrieving a un-ique sequence number (primary key) from the Sequence-Bean, the task is scheduled in the Timer. This number isused as the primary key of the new record in the data-base, where all the information about the task is stored.Persistent data include start and stop time and the exe-cution period. Fig. 3 shows the steps performed by thismethod. The procedure just described stands for time-triggered tasks. In the case of an event-triggered task,

of method ‘‘initialize’’.

od ‘‘schedule’’ (time-triggered).


execution takes place immediately without the need forTimer registrations. Execution data is sent directly tothe TaskHandler.

Method ‘‘remove’’. The method is called for removinga scheduled task from the Timer. After its removal thetask is removed from the database and will not be exe-cuted again.

Each task is implemented as object of the class Taskand has a start time, a stop time and a period. We referto all of them as ‘‘invocation parameters’’. Start is thetime that the scheduled task will be invoked for the firsttime. If not specified, immediate execution is performed.Stop is the time that the task will be invoked for the lasttime and, then, will be removed from the Scheduler. Ifnot specified, infinite execution is assumed. Period de-fines the period in the case of periodic events. If notspecified (or has value equal to 0), indicates that the taskwill be executed at most twice (once at the start and an-other one at the stop time, unless those two are thesame). If not specified (or has value equal to 0) and stopis not specified also the event will take place once at thestart time. There are eight possible combinations of theinvocation parameters, which are presented in Table 2.E stands for ‘‘Exists’’ and N stands for ‘‘Not specified’’.For example, if invocation parameters of a task areENE this means that the task has a start time, the stoptime is set to ‘‘Not specified’’ and has a period differentfrom 0. All these combinations after being processed bythe SchedulerBean, are reduced to two (highlighted inTable 2) for easier handling and better performance.So, if start time is N then as start time is set the currenttime and thus it is transformed to E. The EEN and ENEcases are also transformed to EEE. Apart from invoca-tion parameters and ID, the object Task has anotherattribute which defines the data of the service to be exe-cuted. A service can be represented by any Java objector servlet.

The Timer is the most important component of thearchitecture. It extends the java.util.Timer class. Thisclass provides a scheduling engine that is responsiblefor the execution of tasks. Each Timer object uses a sin-gle background thread for executing all of the timer�stasks. Internally, it uses a binary heap to representits task queue, so the cost to schedule a task is

Table 2Possible combinations of invocation parameters

Start Stop Period

E E E

E E NE N EE N N

N E EN E NN N EN N N

O(logn), where n is the number of concurrently sched-uled tasks.

Tasks may be scheduled for one-time execution(ENN case), or for repeated execution at regular inter-vals (EEE case) in two ways: fixed-delay execution andfixed-rate execution. In fixed-delay execution, each exe-cution is scheduled relative to the actual execution timeof the previous execution. If an execution is delayed forany reason, such as garbage collection or other back-ground activity, subsequent executions will be delayedas well. In fixed-rate execution, each execution is sched-uled relative to the scheduled execution time of the ini-tial execution. If an execution is delayed, two or moreexecutions will occur in rapid succession to ‘‘catch up’’(Joy et al., 2000). In Scheduler we chose fixed-delay exe-

cution because this method is appropriate for activitiesthat require ‘‘smoothness’’ keeping the frequency accu-rate in the short run than in the long run. If a task takesexcessive time to complete, it ‘‘hogs’’ the timer�s taskexecution thread. Therefore, in Scheduler, tasks were de-signed to complete quickly as it will be shown in subse-quent paragraph.

Scheduled tasks can be canceled at any time, usingthe reference ID assigned to them upon their scheduling.Canceling a task results to its removal from both theTimer and the database.

3.3.2. The persistence subsystem

The Persistence Subsystem, comprised of the entitybeans TaskBean and SequenceBean (see Fig. 1), isresponsible for storing data pertaining to time-triggeredtasks in order to maximize the reliability of the schedul-ing service. It consists of two main components. TheTaskBean is the interface of the relational database usedfor storing the task details. It is implemented as a CMP2.0 entity bean. CMPs can be developed much more rap-idly because the EJB container performs storage opera-tions, handles all data access logic and generates theJDBC code. There is no need to implement any persis-tence logic (such as JDBC or SQL/J). This simplifiesdevelopment and enhances scalability offering anabstraction layer over JDBC (Eckel, 2000). TheSequenceBean, which is responsible for generating un-ique identifiers (IDs) for the scheduled tasks, is also con-sidered as part of this subsystem. The generated IDs areused as primary keys for the records stored in the data-base. Stored data includes the start and stop time as wellas the execution period for each task and is used duringthe initialization of the Scheduler.

3.3.3. The task execution subsystem

The basic component of this subsystem is the mes-sage-driven bean TaskHandler, which is bound to a mes-sage queue (see Fig. 1). When a Task ‘‘runs’’, it acts likea JMS client that sends a message to the Queue. Thismessage contains information pertaining to the service


execution. The SchedulerBean has also the same role(i.e., JMS client) in the case of event-triggered tasks.This asynchronous implementation has significant im-pact on performance because tasks complete quicklywithout delay. Specifically, the actual operation per-formed by each scheduled task is degraded to the simpleinsertion of a message in the JMS queue. This is consid-ered as a very lightweight operation, compared to whata service might have involved. Execution of the servicetakes place immediately after the message is consumedfrom the queue and is completely decoupled from theScheduling Subsystem. Thus it does not affect the sched-uler�s performance.

4. Performance evaluation

The performance evaluation of the scheduler compo-nent was performed in a PC powered by an Athlon1.3 MHz CPU. The operating system was Windows2000 Professional, configured to execute a minimumset of background services. JBoss 3.0.4 was the J2EE-compliant application server which was used for boththe development and the performance evaluation ofthe software component. Its embedded HypersonicSQL database was used for persisting tasks. All subsys-tems were deployed on a single application server withno special performance tuning applied.

For the presented analysis, different test cases weredeveloped and used. In most of these tests the time inter-vals between the scheduling of two successive tasks fol-low the exponential distribution (thus, simulating arandom arrival model). The exact testing configurations,as well as the metrics used, are described in the followingsections.

4.1. Metrics

The performance metrics of a scheduler can be dividedin system metrics and user metrics (Bapat, 2001; Jain,1991; Feitelson and Rudolph, 1998). The first categoryincludes throughput, utilization (the fraction of time thatthe system remains busy), makespan and efficacy. Thetwo latter are used mainly in distributed computing envi-ronments hence they were not considered in our case.Utilization was also excluded as it tries to capture howefficiently the system resources are being utilized and isbest suited for low-level schedulers (e.g., CPU schedul-ers). However throughput was measured as it gives a feel-ing of the Scheduler�s capabilities. The second category,which is of prime importance for an application sched-uler, includes turnaround time and average delay. Thesemetrics focus on the measurement of the performancefrom the end user�s perspective. The inclusion of theabove metrics in a test suite, can give a quite clear viewof the scheduler�s capabilities and limitations.

The most important metric in a real-time scheduler isthe actual time that a task execution has delayed. Thedelay metric is defined as the elapsed time between thetrigger of a task and the start of its execution. In ourcase, delay includes both the delay of the timer andthe time a task has been waiting in a queue. One goodthing about delay is that it does not have to deal withthe nature of the jobs (unless they are classified into pri-ority classes). Hence, a simple arithmetic mean is en-ough to capture the implications of average delay(provided that outliers have been excluded from theset of samples). This time, as already stated, should beless than a second for the vast majority of the tasks.Hence, the appropriate time measurement unit is themillisecond. In order to measure the maximum, mini-mum and average task delays and indicate their correla-tion with the scalability of the scheduler, several testswere performed.

The throughput, which is defined as the number ofjobs that are completed per unit time, has also beenmeasured. This metric, is not very important for endusers, but is strongly related with the scalability of thescheduler.

Turnaround time is the time from the moment the usersubmits the job to the system till the system finishes itsexecution. From the user�s perspective, turnaround timeis a crucial metric, which must be minimized. Turn-around time has two components, waiting time and exe-cution time (or run time). The waiting time is the timethat the job spends in a queue waiting to be assignedto a server while the run time is the time the host takesto complete executing the task. In the context of thepresent test suite, we measure the turnaround times forevent-triggered tasks that are submitted initially to aheavily loaded scheduler and subsequently to an idleone. Finally we introduce a new metric, named Normal-

ized Delay. This metric is defined as follows:

Normalized Delay ¼ delay per task group

group periodð1Þ

where task group stands for a set of tasks with the sameexecution period (called group period). The delay in thenumerator can be either an average or maximum value.Normalized Delay gives a qualitative view of the delaysimposed by the Scheduler.

4.2. Test descriptions and results

Before performing the actual tests, test cases wereexamined in order to determine the behavior of theScheduler under heavy load conditions. The results ofthese stress tests allowed the determination of the limitsfor the performance of our Scheduler. During the initialstress test 300 tasks were scheduled for simultaneousexecution (at the same millisecond). The delays observedwere between 0 and 30 ms and the average delay was


approximately 19 ms. This time value is an approxima-tion of the minimum interval required between the exe-cutions of two successive tasks so as to keep delay in lowlevels. It also gives a rough approximation of the maxi-mum throughput the Scheduler can handle before delaystarts to increase exponentially, which was found to bein the order of 50 tasks/s (1000 ms divided by a 19 mstask interval). Indeed, when 50 tasks were scheduled,each one of them having an 20 ms interval from the pre-vious one, the average delay was 17 ms. Subsequent sim-ilar experiments showed that the delay is a linearfunction of the number of tasks scheduled.

4.2.1. Test 1: delay and normalized delay (ND)This test demonstrates the delays imposed on the

tasks by the scheduler. Apparently, hourly or daily peri-odic tasks, even in a large number, would not induce anoverload problem on the scheduler. We are interested inmore frequent executing activity. In this respect, the taskperiods were randomly selected from the set[2, 3,5,10,30] seconds. It should be noted that, such peri-ods are not representative for real-world tasks, but areconsidered appropriate for the desired load tests.

Fig. 4. Average and maximum delay.

Table 3Normalized delays (average/max) for each period group

Tasks ND2 ND3

100 0.02/0.57 0.01/0.25200 0.06/0.69 0.03/0.43300 0.04/0.39 0.04/0.30400 0.05/0.76 0.04/0.27500 0.09/0.54 0.05/0.34600 0.17/1.14 0.07/0.34700 0.13/0.98 0.06/0.33800 – –900 – –1000 – –

As tasks are assumed to commence execution at ran-dom times (thus simulating random arrivals) the intervaltime X between the scheduling of two successive taskswas modeled as exponentially distributed variable withmean arrival rate k = 5 tasks/s. The formula used is:

X ¼ �1

klnðuÞ; k : arrival rate; u : a uniformly

distributed random number in the ½0;1� interval ð2Þ

Provision has also been taken so as not to exceed thelimit of 50 task executions/s. Thus, the number of tasksin each period group was adjusted accordingly. Fig. 4depicts the average and maximum delay observed. Theminimum delay was kept stable at 10 ms, which reflectsthe delay for inserting and consuming a message fromthe queue. Table 3 shows the normalized delays. NDxdenotes the normalized delay of the tasks with periodx seconds. These values were calculated from samplestaken during the last minute of test execution, whenall the tasks were active. The average throughput duringthis minute was 2500 tasks/min (or 41.67 tasks/s). A firstobservation from this test is that smaller values of thegroup period lead to more rapid congestion, which, ofcourse, is normal. However for period values in the or-der of 30 s, which again can be considered excessive forapplication-level scheduling, the average ND remainssteadily very low (<0.02), while its maximum value doesnot exceed 0.1. Taking into consideration that typicalperiods for scheduling application-level services are nor-mally much bigger, we can safely conclude that theScheduler does not impose bottleneck to a service provi-sioning system.

4.2.2. Test 2: delay with variable offered load

In this test, we exceeded the 50 tasks/s limit and ob-serve how performance varies. The offered load (tasks/s) is increased on steps of 5 tasks/s and the periods(the same as in Test 1) are evenly distributed amongthem.

As shown in Fig. 5, maximum delay increases sub-stantially as throughput increases. Moreover, Fig. 6shows that the delay is a rough approximation of a lin-

ND5 ND10 ND30

0.01/0.17 0.00/0.13 0.00/0.040.03/0.31 0.02/0.15 0.01/0.060.02/0.14 0.01/0.10 0.01/0.050.02/0.28 0.02/0.22 0.01/0.080.03/0.40 0.02/0.25 0.01/0.060.03/0.26 0.02/0.19 0.01/0.050.06/0.58 0.03/0.29 0.02/0.080.16/0.71 0.04/0.34 0.02/0.060.09/0.43 0.03/0.16 0.02/0.070.03/0.12 0.02/0.22 0.01/0.09

Fig. 5. Maximum delay.

Fig. 6. The delay for 90 tasks/s.

Fig. 7. The delay for 50 tasks/s.

Table 4Turnaround times

Averageturnaroundtime (ms)

Maximumturnaroundtime (ms)

Tasks withexecutiontimes >20 ms (%)

Idle Scheduler 1.6 1.6 <1Loaded Scheduler 3–50 800–2640 �10

Table 5Periods of tasks for Test 4

Percentage of scheduled tasks Period (min)

1 0.54 15 220 570 10


ear function of time (or task executions). The solid lineshows the first execution of the last scheduled task,whereas the dashed line shows the time this task shouldnormally have been executed. If the offered load is high-er than 50 tasks/s, the delay systematically increases. Fi-nally, Fig. 7 shows the delay values and behavior for a‘‘normal’’ throughput of 50 tasks/s. In this figure, thefirst execution of the last scheduled task and the timeit should normally have been executed are almost iden-tical (dashed line), as there is only a 23 ms delay betweenthem.

4.2.3. Test 3: turnaround time of event-triggered tasks

This test is quite similar to Test 1. The only differenceis that during its last minute around 360 event-triggeredtasks were scheduled at random intervals. The rationalebehind this test case was to measure the turnaroundtimes of these event-triggered tasks. The measured aver-age and maximum turnaround times are depicted in

Table 4, as well as the percentage of tasks with executiontimes greater than 20 ms. Finally these measurementsare compared with those measured on an idle scheduler.

4.2.4. Test 4: heavy load conditions

This test case aims to demonstrate the scheduler�sbehavior under more realistic conditions regarding theservice provisioning context, which is after all the tar-geted environment for the Scheduler. The periods shownin Table 5 were used.

The number of scheduled tasks was also increasedsignificantly (10,000, 15,000 and 20,000 tasks) and thescheduling rate was raised to 10 tasks/s. To comparethe results of the experiments both the delay and thenumber of scheduled tasks were monitored, 40 min afterthe initial scheduling.

For 10,000 tasks, the results were quite satisfactory:the average delay was below 200 ms, and all the taskshave been scheduled 15 min after the start. For 15,000

Fig. 9. Delay of a standalone Timer.


tasks, the delay was very short (below 300 ms) until thenumber of scheduled tasks reached 13,800. After thatpoint the delay increased rapidly, and 40 min after thetest start, delay was about 8 min with only 14,600 outof the 15,000 tasks having been scheduled. Finally, for20,000 tasks the delay started to increase uncontrollablywhen the number of tasks reached 12,000 and after40 min of test execution it reached the 10 min, whileonly 12,350 tasks had been scheduled.

The above results are quite promising, as the majorityof the tasks in real-world conditions should have evenlarger periods.

4.2.5. Final commentsThe final experiment was a variation of Test 3. Its

only difference was that instead of submitting event-trig-gered tasks to the scheduler, we scheduled the samenumber of tasks. Our intention was to estimate the im-pact that the scheduling procedure, with the databaseinsertion overhead, would have on the delay times.The results showed that the imposed overhead did notlead to significant performance degradation.

In general, what became obvious from this study, isthat the main factor that affects the performance is notthe number of scheduled tasks but the load that was of-fered by the scheduler�s clients. Another observation isthat the task delays, in tests with low offered load, forma saw-like pattern (see Fig. 8).

Fig. 9 proves that the Timer is responsible for thisbehavior, as it shows that the delay of a single Timer,that executes dummy tasks, follows the same pattern.From this and other similar experiments, we can safelyconclude that the Timer imposes a maximum 10 ms de-lay, regardless of the number of scheduled tasks.

Moreover, in Fig. 8 only a small percentage of thetasks have delay higher than the average. This percent-age is manifested in the plot through the shown peaks

Fig. 8. Delays of 1000 non-overlapping tasks.

and is, mainly, due to operating system services, whichrun on the background. The times shown in this figurehave been collected after the execution of 1000 non-overlapping tasks (the tasks were triggered sequentially).

The aforementioned results prove that only a mini-mal performance overhead is imposed by the Scheduler.In Section 6 some possible future improvements foroptimizing the operation of the Scheduler are discussed.

5. Scheduler application in mobile service provision

The presented Scheduler architecture has been devel-oped as an integrated component of PoLoS (Ioannidis etal., 2003), a generic location based services (LBS) plat-form.2 The main objectives of PoLoS are the develop-ment, deployment and provision of LBS. It can bedeployed over any J2EE compliant server, thus beingan inherently open solution. Aiming to provide an inte-grated solution for delivering LBSs, PoLoS featuresmultiple interfaces with positioning systems, GIS (Geo-graphic Information System) servers or other networkelements. It supports all major transport protocols fromthe wireless domain (e.g., HTTP, WAP or SMS), thusallowing access from a wide range of end terminals, suchas mobile phones, smart phones, personal digital assis-tants (PDAs) and notebooks. PoLoS aims to be inde-pendent from the underlying technologies and asdiverse as possible and towards this direction the posi-tioning interface covers a wide spectrum of existing posi-tioning techniques, from 2G/3G and WLANenvironments, while GPS location is also supported

2 The work presented in this paper has been performed in theframework of the project IST-2001-35283 ‘‘PoLoS’’, which is partlyfunded by the European Commission and the Swiss BBW (Bundesamtfur Bildung und Wissenschaft).

Fig. 10. PoLoS overall architecture.


(Prigouris et al., 2003). Moreover interfaces with GISsystems, SMS gateways and other external entities havebeen developed using the Web Services model and stan-dardized frameworks such as 3GPP�s OSA (Open Ser-vice Access) (OSA, 2002). For assisting the servicecreation process PoLoS offers an Integrated Develop-ment Environment (IDE), which supports the processof writing new services and deploying them inside theprovisioning platform. Services are created through anXML-based Service Creation Language (SCL), whichwas defined specifically for this reason (Ioannidiset al., 2004).

The heart of the PoLoS architecture is its Kernel(Spanoudakis et al., 2003), which is responsible for coor-dinating the collaboration between the peripheral enti-ties (e.g., GIS server, positioning systems, billingsystems, interfaces components) and its internal compo-nents. Moreover the kernel handles issues like servicedeployment, service management and service removalby maintaining interfaces towards the appropriate func-tional entities. A bird�s eye view of the PoLoS frame-work is provided in Fig. 10. The Scheduler is a coremodule of the Kernel, which supports the Push para-digm (Chevest et al., 2002). The scheduler acts as a dum-my client, on behalf of the user, which, triggers theexecution of services, collects the results and dispatchesthem to the designated end user via the available WAPor SMS interface. Registrations to the Scheduler are per-formed through a convenient management interface,which allows users to register services for future execu-tion according to the different invocation patterns sup-ported by the scheduling mechanism.

6. Scalability issues

Every enterprise system should be scalable enough inorder to support large numbers of concurrent users. Theproposed scheduler is no exception to this rule, since itsmain target application domain is service provisioning.

From the discussed experiments we observed that forvery high loads (see Test 4 in Section 4.2), the applica-tion server resources (i.e., computing power, memory)were saturated. According to the Java documentationthe Timer class, has high capacity which does not de-grade the performance of the system (‘‘. . .this class scalesto large numbers of concurrently scheduled tasks; thou-

sands should present no problem. . .’’). The EJB compo-nents, on the other hand, although ‘‘intelligently’’managed by the EJB container, certainly, impose scala-bility limitations.

The most common way to address scalability isthrough load-balancing techniques. Load-balancing re-fers to the distribution of workload to multiple applica-tion instances executing in different physical systems.Clustering is probably the most well-known load-bal-ancing technique used by modern application servers.Unfortunately, scalability design issues are not, yet, cov-ered by the J2EE specification. Thus, in order to make aclustered version of the Scheduler we have to considersome additional design aspects. The main challengeregarding clustered (time-triggered) applications is theexistence of the Timer component itself.

Two different scenarios can be considered for han-dling potential scalability problems that may arise:The first scenario, which is depicted in Fig. 11, involvesthe deployment of the Timer outside the cluster of theEJB containers. The Timer is a singleton object, com-mon for all the members of the cluster. Each containerwithin the cluster (hereinafter referred to as clusternode) hosts the SchedulerBean, the entity beans (i.e.,TaskBeans) and the TaskHandler along with the corre-sponding Queue (i.e., one per cluster node). Placementof the Timer outside the cluster is justified on the basisof its increased scalability and the fact that task registra-tions are not demanding and resource consuming oper-ations. In this scenario, the SchedulerBean, has to beenhanced with additional methods, since apart fromtask registration, it handles the queue dispatching pro-cess, inside its local cluster node.

Fig. 12. Clustering with local timers.

Fig. 11. Clustering with a global timer.


In the second scenario, which is illustrated in Fig. 12,all components of the Scheduler�s architecture, includingthe Timer, are clustered. This means that each clusternode has a local Timer, which is used for triggering tasks.An important issue, with this approach is that assign-ment of tasks to cluster nodes is handled by the load-bal-ancing engine of the application server, without takinginto account any information about the scheduled tasks.Since, such engines usually adopt simplistic mechanisms

for distributing the experienced load (e.g., a round-robinscheme), it is possible that certain cluster nodes becomeoverloaded, while others experience lighter load. Forexample, imagine the extreme case where we have a clus-ter of two nodes and the scheduling requests arrive insuch a manner, that for each two arriving requests thefirst one is resource-demanding (i.e., requesting fre-quent execution for an extensive time period), while thesecond is significantly lighter (i.e., requesting infrequent


execution or execution for a much shorter period). Aftera while, the first cluster node will reach saturation, whilethe second node will have very low utilization. In orderto avoid, such unbalanced operation, each Timer insidethe proposed cluster, does not perform direct task execu-tion, through its local queue, but rather directs each trig-gered task back to the external load-balancing engine.Hence, the task execution process is decoupled from taskscheduling and is redistributed between available con-tainers, thus, increasing the fairness and efficiency ofthe system. As in the previous scenario, the Secheduler-Bean should be enhanced with functionality responsiblefor the queue dispatching process.

The discussion in this section highlights the impor-tant issue of clustering a scheduling service. The J2EEcommunity has recognized that this topic is of highinterest for enterprise applications and has included acontainer-managed Timer Service in its EJB 2.1 specifi-cation. Such service is persistent and has been adoptedby several vendors. However, it is more coarse-grainedthan the default Java Timer and not suited for real-timeevents.

7. Further work

Results of performance analysis showed that the mes-sage queue, to which the TaskHandler is bound, consti-tutes a bottleneck. When a large amount of tasks arrivesin the queue a lot of time elapses before these tasks areconsumed by TaskHandler. Multiple message queuescan be used to eliminate this undesirable delay. Tasksfor execution will be distributed among these queuesand as a result the wait time will be effectively reduced.But this is useless when all queues are maintained by thesame server (i.e., in a single machine with only oneCPU). A clustered environment where queues, Task-

Handlers (and possibly Timers), reside in different serv-ers (and machines) would be ideal for such animplementation. Installing and testing the Scheduler insuch a multi-CPU distributed environment is in ourimmediate future plans.

Improvements can be done also in the section of per-sistence. We can use the Java Data Objects (JDO) APIthat is a standard interface-based Java model abstrac-tion of persistence. JDO (Tyagi et al., 2003) is used todirectly store Java domain model instances in the data-base. CMP beans require active transactions for all busi-ness methods. Non-transactional access is not standardor portable. JDO allows choosing whether transactionsare needed. JDO requires insertions, removals and up-dates to be performed within transactions, but read-onlyapplications, such as caching, can also be implementedwithout transactions. The most important benefits whenusing JDO however include ease of use, high perfor-mance, and integration with EJBs.

8. Conclusions

We have presented a working scheduling model,which supports the scheduled execution of services orother tasks and is targeted to the enterprise domain.Automated and scheduled task execution is an essentialingredient of many modern enterprise systems, as itspresence can significantly augment their applicationrange. In the mobile market, where new types of servicesare continuously emerging everyday, such systems canprovide a real boost to their absorption and use by awide portion of end users. Within the LBS context, theability to schedule the delivery of service content, whilethe user is moving between locations, either on a timebasis or when certain events occur (e.g., entering anarcheological site or a commercial center) can lead tothe development of next-generation value added ser-vices. A fine-grained architecture of a Scheduler was pre-sented along with the requirements that guided usthroughout its design. Implementation aspects were cov-ered in detail while, an extensive performance evaluationwas performed in order to demonstrate the scalability,reliability and efficiency of the architecture, thus, assur-ing its applicability for enterprise-scale systems. The factthat its design and implementation was based on soundtechniques and state-of-the-art open technologies makesit an ideal solution for integration with existing serviceprovisioning platforms running over heterogeneousinfrastructures.

References

Bapat, S., 2001. Investigating Metrics Used in Evaluating Schedulers.Ohio State University.

Bhattacharya, B., 2003. Pulsar Scheduler J2EE. Available from:<j2eescheduler.sourceforge.net>.

Cattell, R. et al., 2000. Java 2 Platform, Enterprise Edition: platformand component specifications. Addison-Wesley Pub Co, USA.

Chevest, K., Mitchell, K., Davies, N., 2002. Exploring context-awareinformation push. Personal and Ubiquitous Computing 6 (4), 276–281.

Dale, G., 2003. Kronova eScheduler Concepts Guide. Indus Consul-tancy Services, Inc.

Eckel, B., 2000. Thinking in Java, second ed. Prentice-Hall, USA.Feitelson, D., Rudolph, L., 1998. Metrics and benchmarking for

parallel job scheduling. In: Job Scheduling Strategies for ParallelProcessing. Lecture Notes in Computer Science, vol. 1459.Springer-Verlag, pp. 1–24.

Franklin, M., Zdonik, S., 1998. Data in your face: push technology inperspective. ACM SIGMOD International Conference on Man-agement of Data, Seattle, Washington, USA, pp. 516–519.

Goodwill, J., 2001. Apache Jakarta-Tomcat. Apress Publishing,Berkeley.

Gulcu, C., 2003. The complete log4j manual. QOS.ch, Lausanne,Switzerland.

Ioannidis, A., Priggouris, I., Marias, I., Hadjiefthymiades, S., Faist-Kassapoglou, C., Hernandez, J., Merakos, L., 2003. PoLoS:integrated platform for location-based services. In: Proceedingsof the IST Mobile and Wireless Communications Summit,Portugal.

http://j2eescheduler.sourceforge.net


Ioannidis, A., Spanoudakis, M., Sianas, P., Priggouris, I., Hadjieft-hymiades, S., Merakos, L., 2004. Using XML and relatedstandards to support Location Based Services. In: Proceedings ofSAC�2004, 19th ACM Symposium on Applied Computing, WebTechnologies and Applications Special Track, Nicosia, Cyprus.

Jain, R., 1991. The Art of Computer Systems Performance Analysis:Techniques for Experimental Design, Measurement, Simulation,and Modeling. Wiley- Interscience, New York, USA.

Joy, B. et al., 2000. Java(TM) Language Specification, second ed.Addison-Wesley Pub Co, USA.

OSA (Open Service Access), 2002. Application Programming Interface(API) Part 1: Overview. Final draft ETSI ES 202 915-1 V1.1.1.

Prigouris, N., Papazafeiropoulos, G., Marias, I., Hadjiefthymiades, S.,Merakos, L., 2003. Building a generic broker for location retrieval.In: Proceedings of the IST Mobile and Wireless CommunicationsSummit, Portugal.

Roman, E. et al. (Eds.), 2002. Mastering Enterprise JavaBeans,second ed. Wiley Computer Publishing, USA.

Sims, D., 2002. Job Scheduling in J2EE Applications. Sims Comput-ing, Inc., USA.

Spanoudakis, M., Batistakis, A., Priggouris, I., Ioannidis, A., Had-jiefthymiades, S., Merakos, L., 2003. Extensible platform forlocation based services provisioning. In: Proceedings of the 3rdInternational Workshop on Web and Wireless GeographicalInformation Systems (W2GIS 2003), Rome, Italy.

Stark, S. et al., 2002. Jboss Administration and Development, seconded. Jboss Group LLC, Atlanta, USA.

Tyagi, S. et al., 2003. Core Java Data Objects. Prentice-Hall, USA.

Vassileios Tsetsos received his B.Sc. in Informatics from the Depart-ment of Informatics and Telecommunications at the University ofAthens, Greece, in 2003. He is currently a postgraduate student at thesame Department (Communication Systems and Data Networksdivision). He is involved as software developer in the PoLoS (Inte-grated Platform for Location Based Services) project implementedin the context of IST. His research interests are in the areas of mobileand pervasive computing, distributed computing, and web applica-tions.

Odysseas Sekkas received his B.Sc. in Informatics from the Depart-ment of Informatics and Telecommunications at the University ofAthens, Greece, in 2003. He was accepted in the M.Sc. program inCommunication Systems and Data Networks from the same Depart-ment. Currently, he is in the first year of his M.Sc. studies. He isinvolved as programmer in the development of the PoLoS (IntegratedPlatform for Location Based Services) project implemented in the

context of IST. His research interests include wireless and mobilecomputing, QoS and mobility support for IP networks, and webengineering.

Ioannis Priggouris received his B.Sc. in Informatics from the Depart-ment of Informatics and Telecommunications at the University ofAthens, Athens—Greece in 1997 and his M.Sc. in CommunicationSystems and Data Networks from the same Department in 2000. Overthe last two years he has been a Ph.D. candidate in the department.Since 1999, he has been a member of the Communication NetworksLaboratory (CNL) of the University of Athens. He has participated inthe RAINBOW (Radio Access INdependent BrOadband on Wireless)and the EUROCITI (EUROpean CITIes platform for on-line trans-action services) projects implemented in the context of ACTS and ISTcorrespondingly, as well as in several other national and Europeanprojects. Currently, he is extensively involved in the development of thePoLoS (Integrated Platform for Location Based Services) projectimplemented in the context of IST. His research interests are in theareas of mobile computing, QoS and mobility support for IP networksand location-sensitive resource management. He is the author of manypublications in the above areas.

Stathes Hadjiefthymiades received his B.Sc. in Informatics from theDepartment of Informatics at the University of Athens, Greece, in1993 and his M.Sc. in Advanced Information Systems from the samedepartment in 1996. In 1999 he received his Ph.D. from the Universityof Athens (Department of Informatics and Telecommunications). In2002 he received a joint engineering-economics M.Sc. degree from theNational Technical University of Athens. In 1992 he joined the Greekconsulting firm Advanced Services Group, Ltd., where he was exten-sively involved in the analysis and specification of information systemsand the design-implementation of telematic applications. In 1995 hebecame a member of the Communication Networks Laboratory of theUniversity of Athens. During the period September 2001–July 2002, heserved as a visiting assistant professor at the University of Aegean,Department of Information and Communication Systems Engineering.In July 2002 he joined the faculty of the Hellenic Open University(Department of Informatics), Patras, Greece, as an assistant professorin telecommunications and computer networks. Since December 2003,he belongs to the faculty of the University of Athens, Department ofInformatics and Telecommunications. He has participated in numer-ous projects realized in the context of EU programs (ACTS, ORA,TAP, and IST), EURESCOM projects, as well as national initiatives.His research interests are in the areas of wireless/mobile computing,web engineering, and networked multimedia applications. He is theauthor of over 70 publications in these areas.

a scheduling framework for enterprise servicescgi.di.uoa.gr/~sekkas/papers/jss_scheduler.pdf · as...

Documents