optimization of resource provisioning

7/29/2019 Optimization of Resource Provisioning

1/14

Optimization of Resource ProvisioningCost in Cloud Computing

Sivadon Chaisiri, Student Member, IEEE,

Bu-Sung Lee, Member, IEEE, and Dusit Niyato, Member, IEEE

AbstractIn cloud computing, cloud providers can offer cloud consumers two provisioning plans for computing resources, namely

reservation and on-demand plans. In general, cost of utilizing computing resources provisioned by reservation plan is cheaper than

that provisioned by on-demand plan, since cloud consumer has to pay to provider in advance. With the reservation plan, the consumer

can reduce the total resource provisioning cost. However, the best advance reservation of resources is difficult to be achieved due to

uncertainty of consumers future demand and providers resource prices. To address this problem, an optimal cloud resource

provisioning (OCRP) algorithm is proposed by formulating a stochastic programming model. The OCRP algorithm can provision

computing resources for being used in multiple provisioning stages as well as a long-term plan, e.g., four stages in a quarter plan and

twelve stages in a yearly plan. The demand and price uncertainty is considered in OCRP. In this paper, different approaches to obtain

the solution of the OCRP algorithm are considered including deterministic equivalent formulation, sample-average approximation, and

Benders decomposition. Numerical studies are extensively performed in which the results clearly show that with the OCRP algorithm,

cloud consumer can successfully minimize total cost of resource provisioning in cloud computing environments.

Index TermsCloud computing, resource provisioning, virtualization, virtual machine placement, stochastic programming.

1 INTRODUCTION

CLOUD computing is a large-scale distributed computingparadigm in which a pool of computing resources isavailable to users (called cloud consumers) via the Internet[1]. Computing resources, e.g., processing power, storage,software, and network bandwidth, are represented to cloudconsumers as the accessible public utility services. Infra-

structure-as-a-Service (IaaS) is a computational servicemodel widely applied in the cloud computing paradigm.In this model, virtualization technologies can be used toprovide resources to cloud consumers. The consumers canspecify the required software stack, e.g., operating systemsand applications; then package them all together intovirtual machines (VMs). The hardware requirement ofVMs can also be adjusted by the consumers. Finally, thoseVMs will be outsourced to host in computing environmentsoperated by third-party sites owned by cloud providers. Acloud provider is responsible for guaranteeing the Qualityof Services (QoS) for running the VMs. Since the computing

resources are maintained by the provider, the total cost ofownership to the consumers can be reduced.

In cloud computing, a resource provisioning mechanismis required to supply cloud consumers a set of computingresources for processing the jobs and storing the data. Cloudproviders can offer cloud consumers two resource provi-sioning plans, namely short-term on-demand and long-termreservation plans. Amazon EC2 [2] and GoGrid [3] are, for

instances, cloud providers which offer IaaS services withboth plans. In general, pricing in on-demand plan ischarged by pay-per-use basis (e.g., 1 day). Therefore,purchasing this on-demand plan, the consumers candynamically provision resources at the moment when theresources are needed to fit the fluctuated and unpredictable

demands. For reservation plan, pricing is charged by a one-time fee (e.g., 1 year) typically before the computingresource will be utilized by cloud consumer. With thereservation plan, the price to utilize resources is cheaperthan that of the on-demand plan. In this way, the consumercan reduce the cost of computing resource provisioning byusing the reservation plan. For example, the reservationplan offered by Amazon EC2 can reduce the totalprovisioning cost up to 49 percent when the reservedresource is fully utilized (i.e, steady-state usage) [4].

With the reservation plan, the cloud consumers a priorireserve the resources in advance. As a result, the under-provisioning problem can occur when the reserved resourcesare unable to fully meet the demand due to its uncertainty.Although this problem can be solved by provisioning moreresources with on-demand plan to fit the extra demand, thehigh cost will be incurred due to more expensive price ofresource provisioning with on-demand plan. On the otherhand, the overprovisioning problem can occur if the reservedresources are more than the actual demand in which part ofa resource pool will be underutilized. It is important for thecloud consumer to minimize the total cost of resourceprovisioning by reducing the on-demand cost and over-subscribed cost of underprovisioning and overprovisioning.To achieve this goal, the optimal computing resource

management is the critical issue.In this paper, minimizing both underprovisioning and

overprovisioning problems under the demand and priceuncertainty in cloud computing environments is our

164 IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 5, NO. 2, APRIL-JUNE 2012

. The authors are with the School of Computer Engineering, NanyangTechnological University (NTU), Nanyang Avenue, Singapore 639798.E-mail: {siva0020, ebslee, dniyato}@ntu.edu.sg.

Manuscript received 3 Apr. 2010; revised 6 Sept. 2010; accepted 28 Jan. 2011;published online 7 Feb. 2011.For information on obtaining reprints of this article, please send e-mail to:[email protected] and reference IEEECS Log Number TSCSI-2010-04-0030.Digital Object Identifier no. 10.1109/TSC.2011.7.

1939-1374/12/$31.00 2012 IEEE Published by the IEEE Computer Society


2/14

motivation to explore a resource provisioning strategy forcloud consumers. In particular, an optimal cloud resourceprovisioning (OCRP) algorithm is proposed to minimize thetotal cost for provisioning resources in a certain time period.To make an optimal decision, the demand uncertainty fromcloud consumer side and price uncertainty from cloudproviders are taken into account to adjust the tradeoffbetween on-demand and oversubscribed costs. This optimaldecision is obtained by formulating and solving a stochasticinteger programming problem with multistage recourse [5].Benders decomposition [6] and sample-average approxima-tion [7] are also discussed as the possible techniques tosolve the OCRP algorithm. Extensive numerical studies andsimulations are performed, and the results show that OCRPcan minimize the total cost under uncertainty.

The major contributions of this paper lie in themathematical analysis which can be summarized as follows:

. The optimal cloud resource provisioning algorithmis proposed for the virtual machine management.

The optimization formulation of stochastic integerprogramming is proposed to obtain the decisionof the OCRP algorithm as such the total cost ofresource provisioning in cloud computing environ-ments is minimized. The formulation considersmultiple provisioning stages with demand and priceuncertainties.

. The solution methods based on Benders decomposi-tion and sample-average approximation algorithmsare used to solve the optimization formulation inan efficient way.

. The performance evaluation is performed which can

reveal the importance of optimal computing re-source provisioning. The performance comparisonamong the OCRP algorithm and the other ap-proaches is also presented.

The proposed mathematical analysis will be useful to thecloud consumers (e.g., organization and company) for themanagement of virtual machines in cloud computingenvironment. The proposed OCRP algorithm will facilitatethe adoption of cloud computing of the users as it canreduce the cost of using computing resource significantly.

The rest of this paper is organized as follows: Relatedworks are reviewed in Section 2. The system model andassumption of cloud computing environment are describedin Section 3. In Section 4, the stochastic programmingformulation of the OCRP algorithm is presented. Section 5presents the Benders decomposition algorithm. Section 6presents the sampling-average approximation approach.Experiments and simulations to evaluate the performanceof the OCRP algorithm are presented in Section 7. Finally,conclusions are stated in Section 8.

2 RELATED WORK

Available resource provisioning options were discussed in[8]. The resource provisioning strategies in distributed

systems were addressed in [9], [10], [11], [12], [13], [14], [15].In [9], an architectural design of on-demand service for gridcomputing was proposed. In [10], a profile-based approachto capture experts knowledge of scaling applications was

proposed in which extra demanded resources can be moreefficiently provisioned. In [11], the concept of resource slotwas proposed. The objective is to address uncertainty ofresources availability. In [12], a binary integer program tomaximize revenues and utilization of resource providerswas formulated. However, [9], [10], [11], [12] did notconsider uncertainty of future consumer demands. In [13],an optimization framework for resource provisioning was

developed. This framework considered multiple client QoSclasses under uncertainty of workloads (e.g., demands ofcomputing resources). The arrival pattern of workloads isestimated by using online forecasting techniques. In [14],heuristic method for service reservation was proposed.Prediction of demand was performed to define reservationprices. In [15], K-nearest-neighbors algorithm was appliedto predict the demand of resources. In contrast, our workspecifies that demands are given as probability distribu-tions. In addition, the price difference between reservationand on-demand plans was not taken into account in allworks in the literature.

As virtualization is a core technology of cloud computing,the problem of virtual machine placement (VM placement)becomes crucial [16], [17], [18], [19], [20]. In [16], the broker-based architecture and algorithm for assigning VMs tophysical servers were developed. In [17], a resource manage-ment consisting of resource provisioning and VM placementwas proposed. In [18], techniques of VM placement andconsolidation which leverage min-max and shares featuresprovided by hypervisors were explored. In [19], a dynamicconsolidation mechanism based on constraint programmingwas developed. This consolidation mechanism was origin-ally designed for homogeneous clusters. However, hetero-geneity which is common in a multiple cloud provider

environment was ignored. Moreover, [16], [17], [18], [19]did not consider uncertainty of future demands and prices.In [20], a dynamic VM placement was proposed. However,the placement in [20] is heuristic-based which cannotguarantee the optimal solution.

Stochastic programming has been developed to solveresource planning under uncertainty [5] in various fields,e.g., production planning, financial management, andcapacity planning. For example, in [21], the authors appliedthe stochastic programming approach for planning ofelectrical power generation and transmission line expansionwhile some uncertainties affecting to the planning are taken

into account. It is shown that stochastic programming is thepromising mathematical tool which is able to address theoptimal decision making in the stochastic environment.However, to the best of our knowledge, the application ofstochastic programming to computing resource provision-ing has never been exclusively studied.

The optimal virtual machine placement (OVMP) algorithmwas proposed [22]. This OVMP algorithm can yield theoptimal solution for both resource provisioning and VMplacement in two provisioning stages. Motivated by thisprevious work, we introduce the OCRP algorithm in thispaper which achieves many improvements. First, theproblem is generalized into the multiple stage formulation.

Second, the different approaches to obtain the solution ofcomputing resource provisioning are considered. Finally,the performance evaluation is extended to consider variousrealistic scenarios.

CHAISIRI ET AL.: OPTIMIZATION OF RESOURCE PROVISIONING COST IN CLOUD COMPUTING 165


3/14

3 SYSTEM MODEL AND ASSUMPTION

3.1 Cloud Computing Environment

As shown in Fig. 1, the system model of cloud computingenvironment consists of four main components, namelycloud consumer, virtual machine (VM) repository, cloudproviders, and cloud broker. The cloud consumer has

demand to execute jobs. Before the jobs are executed,computing resources has to be provisioned from cloudproviders. To obtain such resources, the consumer firstlycreates VMs integrated with software required by thejobs. The created VMs are stored in the VM repository.Then, the VMs can be hosted on cloud providersinfrastructures whose resources can be utilized by theVMs. In Fig. 1, the cloud broker is located in the cloudconsumers site and is responsible on behalf of the cloudconsumer for provision resources for hosting the VMs. Inaddition, the broker can allocate the VMs originally storedin the VM repository to appropriate cloud providers. The

broker implements the OCRP algorithm to make anoptimal decision of resource provisioning.In OCRP, there are multiple VM classes used to classify

different types of VM. Let I & IN11 denote the set of VMclasses. It is assumed that one VM class represents a distincttype of jobs (e.g., one class for web application and the otherfor database application). A certain amount of resources isrequired for running the VM, and this required amount ofresources can be different for VM in different classes. Withthis resource requirement, the cloud broker can reservecomputing resources from cloud providers to be used in thefuture according to the actual demand. This demand can bedetermined as the number of created VMs. In this case, it ispossible that additional resources can be provisionedinstantly from cloud providers if the reserved resources isnot enough to accommodate the actual demand.

Let J & IN1 denote the set of cloud providers. Each cloudprovider supplies a pool of resources to the consumer. LetR denote the set of resource types which can be providedby cloud providers. Resource types can be computingpower (in unit of CPU-hours), storage (in unit of GBs/month), and network bandwidth for Internet data transfer(in unit of GBs/month). Each VM class specifies the amountof resources in each resource type. Let bir be the amount ofresource type r required by the VM in class i

2 I. It is

assumed that every cloud provider prepares facilities, e.g.,virtualization management software, network facility, and

load balancer, to support the consumers hosting VM. Notethat the key notations used in the paper are listed in Table 1.

3.2 Provisioning Plans

A cloud provider can offer the consumer two provisioningplans, i.e., reservation and/or on-demand plans. Forplanning, the cloud broker considers the reservation planas medium- to long-term planning, since the plan has to be

subscribed in advance (e.g., 1 or 3 years [2]) and the plancan significantly reduce the total provisioning cost [4]. Incontrast, the broker considers the on-demand plan as short-term planning, since the on-demand plan can be purchasedanytime for short period of time (e.g., one week) when theresources reserved by the reservation-plan are insufficient(e.g., during peak load).

3.3 Provisioning Phases

The cloud broker considers both reservation and on-demand plans for provisioning resources. These resourcesare used in different time intervals, also called provisioningphases. There are three provisioning phases: reservation,

expending, and on-demand phases. As shown in Fig. 2, thesephases with their actions perform in different points of time(or events) as follows. First in the reservation phase, withoutknowing the consumers actual demand, the cloud brokerprovisions resources with reservation plan in advance. Inthe expending phase, the price and demand are realized,and the reserved resources can be utilized. As a result, thereserved resources could be observed to be either over-provisioned or underprovisioned. If the demand exceedsthe amount of reserved resources (i.e., underprovisioned),the broker can pay for additional resources with on-demandplan, and then the on-demand phase starts.

3.4 Provisioning StagesA provisioning stage is the time epoch when the cloudbroker makes a decision to provision resources by purchas-ing reservation and/or on-demand plans, and also allocatesVMs to cloud providers for utilizing the provisionedresources. Therefore, each provisioning stage can consistof one or more provisioning phases. The number ofprovisioning stages is based on the number of planningepoches considered by the cloud broker, e.g., a yearly planconsists of 12 provisioning stages (i.e., 12 months). Let T &IN1 denote the set of all provisioning stages where jT j ! 2.For resource provisioning under uncertainty, the broker is

assumed to be able to reserve the resources in the firstprovisioning stage. Also, the broker obtains a solution,called recourse action [5], for provisioning resources againstuncertainty parameters (i.e., demand and price) in everystage. These uncertainty parameters in each stage will beobserved by the broker after the resource reservation hasbeen made. The observed uncertainty parameters are calledrealization (e.g., the actual number of created VMs after thejobs are submitted by the consumers). Then, the broker willtake the recourse action according to the realization, e.g.,utilizing the reserved resource and/or provisioning moreresource with on-demand plan.

Fig. 3 shows the relationship between provisioning

phases and provisioning stages, where a yearly plan with12 provisioning stages, namely T fT1; T2; . . . ; T12g, isconsidered. Fig. 3a shows the example of all threeprovisioning phases existing in each stage. In Fig. 3b, each


Fig. 1. System model of cloud computing environment.

1. IN1 f1; 2; 3; . . .g.


4/14

stage may not consist of all provisioning phases. Forexample, the reservation phase is performed in T1 and thenumber of resources are reserved. From T1-T3, the expend-ing phase starts in some points of time when someresources reserved in T1 are utilized. Then, the on-demandphase starts in T3 due to the insufficient reserved resources,and some resources are provisioned with on-demand plan.In T4, the reservation phase starts again and so on.

3.5 Reservation Contracts

A cloud provider can offer the consumer multiple reserva-tion plans with different reservation contracts. Each

reservation contract refers to the advance reservation ofresources with the specific time duration of usage. Forexample, the reservation plan offered by Amazon EC2 hastwo reservation contracts [2], namely 1-year contract and 3-year contract. The certain amount of resources are reservedfor 1 yearin the one-yearcontractand 3 years inthe three-yearcontract starting from the time when they are provisioned.

Let K & IN1 denote the set of all reservation contractswhich are offered by cloud providers. Let Lk denote thetime duration (in unit of provisioning stages) specified inreservation contract k 2 K. Let Tk denote the set of stages atwhich the cloud broker can provision resources by contractk

. Let Fkt be the set of stages at which some resourcesreserved by contract k could be utilized at stage t 2 T.Given the total number of stages jT j, both Tk and Fkt areexpressed as follows:

Tk f1; . . . ; j T j Lk 1g; 1

Fkt fmax1; t Lk 1; . . . ; mint; j T j Lk 1g: 2In Fig. 4, the example of advance reservation for the

yearly plan with 3-month (K1) and 6-month (K2) reservation


Fig. 2. Transition of provisioning phases.

Fig. 3. Relationship between provisioning phases and provisioningstages.

TABLE 1List of Key Notations

Fig. 4. Example of advance reservations with 3-month (K1) and 6-month(K2) contracts.


5/14

contracts is shown (i.e., LK1 3 and LK2 6). The boxesabove the timeline represent the time coverage of somereserved contracts. As shown in (1), TK1 fT1; T2; . . . ; T10gand TK2 fT1; T2; . . . ; T7g are the sets of stages at whichresources can be provisioned by K1 and K2, respectively.Contract K1 is subscribed three times (e.g., T7-T9), whilecontract K2 is subscribed twice (e.g., T5-T10). Fig. 4 alsoshows that some stages can be covered by two (or more)

subscribed contracts, e.g., T1-T3 are covered by contracts K1and K2. In Fig. 4, any set Fkt in (2) can be obtained, e.g.,FK1T4 fT2; T3; T4g, FK1T11 fT9; T10g, and FK2T12 fT7g.3.6 Uncertainty of Parameters

The optimal solution used by the cloud broker is obtainedfrom the OCRP algorithm based on stochastic integerprogramming [5]. Stochastic programming takes a set ofuncertainty parameters (called scenarios), described by aprobability distribution into account. Let denote the set ofall scenarios in every provisioning stage and t denote theset of all scenarios in provisioning stage t. Set is definedas the Cartesian product of all t, namely

Yt2T

t 1 2 jT j: 3

It is assumed that the probability distribution of has finitesupport, i.e., set has a finite number of scenarios withrespective probabilities p! 2 0; 1 where ! is a compositevariable defined as ! !1; . . . ; !jT j 2 . In this paper,demand and price are considered as scenarios in whoseprobability distribution is assumed to be available. Theactual scenario of uncertainty parameter after it is observedby the broker is called realization.

3.7 Provisioning CostsWith three aforementioned provisioning phases, there arethree corresponding provisioning costs incurred in thesephases, namely reservation, expending, and on-demandcosts. The main objective of the OCRP algorithm is tominimize all of these costs while the consumers demand ismet, given the uncertainty of demand and price.

For cloud provider, the price is defined in dollars ($) perresource unit. Let c

Rjkr denote the unit price (i.e., costs to the

consumer) of resource type r subscribed to reservationcontract k provided by cloud provider j in reservationphase of the first provisioning stage. It is assumed that theprice of reservation plan in the first stage is charged by a

fixed one-time fee. The reservation cost cR

ijk is the cost forprovisioning every resource type defined as follows:

cRijk

Xr2R

bircR

jkr : 4

The prices in reservation and expending phases could beadjusted by cloud providers without informing the con-sumer in advance, except the price of the reservation plan inthe first provisioning stage. For instance, the cost of electricpower to supply a cloud providers data center could beincreased by power plants in the next few months, and thecloud provider will be able to increase the costs of

computing resources in the future as well. For the pricesin provisioning stage t given scenario ! in both reservationand expending phases, c

rjkrt! and cejkrt! denote the unit

prices of resource type r with reservation contract k

provided by cloud provider j, respectively. Let crijkt!

and ceijkt!, defined with the similar way as (4), denote thereservation and expending costs for provisioning everyresource type, respectively.

In on-demand phase, let co

jrt! denote the unit price ofresource type r provided by cloud provider j in provision-ing stage t given scenario !. Also, the price can be changedby cloud providers (i.e., uncertain to consumer when theresource is reserved). Let coijt !, defined with the similarway as (4), denote the on-demand cost for provisioningevery resource type. Given VM class i, cloud provider j,provisioning stage t, and scenario !, the expending cost ofany reservation contract k is assumed to be cheaper than theon-demand cost (i.e., c

eijkt! < coijt !).

From the system model and assumption, the optimalcloud resource provisioning algorithm is developed tominimize the total provisioning costs under the price anddemand uncertainty in multiple provisioning stages. In thenext section, the stochastic programming formulation of theOCRP algorithm is presented.

4 STOCHASTIC PROGRAMMING MODEL

In this section, the stochastic programming with multistagerecourse [5] is presented as the core formulation of theOCRP algorithm. First, the original form of stochasticinteger programming formulation is derived. Then, theformulation is transformed into the deterministic equivalentformulation (DEF) which can be solved by traditionaloptimization solver software.

4.1 Stochastic Integer Programming for OCRP

Minimize:

zXi2I

Xj2J

Xk2K

cRijk x

Rijk IE

QxRijk ; !; 5subject to:

xRijk 2 IN0; 8i 2 I; 8j 2 J; 8k 2 K: 6

The general form of stochastic integer program of theOCRP algorithm is formulated in (5) and (6). The objectivefunction (5) is to minimize the cloud consumers totalprovisioning cost. Decision variable xRijk denotes thenumber of VMs provisioned in the first provisioning stage.

In other words, this number refers to as the total amount ofreserved resources. The expected cost under the uncer-tainty is defined as IEQxRijk ; ! where QxRijk ; ! isexpressed as follows:

QxRijk ; ! minYxr

ijkt!;xe

ijkt!;xo

ijt!

CY; Y 2 xRijk ; !: 7In (7), the objective of QxRijk ; ! is to minimize the cost

under uncertainty given scenario ! and xRijk . Such cost is

represented by C and defined in (9). Composite variableY representing the solution of QxRijk ; ! consists ofvariables, namely x

rijkt!, xeijkt!, and xoijt !, which

denote the numbers of VMs provisioned in reservation,expending, and on-demand phases, respectively. SetxRijk ; ! controls the relationship among the variables byconstraints as expressed in (8)-(15). The constraint in (10)



6/14

maintains the amount of resources utilized in expendingphase to be less than or equal to the number of reservedresources as stated in (2). In (11), the constraint implies thatthe reservation in the first stage can be performed withoutany uncertainty. The constraint in (12) ensures that theconsumers demand for VM class i 2 I in stage t 2 T ismet. The constraint in (13) states that the allocation of

resources for VMs must not exceed the maximum resourcecapacity offered by a cloud provider. Constraints (14) and(15) indicate that variables take the values from a set ofnonnegative integer numbers (i.e., IN0).

QxRijk ; ! min CY; 8where

CY Xi2I

Xj2J

Xk2K

Xt2Tk

crijkt!xrijkt!

Xi2IXj2JXt2T Xk2Kceijkt!xeijkt! coijt !xoijt !

;

9

subject to:

xeijkt!

Xt2Fkt

xrijkt

!; 8i2I; 8j2J; 8k2K; 8t2T; 10

xRijk xrijkt!; t 1; 8i2I; 8j2J; 8k2K; 11

Xj2J

Xk2K

xeijkt!xoijt !

!dit!; 8i2I; 8t2T; 12

Xi2I

birXk2K

xeijkt!xoijt!

ajrt!; 8j2J; 8r2R; 8t2T; 13

xrijkt!2 IN0; 8i2I; 8j2J; 8k2K; 8t2Tk; 14

xeijkt!2 IN0; xoijt !2 IN0; 8i2I; 8j2J; 8k2K; 8t2T: 15

4.2 Deterministic Equivalent Formulation

Given a probability distribution of all scenarios in set ,the formulation in (5)-(15) can be transformed into the

deterministic integer programming called deterministicequivalent formulation as expressed in (16)-(23). To solvethis DEF, probability distributions of both price anddemand must be available, i.e., p! in (16). Then, theDEF can be solved by using traditional optimization solversoftware. For example, the formulation is implementedusing MathProg script, and then the script is solved byGNU Linear Programming Kit (GLPK) [23].Minimize:

z Xi2I

Xj2J

Xk2K

cRijk x

Rijk

X!2

Xi2I

Xj2J

Xk2K

Xt2Tk

p!crijkt!xrijkt!

X!2

Xi2I

Xj2J

Xt2T

p!Xk2K

ceijkt!xeijkt!coijt !xoijt !;16

subject to: (6)

xeijkt!

Xt2Fkt

xrijkt

!; 8i2I; 8j2J; 8k2K; 8t2T; 8!2 ; 17

xRijk xrijkt!; t 1; 8i2I; 8j2J; 8k2K; 8!2; 18

Xj2J

Xk2K

xeijkt!xoijt !

!dit!; 8i2I; 8t2T; 8!2; 19

Xi2I

birXk2K

xeijkt!xoijt !

ajrt!; 8j2J;8r2R;8t2T; 8!2;

20

xrijkt!2IN0; 8i2I; 8j2J; 8k2K; 8t2Tk; 8!2; 21

xe

ijkt!2 IN0; 8i2I; 8j2J; 8k2K; 8t2T; 8!2; 22

xoijt !2 IN0; 8i2I; 8j2J; 8t2T; 8!2: 23

5 BENDERS DECOMPOSITION

In this section, the Benders decomposition algorithm [6] isapplied to solve the stochastic programming problemformulated in Section 4. The goal of this algorithm is tobreak down the optimization problem into multiple smallerproblems which can be solved independently and paral-lelly. As a result, the time to obtain the solution of the OCRP

algorithm can be reduced. The Benders decompositionalgorithm can decompose integer programming problemswith complicating variables into two major problems:master problem and subproblem.

Property 1. The DEF derived in (16)-(23) is the problem whosestructure has multiple complicating variables.

Proof. Variables xeijkt! from the DEF defined in (16)-(23)are considered as complicating variables [6]. Sincevariables x

eijkt! exist in constraints (17), (19), and (20),

the variables prevent the decomposability of the DEF. Ifvariables x

eijkt! are given the fixed values are denoted

by xfixijkt

!

, the DEF can be decomposed into two types of

independent optimization subproblems, namely S1 andS2! presented as follows:S1 Minimize:

zr Xi2I

Xj2J

Xk2K

cRijk x

Rijk

X!2

Xi2I

Xj2J

Xk2K

Xt2Tk

p!crijkt!xrijkt!;24

subject to: (6), (17), (18), (21)

xeijkt!xfixijkt !; 8i2I; 8j2J; 8k2K; 8t2T; 8!2; 25

S2! Minimize:zo !

Xi2I

Xj2J

Xt2T

p!coijt !xoijt !; 26



7/14

subject to: (19), (20), (23)

xeijkt! xfixijkt !; 8i2I; 8j2J; 8k2K; 8t2T: 27

From this decomposition, we conclude that the DEF has

the structure with multiple complicating variables. tuFrom Property 1, the problem can be solved by Benders

decomposition algorithm. The algorithm consists of steps

which are performed iteratively. At each iteration, the master

problem constituted by the complicating variables and

subproblems constituted by the other decision variables

are solved, then lower and upper bounds are calculated. The

algorithm stops when optimal solution converges, i.e., the

lower and upper bounds are satisfactorily close to each other.In Fig. 5, the flowchart of Benders decompostion

algorithm is shown. The algorithm for solving OCRP is

presented in four steps (i.e., Step-0 to Step-3) as follows.Step-0: Initialization of the master problem. In Step-0, the

step is the initialization of the master problem. This Step-0is performed only once, while Step-1 to Step-3 arerepeatable in the algorithm. Let denote the iterationcounter and initially set 1. The master problem asexpressed in (28)-(32) is an alternative form of theformulation DEF shown in (16)-(23).

Minimize:

ze X!2

Xi2I

Xj2J

Xt2T

Xk2K

p!ceijkt!xeijkt! ; 28

subject to:

!lb; 29Xj2J

xeijkt!dit!; 8i2I; 8t2T; 8!2; 30

Xi2IXk2K

birxeijkt!ajrt!; 8j2J; 8r2R; 8t2T; 8!2; 31

xeijkt!2 IN0; 8i2I; 8j2J; 8k2K; 8t2T; 8!2; 32

The objective function (28) is directly derived from that in

(16). xeijkt! represents variable xeijkt! in iteration of

master problem, while variable provides the minimumcost given reservation and on-demand costs. This will be

improved in consequent iterations. Initially, can be fixedby constant lb as shown in constraint (29). This lb can beestimated from an economical analysis or historical data of

prior solutions [6]. Constraints (30)-(32) define the boundaryofx

eijkt!. After solving the master problem, the algorithm

proceeds to the Step-1.Step-1: Subproblem solution. In Step-1, multiple subpro-

blems are formulated and solved. Lets assign solutionx

eijkt! obtained from the master problem to variables

xfixijkt !:

Given the fixed solution xfixijkt !, two aforementioned

subproblems, namely, S1 and S2! can be solvedconcurrently.

The subproblem S1 is presented in (24)-(25) in which theobjective function is to minimize the reservation cost. Let

variable rijkt! denote the optimal solution of the dual

problem of S1 in iteration associated with constraint (25).

The solution of rijkt! will be used in Step-3.

The subproblem S2! is presented in (26)-(27) in whichthe objective function is to minimize the on-demand costwhen the realization is set to !. S2! associates with thenumber of scenarios jj, and hence jj subproblems aregenerated. Note that jj is the Cardinality of set . Letvariable

oijkt! denote the optimal value of the dual

problem of S2! in iteration associated with constraint(27). The solution of

oijkt

!

will be used in Step-3.Step-2: Convergence checking. In Step-2, the convergence of

lower and upper bounds of the solutions obtained frommaster problem and subproblems is checked. Both boundsare adjusted in each iteration. The lower bound in iteration denoted as zlb can be obtained from the objectivefunction of the master problem, i.e., zlb ze . The upperbound in iteration denoted by zub can be obtained from

zub ze zr X!2

zo !: 33

Let denote a small tolerance value to verify the

convergence of both lower and upper bounds. The Bendersdecomposition algorithm stops when zub zlb < , whichmeans both bounds are acceptably close to each other andthe optimal solution can be found in iteration . Otherwise,the algorithm proceeds to the next iteration in which Step-3will perform.

Step-3: Master problem solution.

!X!2

Xi2I

Xj2J

Xk2K

Xt2T

rijkt! oijkt!

x

eijkt! xeijkt!

zr X!2 zo !; 2 f

1;. . .

; 1g:

34

Let the iteration counter be increased by 1.Then, the master problem in (28)-(32) can be further relaxed


Fig. 5. Flowchart of Benders decomposition algorithm.


8/14

by additional constraints called Benders cuts [6]. In addition,the solution of the master problem will adjust the cost and also the expending cost according to solution ofx

eijkt!. As shown in (34), the Benders cuts are constructed

from the optimal costs obtained from master problem andsubproblems in the prior iterations. After solving thismaster problem, Step-1 is repeated and the same iterativeprocess continues.

6 SAMPLE-AVERAGE APPROXIMATION

In the case that the number of scenarios is numerous, itmay not be efficient to obtain the solution of the OCRPalgorithm by solving the stochastic programming formula-tion defined in (16)-(23) directly if all scenarios in theproblem are considered. To address this complexity issue,the sample-average approximation (SAA) approach isapplied [7]. This approach selects a set of scenarios, e.g.,N scenarios, where N is smaller than the total number ofscenarios jj. Then, these N scenarios can be solved in adeterministic equivalent formulation. The optimal solutioncan be obtained if N is large enough which can be verifiednumerically.

In this section, the SAA approach is applied toapproximate the expected cost in every considered provi-sioning stage, i.e., QxRijk ; ! in (7). A sampling method(e.g., Monte Carlo [24] and Latin hypercube [25]), is used togenerate scenarios N f!1; . . . ; !Ng, where N denotes thesample size. Let N f1; . . . ; Ng be the set of indices ofsamples. Then, the expected cost can be redefined asshown in (42).

The function QxRijk ; N is the SAA to the objectivefunction in (5). Then, the problem can be transformed into adeterministic equivalent formulation, as called approxima-tion problem (AP) formulation, as expressed in (35)-(41).Minimize:

zNXi2I

Xj2J

Xk2K

cRijk x

Rijk

1

N

Xn2N

Xi2I

Xj2J

Xk2K

Xt2Tk

crijkt!nxrijkt!n

1N

Xn2N

Xi2I

Xj2J

Xt2T

Xk2K

ceijk!nxeijkt!ncoijt !nxoijt !n

35subject to: (6)

xeijkt!n

Xt2Fkt

xrijkt

; 8i2I; 8j2J; 8k2K; 8t2T; 8n2N; 36

xRijk xrijkt!n; t1; 8i2I; 8j2J; 8k2K; 8n2N; 37

Xj2J

Xk2K

xeijkt!n xoijt !n

! dit!n; 8i2I; 8t2T; 8n2N;

38

Xi2I

birXk2K

xeijkt!nxoijt !n ajrt!n;8j2J; 8r2R;8t2T;8n2N;

39

xrijkt!n2IN0; 8i2I; 8j2J; 8k2K; 8t2Tk; 8n2N; 40

xeijkt!n;xoijt !n2 IN0; 8i2I;8j2J;8k2K;8t2T;8n2N:41

Q

xRijk ; N

1

N

XNn1

Q

xRijk ; !n

: 42

Let z and x denote the optimal objective function valueand optimal solution of the original formulation (defined in(5) and (6)), respectively. Let zN and x

N denote the optimal

objective function value and optimal solution of the APformulation, respectively. Note that although zN becomescloser to z when N is large, value zN naturally variesaccording to set of samples N. Therefore, an estimationmethod is required to achieve the SAA lower and upperbounds of the optimal solution. Obviously, zN forms theSAA upper bound of z as follows:

z zN: 43

In addition, the SAA lower bound of z is formed by thefollowing unbiased property

IEzN z: 44For the properties in (43) and (44), bounding method is

required to obtain the estimates of both SAA upper andlower bounds on z with a certain confidence interval. Thenext two following sections present the estimation of theSAA bounds by applying the similar method to that in [7],[21], [24].

6.1 SAA Lower Bound Estimates

The expected value IEzN can be estimated by generatingM independent batches,2 each of size N, denoted as!1;m; . . . ; !N;m where m 2 f1; . . . ; Mg, then solving the APformulation. Let zN;m denote the solution given batch m.Next, the SAA lower bound can be obtained from

LN;M 1M

XMm1

zN;m : 45

Due to (44), this LN;M is an unbiased estimator of the meanIEzN which forms a statistical lower bound for z. Whenthe generated M batches are independent and identicallydistributed (i.i.d.) by the Central Limit Theorem, thedistribution of SAA lower bound estimate converges to anormal distribution N0; 2L,3 namelyffiffiffiffiffi

Mp

LN;M IEzN D N 0; 2L; as M 1; 46

where 2L VarzN4 which can be approximated by thesample variance estimator s2LM as follows:

s2LM 1

M 1XMm1

zN;m LN;M2: 47

Finally, the 1 -confidence interval of the SAA lowerbound can be defined as


2. A batch is a set of samples generated by a sampling technique.3.N 0; 2L denotes the normal distribution with mean 0 and variance 2L.4. VarzN denotes the variance of samples.


9/14

LN;M z=2sLMffiffiffiffiffi

Mp ; LN;M

z=2sLMffiffiffiffiffiM

p !

; 48

where z satisfies ProbfN0; 1 zg 1 . Note that thevalue z=2 from the Z-distribution can be replaced by criticalvalue t=2;M1 from the Students t-distribution ifM is small.

6.2 SAA Upper Bound Estimates

As shown in (43),^zN forms the SAA upper bound of z. Byselecting a solution xN obtained by solving the AP formula-

tion, the SAA upper bound can be estimated by using theunbiased estimator of zN. To achieve this estimator, we cangenerate ~M independent batches, each of size ~N, denoted as!1;m; . . . ; ! ~N;m where m 2 f1; . . . ; ~Mg. Since solution xN isalready fixed to the SAA problem, i.e., zNxN, given ~Nsamples of each generated batch, the objective function (35)can be decomposed into ~N independent subproblems inwhich each of them can be solved independently and inparallel. Thus, these subproblems can be solved moreefficiently than the original AP formulation. After allsubproblems are solved, each batch yields the solution

denoted as z~N;mxN. Then, the SAA upper bound can beestimated from

U~N; ~MxN 1

~M

X~Mm1

z~N;mxN: 49

Again, the distribution of SAA upper bound estimateconverges to a normal distribution N0; 2UxN givensolution xN, namelyffiffiffiffiffi

~Mp

U~N; ~MxN zNxNDN0; 2UxN; as ~M1; 50

where 2

U^xN

Var^zN

^xN which can be approximatedby the sample variance estimator s2U ~M; xN as follows:

s2U ~M; xN 1

~M 1X~Mm1

z~N;mxN U~N; ~MxN2: 51

Finally, the 1 -confidence interval of the SAA upperbound can be obtained from

U~N; ~MxN z=2sU ~M;xNffiffiffiffiffi

~Mp ; U~N; ~MxN z=2sU ~M;xNffiffiffiffiffi

~Mp

" #: 52

6.3 Monte Carlo Sampling

The Monte Carlo sampling technique can be applied togenerate scenarios N. Each scenario is from as follows.First, the random number is uniformly selected from 0; 1.Then, the scenario is derived by the inverse transformationmethod given the random number and cumulative prob-ability distribution of . The same process is iterativelyperformed until the N scenarios are completely chosen.

6.4 Obtaining the Optimal Solution

The optimal solution based on the SAA approach can beobtained when N is sufficiently large. However, we canselect an optimal solution by solving different SAA

problems with different size of N. Until the same solutionis found in these problems, we can choose the solution asthe desired solution. The detail and result of this solutionmethod is presented in the next section.

7 PERFORMANCE EVALUATION

In this section, the performance evaluation of the proposedOCRP algorithm is presented. Two case studies areconsidered in this evaluation, namely two provisioning stageproblem (2-PSP) and 12 provisioning stage problem (12-PSP).The former, 2-PSP, has only two provisioning stages. Weassume that the cloud broker is making a decision for

provisioning resources at the end of year. Under price anddemand uncertainty, the cloud broker performs theadvance reservation of resources in the first stage forbeing used in the next whole year which is the secondstage. Therefore, the 1-year reservation contract is suffi-ciently required by the broker since the contract can coverthe time duration. At the second stage, the price anddemand are observed. Then, the number of reservedresources are utilized and some additional amount ofresources can be provisioned in an on-demand fashion.

For 12-PSP, we consider 12 months in a year and hencethe stage is defined as T fT1; T2; . . . ; T12g. In each stage,the cloud broker can perform an advance reservation of

resources for being used in the next incoming monthswithin these 12 months. Moreover, additional resources canbe provisioned by purchasing on-demand plans if thereserved resources cannot meet the actual demand. For bothcase studies, the optimal solution obtained from the OCRPalgorithm is the amount of reserved resources in differentprovisioning stages (or the first stage for 2-PSP). Since theamount of resources is reserved for the number of VMs, thisoptimal solution can be considered to the number of reservedVMs in other words.

7.1 Experiment Setup

7.1.1 Setting of Cloud Computing Environment

We first present the parameter setting of a cloud computingenvironment used in this performance evaluation. Theenvironment consists of only one cloud consumer (i.e.,organization) who is renting computing resources offered bycloud providers. The consumer has two different types ofapplications representing two distinct VM classes, namelyI fI1; I2g. For instance, I1 is a database server and I2 is aweb server. Each VM class requires different amount ofresources. Processing time units required by a VM in classesI1 and I2 are 8,748 and 6,570 CPU-hours per year,respectively. Permanent storage capacities required by aVM in classes I1 and I2 are 1,920 and 1,200 GBs per year,

respectively. Network bandwidth regarding outbound data-transfer required by a VM in classes I1 and I2 are 24,000 and30,000 GBs per year, respectively. The cost of inbound datatransfer is assumed to be free of charge. Furthermore, asoftware package is needed to be installed in a VM of eachVM class. A software package consists of operating system,database software (for class I1 only), web applicationsoftware (for class I2 only), and other utility software. Thesoftware cost is additionally charged to the consumer as thelicense cost per a running VM. The license software costscorresponding to a VM in classes I1 and I2 are $500 and$1,200, respectively. The consumer is assumed to purchase

these software licenses from software vendors when VMs arerunning in expending and on-demand phases.

The environment consists of four cloud providers,namely J fJ1; J2; J3; J4g. J1 represents the private cloud



10/14

and the other represent the three public clouds. The privatecloud J1 is the data center belonging to the cloud consumer.This data center has housed only 10 physical servers. Eachserver is assumed to offer 100 percent uptime systemavailability, i.e., 24 365 8;760 CPU-hours per year.Therefore, for the whole data center, J1 can offer 87,600CPU-hours as the maximum processing capacity. The otherresource types in J1 is assumed to be abundant to serve allVMs run by the consumer. The total cost to utilize theservers in J1 is only considered from the annual energy cost.

This annual cost is assumed to be the average energy costtaken by Dell PowerEdge M600 blade server as presented in[26]. The cost is calculated as 454:39=1;000 kilowatts(average power consumption per server5) $0:0897 (elec-tric charge per watt-hour)24 (hours per day) 365 (daysper year) % $357 per server per year. For only J1, this cost($357) is considered as the reservation cost to reserveresources for a VM, while the expending cost of processingtime and storage capacity are omitted. Only the cost ofnetwork bandwidth is charged to $0.10 per GB per month ofoutbound data transfer. Furthermore, the on-demand planfor provisioning resources is unavailable in J1.

The public cloud providers (J2, J3, and J4) are assumedto offer unlimited capacity of all resource types, so theconstraint in (13) is omitted. Pricing of resources in providerJ2 is based on the prices defined by Amazon EC2 (inFebruary 2010), and the price of processing time is based onthat of the Small Instance type [2]. However, pricing inproviders J3 and J4 is artificially and reasonably defined.Providers J2 and J3 offer customers both reservation andon-demand plans, while J4 has only an on-demand plan.Providers J2 and J3 offer customers three differentreservation contracts, namely 3-month (3M), 6-month(6M), and 1-year (1Y) contracts. Pricing of resource inexpending and on-demand phases is charged as the pay-per-use basis based on the actual usage per resource unit.The resource unit of processing time is CPU-hour, whileone of storage capacity and network bandwidth is GBs permonth. For the network bandwidth, only the outbound datatransfer is charged, while the inbound one is free of chargein every cloud provider.6 Pricing defined by every cloudprovider is listed in Table 2.

7.1.2 Uncertainty Parameters

In this part, two main uncertainty parameters including theprice of resource and the demand as the number of requiredVMs per VM class, are defined. The price in the on-demand

phase of all resource types can be doubly increased, withprobability 0.1, from the price defined in Table 2. Withprobability 0.9, the price in the on-demand phase remainsunchanged. For the demand uncertainty, the actual re-quired number of VMs in the second provisioning stage ofVM class (i.e., I1 and I2) varies from 1 to 50. The demand ofone VM class is assumed to be the same as the other class.Three distributions of demand are considered in theexperiment, namely discrete normal distribution, uniformdistribution, and distribution from test data. Means of both

normal and uniform distributions are set to 25.50. Thevariance of normal distribution is set to 6, while thevariance of uniform distribution is 208.25. The lastdistribution is generated from the logged data obtainedfrom Institute of High Performance Computing (IHPC) inSingapore. The data are collected from the actual usage ofshared computing resources located in the computer clustermaintained by IHPC. The probability distribution (withmean = 21.64 and variance = 389.93) of the resource usagerepresenting the number of required VMs is derived asshown in Fig. 6.

7.2 Case Study: Two Provisioning Stage Problem

7.2.1 Cost Structure

First, the cost structure to provision resources is studied. Toease the illustration, this study considers only single VMclass I1 and single cloud provider J2. The number ofrequired VMs (i.e., demand) is varied following the normaldistribution. In Fig. 7, given different number of reservedVMs, cost in the first stage called first stage cost (which isactually reservation cost), cost in the second stage calledsecond stage cost including expending and on-demand costs,and total cost, are presented. As expected, the first stagecost increases, as the number of reserved VMs increases.

However, the second stage cost decreases after the demandis realized, since the cloud consumer needs smaller numberof VMs provisioned by on-demand plan. In this case, the


TABLE 2Pricing Defined by Each Cloud Provider

5. We omit the power consumption of cooling system as it is the fixedcost of private cloud in the organization.

6. The charge of inbound data transfer offered by Amazon EC2 has beenfree through June 30, 2010. Fig. 6. The probability distribution of the real data.


11/14

optimal number of reserved VMs can be determined to be30 reserved VMs as shown in Fig. 7, which is the point thatthe total cost is minimum. Clearly, even in this small setting(one VM class and one provider), the optimal solution is nottrivial to obtain due to the demand uncertainty. Therefore,the OCRP algorithm would be required to guarantee theminimum cost to the consumer.

Given the optimal reservation of 30 VMs as shown in

Fig. 7, the comparison between resource provisioning withand without reservation can be made as illustrated inFig. 8. Without reservation, the number of VMs is todynamically provision resources in the second stage byonly purchasing resources in the on-demand plan. Givendifferent demands (or realization of required number ofVMs) from 1 to 50, the cost in resource provision withreservation becomes cheaper than that without reservationdue to the discounted price of processing time. However,the cost with reservation may not always be the cheapest.As shown in Fig. 8, the total cost in the resource provisionwithout reservation is lower than the other one until thedemand is 15 in which the effective reservation begins.This fact indicates that even if the solution is optimal, itcannot guarantee the best solution in all realizations ofobserved parameters. Therefore, the effective way to tacklethe uncertainty is not to search for the best solution forevery possible situation happening in the future, but toobtain the solution which is able to minimize the tradeoffbetween advance reservation and on-demand provisionwhile the uncertainty is carefully considered.

7.2.2 Impact of Probability Distributions

For the next experiments, all parameters of the cloudcomputing environment are applied. The deterministic

equivalent formulation derived in Section 4.2 is implemen-ted and solved by GLPK.

The stochastic effect of demand under different prob-ability distributions is investigated. In Table 3, the numberof reserved VMs (indicated as label N.R. in Table 3)

provisioned in the first stage is presented. Also, theexpected second stage costs incurred due to differentprobability distributions are shown. The costs include thereservation cost (R.C.), expending cost (E.C.), on-demandcost (O.C.), oversubscribed cost (O.S.C.), and total cost. Theprobability distributions include normal distribution(Norm), uniform distribution (Uniform), and distributionfrom test data. Furthermore, three variances of the normaldistribution are considered, namely variance = 4 (Norm v4),6 (Norm v6), and 8 (Norm v8).

In Table 3, we observe that as the variance increases thenumber of VMs also needs to be increased. Therefore, thenumber of reserved VMs in the test data is highest, whilethat in Norm v4 is lowest. Larger variance increases thechance that the demand will be smaller or larger than themean. Consequently, Norm v8 incurs more total cost andreserves more VMs than those in Norm v4 and Norm v6.Again, the increment of number of reserved VMs canensure that the on-demand cost can be minimized.

7.2.3 Comparison with Other Provisioning Algorithms

Next, the comparison between provisioning algorithms isperformed. The algorithms include the proposed OCRP,expected-value of uncertainty provisioning (EVU), max-imum advance reservation provisioning (MaxRes), andnonreservation provisioning (NoRes) algorithms. EVU usesthe average values of uncertainty parameters and solvesthem by a traditional deterministic program. MaxResreserves the maximum number of available VMs, whileNoRes does not reserve any resources. Both MaxRes andNoRes also apply the traditional deterministic program forallocating VMs to cloud providers. All algorithms with thedefined input parameters are coded and solved by GLPK.

The given distributions are applied to the possible scenariosof demand and price, respectively. The solution obtainedfrom each solved algorithm yields the number of reservedVMs (N.R.) and the allocation of VMs to providers. Then, asimulation program is developed to evaluate the solution of


Fig. 7. The optimal solution in a simple cloud computing environment.

Fig. 8. Comparison between total costs of resource provision with andwithout reservation.

TABLE 3Number of Reserved VMs and Costs Given Different Probability Distributions


12/14

each algorithm. The simulation contains 1,000 iterations. Ineach iteration, the random number is uniformly selectedfrom 0; 1. Next, the scenario is derived by the inversetransformation method given the random number andcumulative probability distributions of the scenarios. Theprovisioning costs incurred by purchasing the provisioningplans given by the solution of each algorithm are recorded.After finishing the last iteration, the simulation calculatesthe average costs as presented in Table 4. The costs includereservation cost (R.C.), expending cost (E.C.), on-demandcost (O.C.), oversubscribed cost (OS.C.), and total cost.

In Table 4, the proposed OCRP achieves the lowest totalcost, while NoRes yields the highest total cost due to thehighest on-demand cost. The OCRP algorithm reserves59 VMs (including both classes I1 and I2). AlthoughMaxRes reserves 100 VMs (50 per VM class) to entirelyavoid the higher cost in the on-demand plan, it still incursmuch higher cost than that of OCRP. Additionally, MaxResincurs the highest oversubscribed cost since the reservedresources are unnecessarily overprovisioned. EVU incursthe total cost lower than those of MaxRes and NoResalgorithms. Although the oversubscribed cost of OCRP is

higher than that of EVU, the on-demand cost of the OCRPalgorithm is much lower. Again, it is possible that the on-demand cost can increase due to the price uncertainty. As aresult, the diminution of the on-demand cost is moreimportant. The result of this experiment shows the balancebetween the number of provisioning resources to beacquired in the first and second stages in which OCRPcan provide the most optimal tradeoff.

7.2.4 Bender Decomposition

Fig. 9 shows the bound convergence obtained by solvingBender decomposition algorithm. The adjustment of lowerand upper bounds is performed in each iteration. At

iteration 42, algorithm converges. The optimal solutionobtained from the decomposition is the same as oneobtained by solving DEF without decomposition. Weobserve that the subproblems can be solved efficiently dueto their smaller number of variables and parallelization.

However, solving the master problem requires substantialamount of time since more Benders cuts have to be added.

7.3 Case Study: Twelve Provisioning StageProblem

The SAA approach presented in Section 6 is studied. In this12-PSP, only provider J2 and J3 are considered. It isassumed that the resource price is stable and the demandvaries within set f10; 20; 30; 40; 50g (i.e., 512 244;140;625scenarios for the 12 stages). For sampling data, four sample

sizes are determined, namely N 2 f200; 500; 600; 750g.Then, the Monte Carlo sampling technique generatesdemand realizations for five batches per each size, i.e.,M 5. Lower bound estimate of each sample size iscalculated by solving five SAA problems with respectivebatches. For upper bound estimate, the solution obtainedfrom each solved SAA problem is fixed to another newSAA problem whose sample size is 750, i.e., ~N 750. Tenbatches of the new SAA problem are constructed andsolved, i.e., ~M 10. Then, the solutions obtained from theten batches can be calculated as the upper bound estimate.

In Table 5, the estimates of SAA lower and upper boundsare presented. The optimal solution is found in the samplesize of 750. From this optimal solution, advance reserva-tions with only the 6-month reservation contract are used inonly T1 and T7. That is, 10 VMs will be reserved to providerJ2 in stages T1 and T7 each and 30 VMs will be reserved to


Fig. 9. Convergence of the upper and lower bounds by applying theBenders decomposition.

TABLE 5Estimation of Lower and Upper Bounds of Provisioning

Costs in the 12 Provisioning Stage Problem

TABLE 4Number of Reserved VMs and Average Costs Given Different Resource Provisioning Algorithms


13/14

provider J3 in stages T1 and T7 each. We conclude that 40reserved VMs are only needed in stages T1 and T7 each. Thissolution can avoid the higher on-demand cost, since only 10more VMs could be provisioned with on-demand plan inany provisioning stages.

7.4 Discussion

7.4.1 Experimental Results

1. Balance of costs. We observe that the cloud brokerwith OCRP will minimize on-demand cost ratherthan the oversubscribed cost. Since resource pricingin the on-demand plan is higher and possiblyincreased by cloud providers, the reservation planis more attractive by the cloud broker. However,reserving too many VMs may not be optimal (e.g.,as that of MaxRes from Table 4). Therefore, thetradeoff between on-demand and oversubscribedcosts needs to be adjusted in which OCRP canoptimally perform.

2. Virtual machine outsourcing. As presented in Sec-tion 7.2, the VM outsourcing from a private cloud toa public cloud provider (or public cloud) showsinteresting result. In the experiment, the privatecloud fully utilizes its own resources. Then, extraVMs can be spilled over to public clouds. Purchasingand deploying new hardware to a private cloud maynot be an optimal solution, since the total cost ofownership (TCO) must be considered. To reduce thisTCO, workload outsourcing is the attractive choicewhich is shown from our evaluation.

7.4.2 Implementation Issues

1. Multiple provisioning stages planning issue. As shownin Section 7.3, the OCRP algorithm can be applied tomultiple provisioning stages representing long-termplanning. Since the optimal solution of the firstprovisioning stage depends on multiple probabilitydistributions describing the uncertainty occurring inconsequent time epochs, multiple stages planning isneeded. For example, the workload of some onlinesouvenir shopping websites could be dramaticallyincreased in the high-season consisting of many timeperiods in a year (e.g., Christmas Day, ValentinesDay, etc.). As a result, the websites should provisionresources by considering multiple time epochs (i.e.,provisioning stages) in advance, while reservationcontracts offered by cloud providers can be takeninto account to reduce the provisioning cost.

2. Use of decomposition method. The use of decomposi-tion method for OCRP has to be carefully consid-ered, since the formulation of the OCRP algorithm isa pure integer program which is the NP-hardproblem [5]. Although the subproblems can besolved in parallel, the master problem with theadditional Benders cuts requires considerable com-putational time. The performance improvement ofthe decomposition algorithm will be considered in

the future work.3. Benefit of SAA. Sample-average approximation meth-

od can overcome the provisioning problems with alarge set of scenarios which are impossible to solve

with deterministic equivalent formulation directly.In contrast, the approximation algorithm withestimation of SAA lower and upper bounds canyield tolerably solutions while the problems can bepractically solved in timely manner.

4. Limitation of stochastic programming. Stochastic pro-gramming does not address the method to obtainappropriate probability distributions describing un-certainty (i.e., distributions of scenarios ). How-ever, this limitation can be alleviated by applyingvariance reduction techniques (e.g., importancesampling [27]) to increase the precision of theestimates of uncertainty.

7.4.3 Future Research Direction

For the future work, scenario reduction techniques [28] willbe applied to reduce the number of scenarios. In addition,the optimal pricing scheme for cloud providers withthe consideration of competition in the market will beinvestigated.

8 CONCLUSION

In this paper, we have proposed an optimal cloud resourceprovisioning (OCRP) algorithm to provision resourcesoffered by multiple cloud providers. The optimal solutionobtained from OCRP is obtained by formulating andsolving stochastic integer programming with multistagerecourse. We have also applied Benders decompositionapproach to divide an OCRP problem into subproblemswhich can be solved parallelly. Furthermore, we haveapplied the SAA approach for solving the OCRP problemwith a large set of scenarios. The SAA approach caneffectively achieve an estimated optimal solution even theproblem size is greatly large. The performance evaluationof the OCRP algorithm has been performed by numericalstudies and simulations. From the results, the algorithm canoptimally adjust the tradeoff between reservation ofresources and allocation of on-demand resources. TheOCRP algorithm can be used as a resource provisioningtool for the emerging cloud computing market in which thetool can effectively save the total cost.

ACKNOWLEDGMENTS

This work was done in the Parallel and DistributedComputing Centre (PDCC) of the School of ComputerEngineering, Nanyang Technological University, Singapore.This work was supported by the projects User and DomainDriven Data Analytics and Design and Analysis of CloudComputing for Data Value Chain: Operation ResearchApproach, granted by the A*STAR Thematic StrategicResearch Programme.

REFERENCES[1] I. Foster, Y. Zhao, and S. Lu, Cloud Computing and Grid

Computing 360-Degree Compared, Proc. Grid Computing Envir-

onments Workshop (GCE 08), 2008.[2] Amazon EC2, http://aws.amazon.com/ec2, 2012.[3] GoGrid, http://www.gogrid.com, 2012.[4] Amazon EC2 Reserved Instances, http://aws.amazon.com/ec2/

reserved-instances, 2012.



14/14

[5] F.V. Louveaux, Stochastic Integer Programming, Handbooks inOR & MS, vol. 10, pp. 213-266, 2003.

[6] A.J. Conejo, E. Castillo, and R. Garca-Bertrand, Linear Program-ming: Complicating Variables, Decomposition Techniques in

Mathematical Programming, chapter 3, pp. 107-139, Springer, 2006.[7] J. Linderoth, A. Shapiro, and S. Wright, The Empirical Behavior

of Sampling Methods for Stochastic Programming, Ann. Opera-tional Research, vol. 142, no. 1, pp. 215-241, 2006.

[8] G. Juve and E. Deelman, Resource Provisioning Optionsfor Large-Scale Scientific Workflows, Proc. IEEE Fourth Intl Conf.e-Science, 2008.

[9] Z. Huang, C. He, and J. Wu, On-Demand Service in Grid:Architecture Design, and Implementation, Proc. 11th Intl Conf.Parallel and Distributed Systems (ICPADS 05), 2005.

[10] Y. Jie, Q. Jie, and L. Ying, A Profile-Based Approach to Just-in-Time Scalability for Cloud Applications, Proc. IEEE Intl Conf.Cloud Computing (CLOUD 09), 2009.

[11] Y. Kee and C. Kesselman, Grid Resource Abstraction, Virtualiza-tion, and Provisioning for Time-Target Applications, Proc. IEEEIntl Symp. Cluster Computing and the Grid, 2008.

[12] A. Filali, A.S. Hafid, and M. Gendreau, Adaptive ResourcesProvisioning for Grid Applications and Services, Proc. IEEE IntlConf. Comm., 2008.

[13] D. Kusic and N. Kandasamy, Risk-Aware Limited LookaheadControl for Dynamic Resource Provisioning in Enterprise Com-

puting Systems, Proc. IEEE Intl Conf. Autonomic Computing, 2006.[14] K. Miyashita, K. Masuda, and F. Higashitani, Coordinating

Service Allocation through Flexible Reservation, IEEE Trans.Services Computing, vol. 1, no. 2, pp. 117-128, Apr.-June 2008.

[15] J. Chen, G. Soundararajan, and C. Amza, Autonomic Provision-ing of Backend Databases in Dynamic Content Web Servers, Proc.IEEE Intl Conf. Autonomic Computing, 2006.

[16] L. Grit, D. Irwin, A. Yumerefendi, and J. Chase, Virtual MachineHosting for Networked Clusters: Building the Foundations forAutonomic Orchestration, Proc. IEEE Intl Workshop VirtualizationTechnology in Distributed Computing, 2006.

[17] H.N. Van, F.D. Tran, and J.-M. Menaud, SLA-Aware VirtualResource Management for Cloud Infrastructures, Proc. IEEENinth Intl Conf. Computer and Information Technology, 2009.

[18] M. Cardosa, M.R. Korupolu, and A. Singh, Shares and Utilities

Based Power Consolidation in Virtualized Server Environments,Proc. IFIP/IEEE 11th Intl Conf. Symp. Integrated Network Manage-ment (IM 09), 2009.

[19] F. Hermenier, X. Lorca, and J.-M. Menaud, Entropy: AConsolidation Manager for Clusters, Proc. ACM SIGPLAN/SIGOPS Intl Conf. Virtual Execution Environments (VEE 09), 2009.

[20] N. Bobroff, A. Kochut, and K. Beaty, Dynamic Placement ofVirtual Machines for Managing SLA Violations, Proc. IFIP/IEEEIntl Symp. Integrated Network Management (IM 07), pp. 119-128,May 2007.

[21] P. Jirutitijaroen and C. Singh, Reliability Constrained Multi-AreaAdequacy Planning Using Stochastic Programming with Sample-Average Approximations, IEEE Trans. Power Systems, vol. 23,no. 2, pp. 504-513, May 2008.

[22] S. Chaisiri, B.S. Lee, and D. Niyato, Optimal Virtual MachinePlacement across Multiple Cloud Providers, Proc. IEEE Asia-

Pacific Services Computing Conf. (APSCC), 2009.[23] GNU Linear Programming Kit (GLPK), http://www.gnu.org/

software/glpk, 2012.[24] W.-K. Mak, D.P. Morton, and R.K. Wood, Monte Carlo Bounding

Techniques for Determining Solution Quality in StochasticPrograms, Operations Research Letter, vol. 24, pp. 47-56, 1999.

[25] M.D. McKay, R.J. Beckman, and W.J. Conover, A Comparison ofThree Methods for Selecting Values of Input Variables in theAnalysis of Output from a Computer Code, Technometrics,vol. 21, no. 2, pp. 239-245, 1979.

[26] R. Chheda, D. Shookowsky, S. Stefanovich, and J. Toscano,Profiling Energy Usage for Efficient Consumption, Architecture

J., no. 18, 2008.[27] G.B. Dantzig and G. Infangerm, Large-Scale Stochastic Linear

Programs: Importance Sampling and Benders Decomposition,

Proc. IMACS World Congress on Computation and Applied Math.,1991.[28] H. Heitsch and W. Romisch, Scenario Reduction Algorithms in

Stochastic Programming, J. Computational Optimization andApplications vol 24 pp 187-206 2003

Sivadon Chaisiri received the MEng degreefrom Kasetsart University, Bangkok, Thailand, in2005. He is currently working toward the PhDdegree at Nanyang Technological University,Singapore. His current research interests in-clude cloud computing and distributed systems.He is a student member of the IEEE.

Bu-Sung Lee received the BSc (Hons.) andPhD degrees from the Electrical and ElectronicsDepartment, Loughborough University of Tech-nology, United Kingdom, in 1982 and 1987,respectively. He is currently an associate pro-fessor with the School of Computer Engineering,Nanyang Technological University, Singapore.He was elected the inaugural president ofSingapore Research and Education Networks(SingAREN), 2003-2007, and has been an

active member of several national standards organizations, such as aboard member of Asia Pacific Advanced Networks (APAN) Ltd. In 2010,he held a joint appointment as director, Service Platform Lab, HP LabsSingapore. His research interests include computer networks protocols,distributed computing, network management, and grid/cloud computing.He is a member of the IEEE.

Dusit Niyato received the BE degree from KingMongkuts Institute of Technology Ladkrabang,Bangkok, Thailand, in 1999 and the PhD degreein electrical and computer engineering from theUniversity of Manitoba, Winnipeg, Canada, in2008. He is currently an assistant professor inthe Division of Computer Communications,School of Computer Engineering, NanyangTechnological University, Singapore. His currentresearch interests include the design, analysis,

and optimization of wireless communication, smart grid systems, greenradio communications, and mobile cloud computing. He is a member of

the IEEE.


optimization of resource provisioning

Documents