Zhang et al.

RESEARCH

Service Workload Patterns for QoS-driven CloudResource ManagementLi Zhang1, Yichuan Zhang1, Pooyan Jamshidi2, Lei Xu2 and Claus Pahl2*

Abstract

Cloud service providers negotiate SLAs forcustomer services they offer based on the reliabilityof performance and availability of their lower-levelplatform infrastructure. While availabilitymanagement is more mature, performancemanagement is less reliable. In order to support acontinuous approach that supports the initial staticinfrastructure configuration as well as dynamicreconfiguration and auto-scaling, an accurate andefficient solution is required. We propose aprediction technique that combines a workloadpattern mining approach with a traditionalcollaborative filtering solution to meet the accuracyand efficiency requirements. Service workloadpatterns abstract common infrastructure workloadsfrom monitoring logs and act as a part of afirst-stage high-performant configurationmechanism before more complex traditionalmethods are considered. This enhances currentreactive rule-based scalability approaches and basicprediction techniques by a hybrid predictionsolution. Uncertainty and noise are additionalchallenges that emerge in multi-layered, oftenfederated cloud architectures. We specifically addlog smoothing combined with a fuzzy logicapproach to make the prediction solution morerobust in the context of these challenges.

Keywords: Quality of Service; ResourceManagement; Cloud Scalability; Web and CloudServices; QoS Prediction; Workload PatternMining; Uncertainty

1 INTRODUCTIONQuality of Service (QoS) is the basis of cloud serviceand resource configuration management [1] [2]. Cloud

*Correspondence: cpahl@computing.dcu.ie2IC4 / School of Computing, Dublin City University, Dublin, Ireland

Full list of author information is available at the end of the article

service providers – whether at infrastructure, platformor software level – provide quality guarantees usuallyin terms of availability and performance to their cus-tomers in the form of service-level agreements (SLAs)[3]. Internally, the respective service configuration interms of available resources then needs to make surethat the SLA obligations are met [4]. To facilitate SLAconformance, virtual machines (VMs) can be config-ured and scaled up/down in terms of CPU cores andmemory, deployed with storage and network capabili-ties. Some current cloud infrastructure solutions allowusers to define rules manually to scale up or down tomaintain performance levels.

QoS factors like service performance in terms of re-sponse time or availability may vary depending on net-work, service execution environment and user require-ments, making it hard for providers to choose an ini-tial configuration and scale this up/down to maintainthe SLA guarantees, but also optimising resource util-isation at the same time. We utilise QoS predictiontechniques here, but rather than bottom-up predict-ing QoS from monitored infrastructure metrics [5] [6][7], we reverse the idea, resulting in a novel techniquefor pattern-based resource configuration.

A pattern technique is at the core of the solution.Various types of cloud computing patterns exist [8],covering workload, but also offer types and applicationand management architectures. These patterns link in-frastructure workloads such as CPU utilisation withservice-level performance. Recurring workloads havealready been captured as patterns in the literature [8],but we additionally link these to service quality.

We determine service workload patterns throughpattern mining from resource utilisation logs. Theseservice workload patterns (SWPs) correspond to typ-ical workloads of the infrastructure and map these toQoS values at the service level. A pattern consistsof a relatively narrow range of metrics measured foreach infrastructure concern such as compute, mem-ory/storage and network under which the QoS concernis stable. This can be best illustrated through utilisa-tion rates. Should resources be utilised in a certainrange, e.g., low utilisation of a CPU around 20%, thenthe response-time performance can be expected to be

Zhang et al. Page 2 of 20

high and not impacted negatively by the infrastruc-ture.

These patterns can then be used in the followingway. In a top-down approach, we take a QoS require-ment and determine suitable workload-oriented config-urations that maintain required values. Furthermore,we enhance this with a cost-based selection function,applicable if many candidate configurations emerge.

We specifically look at performance as the QoS con-cern here since dealing with availability in cloud envi-ronments is considered as easier to achieve, but per-formance is currently neglected in practice due to lessmature resource management techniques [4]. We in-troduce pattern mining mechanisms and, based on aQoS-SWP matrix, we define SWP workload configu-rations for required QoS. The accuracy of the solutionto guarantee that the chosen (initially predicted) re-source configurations meet the QoS requirements is ofutmost importance. An appropriate scaling approachis required in order to allow this to be utilised in dy-namic environments. In this paper, we show that thepattern-based approach improves the efficiency of thesolution in comparison with traditional prediction ap-proaches, e.g., based on collaborative filtering. This en-hances existing solutions by automating current man-ual rule-based reactive scalability mechanisms and alsoadvances prediction approaches for QoS, making themapplicable in the cloud with its accuracy and perfor-mance requirements.

Cloud systems are typically multi-layer architectureswith services being provided at infrastructure, plat-form or software application level. Uncertainty andnoise are additional challenges that emerge in thesemulti-layered, often federated clouds architectures. Weextend earlier work [9] to address these challenges.We propose to use log smoothing and a fuzzy logic-based approach to make the prediction solution morerobust in the context of these challenges. Smoothingwill deal with log data variability and will allow detect-ing trends (but adds more noise). Uncertainty oftenoccurs as to the completeness and reliability of mon-itored data, which will here be addressed through afuzzy logic enhanced prediction. We will demonstratethe robustness of the solution against noise and uncer-tainty.

Section 2 outlines the solution and justifies its prac-tical relevance. Section 3 introduces SWPs and howthey can be derived. Section 4 discusses the selectionof patterns as workload specifications for resource con-figuration. The application of the solution for SLA-compliant cloud resource configuration is described inSection 5. Section 6 deals with uncertainty througha fuzzification of the patterns. Section 7 contains an

evaluation in terms of accuracy, performance and ro-bustness of the solution and Section 8 discusses relatedwork.

2 QUALITY-DRIVEN CONFIGURATIONAND SCALING

Cloud resource configuration is the key problem weaddress. We start with a brief discussion of the state-of-the-art and relevant background.

An SLA is typically defined based on availability.Customers expect that the services they acquire willbe always available. Thus, providers usually make clearand reliable claims here. The consensus in the indus-try is that cloud computing providers generally havesolutions to manage availability. Response time guar-antees, on the other hand, are harder to guarantee [4].These types of obligations are more carefully phrasedor fully ignored. A quote to illustrate this is ”We areputting a lot of thought into how we can offer pre-dictable, reliable and specific performance metrics toa customer that we can then [build an] SLA around,”[C. Drumgoole, vice president global operations, Ver-izon Terremark, 2013]. Thus, we specifically focus onperformance, although our solution is in principle ap-plicable to availability as well. From a providers per-spective, the question is how to initially configure andlater scale VMs and other resources for a service suchthat the QoS (specifically response time) is guaranteedand, if additionally possible, cost is optimised. Froman infrastructure perspective, memory, storage, net-work conditions and CPU utilisation impact on QoSsuch as performance and availability significantly. Weconsider data storage size, network throughput andCPU utilization as representatives of data, networkand computation characteristics. Common definitions,e.g., throughput as the rate of successful message deliv-ery over a communication channel or bandwidth, shallbe assumed. Figure 1 illustrates in a simple examplethat values of the three resource configuration factorscan be linked to the respective measured performance.It shows how the performance differs depending on theinfrastructure parameters, but not necessarily in a waythat would be easy to determine and predict.

Figure 1: Measured QoS mappings: Infrastructureto Service ([CPU, network, storage] → Performance).

Zhang et al. Page 3 of 20

The first step is to monitor and record these inputmetrics in system-level resource utilisation logs. Thesecond step is pattern extraction. From repeated ser-vice invocations records (the logs), an association toservice QoS values based on prediction techniques canbe made. An observation based on experiments that wemade is that most services have relatively fixed serviceworkload patterns (SWP):

• The patterns are defined here as ranges of storage,network and CPU processing characteristics thatreflect stable, small acceptable variations of a QoSvalue.• Generally, service QoS keeps steady under a SWP,

allowing this stable mapping between infrastruc-ture input and QoS to be used further.

If we can extract SWPs from service logs or the re-spective resource usage logs (based on pattern min-ing), the associated service quality can be based onusage information using pattern matching and predic-tion techniques. Even if there is no or insufficient usageinformation for a given service, quality values can becalculated using log information of other similar ser-vices, e.g., through collaborative filtering. These twosteps can be carried out offline. The next, first onlinestep is pattern matching, where dynamically a patternis matched in the matrix against performance require-ments. The final step is the (if necessary dynamic) con-figuration of the infrastructure in the cloud.

The hypothesis behind our workload pattern-drivenresource configuration based on required service-levelquality is the stability of variations of quality underSWPs. We assume SLA definitions to establish QoS re-quirements and the charged costs for a service to be de-cided between provider and consumer. Service-specificworkload pattern are mined and constructed whichconsiders environmental characteristics of a service (ina VM) deployment. We experimentally demonstratethat the hybrid technique for QoS-to-SWP mappings(based on pattern matching and collaborative filteringfor missing information) enhances accuracy and com-putational performance and makes it applicable in thecloud. In contrast, traditional prediction techniquescan be computationally expensive and unsuitable forthe cloud.

We limit this investigation to services and infrastruc-ture with some reasonably deterministic behaviour,e.g., classical business or technology management ap-plications. However, we deal with larger substantialuncertainties arising from the infrastructure and plat-form environment in which the services are executed.We also focus on the variability of log data, noise thatoccurs and uncertainties arising from multi-cloud en-vironments.

3 WORKLOAD PATTERNSThe core concept of our solution is a Service WorkloadPattern (SWP). A SWP is a group of service invoca-tion characteristics reflected by the utilised resources.In a SWP, the value of workload characteristics is arange. The QoS is meant to be steady under a SWP.We describe a SWP M as a triple of ranges low to high(as low ∼ high ranges):

M = [ CPUlow ∼ CPUhigh,

Storagelow ∼ Storagehigh,Networklow ∼ Networkhigh ]

(1)

CPU, Storage and Network are common server com-putation, memory and network characteristics that wehave chosen for this investigation [7]. The CPU timeused and utilisation rates are typically the most influ-ential factor. The RAM (memory) utilisation rate andstorage access are equally important. In distributedapplications, network parameters such as bandwidth,latency and throughput have an influence of serviceQoS (we consider here the latter). Note that in prin-ciple, the specific characteristics could be varied.

We initially work with monitored absolute figures forCPU time used, stored data size and network through-put. Later on, we also convert this into normalised util-isation rates with respect to the allocated resources.

3.1 SWP Pattern Mining and ConstructionWe assume service-level execution quality logs in theformat< q1, . . . , qn > and infrastructure-level resourcemonitoring logs < ri1, . . . , r

im > with i = 1, . . . , j for j

different quality aspects (e.g., storage, network, serverCPU utilisation) of the past invocations of the servicesunder consideration, as illustrated in Figure 1. Foreach service, the resource metrics and the associatedmeasured performance are recorded. The challenge isnow to determine or mine combinations of value rangesfor input parameters r that result in stable, i.e., onlyslightly varying performances. The solution is a SWPextraction process that constructs the workload pat-terns.• A SWP is composed of storage, network and

computation characteristics. For these, we takethroughput, data size and CPU utilization as rep-resentatives, respectively.

• We consider the execution (response) time as therepresentative of QoS here.

An execution log records the input data size and exe-cution QoS; a monitoring log records the network sta-tus and Web server status. We reorganize these twologs to find the SWP under which QoS keeps steady.

Zhang et al. Page 4 of 20

Our SWP mining algorithm is based on a generic algo-rithm type, DBSCAN (density-based spatial clusteringof applications with noise). DBSCAN [10] analyses thedensity of data and allocates the data into a cluster ifthe spatial density is greater than a threshold. TheDBSCAN algorithm has two parameters: the thresh-old ε and the minimum number of points MinPts. Twopoints can be in the same cluster if their distance isless than ε. The minimum number of points is alsogiven. We also need a parameter MaxTimeRange, themax time range of a cluster. We expect the range oftime is a cluster that can be steady and that has a sizelimit. When the cluster is too large, e.g., if the rangeexceeds a threshold, the cluster construction shouldbe stopped. The main steps are given in the followingalgorithm 1:

• Select any object p from the object set S and findthe objects set D in which the object is density-reachable from object p with respect to ε andMinPts.• Choose another object without cluster and repeat

the first step

The pattern extraction algorithm is presented in Al-gorithm 1.

We give higher precedence to more recent log entries.Exponential smoothing can be applied to any discreteset of sequential observations xi. Let the sequence ofobservations begin at time t = 0, then simple expo-nential smoothing is defined as follows:

y0 = x0

yt = αxt + (1− α)yt−1, t > 0, 0 < α < 1(2)

The choice of α is important. Close to 1 has nosmoothing effect and gives higher weight to recentchanges and as a result the estimate may fluctuatedramatically. Values of α closer to 0 have a bettersmoothing effect and the estimate is less responsiveto recent changes. We can choose a value like 0.8 asthe default, which is relatively high, but reflects themost recent multi-tenancy situation (which can un-dergo short-term changes). We will discuss this sepa-rately later in more detail in Section 4.4.

3.2 Pattern-Quality Matrix

The input value ranges form a pattern that is linkedto the stable performance ranges in a Quality MatrixMS(M,S) based on patterns M and services S. MS as-sociates a service quality QoSP (Si,Mi) (with P stand-ing for performance) of service Si in S under a patternMj in M .

Algorithm 1 SWP Extraction Algorithm based onDBSCAN.Input: Service Usage InforSet (execution + monitoringlog), ε, MinPts, MaxTimeRange.Output: SWP PatternBase, Pattern-QoS information,PatternQoS.1: for (Infori < CPU,DataSize, ThroughPut, Performance >∈ InforSet do

2: if Infori does not belong to any exist cluster then3: Pj=newPattern(Infori) {create a new pattern with

Infori as seed}4:5: Add( Pj , PatternBase )6: InforSet = InforSet− Infori7: SimInfor = SimilarInfor(InforSet, Infori, ε)

{SimInfor is the information set which includes allthe similar usage information of Infori. Differencesbetween the information in SimInfor and Inforion the characteristics value except execution time areless than ε. n is the number of information items inSimInfor.}

8: InforSet = InforSet− SimInfor9:

10: if n > MinPts then11: {MinPts is min number of exec info in cluster}12: (S1, S2 . . . , Sm) = Divide( SimInfor ) {Divide

SimInfor into different groups.}13:14: Group S1 includes all information of service s115: for (k = 1;K ≤ m;k ++) do16: for Inforj ∈ Sk do17: if MaxTime−MinTime < MaxTimeRange

then18:19: SimInfor = SimilarInfor(InforSet,

Inforj , time,MinPts, ε) {Search similarinfo of Sk in execution information set. If thenumber of similar information item is less thanMinPts, then the density will turn low andtop the loop.}

20: Sk = Sk + SimInfor21: InforSet = InforSet− SimInfor22: end if23: end for24: PatternCharacteristics(Sk) {Organizes the informa-

tion in the cluster and statistics for the ranges ofcharacteristics completes matrix}

25: end for26: end if27: end if28: end for

Zhang et al. Page 5 of 20

MS =

S1 S2 . . . Sm

M1 q1,1 q1,2 . . . q1,mM2 q2,1 q2,2 . . . q2,m. . . . . . . . . . . . . . .Ml ql,1 ql,2 . . . ql,m

(3)

Figure 1 at the beginning illustrated monitoring andexecution logs that capture low-level metrics (CPU,storage, network) and the related service response timeperformance. SWPs Mi then result from the log min-ing process using clustering.

The following is a set of patterns M1 to M3 for thegiven example in Figure 1:

CPU Strg NetwM1 = [ 2.1 ∼ 2.5 , 10 ∼ 11 , 0.1 ∼ 0.2 ]M2 = [ 2.0 ∼ 2.2 , 8 ∼ 30 , 0.2 ∼ 0.4 ]M3 = [ 1.2 ∼ 2.0 , 10 ∼ 20 , 0.1 ∼ 0.5 ]

(4)

For those patterns, we can construct the followingquality matrix MS:

S1 S2 S3

M1 0.2 ∼ 0.5s 1.5 ∼ 1.8sM2 1.4 ∼ 1.8s 1.1 ∼ 1.5sM3 1.5 ∼ 2.1s

(5)

The matrix MS above shows the QoS in this examplefor performance information of all services sj for allpatterns Mi. The quality qij(1 ≤ j ≤ l, 1 ≤ i ≤ m)is the quality of service sj under pattern Mi with thequality value qij defined as follows:• as φ if the service sj has no invocation history

under pattern mi and• as lowij ∼ highij if the service sj has an invoca-

tion history under mi with range ∼.For a pattern M1 = [0.5 ∼ 0.6, 0.2 ∼ 0.4, 30 ∼ 40]

the CPU utilization rate is 0.5−0.6, storage utilizationis 0.2−0.4 and network throughput is 30−40MB. Thesample matrix illustrates the workload pattern to QoSassociation for services. Empty spaces (undeterminednull values) for a service indicate lacking data. In thatcase, a prediction based on similar services is necessary,for which we use collaborative filtering.

3.3 Pattern MatchingFor monitored resource metrics (CPU, storage, net-work), we need to determine which of these influencesperformance the most. This determines the matchedpattern. Let the usage information of service s be asequence xk of data storage D, network throughput

N and CPU utilisation C values mapped to responsetime R for k = 1, . . . , n:

[< x1D, x1N , x

1C >, x1R]

. . .

[< xkD, xkN , x

kC >, xkR]

. . .

[< xnD, xnN , x

nC >, xnR]

(6)

We use response time performance in the log as thereference sequence xR(k), k = 1, . . . , n, and other con-figuration metrics as comparative sequences. Then, wecalculate the association degree of other characteris-tics with response time and use characteristics of aninvocation as standard and carry out a normalizationof the other metrics. Thus, the normalized usage in-formation y is (schematically) for any invocation k:

[< yD(xkD), yN (xkN ), yC(xkC) >, 1] (7)

Response time is the reference sequence x0(k), k =1, . . . , n and the other infrastructure characteristicsare the comparative sequences. We calculate the as-sociate degree of the other three characteristics withresponse time. We take one invocation as standardand then normalise the others. The reference (i = 0)and comparison sequences (i = 1, . . . , 3) are han-dled dimensionless. We obtain standardised sequencesyi(k), i = 0, 1, . . . , 3; k = 1, . . . , n, see Figure 2.

Figure 2: Normalised QoS mappings: Infrastructureto Service ([CPU, network, storage] → Performance).

Next, we calculate absolute differences for the tableabove using

∆oi = |yo(k)− yi(k)| (8)

With Oi here ranging over the quality aspects, weget O1 = D, O2 = N and O3 = C. The resultingabsolute difference sequence is for our 3 quality aspectsthe following:

∆01 = (0, y01(1), . . . , y01(n)),

∆02 = (0, y01(2), . . . , y02(n)),

∆03 = (0, y01(3), . . . , y03(n)),

(9)

Zhang et al. Page 6 of 20

In the next step, we determine a correlation coef-ficient between reference and comparative sequence(using here the correlation coefficient of the Gray rel-evance):

ζoi(k) =∆min + ρ∆max

∆oi(k) + ρ∆max(10)

Here ∆oi(k) = |y0(k)−yik| is the absolute difference,∆min = mini mink∆0i(k) is the minimum differencebetween two poles, ∆max = maxi maxk∆0i(k) is themaximum difference, ρ ∈ (0, 1) is a distinguishing fac-tor. Afterwards, we use the formula

roi =1

n

n∑i=1

ζo1(k) (11)

to calculate the correlation degree between the metrics.Then, we sort the metrics based on the correlationdegree. If r0 is the largest, it has the greatest impacton response time and will be matched prior to othersin the pattern matching process.

Clouds are shared multi-user environments whereusers and applications require different quality set-tings. A multi-valued utility function U can be addedrepresenting the user weighting of a vector Q of qualityattributes q ∈ Q for a matrix mi ∈M as a weighting.This utility function allows a user to customise thematching with user-specific weightings:

Up,m,q : rng(Qm,q)→ [0, 1] (12)

The overall utility can be defined, taking into ac-count the importance or severity of the quality at-tributes ωi for each q ∈ Q :

Um =∑∀qi∈Q

ωiUm,qi(Qm,qi(MS))

Um,q =∑∀p∈p

Up,m,q∑i

ωi = 1, ωi ≥ 0

(13)

Finally, the pattern that optimizes the overall con-figuration utility is determined through the maximumutility calculated as:

maxm∈MsUm (14)

Note, that the utility is based on the three qualityconcerns, but could potentially be extended to takeother factors into account. Furthermore, costs for theinfrastructure can also be taken into account to deter-mine the best configuration. We will define an addi-tional cost function in the cloud configuration Section5 below.

4 QUALITY PATTERN-DRIVENCONFIGURATION DETERMINATION

The QoS-SWP matrix is the tool to determine SLArequirements-compliant SWPs as workload specifica-tions for the resource configuration and re-configurationand re-scaling. For quality-driven configuration, thequestion is: for a given service Si and a given perfor-mance requirement QoSP , what are suitable SWPs toconfigure the execution environment? The executionenvironment is assumed to be a VM image configu-ration with storage and network services samples arediscussed in Section 5. We first determine a few config-uration determination use cases to get a comprehensivepicture where the pattern technique can be used andthen discuss the core solutions in turn.

4.1 Use CasesIn general, there is a possibly empty set of patternsMS(si) for each service si, i.e., some services have us-age information, others have no usage information inthe matrix itself. Consider the sample matrix from theprevious section. Three use cases emerge that indicatehow the matrix can be used:• Configuration Determination – Existing Patterns:

For a service s with monitoring history: Since s1has an invocation history for various patterns fora requested response time of 0.45s, we can returnthis set of patterns including M1 and M3.

• Configuration Determination – Non-existing Pat-terns: For a given service s without history: Sinces2 has no invocation history for a required re-sponse time of 2s, we can utilise collaborative fil-tering for the prediction of settings, i.e., use sim-ilar services to determine patterns for the givenservice [11] [12].

• Configuration Test – For a given triple of SWPvalues and a service s: If a given s1 has an invo-cation history for a required response time of 2sand we have a given workload configuration, wecan test the compliance of the configuration witha pattern using the matrix.

4.2 Pattern-based Configuration DeterminationIf patterns exist that satisfy the performance require-ments, then these are returned as candidate configu-rations. In the next step, a cost-oriented ranking of

Zhang et al. Page 7 of 20

these can be done. We use quality level to cost map-pings that will be explained in Section 5 below. If nopatterns exist for a particular service (which reflectsthe second use case above), then these can be deter-mined by prediction through collaborative filtering, see[7].

QoS Prediction Process. For any service s, if there isinformation of sv under pattern mi, then calculate thesimilarity between other services sj and sv. We canget the k neighbouring services of service sj througha similarity calculation. The set of these k services isS = s′1, s

′2, . . . , s

′k. We fill the null (empty) QoS val-

ues for the target invocation using the information inthis set. Using the information in S, we then calcu-late the similarity of mi with other patterns that havethe information for target service sj . We choose themost similar k′ patterns of mi, and use the informa-tion across the k′ patterns and S to predict the qualityof service sj .

Service Similarity Computation. If there is no infor-mation of sj in a pattern Mi, we need to predict theresponse time qi,j for sj . Firstly, we calculate the simi-larity of sj and services which have information withinpattern Mi ranges. For a service sv in Ii where Ii isthe set of services that have usage information withinpattern Mi we calculate the similarity of sj and sv.We need to consider the impact of configuration en-vironment differences, i.e., redefine common similaritydefinitions. Mvj is the set of workload patterns whichhave the usage information of services sv and sj .

simS(sv, sj) =∑mc∈Mvjj(qc,v − qv)(qc,j − qj)√∑

mc∈Mvj(qc,v − qv)2√∑

mc∈Mvj(qc,j − qj)2(15)

Here, qv is the average quality value for service svand qj is the respective value for sj . From this, we canobtain all similarities between sj and others serviceswhich have usage information within pattern mi. Themore similar the service is to sj , the more valuable itsdata is.

Predicting Missing Data. Missing or unreliable datacan have a negative impact on prediction accuracy.In [13], we considered noise up to 10% to be accept-able. In order to deal with uncertainty beyond this,we calculate the similarity between two services andget the k neighbouring services. Then, we establishthe k-neighbour matrix Nsim, see Equation (16), andcomplete the missing data. Nsim shows the usage in-formation of the k neighbour services of sj under allpatterns, reducing the data space to k columns.

Nsim =

sj s′1 . . . s′kM1 s1j s1,1 . . . s1,m. . . . . . . . . . . . . . .Mi si,j si,1 . . . si,m. . . . . . . . . . . . . . .Ml sl,j sl,1 . . . sl,m

(16)

Empty spaces are filled, if required. Then, we addsi,p as the data of service sp under pattern mi:

si,p = q′p +

∑n∈S′ simn,p × (q′i,n − q′n)∑

n∈S′(|simn,p|)(17)

Again, q′p is the average quality value of sp, andsimn,p is the similarity between sn and sp. Now ev-ery service s ∈ S′ has usage information within allpattern ranges in mi.

Calculating Pattern Similarity and Prediction. Thereis QoS information of k neighbouring services of sj inmatrix Nsim. Some of them are prediction values. Wecan calculate the similarity of pattern mi and otherpatterns using the correction cosine similarity method:

simM (mi,mj) =∑sk∈S(t′i,k − t′i)(t′j,k − t′j)√∑

sk∈S(t′i,k − t′i)2√∑

sk∈S(t′j,k − t′j)2(18)

After determining the pattern similarity, the data ofpatterns with low similarity are removed from Nsim,the set of the first k patterns. The data of these pat-terns are retained for prediction. If pi,j is the data tobe predicted as the usage data of service sj withinpattern mi, it can be calculated.

pi,j = q′i +

∑n∈M ′ simn,i × (q′n,j − q′n)∑

n∈M ′(|simn,i|)(19)

The average QoS of data related to pattern mi is q′iand simn,j is the similarity between patterns mn andmp.

4.3 Pattern-based Configuration TestingWe can use the pattern-QoS matrix to test a standardor any known resource configuration in SWP format(i.e., three concrete values rather than value ranges forthe infrastructure aspects) for instance in the situa-tion outlined above for a service si for which its per-formance is uncertain. This can also be done instead

Zhang et al. Page 8 of 20

of collaborative filtering, as indicated above, if the re-turned set of patterns is empty and a candidate con-figuration is available. Then, the matrix can be usedto determine the respective QoS values, i.e., to pre-dict quality such as performance in our case throughtesting as well.

This situation shall be supported by an algorithmthat matches candidate configurations with storedworkload patterns based on their expected servicequality. The algorithm takes into account whether ornot possibly matching workload patterns exist.

Algorithm 2 Matching Candidate ConfigurationsInput: Service Usage Information of a Service.Output: Metrics Sorted by Correlation Degree.1: Match [ candidate configuration Config = < y1, y2, y3 > of

target service si ] with [characteristics (ranges) < low1 ∼high1, low2 ∼ high2, low3 ∼ high3 > ] of stored patternsMi.

2: if there is a pattern that can be matched then3: return it4: else5: use Gray relevance analysis (Formula (3.7)) to match a

pattern6: end if7: Let the matched pattern be mi

8: Search information about matched pattern mi in matrix M9: if there is QoS information of service si in mi then

10: return it as expected QoS for candidate configuration11: else12: if no related QoS information exists then13: predict QoS by collaborative filtering (4.1)(4.4)14: end if15: end if16: Return

If no patterns exist, existing candidate configurationscan be tested – to enable always a solution, at leastone default configuration should be provided. Alterna-tively, similar services can be considered; these can bedetermined through collaborative filtering and then wewould start again.

4.4 Variability and SmoothingWhile the solution above extracts patterns for a givenlog, some practical considerations shall be taken intoaccount. In log data corresponding to resource work-load measurements, we can usually observe high vari-ations over time, which makes predictions over par-ticularly small time-scales unreliable. The workloadcontains often many short-duration spikes. In orderto alleviate the disturbance caused by these, we canuse smoothing techniques before actually applying thepattern mining based on DBSCAN.

We use a time-series forecasting technique [14] tobetter estimate the workload at some future point intime.• We use double exponential smoothing for the

workload aspect because it can be used to smoothen

the inputs and predict trends in log data. Thismodel takes the number of requests for applicationservices at runtime into account before predictingthe future workload. This is suitable for the threeworkload types CPU, storage and network.

• On the other hand, for estimating response-time,we use single exponential smoothing because forthe oscillatory response-time, we do not need topredict the trend but a smoothed value.

Both the exponential smoothing techniques weightthe history of the workload data by a series of ex-ponentially decreasing factors. An exponential factorclose to one gives a large weight to the first samplesand rapidly makes old samples negligible.

The specific formula for single exponential smooth-ing is:

st = θxt + (1− θ)st−1, t > 0; s0 = x0 (20)

Similarly, the formula for double exponential smooth-ing is:

st = βxt + (1− β)(st−1 + bt−1)bt = γ(st − st−1) + (1− γ)bt−1; 0 < θ, β, γ < 1

(21)

where xt the raw data sequence and st is the out-put of the techniques and θ, β, γ are the smoothingfactors. Note, the number of data points here dependson the control loop intervals and the frequency of theperformance counter retrievals in each loop.

Smoothing helps with significant, but irregular vari-ations of workload. It also help adding more weightto more recent events. In both, cases the benefit ismore reliable prediction of quality at the service level.Single exponential smoothing is specifically used forthese irregular short-term variations of workload re-sulting in oscillatory response-time. These spikes canoften not be dealt with properly in the cloud due tonon-instantaneous provisioning times (this context issometimes referred to as the throttling pattern) [13].Smoothing can build in an adequate response (or non-response) of the system allowing to ignore a certainsituation.

However, smoothing is a kind of noise added to thesystem and we need to address the robustness of theprediction against noise later on in the evaluation.

5 PATTERN-DRIVEN RESOURCECONFIGURATION

This section shall illustrate how the approach can beused in a cloud setting for resource (VM) configura-tion, costing and auto-scaling. Predefined configura-tions for VMs and other resources offered by providers

Zhang et al. Page 9 of 20

as part of standard SLAs could be the following thatrelate to the CPU, storage and network utilisation cri-teria < CPU, Storage,Network > we used in Section3 and 4 for the SWPs.

Gold, Silver and Bronze in Figure 3 are names forthe different service tiers based on different configura-tions that are commonly used in industry. We can addpricing for Pay-as-you-Go (PAYG) and monthly sub-scription fees to the above scheme to take cost-basedselection of configurations into account, see Figure 4.

Figure 3: VM Configuration.

Figure 4: VM Charging Scheme.

We define below a cost function C : Config → Costto formalise such a table. The categories based on theresource workload configurations can now be alignedby the provider with QoS values that are promisedin the SLA – here with response time and availabil-ity guarantees filled in the Configuration − Qualitymatrix CQ:

Gold 0.75 99.99Silver 1.0 99.9Bronze 1.5 99

(22)

In general, the Configuration-Quality matrix is de-fined by

CQ = [cij ] with i : configuration category

and j : quality attribute(23)

A selection function σ determines suitable workloadpatterns Mi for a given quality target q as defined inthe Configuration-Quality matrix and a service sj :

σ(q, s) = {Mi ∈M |MS(q, sj) ∈ qij} (24)

From this set of workload patterns M1, . . . ,Mn, wedetermine the most optimal one in terms of resourceutilisation. For minimum and maximum utilisationthresholds minU and maxU that are derived as inter-val boundaries from the pattern mining process, thebest pattern is selected based on a minimum devia-tion of pattern ranges Mi(q) across all quality factors(based on the overall mean value) from the thresholdaverage value, defined as the mean average deviation(where x indicates the mean value for any expressionx):

mini

√∑(maxU −minU −Mi(q))2 (25)

The thresholds can be set at 60% and 80% of thepattern range averages to achieve a good utilisationwith some remaining capacity for spikes [15].

Based on the best selected SWP Mi with the givenkey metrics, a VM image can be configured accordinglyin terms of CPU, storage and network parameters anddeployed with the service in question. If several SWPsapply to meet performance requirements, then costscan be considered to select the cheapest offer (if thecost in the table reflects in some way the real cost ofprovisioned resources and not only charged costs)

σ(q, s)Cost = miniC(σ(q, s)) (26)

for a cost function C that maps a pattern in σ(q, s) toits cost value. The cost function can create a ranking ofotherwise equally suitable patterns or configurations.

The service-based framework presented in Sections 3and 4 was here applied to the cloud context by link-ing it to standard configuration and payment models.Specific challenges arose from the cloud context thatwe have addressed are:• Standard cloud payment models allow an explicit

costing, which we took into account here throughthe cost function. Essentially, the cost functioncan be used to generate a ranked list of candidatepatterns for a required QoS value in terms of theoperational cost. In [16], we have demonstratedthat different performance result, but also costsvary for a given configuration pattern.

• Cloud solutions are subject to (dynamic) configu-rations, generally both at IaaS and PaaS level.While our configuration here is geared towardstypical IaaS attributes, our implementation workwith Microsoft Azure (see Section 7) also demon-strates the possibility and benefit of PaaS-levelconfiguration. In [16], we have discussed differentPaaS-level storage configurations and their costand performance implications.

Zhang et al. Page 10 of 20

• User-driven scalability mechanisms such as Cloud-Scale or CloudWatch or the AWS Autoscalingtypically work on scaling rules defined on thegranularity of VMs added/removed. Our solutionis based on similar metrics, e.g., GB for storage ornetwork bandwidth, i.e., further automates thesesolutions.

We have briefly mentioned uncertainties that arisefrom cloud environments in Section 2. While we haveneglected this aspect here so far, [13] presents an ap-proach that adds uncertainty handling on top of pre-diction for VM (re-)configuration, which we will adaptfor our problem setting in the next section. Uncertain-ties arise for instance from incomplete or potentiallyuntrusted monitoring data or from varying needs andinterpretations of stakeholders regarding quality as-pects. The approach in [13] adds a fuzzy logic pro-cessing on top of a prediction approach.

6 MANAGING UNCERTAINTYThe QoS-SWP matrix captures predictions in the formof mappings or prediction rules M ×S → Q where forSWPs in M and services in S as antecedents, we as-sociate quality values Q as consequents of those rules.Due to the possibly varying number of recorded map-pings and consequently different patterns for differentservices, there is some uncertainty regarding the re-liability of predictions across the service base causedby possibly insufficient and unrepresentative log data.Sources of uncertainty include:• at early stages, the amount of log data available

is insufficient,• different monitoring tools provide different log

data quality,• temporary variations in workload might render

prediction unreliable.In order to address these uncertainties, we propose a

fuzzy logic-based approach to capture uncertain infor-mation, fuzzify this and infer here quality predictionsbased on the fuzzified recorded log data.

6.1 Fuzzification of Rule MappingsThe hypothesis that motivates our framework is theobservation that utilisation rates for the resources inquestion (CPU, storage, network) are often subopti-mal. We can look at the Q values in the matrix andfind patterns M i that run at 60-80% rate for a givenq ∈ Q. As we might find a number of possible patterns,a degree of uncertainty emerges as either a number ofcandidate patterns for a single service or even for a setof similar services might exist.

Our proposal if to fuzzify all pattern mapings Mi×Sand calculate an optimum quality in a fuzzified spacebefore defuzzifying the result as a concrete pattern

with a predicted quality value. Note that this patternsmight not be in the pattern-quality matrix, i.e., doesnot necessarily reflect actual observations. More con-cretely, we fuzzify M1 ×M2 ×M3 → Q mappings forone quality range q ∈ Q with q = qmin ∼ qmax anda given service s ∈ S. Here, each M i refers to one ofthe three aspects CPU, storage and network, result-ing in a set of m patterns with ranges for the threeaspects that all map to the same quality range q. Thepatterns are the different patterns for a service (or aclass of similar services) that predict a quality rangeq = qmin ∼ qmax:

M11..3 → q

M21..3 → q· · ·

Mm1..3 → q

(27)

This set will be fuzzified, resulting in joint, fuzzy rep-resentation of the merged patterns.

This technique is based on a fuzzy logic approach touncertainty that we developed for cloud auto-scaling[13], but adopted here to service quality prediction.The approach is differently applied here in that insteadof different linguistic concept definitions used in use-defined auto-scaling rules, we have different (impre-cise) workload ranges. Furthermore, instead of differ-ent scaling actions as rule antecedents, we look at dif-fering performance values (moderately varying). Whilein [13], we addressed uncertainty regarding human-specified scalability rules, this is uncertainty regardingthe monitoring systems and their data.

Here, we obtain data from different, but similar ser-vices via the collaborative filtering technique. Differ-ent associated performances (unpredictable variationscaused my factors not considered in the calculation)are determined. Through the variable Y , we will lateron capture predictions such that M1...3I → q ∈ Q Thiscan take into account a targeted 60-80% utilisationrate optimum.

In order to simplify the presentation, we only developthe fuzzy logic approach for the CPU processor loadas core compute resource, linked to performance atthe service level. However, specific characteristics thatwould distinguish CPU, storage and network do notplay any role and, therefore, selecting just one here forsimplification does not restrict the solution.

6.2 Fuzzy Logic Definitions for UncertaintyFuzzy logic is a suitable tool to reflect the uncertaintythat is incorporated in the different patterns for a ser-vice, particularly since these rely on collaborative fil-tering based on similarity with related services.

Zhang et al. Page 11 of 20

Fuzzy logic distinguishes different ways to representuncertainty. A type-1 (T1) fuzzy set associates as afunction a possibility value of some attribute, resultingin a single value depending on the input. A type-2 (T2)fuzzy set is an extension of a type-1 (T1) fuzzy set [17].At a specific value x′ (cf. Figure 5), there is an intervalinstead of a crisp value for each input.

Figure 5: A type-2 fuzzy set based possibilitydistribution.

This leads to the definition of a three dimensionalmembership function (MF), a T2 MF, which charac-terizes a T2 fuzzy set (FS). Note that the core fuzzylogic definitions here are standard definitions in fuzzytheory from the literature such as [18, 19]. Figure 5visualises the definitions.

A T2 FS, denoted by R , is characterized by a T2MF µR(x, u) with

R = {((x, u), µR(x, u)|∀x ∈ X,∀u ∈ Jx, µR(x, u) ≤ 1}(28)

When these values have the same weight, it leads todefinition of an interval type-2 fuzzy set (IT2 FS):

If µR(x, u) = 1 , R is an interval T2 FS, also abbre-viated as an IT2 FS.

Therefore, the membership function MF of a IT2 FScan be fully specified by the two boundary T1 MFs.The area between the two MFs (the grey area in Figure5) characterizes the uncertainty.

The uncertainty in the membership function of anIT2-FS, R, is called footprint of uncertainty (FOU) ofR. Thus, we define:

FOU(R) =⋃x∈X

Jx = {(x, u)|∀x ∈ X,∀u ∈ Jx} (29)

The upper membership function (UMF) and thelower membership function (LMF) of R are two T1-MFs µR(x) and µ

R(x), respectively, that define the

boundary of the FOU.

An embedded fuzzy set Re is a T1 FS that is locatedinside the FOU of R.

6.3 Qualification of Infrastructure Inputs

We need to apply the definitions above to constructIT2 membership functions for our mappings, but firstthe add labels to ranges in order to work with qualifiedrather than quantified ranges. This is common in cloudresource configuration.

Assume the following pattern-to-quality mappingwith normalised input values in [0..100] for CPU, stor-age and network parameters, respectively:

M11..3 = [40− 54], [20− 26], [30− 37]→ q = [1.1− 1.4]

M21..3 = [20− 33], [26− 32], [80− 86]→ q = [0.9− 1.3]

M31..3 = [16− 23], [65− 72], [50− 55]→ q = [1.2− 1.6]

M41..3 = [48− 56], [71− 78], [82− 89]→ q = [1.5− 1.9]

The ranges above reflect observed CPU utilisationranges under which the performance is relatively sta-ble. Ranges with similar performance are clustered. Wenow use a qualified representation of the individualrange clusters. For the CPU processor load, we use fiveindividual labels Very Low, Low, Medium, High andVery High. Thus, for the above mapping, we can mapthe CPU ranges [16-23] to the Very Low label, [20-33]to Low and [40-54] and [48-56] to Medium. The labelsthemselves are suggested by experts and are commonlyused in the configuration of cloud infrastructure solu-tions. While this qualification into concept labels istechnically not necessary, these labels help to visualiseand communicate the pattern-based prediction to thecloud users.

Figure 6 shows the qualified patterns. For each la-bel, the black bar denotes the variation of the meansof each of the pattern value ranges as given by theexperts, e.g., the means of [40-54] and [48-56] for theMedium label. The light grey bars denote the overallvariation of ranges for each label.

The qualified categorisation will in the next step beused to define fuzzy membership functions that fuzzifythe infrastructure input captured in the patterns.

Zhang et al. Page 12 of 20

Figure 6: Qualification of Pattern Ranges beforeFuzzification.

6.4 Defining Membership FunctionsInfrastructure monitoring tools measure the input val-ues for the prediction solution (CPU, storage, net-work) as infrastructure parameters and performanceas service-level data). Their conversion to fuzzy valuesis realized by MFs. In this section, we show how to de-rive appropriate IT2 MFs based on the data extractedfrom the infrastructure and service monitoring logs.We follow the guidelines in [20] in order to constructthe functions. To simplify the investigation, we focuson CPU load and performance only.

As illustrated in Figure 5, we used triangular ortrapezoidal MFs to represent different CPU loadranges:• For each normalized CPU range [CPUmin ∼CPUmax], we define one type-2 MF as in Figure7 for five ranges.• We order these and label them accordingly, e.g.,

with VL to VH labels representing Very Low toLow, Medium, High and Very High CPU loads.

Figure 7: IT2 Membership Functions for CPU load.

Let a and b be the mean values of the interval end-points CPUmin and CPUmax of the labelled rangeswith standard deviations σa and σb, respectively (seeFigure 6). For instance, for the Low, Medium and Highlabel, the corresponding triangular T1 MF is then con-structed by connecting left l, middle m and right r asfollows:

l = (a−σa, 0),m = ((a+b)/2, 1), r = (b+σb, 0) (30)

Accordingly, for Very Low and Very High proces-sor load ranges that border 0 or 100% on the nor-malized scale, the associated trapezoidal MFs can beconstructed by connecting points as follows:

(a− σa, 0), (a, 1)(n, 1), (b+ σb, 0) (31)

The labels above refer to utilisation rates of re-sources, with a maximum value being 100. So, irre-spective of the concrete system, the labels are assignedto normalised utilisation ranges.

As indicated by the standard deviations in Figure 6,there are uncertainties associated with the ends andthe locations of the MFs. As an example, a triangularT1 MF might be defined follows:

l′ = (a− 0.3× σs, 0)m = ((a+ b)/2, 1)r′ = (b+ 0.4× σb, 0)

(32)

These uncertainties cannot be captured by T1 fuzzyMFs. In IT2 MFs on the other hand, the footprint ofuncertainty (FOU) can be captured by the upper andlower MFs (UMF and LMF) for each range, see Figure8. A blurring parameter 0 ≤ α ≤ 1 can determine theFOU (see Figure 6).

Here, we use α = 0.5. Parameter α = 0 reduces IT2MFs to a T1 MFs, while parameter α = 1 results infuzzy sets with the widest possible FOUs.

Figure 8: Locations of the main points of IT2 MFs.

Zhang et al. Page 13 of 20

6.5 The Fuzzy Prediction Process

After constructing the IT2 fuzzy sets with the MFs andthe set of rules for the infrastructure load patterns, theprediction can then start to determine service qualityestimations. The fuzzy prediction technique proceedsin a number of steps as follows (which we will explainin more detail afterwards):

1 The inputs comprising the workload are firstfuzzified.

2 Then, the fuzzified input activates an inferencemechanism to produce output IT2 FSs.

3 Decisions made through the inference mechanismare represented in the form of fuzzy values, whichcannot be directly used as prediction results. Theoutputs are then processed by a type-reducer,which combines the output sets and subsequentlycalculates the center-of-set.

4 The type-reduced FSs are T1 fuzzy sets that needto be defuzzified in the last step to determine thepredicted quality value q.

5 This value q is then passed back to the user as thepredicted quality.

In the first step, we must specify how the numericinputs ui ∈ Ui for the CPU utilisation rates are con-verted to fuzzy sets (a process that called ”fuzzifica-tion” [18]) so that they can be used by the FLS. Here,we use singletons:

µR1=

{1 x = ui

0 otherwise(33)

For the defuzzification step, we use the construct ofa centroid [21]. The centroid of a IT2 FS R is the unionof the centroids of all its embedded T1 fuzzy sets Re:

CR ≡⋃∀Re

c(Re) = [cl(R), cr(R)] (34)

For the type-reducing step we use the center-of-setsconstruct [36]. The center-of-set (cos) type reductionis calculated as follows:

Ycos =⋃

f l∈F l,yl∈CGl

ΣNl=1f

l × y1

ΣNl=1f

l= [yl, yr] (35)

where f l ∈ F l is the firing degree of mapping rule land yl ∈ CGl is the centroid of the IT2 FS Gl. The

centroids cl(R), cr(R) and yl, yr are calculated usingthe KM algorithm from [21].

6.6 Fuzzy Reasoning for Quality Prediction

Our quality mappings that encode the SWPs are ina multi-input single-output format. Because the logrecords of different services may not be similar, manypatterns may be conflicting, i.e., we might have rulemappings with the same antecedent, but different con-sequent values. In this step, mappings with the sameif-part are combined into a single rule. For each map-ping in the patterns retrieved from the logs, we get thefollowing for a sample rule R1:

IF (x1 is Fl1) and ... and (xp is F

lp),

THEN (y is y(tlu))

(36)

where tlu is the index for the result. In order to com-bine these conflicting mappings, we used the averageof all the responses for each mapping and use this asthe centroid of the mapping consequent. Note, thatthe mapping consequents are IT2 FSs. However, whenthe type reduction is used, these IT2 FSs are replacedby their centroids. This means that we represent themas intervals [yn, yn] or crisp values when yn = yn. Thisleads to rules with the following form for an arbitraryrule Rl:

IF the CPU workload (x1) is Fi1 ,

AND the network load (x2) is Gi2 ,

AND the storage utilisation (x3) is Hi3 ,THEN qi is the predicted value.

where C is the value of associated consequent, i.e., herea quality value for performance in ms, and wl

u is theweight associated with the u-th consequent of the lthmapping. Therefore, each qi can be computed.

In a concrete example, we now illustrate the detailsof the prediction process. Assume that the normal-ized values regarding the three input parameters arex1 = 40, x2 = 50 and x3 = 55, respectively – see thesolid lines for the workload in Figure 9 as a sample fac-tor. For x1 = 40, two IT2 FSs for the processor work-load ranges F2 = Low and F2 = Medium with the de-grees [0.3797, 0.5954] and [0.3844, 0.5434] are fired[1].Similarly, for x2 = 50 , three IT2 FSs for the per-formance ranges G3 = Medium, G4 = Slow, andG5 = V ery Slow with the firing degrees [0, 0.1749],[0.9377, 0.9568] and [0, 0.2212] are fired. For x3 = 55similar values would emerge. Intuitively, the lower and

[1]The term ’fired’ is commonly used for rules, generallyassuming the consequents of the rules being actionsthat are fired. We have kept the term, but it applieshere only to mappings to quality values.

Zhang et al. Page 14 of 20

upper values of the intervals can be computed by find-ing the y-intercept of the solid lines in the Figure, re-spectively, with the LMF and the UMF of the crossedFSs. As a result, six pattern mappings might resultthat are fired (here a sample is given):

M8 : (F2, G3, H1),M9 : (F2, G4, H4),M10 : (F2, G5, H4),M13 : (F3, G3, H2),M14 : (F3, G4, H1),M15 : (F3, G5, H2)

(37)

Figure 9: IT2 MFs of for CPU workload ranges.

The firing intervals are computed using the meet op-eration [17]. For instance, the firing interval associatedto the pattern M9 for the CPU attribute is:

m9 = µM9

2

(x′1)⊗µG9

4

(x′2) = 0.3797×0.9377 = 0.3560

(38)

The output can be obtained using the center-of-set:

Yl(40, 50, 55) =[yl(40, 50, 55), yr(40, 50, 55)] =[0.9296, 1.1809]

(39)

The defuzzified output can be calculated as follows:

Y (40, 50, 55) =0.9296 + 1.1809

2= 1.0553 (40)

Thus, we can finally calculate the predicted qualityY (x1, x2, x3) for all the possible normalized values ofthe input infrastructure parameters (i.e., x1 ∈ [0, 100],x2 ∈ [0, 100], x3 ∈ [0, 100] for CPU, storage and net-work).

7 EVALUATIONOur proposed solution builds on a core pattern-basedquality prediction solution that can be used for cloudconfiguration in an environment where there is uncer-tainty about concerns such as correctness or complete-ness. We have implemented an experimental test en-vironment to evaluate the accuracy, efficiency and ro-bustness of the solution in the context of uncertaintyand dynamic processing needs.

7.1 Implementation Architecture and EvaluationSettings

The implementation of the prediction and configura-tion technique covers different parts – off-line and on-line components:• The pattern determination and the construction

of the patterns-quality matrix is done off-linebased on monitoring logs. The matrix is neededfor dynamic configuration and can be updated asrequired in the cloud system. For the prediction,the accuracy is central. As the construction is off-line, performance overhead for the cloud environ-ment is, as we will demonstrate, negligible.

• The actual prediction through accessing the ma-trix is done in a dynamic cloud setting as part of ascaling engine that combines prediction and con-figuration. Here the acceptable performance over-head for the prediction needs to be demonstrated.

• Uncertainties and noise are additional problemsthat arise in heterogeneous, cross-organisationalcloud architectures. Robustness in the presence ofthese challenges needs to be demonstrated.

For both the accuracy and performance concern, weuse a standard prediction solution, collaborative fil-tering (CF) as the benchmark, which is widely usedand analysed in terms of these properties, cf. [11] [12]or [5] [6]. We have implemented a simulation envi-ronment with a workload generator to evaluate accu-racy of the prediction approach and the performanceof the prediction-based configuration. Test data is de-rived from sources such as [22] where quality met-rics (response time, throughput) are collected for 5825Web services with 339 users. We selected 100 applica-tion services from three different categories, each cate-gory either sensitive to data size, network throughputor CPU utilization. Figure 10 describes the testbedwith the monitoring and SWP extraction solution forWeb services. The primary concern is the accuracy ofthe pattern extraction and pattern-based predictionof performance for deployed services. Furthermore, asdynamic reconfiguration, i.e., auto-scaling, is an aim,also performance needs to be looked at.

We have tested our scalability management in Mi-crosoft Azure. We have implemented a range of stan-dard applications, including an online shopping appli-cation, a services management solution and a video

Zhang et al. Page 15 of 20

processing feature to determine the quality metricsfor different service and infrastructure configurationtypes. For the first two, we used the Azure Diagnos-tics CSF to collect monitoring data (Figure 10). Wealso created an additional simulation environment togather a reliable dataset without interference from un-controllable cloud factors such as the network. Con-crete applications systems we investigated are the fol-lowing: a single-cloud storage solutions for online shop-ping applications [16] and a multi-cloud HADR solu-tion (high-availability disaster recovery system [23].This work has resulted in a record of configura-tion/workload data combined with performance data– as for instance shown in [16] where Figs. 3-6 captureresponse time for different service types and Figs. 7-8show infrastructure concerns such as CPU and stor-age aspects. In that particular case, 4100 test runswere conducted, using a 3-service application with be-tween 25 and 200 clients requesting services. AzureCSF telemetry was used to obtain monitoring data.This data was then looked at to identify our workloadpatterns (CPU, Storage, Network) → Performance.

7.2 Cloud Application Workload PatternsA number of different application workload patternsare described in the literature [8, 13]. These includestatic, periodic, one-in-a-lifetime, unpredictable, con-tinuously changing as in [8], or slowly varying, quicklyvarying, dual phase, steep tri-phase, large variationand big spike [13]. We need to make sure that thisexperimental evaluation does indeed cover the appli-cation workload pattern in order to guarantee generalapplicability. We followed [13] and induced these sixpatterns in our experiments. Based on the successfulpattern coverage, we can conclude that the techniqueis applicable for a range of the most common situa-tions.

We have implemented our prediction mechanism indifferent platforms. Work we described in [13] dealswith how to implement this in an auto-scaling solutionsuch as Amazon AWS where Amazon-monitored work-load and performance auto-scaling metrics are con-sidered together with a prediction of anticipated be-haviour to configure the compute capabilities.

7.3 AccuracyReliable performance guarantees based on configura-tion parameters is the key aim – real performanceneeds to match the expected or promised one for theprovider to fulfill the SLA obligations. Accuracy in vir-tualisation environments is specifically challenging [24]due to resource contentions because of the layered ar-chitecture, shared resources and distribution.

Accuracy of prediction is measured in terms of devia-tion from the real behaviour. The metric we use here is

based on the mean absolute error (MAE) between pre-diction (SLA imposed) and real response time, whichis the normal choice to measure prediction accuracy.Different characteristics of QoS have different ranges.Consequently, we use NMAE (the Normalized MeanAbsolute Error) instead of the MAE. The smaller theNMAE, the more accurate is the prediction. We com-pare our solution with similar methods based on tradi-tional prediction methods in terms of matrix density.This covers different situations from situations wherelittle is known about the services (density is low) andsituations where there is a reliable base of historic datafor pattern extraction and prediction (high density).

Figure 11: Accuracy Evaluation.

In earlier work, we included average-based predica-tions and classical collaborative filtering (CF) in acomparison with our own hybrid method (MCF) ofmatrix-based matching and collaborative filtering [7].The NMAE of k = 15 and k = 18 (higher or lowerks are not interesting as lower values are statisticallynot significant and higher ones only show a stabili-sation of the trend) shows an accuracy improvementfor our solution compared to standard prediction tech-niques, even without utility function and exponentialsmoothing, see Figure 11. For this evaluation here, wealso include time series (TS) into the comparison. Forthe evaluation, we considered some noisy data whichcannot be in any pattern. We also removed invocationdata and then predicted it using the CF, MCF andalso the TS time series method from [25].

We can observe that an increase of the dataset sizeimproves the accuracy significantly. In all cases, ourMCF approach outperforms the other ones by result-ing in less prediction errors.

7.4 Efficiency Overhead (Runtime)For automated service management in the contextof cloud auto-configuration and auto-scaling we needsufficient performance of the extraction and matchingapproach itself. To be tested in this context are theperformance of three components:

Zhang et al. Page 16 of 20

Figure 10: Evaluation Architecture.

1 SWP Extraction from Logs (Matrix Determina-tion)

2 Configuration-Pattern Matching (Existing Pat-terns)

3 Collaborative Filtering (Non-Existing Patterns)For cases 1 and 2, we determined 150 workload pat-

terns from 2400 usage recordings. We tested the algo-rithm on a range of different datasets extracted froma number of documented benchmarks and test cases.Com-pared to other work based on the TS and CFsolutions, the matrix for collaborative computation isreduced from 2400*100 to 150*100, which reduces ex-ecution time significantly by the factor 16. For case 3,only when a matched pattern provides no informationfor a target service, the calculation for collaborationprediction is required, see Figure 12, where we com-pare prediction performance with (MCF) as describedin Sections 3 and 4 (using Algorithms 1 and 2) andwithout (CF) the pattern-based matrix utilisation.

7.5 Robustness against Noise and UncertaintyExponential smoothing causes noise and results insome unavoidable errors. We need to demonstrate thatour prediction solution is resilient against input noises,one of which is the estimation error through smooth-ing.

We experimentally evaluated the robustness againstnoise. We observed that the worst estimation errorhappens for large variation and quickly varying pat-terns and is less than 10% of the actual workload.Based on this, we injected a white noise to the input

Figure 12: Performance Evaluation.

measurement data (i.e., x1) with an amplitude of 10%.We ran RMSE (root-mean-square error – measure ofthe differences between values predicted by a model oran estimator and the values actually observed) mea-surements for each levels of blurring, and for each mea-surement, we used 10,000 data items as input. We useddifferent blurring values. Two observations emerged:• Firstly, the error of control output produced by

the prediction technique is less than 0.1 for theblurring levels.

• Secondly, the error of control output is decreasingwhen we configured the technique with a higherblurring.

A higher blurring as part of the smoothing leads toa bigger FOU in the uncertainty management, where

Zhang et al. Page 17 of 20

FOU is a representative for the supporting levels of un-certainty. Therefore, the designer should make a choicein terms of the level of uncertainty that is acceptable.Note that in some circumstances an overly wide FOUresults in a performance degradation. However, we canstate that these observations provide enough evidencethat robustness against input noise is maintained. Us-ing IT2 FLS in the uncertainty management actuallyalleviates here the impact of noise.

7.6 Threats to ValiditySome threats to the validity of the observations hereexist. The usefulness of the approach relies on the qual-ity of the workload patterns. While the existence ofthe patterns can be guaranteed by simply defining theindividual ranges in the broadest possible way, thusresulting in some over-arching default patterns, theirusefulness will only emerge if they denote sufficientlynarrow ranges to allow the resources to be utilised inan equally narrow band. For instance, a band of 60 to80% is aimed at for processor loads.

While this has emerged to be possible for traditionalsoftware services that are computationally intensive orare typical text and static content-driven applications,multi-media type applications for instance with differ-ent network consumption patterns require more inves-tigation.

7.7 Summary of EvaluationThus, to conclude the evaluation, the computationaleffort for the dynamic prediction is decreased to alarge extent due to the already partially filled matrix.As already explained, the performance of the patternextraction and matrix construction (DBSCAN basedclustering and collaborative filtering) can be computa-tionally expensive, but can be done offline and only thematrix-based access (as demonstrated in the perfor-mance figure above) impacts on the runtime overheadfor the configuration. However, as the results show,our methods overhead increases only slowly even if thedata size increases substantially. Consequently, the so-lution in this setting is no more intrusive than a re-active rule-based scalability solution such as AmazonAWS Auto Scaling that would also follow our proposedarchitecture.

We specifically looked at the accuracy and robust-ness of the solution. The accuracy of the core solu-tion based on the matrix is better than a traditionalcollaborative filtering approach. However, in cloud en-vironments, other concerns needed to be addressedin addition. In order to address uncertainty due todifferent and possible incomplete and unreliable logdata, we added smoothing and fuzzification. We havedemonstrated that these features improve the robust-ness against external influence factors.

8 RELATED WORKQoS-based service selection in general has been widelycovered. There are three main categories of prediction-based approaches for selection.• The first one covers statistical and other mathe-

matical methods, which are often adopted for sim-plicity [1] [2] [3] [26] [27]. Others, e.g., in the con-text of performance modeling use mathematicalmodels such as queues.

• The second category selects services based on userfeedback and reputation [28] [29]. It can avoid ma-licious feedback, but does not consider the im-pact of SLA requirements and the environmentand cannot customise prediction for users.

• The third category is based on collaborative filter-ing [11][12] [30], which is a widely adopted recom-mendation method [31] [32] [33], e.g., [32] summa-rizes the application of collaborative filtering indifferent types of media recommendation. Here,we combine collaborative filtering with serviceworkload patterns, user requirements and SLAobligations and preferences. This considers differ-ent user preferences and makes prediction person-alized, while maintaining good performance re-sults.

To demonstrate that our solution is an advancementcompared to existing work on prediction accuracy, wehad singled out two approaches for categories 1 and 3for the evaluation above.

8.1 General Prediction ApproachesSome works integrate user preferences and user charac-teristics into QoS prediction [11] [12] [30] [5], e.g. [11][12] propose prediction algorithms based on collabo-rative filtering. They calculate the similarity betweenusers by their usage data and predict QoS based onuser similarity. This method avoids the influence ofthe environment factor on prediction. However, eventhe same user will have different QoS experiences overtime depending on the configuration of the executionenvironment or will work with different input data.Current work generally does not consider user require-ments. Another current limitation of current solutionsis low efficiency as we demonstrated. Our work in [13]is a direction based on fuzzy logic to take user scala-bility preferences into account for a cloud setting.

In [34] [7], pattern approaches are proposed. [34]suggests pattern-based management for cloud config-uration management, but without a detailed solution.[7] is about bottom-up QoS prediction for standardservice-based architectures, while in this paper QoSrequirements are used to predict suitable workload-oriented configurations taking specifically cloud con-cerns into consideration. We added additionally expo-nential smoothing and utility functions and the cost

Zhang et al. Page 18 of 20

analysis here, but draw on some evaluation resultsfrom [7] in comparison to standard statistical meth-ods.

8.2 Cloud Quality Prediction and Scalability

Various model-based predictive approaches are in usefor cloud quality and resource management, includ-ing machine learning techniques for clustering, fuzzyclustering, Kalman filters, or low band filters, usedfor model updates which are then used for optimiza-tion and integrate with controllers Supporting cloudservice management can automatically scale the in-frastructure to meet the user/SLA-specified perfor-mance requirements, even when multiple user applica-tions are running concurrently. Jamshidi et al. [13] dealwith multi-user requirements as part of an uncertaintymanagement approach, which is based on predictionas only exponent smoothing. Ghandi et al. [35] alsoleverage application level metrics and resource usagemetrics to accurately scale infrastructure. They useKalman filtering to automatically learn changing sys-tem parameters and to proactively scale the infrastruc-ture, but have less of a performance gain than throughpatterns in our solution. Another work in this direc-tion is [36], where the solution aims to automaticallyadapt to unpredicted conditions by dynamically up-dating a Kriging behaviour model. These deal withuncertainty concerns that we have excluded. However,an integration of both directions would be beneficial inthe cloud. These approaches can add the uncertaintymanagement solutions required.

8.3 Uncertainty Management in Clouds

Uncertainty is a common problem, particularly in openmulti-layered and distributed environments such as thecloud applications. In [13], fuzzy logic is used to dealwith auto-scaling in cloud infrastructures. We haveadopted some of the principles here, but applied themto quality prediction here rather than resource alloca-tion there. A similar fuzzy controller for resource man-agement has been suggested in [37] based on adaptiveoutput amplification and flexible rule selection. How-ever, this solution is based on T1 FLS and does notaddress uncertainty.

The proposed method in this paper takes full ac-count of user requirements (reflected in SLA obliga-tions for the provider), the network and computationalfactors. It abstracts the service workload pattern tokeep the service QoS steady. When user/SLA require-ments are known, prediction-base configuration can bedone based on matched patterns. This approach is ef-ficient and reduces the computational overhead.

9 CONCLUSIONSWeb or cloud services [38] usually differ with respectto QoS characteristics. Relevant service-level qualitiesare response time, execution cost, reliability, or avail-ability. There are many factors that impact on QoS[39]. They depend not only on the service itself, butalso how it is deployed. Some factors are static, someare run-time static, the others are dynamic. Run-timestatic and dynamic factors like client load, server load,network channel bandwidth or network channel delayare generally uncertain, but can be influenced by suit-able configuration in virtualised environments such asthe cloud. Most factors can be monitored, and their im-pact on service-level quality can be calculated as partof a service management solution. Service managementin cloud environments requires SLAs for individualusers to be managed continuously through dynamicplatform and infrastructure configuration, based onmonitored QoS data.

We provided a solution that links defined SLA obli-gations for the provider in terms of service perfor-mance with lower-level metrics from the infrastructurethat facilitates the provisioning of the service. Our so-lution enables cloud workload patterns to be associ-ated to performance requirements in order to allowthe requirements to be met through appropriate con-figuration. Note, that the investigation was based onthe three quality concerns, but could potentially beextended to take other factors into account.

Performance management is still a problem in thecloud [4] [40]. While availability is generally man-aged and, correspondingly, SLA guarantees are made,reliably guaranteeing performance is not yet solved.Through a mining approach we can extract resourceworkload patterns from past behaviour that match theperformance requirement and allow a reliable predic-tion of a respective configuration for the future.

What our approach signifies is the increased level ofnoise and uncertainty in cloud environments. Uncer-tainty emerges as different user roles are involved inconfiguring and running cloud applications, both onthe consumer and provider side, and furthermore, dataextracted from the cloud is uncertainty regarding itscompleteness and reliability as it might originate fromdifferent tools at different layers, possibly controlledby different providers or users. Noise occurs naturallyin this setting, but is worsened by a need to apply forinstance smoothing to address variability and detecttrends.

There are wider implications that would still needmore work for any solution to be practically viable.We started with three resource concerns, choosing onefor each category. For the fuzzification, we focused onone concern to reduce complexity. Our experiments

Zhang et al. Page 19 of 20

with concrete cloud platforms (Openstack and Azure)also show the need considerable integration work withplatform services for monitoring and resource manage-ment.

In order to further fine-tune the approach in the fu-ture, we could take more infrastructure metrics intoaccount. More specific cloud infrastructure solutionsand more different use cases shall be used on the ex-perimental side to investigate whether different pat-terns emerge either for different resource provisioningenvironments or for different application domains andconsumer customisations [41].

Competing interests

The authors declare that they have no competing interests.

Author’s contributions

LX and YZ have worked on the collaborative filtering solution presented in

Sections 3 and 4. LX (supervised by CP) has mapped this to the cloud

computing context (Sections 2 and 5). PJ is the central contributor of

Section 6 (work that has been supervised by CP). CP is responsible for

Introduction and Conclusions. All authors have contributed to the

evaluation. All authors read and approved the final manuscript.

Acknowledgements

This research has been supported by the Fundamental Research Funds for

the Central Universities of China (grant N130317005), by the National

Natural Science Foundation of China (grant 61402090), and the Irish

Centre for Cloud Computing and Commerce IC4, an Irish national

Technology Centre funded by Enterprise Ireland, the Irish Industrial

Development Authority, and by Science Foundation Ireland (International

Strategic Cooperation Award Grant Number SFI/13/ISCA/2845).

Author details1Software College, Northeastern University, Shenyang, China. 2IC4 /

School of Computing, Dublin City University, Dublin, Ireland.

References1. Cardoso, J., Miller, J., Sheth, A., Arnold, J.: Quality of service for

workflows and web service processes. Journal of Web Semantics 1,

281–308 (2004)

2. Kritikos, K., Plexousakis, D.: Requirements for qos-based web service

description and discovery. IEEE Transactions on Services Computing

2(4), 320–337 (2009)

3. Ye, Z., Bouguettaya, A., Zhou, X.: Qos-aware cloud service

composition based on economic models. In: Proceedings of the 10th

International Conference on Service-Oriented Computing. ICSOC’12,

pp. 111–126. Springer, Berlin, Heidelberg (2012)

4. Chaudhuri, S.: What next?: A half-dozen data management research

goals for big data and the cloud. In: Proceedings of the 31st

Symposium on Principles of Database Systems. PODS ’12, pp. 1–4.

ACM, New York, NY, USA (2012)

5. Zhang, L., Zhang, B., Liu, Y., Gao, Y., Zhu, Z.-L.: A web service qos

prediction approach based on collaborative filtering. In: Services

Computing Conference (APSCC), 2010 IEEE Asia-Pacific, pp. 725–731

(2010)

6. Zhang, L., Zhang, B., Na, J., Huang, L., Zhang, M.: An approach for

web service qos prediction based on service using information. In:

Service Sciences (ICSS), 2010 International Conference On, pp.

324–328 (2010)

7. Zhang, L., Zhang, B., Pahl, C., Xu, L., Zhu, Z.: Personalized quality

prediction for dynamic service management based on invocation

patterns. 11th International Conference on Service Oriented

Computing ICSOC 2013 (2013)

8. Fehling, C., Leymann, F., Retter, R., Schupeck, W., Arbitter, P.:

Cloud Computing Patterns - Fundamentals to Design, Build, and

Manage Cloud Applications. Springer, Vienna, Austria (2014)

9. Zhang, L., Zhang, Y., Jamshidi, P., Xu, L., Pahl, C.: Workload

patterns for quality-driven dynamic cloud service configuration and

auto-scaling. In: Utility and Cloud Computing (UCC), 2014

IEEE/ACM 7th International Conference On, pp. 156–165 (2014)

10. Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A density-based algorithm

for discovering clusters in large spatial databases with noise. In: Proc.

of 2nd International Conference on Knowledge Discovery And, pp.

226–231 (1996)

11. Shao, L., Zhang, J., Wei, Y., Zhao, J., Xie, B., Mei, H.: Personalized

qos prediction forweb services via collaborative filtering. In: Web

Services, 2007. ICWS 2007. IEEE International Conference On, pp.

439–446 (2007)

12. Zheng, Z., Ma, H., Lyu, M.R., King, I.: Qos-aware web service

recommendation by collaborative filtering. IEEE Trans. Serv. Comput.

4(2), 140–152 (2011)

13. Jamshidi, P., Ahmad, A., Pahl, C.: Autonomic resource provisioning for

cloud-based software. In: Proceedings of the 9th International

Symposium on Software Engineering for Adaptive and Self-Managing

Systems. SEAMS 2014, pp. 95–104. ACM, New York, NY, USA

(2014)

14. Kalekar, P.S.: Time series forecasting using holt-winters exponential

smoothing. Technical report, Kanwal Rekhi School of Information

Technology (2004)

15. Zhao, H., Li, X.: Resource Management in Utility and Cloud

Computing. Springer, New York, USA (2013). Springer Briefs in

Computer Science

16. Xiong, H., Fowley, F., Pahl, C., Moran, N.: Scalable architectures for

platform-as-a-service clouds: Performance and cost analysis. 8th

European Conference on Software Architecture, ECSA 2014 (2014)

17. Mendel, J.M.: Type-2 fuzzy sets and systems: An overview [corrected

reprint]. Computational Intelligence Magazine, IEEE 2(2), 20–29

(2007)

18. Mendel, J.M., John, R.I., Liu, F.: Interval type-2 fuzzy logic systems

made simple. Fuzzy Systems, IEEE Transactions on 14(6), 808–821

(2006)

19. Wu, D.: On the fundamental differences between interval type-2 and

type-1 fuzzy logic controllers. Fuzzy Systems, IEEE Transactions on

20(5), 832–848 (2012)

20. Liang, Q., Karnik, N.N., Mendel, J.M.: Connection admission control

in atm networks using survey-based type-2 fuzzy logic systems.

Systems, Man, and Cybernetics, Part C: Applications and Reviews,

IEEE Transactions on 30(3), 329–339 (2000)

21. Karnik, N.N., Mendel, J.M.: Centroid of a type-2 fuzzy set.

Information Sciences 132(14), 195–220 (2001)

22. Zheng, Z., Zhang, Y., Lyu, M.R.: Distributed qos evaluation for

real-world web services. In: Web Services (ICWS), 2010 IEEE

International Conference On, pp. 83–90 (2010)

23. Xiong, H., Fowley, F., Pahl, C.: An architecture pattern for multi-cloud

high availability and disaster recovery. In: Workshop on Federated

Cloud Networking FedCloudNet’2015 (2015)

24. Gmach, D., Rolia, J., Cherkasova, L., Kemper, A.: Workload analysis

and demand prediction of enterprise data center applications. In:

Proceedings of the 10th International Symposium on Workload

Characterization. IISWC ’07, pp. 171–180. IEEE Computer Society,

Washington, DC, USA (2007)

25. Cavallo, B., Di, P., Canfora, G.: An empirical comparison of methods

to support qos-aware service selection. In: the 2nd International

Workshop on Principles of Engineering Service-Oriented Systems

(2010)

26. Alrifai, M., Skoutas, D., Risse, T.: Selecting skyline services for

qos-based web service composition. In: Proceedings of the 19th

International Conference on World Wide Web. WWW ’10, pp. 11–20.

ACM, New York, NY, USA (2010)

27. Zeng, L., Benatallah, B., H.H. Ngu, A., Dumas, M., Kalagnanam, J.,

Chang, H.: Qos-aware middleware for web services composition. IEEE

Trans. Softw. Eng. 30(5), 311–327 (2004)

28. Vu, L.-H., Hauswirth, M., Aberer, K.: Qos-based service selection and

ranking with trust and reputation management. In: Proceedings of the

2005 Confederated International Conference on On the Move to

Meaningful Internet Systems. OTM’05, pp. 466–483. Springer, Berlin,

Heidelberg (2005)

Zhang et al. Page 20 of 20

29. Li, Y., Zhou, M.-H., Li, R.-C., Cao, D.-G., Mei, H.: Service selection

approach considering the trustworthiness of qos data. Journal of

Software, 2620–2627 (2008)

30. Wu, G., Wei, J., Qiao, X., Li, L.: A bayesian network based qos

assessment model for web services. In: Services Computing, 2007. SCC

2007. IEEE International Conference On, pp. 498–505 (2007)

31. Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based

collaborative filtering recommendation algorithms. In: Proceedings of

the 10th International Conference on World Wide Web. WWW ’01,

pp. 285–295. ACM, New York, NY, USA (2001)

32. Zeng, C., Xing, C.-x., Zhou, L.-z.: A survey of personalization

technology. Journal of Software 13(10), 1952–1961 (2002)

33. Xu, H.L., Wu, X.: Comparison study of internet recommendation

system. Journal of Software, 350–362 (2009)

34. Srinivas, D.: A patterns/workload-based approach to the cloud. DIMS

Lightning Talk, IBM (2013)

35. Gandhi, A., Harchol-Balter, M., Raghunathan, R., Kozuch, M.A.:

Autoscale: Dynamic, robust capacity management for multi-tier data

centers. Transactions on Computers Systems 30(4), 14–11426 (2012)

36. Gambi, A., Toffetti Carughi, G., Pautasso, C., Pezze, M.: Kriging

controllers for cloud applications. IEEE Internet Computing 17(4),

40–47 (2013)

37. Rao, J., Wei, Y., Gong, J., Xu, C.-Z.: Dynaqos: Model-free self-tuning

fuzzy control of virtualized resources for qos provisioning. In: Quality

of Service (IWQoS), 2011 IEEE 19th International Workshop On, pp.

1–9 (2011)

38. Pahl, C., Xiong, H.: Migration to paas clouds c migration process and

architectural concerns. 7th International Symposium on Maintenance

and Evolution of Service-Oriented and Cloud-Based Systems

(MESOCA 2013) (2013)

39. Lelli, F., Maron, G., Orlando, S.: Client side estimation of a remote

service execution. In: Proceedings of the 2007 15th International

Symposium on Modeling, Analysis, and Simulation of Computer and

Telecommunication Systems. MASCOTS ’07, pp. 295–302. IEEE

Computer Society, Washington, DC, USA (2007)

40. Jamshidi, P., Ahmad, A., Pahl, C.: Cloud migration research: A

systematic review. IEEE Transactions on Cloud Computing 1(2),

142–157 (2013)

41. Wang, M., Bandara, K.Y., Pahl, C.: Process as a service distributed

multi-tenant policy-based process runtime governance. In: Proceedings

of the 2010 IEEE International Conference on Services Computing.

SCC ’10, pp. 578–585. IEEE Computer Society, Washington, DC, USA

(2010)