Systematically Deriving Quality Metrics for CloudComputing Systems

Matthias Becker* Sebastian Lehrig Steffen Becker

{matthias.becker|sebastian.lehrig|steffen.becker}@{*uni-paderborn|informatik.tu-chemnitz}.de*Heinz Nixdorf Institute Software Engineering Chair

University of Paderborn, Paderborn, Germany Chemnitz University of Technology, Chemnitz, Germany

ABSTRACTIn cloud computing, software architects develop systems forvirtually unlimited resources that cloud providers accounton a pay-per-use basis. Elasticity management systems pro-vision these resources autonomously to deal with changingworkload. Such changing workloads call for new objectivemetrics allowing architects to quantify quality propertieslike scalability, elasticity, and efficiency, e.g., for require-ments/SLO engineering and software design analysis. Inliterature, initial metrics for these properties have been pro-posed. However, current metrics lack a systematic deriva-tion and assume knowledge of implementation details likeresource handling. Therefore, these metrics are inapplicablewhere such knowledge is unavailable.

To cope with these lacks, this short paper derives metricsfor scalability, elasticity, and efficiency properties of cloudcomputing systems using the goal question metric (GQM)method. Our derivation uses a running example that out-lines characteristics of cloud computing systems. Eventually,this example allows us to set up a systematic GQM plan andto derive an initial set of six new metrics. We particularlyshow that our GQM plan allows to classify existing metrics.

Categories and Subject DescriptorsD.2.8 [Software Engineering]: MetricsPerformance mea-sures; D.2.11 [Software Engineering]: Software Architec-turesLanguages

Keywordscloud computing; scalability; elasticity; efficiency; metric;SLO; analysis; GQM

1. INTRODUCTIONIn cloud computing, software architects develop appli-

cations on top of compute environments being offered bycloud providers. For these applications, the amount of of-fered resources is virtually unlimited while elasticity man-agement systems provision resources autonomously to dealwith changing workloads. Furthermore, providers bill pro-

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.ICPE15, Jan. 31Feb. 4, 2015, Austin, Texas, USA.Copyright c 2015 ACM 978-1-4503-3248-4/15/01 ...$15.00.http://dx.doi.org/10.1145/2668930.2688043

visioned resources on a per-use basis [1]. As a consequenceof these characteristics, architects want their applicationsto use as few resources as possible in order to save moneywhile still maintaining the quality requirements of the sys-tem. Quality properties that focus directly on these aspectsare scalability, elasticity, and efficiency [9].

These quality properties need to be quantified for require-ments engineering and software design analysis by means ofsuitable metrics. For instance, cloud consumers and cloudproviders need to negotiate service level objectives (SLOs),i.e., metrics and associated thresholds [7]. Such SLOs haveto consider characteristics like changing workloads (howfast can an application adapt to a higher workload?) andpay-per-use pricing (how expensive is serving an additionalconsumer?). However, no established metrics for require-ments/SLO engineering and software design analysis exist.Current metrics assume knowledge of implementation de-tails as they focus on the application at run-time [9] and lacka systematic derivation making such limitations explicit.

In literature, classical performance-oriented metrics [4]like response time and throughput are insufficient for situa-tions relevant for cloud computing applications. First, theydo not take changing workloads into account, e.g., metricsto describe reaction times to system adaptations are miss-ing. Second, the degree to which systems match resourcedemands to changing workloads cannot be quantified. Morerecent work [9] proposes metrics for such characteristics thatassume knowledge of implementation details like resourcehandling. These metrics are inapplicable when such knowl-edge is unavailable, e.g., in early design phases and for SLOs.

To cope with these lacks, we systematically derive an ini-tial set of scalability, elasticity, and efficiency metrics usingthe goal question metric (GQM) method [15]. First, we il-lustrate the characteristics of cloud-aware applications usinga running example scenario and, subsequently, derive met-ric candidates from this scenario. Second, by generalizingour metric candidates, we develop a first set of six metrics.Third, we show that our GQM plan allows to systematicallyclassify existing metrics and makes their limitations explicit.

This paper contributes our systematic GQM plan, includ-ing classifications of our six and related work metrics.

This short research paper is organized as follows. Sec-tion 2 gives definitions of the considered quality propertiesand Sec. 3 introduced the running example system and itsrequirements. The system is implemented as cloud-awareapplication. In Sec. 4, we systematically derive a set of newmetrics using the GQM method. In Sec. 5, we put our met-rics in relation to existing metric proposals in literature andclassify these metrics using our GQM plan in Sec. 6. Finally,Sec. 7 concludes the paper and highlights future work.

169

2. DEFINITIONSBecause we distinguish scalability, elasticity, and efficiency

throughout our paper, we first give the definitions of theseproperties based on the work of Herbst et al. [9]: Scalabil-ity is the ability of the system to sustain increasing work-loads by making use of additional resources [9]; Elasticityis the degree to which a system is able to adapt to workloadchanges by provisioning and deprovisioning resources in anautonomous manner, such that at each point in time theavailable resources match the current demand as closely aspossible [9]; and Efficiency expresses the amount of re-sources consumed for processing a given amount of work [9].We use these definitions throughout this paper to derivemetrics for each quality property. In the next section, weexemplify these definitions based on our running example.

3. MOTIVATING EXAMPLEAs an example scenario, we consider a simplified online

bookshop. An enterprise assigns a software architect to de-sign this shop, given the following requirements:

Rfct: Functionality In the shop, customers shall be ableto browse and order books.

Rscale: Scalability The enterprise expects an initial cus-tomer arrival rate of 100 customers per minute. It fur-ther expects that this rate will grow by 12% in the firstyear, i.e., increase to 112 customers per minute. In thelong run, the shop shall therefore be able to handle thisincreased load without violating other requirements.

Relast: Elasticity The enterprise expects that the contextfor the bookshop repeatedly changes over time. For ex-ample, it expects that books sell better around Christ-mas while they sell worse around the holiday seasonin summer. Therefore, the system shall proactivelyadapt to anticipated changes of the context, i.e., main-tain a response time of 3 seconds or less as well aspossible. For non-anticipated changes of the context,e.g., peak workloads, the system shall re-establish aresponse time of 3 seconds or less within 10 minutesonce a requirement violation is detected.

Reff : Efficiency The costs for operating the bookshop shallonly increase (decrease) by $0.01 per hour when thenumber of customers concurrently using the shop in-creases (decreases) by 1. In other words, the marginalcost of the enterprise for serving an additional cus-tomer shall be $0.01.

Requirements Rscale, Relast, and Reff are typical reasonsto operate a system in an elastic cloud computing environ-ment [9], i.e., an environment that autonomously provisionsthe required amount of resources to cope with contextualchanges. Thus, the software architect designs the shop asa 3-layer Software as a Service (SaaS) application operatingin a rented Infrastructure as a Service (IaaS) cloud comput-ing environment that provides replicable virtual servers (seeFig. 1). The three layers involve the typical layers of webapplications: presentation, application, and data layer.

The architect designs each SaaS layer such that it canconsume a higher/lower quantity of IaaS services to sustainchanging workloads. Technically, he ensures that each SaaSlayer can be replicated and load-balanced over additionalIaaS virtual servers (scale-out) or be removed again (scale-in). Therefore, properties like scalability (Rscale), elasticity

Replicable

Data Layer

Replicable

Application LayerReplicable

Presentation LayerSaaS Environ-

ment

IaaS Environ-

ment

Book Management

Book Database

Replicable Virtual Servers

Enterprise /IaaS Consumer

IaaS Provider

SLOs

Book Shop Frontend

...

Figure 1: Overview of the simplified online bookshop.

(Relast), and efficiency (Reff ) of the bookshop are inher-ently coupled with corresponding properties of the underly-ing IaaS environment. The enterprise (IaaS consumer) andthe IaaS provider have, thus, to agree on measurements andthresholds for these properties. Typically, consumer andprovider achieve an agreement based on negotiated SLOs asexemplified by Rscale, Relast, and Reff . However, currentlythere is a lack of agreed-on metrics for scalability, elastic-ity, and efficiency in the context of cloud computing. Thislack results in too few SLOs or SLOs that cannot be quanti-fied and checked by cloud consumers. In consequence, thereis the risk that requirements Rscale, Relast, and Reff can-not be fulfilled due to a non-anticipated (and contractuallyuncheckable) behavior of the underlying IaaS environment.

4. DERIVING METRICS WITH GQMWe derive our metrics with the goal question metric (GQM)

method [15] in a systematic top-down fashion by first defin-ing the goal to analyze cloud computing system designs, e.g.,for design analysis or SLO specification (conceptual level).Second, we formulate questions that help achieving the goal(operational level). Finally, we identify metrics that allowus to answer the questions (quantitative level).

4.1 GoalTable 1 shows the goal in form of the GQM goal definition

template. The metrics we want to define in this paper shallhelp to analyze (purpose) the scalability, elasticity, and ef-ficiency (issue) of cloud computing system designs (object)from the viewpoint of a software architect (viewpoint).

Note that the derived metrics are not necessarily general-izable or applicable for different objects or viewpoints. Forexample, cloud computing vendors may perceive the effi-ciency of cloud system software different than software ar-chitects and, thus, need different metrics.

Table 1: Goal definition according to GQM plan

Purpose AnalyzeIssue the scalability, elasticity, and efficiency ofObject cloud computing system designsViewpoint from a software architects viewpoint

4.2 QuestionsNext, we define questions that help achieving our defined

goal. We define questions for each quality property sepa-rately. All questions are indicator questions for the accord-ing quality property as defined in Sec. 2.

170

Questions for Scalability.According to the definition, scalability is an ability, i.e.,

a system is either scalable or it is not. Hence, we definequestions whether a system is able to fulfill its requirementsunder increasing workload. Moreover, the increase rate ofthe workload can be a relevant context factor.

Q1scale Does the system fulfill its requirements when theworkload increases (from workload WLX to WLY )?

Q2scale Does the system fulfill its requirements when theworkload increases with rate R (from workload WLXto WLY within time t)?

Questions for Elasticity.Elasticity is, according to the definition, the degree to

which a system is able to autonomously adapt to workloadchanges. Thus, we define questions that consider the timeit takes for the system to adapt.

Q3elast How often does the system violate its requirementsunder workload WLX in time period t?

Q4elast From a point the system violates its requirements,how long does it take before the system recovers to astate in which its requirements are met again?

Questions for Efficiency.Efficiency relates the amount of demanded resources to

the amount of work requested. Hence, we formulate ques-tions that ask for this relation.

Q5eff How close to the actual resource demand can thesystem align the resource provisioning?

Q6eff What is the amount of resources, autonomously pro-visioned by the system, for a given workload WLX?

4.3 MetricsIn this section, we summarize general requirements for

metrics and concrete requirements for cloud computing sys-tem metrics. We then derive our metrics that answer thequestions from Sec. 4.2 in two steps. In the first step, wederive exemplary metrics (EM) for scalability, elasticity, andefficiency for the example system in Sec. 3. In the secondstep, we generalize these metrics to answer the questions forarbitrary cloud computing systems.

4.3.1 Requirements for Derived MetricsThe metrics we define have to meet four typical charac-

teristics of metrics [7] in order to be applicable by softwarearchitects: (1) quantifiability, (2) repeatability, (3) compa-rability, and (4) easy obtainability. Additionally, we requireour metrics to be (5) context dependent to reflect the contextdependency of cloud computing systems. Figure 2 showsparts of the context that impact the quality of a cloud com-puting system as a feature diagram. This context coversthe systems workload, i.e., work and load, and deployment,i.e., replication of components, processor speed, memorysize, network speed, etc. This entire context is subject tochange at run-time in a typical cloud computing environ-ment. Hence, metrics need to have a context parameter toenable the comparison of implementations in different con-texts. For example, system SA may have less SLO violations

per minute with workload WLX (context) but more SLO vi-olations per minute with workload WLY (different context)compared to system SB .

4.3.2 Derived Metrics from Example ScenarioIn this section, we define exemplary metrics (EM) that

can be used to answer the questions from Sec. 4.2. In thissection, we restrict these metrics to evaluate whether therequirements for the example scenario in Sec. 3 are fulfilled.In Sec. 4.3.3, we generalize these metrics for arbitrary cloudcomputing systems.

The first requirement Rfct in our example scenario is ageneral requirement that defines the basic functionality ofthe bookshop. However, metrics for functional requirementsare out of the scope of this paper.

Exemplary Scalability Metrics.Rscale specifies the systems required scalability, i.e., the

systems ability to make use of additional resources at in-creasing workloads. Hence, this requirement is dependenton the context, e.g., the load specified as arrival rates. Thebook enterprise has to specify arrival rates, e.g., by estimat-ing current sales, sale trends, and seasonal sale variability.

The bookshops software architect checks whether the de-signed system fulfills requirement Rscale by evaluating ques-tions Q1scale and Q2scale for this design. A metric EMscale,defined as the maximum workload the system can handlewithout violating requirement Rscale, answers question Q1scale.For example, the software architect can measure this metricby predicting the performance for the bookshop design withthe increased workload of 12% as expected by the book en-terprise (predictions can, e.g., be conducted using queuingnetworks, c.f. [14]).

The enterprise does not specify at which rate the work-load increases, e.g., linearly or exponentially. Therefore, toanswer question Q2scale, the bookshops software architectneeds a metric EMrate, defined as the rate a system canscale up to a certain maximum workload. For example, thesoftware architect can measure this metric by predicting theperformance of the bookshop design under increasing work-load, e.g., a linear increasing workload of one additional cus-tomer per month. Afterwards, the architect can discuss thequantified requirement with the enterprise.

Exemplary Elasticity Metrics.Relast specifies the bookshops required elasticity, i.e., the

degree to which the bookshop is able to adapt to workloadchanges by autonomously provisioning and deprovisioningcloud resources. In general, these workload changes (con-text) can be either anticipated or impossible to anticipate.As described in Sec. 3, the enterprise can anticipate vari-ability of the bookshops customers demand, i.e., the enter-prise estimates to sell more books around Christmas than inmid-summer. Other short-term variations of the workloadcannot be anticipated, e.g., peak workloads.

The bookshops software architect can evaluate whetherRelast is fulfilled by answering questions Q3elast and Q4elastfor the bookshop design. The cloud computing system canpotentially cope with both, anticipated and non-anticipated,workload variability. However, considering the fact that re-sources can be available with delay, the system will likelyviolate requirement Relast until the time additionally provi-sioned resources are available. A metric EMviol, defined as

171

Context

Workload

Work

OperationwCalls Inputw(Parameter)

Load

RequestwRate

Deployment

Replication Processor Memory Network

ConcurrentwUsers

Figure 2: Feature diagram showing aspects of context.

number of requirement violations for a given workload, is amean for the software architect to answer question Q3elast.A metric EMadapt, defined as the time between detection ofa requirement violation and the time when the requirementis fulfilled again, e.g., by autonomous resource provisioning,answers question Q4elast. For example, the bookshops ar-chitect can measure both metrics, EMviol and EMadapt, bysimulating the bookshops autonomous behavior using self-adaptive queuing networks [2].

Exemplary Efficiency Metrics.Reff specifies the required efficiency of the bookshop re-

garding its resource consumption. The bookshops autonomousprovisioning and deprovisioning adapts the resource con-sumption to the bookshops workload, i.e, the workload isimportant context here as well. Furthermore, the operationcosts of the bookshop depend on this resource consumptionas the enterprise has to pay for the cloud resources in apay-per-use fashion.

The software architect can evaluate whether the Reff isfulfilled by answering questions Q5eff and Q6eff . A metricEMclose can be defined as the difference between the book-shops amount of provisioned resources for a workload WLXand the minimum amount of resources required to cope withthat workload WLX without violating the stated require-ments. The bookshops software architect can use this met-ric to answers question Q5eff . A metric EMprov, defined asthe amount of resources provisioned for a workload WLX ,answers question Q6eff . Whereas the marginal costs can becalculated directly from the metric EMprov, EMclose sup-ports the software architect to determine whether the re-quired marginal costs are achievable. Again, both metricscan be measured by simulating the bookshops autonomousbehavior under variable workload, for example.

4.3.3 Generalized MetricsWe define initial new metrics that we derived from the

six exemplary metrics (EM) from the previous subsection.For each metric, we define the corresponding quality prop-erty it quantifies, what is being measured, on which con-textual properties the measurement depends, and the scopeand unit of the measurement result. To guarantee objectiv-ity, our metrics rely only on externally observable propertiesof a system, e.g., costs per time. Reproducibility is guaran-teed by specifying the contextual properties on which themetric depends. Thus, context dependency is also guaran-teed. By defining the measurement results scope as ordinalnumbers, we guarantee that the metric reflects a testablequantification of the quality property. Additionally, we pro-

vide a small example for each metric. Table 2 summarizesthe following generalized metrics.

Scalability Metrics.Scalability Range (ScR) Based on EMscale, we define

scaling range as a scalability metric that reflects a cloudcomputing systems ability to achieve its SLOs in a certainworkload range, e.g., a range of request rates. For each singleworkload within this range, the system achieves its SLOs.The workload range is defined as a maximum workload. Forexample, a perfectly scalable system SA can scale up to ainfinite request rate, i.e., ScRSA = req/min. A lessscalable system, for example, scales up to a request rate of112 requests per minute, i.e., ScRSB = 112 req/min.

Scalability Speed (ScS) Based on EMrate, we definescalability speed ScS as a workload range with a maximalchange rate in which a system can scale. This metric is ascalability metric which additionally considers the rate atwhich a system can scale. That is, the metric defines thata system is able to achieve its SLOs at each time when theworkload changes at a maximal changing rate. The rate isdefined by a maximum workload and an increase rate. Forexample, a scalable system can scale up to a request rate of112 req/min with a linear increase rate of 1 additional re-quests per month, i.e., ScSSA = (112 req/min, 1 req/month).

Elasticity Metrics.Mean Time To Quality Repair (MTTQR) We de-

rive the mean time to quality repair metric from EMadapt,the measure how quickly a system can adapt to workloadchanges. It is a measure for elasticity and depends on aworkload delta, i.e., the increase/decrease between two work-loads. MTTQR defines the mean time a system needs to re-establish its SLOs when the workload increases/decreases fora defined workload delta specified as factor (real number).Hence, MTTQR is measured in time units. Since it definesa mean time, MTTQR is specific for a specified time framein which the mean is calculated. For example, with the sameworkload increase factor of 1.2, a perfectly elastic system willadapt itself to the increasing workload within zero time, i.e.,MTTQRSA,1d(1.2 req/s) = 0 min. A less elastic system,e.g., will need a mean time of 10 min (calculated over oneday) to adapt itself to increasing workload after it detects theworkload increase, i.e., MTTQRSB ,1d(1.2 req/s) = 10 min.

Number of SLO violations (NSLOV ) The numberof SLO violations in a defined time interval is derived fromEMviol. This metric measures elasticity of a system. Theworkload delta is specified as a factor (real number) as well.

172

Table 2: Derived quality property metrics for cloud computing systems

Metric Unit Example

Scalability MetricsScalability Range (ScR) max The system scales up to 112 req./minScalability Speed (ScS) (max, rate) The system scales up to 112 req./min with linear increase rate 1

req./month

Elasticity MetricsNumber of SLO Violations (NSLOV ) 1/[time unit] 42 SLO (response time) violations in 1 hourMean Time To Quality Repair (MTTQR) [time unit] 30 seconds for an additional 10 requests/hour

Efficiency MetricsResource Provisioning Efficiency (RPE) [0, ] 10% more resources than actual resource demandMarginal Cost (MC) [monetary unit] $1.00 for an additional 100 requests/hour

NSLOV reflects how often a system violates its SLOs whenworkload changes at a given rate, measured as a real num-ber. For example, with a workload increase factor of 1.2, aperfectly elastic system would have 0 SLO violations perrequest, i.e., NSLOVSA(1.2 req/s) = 0. In contrast, anon-elastic system will violate its SLO for each request, i.e.,NSLOVSA(1.2 req/s) = 1.

Efficiency Metrics.Resource Provisioning Efficiency (RPE) We define

resource provisioning efficiency (RPE) as a metric to mea-sure a systems efficiency in a specified workload delta basedon EMclose. That is, the metric measures the mismatch be-tween actual resource utilization and resource demand whilethe workload is changing. We measure this mismatch in per-centage, i.e., as a real number. A perfectly efficient systemwill adapt its resource demand exactly to the resource de-mand at all times. For example, if the workload increaseswith factor 1.2 the system will provision exactly that amountof additional resources required to cope with this additionalworkload, i.e., RPESA(1.2 req/s) = 0.

Marginal costs (MC) We derive the marginal costs fora specified workload delta from EMprov. Marginal costs arethe operation costs to serve one additional workload unit,thus, measuring the efficiency of a cloud computing system.For example, the operation costs to serve 20% additionalrequests per second (factor 1.2) can be $1.00 for system SA,i.e., MCSA(1.2) = $1.00.

5. RELATEDWORKIn this section, we present related work that also targets

metrics for scalability, elasticity, and efficiency in the contextof cloud computing and SLO specification. Where applica-ble, we classify found metrics using our GQM plan in Sec. 6.

In the area of scalability metrics, Bondi [5] further di-vides scalability into structural scalability (ability to ex-pand in a chosen dimension without major modification)and load scalability (ability of a system to perform grace-fully as the offered traffic increases). In terms of this clas-sification, we focus on load scalability in our work. How-ever, in contrast to Bondi, we also provide concrete metricsfor load scalability and apply these to the cloud computingcontext. Duboc et al. [6] formally define scalability require-ments and provide a method to analyze a software modelfor scalability obstacles. A scalability obstacle could also bedefined and detected using our metrics. In contrast to theapproach from Duboc et al., our metrics are closer to SLOs

in their current form. Jogalekar and Woodside [11] presenta scalability metric for general distributed systems. Theirscalability metric also includes an efficiency measure. In ourwork, we distinguish between scalability and efficiency astwo different metrics. Furthermore, in contrast to Jogalekarand Woodside, our metrics are focused on cloud computingenvironments with their particular characteristics.

Herbst et al. [9] provide a set of elasticity metrics basedon speed and precision (w.r.t. avoiding under- and overprovi-sioning) of scale-in and -out. Because their goal is a bench-marking methodology for elasticity, they can assume fullknowledge about the resource usage of the benchmarked ap-plication. However, in our case, we assume that this knowl-edge is unavailable because details on resources are imple-mentation decisions. In contrast, we consider requirementsspecified between cloud consumer and provider. This lack ofknowledge necessarily leads to different metrics as by Herbstet al., e.g., considering SLO violations instead of resource us-age transparent to consumers. Folkers et al. [8] and Islam etal. [10] both provide elasticity metrics that meet this require-ment regarding knowledge. However, they lack the distinc-tion between elasticity and efficiency because they both usecost metrics for elasticity, thus, eliminating the possibilityto investigate both properties in separation.

Roloff et al. [12] define basic efficiency metrics for highperformance computing in cloud computing environments.They define cost efficiency as the product of costs per hourand average performance. In contrast to our work, they ne-glect the context, e.g., actual workload, and only take theaverage performance. Berl et al. [3] address energy efficiencyin all technical components of cloud computing, e.g., servers,networks as well as network protocols. We do not addressenergy efficiency directly but only resource provisioning ef-ficiency in this paper. Investigating whether efficient pro-visioning of resources positively correlates with energy effi-ciency is left as future work.

These related works focus on single quality properties. Incontrast, the Cloud Services Measurement Initiative Consor-tium (CSMIC) provides a standard measurement frame-work, called the Service Measurement Index (SMI), thatcovers all quality properties considered important for cloudcomputing [13]. CSMIC particularly provides metrics forthese quality properties, intended to be used by cloud con-sumers and cloud providers. Their framework allows for astructured classification of quality properties, similar to ourGQM plan. However, CSMIC does not address the deviationof their suggested metrics, thus, leaving open what limita-tions come with their metrics and whether other metrics arefeasible as well.

173

6. CLASSIFICATIONOFMETRICS IN RE-LATEDWORK

In this section, we classify metrics identified in relatedwork (Sec. 5) by relating these metrics to the questions ofour GQM plan. In doing so, we show that we can systemat-ically make assumptions and limitations of existing metricsexplicit. Note that we classify only a few related metrics,for illustration.

Scalability metric by Jogalekar and Woodside [11]Jogalekar and Woodside provide a scalability metricthat can directly be used to answer Q1scale (Doesthe system fulfill its requirements when the workloadincreases (from workload WLX to WLY )?). Theyuse workload as the main input context factor to theirmetric, thus, complying to our scalability question.As an output, they determine a productivity factorthat states whether system requirements can be ful-filled (using a threshold for this factor). Also this ideacomplies to our question. However, to calculate theproductivity factor, they require knowledge about theoperation costs for a given workload. In contrast, wedo not limit our scalability metrics to this knowledgeabout costs and require it for elasticity metrics only.

Elasticity and efficiency metrics by Herbst et al. [9]Herbst et al. provide metrics that consider speed andprecision (w.r.t. avoiding under- and overprovisioning)of scale-in and -out. Their ideas on speed can beused to answer our elasticity question Q4elast (Froma point the system violates its requirements, how longdoes it take before the system recovers to a state inwhich its requirements are met again?). However,to detect requirement violations, they assume to havealready an implemented system at hand; design-time(e.g., simulation-based) approaches do not work withtheir metric. Their ideas on precision fit to our elastic-ity question Q5eff (How close to the actual resourcedemand can the system align the resource provision-ing?). Again, their metric requires an implementedsystem to determine the actual resource usage.

7. CONCLUSIONIn this short research paper, we argue for the need of novel

metrics for quality properties of cloud computing systems.Using the GQM method, we systematically derive an initialset of six metrics for scalability, elasticity, and efficiency.Moreover, by using our GQM plan, we classify existing met-rics to make their limitations explicit.

Our systematically derived metrics help software archi-tects, requirements engineers, testers, etc. to design and an-alyze cloud computing systems. Our GQM plan helps themto consider limitations of such metrics during these tasks.

Future work is directed towards the development of anal-ysis methods and tools that enable software architects toverify the fulfillment of scalability, elasticity, and efficiencyrequirements of their cloud computing applications alreadyat design time. Understanding limitations of metrics is es-sential for this purpose.

8. ACKNOWLEDGMENTSThis work is supported by the German Research Founda-

tion (DFG) within the Collaborative Research Centre On-

The-Fly Computing (SFB 901). The research leading tothese results has received funding from the European UnionSeventh Framework Programme (FP7/2007-2013) under grantno 317704 (CloudScale).

9. REFERENCES[1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph,

R. Katz, A. Konwinski, G. Lee, D. Patterson,A. Rabkin, I. Stoica, and M. Zaharia. A view of cloudcomputing. Commun. ACM, 53(4):5058, Apr. 2010.

[2] M. Becker, S. Becker, and J. Meyer. SimuLizar:Design-time modelling and performance analysis ofself-adaptive systems. In Proceedings of SoftwareEngineering 2013 (SE2013), Aachen, 2013.

[3] A. Berl, E. Gelenbe, M. Di Girolamo, G. Giuliani,H. De Meer, M. Q. Dang, and K. Pentikousis.Energy-efficient cloud computing. The ComputerJournal, 53(7):10451051, 2010.

[4] G. Bolch, S. Greiner, K. S. Trivedi, and H. de Meer.Queueing Networks and Markov Chains: Modeling andPerformance Evaluation With Computer ScienceApplications. 1998.

[5] A. B. Bondi. Characteristics of scalability and theirimpact on performance. In WOSP 00, pages 195203,New York, NY, USA, 2000. ACM.

[6] L. Duboc, E. Letier, and D. S. Rosenblum. Systematicelaboration of scalability requirements throughgoal-obstacle analysis. IEEE Transactions on SoftwareEngineering, 39(1):119140, Jan. 2013.

[7] T. Erl, Z. Mahmood, and R. Puttini. CloudComputing: Concepts, Technology & Architecture.Prentice Hall, 2013.

[8] E. Folkerts, A. Alexandrov, K. Sachs, A. Iosup,V. Markl, and C. Tosun. Benchmarking in the cloud:What it should, can, and cannot be. In TPCTC, pages173188, 2012.

[9] N. R. Herbst, S. Kounev, and R. Reussner. Elasticity:What it is, and What it is Not. In ICAC 13, 2013.

[10] S. Islam, K. Lee, A. Fekete, and A. Liu. How aconsumer can measure elasticity for cloud platforms.In Proceedings of the 3rd ACM/SPEC InternationalConference on Performance Engineering, ICPE 12,pages 8596, New York, NY, USA, 2012. ACM.

[11] P. Jogalekar and M. Woodside. Evaluating thescalability of distributed systems. IEEE Trans.Parallel Distrib. Syst., 11(6):589603, June 2000.

[12] E. Roloff, M. Diener, A. Carissimi, and P. Navaux.High performance computing in the cloud:Deployment, performance and cost efficiency. InCloudCom 12, pages 371378, 2012.

[13] J. Siegel and J. Perdue. Cloud services measures forglobal use: The service measurement index (smi). InSRII Global Conference (SRII), 2012 Annual, pages411415, July 2012.

[14] C. U. Smith. Performance engineering of softwaresystems. Software Engineering Institute series insoftware engineering. Addison-Wesley, 1990.

[15] R. van Solingen, V. Basili, G. Caldiera, and H. D.Rombach. Goal Question Metric (GQM) Approach.John Wiley & Sons, Inc., 2002.

174