exploiting nonstationarity for performance...
TRANSCRIPT
Exploiting Nonstationarity for Performance Prediction
Christopher StewartU. Rochester CS Dept.
Terence KellyHewlett-Packard Labs
Alex ZhangHewlett-Packard Labs
ABSTRACTReal production applications ranging from enterprise applicationsto large e-commerce sites share a crucial but seldom-noted charac-teristic: The relative frequencies of transaction types intheir work-loads arenonstationary, i.e., the transaction mix changes over time.Accurately predicting application-level performance in business-critical production applications is an increasingly important prob-lem. However, transaction mix nonstationarity casts doubton thepractical usefulness of prediction methods that ignore this phe-nomenon.
This paper demonstrates that transaction mix nonstationarity en-ablesa new approach to predicting application-level performanceas a function of transaction mix. We exploit nonstationarity tocircumvent the need for invasive instrumentation and controlledbenchmarking during model calibration; our approach relies solelyon lightweight passive measurements that are routinely collected intoday’s production environments. We evaluate predictive accuracyon two real business-critical production applications. The accuracyof our response time predictions ranges from 10% to 16% on theseapplications, and our models generalize well to workloads very dif-ferent from those used for calibration.
We apply our technique to the challenging problem of predict-ing the impact of application consolidation on transactionresponsetimes. We calibrate models of two testbed applications running ondedicated machines, then use the models to predict their perfor-mance when they run together on a shared machine and serve verydifferent workloads. Our predictions are accurate to within 4% to14%. Existing approaches to consolidation decision support pre-dict post-consolidationresource utilizations. Our method allowsapplication-level performanceto guide consolidation decisions.
Categories and Subject DescriptorsC.4 [Performance of Systems]: Modeling techniques
General TermsExperimentation, Management, Measurement, Performance
Keywordsenterprise, internet services, LAR regression, mutli-tier, noninva-sive, nonstationarity, performance prediction, realistic workloads
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.EuroSys’07,March 21–23, 2007, Lisboa, Portugal.Copyright 2007 ACM 978-1-59593-636-3/07/0003 ...$5.00.
1. INTRODUCTIONModern distributed applications continue to grow in scale and
complexity. Distributed enterprise applications are furthermore as-suming a growing role in business-critical operations. Understand-ing the performance of such applications is consequently increas-ingly difficult yet increasingly important due to their economic value.This paper considers the problem of performance predictionin dis-tributed applications: Given forecasts of future application work-load, we seek to predict application-level response times.A goodsolution to this problem will enable operators to explore a widerange of important “what-if” scenarios, e.g., “How will responsetimes change if the the number visitors at my Web site doublesandthe buy:browse ratio increases by 50%?” We do not address thecomplementary problem of workload forecasting, but we showthatif accurate workload forecasts are available they can be mappeddirectly to accurate performance predictions.
The workloads of the real production applications that we seek tomodel share a crucial but seldom-noted characteristic: thetransac-tion mixes in these workloads are highlynonstationaryin the sensethat the relative frequencies of transaction types vary considerablyover time. This is a problem for most conventional performancemodels, which implicitly assume that transaction mix is stationary,because the system resource demands of different transaction typesare usually very different in real applications.
Our approach leverages earlier work that focused on retrospec-tively explainingperformance in terms of transaction mix [23]. Weincorporate queueing-theoretic extensions into the earlier techniqueto obtain a method suitable for prospectivelypredictingfuture per-formance as a function of transaction mix. One novel featureofour approach is that whereas performance models in prior systemsliterature include ascalar measure of workload intensity, we de-scribe workload using a transaction-mixvector. Another novel fea-ture is that we exploit transaction mix nonstationarity to circumventthe need for invasive instrumentation and controlled benchmarkingduring model calibration.
Our approach is practical for real production systems and canbe applied to a wide range of applications. Our models are cal-ibrated using purely passive measurements that are routinely col-lected in today’s real production applications. Furthermore, theywork well under a wide range of workload conditions and a widevariety of application architectures, including locally distributedmulti-tier E-commerce applications and globally-distributed high-availability enterprise applications.
We compare our proposed method with several alternatives, eval-uating their ability to predict response times in two very differentreal production applications: the Web shopping site of a major re-tailer and a business-critical internal enterprise application. Ourmethod accurately predicts response times for both applications.
Furthermore our performance models generalize well to regions ofworkload space very different from those present in the calibrationdata. We demonstrate that transaction mix models achieve sub-stantially greater accuracy than similar models that employ scalarmeasures of workload intensity.
Finally, we apply our method to the challenging problem of pre-dicting response times in applications that areconsolidatedonto ashared infrastructure, subject to a severe handicap: we must cal-ibrate our models using only lightweight passive observations ofthe applications running on dedicated machines prior to consoli-dation. We evaluate our performance predictions in consolidatedenvironments using a testbed of benchmark applications, since realproduction applications were unavailable for experimentation. Ourpredictions are remarkably accurate according to two measures thatpenalize inaccuracy in very different ways. The current state of theart in consolidation decision support both in practice and in the re-search literature predicts theresource utilizationeffects of consoli-dation. We present a practical way to incorporateapplication-levelperformanceinto consolidation decision-making.
The remainder of this paper is organized as follows: Section2describes the prevalence of transaction mix nonstationarity in real-world workloads, the problems it poses for many conventional per-formance models, and the opportunities it creates that we exploit.Section 3 presents our approach to performance prediction,definesour main accuracy measure, and describes an accuracy-maximizingmodel calibration procedure. Section 4 describes the applicationsused in our tests and presents empirical results on the accuracy ofour predictions. Section 5 applies our models to the challengingproblem of predicting the performance of applications thatare con-solidated onto a shared infrastructure. Section 6 reviews relatedwork, and Section 7 concludes with a discussion.
2. TRANSACTION MIX NONSTATIONAR-ITY IN REAL WORKLOADS
It is well known that thevolumeof demand in production appli-cations naturally fluctuates on several time scales (e.g., daily andweekly cycles). Similarly, there is little reason for the transactionmix of real applications to remain constant over time. In this sec-tion, we describetransaction mix nonstationarityin two real pro-duction applications (Section 4.1 describes the applications them-selves in detail). An investigation into the factors that influencenonstationarity in real applications is orthogonal to our goal of per-formance prediction, so we leave it for future work.
Figures 1 and 2 illustrate time variations in transaction mix. Thefirst is a scatterplot of the relative frequencies of the two most com-mon transaction types of the “VDR” application in 5-minute timewindows. Note that nearly every possible combination is present(the upper right corner of the plot must be empty because the sumof the two fractions cannot exceed 1). Figure 2 is a time seriesof the fraction of “ACME” transactions that are of type “add-to-cart” in 5-minute windows. It shows that this fraction varies overtwo orders of magnitude during a four-day period (note that thevertical scale is logarithmic). The transaction mix nonstationarityevident in these figures is not an artifact of 5-minute time windows;it remains when we aggregate measurements into much longer in-tervals. Figure 4 shows the fraction of VDR transactions dueto themost common transaction type in hour-long windows over a periodof several days; the fraction ranges from less than 5% to over50%.Plots using longer aggregation intervals are qualitatively similar.
One implication of transaction mix nonstationarity is thatthe fullspectrum of workloads for which we must predict performancemaynot be available during model calibration. Performance models
must thereforegeneralizewell to workloads that are very differentfrom those used for calibration. Furthermore, a convincingvali-dation of a performance prediction method requires nonstationaryworkloads, because stationary workloads differ qualitatively fromreal-world workloads.
Synthetic workload generators used in benchmarking and sys-tems research typically employ first-order Markov models todeter-mine the sequence of transactions submitted by client emulators;examples include the standard TPC-W workload generator [47] andthe RUBiS workload generator [38]. This approach cannot yieldthe kind of markedly nonstationary workloads that we observe inreal production applications, because the long-term relative stateoccupancy probabilities of first-order Markov processes are station-ary [43]. Figure 5 shows the relative fractions of the two most com-mon transaction types in the workload generated by the default RU-BiS generator during a 5-hour run, in 5-minute windows.Nearlyall of the 60+ data points lie on top of one another. Plots of differ-ent transaction type pairs aggregated into different time windowsare qualitatively similar.
What are the implications of nonstationarity for performancemodeling? We define ascalar performance modelas one that ig-nores transaction mix in workload and instead considers only ascalar measure of workload intensity, e.g., arrival rate. Nonstation-arity clearly poses serious problems for scalar performance mod-els. For example, consider an application whose workload consistsof equal numbers of two transaction types: type A, which placesheavy demands on system resources, and type B, which has lightdemands. Suppose that we want to predict the application’s perfor-mance if the total number of transactions increases by 50%. Scalarmodels may work well if the relative proportion of the two trans-action types remains equal. However such models are unlikely toyield accurate predictions if the transaction mix changes:Perfor-mance will differ dramatically if the number of type-A transactionsdoubles and the number of type-B remains constant, or vice versa.Of course, evaluations of scalar performance models using first-order Markov workload generators will not expose this problem.Stationary test workloads mask the deficiencies of scalar perfor-mance models.
This paper employstransaction mix modelsthat predict application-level performance based on transaction counts by type. These mod-els have a number of attractive features: they are “semanticallyclear” in the sense that their free parameters have intuitive interpre-tations; they yield accurate performance predictions under a widerange of circumstances; and the computational procedures used formodel calibration are fairly straightforward. However it is non-stationarity that makes our approach particularly practical, becausenonstationarity allows us to calibrate our models using only light-weight passive measurements that are collected in today’s real pro-duction environments. We describe the opportunities that nonsta-tionarity creates for calibration in greater detail in Section 3.5, afterwe describe our performance models.
3. TRANSACTION MIX MODELSThis section describes our transaction mix performance models
and several variants and alternatives against which we shall latercompare them. All models have the same general high-level form:
P = F~a(~W) (1)
whereP is a scalar summary of application performance,F spec-ifies the functional form of our models,~a is a vector of calibratedparameter values, and~W is a vector of workload characteristics.This section explains the development of our approach. Section 3.1justifies our basic assumptions in terms of the measured properties
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
frac
tion
of 2
nd m
ost p
opul
ar tr
ansa
ctio
n
fraction of most popular transaction
0.001
0.01
0.1
1
0 24 48 72 96 120
frac
tion
add-
to-c
art
time (hours)
0
0.25
0.5
0.75
1
0 10 20 30 40 50 60 70 80 90 100
P[X
<=
x]
utilization seen by arriving transactions
ACME CPU
VDR CPU
ACMEdisk
Figure 1: Fractions of VDR transactions,two most common types. Each point rep-resents a different 5-minute interval.
Figure 2: Fraction of ACME “add-to-cart” transactions vs. time. Each pointrepresents a different 5-minute interval.
Figure 3: CDFs of resource utilizationsencountered by arriving transactions inACME and VDR applications.
0 0.25 0.5
0.75
0 20 40 60 80 100 120 140 160 180 200 0 2000 4000 6000 8000
frac
tion
due
tom
ost p
opul
ar ty
pe
# tr
ansa
ctio
nsdu
ring
hour
time (hours) 0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
frac
tion
2nd
mos
t pop
ular
type
fraction most popular type
Figure 4: Fraction of VDR transactions due to most commontype (heavy line); total transaction volume (light line), 1-hr win-dows.
Figure 5: Fraction of RUBiS transactions due to two most com-mon types using default generator. The figure is not empty;note the tight cluster of points at coordinates (0.3, 0.3). Thisworkload is highly stationary.
of real applications. Section 3.2 presents our performancemodels.Section 3.4 defines the accuracy measure that we seek to optimize,and Section 3.5 explains how we calibrate our models to maximizeaccuracy according to this measure.
3.1 AssumptionsWe begin with the following observations about modern dis-
tributed applications:
1. Workload consists of request-replytransactions.2. Transactions occur in a small number oftypes(e.g., “log
in,” “browse,” “add-to-cart,” “checkout” for an E-commercesite).
3. Transaction types strongly influence system resource demands(e.g., “checkout” transactions at an E-commerce site requiremore CPU than browsing).
4. Resources are adequately provisioned or over-provisioned inbusiness-critical production applications.
5. Transaction mix is nonstationary.
The first two observations apply to every commercially-importantdistributed production application that we have encountered. Thethird property arises because transaction types often determine therun-time code path through application logic, which in turnstronglyinfluences resource service demands. The fourth property, ade-quate resource provisioning, is a fundamental requirementof ca-pacity planning in business-critical applications. By design, allo-cated capacity is generous relative to offered workload; heavy loadand overload represent serious failures of the capacity planning pro-cess. Fortunately such failures are rare because capacity planningfor intra-enterprise applications can often exploit good estimates ofthe total user population and anticipated usage patterns.
Even in server-consolidation scenarios where elevating resourceutilization is an explicit goal, practitioners are advisedto keeppeakutilizations of resources such as CPU below 70% [14]. In practice,enterprise system operators are typically even more cautious than
this conservative guideline. Figure 3 shows cumulative distribu-tions of resource utilizations encountered by arriving transactionsin the two distributed production applications used in our investiga-tion, “ACME” and “VDR.” Transactions arriving at these two verydifferent applications operated by two different firms rarely find uti-lization at any resource in excess of 35%; utilization greater than50% is almost never encountered. This implies that whilequeueingtimes at resources such as CPUs and disks should not be ignored,servicetimes will often account for much of overall transaction re-sponse times. Another implication is that non-queueing congestioneffects associated with very heavy load, e.g., cache interference,are likely to be rare in practice.
Together with our first three assumptions, Figure 3 suggestsaradically simple performance model that accounts for transactionservice times but ignores queueing entirely. Our previous work re-ports that such a model works surprisingly well in practice.Specif-ically, the sum of response times across all transactions within aspecified time interval is well explained by transaction mixalone [23].However, Figure 3 also suggests that waiting times are sometimesnon-negligible, and our approach in this paper models waiting times.
In summary, we observe that transaction mix alone is a powerfulperformance predictor; it will be the~W of Equation 1. We havealso seen that queueing can be non-negligible, and our models willexplicitly account for waiting times in addition to servicetimes.Our performance measureP will be aggregate transaction responsetime within short time windows (e.g., 5-minute intervals);this caneasily be converted toaverageresponse time because we know thenumber of transactions within each window. After specifying theform of our modelsF and defining our measures of model accuracy,we describe how to obtain accuracy-maximizing parameters~a byexploiting naturally-occurring nonstationarity.
3.2 ModelsThis section develops a series of three performance models of
increasing sophistication and breadth of applicability. The Basic
model of Section 3.2.1 takes into account transaction mix alone; itis taken from previous work [23]. Section 3.2.2 extends the Ba-sic model to explicitly incorporate queueing delays. The Extendedmodel does not conform to the template of Equation 1, however,because its inputs include resource utilizations as well astransac-tion mix. While the Extended model may offer improved accuracywhen used to retrospectivelyexplainperformance, it cannot be usedto predictperformance given workload forecasts alone. The Com-posite model of Section 3.2.3 corrects this deficiency by modelingresource utilizations in terms of transaction mix and incorporatingthe utilizations thus obtained into the Extended model. Finally, ourempirical evaluations will include variants of the Basic, Extended,and Composite models that use only a scalar measure of workloadintensity rather than a vector describing transaction mix.
3.2.1 Basic ModelWe divide time into short non-overlapping intervals, e.g.,5 min-
utes. For intervali let Ni j denote the number of transactions of typej that began during the interval and letTi j denote the sum of theirresponse times. Our Basic model has the form
yi = ∑j
Ti j = ∑j
α jNi j (2)
whereyi is the sum of all transaction response times during inter-val i. Note that no intercept term is present in Equation 2, i.e.,we constrain the model to pass through the origin: Aggregatere-sponse time must be zero for intervals with no transactions.Valuesof model parametersα j are obtained through model calibration; leta j denote these calibrated values. Intuitively, calibrated parametersa j represent typicalservicetimes for the various transaction types,summed over all service and delay centers on the transaction’s ex-ecution path.
For given model parametersa j and observed transaction mixNi jat timei, let
yi = F~a(~Ni) = ∑j
a jNi j (3)
denote thefitted valueof the model at timei. If the Ni j repre-sent past workload, ˆyi can be interpreted as the model’s guess ofwhat aggregate response time should have been during interval i.If instead the given transaction mix is a forecast of future work-load, the fitted value represents the model’s performance predic-tion. Note that since the total number of transactions within an in-terval is known—it is simply∑ j Ni j —one can convert a fitted valueyi representingaggregateresponse time into anaverageresponsetime.
Our Basic model can be thought of as an open queueing net-work containing a single service station with an infinite number ofservers. Waiting cannot occur in such a system, and Equation2does not explicitly model waiting times.
3.2.2 Extended ModelWe extend the Basic model of Equation 2 by adding terms rep-
resenting waiting times, as follows:
yi =n
∑j=1
α jNi j +∑r
(
1λi
·U2
ir
1−Uir
)
·n
∑j=1
Ni j . (4)
The rightmost term represents waiting times in an M/M/1 queue,with one queue per resource;Uir denotes the utilization of resourcer during intervali. The naıve approach of adding utilizations assimple linear terms has no basis in queueing theory, but we shallcompare our approach with this alternative (see the discussion ofTable 4 in Section 4.2.1).
CPU
Network
Disk
CPU
Network
Disk
App Server DB Server
Figure 6: Extended queueing model: One station per resourceat each tier.
To derive Equation 4, we note that the term1λi·
U2ir
1−Uirrepresents
the average waiting time (per transaction at resourcer over inter-val i) in an M/M/1 queueing model, whereλi is the arrival rateof transactions of all types in intervali. We multiply this term bythe number of transactions of all types in intervali to obtain thesum of waiting times, to agree with the left-hand-side of theequa-tion. Realizing thatλi = ∑n
j=1 Ni j /L whereL is the interval lengthin seconds, one can further simplify the sum-of-waiting-time term
to ∑r L ·U2
ir1−Uir
. Finally, since the sum-of-waiting-time term on theright-hand-side of Equation 4 does not involve any unknown pa-
rametersα j , one can regresszi ≡ yi −∑r L ·U2
ir1−Uir
against the trans-action mix∑n
j=1 α jNi j .Figure 6 depicts our Extended model as an open queueing net-
work consisting of a single-server station for each resource (e.g.,CPU, disk, network) at each tier (e.g., application server,databaseserver). Although the figure shows only two tiers with three re-sources each, the model can accommodate additional tiers (e.g., fora Web server) and additional resources. The Extended model cap-tures the distributed aspect of an application by explicitly includingnetwork queueing effects.
3.2.3 Composite ModelBecause it relies on resource utilizations, the Extended model of
Equation 4 cannot be used to predict performance based on transac-tion mix Ni j alone. Our Composite model overcomes this difficultyby estimating utilizations as weighted sums of transactioncounts:
Uir = β0r +∑j
β jr Ni j (5)
whereβ jr represents the service demand of transaction typej onresourcer. The total service demand placed on the resource isthe sum of service demands of all transaction types. As with theBasic response time model of Section 3.2.1, we obtain for eachresourcer parametersb jr corresponding to theβ jr during modelcalibration. Unlike the model of Equation 2, however, we includean intercept termβ0r in our utilization models, because real systemresources are not entirely idle even in the complete absenceof ap-plication workload. Equation 5 generalizes the familiar UtilizationLaw; specifically, it reduces to the Utilization Law in the specialcase of one transaction type and no intercept term.
Once we have obtained utilization estimatesUir from a calibratedutilizationmodel (Equation 5), we substitute these into a calibratedExtendedmodel to obtain aCompositemodel of aggregate responsetime as a function of transaction mixNi j . In rare cases whereUir <
0 or Uir ≥ 1, we correct the utilization estimate to zero or 1− ε,respectively.
3.2.4 Scalar ModelsRecent queueing models of distributed application performance
rely on ascalarmeasure of workload intensity that ignores transac-tion types [41,48]. What additional predictive power do we obtainby using a transaction-mix vector? To address this question, ourempirical evaluations will compare our models with Scalar variants
that use only the totalnumberof transactions in each time interval.For example, the Scalar variant of the Basic model is
yi = ∑j
Ti j = αNi (6)
whereNi = ∑ j Ni j is the total number of transactions that occurredduring time intervali.
3.3 DiscussionOur approach contains a number of simplifications that deserve
mention. We account for waiting times at each resource usinganexpression for a single-server queue, whereas many real productionapplications run on systems with multiple CPUs and disks at eachtier. Previous work made similar simplifying assumptions [41],and we find that in practice this approach works well. Our modelassumes an open network in which requests exit after service. Aclosed network model would require us to model client “thinktimes,”as in some previous models of distributed applications [28,48].
Our queueing networks implicitly assume that transactionsdonot recirculate among resources; our models aggregate all servicetimes from all of a transaction’s visits to a resource, rather thanexplicitly modeling visits separately. More sophisticated queueingmodels take into account recirculation among service stations [20],but such models require detailed information about how transac-tions move among resources, which is often not available in prac-tice. Tools to gather this information exist as research prototypes,e.g., Magpie [7], but few real production systems are currently in-strumented to measure fine-grained transaction resource visits. Wehave designed our models to require for calibration only data thatis routinely collected on today’s real production applications.
Our use of open queuing network models is in part motivated bypractical considerations: It is often difficult in practiceto obtainfor real production applications the client session information re-quired to calibrate closed models, and therefore the use of an openmodel facilitates more thorough empirical validation thanwould bepossible if we employed a closed model. However there are goodreasons for preferring open models for their own sake, e.g.,theyare relatively simple. More importantly, open models are more ap-propriate if transient overloads are possible because openmodelsdo not inherently restrict the number of concurrent transactions ina system. See Schroederet al. for a detailed discussion of the im-plications of open versus closed models [40].
For all of the production and testbed applications considered inthis paper, transaction types are given—they can easily be inferredby, e.g., inspecting the request URL. In our experience suchtrans-action type information is sufficient for our method and is readilyavailable in practice. Our approach remains applicable if transac-tion types are not given as long as a classifier can be constructedthat maps transactions to types that reflect their system resourceservice demands. Transaction type classification is an orthogonalproblem to our modeling interests and has been the subject ofex-tensive research (see Section 6.1).
We have implicitly assumed that an application’s set of transac-tion types is fixed and the relationship between transactiontype andresource demands is stable. This is not a restrictive assumption inpractice because the time required to re-calibrate new performancemodels is short compared to the time scales on which applicationlogic and transaction structure changes. In our experiencewithreal production applications in the enterprise, changes toapplica-tion structure and configuration are normally rare; stakeholders inbusiness-critical applications do not undertake such modificationslightly or frequently. Even if transaction types or their resource de-mands change completely, it takes only a day or two to gather suf-
ficient data to calibrate completely new performance models. Lessdrastic changes are easier to handle: Model calibration itself takesless than a second, and therefore continuous re-calibration (e.g.,at the conclusion of every 5-minute measurement interval) can beused to track gradual drift in the workload/performance relation-ship.
A final simplification is that our models ignore interaction ef-fects across transaction types and implicitly assume that queueingis the only manifestation of congestion. However queueing doesnot describe certain kinds of resource contention, e.g., cache inter-ference. “Checkout” transactions, for instance, may require moreCPUservicetime during heavy browsing if the latter reduces pro-cessor cache hit rates for the former. Our models do not accountfor such effects. The question of whether our simplifying assump-tions areoversimplificationsis ultimately an empirical one, whichwe address in Section 4.
3.4 Accuracy MeasuresIf yi is the actual measured aggregate response time during in-
terval i and yi is the fitted value obtained from a calibrated per-formance model, letei = yi − yi denote theresidual (model error)at time i. We measure model accuracy in terms of intuitive func-tions of the residuals. We cannot use the conventional coefficient ofmultiple determinationR2 to assess model accuracy; it is not mean-ingful because Equation 2 and Equation 4 lack intercept terms [33,p. 163].
Our overall figure of merit, normalized aggregate error, general-izes the familiar, intuitive concept of absolute percent error:
normalized aggregate error≡∑i |ei |
∑i yi(7)
Consider, for example, asingle(y, y) pair: if y = 100 and ˆy = 105,then normalized aggregate error is 0.05, indicating that the model’sprediction is off by 5%. We say that model parameters for Equa-tion 2 or Equation 4 areoptimal if they minimize error as definedby Equation 7.
In addition to our overall figure of merit we shall also reportthe distribution of normalized residuals|ei |/yi , scatterplots of(y, y)pairs, and order statistics on the normalized residuals. Each of thesemeasures offers different insight into model accuracy and penal-izes inaccuracy in a different way. For example, the distributionof |ei |/yi penalizes even small residuals if the corresponding mea-surements are small, whereas Equation 7 penalizes residuals whosemagnitude is large even if|ei | is small relative to the correspondingyi . A good model is accurate according to both measures.
3.5 CalibrationThe input to calibration is a data set consisting of aggregate re-
sponse timesyi and transaction mixes~Ni = (Ni1,Ni2, . . .). For ourExtended and Composite models we furthermore require resourceutilizationsUir . These inputs correspond to readily available andpurely passive measurements of applications and their underlyingsystem resources. A good rule of thumb is that model calibra-tion requires roughly ten times as many measurement intervals ias transaction types [33]. If measurements are taken at 5-minuteintervals, a few days suffice to collect enough data to calibrate ourmodels of the production applications that we study.
The output of calibration is a set of calibrated parameter valuesa j corresponding to theα j parameters of the Basic and Extendedmodels and, for a Composite model, calibrated parameter valuesb jr corresponding to theβ j parameters of the utilization model(Equation 5).
The goal of calibration is to compute parameters that maximizemodel accuracy. The denominator in Equation 7 is a constant,soto achieve optimal accuracy a calibrated model must minimize thenumerator, i.e., the sum of absolute residuals. This is a specialcase of linear programming, for which specialized variantsof thesimplex algorithm have been developed; we use the algorithmofBarrodale & Roberts [8]. The algorithm yields model parametersthat optimize retrospective explanatory accuracy with respect to thedata used for calibration. This exercise is sometimes knownasleast absolute residuals(LAR) regression. Ordinary least squares(OLS) regression minimizes the sum ofsquaredresiduals, and itcan be shown that a model with OLS parameters can havearbitrar-ily worseaccuracy than an optimal model according to the measureof Equation 7. In practice we find that LAR-calibrated modelsaremore accurate than their OLS-calibrated counterparts.
Another advantage of LAR is that it isrobust, i.e., it resists theinfluence of extreme values in the calibration data set. By contrast,OLS is far more sensitive to distortion by outliers. A wide varietyof robust regression procedures are available; several arevariantsof OLS and LAR [34, 52]. We prefer plain-vanilla LAR becauseit guarantees optimal retrospective accuracy, because it is concep-tually simple and easy to explain, and because it involves notun-able parameters. The only disadvantage of LAR is that numericalsolvers are not as widely available. However, as reported previ-ously, the accuracy gain over OLS outweighs the inconvenienceof LAR [23]. A final advantage of using linear programming formodel calibration is that it is easy to add additional constraints,e.g., on the values of parameters. Extensions of elementarysta-tistical techniques such as least-squares regression can sometimesachieve similar capabilities, but in our experience they donot offerthe generality, convenience, and flexibility of linear programming.
3.5.1 The Role of NonstationarityModel calibration in our approach exploits variations in transac-
tion volume, transaction mix, and resource utilization in the calibra-tion data—i.e., the kind of nonstationarity found in real workloads.At the other extreme, it is easy to show that lightweight passivemeasurements of response times and utilizations collectedunderperfectlystationaryworkload cannot be used for model calibration.The essential difficulty is that the optimization problem that the re-gression procedure seeks to solve lacks a unique solution. Typicalimplementations of OLS regression, for example, fail in such casesbecause they attempt to invert a singular (non-invertible)matrix.
A simple example conveys intuition for the insurmountable prob-lems created by perfectly stationary transaction mix. Consider thefollowing utilization and transaction mix measurements:
time utilization number of transactionsinterval i U type A type B
1 u1 5 72 u2 10 143 u3 15 214 u4 20 28
These data cannot be used to calibrate a utilization model (Equa-tion 5) regardless of the calibration procedure used, andregardlessof the utilization measurements ui . The problem is that the transac-tion count information is essentially the same in each row becausethe counts in rows 2 through 4 are multiples of those in row 1 andthe A:B ratio is everywhere the same—i.e., the transaction mix isstationary. This makes it impossible to determine whether Ais aheavyweight transaction and B is lightweight or vice versa.Thesituation is the same if the data include aggregate responsetimesrather than utilization measurements and our goal is to calibrate aBasic model (Equation 2) or an Extended model (Equation 4).
If a first-order Markov model generates workload for calibrationand if measurement intervals contain a reasonably large number ofrequests, the result will almost certainly benearly-stationarywork-load. This in turn causesmulticollinearity, a regression pathologythat arises when predictor variables are mutually correlated. Thenet effect is that predictive accuracy will suffer regardless of theregression procedure used. On the other hand, our empiricalresultsshow that naturally-occurring workloads have sufficient transactionmix nonstationarity to allow us to calibrate very accurate modelsusing passive measurements of utilizations and response times.
In summary, the nonstationarity of real workloads makes pos-sible our lightweight model calibration approach, which relies onpassive measurements and requires no invasive system or applica-tion instrumentation or controlled benchmarking.Nonstationarityallows model calibration to substitute data analysis for invasivemeasurement.
4. VALIDATIONWe calibrate our models and evaluate their retrospective explana-
tory accuracy on the first half of each validation data set. Wethenapply the calibrated models to the transaction mix in each time in-terval i of the second half to obtain fitted values ˆyi . Finally, wecompare these ˆyi with observedyi to evaluate prospective predic-tion accuracy. This section first describes the two real productiondata sets used in our evaluation and an additional data set collectedin a lab environment, then presents our results.
4.1 Data SetsTable 1 summarizes our data sets. The first two were collected
on distributed production applications serving real customers andreal enterprise users, respectively. Together these data sets severelychallenge the ability of our method to generalize along several di-mensions. They differ in terms of the nature of the application,the extent of geographic distribution, and workload. Theirwork-loads exhibit the nonstationarities described in Section 2. Due tospace limitations we provide an abbreviated description ofthe datasets here; additional information on all three data sets is availablein [24].
Pronounced workload periodicity is present in all of our datasets. Referring to the light data series and right-hand vertical axisin Figure 4, we see that workload ranges from under 100 to roughly7,300 transactions per hour. The ACME data set shows a similardaily cycle with comparably wide variation. Other periodicities andnonstationarities are present in all of our data sets.
4.1.1 ACME: DotCom-Era Web ShoppingThe “ACME” data set is taken from the busiest of seven servers
comprising a major US retailer’s Web shopping site. This server ac-counts for roughly 38% of all ACME transactions during the mea-surement period. ACME was typical of large E-commerce sitescirca 2000. For confidentiality reasons, the researchers who stud-ied the ACME site were not permitted to disclose the details of itshardware and software infrastructure. However, an extensive work-load characterization is available [3]. The ACME data set includesmeasurements of transaction response times and system resourceutilizations collected at the application server tier; it does not in-clude requests for static Web pages. Each transaction is further-more tagged as a cache hit or a miss, and we treat hits and missesof the same transaction type as two different types.
4.1.2 VDR: Modern Enterprise ApplicationFigure 7 depicts the architecture of the globally-distributed VDR
application. VDR is a high-availability business-critical internal
Data Production Transactions: Trans’ns/min Resp time (sec) Type ofSet Dates Duration Number Types Mean Median Mean Median Application
ACME July 2000 4.6 days 1,180,430 93 182.2 183.0 0.929 0.437 Web Retail ShoppingVDR Jan 2005 7.8 days 666,293 37 59.4 56.4 1.289 1.236 Business-critical Enterprise
PetStore April 2004 38 hours 4,920,642 10 2147.8 3163.8 0.096 0.040 Sample application
Table 1: Summary of production data sets.
client client client
WAN
App Srvr App Srvr
Loadbalancer
Primary DB Backup DB
client client client
WAN
App Srvr App Srvr
Loadbalancer
Primary DB Backup DB
client client client
WAN
App Srvr App Srvr
Loadbalancer
Primary DB Backup DB
WANOracle DB replication
AMERICAS EUROPE ASIA
DB failover DB failover
Man
aged
App
Ser
ver
Pro
vide
r
Figure 7: VDR application architecture.
HP application serving both external customers and HP usersonsix continents. Its system architecture therefore incorporates re-dundancy and fail-over features both locally and globally.Regionalhubs in Atlanta, Swindon, and Singapore respectively servetheAmericas, Europe, and Asia regions. All application serverhostsare HP 9000/800 servers running HP-UX B.11.11. The Americasregion has two such hosts with 16 CPUs and 64 GB RAM each.The European region has three, with 16 CPUs and 32 GB RAMeach, and the Asia region has two, with 12 CPUs and 20 GB RAMeach. All of the app servers ran BEA WebLogic. We have lessdetailed information about hosts at the database tier, but we knowthat they are similar in number and specifications to those attheapp server tier and that they ran Oracle 9i.
VDR operators and system architects have told us that VDRtransactions are relatively “heavyweight” in the sense that they placesubstantial demands on system resources. The VDR data set in-cludes both transaction records and system resource utilization mea-surements collected atbothapplication server and database servertiers, derived respectively from application-level logs and Open-View Performance Agent (OVPA).
4.1.3 PetStore: Sample ApplicationThe PetStore data set was collected on a testbed applicationserv-
ing a highly variable and highly nonstationary workload. The totalrequest rate and the ratios of transaction type frequenciesvary si-nusoidally, and the total rate increases over time so that the peaksrepresent transient overloads (“flash crowds”) during which offeredworkload exceeds the system capacity. Our models implicitly as-sume that load doesnot exceed system capacity, so this data setprovides an extraordinary challenge to our approach. The PetStoredata set and the environment in which it was collected are describedin detail in earlier publications [15,24]; due to space limitations wedo not repeat these descriptions here. The PetStore data setwasgenerated long before the present investigation began and thereforewas not (consciously or otherwise) tailored to the methodology pro-posed in this paper.
4.2 ResultsWe calibrate our models on the first half of each data set, and
also measure their retrospective explanatory accuracy on the firsthalf. We then evaluate the predictive accuracy of the calibratedmodels using the transaction mixes in the second half of eachdataset. Transaction mix nonstationarity therefore implies that predic-
VDR ACME PetStoreScalar TMIX Scalar TMIX Scalar TMIX
BasicOLS 0.1485 0.1002 0.2034 0.1308 0.4403 0.3605LAR 0.1462 0.0940 0.2029 0.1281 0.3646 0.3230
ExtendedOLS 0.1478 0.0997 0.2012 0.1213 0.2320 0.2897LAR 0.1454 0.0936 0.2006 0.1185 0.1978 0.2221
Table 2: Retrospective explanatory accuracy: Normalized ag-gregate error ∑i |ei |/∑i yi .
tive evaluations will involve workloads different than those usedin calibration. Nonstationarities provide a challenging and credi-ble evaluation of the extent to which models generalize beyond thecalibration data.
4.2.1 Retrospective ExplanationTable 2 summarizes the explanatory accuracy of eight models
on our three data sets according to our overall figure of merit, nor-malized aggregate error (Equation 7). The table includes both thestandard transaction mix model (TMIX) versions of our BasicandExtended models as well as Scalar variants that ignore transactiontypes and use only the total number of transactions. Both OLSandLAR regression are used to calibrate each model variant.
First, the transaction mix models consistently outperformtheirscalar counterparts by a wide margin for the production datasets:15% vs. 10% error for VDR and 20% vs. 13% error for ACMEregardless of the calibration procedure. Even for the deliberatelyoverloaded PetStore, which violates flow balance, the Extendedtransaction mix model calibrated with LAR has only a 22% error.We achieve substantially greater accuracy by exploiting transac-tion mix.
Second, the Extended model, which accounts for queueing, out-performs the Basic model, which does not. Our Basic model achievesnearly the same accuracy for both production applications as theExtended model. This is what we expect because queueing is de-liberately minimized in production applications (see Figure 3 andthe discussion in Section 3.1). For the heavily-loaded PetStore data,in which queueing is a first-order effect, the Extended modelper-forms substantially better than the Basic model (22% vs. 32%whenLAR is used).
Third, normalized aggregate error is lower when LAR regressionis used. Outliers (extreme data values) are present in both data setsand are particularly numerous and large in the PetStore data. LARis a robust estimation procedure and is less sensitive to outliers thanOLS, and therefore yields more accurate models according toouraccuracy measure.
We also evaluated model variants that include an intercept termin Equations 2 and 4. Such models are “wrong” from a queueing-theoretic perspective because they imply nonzero aggregate responsetimes even when no transactions occur. However an interceptcanincrease retrospective accuracy but cannot reduce it, so wemightbe tempted to include one if we are willing to trade “sanity” foraccuracy. We found that the benefits of including an intercept arevery limited, and that the principled models with no intercept arenearly as accurate.
0.1
1
10
102
103
104
105
0.1 1 10 102 103 104 105fitte
d su
m o
f res
pons
e tim
es (
seco
nds)
actual sum of response times (seconds)
normalepisode 1episode 2episode 3
y = xy = x/2y = 2x
Figure 8: A performance anomaly in the “FT” application.
VDR ACME PetStoreScalar TMIX Scalar TMIX Scalar TMIX
∑i |ei |∑i yi
Basic 0.1621 0.1226 0.1749 0.1470 0.6528 0.6257Composite 0.1606 0.1218 0.1723 0.1305 0.3702 0.4173median|ei |/yi
Basic 0.1418 0.0963 0.1724 0.1378 0.5247 0.5056Composite 0.1420 0.0931 0.1664 0.1230 0.1708 0.2365
Table 3: Prospective prediction accuracy: Normalized aggre-gate error ∑i |ei |/∑i yi and median|ei |/yi .
4.2.2 Application: Performance Anomaly DetectionOur previous work explains how accurate retrospective explana-
tory performance models can be useful [23]. The most obviousap-plication isperformance anomaly detection, i.e., identifying whenperformance is surprising, given workload. Knowing whether work-load explains performance can guide our choice of diagnostic tools:Ordinary overload might recommend bottleneck analysis, whereasdegraded performancenot explained by workload might suggest afault in application logic or configuration.
We have shown that a real performance bug episode in a realdistributed production application appears as a prominentperfor-mance anomaly. Figure 8 shows a scatterplot of(yi , yi) pairs gen-erated by a Basic model of the “FT” application during a periodwhen application operators reported episodes of a performance bugdue to a misconfiguration in a concurrency parameter. One episodecorresponds to the prominent clusters of points in the lowerrightcorner of the figure. FT is a globally-distributed enterprise applica-tion that resembles VDR in several respects. See [23] for details onthe FT application and on this case.
Model calibration takes under one second for large data sets,so our method can be used to detect anomalies in real time bysimply recomputing a new model at the conclusion of each timeinterval (e.g., every 5 minutes) using a large moving windowofhistorical data (e.g., from the previous week or month). Thedatapoint corresponding to the most recent interval may then be deemedanomalous if the overall accuracy of the model is good but themostrecent performance observationyi disagrees substantially with themodel’s fitted value ˆyi .
4.2.3 Prospective PredictionTable 3 summarizes predictive accuracy results for our Basic and
Composite models calibrated using LAR regression; the table alsoincludes Scalar variants of both models. We do not present resultsfor OLS calibration because they do not alter the qualitative con-clusions we drew from Table 2: LAR works better, sometimes byasubstantial margin.
In addition to normalized aggregate error, Table 3 shows theal-ternative accuracy measure discussed in Section 3.4: the median ofthe distribution of normalized absolute residuals|ei |/yi . The twomeasures differ in how they penalize inaccuracy. Normalized ag-gregate error severely punishes even a single large residual but may“forgive” many small residuals, even those where|yi − yi | is largein relation toyi . Our other accuracy measure, median|ei |/yi , hasthe opposite tendency: It forgives a large residual if the correspond-ing yi is also large, but it penalizes us if the residual is often largein relation toyi .
Our prospective prediction results are consistent with ourretro-spective explanation results: First, transaction mix models (TMIX)outperform their Scalar counterparts by a very large marginfor bothreal production applications by both accuracy measures. Even theBasic TMIX achieves normalized aggregate error under 15% forboth real production data sets, and its individual performance pre-dictions yi are within 14% of the true valueyi half of the time.The Basic model leaves little room for improvement when appliedto real production applications; as we would expect, the Compos-ite model offers relatively modest accuracy improvements for thereal production applications, which are intentionally very lightlyloaded (Figure 3). Queueing delays are likely to be small in rela-tion to service times in such situations, so we gain relatively littleby modeling queueing. However for the deliberatelyoverloadedPetStore the situation is very different, as we would expect: TheComposite model consistently achieves far better accuracyby bothof our accuracy measures.
The relative benefits of incorporating transaction mix and queu-ing in the testbed application and the production applications areconsistent with our expectations: PetStore is heavily loaded andhas few transaction types; as we expect, the greatest increase inaccuracy occurs when we move from the Basic model to the Com-posite model. The production applications are lightly loaded andhave many transaction types; not surprisingly, the greatest increasein accuracy occurs when we move from a Scalar model to a TMIXmodel.
The combined benefits of accounting for both queueing and trans-action mix in production applications are striking. Compared withthe TMIX/Composite models, the Scalar/Basic models have sub-stantially worse normalized aggregate error: 33% greater error forVDR and 34% greater error for ACME. The differences are evenlarger when we consider median|ei |/yi : 52% greater error for VDRand 40% greater error for ACME.
Figures 9, 10, and 11 illustrate both retrospective and predictiveaccuracy for a Composite model of the VDR application calibratedwith LAR regression. In all cases, retrospective fitted values orresiduals are shown in blue and prospective fitted values areshownin red. Figure 9 presents a time series of observed aggregatere-sponse timesyi overlaid on fitted values ˆyi . Overall, the latter trackthe former quite closely. Residuals are not markedly largerforprospective performance prediction than for retrospective perfor-mance explanation. For the VDR data set, our model generalizeswell from historical data used for calibration to future workloaddata used for prediction.
Figure 10 shows a scatterplot of(yi , yi) pairs. The three straightdiagonal lines in the figure are they = x diagonal, indicating per-fect prediction, flanked byy = 2x andy = x/2. Although observedvaluesyi range over three orders of magnitude, fitted values ˆyi al-most always agree to within a factor of two, and are usually within15% of the trueyi value. Finally, Figure 11 shows the full distri-butions of|ei |/yi for both explanatory and predictive models. OurComposite model is highly accurate for retrospectively explainingperformance, and the figure shows that residuals remain remark-
0 500 1000 1500 2000
050
010
0015
0020
00
5−minute interval (train+test)
sum
_of_
resp
_tim
esactualfitted (on train)predicted (on test)
2 5 10 20 50 100 500 2000
5 e
−01
5 e
+00
5 e
+01
5 e
+02
Actual sum_of_resp_times
Pre
dict
ed s
um_o
f_re
sp_t
imes
on train dataseton test dataset
0.0 0.5 1.0 1.5
0.0
0.2
0.4
0.6
0.8
1.0
Absolute Percentage Error (APE): |Predict−Actual|/Actual
Cum
ul. D
ensi
ty F
unct
ion
(CD
F)
on train dataseton test dataset
Figure 9: Time series ofyi and yi . Figure 10: Scatterplot of (yi , yi) pairs. Figure 11: CDF of |ei |/yi .
VDR ACME PetStoreBasic 0.1226 0.1470 0.6257naıveUr 0.1221 0.2193 0.5794Composite 0.1218 0.1305 0.4173
Table 4: Incorporating utilization naıvely vs. correctly: Nor-malized aggregate error.
ably small when the model is used to predict performance based ontransaction mix.
4.2.4 Generalizing to New ConditionsWe performed an additional experiment to evaluate the ability of
our Composite model to predict performance in a real applicationunder workload conditions very different from those of the cali-bration data. We sorted the VDR data set in ascending order onapplication server CPU utilization. The first half of the resultingre-ordered data set contains time intervals during which CPU uti-lization on the app server was very low. The second half containstime intervals during which utilization was much higher. Figure 12shows the distributions of CPU utilizations in both halves of there-ordered VDR data set.
We calibrated a Composite model on the first half of the re-ordered VDR data, when load was very light (CPU utilization be-tween 4% and 18%). We then evaluated the model’s predictiveaccuracy on the second half, when load was much heavier (utiliza-tion between 18% and 45%). The model’s predictive accuracy wasremarkably high under this challenging test: normalized aggregateerror∑i |ei |/∑i yi was under 9.7%; most predictions ˆyi were within8% of the true valueyi . Furthermore, the model’s predictive accu-racy did not vary with load: Figure 13 shows a scatterplot of errorper time interval|ei |/yi versus CPU utilization on the app server.The figure shows that accuracy is not noticeably worse under highutilization (i.e., there is no upward trend to the right).
4.2.5 Modeling UtilizationA nonspecialist in queueing theory might wonder why we do
not simply incorporate resource utilizations into our performancemodel by addinglinear Ur terms rather than the mysteriousU2/(1−U) terms of Equation 4. Table 4 compares the predictive accuracyof our Basic and Composite models with a model that incorporatesutilization in the naıve way. We see that the naıve approach some-times improves upon our Basic model. However the “correct” ap-proach of our Composite model yields still better accuracy.Severalsimilar cases not reported here tend toward the same conclusion:Embellishing the Basic model in haphazard ways sometimes offersmodest advantages, but amendments with sound theoretical justi-fications (as in our Extended and Composite models) yield betterresults overall.
5. PERFORMANCE-AWARE APPLICATIONCONSOLIDATION
Enterprise systems that comprise multiple applications often ex-ecute each sub-application on separate machines to isolatethe ef-fects of software faults and workload spikes. Compared withsuchmachine-granularity application boundaries, application consolida-tion (i.e., executing multiple applications on one machine) has sev-eral advantages including better resource utilization andlower man-agement and maintenance overheads. However, workload fluctua-tions in consolidated environments can have complex effects onapplication-level performance that reduce the overall predictabilityof the system. In this section, we use our transaction mix model topredict application-level performance amidst contentionfor sharedphysical resources.
5.1 Consolidation ModelWe predict system-wide performance in consolidated environ-
ments by combining the Composite transaction mix models of eachtarget application running in isolation. We concatenate the transac-tion mix vectors and sum the resource utilizations in the Compositetransaction mix models of the target applications. More formally,the system-wide sum of response times for two consolidated appli-cations in intervali is:
yi =n′+n′′
∑j=1
α jNi j +∑r
(
1λi
(U ′ir +U ′′
ir )2
1− (U ′ir +U ′′
ir )
)
·n′+n′′
∑j=1
Ni j
where superscripts (′ and ′′) distinguish between the two appli-cations. For brevity, we show a unifieda j and Nj which repre-sent concatenation of each application’s individual parameters. Thevariablen represents the number of transaction types,Ur representsthe utilization of resourcer, andλ represents the post-consolidationarrival rate of both applications. Note that the unificationof severalComposite models can trivially be extended to handle the consol-idation of more than two applications, though we do not provideempirical results on this more general case. We acknowledgethatadditive measures of resource utilization may not account for someadditional costs of resource sharing (e.g., context switching). Wehave achieved accurate performance predictions without consider-ing such effects, though their impact may become more significantas the degree of consolidation increases. Likewise, we assume anapplication’s resource requirements do not decrease afterconsoli-dation, which can happen if the target applications interact or sharedata.
The resulting transaction mix model can be manipulated to de-rive performance predictions for each application under consolida-tion by considering the sum of the transaction types correspondingto a target application. Specifically, we extract the per-application
0.0 0.1 0.2 0.3 0.4
0.0
0.2
0.4
0.6
0.8
1.0
App server utilization
Cum
ul. D
ensi
ty F
unct
ion
(CD
F)
0.20 0.25 0.30 0.35 0.40 0.45
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
App server utilization
|Act
ual −
Pre
dict
ed|/A
ctua
l
Figure 12: CDFs of app server CPU utilization in both halvesof re-ordered VDR data set.
Figure 13: Prediction error |ei |/yi versus app server CPU uti-lization in re-ordered VDR data set.
performance under consolidation as follows:
y′i =n′
∑j=1
α jNi j +∑r
(
1λi
(U ′ir +U ′′
ir )2
1− (U ′ir +U ′′
ir )
)
·n′
∑j=1
Ni j .
y′′i =n′+n′′
∑j=n′+1
α jNi j +∑r
(
1λi
(U ′ir +U ′′
ir )2
1− (U ′ir +U ′′
ir )
)
·n′+n′′
∑j=n′+1
Ni j .
5.2 Realistic Workload GenerationNot surprisingly, the ACME and VDR administrators did not al-
low us to perform consolidation experiments using their applica-tions, so we built our own testbed from benchmark applications.Unfortunately, the workload generators shipped with our bench-mark applications produced stationary workloads, which, as wehave seen, differ qualitatively from workloads observed inthe wild.We developed our own workload generator which applies realisticnonstationary workloads to our testbed by mimicking the transac-tion type frequencies in the traces of real applications (e.g., ACMEand VDR). First, we separately ranked the transaction typesin bothour real and testbed applications according to their popularity inthe real trace and workload generator probabilities respectively. Wecreated a synthetic trace that imitates the the nonstationarity of realworkloads by replacing each transaction type in the real trace withthe transaction type with the same popularity rank in the testbedapplication. Finally, our workload generator also mimics the sea-sonality of real workloads by varying the request rate accordingto a sawtooth pattern. Henceforth references to a real application(i.e., ACME or VDR) preceded by “M-” will indicate a mimickedworkload based on the named dataset.
5.3 EvaluationOur testbed consists of two benchmark applications. The Rice
University Bidding System (RUBiS) [38] is an online auctionbench-mark that consists of 22 transaction types that provide servicessuch as browsing for items, placing bids, or viewing a user’sin-formation. The StockOnline [42] trading system comprises sixtransaction types corresponding to buying, selling, viewing prices,viewing holdings, updating account information, and creating newusers. Each application runs on top of the JBoss 4.0.2 applicationserver [21] and accesses MySQL [32] as the back-end database.The application server and database run on separate machines. Re-source consumption at the database is negligible in our testbed, sowe focus on the consolidation of the application server tier.
Our experiments are run on a three-machine cluster. Each nodeconsists of four 2.4-GHz CPUs with Intel Hyperthreading technol-ogy and 6 GB of main memory. We use the Linux 2.6.9 kernel
distributed by Red Hat. All of the experiments mentioned in thissection were run for five hours. Matching our observations oftheACME workload, each test was configured to exhibit eight com-plete sawtooth fluctuations in request rate. Unless otherwise noted,request rates were set for low CPU utilization (15–25%) wheneachapplication ran in isolation. Measurements were collectedevery 30seconds using the SAR system monitor. Each respective testbedapplication was calibrated using the M-ACME imitation workload.
Table 5 shows the accuracy of our consolidation predictionsus-ing the metrics discussed in Section 4.2.3. We show one consol-idation scenario where the calibration and evaluation workloadsare the same (M-ACME) and two scenarios in which the work-load under consolidation differs from the calibration workload (M-ACME). Under the same calibration and evaluation workloadstheaggregate error is 5.5% and 11% for RUBiS and Stock respectively.We also observe that the composition of transaction mix models forconsolidation is robust to changes in workload; predictingperfor-mance on completely different workloads from the calibration setstill yields aggregate errors within 9% and 14%. We note thatthenormalized errors for the StockOnline application are about twicethat of RUBiS. Upon further investigation, we found that StockOn-line is developed in a fashion such that transaction type does notprovide much distinctive information about resource demands. Inthis sense, StockOnline violates one of our fundamental assump-tions about applications, yet we are still able to report normalizedaggregate errors of between 11% and 14%.
Figure 14 shows the cumulative distribution of absolute percenterror when the RUBiS and StockOnline applications are consoli-dated and subjected to the M-VDR workloads. More than 97.6%and 74.2% of RUBiS and StockOnline predictions respectively arewithin 20% actual response time. The calibration and evaluation ofStockOnline on different workloads represents a significant chal-lenge for our model yet 98% of performance predictions are within40% of the actual response time.
We also wish to demonstrate the robustness of our consolidationprediction under heavy load. We increased the workload intensityby ramping up utilizations such that CPU utilizations reached 70%on the application server. Figure 15 shows the cumulative distri-bution of the normalized residual error in our heavy-load consol-idation scenario. We report normalized aggregate error of 12.8%and 11.7% for RUBiS and StockOnline respectively. While theseresults indicate the robustness of our consolidation model, we notethe limitations of our queueing model: As we noted with the Pet-Store application, our predictions are not intended for systems inwhich a resource is frequently saturated. Consolidation experi-ments under such situations gave performance predictions with un-
RUBiS Accuracy Stock AccuracyConsolidated Consolidated Normalized Median of Normalized Median of
RUBiS Stock Aggregate Normalized Aggregate Normalized
Workload Workload Error (∑i |ei |∑i yi
) Absolute Residuals (|ei |/yi ) Error (∑i |ei |∑i yi
) Absolute Residuals (|ei |/yi )
M-ACME M-ACME 0.0549 0.0383 0.1064 0.1263M-ACME M-VDR 0.0724 0.0706 0.1391 0.1184M-VDR M-VDR 0.0810 0.0626 0.1349 0.1145
Table 5: Application-level performance prediction accuracy in consolidated environments. Predictions are based on measurementsof each application in isolation. In all cases model calibration employed the M-ACME workload.
0.0 0.2 0.4 0.6 0.8
0.0
0.2
0.4
0.6
0.8
1.0
sumRespTime Absolute Percentage Error (APE): |Predict−Actual|/Actual
Cum
ulat
ive
Den
sity
Fun
ctio
n (C
DF
)
RubisStock
Figure 14: CDF of absolute percent error (|ei |/yi ) under differ-ent calibration and evaluation workloads.
acceptable (41%) normalized standard error; since this result is ex-pected it is not shown here.
6. RELATED WORKThis section reviews literature on several topics related to our
work: We begin with a survey of workload characterization stud-ies that have identified regularities in modern applicationwork-loads and then consider workload generators used in research and incommercial benchmarking. We then review instrumentation toolsand techniques that can be used to calibrate performance models,theoretical performance models themselves, and the application ofsuch models to practical performance prediction problems.Finally,we summarize the state of the art in server consolidation researchand practice. A review of literature on an important application ofour modeling technique, performance anomaly detection, isavail-able in [23]. A review of literature on LAR regression is availablein [24].
6.1 Workload CharacterizationPrevious research has characterized many aspects of workload in
modern transactional applications, e.g., Web server workloads [4]and Web user sessions [2]. Menasceet al.propose first-order Markovmodels of customer accesses at e-commerce sites [30] and describestatistically self-similar arrival patterns at such sites[29]. Someforms of nonstationarity have been observed in workloads, e.g., inInternet traffic [12] and in the diurnal cycles of requests toWebsites [5]. However to the best of our knowledge nonstationarity inthe mix of application-level transaction types is not considered inprevious research.
Many characterization studies lead directly to prescriptions forimproved performance. Breslauet al. show that the Zipf-like dis-tribution of Web document access frequencies implies that acertain
0.0 0.2 0.4 0.6 0.8
0.0
0.2
0.4
0.6
0.8
1.0
sumRespTime Absolute Percentage Error (APE): |Predict−Actual|/Actual
Cum
ulat
ive
Den
sity
Fun
ctio
n (C
DF
)
RubisStock
Figure 15: CDF of absolute percent error (|ei |/yi ) under heavynon-saturating workload.
Web cache removal policy is optimal [11]. Arlitt & Williamson findsupport for size-based Web cache removal policies in the size dis-tributions of Web documents [5]. Junget al.characterize transientoverload events (“flash crowds”) and suggest ways for systemde-fenses to distinguish them from denial-of-service attacks[22].
Clustering techniques are frequently employed to simplifywork-load for performance modeling [6]. Often workload units areclus-tered according to their resource demands as in Magpie [7] and inEspositoet al. [18], but occasionally clustering is applied to otheraspects of workload, e.g., customers [51].
Our contribution is to recognize that in many modern applica-tions transaction types effectively cluster workload elements ac-cording to their resource demands, and that nonstationarity in trans-action mix presents an opportunity to calibrate performance modelswithout invasive instrumentation or controlled benchmarking.
6.2 Workload GenerationCommercial synthetic workload generators are used to evalu-
ate the performance of live applications and systems [31] and instandard benchmarks such as TPC-W [47]. To the best of ourknowledge, such tools generate workloads with stationary trans-action mixes. Schroederet al. survey workload generators usedin research, emphasizing the distinction between open and closedgenerators [40]. These tools allow users to configure arrival rateparameters but do not facilitate the generation of workloads withnonstationary transaction mixes. To the best of our knowledge, theonly workload generator that does so is the SWAT tool of Krishna-murthyet al. [25, 26]. SWAT employs sophisticated mathematicalprogramming techniques to construct a mix of sessions that con-forms to user-specified aggregate workload criteria; nonstationarityfollows from the use of recorded sessions from a real productionsystem. SWAT strives to provide both the fidelity of simple trace
replay with the control and configurability of conventionalwork-load generators.
Our contribution is to recognize that the stationary workloadsproduced by conventional workload generators make parameter es-timation for performance modelingmore difficult than naturally-occurring nonstationary workloads. Indeed, our results show thatcontrolled experiments using synthetic workload generators are notnecessary to calibrate performance models. Accurate performancemodels may be calibrated using the kinds of lightweight passivemeasurements routinely collected in today’s production systems—transaction logs and utilization logs.
We furthermore report that the simple expedient of replayingtransaction logs from real production applications with the trans-action types re-named to suit a testbed application (Section 5.2)yields good results.
6.3 Instrumentation & MeasurementA wide range of commercial performance measurement tools are
available. Sauerset al. provide a candid survey of the strengthsand limitations of one vendor’s products [39]. Some tools pro-vide insight into transaction execution paths and resourceusagenoninvasively—without application source code modifications—via instrumented middleware [19]. If source code is available, ap-plications may be systematically instrumented to record more de-tailed transaction execution information, e.g., using theARM in-strumentation standard [45]. Instrumentation to characterize trans-action resource demands in greater detail and with lower overheadremains an active research area [7,27,46].
Our contribution is to recognize that transaction mix nonstation-arity in real-world workloads enables us to use very lightweightmeasurements to characterize the resource demands of transactiontypes for the purpose of calibrating performance models.
6.4 Queueing TheoryQueueing networks are the subject of a large theoretical liter-
ature; see Bolchet al. for a lengthy survey [10]. Jain describesapplications of queueing theory to computer system performanceanalysis [20]. The approaches that Jain surveys differ fromoursin several key respects: Jain emphasizes the design of controlledexperiments for performance analysis, and an underlying assump-tion throughout much of the book is that systematic benchmarkingis possible. Furthermore most of Jain’s queueing network modelsassume far more detailed information about transaction behaviorthan is available in many practical situations. For instance, it is fre-quently assumed that the number of times a transaction visits var-ious resources and the distribution of service times at eachstationcan be measured directly. Our work proceeds from the assump-tion that lightweight passive measurements of transactionresponsetimes and resource utilizations are all that is available.
Operational analysis is a branch of queueing theory that attemptsto avoid probabilistic assumptions about system workload (e.g.,Poisson arrivals and exponentially-distributed service times) andrely solely upon measurable quantities [17]; Little’s Law and theUtilization Law are classic examples of operational laws. MeanValue Analysis (MVA) restricts attention to the averages (as op-posed to the full distributions) of performance measures [35]. Ro-lia & Sevcik introduce a variant of MVA designed to accommodatesoftwareservers in addition to conventional hardware service sta-tions [36]. Like conventional MVA, this method pertains to closedqueueing networks, whereas we employ open network models.
Generalizations of queueing-theoretic models and MVA addressmulticlassnetworks [9]. Workload classes are often interpretedas categories such as “batch,” “terminal,” and “transaction,” but
classes can also be used to represent different transactiontypes.One problem with existing multiclass methods for our purposes isthat most assume a closed network—our production traces do notinclude sufficient information about client sessions for usto employa closed model. Another problem is that the computational cost ofcomputing exact solutions to MVA using conventional algorithmsincreases rapidly with the number of classes. A more efficient algo-rithm has appeared recently [13] but it is formidably complex anddifficult to implement. Multiclass models with more than a handfulof classes are rarely used in practice due to their complexity andthe computational cost of computing solutions.
Our contribution is to introduce a computationally tractable andconceptually simple performance model for open networks thattakes transaction mix into account, that models multiple servicecenters, that is easy to calibrate, and that yields accurateresponsetime predictions for real production applications.
6.5 Applied Performance PredictionThis section reviews in depth two recent papers that apply queue-
ing models to distributed applications, highlighting similarities andcontrasts with respect to our work. We refer the reader to their ex-cellent literature reviews for recent, broad, and thoroughsurveys ofrelated work in this field [41,48].
Urgaonkaret al. model multi-tier Internet services as product-form queueing networks and employ mean value analysis to com-pute average response times [48]; in some respects this workis sim-ilar to that of Liuet al. [28]. The Urgaonkaret al. model assump-tions differ from ours in several details. For instance, Urgaonkaret al. explicitly model concurrency limits whereas we do not. Weassume an open queueing network whereas Urgaonkaret al. as-sume a closed network. We explicitly model distinct physical re-sources such as CPUs and disks whereas Urgaonkaret al.associatea single queue with each tier. The models differ in their assump-tions about how requests recirculate among tiers; compare our Fig-ure 6 with their Figure 3 [48, p. 294]. An important difference isthat their method requires more diverse model parameter estimatesthan ours, including request visit ratios at each tier, service timesat each tier, user think times, and certain other parametersrelatedto congestion effects. Urgaonkaret al. report that their approachyields accurate average response time estimates for two sample ap-plications (RUBiS and Rubbos) subjected to stationary syntheticworkloads in a testbed environment; they do not report validationresults on real production applications.
Stewart & Shen present a performance model of distributed In-ternet applications based on “profiles” that summarize how appli-cation software components and their workloads place demands onunderlying system resources [41]. Their model also accounts forinter-component communications and component placement.Thiswork shares some features in common with our approach. For in-stance, Stewart & Shen account for waiting times at servers us-ing an M/G/1 model; we employ a similar model in Equation 4.They estimate the resource demands of components by fitting lin-ear models to benchmark data. However, they describe workloadby a constant scalar arrival rate, whereas we use a time-varying vec-tor of per-type transaction counts. Stewart & Shen report that theirmost sophisticated model variant predicts average response timesto within 14%. Their validation uses testbed applications (RUBiSandStockOnline) and stationary synthetic workloads.
An important difference with respect to our work is that themethod of Stewart & Shen requires very extensive calibration: Theresource consumption profile of each component must be estimatedvia controlled benchmark experiments, and inter-component com-munication overheads must also be measured. They place each
profiled componenton a dedicated machineduring calibration andrequire at least one benchmark run per component. For their fullmodel, O(N2) benchmark runs are required to estimate pairwiseinter-component communication costs [41, p. 75]. We exploit non-stationarity to obtain similar performance profiles using only light-weight passive measurements of running production systems: Thecoefficients of our utilization model (Equation 5) correspond closelyto those in the “component resource profiles” of Stewart & Shen(see Figure 2 and Tables 1 and 2 in [41]. Another difference isthat we do not require knowledge of internal application compo-nent structure; we use only externally-visible transaction types.
We emphasize two important differences between our evaluationexperiments and those presented in Urgaonkaret al.and in Stewart& Shen. First, as noted above, we have employed two real produc-tion traces for our evaluations; they have used only testbedappli-cations. The workload of our applications isnonstationaryin sev-eral key parameters, including both workload intensity andtrans-action mix. By contrast, Stewart & Shen and Urgaonkaret al.em-ploy synthetic workloads reminiscent of classic steady-state bench-marks both for model calibrationand for evaluation. The transac-tion mixes in their synthetic workload (e.g., the buy:browse ratioin their synthetic e-commerce workloads) remainconstantduringboth calibration and evaluation. We believe that our nonstationaryworkload yields a far more challenging and more realistic test of aperformance model’s generalizability and predictive accuracy.
Our empirical evaluations could not include comparisons withthe methods of Stewart & Shen and of Urgaonkaret al. for tworeasons: First, the input-output behavior of the three models is suf-ficiently different to preclude a true apples-to-apples comparison.More importantly, the other two approaches require far moreexten-sive calibration data than is available in our data sets (ourcurrenttestbed at HP does not permit the same instrumentation as used inStewart & Shen; e.g., kernel modifications are not allowed).How-ever we do compare our preferred approach with alternativesthat,like the models of Stewart & Shen and of Urgaonkaret al., employa scalar measure of workload intensity (Section 3.2.4). We foundthat transaction mix models offer substantially higher accuracy thantheir Scalar counterparts (Section 4.2).
6.6 ConsolidationThe computing trends of the 1980s and 1990s led to decentral-
ized IT infrastructures that can be more difficult to manage andless cost-effective than their centralized predecessors.Server con-solidation attempts to increase resource utilization while reducingthe costs of hardware, data center floor space, power, cooling, andadministration. The resource cost benefits of consolidation aloneare potentially attractive: Andrzejaket al. studied CPU utilizationin six enterprise data centers containing roughly 1,000 CPUs andfound that consolidation could reduce the number of CPUs neededduring peaks by 53% and the mean number required by 79% [1].
Administrators know that systems can be overloaded if the sumof consolidated application resource demands is excessive. Prac-titioners are advised to rely on rough guidelines for total resourceutilization, e.g., “avoid peak CPU utilization over 70%” [14]. Com-mercial capacity planning decision support aids may employmoresophisticated time-series analysis of pre-consolidationhistorical data,but their suggestions are based on considerations of resource uti-lizations [44,50].
Recent research on consolidation decision support also basesrecommendations on resource utilization. Roliaet al. analyze his-torical utilization data to provide statistical guarantees on post-con-solidation utilization [37]. Urgaonkaret al.profile applications ondedicated nodes to estimate resource demands and “pack” appli-
cations to maximize revenue while controlling the potential for re-source overload [49].
The state of the art in research and in practice is to base consol-idation decisions on considerations of resource utilizations and ei-ther ignore application-level workload or model it as a scalar quan-tity. This is problematic because the relationship betweenapplication-level performance and utilization is complex, and because both de-pend on transaction mix. Our contribution is a practical wayto ob-tain accurate predictions ofresponse timesin transactional applica-tions, thus allowing consolidation decisions to consider application-level performance as well as system resource utilization.
7. CONCLUSIONSThe global geographic distribution, organizational decentraliza-
tion, opaque component structures, and unprecedented scale of mod-ern application architectures confound performance modeling inchallenging new ways. Performance prediction in business-criticalapplications, however, remains an important problem due tothegrowing economic importance of these applications. This paperpresents a practical, versatile, and accurate approach to predictingapplication-level response times in complex modern distributed ap-plications. Our method exploits naturally-occurring workload non-stationarity to circumvent the need for invasive instrumentation orcontrolled benchmarking for model calibration. It relies solely onmeasurement data that is routinely collected in today’s productionenvironments. Our method can be adapted to a wide range of ap-plications, and calibrated models generalize well to new regions ofworkload/performance space. It is novel in its use of transactionmix to predict performance, and we have shown that transactionmix is a far more powerful predictor of application performanceunder realistic conditions than scalar workload volume.
Our empirical results show that our method predicts responsetimes in real production applications to within 16% by two very dif-ferent accuracy measures. A model of a real production applicationcalibrated under light load predicts performance under heavy loadto within 10%. Our results show that if accurate workload fore-casts are available, they can be mapped directly to accurateperfor-mance predictions. Furthermore we predict response times of con-solidated applications to within 4% to 14% based on passive pre-consolidation measurements, even when workload changes dramat-ically between calibration and evaluation. Whereas existing ap-proaches to consolidation decision support consider only resourceutilization, our approach enables application-level response timesto guide consolidation decisions.
AcknowledgmentsMartin Arlitt supplied the ACME data set. Ira Cohen, Julie Symons,and the second author collected the FT and PetStore data setsforseparate projects [15,16]. We thank the operators of the ACME, FT,and VDR production systems for providing anonymized trace data.Hsiu-Khuern Tang answered questions on statistical matters. ArjunNath provided valuable assistance to our testbed experiments. Weare deeply grateful to Narayan Krishnan and Eric Wu for theirex-traordinarily assistance in setting up and administering the clusterwe used for consolidation tests. We thank David Oppenheimer,Jerry Rolia, and Bhuvan Urgaonkar for many insightful discus-sions of performance modeling and its applications, and we thankKim Keeton, Kai Shen, Zhikui Wang, Xiaoyun Zhu, Sharad Sing-hal, and the anonymous reviewers for reading drafts and offeringmany helpful suggestions. The first author acknowledges supportfrom the U.S. National Science Foundation CAREER Award CCF-0448413.
8. REFERENCES[1] A. Andrzejak, M. Arlitt, and J. A. Rolia. Bounding the resource
savings of utility computing models. Technical ReportHPL-2002-339, HP Labs, Dec. 2002.
[2] M. Arlitt. Characterizing Web user sessions.Performance EvaluationReview, 28(2), Sept. 2000.
[3] M. Arlitt, D. Krishnamurthy, and J. A. Rolia. Characterizing thescalability of a large web-based shopping system.ACM Trans. onInternet Tech, 1(1):44–69, Aug. 2001.
[4] M. Arlitt and C. Williamson. Web server workload characterization:The search for invariants. InSIGMETRICS, May 1996.
[5] M. F. Arlitt and C. L. Williamson. Internet Web servers: Workloadcharacterization and performance implications.IEEE/ACM Trans. onNetworking, 5(5):631–644, Oct. 1997.
[6] H. P. Artis. Capacity planning for MVS computer systems.Performance Evaluation Review, 8(4):45–62, 1979.
[7] P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using Magpie forrequest extraction and workload modelling. InOSDI, pages 259–272,Dec. 2004.
[8] I. Barrodale and F. Roberts. An improved algorithm for discrete L1linear approximations.SIAM Journal of Numerical Analysis,10:839–848, 1973.
[9] F. Baskett, K. M. Chandy, R. R. Muntz, and F. G. Palacios. Open,closed, and mixed networks of queues with different classesofcustomers.J. ACM, 22(2):248–260, Apr. 1975.
[10] G. Bolch, S. Greiner, H. de Meer, and K. S. Trivedi.QueueingNetworks and Markov Chains. Wiley, 1998.
[11] L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web cachingand Zipf-like distributions: Evidence and implications. InINFOCOM, Mar. 1999.
[12] J. Cao, W. S. Cleveland, D. Lin, and D. X. Sun. On thenonstationarity of Internet traffic. InSIGMETRICS, pages 102–112,June 2001.
[13] G. Casale. An efficient algorithm for the exact analysisof multiclassqueueing networks with large population sizes. InSIGMETRICS,pages 169–180, June 2006.
[14] A. Cockcroft and B. Walker.Capacity Planning for Internet Services.Sun Press, 2001.
[15] I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase.Correlating instrumentation data to system states: A building blockfor automated diagnosis and control. InOSDI, Oct. 2004.
[16] I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox.Capturing, indexing, clustering, and retrieving system history. InSOSP, Oct. 2005.
[17] P. J. Denning and J. P. Buzen. The operational analysis of queueingnetwork models.ACM Computing Surveys, 10(3):225–261, Sept.1978.
[18] A. Esposito, A. Mazzeo, and P. Costa. Workload characterization fortrend analysis.Performance Evaluation Review, 10(2), July 1981.
[19] Hewlett-Packard. OpenView Transaction Analyzer, Sept. 2006.http://h20229.www2.hp.com/products/tran/.
[20] R. Jain.The Art of Computer Systems Performance Analysis. JohnWiley & Sons, 1991.
[21] The JBoss J2EE Application Server.http://www.jboss.com.[22] J. Jung, B. Krishnamurthy, and M. Rabinovich. Flash crowds and
denial of service attacks: characterization and implications for CDNsand Web sites. InWWW, May 2002.
[23] T. Kelly. Detecting performance anomalies in global applications. InUSENIX WORLDS, Dec. 2005.
[24] T. Kelly and A. Zhang. Predicting performance in distributedenterprise applications. Technical Report HPL-2006-76, HP Labs,May 2006.http://www.hpl.hp.com/techreports/2006/HPL-2006-76.html.
[25] D. Krishnamurthy, J. A. Rolia, and S. Majumdar. SWAT: A tool forstress testing session-based Web applications. InComputerMeasurement Group Conf., Dec. 2003.
[26] D. Krishnamurthy, J. A. Rolia, and S. Majumdar. A syntheticworkload generation technique for stress testing session-basedsystems.IEEE Trans. Software Engineering, 32(11):868–882, Nov.2006.
[27] J. R. Larus. Whole program paths. InPLDI, May 1999.
[28] X. Liu, J. Heo, and L. Sha. Modeling 3-tiered web applications. InProc. MASCOTS, Sept. 2005.http://www.cs.mcgill.ca/∼xueliu/publications/MASCOTS05 Modeling.pdf.
[29] D. Menasce, V. Almeida, R. Riedi, F. Ribeiro, R. Fonseca, andW. Meira. In search of invariants for e-business workloads.In ACME-Commerce Conf., Oct. 2000.
[30] D. A. Menasce, V. A. F. Almeida, R. Fonseca, and M. A. Mendes. Amethodology for workload characterization of e-commerce sites. InACM E-Commerce Conf., Nov. 1999.
[31] Mercury Interactive. Loadrunner load tester, Sept. 2006.http://www.mercury.com/us/products/performance-center/loadrunner/.
[32] MySQL database.http://www.mysql.com.[33] J. Neter, M. H. Kutner, C. J. Nachtsheim, and W. Wasserman.
Applied Linear Statistical Models. Irwin, fourth edition, 1996.[34] C. R. Rao and H. Toutenburg.Linear Models: Least Squares and
Alternatives. Springer, 1999.[35] M. Reiser and S. Lavenberg. Mean value analysis of closed
multichain queuing networks.J. ACM, 27(2):313–322, Apr. 1980.[36] J. A. Rolia and K. C. Sevcik. The method of layers.IEEE Trans.
Software Engineering, 21(8):689–700, Aug. 1995.[37] J. A. Rolia, X. Zhu, and M. Arlitt. Resource access management for a
resource utility for commercial applications. InInt’l Sympos. onIntegrated Network Mgmt. (IM), Mar. 2003.
[38] Rice University Bidding System (RUBiS), Mar. 2004.http://rubis.objectweb.org/.
[39] R. F. Sauers, C. P. Ruemmler, and P. S. Weygant.HP-UX 11i Tuningand Performance. Prentice Hall, 2004.
[40] B. Schroeder, A. Wierman, and M. Harchol-Balter. Open versusclosed: A cautionary tale. InNSDI, pages 239–252, May 2006.
[41] C. Stewart and K. Shen. Performance modeling and systemmanagement for multi-component online services. InNSDI, pages71–84, May 2005.
[42] The StockOnline Benchmark.http://forge.objectweb.org/projects/stock-online.
[43] D. W. Stroock.An Introduction to Markov Processes. Springer, May2005.
[44] R. Talaber. Server consolidation assessments with VMwareCapacityPlanner, Oct. 2005.http://download3.vmware.com/vmworld/2005/pac196.pdf.
[45] The Open Group. Application response measurement (ARM), Sept.2006.http://www.opengroup.org/management/arm.
[46] E. Thereska, B. Salmon, J. Strunk, M. Wachs, M. Abd-El-Malek,J. Lopez, and G. R. Ganger. Stardust: Tracking activity in adistributed storage system. InProc. SIGMETRICS, pages 3–14, June2006.
[47] Transaction Processing Performance Council. TPC-W benchmark,Apr. 2005.http://www.tpc.org/tpcw/.
[48] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi.An analytical model for multi-tier Internet services and itsapplications. InSIGMETRICS, pages 291–302, June 2005.
[49] B. Urgaonkar, P. Shenoy, and T. Roscoe. Resource overbooking andapplication profiling in shared hosting platforms. InOSDI, Dec.2002.
[50] VMWare. Server capacity planning and consolidation, Oct. 2005.http://www.vmware.com/news/releases/vac services.html.
[51] Q. Wang, D. J. Makaroff, and H. K. Edwards. Characterizingcustomer groups for an e-commerce website. InACM E-CommerceConf., May 2004.
[52] R. R. Wilcox.Introduction to Robust Estimation and HypothesisTesting. Elsevier, second edition, 2005.
$Id: wpm.tex,v 1.115 2007/04/10 23:23:37 kterence Exp $