managing web server performance with autotune agentsbrecht/servers/readings-new/diao.pdf ·...

Managing Web serverperformance withAutoTune agents

by Y. DiaoJ. L. HellersteinS. ParekhJ. P. Bigus

Managing the performance of e-commercesites is challenging. Site content changesfrequently, as do customer interests andbusiness plans, contributing to dynamicallyvarying workloads. To maintain goodperformance, system administrators musttune their information technology environmenton an ongoing basis. Unfortunately, doing sorequires considerable expertise and increasesthe total cost of system ownership. In thispaper, we propose an agent-based solutionthat not only automates the ongoing systemtuning but also automatically designs anappropriate tuning mechanism for the targetsystem. We illustrate this in the context ofmanaging a Web server. There we study theproblem of controlling CPU and memoryutilization of an Apache® Web server usingthe application-level tuning parametersMaxClients and KeepAlive, which are exposedby the server. Using the AutoTune agentframework under the Agent Building andLearning Environment (ABLE), we constructagents to fully automate a control-theoreticmethodology that involves model building,controller design, and run-time feedbackcontrol. Specifically, we design (1) a modelingagent that builds a dynamic system modelfrom the controlled server run data, (2) acontroller design agent that uses optimalcontrol theory to derive a feedback controlalgorithm customized to that server, and (3) arun-time control agent that deploys thefeedback control algorithm in an on-line real-time environment to automatically manage

the Web server. The designed autonomicfeedback control system is able to handle thedynamic and interrelated dependenciesbetween the tuning parameters and theperformance metrics with guaranteed stabilityfrom control theory. The effectiveness of theAutoTune agents is demonstrated throughexperiments involving variations in workload,server capacity, and business objectives. Theresults also serve as a validation of the ABLEtoolkit and the AutoTune agent framework.

The increasing complexity of computing systems andapplications demands a correspondingly larger hu-man effort for system configuration and performancemanagement. This manual effort can be time-consuming and error-prone, and requires highlyskilled personnel, making it costly. Autonomic com-puting1 uses the analogy of the human autonomicnervous system to suggest the use of a higher levelof automation and self-management capability incomputing systems.

The complexity and importance of developing au-tonomic computing systems has attracted research

�Copyright 2003 by International Business Machines Corpora-tion. Copying in printed form for private use is permitted with-out payment of royalty provided that (1) each reproduction is donewithout alteration and (2) the Journal reference and IBM copy-right notice are included on the first page. The title and abstract,but no other portions, of this paper may be copied or distributedroyalty free without further permission by computer-based andother information-service systems. Permission to republish anyother portion of this paper must be obtained from the Editor.

DIAO ET AL. 0018-8670/03/$5.00 © 2003 IBM IBM SYSTEMS JOURNAL, VOL 42, NO 1, 2003136

efforts of a theoretical as well as an applied nature.In particular, control theory is emerging as a prom-ising cornerstone to provide a rigorous mathemat-ical foundation for designing and analyzing auto-nomic controllers, which can reduce the work ofsystem administrators and provide guaranteed con-trol performance. Some applications of control the-ory to computing systems include flow and conges-tion control,2–5 differentiated caching and Webservice,6,7 multimedia streaming,8 Web server per-formance,9 e-mail server control,10,11 and distributedresource allocation on the grid.12 These applicationsall provide a degree of autonomic behavior by pro-viding algorithms to automatically control some as-pect of a computing system’s operation. However,a common theme in all this work is that applying con-trol techniques requires significant modeling and de-sign work, which is typically a manual processconducted by the system designer. In the spirit ofreducing all manual intervention, we propose to au-tomate this phase of the deployment of control sys-tems as well.

In this paper, we describe an agent-based autonomicfeedback control system that uses high-level inputsfrom the human system administrators to not onlycontrol a computing system, but also to automati-cally design a controller suitable for that system. Weillustrate this in the context of an Apache** Webserver. Using the Agent Building and Learning Envi-ronment (ABLE),13 AutoTune14 agents are built for(1) modeling the behavior of an Apache Web server,(2) designing the feedback control law, and (3) inon-line operation, adjusting the server parametersin response to workload variations. These agents co-operate at different phases of the life cycle of theautonomic control system to achieve the function ofautomatically controlling the Web server.

The remainder of the paper is organized as follows.The next section describes the background of serverself-tuning in the context of Apache Web servers.The section “Server self-tuning with AutoTuneagents” introduces ABLE-based AutoTune agentsand details the architecture and algorithms used toautomate server tuning. The experimental results aredescribed in the section “Experimental assessment,”comparing the performance of the Apache servercontrolled by the proposed AutoTune controller anda heuristic manual controller, particularly when sig-nificant workload variations exist. Finally, our con-clusions are presented.

Apache Web server and performancetuning

The Apache15 Web server is the most popular Webserver in use today,16 making resource managementfor such a server an important problem. Version 1.3.xof the server on UNIX** is structured as a master pro-cess and a pool of worker processes. The master pro-cess monitors the health of the worker processes andmanages their creation and destruction. The workerprocesses are responsible for communicating withWeb clients and generating responses, and oneworker process can handle at most one connectionat a time. The number of worker processes is lim-ited by the parameter MaxClients, thereby throttlingthe Web server’s throughput. Worker processes cy-cle through three states: idle, waiting, and busy. Aworker is in an idle state if no Transmission ControlProtocol (TCP) connection from the client has beenmade to it.

Once a TCP connection is accepted, the worker pro-cess is either waiting for a HyperText Transfer Pro-tocol (HTTP) request from the client, or is busy inprocessing the client request. According to persist-ent connections in HTTP/1.1, 17 the established TCPconnection remains open between consecutive HTTPrequests (which eliminates the overhead for settingup one connection for each request as in HTTP/1.0).This persistent connection can either be terminatedby the client or by the master process, if the waitingtime of a worker process exceeds the maximum al-lowed time specified by the parameter KeepAlive.

The performance of the Apache Web server can bemeasured by different metrics, such as end-user re-sponse times or utilization of various resources onthe server. Selection of appropriate performancemetrics depends not only on management objectivesbut also on metric availability. From the point of viewof guaranteeing quality of service, bounding the end-user response times is desired. However, end-userresponse time is a client-side metric, and additionalinstrumentation such as a probing station needs tobe added. This would increase server load and raiseother issues, including accuracy and recentness ofthe available measurements. (Refer to Reference 18for a discussion of and control strategies for man-aging the end-user response time.) In this paper, wequantify the server performance using server-sidemetrics, CPU utilization and memory utilization,which are easy to measure on the server and asso-ciated with business needs as well.

IBM SYSTEMS JOURNAL, VOL 42, NO 1, 2003 DIAO ET AL. 137

System administrators typically maintain a certainutilization level on the server, high enough to effi-ciently utilize system resources but not so high thatit causes thrashing and failures as a result of over-utilization. Good end-user response times are en-sured by reserving sufficient capacity to handle work-load surges. However, the system utilization cannotbe directly set in an Apache server. Instead, the ad-ministrators must operate indirectly by adjusting cer-tain tuning parameters, among which MaxClientsand KeepAlive are commonly used. A higher Max-Clients value allows the Apache server to processmore client requests, and increases both CPU andmemory utilizations. Also, decreasing the value ofKeepAlive potentially allows worker processes to bemore active, which directly results in higher CPU uti-lization and indirectly increases memory utilization,since more clients can connect to the server.9

In principle, the desired and feasible CPU and mem-ory utilizations can be achieved by properly select-ing the tuning parameters MaxClients and Keep-Alive, but in practice, it is time-consuming, error-prone, and skills-intensive to adjust these parametersmanually. Moreover, this tuning work has to be re-peated as the workload changes or the server is re-configured for more CPU and memory. A change ofWeb site contents may also affect the CPU and mem-ory usage per request and can also require differentMaxClients and KeepAlive settings. We illustrate thedrawbacks of manual tuning using modified versionsof our agents and the testbed, both of which are de-scribed later. In essence, our modifications to theApache server allow us to change the MaxClientsand KeepAlive values without restarting the server,and the agents provide the graphical user interface(GUI) for manually setting the values.

Suppose the administrator wants to have the desiredCPU level at 0.5 and memory at 0.6. Manually tuningthe Apache server is a trial-and-error process andcan be quite time-consuming. Due to the interrela-tionships between the tuning parameters and per-formance metrics, it may not be easy to find theproper KeepAlive and MaxClients settings.

In Figure 1, the values for the tuning parametersKeepAlive and MaxClients are shown in the top twoInspector windows, and the bottom two Inspectorwindows show the corresponding effects on the per-formance metrics CPU and memory utilization. Thesystem is running in our testbed and is subjected toa synthetic workload, both of which are describedin the section “Apache testbed and workload gen-

erator.” The y-axis shows the measured values, andthe x-axis indicates the time, which is measured bycontrol intervals (i.e., sampling intervals). The con-trol interval is five seconds; every five seconds thetuning parameters (if they are changed) are sent tothe Apache server, and the CPU and memory valuesof the Apache server are also provided to the Apacheadaptor for the system administrator to check thecontrol results.

As Figure 1 indicates, arbitrarily selecting Keep-Alive � 2 and MaxClients � 100 will not yield CPUand memory utilization close to the desired values.The tuning parameters and the performance met-rics are interrelated; for instance, increasing Max-Clients to 150 causes increases in both CPU and mem-ory utilizations. Manually tuning the Apache Webserver using these controls is possible by using thefollowing heuristic. If we increase MaxClients, bothCPU and memory utilization will increase. If we in-crease KeepAlive, CPU utilization will decrease.Thus, we can use MaxClients to adjust utilization un-til the memory is at the desired level, and then getthe desired CPU utilization by adjusting KeepAlive.Using these tuning heuristics, in order to achieve CPU� 0.5 and MEM � 0.6, after several tries the valuesMaxClients � 400 and KeepAlive � 10 are found,which drive CPU and memory close to the desiredvalues.

As mentioned earlier, a variation in workloads mayaffect this relationship between the tuning param-eters and system utilizations, and thus change the“optimal” values for KeepAlive and MaxClients. Forexample, as shown in Figure 2, when additional work-load (requests for dynamic Web pages) is includedaround the twentieth control interval, we observethat the previous setting of tuning parameters causesboth CPU and memory utilizations to deviate fromthe desired levels. Therefore, the system adminis-trator will need to tune the server parameters again,and this can become quite tedious if the workloadis changing frequently.

It is worth mentioning that in this paper we use adesired CPU level of 0.5 and memory of 0.6 simplyfor the purpose of illustrating the tuning process.How to choose the desired utilization level practi-cally depends on business needs and system config-urations, and thus is out of the scope of this paper.In this paper, we focus on how to automatically con-struct quantitative models and conduct tuning con-trol. This is important, since the relationship betweenthe performance metrics and tuning parameters is

DIAO ET AL. IBM SYSTEMS JOURNAL, VOL 42, NO 1, 2003138

only qualitatively known, and extensive manual tun-ing is required and needs to be repeated when work-load changes. In other words, the proposed Auto-Tune agents increase the automation level of servertuning by automating the lower-level tuning (e.g., ad-justing server parameters MaxClients and Keep-Alive). This will not eliminate the role of the systemadministrators, but can reduce their work and helpthem focus on higher-level performance metric goals(e.g., desired CPU and memory utilization).

Server self-tuning with AutoTune agents

Based on the previous section, it is clearly desirableto automate the adjustment of the MaxClients andKeepAlive values, both at system startup and on anongoing basis in response to changing workload.More broadly, we would like to reduce the work of

the system administrator by performing such tuningwork with an autonomic agent. Rather than constructa special-case agent for this particular set of tuningcontrols and this particular software system, we pro-pose a more general architecture that can be appliedto many other systems as well. We use the Apacheserver as an illustration for this general framework.

Our solution consists of multiple agents that auto-mate the entire methodology of controller design andalso perform the on-line system control. These agentsare implemented using the Agent Building andLearning Environment (ABLE), and in particular byimplementing specialized AutoTune agents. Below,we present an overview of ABLE and the basic Au-toTune architecture, followed by a description of ouragents and the control-theory-based algorithms that

Figure 1 Results of manually tuning the Apache Web server


are essential to exploit the learning and control abil-ities of the autonomic agents.

ABLE framework and base AutoTune agent. TheAgent Building and Learning Environment (ABLE)is a Java**-based toolkit for developing and deploy-ing hybrid intelligent agent applications.19 It providesa comprehensive library of intelligent reasoning andlearning components packaged as Java beans (knownas AbleBeans) and a lightweight Java agent frame-work to construct intelligent agents (known as Able-Agents). The AbleBean Java interface defines a setof common attributes (name, comment, state, etc.)and behaviors (standard processing methods such asinit�, reset�, process�, and quit�), which allowsAbleBeans to be connected to form AbleAgents. AJava Swing-based GUI, AbleEditor, is also providedfor creating and configuring AbleBeans, and for con-

structing and testing the AbleAgents built from them.For most AbleBeans, the user-interface is througha GUI component known as a “Customizer” that al-lows the user to set and view parameters related tothe bean.

The base AutoTune agent is a function-specific Able-Agent for autonomic computing. Inspired by humanbiology, the AutoTune agent is based on an archi-tecture that combines several elements that are use-ful in building systems to react to a dynamic envi-ronment. The AutoTune agent contains two basicbuilding blocks (AbleBeans): the AutotuneCon-troller bean and the AutotuneAdaptor bean, asshown in Figure 3. The AutotuneController bean de-fines control strategies (such as learning the behav-ior of the target system or providing actions to amendabnormal situations). Its Customizer GUI allows the

Figure 2 Effects of dynamic workloads on manually tuned Apache server


system administrator to configure the control strat-egy in advance or on the fly. The AutotuneAdaptorbean interfaces with the target system to get the ser-vice level metrics and to set the tuning parameters.Another Customizer GUI is provided for the systemadministrator to manually set the tuning parameters(for example, for testing purposes, or when the Au-totuneController is inactive). This decoupling allowsthe same AutotuneController to be used with a va-riety of systems, simply by choosing the appropriateAutotuneAdaptor to interface with the respectivesystem.

The execution of the AutotuneController and Au-totuneAdaptor beans is managed by the AutoTuneagent through the Agent Administrator. In partic-ular, the Agent Administrator handles the timer fa-cility and asynchronous event processing functionthat allow the AutotuneController and Autotune-Adaptor to run autonomously, by periodically pro-cessing control functions and communicating withthe target system. The AutoTune agent-level Cus-tomizer allows the system administrator to separatelyset the control interval (used by the AutotuneCon-troller bean) and sample interval (used by the Au-

totuneAdaptor bean), which can be different fromeach other. A set of AutotuneMetric classes is de-fined to represent the state/performance of the tar-get system (service-level metrics), the tuning param-eters of the target system (tuning control metrics),and the parameters of the control strategy (config-uration metrics). These metrics can be read or writ-ten by the AutotuneController or AutotuneAdap-tor. They are managed as a collection by theAutoTune Metric Manager for interactions betweenthe two component beans, and can also be selectivelysaved to a historical data repository.

ABLE and the AutoTune architecture were designedto be extensible and to allow rapid deployment ofagent-based solutions. Our experience with using thisinfrastructure validates the usefulness of this archi-tecture and the ABLE toolkit.

AutoTune agents for server self-tuning. For auto-matic tuning of server configuration parameters, analgorithm is needed. A complete self-tuning solutionboth automates the algorithm design and includesan executable on-line algorithm for doing the param-

Figure 3 Architecture of the base AutoTune agent

AutoTune AGENT

AGENT ADMINISTRATOR

AGENT CUSTOMIZER

METRIC MANAGER

AutotuneController BEAN AutotuneAdaptor BEAN

CONTROLLER PROCESSING METHOD

CONTROLLER CUSTOMIZER

ADAPTOR PROCESSING METHOD

ADAPTOR CUSTOMIZER

setTuningControl( )

DATA REPOSITORY

TARGET SYSTEM

AutoTune AGENT


eter adjustments. We follow a standard control-the-oretic methodology for the design process, consist-ing of (1) system modeling and (2) controller design.Thus, our solution consists of three AutoTune agentsthat implement the phases of automatic feedbackcontroller design and deployment: modeling, con-troller design, and run-time control, as shown in Fig-ure 4. The modeling and design phases are per-formed in a “testing” (or nonproduction) mode,whereas the run-time control is active when the sys-tem is “live” (or in production mode).

In order to gain understanding of the dynamic be-havior of the server, a modeling agent is first applied.The modeling agent varies the tuning parametersMaxClients and KeepAlive of the Apache Webserver and records the resulting server performancemetrics (CPU and memory utilization). The collected

time-series data contain information about the dy-namic relationship between the tuning parametersand the performance metrics. Using this modelingdata, a first-order dynamic model is automaticallybuilt by utilizing system identification techniques (seethe section “Modeling agent,” below).

The system model is passed to the controller designagent to conduct model-based controller design.Based on this model, a linear quadratic regulation(LQR) controller is synthesized using design criteriathat are specified through the Customizer GUI. Theoutput of the controller design agent is a set of con-troller parameters that are passed to the run-timecontrol agent.

These controller parameters are used by the run-timecontrol agent to dynamically adjust the MaxClientsand KeepAlive tuning parameters to achieve the de-

Figure 4 Architecture of the AutoTune agents

SYSTEM DESIGNER

SYSTEM ADMINISTRATOR

EXPERIMENTPARAMETERS

MODELING AGENT

CONTROL DATA

DESIRED UTILIZATION

RUN-TIME CONTROL AGENT

APACHE WEBSERVER

CONTROLLERPARAMETERS

DESIGNCRITERIA

CONTROL DESIGN AGENT

EXCITINGSIGNALGENERATOR

CONTROLLER PARAMETERS

SYSTEM ID/CONTROLLERDESIGN

FEEDBACKCONTROLLER

LOGADAPTOR

APACHEADAPTOR

APACHEADAPTOR

MODELING DATA


sired utilization levels. These desired levels are spec-ified by the system administrator (through the run-time agent’s Customizer GUI).

Modeling agent. A good design for the feedback con-troller relies on a mathematical model of the targetsystem. One approach to building this model is touse first principles (e.g., creating a queuing modelof the Apache server). However, this approach re-quires considerable sophistication as well as detailedknowledge about the inner working mechanism ofthe server. Instead of proceeding from first princi-ples, an empirical approach is taken in the model-ing agent for quantifying the relationship betweenthe tuning parameters and performance metrics. Thisempirical modeling approach is referred to as thesystem identification technique,20 also known as the“black box” approach, and is particularly suitable foruse with autonomic agents. Since the model is builtfrom the collected input/output data without requir-ing knowledge of the internals of the target system(i.e., the Apache server), this technique is easily ap-plied to a wide variety of systems.

To perform system identification, the tuning param-eters are varied in a deliberate, predetermined pat-tern so that the dynamics of the systems (i.e., the re-lationship between the tuning parameters and systemutilization) are “excited” and represented in theinput/output data pairs. This means that the param-eter variations should satisfy two properties: uniformcoverage (covering the space of possible variationof tuning parameters) and persistent excitation (con-taining enough frequency components to excite allof the system modes).20 Our experience shows thata first-order model constructed by varying the tun-ing parameters in a (discrete) sinusoid pattern is usu-ally sufficient for modeling the dynamics of queuing-based computing systems. Note that sine wave signalsare used because any signal can be approximatedwith a combination of sine waves of different mag-nitude and frequency. Also note that in order to buildthe model correctly, the number of frequencies inthe exciting signal should be at least equal to the or-der of the model (this is referred to as “persistentexcitation” in control theory, and the necessity of ex-citing all the modes and dynamic behaviors of thesystem has been proven20).

We can build the system model by fitting theinput/output data into the following model form:

� CPUk�1

MEMk�1� � A � � CPUk

MEMk� � B � � KeepAlivek

MaxClientsk�(1)

where CPUk and MEMk denote the values of CPU andmemory utilizations at the k-th time interval, andKeepAlivek and MaxClientsk denote the values ofKeepAlive and MaxClients at the k-th time inter-val. The 2 � 2 matrices A and B include modelingparameters and can be identified using the leastsquares method.20 Note that this model is a linearmodel and, moreover, that the modeling work is onlyconducted once for a representative workload.Whereas the assumption of linearity can affect themodel’s accuracy, using linear models reduces mod-eling complexity and is easier to extend for model-ing high-dimension systems with more metrics.Moreover, using linear models also facilitates the de-sign of robust controllers, which can better toleratemodel inaccuracy (as shown in the section “Exper-imental assessment”), so that there is no need to re-build the model once the workload changes.

The above modeling procedure is wrapped into themodeling agent for automatically building the sys-tem model. The user is only required to answer twosystem-related questions (via the Customizer GUI):(1) the effective ranges of the tuning parameters, and(2) the maximum delay required for the tuning pa-rameters to take full effect on the performance met-rics. Generally, the effective range is known fromserver parameter definitions. The maximum delayrefers to how long it will take for the performancemetrics to reach the new steady state if the tuningparameters are changing from one end of the effec-tive range to the other end. This can be easily mea-sured by conducting a test run. The answers to thesetwo questions are used to determine the magnitudesand time periods of the sinusoid variation pattern.The magnitudes are chosen to cover the effectiveranges of the tuning parameters, and the time pe-riods are chosen to be greater than twice the max-imum delay, to ensure the coverage of the tuningparameter space. The answers to these questionsneed not be very accurate in order to get a goodmodel. This is because the model’s inaccuracy canbe compensated for by a good feedback controlscheme, which will be discussed in the next section.The modeling agent will automatically run the tar-get system (the Apache server in our case) under agiven workload, vary the parameters according to thispattern, collect the performance metrics from theserver, and construct the model from the data.


Due to the process limit of the operating system ofour server, MaxClients can only take an integer valuein the range of [1, 1024]. The tuning parameter Keep-Alive is in integral seconds with a minimum of 1 sec-ond. No maximum value is enforced for KeepAlive,but KeepAlive values larger than 50 have very smalleffects on system utilization. Thus, the range of KeepAlive can be specified as [1, 50]. The maximum timefor the tuning parameters to take effect is roughlyestimated from test run results; we use 10 minutesfor MaxClients and 20 minutes for KeepAlive. Sincethe Apache server model is a second-order model,two sine waves are constructed for varying these pa-rameters.

Run-time control agent. The run-time control agentimplements a state feedback controller21 that makescontrol decisions based on feedback of error, whichis defined as the difference between the desired andmeasured system utilization. The accumulated er-ror over previous time intervals is also used to in-crease the robustness of the controller. The feed-back control law is of the following form:

� KeepAlivek

MaxClientsk� � KP � � eCPU,k

eMEM,k� � KI � �

j�1

k�1 � eCPU, j

eMEM, j�(2)

where eCPU,k and eMEM,k are the differences betweenthe desired CPU and memory utilizations and themeasured values at the k-th time interval. This con-trol law is characterized by the controller parame-ters, the 2 � 2 matrices KP (proportional control gainfor fast response) and KI (integral control gain forremoving steady-state error). These matrices are au-tomatically derived by the controller design agentdiscussed in the next section. The performance ofthe controller, such as the settling time (the time torecover from a disturbance) can be determined an-alytically from the closed loop system model (a modelcomposed of both the system model generated bythe modeling agent and the feedback control law).21

Controller design agent. The controller design agentuses the linear quadratic regulator approach fromoptimal control theory22 to design the parametersKP and KI of the run-time control agent. This de-sign relies on the system model obtained from themodeling agent. In particular, the controller designagent chooses the controller parameters based onminimizing the following quadratic cost function,

J�KP, KI�

� �k�1

�

�eCPU, k eMEM, k vCPU, k vMEM, k � Q � �eCPU, k

eMEM, k

vCPU, k

vMEM, k�

� �KeepAlivek MaxClientsk � R � � KeepAlivek

MaxClientsk�

subject to the dynamic system model in Equation 1and the control law in Equation 2. The variablesvCPU, k and vMEM, k are defined as the accumulated er-ror of CPU (vCPU, k � ¥ j�1

k�1 eCPU, j) and memory(vMEM, k � ¥ j�1

k�1 eMEM, j) utilizations, respectively. Anumerical algorithm for solving the Riccati equationis included in the controller design agent, and it al-lows us to compute the “optimal” KP and KI thatminimize the above cost function. This raises thequestion as to what values to use for the weightingmatrices Q and R of the above cost function. In prin-ciple, Q and R perform some scaling functions in ad-dition to determining a trade-off between control er-ror and control variability.

Since it is not very intuitive or user-friendly to havethe system designer specify these Q and R matricesdirectly, we use some more meaningful inputs to de-rive the proper Q and R matrices. The system de-signer is required only to answer two system-relatedquestions through the Customizer GUI: (1) what arethe valid ranges for the tuning parameters and per-formance metrics (e.g., [0, 1] for CPU and memory,[1, 50] for KeepAlive, and [1, 1024] for MaxClients),and (2) what are the ranges of random fluctuationsfor the performance metrics (e.g., 10 percent for CPUand 2 percent for memory). Like the two questionsasked for the modeling agents, the answers to theabove two questions need not be very accurate, dueto the robustness of the controller. We use the fol-lowing heuristics to determine Q and R.

First, it is sufficient to use only the diagonal formsof the 4 � 4 weighting matrix Q and the 2 � 2 ma-trix R to define the above trade-off; that is Q �diag(q1 , q2 , q3 , q4) and R � diag(r1 , r2). Second,specify the error weights q1 , q2 and tuning param-eter weights r1, r2 in such a way that the errors eCPU, k

and eMEM, k and tuning parameters (KeepAlivek andMaxClientsk) have the same order of magnitude inthe cost function. For example, since the ranges forCPU and MEM are both [0, 1], for KeepAlive [1, 50],


and for MaxClients [1, 1024], we choose q1 � 1, q2 �2, r1 � 1/50 2 and r2 � 1/1000 2. Third, determinethe accumulated error weights q3 , q4 according tothe variability of the performance metrics: the higherthe variability of the metrics, the lower the accuracyof their control effects. Since the CPU utilization usu-ally has 10 percent random fluctuation and the fluc-tuation for the memory utilization is usually quitesmall (less than 2 percent), we choose q3 � 1/10 2

and q4 � 1/ 2 2. Although these rules of thumb fordetermining Q and R may be a little involved, wehave wrapped them into the controller design agent,so that the system designers are not exposed to them.

Implementation of self-tuning agents. Note that allof the above three agents have the same basic struc-ture, and that they are constructed by extending thebase AutoTune agent discussed earlier. In the mod-eling agent, the exciting signal generator bean is ex-tended from the AutotuneController bean inFigure 3, by overriding the process� method toimplement the specific exciting signal function andthe system identification function (discussed in thesection “Modeling agent”). Similarly, the ApacheAdaptor bean is an extension of the AutotuneAdap-tor bean, and it implements the socket connection(see the section “Apache testbed and workload gen-erator”) with the Apache Web server being used bothfor setting the tuning parameters and getting the per-formance metrics. We use the Customizer GUI fa-cility to allow the system designer to specify the ex-perimental parameters (for details, see the section“Modeling agent”). The run-time control agent hasits own extension of the AutotuneController and acorrespondingly different Customizer GUI. This agentcan, in fact, reuse the exact same ApacheAdaptorbean for communicating with the Apache server. Forthe controller design agent, the AutotuneControllerimplements the design automation, and it uses a dif-ferent Adaptor to read the model parameters fromthe modeling agent.

The ABLE framework and Autotune Agent architec-ture have thus proved to be general and sufficientto have allowed us to easily construct all three agentsfrom the same framework. Further, by allowing re-use of the ApacheAdaptor bean between the mod-eling and the run-time agents, it reduces the amountof work that needs to be done. The server self-tun-ing agents increase the automation level of servertuning and require minimal human intervention,from the system designer or from the system admin-istrator. The system designer must provide somehigh-level information to guide the design process:

the experiment parameters and the controller de-sign criteria. The system administrator needs onlyto specify the control objective (the desired systemutilization levels). Once this information is given, theagents can automatically build the system model, de-sign the feedback controller, and apply the control-ler to dynamically tune the server.

Experimental assessment

In this section, we describe our testbed and syntheticworkload generator and present our experimentalresults.

Apache testbed and workload generator. OurApache testbed consists of one server machine run-ning Linux** (kernel v2.2.16) and the Apache HTTPserver v1.3.19, and one or more client machines run-ning synthetic workload generators. The Apacheserver code was modified in order to enable dynamiccontrol. In particular, the tuning parameters Max-Clients and KeepAlive were moved from the staticvariables (read from a configuration file at startup)into the in-memory scoreboard so that they couldbe modified on the fly and picked up by the masterprocess. Further, a metric access functionality wasimplemented as a separate process that communi-cates with other Apache processes through an in-memory scoreboard (to minimize Apache sourcecode changes and performance overheads). Also, aGET/SET interface over a socket connection wasadded to receive the tuning parameters and send per-formance metrics to the external autonomic agents(so as not to compete with normal HTTP traffic).

The synthetic workload generator simulates the ac-tivity of many clients. The workload model used togenerate synthetic transactions was based on theWAGON (Web trAffic GeneratOr and beNchmark)model,23 which has been validated in extensive stud-ies of production Web servers. The “httperf” pro-gram24 was used to generate synthetic HTTP requeststhat conform to this model. The Web site file accessdistributions are from the Webstone 2.5 referencebenchmark.25 Both static and dynamic workloadswere used. The static workload clients requestedstatic Web pages, and the session arrivals followeda Poisson distribution with a rate of 20 sessions persecond. For the dynamic workload, the clients re-quested dynamic Web pages generated through CGI(Common Gateway Interface), and the session ar-rivals also followed a Poisson distribution but witha rate of 10 sessions per second. (A detailed descrip-


tion of the Apache testbed and workload generatorcan be found in Reference 9.)

Experimental results. We assess the performance ofthe autonomic control system by comparing the per-formance achieved through automation with thatachieved by manual tuning as presented in the sec-tion “Apache Web server and performance tuning.”We use different workload scenarios to validate thetuning results against workload variations.

First, the modeling agent was deployed under thestatic workload to build a dynamic system model.This was followed by running the controller designagent to come up with KP and KI for the run-timecontroller. Finally, the run-time feedback control-ler agent was started to dynamically tune the serverparameters. The control interval (adaptation period)

was 5 seconds. Note that the adaptation period wasmuch shorter than the maximum times (i.e., 10 min-utes for MaxClients and 20 minutes for KeepAlive).Usually, this parameter should be smaller than one-twentieth of the maximum time, so that the systemdynamics can be captured and the controller can havetime to react. Here, we made it even smaller, in or-der to achieve faster tuning. Experiments have shownthat changing KeepAlive does not introduce anyperformance overhead, but changing MaxClients willcause a slight increase in CPU utilization, due to theoverhead of adding or removing worker processes.However, these overheads are negligible, especiallywhen the change in MaxClients is not large (smallerthan 100). This is usually the case, and is guaran-teed by putting a bound on the controller output. Inaddition, from the perspective of system stability,having a small control interval will not lead to an

Figure 5 Results of autonomically tuning the Apache Web server


oscillatory response or cause instability problems be-cause each adaptation step (the change of tuning pa-rameters) is calculated based on the model, whichis also obtained at the same control interval, and be-cause the controller is designed to guarantee the sta-bility of the adaptation and optimize the adaptationperformance.

In Figure 5, we show the results of using the run-time control agent. The tuning parameters start atMaxClients � 600 and KeepAlive � 60. Without hu-man intervention, it takes around 50 control inter-vals for the tuning parameters to converge to the val-ues which result in 0.5 CPU utilization and 0.6memory utilization.

The robustness and autonomy of the feedback con-trol system allows the run-time control agent to adapt

to different workloads without having to rerun themodeling and controller design agents to come upwith new run-time controller parameters. Figure 6shows the scenario where the same run-time con-trol agent as in Figure 5 is used, but now the dynamicworkload is added around the 20th control interval(similar to Figure 2). The added HTTP requests fordynamic Web pages consume more system resources,causing large increases in CPU utilization, and slightincreases in memory utilization as well. This resultsin differences between the desired and observed uti-lizations and causes the run-time control agent tostart changing the tuning parameters to compensate.In particular, a larger KeepAlive value is used to de-crease the CPU level and the MaxClients value is ad-justed temporarily according to the dynamics of theserver. The CPU and memory utilizations come backto the desired values after 20 control intervals. (The

Figure 6 Performance of the AutoTune controller for the Apache Web server under dynamic workloads


performance of the autonomic feedback control sys-tem has also been validated through some other ex-periments such as using different workload patterns,with different Web server configurations, and hav-ing different CPU and memory objectives; however,in the interest of brevity we do not include those re-sults here.)

Conclusions

Complex information technology (IT) systems thatrequire manual intervention for configuration andtuning increase the cost of ownership. Specifically,managing the performance of e-commerce sites ischallenging, especially with dynamically varyingworkloads. To maintain good performance, systemadministrators must tune their IT environment onan ongoing basis. The autonomic computing driveaims to reduce costs by increasing the level of au-tomation for such tasks, thereby reducing the man-ual intervention required.

In this paper, we have proposed an agent-based so-lution for not only automating the ongoing systemtuning but also for automatically designing an ap-propriate tuning mechanism for the target system.We use the ABLE toolkit and the AutoTune agentframework to facilitate the construction of auton-omous agents for autonomic performance manage-ment. These agents automate a control-theoreticmethodology of controller design, that is, the Au-toTune agents automate the procedure of modelbuilding, controller design, and run-time feedbackcontrol. We described the methodology used to de-sign a model-based feedback controller, which canhandle the dynamic and interrelated dependenciesbetween tuning knobs and performance metrics. Theeffectiveness of the design automation as well as theresulting tuning mechanism has been demonstratedthrough experiments showing the feedback-drivencontroller to be robust and adaptable to situationsother than the one for which it was designed. Oursystem thus allows a system administrator to auto-mate the process of designing a tuning mechanismand provides a readily available run-time agent toperform the real-time system monitoring and tun-ing. Thus, it is clearly preferable to the manual tun-ing approach that is commonly used today.

The component-based toolkit of ABLE and the gen-eral framework provided by the AutoTune agent ar-chitecture were invaluable in allowing rapid construc-tion of these three agents. In the future, we look toleverage this plug-and-play capability to automate

tuning for a variety of systems, ranging from singledatabase and application servers to distributed serverfarms. We would like to verify that the automatedmethodology we have incorporated in these agentsis indeed applicable across this wide variety of sys-tems. Further, we have not yet exploited the hier-archical and inter-agent capabilities of AutoTune,and much work remains in designing and automat-ing effective control mechanisms for enterprise-scaledistributed systems.

**Trademark or registered trademark of Apache Digital Corpo-ration, The Open Group, Sun Microsystems, Inc., or Linus Tor-valds.

Cited references

1. P. Horn, Autonomic Computing: IBM’s Perspective on the Stateof Information Technology, IBM Corporation (October 15,2001); available at http://www.research.ibm.com/autonomic/manifesto/autonomic_computing.pdf.

2. D.-M. Chiu and R. Jain, “Analysis of the Increase and De-crease Algorithms for Congestion Avoidance in ComputerNetworks,” Computer Networks and ISDN Systems 17 (1989).

3. S. Keshav, “A Control-Theoretic Approach to Flow Control,”Proceedings of the Conference on Communications Architec-ture and Protocols (ACM SIGCOMM ’91), ACM, New York(1991).

4. S. Mascolo, “Classical Control Theory for Congestion Avoid-ance in High-speed Internet,” Proceedings of the 38th Con-ference on Decision and Control, IEEE (1999).

5. C. V. Hollot, V. Misra, D. Towsley, and W. B. Gong, “A Con-trol-Theoretic Analysis of RED,” Infocom 2001, IEEE, An-chorage, AK (2001), pp. 1510–1519.

6. Y. Lu, A. Saxena, and T. F. Abdelzaher, “Differentiated Cach-ing Services: A Control-Theoretic Approach,” Proceedings ofthe 21st International Conference on Distributed Computing Sys-tems, Phoenix, AZ (2001).

7. C. Lu, T. F. Abdelzaher, J. A. Stankovic, and S. H. Son,A Feedback Control Architecture and Design Methodology forService Delay Guarantees in Web Servers, Technical ReportCS-2001-06, University of Virginia, Department of ComputerScience (2001).

8. B. Li and K. Nahrstedt, “Control-based Middleware Frame-work for Quality-of-Service Applications,” IEEE Journal onSelected Areas in Communications 17, No. 9, 1632–1650(1999).

9. Y. Diao, N. Gandhi, J. L. Hellerstein, S. Parekh, and D. M.Tilbury, “Using MIMO Feedback Control to Enforce Pol-icies for Interrelated Metrics with Application to the ApacheWeb Server,” Proceedings of the 8th IEEE/IFIP Network Op-erations and Management Symposium, Florence, Italy (2002).

10. S. Parekh, N. Gandhi, J. L. Hellerstein, D. M. Tilbury, andJ. P. Bigus, “Using Control Theory to Achieve Service LevelObjectives in Performance Management,” Proceedings of the7th IEEE/IFIP Symposium on Integrated Network Management,Seattle, WA (2001), pp. 841–854.

11. Y. Diao, J. L. Hellerstein, and S. Parekh, “A Business-Ori-ented Approach to the Design of Feedback Loops for Per-formance Management,” 12th International Workshop onDistributed Systems: Operations and Management (2001).

12. R. L. Ribler, J. S. Vetter, H. Simitci, and D. A. Reed, “Au-topilot: Adaptive Control of Distributed Applications,” Pro-


ceedings of the 7th IEEE Symposium on High-Performance Dis-tributed Computing (1998).

13. J. P. Bigus, D. A. Schlosnagle, J. R. Pilgrim, W. N. Mills III,and Y. Diao, “ABLE: A Toolkit for Building Multiagent Au-tonomic Systems,” IBM Systems Journal 41, No. 3, 350–371(2002).

14. J. P. Bigus, J. L. Hellerstein, and M. S. Squillante, “Auto Tune:A Generic Agent for Automated Performance Tuning,” Pro-ceedings of the International Conference on Practical Applica-tion of Intelligent Agents and Multi-Agents (PAAM) (2000).

15. Apache Software Foundation. See http://www.apache.org.16. Netcraft, “Netcraft Web Server Survey,” http://www.

netcraft.com/survey.17. R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter,

P. Leach, and T. Berners-Lee, Hypertext Transfer Protocol—HTTP/1.1, RFC 2616, Internet Engineering Task Force(IETF), June 1999; see http://www.ietf.org/rfc/rfc2616.txt.

18. Y. Diao, J. L. Hellerstein, and S. Parekh, “Optimizing Qual-ity of Service Using Fuzzy Control,” Proceedings of the 13thIFIP/IEEE International Workshop on Distributed Systems Op-erations and Management (2002).

19. J. P. Bigus, “The Agent Building and Learning Environment,”Proceedings of the Fourth International Conference on Auton-omous Agents (2000), pp. 108–109.

20. L. Ljung, System Identification: Theory for the User, PrenticeHall, Upper Saddle River, NJ (1999).

21. G. F. Franklin, J. D. Powell, and M. L. Workman, Digital Con-trol of Dynamic Systems, Addison-Wesley Publishing Co.,Reading, MA (1998).

22. W. H. Fleming and R. W. Rishel, Deterministic and Stochas-tic Optimal Control, Springer Verlag (1996).

23. Z. Liu, N. Niclausse, C. Jalpa-Villanueva, and S. Barbier, Traf-fic Model and Performance Evaluation of Web Servers, Tech-nical Report 3840, Institute National de Recherche en In-formatique et en Automatique (INRIA) (1999).

24. D. Mosberger and T. Jin, “httperf: A Tool for Measuring WebServer Performance,” Proceedings of the First Workshop onInternet Server Performance (WISP 98), ACM (1998), pp. 59–67.

25. I. Mindcraft, “Webstone 2.5 Web Server Benchmark” (1998).See http://www.mindcraft.com/webstone/.

Accepted for publication October 16, 2002.

Yixin Diao IBM Research Division, Thomas J. Watson ResearchCenter, P.O. Box 704, Yorktown Heights, New York 10598 (elec-tronic mail: [email protected]). Dr. Diao is a research staff mem-ber at the IBM T. J. Watson Research Center. He received hisPh.D. degree in electrical engineering from Ohio State Univer-sity in 2000. He has authored more than 20 papers, and his re-search interests include computer performance management, in-telligent systems and control, adaptive systems, and stabilityanalysis.

Joseph L. Hellerstein IBM Research Division, Thomas J. WatsonResearch Center, P.O. Box 704, Yorktown Heights, New York 10598(electronic mail: [email protected]). Dr. Hellerstein is a researchstaff member at the IBM T. J. Watson Research Center, wherehe manages the adaptive systems department. He received hisPh.D. degree from the University of California in Los Angeles.Since then, his research has addressed various aspects of man-aging service levels, including: predictive detection, automateddiagnosis, expert systems, and the application of control theoryto resource management. Dr. Hellerstein has published approx-

imately 60 papers and an Addison-Wesley book on expert sys-tems.

Sujay Parekh IBM Research Division, Thomas J. Watson ResearchCenter, P.O. Box 704, Yorktown Heights, New York 10598 (elec-tronic mail: [email protected]). Mr. Parekh is a research asso-ciate at the IBM T. J. Watson Research Center. He received hisB.S. degree in computer science and B.A. in mathematics at Cor-nell University in 1993, and his M.S. degree in computer scienceat the University of Washington in 1996. His research interestscenter around automating both simple and complex computingsystems, and have included work in artificial intelligence plan-ning, machine learning, computer architecture, scheduling algo-rithms, and feedback-driven control.

Joseph P. Bigus IBM Research Division, Thomas J. Watson Re-search Center, P.O. Box 704, Yorktown Heights, New York 10598(electronic mail: [email protected]). Dr. Bigus is a Senior Tech-nical Staff Member at the IBM T. J. Watson Research Center,where he is the project leader on the ABLE research project. Heis a member of the IBM Academy of Technology and an IBMMaster Inventor, with over 20 U.S. patents. Dr. Bigus was an ar-chitect of the IBM Neural Network Utility and Intelligent Minerfor Data products. He received his M.S. and Ph.D. degrees incomputer science from Lehigh University and a B.S. degree incomputer science from Villanova University. Dr. Bigus’s currentresearch interests include learning algorithms and intelligentagents, as well as multiagent teams and their applications to adap-tive system modeling and control, data mining, and decision sup-port.


managing web server performance with autotune agentsbrecht/servers/readings-new/diao.pdf ·...

Documents