Server Consolidation Using OpenVZ: Performance Evaluation Mohuiddin Ahmed, Showayb Zahda, Majed Abbas

Computer Science Department International Islamic University Malaysia

Abstract—Virtualization is abstracting computer resources. It is the ability to run multiple operating systems simultaneously on a single physical machine. In this paper we evaluate the performance of the operating system level virtualization software (OpenVZ). The evaluation will be based on Web hosting, in other words, we evaluate the performance of Web servers on two different machines (dubbed as Baru and Lama) and we observe the behavior of OpenVZ and its scalability for Web servers on different architectures (32 bits and 64 bits). The study evaluates the CPU, Memory and Throughputs under different loading conditions. Index Terms — OpenVZ, performance, server consolidation, virtualization.

I. INTRODUCTION

“Server consolidation is an approach to the efficient usage of computer server resources in order to reduce the total number of servers or server locations that an organization requires”[1]. Simply put, server consolidation is the combination of several servers (i.e. Web Server, Mail Server, Java Server, etc) into one physical machine, in order to, save cost and utilize the computer resources. One way to achieve server consolidation is to use virtualization. Virtualization as a word does not exists in most English dictionaries. However, the closest word to it in Oxford Concise Dictionary is “virtual” which means “Computing: not physically existing as such but made by software to appear to do so” [2]. However, virtual is not a new term in the computer jargons, it was used before such as virtual memory, virtual reality, etc. Virtualization in our context according to [3] is defined as “the abstraction of computer resources” and “abstraction is the process of separating hardware functionality from the underlying hardware.” In other words, virtualization is sharing, and partitioning the resources of computer hardware amongst more than one running operating systems on a single physical machine simultaneously. Each one of the running operating systems is called a Virtual Machine (VM). Virtualization offers the ability to run multiple operating systems (Linux, Windows, MacOS X) on one physical machine at the same time. Currently, virtualization is grabbing the attention of the data centers as well as the small and medium enterprises due to its enormous advantages. This led processor manufacturer mainly Intel and AMD to develop processors to support virtualization i.e. Intel VT and AMD-V respectively.

[1] The corresponding author.

Indeed, virtualization brings a huge economical value to the organizations. Mainly, it reduces the Total Cost of Ownership (TCO) by consolidating a bunch of physical servers into one physical server. For example, 10 Web servers can use one physical machine instead of using 10 physical machines. This reduces a lot of cost -beside the cost of the machines- such as electricity bills, space, air conditioning, etc. In addition, organizations can couple virtualization with open source software solutions (e.g. OpenVZ, Xen) to achieve more reduction in TCO and not to bother anymore about licensing issues. This is handy when multiple virtual machines or virtual machine backups are kept online for redundancy, restoration, or disaster recovery purposes [4].

Virtualization technology can be implemented in several techniques namely, hardware emulation, full virtualization, paravirtualization, operating system level virtualization, application virtualization, and desktop virtualization. However, the discussion about the differences amongst them is beyond this paper. But, a brief discussion about operating system level virtualization will be provided in OpenVZ section.

Besides cost savings, virtualization in general has great advantages for the organizations, due to the fact that the virtualization layer isolates and separates the VMs. None of the VMs is aware about the existence of other VMs on the same machine. They seem to each other as if they were running on different physical machines. The isolation provides higher availability and security for the organization. If one of the VMs crashes or goes down, it has no effects on other VMs. Moreover, system administrators can fire up a VM in less than a minute. As for security, running different services on different VMs improves the security. For instance, if an intruder manages to hack one server, only that particular server goes down and others will remain running as if nothing happened.

Live migration of VMs is yet another great advantage of virtualization. Assume one of the VMs is exhausting the hardware resources, then the administrator can migrate the server into another physical machine in less than one minute. Or if the physical machine should go down for upgrade or maintenance, the VMs can be migrated to another machine. Therefore, migration boosts the availability of the running servers.

Software developers and testers use virtualization to run several platforms to develop and test the products on (i.e. Linux, Windows, MacOS X). It reduces the time of installing the platforms as well as increasing the number of platforms

Proceedings of 11th International Conference on Computer and Information Technology (ICCIT 2008)25-27 December, 2008, Khulna, Bangladesh

1-4244-2136-7/08/$20.00 ©2008 IEEE 341

that can be used for testing and development. Virtualization is indeed a very helpful technology for

education purposes. Now, universities and educational institutions no longer need to buy extra machines to cater their student’s need of running several operating systems. Moreover, each student can have a root access which allows them to have full authority over the system.

Virtualization has disadvantages too; a big damage to all the running servers might occur if the power of the host machine goes down. Or, if the administrator is upgrading the system hardware, he/she has to shutdown the machine. However, by using migration, the system administrators can easily migrate the VMs to another physical machine in less than one second. The rest of the paper is organized as follows: in section II OpenVZ is briefly described, in section III we describe the methodology of the performance evaluation, section IV describes the test bed and its components, in section V we analyze the test results followed by concluding remarks.

II. OPENVZ

OpenVZ is a free and open source virtualization solution. It is the basis of the commercial virtualization solution Virtuozzo from Parallel (formerly known as SWsoft). OpenVZ implements the operating system level virtualization technique which virtualizes servers on the operating system (kernel) layer [5]. In other words, OS-level Virtualization adds a new layer on top of the operating system. This layer manages all the issues related to OpenVZ containers and resource management.

Fig. 1. Operating System level Virtualization [5]

OpenVZ has its own terminologies regarding virtualization technology. The physical machine that is running OpenVZ is called a Host System or Hardware Node. A running instance of the operating system that partitions and shares the resources of the hardware node is called a container, Virtual Environment (VE), and/or Virtual Private Server (VPS) [6]. In this paper we will refer to the running instance as a container. Operating system-level virtualization shares the kernel amongst all running containers. The kernel (OpenVZ kernel creates the virtualization layer) bears the duty of managing and partitioning the resources of the computer, and isolating

the containers from each other. However, sharing one kernel introduces a drawback. In case the kernel crashes all the containers are down. The performance of OpenVZ is closed to the native performance of the machine. The overhead only consumes about 1-3% of the resources. This indeed makes OpenVZ a good choice for server consolidation especially for web hosting [7]. OpenVZ is based on Linux operating system and it only runs in Linux distributions. Moreover, OpenVZ can also be tested without installation using OpenVZ LiveCD1. It takes a few minutes to setup one container using OpenVZ. This is because of the usage of Operating System Templates. An OS-template is a set of packages from some Linux distribution used to populate a VE. With OpenVZ, different distributions can co-exist on the same hardware box, so multiple OS templates are available. [6] Furthermore, changing the configuration of any container can be done on the fly. There is no need to restart any container in order to give or take from it some resources. OpenVZ comes with a command line control utilities that allow the administrators to create, start, stop, destroy and migrate a container. The utilities also allow him/her to set the quota, manage the resources and execute commands on the containers from outside the container itself. For instance, “vzsplit” is a command that creates a configuration file which splits the resources of the computer equally amongst a number of containers the user specifies (vzsplit -n 4).

Fig. 2. WebVZ Management Tool WebVZ is a web management tool for OpenVZ and was

developed by the authors and available in [8] under GPL GNU. WebVZ allows system administrators to control the containers locally and remotely. Fig 2 shows a screenshot of WebVZ. Resource allocation among the containers is done on the basis of demand. If there is need to migrate a container to another server due to resource hogging, this can easily be attained by migration to a different machine which is lightly

1 Live CD: is an operating system that runs without the need for

installation on the hard disk and it does not have any effect on the existing data. Once the CD is ejected, everything goes back to its original state

342

loaded. Fig 3 shows the output of the command “vzlist” (list all containers)

Fig 3 vzlist output

III. METHODOLOGY In order to test the migration between containers we injected http queries from a server and then used a wrapper to increase the number of requests until the system chokes or crashes. In order to evaluate the performance of the containers on OpenVZ kernel, first we evaluated the performance of both machines (we named them Baru and Lama respectively) as a single unit without running OpenVZ kernel so as to know how many requests per second each machine can handle and to find out the relationships between the containers and the Host System. Second, we evaluated the containers by scaling the number of containers from 4 to 8, and finally to 12 on the two machines. All the containers receive the same amount of requests per second concurrently. In other words, each container for example is receiving 100 connections per second at the same time. In order to measure the performance of each running Web server we used a free and open source HTTP performance measurement tool from HP called “httperf” version 0.9.0. According to the manual page of httperf “httperf speaks the HTTP protocol both in its HTTP/1.0 and HTTP/1.1 flavors and offers a variety of workload generators. While running, it keeps track of a number of performance metrics that are summarized in the form of statistics that are printed at the end of a test run. The most basic operation of httperf is to generate a fixed number of HTTP GET requests and to measure how many replies (responses) came back from the server and at what rate the responses arrived.” However, httperf is limited to one test at a time which generates a fixed number of HTTP GET. In order to run httperf several times and increase the number of HTTP GETs each time, another tool called “autobench” can be used which automates web server benchmarking using httperf. According to autobench version 2.1.1 manual page “autobench runs httperf against the specified host or hosts, ramping up the number of requested connections, and logging the results in TSV or CSV format files.”

In order to capture the reading of the CPU and the RAM we used “top -b -d 1” to read the values every second. While the throughput of each test was reported by “httperf’ for plotting and display the result, “bench2graph” is a tool comes with autobench, was used to plot the output of the tests.

This paragraph explains how autobench runs httperf. (E.g. autobench --single_host --host1 192.168.0.1 --low_rate 50 --high_rate 100 --rate_step 10 --timeout 7 --file ve1.tsv). This command will run “httperf” 5 times. First time httperf initiates 50 connections per second until the total number of connections reaches 5000 connections (the default) which is the total number of connections that are initiated during one test, and each connection contains 10 requests (the default). In the second test httperf initiates 60 connections per second until the total number of connections reaches 5000 and so on, each time the number of connections per second increases according to the specified rate step value. The output of httperf is shown in Fig 4 which is extracted and tabulated by autobench as shown in Fig 5.

Fig 4 httperf output sample

Fig. 5. autobench output sample

IV. TESTBED The tests were conducted on two different servers. First server’s specifications (dubbed as Baru) were Intel Core 2 Duo 1.83 GHz, 2 GB of RAM, and 100 Megabit network interface (64 bit machine). And second server’s specifications (dubbed as Lama) were Intel Pentium 4 2.6 GHz, 1 GB of RAM, and 100 Megabit network interface (32 bits machine). Both machines hosted OpenVZ on Centos 5. The tests were conducted from two clients both running Ubuntu 7.10. Fig 6 portrays the architecture of the test bed. The tests were conducted based on the instructions and practices of [9]. We conducted the tests in isolated network environment.

343

The following configuration of “autobench” was done to measure the performance. When we benchmarked the whole machine, the total number of connections to be sent to the machine was set to 20,000. The number of requests in each connection was set to 10. The client time out was set to 7 seconds. However, when we benchmarked the containers, for each container, 5,000 connections were sent in each test. And the rest of the configuration parameters were the same as above. OpenVZ kernel version 2.6.18, and Centos 5 templates from OpenVZ website were used and Apache web server version 2.2.3 was used as the Web server. The http queries were generated by using Httperf version 0.9.0 and autobench 2.1.2. We conducted various tests by scaling the number of containers from 4 to 8 and finally to 12. The configurations of Apache were left to the default except the “KeepAlive Off” which we changed to “On” in order to allow multiple requests in one connection.

Fig. 6. The Test Bed architecture

The configuration files of all the containers were the same and a sample is shown in Fig 7.

ONBOOT="yes" # UBC parameters (in form of barrier:limit) KMEMSIZE="11055923:11377049" LOCKEDPAGES="256:256" PRIVVMPAGES="65536:69632" SHMPAGES="21504:21504" NUMPROC="240:240" PHYSPAGES="0:2147483647" VMGUARPAGES="33792:2147483647" OOMGUARPAGES="26112:2147483647" NUMTCPSOCK="360:360" NUMFLOCK="188:206" NUMPTY="16:16" NUMSIGINFO="256:256" TCPSNDBUF="1720320:2703360" TCPRCVBUF="1720320:2703360" OTHERSOCKBUF="1126080:2097152" DGRAMRCVBUF="262144:262144" NUMOTHERSOCK="360:360" DCACHESIZE="3409920:3624960" NUMFILE="9312:9312" AVNUMPROC="180:180" NUMIPTENT="128:128" # Disk quota parameters (in form of softlimit:hardlimit) DISKSPACE="1048576:1153024" DISKINODES="200000:220000" QUOTATIME="0" # CPU fair sheduler parameter CPUUNITS="1000"

Fig. 7. Configuration of a typical container

V. DISCUSSION

A. OpenVZ There are general observations about OpenVZ that we would like to talk about in this section.

It is easy to fire up a new container using OpenVZ. It would take about one minute. Moreover, a container occupies only a small amount of memory. However, this depends on the size of the distribution as well as the running services. In our case, we only fired up the web server and the default services that run on the Centos 5 template from OpenVZ Website. We found that each VE takes around 25 MB of RAM and about 400 MB of hard disk. Accordingly only 20 containers can be running web servers using 450 MB of RAM and about 8 GB of hard disk.

We ran 55 containers on Baru and 23 on Lama at the same time. This almost filled the memory of each machine respectively. Moreover, we ran the containers with no application overhead within itself.

B. The host machine (Baru) After running “autobench” (from 1000 to 7000 requests per second), the machine managed to handle about 6000 requests per second with no error. This amount of concurrent requests has saturated the CPUs of the machine. However, the throughput was not saturated due to the size of the files (~80 Kilo bytes). Figure 8 shows the results of “autobench” which uses httperf to benchmark the machines. In Fig 8 the X-axis represents the number of requests per second that httperf is supposed to send each test, and Y-axis represents different values that changes based on the figure’s keys. In the following paragraph, we will describe what each line means. Sent request per second is the number of requests that was sent to the machine each second. Average reply rate per second is the average number of replies from the server per second. Response time is the reply time of the request in milliseconds. Throughput is the size of the bandwidth that was used during each test in Kilo bit per second. And finally we observed the percentage of errors for each test as packet loss. As for Fig 8 we see that the first test sent 1000 requests per second and the replies were all intact. In the next test which was 1500 requests per second also it was fine. Autobench conducted 14 tests starting from 1000 requests per second and ending at 7000 requests per second. Each test autobench increases the number of requests per second by 500. So, we see that until 6000 requests per second everything was fine and no errors occurred. After 6000 requests per second we notice there is a drop in the number of requests sent to the server which is a bit steep. And so did every thing. The drop occurred due to the CPUs saturation which was utilized 100%. Fig 9 shows the CPU usage in relation with the time. X-axis represents the time line of the test. We captured the reading of the CPU each second for the whole duration of the tests. Y-axis shows the percentage of the idle CPU. Why the idle? Because the percentage of the idle CPU show the percentage of the free portion of CPU, we can see whether the whole system including the web server is utilizing the CPU or not.

344

Fig 9 shows the CPU idle percentage. We observe that at the beginning of the test the usage of the CPU was about 10%. This value keeps fluctuating each second and keeps going down until it reaches 0% which means that the CPU is fully used. In this test, this point was reached at approximately at 210 seconds and error occurrences were increasing from this point and onward (i.e. 6500 requests per second and beyond).

Fig 10 shows the usage of the RAM of the whole duration of this test.

The results indicate that Baru machine can handle about 6000 requests per second and the CPU was saturated. However, in reality, usually benchmarking is done up to 80% of the CPU usage.

C.The host machine (Lama) Since we are testing the performance of OpenVZ on two different machines, section A presented the results of the first machine (Baru) whereas this section presents the second machine (Lama). In this section we will not present the graphs and we will only mention the results. After running “autobench”, the machine managed to handle about 3300 requests per second with no error. This amount of concurrent requests has saturated the CPU of the machine. However, the throughput was not saturated due to the size of the files (~80 Kilo bytes).

Fig. 8. The performance of Baru.

D. OpenVZ on Baru and Lama In this section we will describe the results of running

OpenVZ on the two machines. The results are tabulated but brief discussion is provided here about the result for clarification. As mentioned before we scaled the number of containers from 4 to 8 and finally to 12 and we conducted the tests on them so as to find the relationships between the machine and OpenVZ. Surprisingly, the two machines have had the same relationship with OpenVZ. The results are described first and later we show the relationship between the machine and OpenVZ.

Running 4 containers on Baru: We set up Baru with 4 identical containers and we ran

autobench against them concurrently.

In other words, all the containers receive the same load at the same time. Fig 11 shows the results of autobench on one of the containers (the other containers have almost an identical graphs so we present one of them here). As the graph shows autobench starts from 200 requests and ends at 2000 requests. However, the errors started to occur at about 1600 requests per second. This means that the machine is receiving about 6400 requests per second at the same time. Note that this number is about the same as the performance of machine Baru in section A. Surprisingly, the CPU usage (Fig 12) was almost the same as in section A and is saturated at the same the error occurrences. As for the RAM (Fig 13) it has also the same usage as in section A.

Fig. 9. CPU Usage of Baru

Fig. 10. RAM usage of Baru

Running 8, and 12 containers on Baru: The results of 8 and 12 containers were almost the same as

when we were using four containers. In 8 containers, each container handled about 750 requests per second, and in 12 containers, each container handles 500 requests per second with no errors. The graphs for the CPU and RAM were the same as Baru machine and 4 containers. Table 1 shows the relationship between the performance of Baru and 4, 8, and 12 containers. We can see from the table that OpenVZ has nearly 0% overhead. This means that we do not lose any hardware resources when we are running OpenVZ.

345

TABLE 1 Performance of Baru and its containers

Machine Number of servers

Requests per second

Total requests per second

Baru 1 ~6000 ~6000 4 containers 4 ~1600 ~6400 8 containers 8 ~750 ~6000 12 containers 12 ~500 ~6000

Fig. 11. Performance of one of the containers

Fig. 12. CPU usage of 4 containers

Fig. 13. RAM usage of the 4 containers

As for Lama we mentioned before that it handled about 3300 requests per second. So, when we ran 4 containers on it we found that 700 requests per seconds were handled by each container which means 700 * 4 = 3200 requests per second which is about the same number of requests that were handled by Lama. Note that we did not manage to run 12 containers on Lama due to RAM limitations.

TABLE 2

Performance of Lama and its containers Machine Number of

servers Requests per second

Total requests per second

Lama 1 ~3300 ~3300 4 containers 4 ~700 ~3200 8 containers 8 ~400 ~3200

VI.CONCLUSION We found that when using OpenVZ, almost no overhead on the computer resources and it shares the resources of the computer fairly amongst the running containers. Moreover, we found that both machines (Baru and Lama) can handle about 6000 and 3300 requests per second respectively. And so do containers of OpenVZ when we run all of them concurrently on the both Baru and Lama. This means that OpenVZ performance does not change with the change of the architecture (32 bits and 64 bits). We can safely say that if we want to run 15 Web servers using OpenVZ containers, each VE can handle 400 requests per second at the same time (this is on Baru). The WebVZ is a very useful tool for administrating OpenVZ containers both locally and remotely.

REFERENCES [1] Information Technology @ Johns Hopkins-Glossary of

Technical Terms and Acronyms. Available http://it.jhu.edu/glossary/pqrs.html

[2] Concise Oxford English Dictionary (Eleventh Edition). Computer Software.

[3] Wolf C., Halter E. (2005). Virtualization from the desktop to the enterprise. Apress, 2005. p.1.

[4] Shields G., The Shortcut Guide to Selecting the Right Virtualization Solution, electronic book. Available: http://nexus.realtimepublishers.com/SGSRVS.htm

[5] OS Virtualization. Available: http://www.parallels.com/en/products/virtuozzo/os/

[6] OpenVZ Wiki. Available: http://wiki.openvz.org/Category:Definitions[7] Padala P., Xiaoyun Zhu, Zhikui Wang, Singhal S., Shin K.

Performance Evaluation of Virtualization Technologies for Server Consolidation. HP Laboratories Palo Alto, April 11, 2007. Available: http://www.hpl.hp.com/techreports/2007/HPL-2007-59.pdf

[8] WebVZ. Available: http://webvz.sourceforge.net [9] The Linux HTTP Benchmarking HOWTO. Available:

http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/

346