[ieee 2013 international conference on cloud & ubiquitous computing & emerging technologies...

An Inter-VM Communication Model Supporting Live Migration

Mrugani Kurtadikar (Author) Department of Computer Science and Information

Technology College of Engineering, Pune

Pune, India [email protected]

Pooja Toshniwal (Author)

Department of Computer Science and Information

Apurva Patil (Author) Department of Computer Science and Information

Technology College of Engineering, Pune

Pune, India [email protected]

Jibi Abraham (Author)

Department of Computer Science and Information Technology Technology College of Engineering, Pune College of Engineering, Pune Pune, India Pune, India [email protected] [email protected]

Abstract— Virtualization technology in cloud environment is characterized by the property of sharing system resources among multiple operating systems. It also maintains isolation between the virtual machines. In virtualized environment, network intensive applications such as web services or databases are being consolidated onto a single host. In these applications, an efficient communication between virtual machines (VMs) is required. Isolation is very important for security reasons but, it increases inter-VM communication overheads. This paper proposes a protocol for efficient and transparent inter-VM communication. It optimizes the inter-VM communication between co-located VMs. It provides synchronization and support for live migration of communicating VMs.

Keywords- Virtualization; IVSHMEM; Hypervisor; TCP/IP

I. INTRODUCTION

Virtualization in cloud environment is emulation of underlying resources to multiple execution environments. These execution environments are called virtual machines [2]. The host machine is the actual machine on which the virtualization takes place and the guest machine is the virtual machine. VM is a software environment in which an operating system or program can be installed and run. Core of any virtualization technology is Hypervisor or Virtual Machine Manager (VMM). Hypervisor manages system resources like processor, memory and network and allocates them to virtual machines as requested. At the same time, it maintains isolation between virtual machines [5].

There are two types of virtualization [9]: Full Virtualization - Hypervisor controls the

hardware resources and emulates it to the guest operating system. In full virtualization, guest Operating System (OS) does not require any modification. Kernel based Virtual Machine (KVM) is an example of full virtualization [11].

Paravirtualization - In paravirtualization, hypervisor controls the hardware resources and provides API to the guest operating system to access the hardware. In paravirtualization, guest OS requires modification to access the hardware resources. Xen is an example of paravirtualization technology [10].

Host machine can access system resources directly but virtual machine uses hypervisor to access them. Isolation constraint between virtual machines does not reveal the existence of one VM from another on the same host. This limits the maximum achievable communication throughput between the two virtual machines running on the same physical host.

The default communication mechanism between two co-located VMs (VMs residing on the same host) is through their virtual network interfaces. It results in low network throughput and high CPU usage due to the complete traversal through network datapath. One approach to reduce this overhead is to use a simple shared memory channel [6] between the co-located communicating VMs. But it has few challenges like providing security and continuing the inter-VM communication when one of the communicating VM gets migrated to a different host due to load balancing. The goal of the proposed work is to consider the issues discussed above and to provide an efficient and transparent inter-VM communication in KVM.

Section 2 describes approaches of inter-VM communication through shared memory and network datapath. Section 3 gives an overview of related work on shared memory communication. Section 4 describes the proposed protocol in detail and finally concludes the paper.

II. INTER-VM COMMUNICATION-APPROACHES

CHALLENGES

Virtual Machines are rapidly finding their way into data centers, enterprise service platforms, high performance

2013 International Conference on Cloud & Ubiquitous Computing & Emerging Technologies

978-0-4799-2235-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CUBE.2013.22

63


978-0-4799-2235-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CUBE.2013.22

63


978-0-4799-2235-2/13 $31.00 © 2013 IEEE

DOI 10.1109/CUBE.2013.22

63


978-0-4799-2235-2/13 $31.00 © 2013 IEEE

DOI 10.1109/CUBE.2013.22

63

computing (HPC) clusters, and even end-user desktop environments [4]. For example, a distributed HPC application may have two processes running in different VMs that need to communicate using messages [7]. Similarly, a web server running in one VM may need to communicate with database server in another VM. There may be a need to transfer a file from one VM to another VM. A performance requirement of such applications is the existence of an efficient means of communication between VMs. The two approaches for communication are as follows:

Network Datapath - The default communication mechanism between VMs is through their virtual network interfaces [6]. In this approach, the communicating processes have to communicate via sockets using network stack. So, this communication must traverse the default TCP/IP stack at both the sender and receiver VMs.

Shared Memory - This mechanism is useful when VMs are co-located. In this, POSIX memory object on the host can be shared with the virtual machines. The VMs can communicate using this shared memory.

This approach bypasses the default network stack and requires fewer hypercalls [10].

There are many challenges with inter-VM communication using shared memory as:

Security - The shared memory can be accessed by the host as well as the guest. Data intended for one VM can be maliciously altered or intercepted by another VM. This is a security breach. Hence, access to shared memory region needs to be controlled [8].

VM Migration - To increase the performance of cloud computing systems, VMs in communication may get migrated due to reasons like load balancing. Even if one of the communicating VM gets migrated, the communication should be continued [8].

III. RELATED WORK

There are many standard papers in literature discussing about usage of shared memory for inter-VM communications when VMs are located on same host.

The authors of paper [6] describes a basic design of shared memory implementation for inter-VM communication between co-located VMs. It describes implementation of virtual device for efficient message passing between VMs which share the same host. But it had limitations such as live migration.

The authors of paper [8] states the various challenges observed with inter-VM communication. The various challenges observed are security, data integrity, denial of service, transparency issues and migration support.

The authors of paper [5] proposes zero copy protocol for inter-VM communication using shared memory. It also suggests extending the work to provide live migration feature.

Cam Macdonell has designed IVSHMEM device which provides inter-VM shared memory communication between co-located VMs [11]. IVSHMEM (Inter-VM SHared MEMory) is designed to share memory blocks among multiple VMs. Shared memory does not belong to any guest, but is a POSIX memory object on the host. This shared memory from the host is mapped into the guests as a PCI device. It provides zero copy overhead [5]. The overall architecture of IVSHMEM device is shown in Figure 1. The communicating VMs need to map this memory in their own virtual address space using mmap system call. In KVM, IVSHMEM is included in Qemu modules in versions later to 0.140 [10]. As shown in Figure 1, shared memory object abcd from host is shared with VM1 and VM2. Multiple IVSHMEM devices can be shared with VMs located on the same host.

IV. PROPOSED PROTOCOL

In order to understand the performance and effectiveness of the inter-VM communication techniques available, a few prototype experiments have been conducted.

For experimentation, file transfer application is chosen to check the utility of shared memory device. On two co-located VMs, the performance of shared memory device for transferring files of different sizes is compared with that of

Figure 1: IVSHMEM device shared between VM1 and VM2. TCP/IP stack and it is also compared with TCP/IP communication performance when the two VMs are on different hosts. The size of shared memory device is kept constant as 1GB and the file size is varied from 10MB to 600MB. One VM writes the file completely into shared

64646464

memory and other VM reads from shared memory. The total time for reading and writing is measured separately using time command of GNU/Linux. For measuring TCP/IP performance, Iperf tool is used. Iperf is a tool for measuring various network criterion[5].

The graph of file size versus latency for IVSHMEM and Iperf is shown in Figure 2. The read and write times for file through IVSHMEM have been measured separately. Hence the graph for file transfer through IVSHMEM is ideal without interrupts. As seen from the graph, the latency of transferring file from one VM to another VM using IVSHMEM is much less than that of the TCP/IP stack. Since the VMs are co-located, the TCP/IP stack performance is optimized, still the latency for TCP/IP is higher than that of IVSHMEM. The third graph shows the latency of file transfer when the two communicating VMs are on different hosts and are communicating through network datapath. The graphs in Figure 2 in conclusion show that the latency of file transfer application is minimum when the two communicating VMs are co-located and communicating through shared memory as compared to when they are communicating through network stack, which gets optimized when the communicating VMs are on same host and the maximum latency is observed in the case when the communicating VMs are on different hosts and are communicating through network stack. Even though the shared memory provides the best performance with respect to latency, there are many limitations. It does not support any memory management mechanism[10]. It is only a piece of region shared by the guests. It contains raw data written by sender VM. The destination VM does not understand from where to start reading the data. Hence, there is no synchronization of activities between sender and receiver VMs. In order to overcome the limitations of shared memory device and higher latency of network stack, the proposed protocol aims at combining the two approaches resulting into better performance. There are three different scenarios to be considered viz. (1) The communicating VMs are co-located throughout their communication. In this case, they will communicate only through shared memory. (2) The communicating VMs are initially co-located but later due to load balancing; one of the VMs gets migrated to different host. In this case, initially the VMs will communicate using shared memory and later on after migration, they will communicate through network stack. (3) Initially the two communicating VMs are on different hosts and later on, due to load-balancing, one of the VM may get migrated to the same host on which the other VM is residing. In this case, the VMs initially will communicate through network stack and later on, through shared memory. (4) The communicating VMs are on different host throughout their communication. In this case, they will communicate only through network stack. Considering the above scenarios, the proposed protocol aims to achieve these requirements through (1) automatic discovery of co-located VMs, (2) synchronization during communication while using shared memory and, (3) support for live migration without any

discontinuity of inter-VM communication in a transparent manner. The protocol definition states that transparent inter-VM communication can be carried out through the use of IVSHMEM and network stack while providing synchronization between VMs and supporting live migration of VMs.

Figure 2: File size versus Latency graph

A. Auto Discovery of co-located VMs

In KVM, due to extreme isolation, the main challenge is to identify whether the communicating VMs are co-resident or not. Hence, in order to achieve transparency, a novel approach is proposed as illustrated in Figure 3.

Figure 3: Auto-discovery of co-located VMs

65656565

As shown in Figure 3, there is an application running on

host which will continuously monitor for live VMs. This application is used to detect live migration of VMs. It will inform server VM of migrated VMs. When the client VM requests for file from server, the server checks whether the file exists. If the file exists, the server checks if the client VM is co-located using shared memory data. If the client VM is located on the same host as of the server VM, then they communicate through shared memory. Otherwise they communicate through network datapath. This mechanism provides an easy way to identify if VMs are co-located

B. Synchronization

If the VMs are co-located and are communicating using shared memory, the main challenge is to synchronize the communication between the client and the server VMs. For this, the shared memory device is divided into a number of chunks of suitable size. The protocol assumes that the shared memory device acts like circular buffer as shown in Figure 4. The file is also divided into number of chunks. The file chunk size and device chunk size must be the same. The server VM writes data chunk-by-chunk into the device. The file chunk is mapped into device chunk by using the formula: (device-chunk- id)=(file-chunk-id)% (no of chunks of device) Consider that the device is divided into four chunks and file into eight chunks as shown in Figure 4. Hence, File chunks 0, 1, 2, 3, 4, 5, 6, 7 and device chunks are 0,1,2,3. Hence the file chunks 0,1,2,3 will be written into device chunks 0,1,2,3 respectively. From the formula (0%4 = 0, 1%4 = 1, 2%4 = 2, and so on). Hence the remaining chunks of file will be mapped as (4%4 = 0, 5%4 = 1, 6%4 = 2 and 7% 4 = 3). Therefore, file chunks 4,5,6,7 will be mapped again to 0,1,2,3. For each chunk written into the device, server sends a packet containing file-chunk-id and chunk-type to the client. The chunk-type may be START, INTERMEDIATE or END. Hence, on receiving the packet, client can start reading the corresponding chunk from device.

Figure 4: IVSHMEM as Circular Ring Buffer

After reading the chunk, client acknowledges the server which also avoids unnecessary polling on the device. The server may free the corresponding chunk of device and reuse it. If the packet type is END, the client reads the last chunk and closes the connection. The server later flushes the device. The overall process is shown as a sequence diagram in Figure 5. Initially all the device chunks are free. The server VM writes the chunk id into the first few bytes of shared memory and then writes the data. Thus for first chunk, 0 is written and then the corresponding data D0 is written into the shared memory. The server then acknowledges the client that chunk 0 is written into the shared memory. The server continues writing next 3 chunk into the shared memory since the device chunks are free. The client then reads first chunk from shared memory and notifies server to free the first chunk. For writing fifth chunk, the server rewinds write pointer to the beginning. Since, the first chunk of device is free, the server overwrites the first chunk by fifth chunk of device. Hence the process continues.

C. VM Migration

Load balancing helps to increase the overall performance of cloud computing systems, which may lead the communicating VMs to get migrated. The VM itself does not know that it is being migrated. Hence, it becomes necessary that VMs continue their inter-VM communication even after migration. Since the migration of VM in KVM-Qemu is very transparent [3], the virtual machines are not aware of the host on which they are located and hence they don't find any difference after migration. Thus, the main challenge is to continue the inter-VM communication transparently even if one of the communicating VMs gets migrated. The two observed scenarios considered in this paper are as follows:

1) Co-located VMs: Initially, the two communicating VMs are on the same

host and they are communicating through shared memory. If one of the VMs gets migrated to another host, the requirement is that the VMs must still communicate seamlessly using network datapath [9] as shown in Figure 6. The proposed protocol aims to detect if one of the co-located VMs which were communicating using shared memory is migrated to some other host. As discussed earlier, the file to be sent and shared memory device is divided into chunks. Hence, when server VM sends a chunk of data to client VM using shared memory, it will first write the corresponding chunk id into the first few bytes of shared memory. On receiving the notification from server VM, client VM will verify the chunk id from the device chunk with the chunk id it expected from server. If the chunk ids match, then client will read the corresponding chunk of file from shared memory. Otherwise, it will notify the server to communicate through network datapath. If the client VM is

66666666

migrated to another host, it will detach from shared memory on earlier host and attach to shared memory on new host. Hence, when it will try to read a particular chunk from shared memory, it will not find the particular chunk into the shared memory on new host. Hence, it will notify the server VM to send data through network stack.

As shown in Figure 5, the server writes chunk id and then the data is written into shared memory. If after reading chunk 1, the client VM is migrated, the client will detach from earlier host and attach to IVSHMEM on new host. Hence, it will not find chunk 2 into shared memory on new host. Hence, in that case the rest communication takes place through network stack. The client will notify the server that the chunk ids do not match. Hence, it should send data through network stack. In this way, the protocol can easily detect migration and achieve transparency.

Figure 5: Synchronization for Data Transfer through IVSHMEM

2) Remote VMs Initially if the two communicating VMs are on different hosts, they will communicate through network datapath. Now, if one of them gets migrated and now both the communicating VMs are on the same host, their co-location can be easily detected by other VM on getting a notification from host application. Hence, once co-location is detected, the client and server VM will now communicate using shared memory.

The host application will continuously monitor for live VMs. It is used for monitoring incoming migrations. This application has access to the shared memory. It is assumed that first few bytes of shared memory are reserved for locking purposes. The host application monitors incoming migrations with the help of reserved bytes. Hence, the host application can detect the incoming migrations and notify registered VMs. If the server and client VM are located on different hosts, and if the client VM is migrated to the same host as server VM, now the VMs are co-located. Hence, the server VM will get a notification from host application of incoming migration. It can now confirm that the client VM is co-located. Hence, the rest of communication can take place through shared memory as long as they are co-located.

Figure 6: Communication -Before and After migration

V. PERFOMANCE EVALUATION

The IVSHMEM readings shown in Figure 2 are taken without protocol. The readings were taken manually without synchronization between server and client. Hence, another experiment was conducted to measure the performance after implementation of protocol. The experimental setup consists of 2 VMs located on same host. The IVSHMEM device size is kept constant at 100 MB. Various files of sizes varying from 10 MB to 1GB are transferred from server VM to client VM using the protocol discussed in section 4. The latency for file transfer has been measured using time command of GNU/Linux. The TCP/IP performance is measured using iperf tool. The graph is plotted for IVSHMEM readings with protocol, iperf readings for VMs on same host and different host as shown in Figure 7. It has been observed that there is significant improvement in performance of inter-VM communication. The latency for IVSHMEM using protocol is less as

67676767

compared to iperf. Also, there is proper synchronization between client VM and server VM. Due to parallel reading and writing of data into shared memory device, the time required for file transfer has been reduced. It has been observed that the protocol works correctly even if client or server VM is migrated to another host. There is proper switching between shared memory communication and network communication after live migration. The proposed protocol overcomes various limitations of bare IVSHMEM as:

Synchronization between client and server VM. Support for file size greater than device size. Support for migration.

The proposed protocol achieves: 1. 20 % higher performance as compared to

IVSHMEM without protocol. 2. 50 % higher performance as compared to TCP/IP

stack.

Figure 7: Comparison of IVSHMEM with Iperf

VI. CONCLUSION AND FUTURE WORK

The proposed protocol aims to provide efficient and transparent model for inter-VM communication between any two VMs. It provides synchronization between sender VM and receiver VM while using shared memory. The protocol is designed to support communication even during live migration. In future, the proposed protocol can be extended to provide security of data written into shared memory device and also to support multiple IVSHMEM devices.

REFERENCES [1] Daniel Nurmi, Rich Wolski, Chris Grzegorczyk, Graziano Obertelli,

Sunil Soman, Lamia Youseff, and Dmitrii Zagorodnov, The Eucalyptus Open-source Cloud-Computing System, Proceedings of Cloud Computing and Its Applications, 2008.

[2] Peter Mell, Timothy Grance, The NIST Definition of Cloud Computing (Draft), Version 15, 10-7-09, National Institute of Standards and Technology, Information Technology Laboratory. K. Elissa, “Title of paper if known,” unpublished.

[3] Avi Kivity, Yani Kamay, Dor Laor, Uri Lublin, and Anthony Liguori, KVM: the Linux Virtual Machine Monitor, http://www.kernel.org. Ottawa, Ontario Canada. June 27th–30th, 2007.

[4] Jian Wang, Survey of State–of–the-art in Inter-VM Communication Mechanisms, Research Proficiency Report, Binghamton University.

[5] Hamid Reza Mohebbi, Omid Kashefi, Mohsen Sharifi, ZIVM: A Zero-Copy Inter-VM Communication Mechanism for Cloud Computing, Computer and Information Science,Vol. 4, No. 6; November 2011.

[6] Fran¸ cois Diakhat´e, Marc Perache, Raymond Namyst, and Herve Jourdren, Efficient Shared Memory Message Passing for Inter-VM Communications, E. C ́sar et al. (Eds.): Euro-Par 2008 Workshops, LNCS 5415, pp. 53–62, 2009.

[7] W. Huang, M. Koop, Q. Gao, and D.K. Panda. Virtual Machine Aware Communication Libraries for High Performance Computing, Proceedings of SuperComputing, Reno, NV, Nov. 2007.

[8] Carl Gebhardt and Allan Tomlinson, Challenges for Inter Virtual Machine Communication, Technical Report , RHUL–MA–2010–12 , 01 September 2010 , Available: http://www.rhul.ac.uk/mathematics/techreports.

[9] Vaibhao Vikas Tatte, Shared Memory Based Communication Between Collocated Virtual Machines, M.Tech. Dissertation Report Communication Between Collocated Virtual Machines, Department of Computer Science and Engineering Indian Institute of Technology, Bombay, 2011.

[10] Xen Community, Xen Hypervisor. http://www.xen.org/. [11] KVM, Kernel Based Virtual Machine - Home Page.

http://www.linux-kvm.org/.

68686868

[ieee 2013 international conference on cloud & ubiquitous computing & emerging technologies...

Documents