KVM/ARM: Experiences Building the Linux ARM Hypervisor

Christoffer Dall and Jason Nieh{cdall, nieh}@cs.columbia.edu

Department of Computer Science, Columbia UniversityTechnical Report CUCS-010-13

April 2013

Abstract

As ARM CPUs become increasingly common in mo-bile devices and servers, there is a growing demandfor providing the benefits of virtualization for ARM-based devices. We present our experiences building theLinux ARM hypervisor, KVM/ARM, the first full sys-tem ARM virtualization solution that can run unmodifiedguest operating systems on ARM multicore hardware.KVM/ARM introduces split-mode virtualization, allow-ing a hypervisor to split its execution across CPU modesto take advantage of CPU mode-specific features. Thisallows KVM/ARM to leverage Linux kernel services andfunctionality to simplify hypervisor development andmaintainability while utilizing recent ARM hardwarevirtualization extensions to run application workloads inguest operating systems with comparable performanceto native execution. KVM/ARM has been successfullymerged into the mainline Linux 3.9 kernel, ensuring thatit will gain wide adoption as the virtualization platformof choice for ARM. We provide the first measurementson real hardware of a complete hypervisor using ARMhardware virtualization support. Our results demonstratethat KVM/ARM has modest virtualization performanceand power costs, and can achieve lower performance andpower costs compared to x86-based Linux virtualizationon multicore hardware.

1 IntroductionARM-based devices are seeing tremendous growthacross smartphones, netbooks, and embedded comput-ers. While ARM CPUs have benefited from their advan-tages in power efficiency in these markets, ARM CPUsalso continue to increase in performance such that theyare now within the range of x86 CPUs for many classesof applications. This is spurring the development of newARM-based microservers and an upward push of ARMCPUs into traditional server and PC systems.

Unlike x86-based systems, a key limitation of ARM-based systems has been the lack of support for virtual-

ization. To address this problem, ARM has introducedhardware virtualization extensions in the newest ARMCPU architectures. ARM has benefited from the hind-sight of x86 in its design. For example, nested pagetables, not part of the original x86 virtualization hard-ware, are standard in ARM. However, there are impor-tant differences between ARM and x86 virtualization ex-tensions such that x86 hypervisor designs may not bedirectly amenable to ARM. These differences may alsoimpact hypervisor performance, especially for multicoresystems, but have not been evaluated with real hardware.

We describe our experiences building KVM/ARM,the ARM hypervisor in the mainline Linux kernel.KVM/ARM is the first hypervisor to leverage ARMhardware virtualization support to run unmodified guestoperating systems (OSes) on ARM multicore hardware.Our work makes four main contributions. First, we in-troduce split-mode virtualization, a new approach to hy-pervisor design that splits the core hypervisor so that itruns across different CPU modes to take advantage of thespecific benefits and functionality offered by each CPUmode. This approach provides key benefits in the con-text of ARM virtualization. ARM introduces a new CPUmode for running hypervisors called Hyp mode, but Hypmode has a different feature set from other CPU modes.Hypervisors provide many aspects of OS functionality,but standard OS mechanisms in Linux would have to besignificantly redesigned to run in Hyp mode. Our split-mode virtualization mechanism allows a hypervisor touse Hyp mode to leverage ARM hardware virtualizationfeatures, but also run in normal privileged CPU modes,allowing it to coexist with other OS functionality.

Second, we designed and implemented KVM/ARMfrom the ground up as an open source project that wouldbe easy to maintain and integrate into the Linux ker-nel. For example, by using our split-mode virtualiza-tion, we can leverage the existing KVM hypervisor in-terface in Linux and can reuse substantial pieces of ex-isting kernel code and interfaces to reduce code duplica-tion. KVM/ARM was accepted as the ARM hypervisorof the mainline Linux kernel as of the Linux 3.9 kernel,

1

ensuring its wide adoption and use given the dominanceof Linux on ARM platforms. Based on our open sourceexperiences, we offer some useful hints on transferringresearch ideas into implementations likely to be adoptedby existing open source communities.

Third, we demonstrate the effectiveness ofKVM/ARM on real multicore ARM hardware. Ourresults are the first measurements of a hypervisor usingARM virtualization support on real hardware. Wecompare against the standard widely-used Linux x86KVM hypervisor and evaluate its performance overheadfor running application workloads in virtual machines(VMs) versus native non-virtualized execution. Ourresults show that KVM/ARM achieves comparableperformance overhead in most cases, and significantlylower performance overhead for two important appli-cations, Apache and MySQL, on multicore platforms.These results provide the first comparison of ARMand x86 virtualization extensions on real hardware toquantitatively demonstrate how the different designchoices affect virtualization performance. We showthat KVM/ARM also provides power efficiency benefitsover Linux x86 KVM.

Finally, we make several recommendations regardingfuture hardware support for virtualization based on ourexperiences building and evaluating a complete ARMhypervisor. We identify features that are important andhelpful to reduce the software complexity of hypervisorimplementation, and discuss mechanisms useful to max-imize hypervisor performance, especially in the contextof multicore systems.

This technical report describes our experiences de-signing, implementing, and evaluating KVM/ARM. Sec-tion 2 presents an overview of the ARM virtualizationextensions and a comparison with x86. Section 3 de-scribes the design of the KVM/ARM hypervisor. Sec-tion 4 discusses the implementation of KVM/ARM andour experiences releasing it to the Linux community andhaving it adopted into the mainline Linux kernel. Sec-tion 5 presents experimental results quantifying the per-formance and energy efficiency of KVM/ARM, as wellas a quantitative comparison of real ARM and x86 vir-tualization hardware. Section 6 makes several recom-mendations about designing hardware support for virtu-alization. Section 7 discusses related work. Finally, wepresent some concluding remarks.

2 ARM Virtualization Extensions

Because the ARM architecture is not classically virtual-izable [20], ARM has introduced hardware virtualizationsupport as an optional extension in the latest ARMv7architecture [4] and a mandatory part of the upcoming

64-bit ARMv8 architecture. The Cortex-A15 [2] is anexamples of current ARMv7 CPUs including hardwarevirtualization extensions. We present a brief overview ofthe ARM virtualization extensions.

2.1 CPU Virtualization

Figure 1 shows the CPU modes on the ARMv7 archi-tecture, including TrustZone (Security Extensions) anda new CPU mode called Hyp mode. TrustZone splits themodes into two worlds, secure and non-secure, which areorthogonal to the CPU modes. A special mode, monitormode, is provided to switch between the secure and non-secure worlds. Although ARM CPUs always power upstarting in the secure world, ARM bootloaders typicallytransition to the non-secure world at an early stage andsecure world is only used for specialized use cases suchas digital rights management. TrustZone may appearuseful for virtualization by using the secure world for hy-pervisor execution, but this does not work because thereis no support for trap-and-emulate. There is no means totrap operations executed in the non-secure world to thesecure world. Non-secure software can therefore freelyconfigure, for example, virtual memory. Any softwarerunning in the non-secure world therefore has access toall non-secure memory, making it impossible to isolatemultiple VMs running in the non-secure world.

Non-Secure state

PL0 User

PL1 Kernel

PL2 Hyp

Monitor Mode (Secure PL1)

Secure state

PL0 User

PL1 Kernel

Figure 1: ARMv7 CPU modes.

Hyp mode was introduced as a trap-and-emulatemechanism to support virtualization in the non-secureworld. Hyp mode is a CPU mode that is strictlymore privileged than other CPU modes, user and ker-nel modes. Without Hyp mode, the OS kernel runningin kernel mode directly manages the hardware and cannatively execute sensitive instructions. With Hyp modeenabled, the kernel continues running in kernel mode butthe hardware will instead trap into Hyp mode on varioussensitive instructions and hardware interrupts. To runVMs, the hypervisor must at least partially reside in Hypmode. The VM will execute normally in user and ker-

2

nel mode until some condition is reached that requiresintervention of the hypervisor. At this point, the hard-ware traps into Hyp mode giving control to the hyper-visor, which can then manage the hardware and providethe required isolation across VMs. Once the conditionis processed by the hypervisor, the CPU can be switchedback into user or kernel mode and the VM can continueexecuting.

To improve performance, ARM allows many traps tobe configured so they trap directly into a VM’s kernelmode instead of going through Hyp mode. For exam-ple, traps caused by normal system calls or undefinedexceptions from user mode can be configured to trap to aVM’s kernel mode so that they are handled by the guestOS without intervention of the hypervisor. This avoidsgoing to Hyp mode on each system call or undefined ex-ception, reducing virtualization overhead.

2.2 Memory Virtualization

In addition to virtualizing the CPU, ARM provides hard-ware support to virtualize physical memory. When run-ning a VM, the physical addresses managed by the VMare actually guest physical addresses and need to betranslated into machine addresses, or host physical ad-dresses. Similarly to nested page tables on x86, ARMprovides a second set of page tables, Stage-2 page tables,configurable from Hyp mode. Stage-2 translation can becompletely disabled and enabled from Hyp mode. Stage-2 page tables use ARM’s new LPAE page table format,with subtle different requirements than the page tablesused by kernel mode.

Similar to standard page tables used to translate vir-tual addresses to physical addresses used by the VM,ARM virtualization extensions provide a second set ofpage tables to perform a translation from guest physi-cal addresses to host physical addresses. These Stage-2translation tables have a format closely resembling, butnot identical to, the normal virtual-to-physical page ta-bles, and are stored in normal memory and walked byhardware to translate guest physical addresses into hostphysical addresses.

Figure 2 shows the complete address translationscheme. Three levels of page tables are used for Stage-1translation from virtual to guest physical addresses, andfour levels of page tables are used for Stage-2 translationfrom guest to host physical addresses. Stage-2 transla-tion can be entirely enabled or disabled using a bit inthe Hyp Configuration Register (HCR). The base regis-ter for the Stage-2 first-level (L1) page table is speci-fied by the VirtualizationTranslation Table Base Regis-ter (VTTBR). Both registers are only configurable fromHyp mode.

L1

L2

L3

TTBR VA

Page

Table

Table

L1

Table L2

Table L3

Page

Stage-1

Stage-2

VTTBR

L4

Page PA

GPA

Figure 2: Stage-1 and Stage-2 page table walk onARMv7 using the LPAE memory long format descrip-tors. The virtual address (VA) is is first translated intoa guest physical address (GPA) and finally into a hostphysical address (PA).

2.3 Interrupt Virtualization

ARM defines the Generic Interrupt Controller (GIC) ar-chitecture [3], which now includes virtualization support(VGIC). It receives interrupts from devices and deter-mines if and on which CPU cores to raise correspond-ing interrupt signals. The GIC has a distributor and perCPU interfaces. The distributor determines which CPUsreceive an interrupt and routes it to the respective CPUinterfaces. CPUs access the distributor over a Memory-Mapped I/O (MMIO) interface to enable and set priori-ties of interrupts. CPU interfaces raise interrupt signalson the respective CPUs and the CPUs access the MMIOCPU interface to acknowledge (ACK) and complete in-terrupts, known as signaling an End-Of-Interrupt (EOI).Figure 3 shows an overview of the GIC architecture.

Each interrupt signal raised via the CPU interface canbe configured to trap to either Hyp or kernel mode. Trap-ping all interrupts to kernel mode and letting OS soft-ware running in kernel mode handle them directly is ef-ficient, but does not work in the context of VMs, becausethe hypervisor loses control over the hardware. Trappingall interrupts to Hyp mode ensures that the hypervisor re-tains control, but requires emulating virtual interrupts insoftware to signal events to VMs. This is cumbersometo manage and expensive because interrupt processing,such as ACKing and EOIing, must now go through thehypervisor.

ARM introduces hardware support for virtual inter-rupts to reduce the number of traps to Hyp mode. Hard-ware interrupts trap to Hyp mode to retain hypervisorcontrol, but virtual interrupts trap to kernel mode so thatguest OSes can ACK, mask, and EOI them without trap-ping into the hypervisor. Each CPU has a virtual CPU

3

CPU 0

CPU 1

MMIO Interface

IRQ/FIQ

IRQ/FIQ

GIC

Control Interface

CPU Interface

CPU Interface

Control Interface

Distributor

Devices

PPI

SPI

Distributor Interface

SGI

List Registers

List Registers

Virtual CPU Interface

Virtual CPU Interface

Figure 3: ARM Generic Interrupt Controller (GIC)overview. Devices inject interrupts to the GIC distribu-tor. Interrupts can either be Private Peripheral Interrupts(PPIs), Shared Peripheral Interrupts (SPIs), or SoftwareGenerated Interrupts (SGIs). The distributor determineswhich CPU(s) receives the interrupts and can raise ei-ther the IRQ or FIQ line to those CPU(s). CPUs canconfigure both the CPU interfaces and the distributorusing a set of memory mapped configuration registers.The virtualization support consists of a virtual CPU in-terface, which can be accessed directly from guests andprogrammed by the hypervisor, to generate virtual inter-rupts.

interface that the guest OS can interact with throughMMIO without trapping to Hyp mode, and a virtual CPUcontrol interface, which can be programmed to raise vir-tual interrupts using a set of list registers, which are onlyaccessed by the hypervisor. In the common case, emu-lated virtual devices are used to emulate physical devicesand when they raise interrupts, these are treated as virtualinterrupts, which are programmed into the list registers,causing the GIC to trap the VM directly to kernel modeand lets the guest ACK and EOI the interrupt. Addition-ally, if a physical device is directly assigned to a VM, thelist registers can be programmed so that virtual interruptsgenerated as a consequence of a physical interrupt fromthat specific device can be ACKed and EOIed directly bythe guest OS instead of needing to trap to the hypervisor.

Virtual interrupts allow guest OSes to handle themonce they have been delivered to the CPU, but provide nomechanism to access the distributor. All accesses fromVMs to the distributor will cause traps to Hyp mode, andthe hypervisor must emulate a virtual distributor. For ex-ample, when a guest multicore OS generates a softwareinterrupt that results in an Inter-Processor Interrupt (IPI)

between cores, it writes to the Software Generated In-terrupt (SGI) register on the distributor, which then for-wards the interrupt to the corresponding CPU interface.The write to a distributor register will trap to Hyp mode,so the hypervisor can emulate it in software.

2.4 Timer VirtualizationARM defines the ARM Generic Timers which includesupport for timer virtualization. Generic timers providea counter that measures passing of time in real-time, anda timer for each CPU, which is programmed to raise aninterrupt to the CPU after a certain amount of time haspassed, as per the counter. Timers are likely to be usedby both hypervisors and guest OSes, but to provide isola-tion and retain control, the timers used by the hypervisorcannot be directly configured and manipulated by guestOSes. Such timer accesses from a guest OS would needto trap to Hyp mode, incurring additional overhead fora relatively frequent operation for some workloads. Hy-pervisors may also wish to virtualize VM time, which isproblematic if VMs have direct access to counter hard-ware.

ARM provides virtualization support for the timers byintroducing a new counter, the virtual counter and a newtimer, the virtual timer. A hypervisor can be config-ured to use physical timers while VMs are configuredto use virtual timers. VMs can then access, program,and cancel virtual timers without causing traps to Hypmode. Access to the physical timer and counter fromkernel mode is controlled from Hyp mode, but softwarerunning in kernel mode always has access to the virtualtimers and counters. Additionally, Hyp mode configuresan offset register, which is subtracted from the physicalcounter and returned as the value when reading the vir-tual counter.

2.5 Comparison with x86There are a number of similarities and differences be-tween the ARM virtualization extensions and those forx86 from Intel and AMD. Intel and AMD extensions arevery similar, so we just compare ARM and Intel. BothARM and Intel introduce a new CPU mode for support-ing trap and emulate, but they are quite different. ARM’sHyp mode is a separate and strictly more privileged CPUmode than previous user and kernel modes. In contrast,Intel has root and non-root modes [15] which are orthog-onal to its CPU protection modes, and can trap oper-ations from non-root to root mode. Intel’s root modesupports the same full range of user and kernel modefunctionality as its non-root mode, whereas ARM’s Hypmode is a strictly different CPU mode with its own setof features. A hypervisor using ARM’s Hyp mode has

4

an arguably simpler set of features to use than the morecomplex options available with Intel’s root mode.

Both ARM and Intel trap into their respective Hyp androot modes, but Intel provides specific hardware supportfor a VM control block which is automatically savedand restored when switching to and from root mode us-ing only a single instruction. This is used to automat-ically save and restore guest state when switching be-tween guest and hypervisor execution. In contrast, ARMprovides no such hardware support and any state thatneeds to be saved and restored must be done explicitly insoftware. This provides some flexibility in what is savedand restored in switching to and from Hyp mode. For ex-ample, trapping to ARM’s Hyp mode is potentially fasterthan trapping to Intel’s root mode if there is no additionalstate to save.

ARM and Intel are quite similar in their support forvirtualizing physical memory. Both introduce an addi-tional set of page tables is introduced to translate guest tohost physical addresses. ARM benefited from hindsightin including Stage-2 translation whereas Intel did not in-clude its equivalent Extended Page Table (EPT) supportuntil its second generation virtualization hardware.

ARM’s support for virtual interrupts and timers haveno real x86 counterpart. EOIing and masking interruptsby a guest OS on x86 require traps to root mode, whereasARM’s virtual interrupts avoid the cost of trapping toHyp mode for those interrupt handling mechanisms. Ex-ecuting similar timer functionality by a guest OS on x86will incur additional traps to root mode compared to thenumber of traps to Hyp mode required for ARM. Read-ing a counter, however, is not a privileged operation onx86 and does not trap, even without virtualization sup-port in the counter hardware.

3 Hypervisor Architecture

KVM/ARM is the first complete open source hypervi-sor for the ARM architecture. Its relative simplicityand rapid completion was faciliated by specific designchoices that allow it to leverage substantial existing in-frastructure despite differences in the underlying hard-ware. Any hypervisor comprises components includingcore CPU and memory virtualization, a VM scheduler, amemory allocator, an I/O emulation layer, and a manage-ment interface. Instead of reinventing and reimplement-ing such complex core functionality in the hypervisor,and potentially introduce tricky and fatal bugs along theway, KVM/ARM leverages existing infrastructure in theLinux kernel. KVM/ARM builds on the popular KVMinterface in the kernel to provide a high degree of codereuse for CPU scheduling, memory management, anddevice emulation.

While a standalone hypervisor design approach hasthe potential for better performance and a smallerTrusted Computing Base (TCB), this approach is lesspractical on ARM for many applications. ARM hard-ware is in many ways much more diverse than x86.Hardware components are often tightly integrated inARM devices in non-standard ways by different devicemanufacturers. ARM hardware lacks features for hard-ware discovery such as a standard BIOS or a PCI bus,and there is no established mechanism for installing low-level software on a wide variety of ARM platforms.Linux, however, is supported across almost all ARMplatforms and by integrating KVM/ARM with Linux,KVM/ARM is automatically available on any devicerunning a recent version of the Linux kernel. This is incontrast to standalone approaches such as Xen [6], whichmust actively support every platform on which they wishto install the Xen hypervisor.

3.1 Split-mode Virtualization

Simply running a hypervisor entirely in ARM’s Hypmode is attractive since it is the most privileged level.However, since KVM/ARM leverages existing kernel in-frastructure such as the scheduler, running KVM/ARMin Hyp mode implies running the Linux kernel in Hypmode. This is problematic for at least two reasons. First,low-level architecture dependent code in Linux is writtento work in kernel mode, and would not run unmodified inHyp mode, because Hyp mode is a completely differentCPU mode from normal kernel mode. The significantchanges that would be required to run the kernel in Hypmode would be very unlikely to be accepted by the Linuxkernel community. More importantly, to preserve com-patibility with hardware not providing Hyp mode and torun Linux as a guest OS, low-level code would have tobe written to work in both modes, potentially resultingin slow and convoluted code paths. As a simple exam-ple, a page fault handler needs to obtain the virtual ad-dress causing the page fault. In Hyp mode this addressis stored in a different register than in kernel mode.

Second, running the entire kernel in Hyp mode wouldadversely affect native performance. For example, Hypmode has its own separate address space. Whereas ker-nel mode uses two page table base registers to providethe familiar 3GB/1GB split between user address spaceand kernel address space, Hyp mode uses a single pagetable register and can therefore not have direct access tothe user space portion of the address space. Frequentlyused functions to access user memory would require thekernel to explicitly map user space data into kernel ad-dress space and subsequently perform necessary tear-down and TLB maintenance operations, which would re-sult in poor native performance on ARM.

5

Note that these problems with running a Linux hy-pervisor using ARM Hyp mode do not occur for x86hardware virtualization. Because x86 root mode is or-thogonal to its CPU privilege modes, it is possible to runthe entire Linux kernel in root mode as a hypervisor be-cause the same set of CPU modes available in non-rootmode are available in root mode. Nevertheless, giventhe widespread use of ARM and the advantages of Linuxon ARM, finding an efficient virtualization solution forARM that can leverage Linux and take advantage of thehardware virtualization support is of crucial importance.

KVM/ARM introduces split-mode virtualization, anew approach to hypervisor design that splits the corehypervisor so that it runs across different CPU modesto take advantage of the specific benefits and function-ality offered by each CPU mode. As applied to ARM,KVM/ARM uses split-mode virtualization to leveragethe ARM hardware virtualization support enabled byHyp mode, while at the same time leveraging existingLinux kernel services running in kernel mode. Split-mode virtualization allows KVM/ARM to be integratedwith the Linux kernel without intrusive modifications tothe existing code base.

This is done by splitting the hypervisor into two com-ponents, the lowvisor and the highvisor, as shown in inFigure 4. The lowvisor is designed to take advantageof the hardware virtualization support available in Hypmode to provide three key functions. First, the lowvi-sor sets up the correct execution context by appropriateconfiguration of the hardware, and enforces protectionand isolation between different execution contexts. Thelowvisor directly interacts with hardware protection fea-tures and is therefore highly critical and the code baseis kept to an absolute minimum. Second, the lowvi-sor switches from one execution context to another asneeded, for example switching from a guest executioncontext to the host execution context. We refer to anexecution context as a world, and switching from oneworld to another as a world switch, because the entirestate of the system is changed. Since the lowvisor con-figures the hardware, only it can be responsible for thehardware reconfiguration necessary to perform a worldswitch. Third, the lowvisor provides a virtualization traphandler, which handles interrupts and exceptions thatmust trap to the hypervisor. The lowvisor performs onlythe minimal amount of processing required and defersthe bulk of the work to be done to the highvisor after aworld switch to the highvisor is complete.

The highvisor runs in kernel mode and can thereforedirectly leverage existing Linux functionality such as thescheduler. Further, the highvisor can make use of stan-dard kernel software data structures and mechanisms toimplement its functionality, such as locking mechanismsand memory allocation functions. This makes higher-

Host Kernel KVMHighvisor

Host User QEMUPL 0 (User)

PL 1 (Kernel)

PL 2 (Hyp)

VM Kernel

VM User

Trap

Lowvisor

Trap

Figure 4: KVM/ARM system architecture.

level functionality easier to implement in the highvisor.For example, while the lowvisor provides the low-levelmechanism to switch from one world to another, thehighvisor handles Stage-2 page faults from the VM andperforms instruction emulation.

Because the hypervisor is split across kernel mode andHyp mode, switching between a VM and the hypervisorinvolves multiple mode transitions. A trap to the hyper-visor while running the VM will first trap to the lowvisorrunning in Hyp mode. The lowvisor will then cause an-other trap to run the highvisor. Similarly, going fromthe hypervisor to a VM when running the highvisor re-quires trapping from kernel mode to Hyp mode, and thenswitching to the VM. As a result, split-mode virtualiza-tion incurs a double trap cost in switching to and fromthe hypervisor. On ARM, the only way to perform thesemode transitions to and from Hyp mode is by trapping.However, as shown in Section 5, the cost of this extratrap is not a significant performance cost on ARM.

KVM/ARM uses a memory mapped interface to sharedata between the highvisor and lowvisor as necessary.Because memory management can be complex, weleverage the highvisor’s ability to use the existing mem-ory management subsystem in Linux to manage mem-ory for both the highvisor and lowvisor. Managingthe lowvisor’s memory involves additional challengesthough, because it requires managing Hyp mode’s sep-arate address space. One simplistic approach would beto reuse the host kernel’s page tables and also use themin Hyp mode to make the address spaces identical. Thisunfortunately does not work, because Hyp mode uses adifferent page table format from kernel mode. Therefore,the highvisor explicitly manages the Hyp mode page ta-bles to map any code executed in Hyp mode and any datastructures shared between the highvisor and the lowvisorto the same virtual addresses in Hyp mode and in kernelmode.

Similarly, lowvisor code must also be explicitlymapped to Hyp mode to run in Hyp mode. Extra caremust be taken to either link separate C object files, whichcannot call standard kernel functions, into sections that

6

can be mapped into Hyp mode, or write all the code run-ning in Hyp mode in assembly to have a single blockof code that can be mapped into Hyp mode. Since thelowvisor operations are low-level in nature and wouldrequire substantial portions of assembly even if C wasused, we opted for the latter approach of writing alllowvisor code in a separate assembly file linked at a spe-cific section by a linker script, making it easy to map itinto Hyp mode.

3.2 CPU Virtualization

To virtualize the CPU, KVM/ARM must present an in-terface to the VM which is essentially identical to theunderlying real hardware CPU, while ensuring that thehypervisor remains in control of the hardware. This in-volves ensuring that software running in the VM musthave persistent access to the same register state as soft-ware running on the physical CPU, as well as ensuringthat physical hardware state associated with the host ker-nel is persistent across running VMs. Register state notaffecting the hypervisor can simply be saved to and re-stored from memory when switching from a VM to thehost and vice versa. KVM/ARM configures access to allother sensitive state to trap, so it can be emulated in thehighvisor.

Table 1 shows the state belonging to the hypervisor ora VM, and KVM/ARM’s virtualization method for eachstate category. KVM/ARM performs trap and emulateon sensitive instructions and when accessing hardwarestate that could affect the hypervisor or would leak in-formation about the hardware to the VM that violates itsvirtualized abstraction. For example, KVM/ARM trapsif a VM executes the WFI instruction, which causes theCPU to power down, because such an operation shouldonly be performed by the hypervisor to maintain controlof the hardware. Because trap and emulate can be expen-sive, KVM/ARM uses save/restore of registers wheneverpossible to reduce the frequency of traps by leveragingARM hardware support for virtualization. For example,access to control registers such as the Stage-1 page ta-ble base register is configured not to trap to Hyp modebut are instead saved and restored, allowing a VM to ac-cess hardware state directly whenever the hardware sup-ports it and avoiding traps on common guest OS opera-tion such as a context switch.

Register state used by a VM needs to be contextswitched if it will be used by the hypervisor or anotherVM. In particular, once a VM traps to the hypervisor,the hypervisor may choose to run another VM. How-ever, in many cases, the hypervisor is likely to run thesame VM again. If the register state will not be used bythe hypervisor, it can be switched lazily only when an-other VM is run instead of switching the state on every

Action Nr. State

Save/Restore

38 General Purpose (GP) Registers26 Control Registers16 VGIC Control Registers

4 VGIC List Registers2 Arch. Timer Control Registers

32 64-bit VFP registers4 32-bit VFP Control Registers

Trap-and-emulate

- CP14 Trace Registers- WFI Instructions- SMC Instructions- ACTLR Access- Cache ops. by Set/Way- L2CTLR / L2ECTLR Registers

Table 1: VM and host state on a Cortex-A15 ARMv7CPU.

switch between the hypervisor and the VM. For exam-ple, the hypervisor is not likely to perform floating pointoperations, but there is a significant amount of floatingpoint state in the VFP registers to save and restore. To re-duce the cost of trapping to the hypervisor, KVM/ARMdoes a lazy context switch for floating point state. Whenswitching to a VM from the hypervisor, KVM/ARMconfigures the hardware to trap on all access to the VFPstate. When the VM then accesses VFP state, it trapsinto the hypervisor, which only then context switches theVFP registers and removes the trap configuration for fu-ture accesses by the VM, until the next context switchback to the hypervisor. Since all context switching isdone by the lowvisor, no double trap is incurred on alazy context switch. If the VM does not access VFP reg-isters, KVM/ARM does not need to context switch theVFP registers. Other state may also benefit from lazycontext switching, but for simplicity, we have only im-plemented this functionality for floating point state in thecurrent unoptimized release.

The difference between running inside a VM in kernelor user mode and running the hypervisor in kernel or usermode is determined by how the virtualization extensionshave been configured by Hyp mode during the worldswitch. A world switch from the highvisor to a VM per-forms the following actions: (1) store all highvisor GPregisters on the Hyp stack, (2) configure the VGIC forthe VM, (3) configure the timers for the VM, (4) save allhighvisor-specific configuration registers onto the Hypstack, (5) load the VM’s configuration registers onto thehardware, which can be done without affecting currentexecution, because Hyp mode uses its own configurationregisters, separate from the highvisor state, (6) config-ure Hyp mode to trap floating-point operations for lazycontext switching, trap interrupts, trap CPU halt instruc-tions (WFI/WFE), trap SMC instructions, trap specificconfiguration register accesses, and trap debug register

7

accesses, (7) write VM-specific IDs into shadow ID reg-isters, (8) set the Stage-2 page table base register (VT-TBR) and enable Stage-2 address translation, (9) restoreall guest GP registers, (10) trap into either user or kernelmode.

The CPU will stay in the VM world until an event oc-curs, which triggers a trap into Hyp mode. Such an eventcan be caused by any of the traps mentioned above, aStage-2 page fault, or a hardware interrupt. Since theevent requires services from the highvisor, either to em-ulate the expected hardware behavior for the VM or toservice a device interrupt, KVM/ARM must perform an-other world switch back into the highvisor. The worldswitch back to the highvisor performs the following ac-tions: (1) store all guest GP registers, (2) disable Stage-2 translation, (3) configure Hyp mode to not trap anyregister access or instructions, (4) save all VM-specificconfiguration registers, (5) load the highvisor’s config-uration registers onto the hardware, (6) configure thetimers for the highvisor, (7) save VM-specific VGICstate, (8) restore all highvisor GP registers, (9) trap intokernel mode.

3.3 Memory Virtualization

KVM/ARM provides memory virtualization by ensur-ing that a VM cannot access physical memory belong-ing to the hypervisor or other VMs, including any sen-sitive data. KVM/ARM uses Stage-2 translation to con-trol physical memory access within a VM by configur-ing the Stage-2 translation page tables to only allow ac-cess to certain regions of memory; other accesses willcause Stage-2 page faults which trap to the hypervisor.Since Stage-2 translation can only be configured in Hypmode, its use is completely transparent to the VM. Whenthe hypervisor performs a world switch to a VM, it en-ables Stage-2 translation and configures the Stage-2 pagetable base register accordingly. Since Stage-2 transla-tion is disabled by the lowvisor when switching back tothe highvisor, the hypervisor has unfettered access to allphysical memory, including the Stage-2 page tables foreach VM. Although both the highvisor and VMs can runin kernel mode, Stage-2 translations ensure that the high-visor is protected from any access by the VMs.

KVM/ARM uses split mode virtualization to lever-age existing kernel memory allocation, page referencecounting, and page table manipulation code. WhereKVM/ARM handles most Stage-2 page faults by simplycalling get_user_pages and map the returned pageto a guest, a bare-metal hypervisor would be forced toeither statically allocate memory to VMs or write a newmemory allocation subsystem.

3.4 I/O Virtualization

KVM/ARM leverages existing QEMU and Virtio [22]user space device emulation to provide I/O virtualiza-tion. At a hardware level, all I/O mechanisms on theARM architecture are based on load/store operations toMMIO device regions. With the exception of devicesdirectly assigned to VMs, all hardware MMIO regionsare inaccessible from VMs. KVM/ARM uses Stage-2translations to ensure that physical devices cannot be ac-cessed directly from VMs. Any access outside of RAMregions allocated for the VM will trap to the hypervisor,which can route the access to a specific emulated de-vice in QEMU based on the fault address. This is some-what different from x86, which uses x86-specific hard-ware instructions such as inl and outl for port I/Ooperations in addition to MMIO. As we show in Sec-tion 5, KVM/ARM achieves low I/O performance over-head with very little implementation effort.

3.5 Interrupt Virtualization

KVM/ARM leverages its tight integration with Linux toreuse existing device drivers and related functionality,including handling interrupts. When running in a VM,KVM/ARM configures Hyp mode to trap hardware in-terrupts to Hyp mode, and performs a world switch tothe highvisor to handle the interrupt, so that the hypervi-sor remains in complete control of hardware resources.When already running in the highvisor, KVM/ARM con-figures Hyp mode to trap interrupts directly to kernelmode, avoiding the overhead from going through Hypmode. In both cases, essentially all of the work is done inthe highvisor by reusing Linux’s existing interrupt han-dling functionality.

However, VMs must receive notifications in the formof interrupts from emulated devices and multicore guestOSes must be able to send virtual IPIs from one virtualcore to another. KVM/ARM uses the VGIC to injectvirtual interrupts into VMs and to reduce the number oftraps to Hyp mode. As described in Section 2, virtualinterrupts are raised to virtual CPUs by programmingthe list registers in the MMIO virtual CPU control in-terface. KVM/ARM configures the Stage-2 page tablesto prevent VMs from accessing the control interface, andthereby ensures that only the hypervisor can program thecontrol interface. However, virtual interrupts cannot nec-essarily be programmed directly into list registers, be-cause another VM may be using the physical CPU, orbecause there are more virtual interrupts than availablelist registers. Further, most devices are not aware of theaffinity of interrupts to CPU cores, but simply raise in-terrupts to the GIC distributor. Consequently, a softwarelayer must exist between the emulated devices and the

8

virtual CPU interface.KVM/ARM introduces the virtual distributor, a soft-

ware model of the GIC distributor as part of the high-visor. The vitual distributor exposes an interface to userspace, so emulated devices in user space can raise virtualinterrupts to the virtual distributor. The virtual distribu-tor keeps internal software state about the state of eachinterrupt and uses this state whenever a VM is scheduledto program the list registers to inject virtual interrupts.For example, if the hardware only exposes four list reg-isters, but there are eight pending virtual interrupts, thevirtual distributor will choose the four interrupts withhighest priority and program the list registers with thosefour interrupts. Additionally, guest OS software runninginside the VM may program the distributor to configurepriorities, disable certain interrupts, or to send IPIs toother virtual CPUs. Such operations will trap to the hy-pervisor and be routed to the virtual distributor, whichwill populate the effects to its software model or directlygenerate virtual interrupts through the list registers.

Ideally, the virtual distributor only accesses the hard-ware list registers when necessary, since device MMIOoperations are typically significantly slower than cachedmemory accesses. A complete context switch of the listregisters is necessary when scheduling a different VMto run on a physical core, but unnecessary when simplyswitching between a VM and the hypervisor since thehypervisor disables all access to the virtual CPU inter-face when it runs. However, since the virtual distribu-tor is a complicated piece of software with shared stateacross multiple processes and physical CPUs, the ini-tial unoptimized version of KVM/ARM uses a simplifiedapproach which completely context switches all VGICstate including the list registers on each world switch.

3.6 Timer Virtualization

Reading counters and programming timers are frequentoperations in many OSes to manage process schedulingand to regularly poll device state. For example, Linuxreads a counter to determine if a process has expiredits time slice, and programs timers to ensure that pro-cesses don’t exceed their allowed time slices. Applica-tion workloads also often leverage timers for various rea-sons. Trapping to the hypervisor for each such operationis likely to incur noticeable performance overheads, andallowing a VM direct access to the time-keeping hard-ware typically implies giving up timing control of thehardware resources as VMs can disable timers and con-trol the CPU for extended periods of time.

KVM/ARM leverages ARM’s hardware virtualizationfeatures of the generic timers to allow VMs direct ac-cess to reading counters and programming timers with-out trapping to Hyp mode while at the same time ensur-

ing the hypervisor remains in control of the hardware.Since access to the physical timers is controlled usingHyp mode, any software controlling Hyp mode has ac-cess to the physical timers. KVM/ARM maintains hard-ware control by using the physical timers in the hyper-visor and disallowing access to physical timers from theVM. We note that unmodified legacy Linux kernels onlyaccess the virtual timer and can therefore directly accesstimer hardware without trapping to the hypervisor.

Unfortunately, due to architectural limitations, the vir-tual timers cannot directly raise virtual interrupts, butwill instead always raise hardware interrupts, which trapto the hypervisor. KVM/ARM detects when a virtualtimer programmed by a VM expires, and injects a cor-responding virtual interrupt to the VM, performing allhardware ACK and EOI operations in the highvisor. Fur-ther, the hardware only provides a single virtual timer perphysical CPU, and multiple virtual CPUs may be multi-plexed across this single hardware instance. To supportvirtual timers in this scenario, KVM/ARM detects un-expired timers when a VM traps to the hypervisor andleverages existing OS functionality to program a soft-ware timer at the time when the virtual timer would haveotherwise fired, had the VM been left running. Whensuch a software timer fires, a callback function is exe-cuted, which raises a virtual timer interrupt to the VMusing the virtual distributor described above.

4 Implementation and AdoptionWe have successfully integrated our work into the Linuxkernel and KVM/ARM is now the standard ARM hyper-visor on Linux platforms, as it is included in every kernelbeginning with version 3.9. We share some lessons welearned from our experiences in hopes that they may behelpful to others in getting research ideas widely adoptedby an existing open source community.

Code maintainability is key. It is a common miscon-ception that a research software implementation provid-ing potential improvements or interesting new featurescan simply be open sourced and thereby quickly inte-grated by the open source community. An importantpoint that is often not taken into account is that any im-plementation must be maintained. If an implementationrequires many people and much effort to be maintained,it is much less likely to integrated into existing opensource code bases. Because maintainability is so cru-cial, reusing code and interfaces is important. For exam-ple, KVM/ARM builds on existing infrastructure such asKVM and QEMU, and from the very start we prioritizedaddressing code review comments to make our code suit-able for integration into existing systems. An unexpected

9

but important benefit of this decision was that we couldleverage the community for help to solve hard bugs orunderstand intricate parts of the ARM architecture.

Be a known contributor. Convincing maintainers tointegrate code is not just about the code itself, but alsoabout who submits it. It is not unusual for researchersto complain about kernel maintainers not accepting theircode into Linux only to have some known kernel devel-oper submit the same idea and have it accepted. Thereason is an issue of trust. Establishing trust is a catch-22. On the one hand one must be well-known to sub-mit code; yet one cannot become known without sub-mitting code. One way to do this is to start small. Aspart of our work, we also made various small changes toKVM to prepare the support for ARM, which includedcleaning up existing code to be more generic and im-prove cross platform support. The KVM maintainerswere glad to accept these small improvements, whichgenerated goodwill and helped us become known to theKVM community.

Make friends and involve the community. Opensource development turns out to be quite a social enter-prise. Networking with the community helps tremen-dously, not just online, but in person at conferences andother venues. For example, at an early stage in the de-velopment of KVM/ARM, we traveled to ARM head-quarters in Cambridge, UK to establish contact withboth ARM management and the ARM kernel engineer-ing team, who both contributed to our efforts.

As another example, an important issue in integrat-ing KVM/ARM into the kernel was agreeing on variousinterfaces for ARM virtualization, such as reading andwriting control registers. Since it is an established pol-icy to never break released interfaces and compatibilitywith user space applications, existing interfaces cannotbe changed, and the community puts great efforts intodesigning extensible and reusable interfaces. Decidingon the appropriateness of an interface is a judgment calland not an exact science. We were fortunate enough toreceive help from well-known and respected kernel de-velopers such as Rusty Russell, who helped us drive boththe implementation and communication about our inter-faces, specifically for the purpose of user space save andrestore of registers, a feature useful for both debuggingand VM migration. Working with an established devel-oper like Rusty was a tremendous help, both because wecould leverage his experience, but also because he has astrong voice in the kernel community.

Involve the community early. An important issue indeveloping KVM/ARM was how to get access to Hyp

mode across the plethora of available ARM SoC plat-forms supported by Linux. One approach would be toinitialize and configure Hyp mode when KVM is initial-ized, which would isolate the code changes to the KVMsubsystem. However, because getting into Hyp modefrom the kernel involves a trap, early stage bootloadermust have already installed code in Hyp mode to handlethe trap and allow KVM to run. If no such trap handlerwas installed, trapping to Hyp mode could end up crash-ing the kernel. We worked with the kernel community todefine the right ABI between KVM and the bootloader,but soon learned that agreeing on ABIs with SoC ven-dors had historically been difficult.

In collaboration with ARM and the open source com-munity, we reached the conclusion that if we simply re-quired the kernel to be booted in Hyp mode, we wouldnot have to rely on fragile ABIs. The kernel then sim-ply tests when it starts up whether it is in Hyp mode, inwhich case it installs a trap handler to provide a hookto re-enter Hyp mode at a later stage. A small amountof code must be added to the kernel boot procedure, butthe result is a much cleaner and robust mechanism. If thebootloader is Hyp mode unaware and the kernel does notboot up in Hyp mode, KVM/ARM will detect this andwill simply remain disabled. This solution avoids theneed to design a new ABI and it turned out that legacykernels would still work, because they always make anexplicit switch into kernel mode as their first instruction.These changes were merged into the mainline Linux 3.6kernel, and official ARM kernel boot recommendationswere modified to recommend that all bootloaders bootthe kernel in Hyp mode to take advantage of the new ar-chitecture features.

Know the chain of command. There were multiplepossible upstream paths for KVM/ARM. Historically,other architectures supported by KVM such as x86 andPowerPC were merged through the KVM tree directlyinto Linus Torvalds’s tree with the appropriate approvalof the respective architecture maintainers. KVM/ARM,however, required a few minor changes to ARM-specificheader files and the idmap subsystem, and it was there-fore not clear whether the code would be integrated viathe KVM tree with approval from the ARM kernel main-tainer or via the ARM kernel tree. Russell King is theARM kernel maintainer, and Linus pulls directly fromhis ARM kernel tree for ARM-related code. The situ-ation was particularly interesting, because Russell Kingdid not want to merge virtualization support in the main-line kernel [17] and he did not review our code. Atthe same time, the KVM community was quite inter-ested in integrating our code, but could not do so withoutapproval from the ARM kernel maintainer, and RussellKing refused to engage in a discussion about this proce-

10

dure.

Be persistent. While we were trying to merge our codeinto Linux, a lot of changes were happening aroundLinux ARM support in general. The amount of churnin SoC support code was becoming an increasingly bigproblem for maintainers, and great efforts were under-way to reduce board specific code and support a singleARM kernel binary bootable across multiple SoCs. Inlight of these ongoing changes, getting enough time fromARM kernel maintainers to review the code was chal-lenging, and there was even extra pressure on the main-tainers to be highly critical of any new code merged intothe ARM tree. Therefore, we had no choice but to keepmaintaining and improving the code, and regularly sendout updated patch series that followed upstream kernelchanges. Eventually, Will Deacon, one of the ARMkernel engineers listed in the ARM kernel maintainersfile, made time for several comprehensive and helpfulreviews, and after addressing his concerns, he gave ushis approval of the code. After all this, when we thoughtwe were done, we finally received some feedback fromRussell King.

When MMIO operations trap to the hypervisor, thevirtualization extensions populate a register which con-tains information useful to emulate the instruction(whether it was a load or a store, source/target registers,and the length of MMIO accesses). A certain class ofinstructions used by older Linux kernels do not popu-late such a register. KVM/ARM therefore loads the in-struction from memory and decodes it in software. Eventhough the decoding implementation was well tested andreviewed by a large group of people, Russell King ob-jected to including this feature. He had already imple-mented multiple forms of instruction decoding in othersubsystems and demanded that we either rewrite signif-icant parts of the ARM kernel to unify all instructiondecoding to improve code reuse, or drop the MMIOinstruction decoding support from our implementation.Rather than pursue a rewriting effort that could drag onfor months, we had no choice but to abandon the other-wise well-liked and useful code base. We can only spec-ulate about the true motives behind this decision, as theARM kernel maintainers would not engage in a discus-sion about the subject.

After 15 main patch revisions and more than 18months, the KVM/ARM code was successfully mergedinto Linus’s tree via Russell King’s ARM tree in Febru-ary 2013. In getting all these things to come together inthe end before the 3.9 merge window, the key was havinga good relationship with many of the kernel developersto get their help, and being persistent in continuing topush to have the code merged in the face of various chal-lenges.

5 Experimental Results

We present some experimental results that quantify theperformance of KVM/ARM on real multicore ARMhardware. We evaluate the virtualization overhead ofKVM/ARM compared to direct execution by runningboth microbenchmarks and real application workloadswithin VMs and directly on the hardware. We measureboth the performance and energy virtualization costs ofusing KVM/ARM. We also compare the virtualizationand implementation costs of KVM/ARM versus KVMx86 to demonstrate the effectiveness of KVM/ARMagainst a more mature hardware virtualization platform.These results provide the first real hardware measure-ments of the performance of ARM hardware virtual-ization support as well as the first comparison betweenARM and x86.

5.1 Methodology

We used one ARM and two x86 platforms for our mea-surements. ARM measurements were obtained using anInsignal Arndale board [14] with a dual core 1.7GHzCortex A-15 CPU on a Samsung Exynos 5250 SoC. Thisis the first and most widely used commercially availabledevelopment board based on the Cortex A-15, the firstARM CPU with hardware virtualization support. On-board 100Mb Ethernet is provided via the USB bus andan external 120GB Samsung 840 series SSD drive wasconnected to the Arndale board via eSATA. x86 mea-surements were obtained using both a low-power mobilelaptop platform and an industry standard server platform.The laptop platform was a 2011 MacBook Air with adual core 1.8GHz Core i7-2677M CPU, an internal Sam-sung SM256C 256GB SSD drive, and an Apple 100MbUSB Ethernet adapter. The server platform was a ded-icated OVH SP 3 server with a dual core 3.4GHz In-tel Xeon E3 1256v2 CPU, two physical SSD drives ofwhich only one was used, and 1GB Ethernet connectedto a 100Mb network infrastructure.

Given the differences in hardware platforms, our focuswas not on measuring absolute performance, but ratherthe relative performance differences between virtualizedand direct execution on each platform. Since our goal isto evaluate hypervisors, not raw hardware performance,this relative measure provides a useful cross-platformbasis for comparing the virtualization performance andpower costs of KVM/ARM versus KVM x86.

To provide comparable measurements, we kept thesoftware environments across all hardware platforms thesame as much as possible. Both the host and guest VMson all platforms were Ubuntu version 12.10. Becausethe Linux 3.9 kernel is not yet released as of the time ofthis writing, device support is not complete and there-

11

fore it does not run across all of the hardware platformswe used. As a result, we used the Linux 3.6 kernel forour experiments, with patches for KVM/ARM rebasedon top of the source tree. Since the experiments wereperformed on a number of different platforms, the kernelconfigurations had to be slightly different, but all com-mon features were configured similarly across all plat-forms. In particular, Virtio drivers were used in the guestVMs on both ARM and x86. All systems were config-ured with a maximum of 1.5GB of RAM available to therespective guest VM or host being tested. Furthermore,all multicore measurements were done using two physi-cal cores and single-core measurements were configuredwith SMP disabled in the kernel configuration of boththe guest and host system; hyperthreading was disabledon the x86 platforms. CPU frequency scaling was dis-abled to ensure that native and virtualized performancewas measured at the same clock rate on each platform.

apache Apache v2.2.22 Web server runningApacheBench v2.3, which measures numberof handled requests per seconds serving theindex file of the GCC 4.4 manual using 100concurrent requests

mysql MySQL v14.14 (distrib 5.5.27) running theSysBench OLTP benchmark using the de-fault configuration

memcached memcached v1.4.14 using the memslapbenchmark with a concurrency parameter of100

kernelcompile

kernel compilation by compiling the Linux3.6.0 kernel using the vexpress defconfig forARM using GCC 4.7.2 on ARM and theGCC 4.7.2 arm-linux-gnueabi- crosscompilation toolchain on x86

untar untar extracting the 3.6.0 Linux kernel im-age compressed with bz2 compression usingthe standard tar utility

curl 1K curl v7.27.0 downloading a 1KB randomlygenerated file 1,000 times from the respec-tive iMac or OVH server and saving the re-sult to /dev/null with output disabled,which provides a measure of network latency

curl 1G curl v7.27.0 downloading a 1GB ran-domly generated file from the respectiveiMac or OVH server and saving the resultto /dev/null with output disabled, whichprovides a measure of network throughput

hackbench hackbench [19] using unix domain sock-ets and 100 process groups running with 500loops

Table 2: Benchmark applications workloads.

For measurements involving the network and anotherserver, 100Mb Ethernet was used on all systems. The

ARM and x86 laptop platforms were connected us-ing a Netgear GS608v3 switch, and a 2010 iMac witha 3.2GHz Core i3 CPU with 12GB of RAM runningMac OS X Mountain Lion was used as a server. Thex86 server platform was connected to a 100Mb port inthe OVH network infrastructure, and another identicalserver in the same data center was used as the server.While there are some differences in the network infras-tructure used for the x86 server platform because it iscontrolled by someone else, we do not expect these dif-ferences to have any significant impact on the relativeperformance between virtualized and native execution.

We present results for four sets of experiments. First,we measured the cost of various micro-architecturalcharacteristics of the hypervisors using custom smallguest OSes only written for testing aspects of the respec-tive systems. We further instrumented the code on bothKVM/ARM and KVM x86 to read the cycle counter atspecific points along critical paths to more accurately de-termine where overhead time was spent.

Second, we measured the cost of a number of com-mon low-level OS operations using lmbench [18] v3.0on both single and multicore. When running lmbenchon multicore configurations, we pinned each benchmarkprocess to a separate CPU to measure the true overheadof interprocessor communication in VMs on multicoresystems.

Third, we measured real application performance us-ing a variety of workloads running both natively and inVMs, and compared the normalized performance acrossARM and x86 hypervisors. Table 2 shows the applica-tions we used.

Fourth, we measured energy efficiency using the sameeight application workloads used for measuring appli-cation performance. ARM power measurements wereperformed using an ARM Energy Probe [5] and mea-sure power consumption over a shunt attached to thepower supply of the Arndale board. Power to the exter-nal SSD was delivered by attaching a USB power cableto the USB ports on the Arndale board thereby factor-ing storage power into the total SoC power measured atthe power supply. x86 power measurements were per-formed using the powerstat tool, which reads ACPIinformation. powerstat measures total system powerdraw from the battery, so power measurements on thex86 system were run from battery power and could onlybe run on the x86 laptop platform. Although we did notmeasure the power efficiency of the x86 server platform,it is expected to be much less efficient that the x86 lap-top platform, so using the x86 laptop platform providesa conservative comparison of energy efficiency againstARM. The display and wireless features of the x86 lap-top platform were turned off to ensure a fair comparison.Both tools reported instantaneous power draw in watts

12

with a 10Hz interval. These measurements were aver-aged and multiplied by the duration of the test to obtainan energy measure.

5.2 Performance Measurements

Micro Test ARM ARM no x86 x86vgic/vtimers laptop server

Hypercall 4,917 2,112 1,263 1,642Trap 27 27 632 821I/O Kernel 6,248 2,945 2,575 3,049I/O User 6,908 3,971 8,226 10,356IPI 10,534 - 13,670 16,649EOI 9 - 1,713 2,195

Table 3: Micro-architectural cycle counts.

Table 3 presents various micro-architectural costs ofvirtualization using KVM/ARM on ARM and KVM x86on x86. Measurements are shown in cycles instead oftime to provide a useful comparison across platformswith different CPU frequencies. We show two numbersfor the ARM platform where possible, with and withoutVGIC and virtual timers support.

Hypercall is the cost of two world switches, goingfrom the VM to the host and immediately back againwithout doing any work in the host. KVM/ARM takesthree to four times as many cycles for this operation ver-sus KVM x86 due to two main factors. First, saving andrestoring VGIC state to use virtual interrupts is quite ex-pensive on ARM; available x86 hardware does not yetprovide such mechanism. The ARM no VGIC/vtimersshows measurements without the cost of saving andrestoring VGIC state and this accounts for over half ofthe cost of a world switch on ARM. Second, x86 pro-vides hardware support to save and restore state on theworld switch, which is much faster. ARM requires soft-ware to explicitly save and restore state, which providesgreater flexibility, but higher costs. Nevertheless, with-out the VGIC state, the world switch costs are only about500 cycles more than the hardware accelerated worldswitch cost on the x86 server platform.

Trap is the cost of switching the hardware mode fromthe VM into the respective CPU mode for running thehypervisor, Hyp mode on ARM and root mode on x86.ARM is much faster than x86 because it only needs tomanipulate two registers to perform this trap, whereasthe cost of a trap on x86 is the same as the cost of aworld switch because the same amount of state is savedby the hardware in both cases. The trap cost on ARMis a very small part of the world switch costs, indicatingthat the double trap incurred by split-mode virtualizationon ARM does not add much overhead.

I/O Kernel is the cost of an I/O operation (inl on x86,

MMIO load on ARM) to a device, which is emulated in-side the kernel and returning to the VM. I/O User showsthe cost of issuing an I/O read operation from a deviceemulated in user space, adding to I/O Kernel the cost oftransitioning from the kernel to a user space process onthe host for I/O. This is representative of the cost of usingQEMU. Since these operations involve world switches,saving and restoring VGIC state is again a significantcost on ARM. KVM x86 is much faster than KVM/ARMon I/O Kernel, but slower on I/O User. This is becausethe hardware optimized world switch on x86 constitutesthe majority of the cost of performing I/O in the kernel,but transitioning from kernel to a user space process onthe host side is more expensive on x86 than on ARM.

IPI is the cost of issuing an IPI to another virtual CPUcore when both virtual cores are running on separatephysical cores and both are actively running inside theVM. The IPI test measures the time starting from send-ing an IPI until the other virtual core responds to theIPI; it does not include time spent by the other core sig-naling completion of the interrupt to the interrupt con-troller. ARM is somewhat faster because the underlyinghardware IPI on x86 is expensive, and x86 APIC MMIOoperations require KVM x86 to perform instruction de-coding not needed on ARM.

EOI is the cost of completing an interrupt on both plat-forms. It includes both interrupt acknowledgment andcompletion on ARM, but only completion on the x86platform. ARM requires an additional operation, the ac-knowledgment, to the interrupt controller to determinethe source of the interrupt. x86 does not because thesource is directly indicated by the interrupt descriptortable entry at the time when the interrupt is raised. How-ever, the operation is roughly 200 times faster on ARMthan x86 because there is no need to trap to the hyper-visor on ARM because of VGIC support for both opera-tions. On x86, the EOI operation must be emulated andtherefore causes a trap to the hypervisor. This operationis required for every virtual interrupt including both vir-tual IPIs and interrupts from virtual devices.

Figures 5a and 5b show normalized performance forrunning lmbench in a VM versus running directly on thehost. Figure 5a shows that KVM/ARM and KVM x86have similar virtualization overhead in a single core con-figuration. For comparison, we also show KVM/ARMperformance without VGIC/vtimers. Overall, usingVGIC/vtimers provides slightly better performance ex-cept for the pipe and context switch workloads where thedifference between using and not using VGIC/vtimers issubstantial. The high overhead without VGIC/vtimers inthese cases is caused by updating the runqueue clock inthe Linux scheduler every time a process blocks, sincereading a counter traps to user space without vtimerson the ARM platform. We verified this by running the

13

11.19  

10.65  

0.00  

1.00  

2.00  

3.00  

4.00  

5.00  

6.00  

7.00  

8.00  

9.00  

10.00  

fork  

exec  

ctxsw  2p  

ctxsw  8p  

ctxsw  16p  

pipe  

prot  fault  

page  fault  

af_unix   tcp

 

ARM  

ARM  no  vgic/vDmers  

x86  Laptop  

x86  Server  

(a) UP VM normalized lmbench performance

10.31  

12.42  

15.68  

0.00  

1.00  

2.00  

3.00  

4.00  

5.00  

6.00  

7.00  

8.00  

9.00  

10.00  

fork  

exec  

ctxsw  2p  

ctxsw  8p  

ctxsw  16p  

pipe  

prot  fault  

page  fault  

af_unix   tcp

 

ARM  

ARM  no  vgic/vDmers  

x86  Laptop  

x86  Server  

(b) SMP VM normalized lmbench performance

0.00  

0.50  

1.00  

1.50  

2.00  

2.50  

3.00  

Apache  

MySQL  

memcach

ed  

kernel  co

mpile   unta

r  curl1

k  curl1

g  

hackben

ch  

ARM   ARM  no  vgic/vBmers   x86  Laptop   x86  Server  

(c) UP VM normalized application performance

0.00  

0.50  

1.00  

1.50  

2.00  

2.50  

3.00  

Apache  

MySQL  

memcach

ed  

kernel  co

mpile   unta

r  curl1

k  curl1

g  

hackben

ch  

ARM   ARM  no  vgic/vBmers   x86  Laptop   x86  Server  

(d) SMP VM normalized application performance

Figure 5: UP and SMP performance measurements. All results are normalized virtual relative to native with lowerbeing less overhead.

workload with VGIC support, but without vtimers, andwe counted the number of timer read exits when runningwithout vtimers support.

Figure 5b shows more substantial differences in virtu-alization overhead between KVM/ARM and KVM x86in a multicore configuration. KVM/ARM has less over-head than KVM x86 fork and exec, but more for pagefaults. Both systems have the worst overhead for thepipe and context switch workloads, though KVM x86 ismore than three times worse for pipe. These differencesare due to the cost of repeatedly sending an IPI from thesender of the data in the pipe to the receiver for each mes-sage. x86 not only has higher IPI overhead than ARM,but it must also EOI each IPI, which is much more ex-pensive on x86 than on ARM because this requires trap-ping to the hypervisor on x86 but not on ARM. Withoutusing VGIC/vtimers, KVM/ARM also incurs high over-

head comparable to KVM x86 because it then also trapsto the hypervisor to EOI the IPIs.

Figures 5c and 5d show normalized performance forrunning application workloads in a VM versus runningdirectly on the host. Figure 5c shows that KVM/ARMand KVM x86 have similar virtualization overheadacross all workloads in a single core configuration, butFigure 5d shows that there are more substantial dif-ferences in performance on multicore. On multicore,KVM/ARM has significantly less virtualization over-head than KVM x86 on Apache and MySQL. Overall onmulticore, KVM/ARM performs within 10% of runningdirectly on the hardware for most application workloads,and the highest virtualization overheads for Apache andMySQL are still significantly less than the more ma-ture KVM x86 system. KVM/ARM’s split-mode vir-tualization design allows it to leverage ARM hardware

14

support with comparable performance to a traditionalhypervisor using x86 hardware support. The measure-ments also show that KVM/ARM performs better overallwith than without ARM VGIC/vtimers support. UsingVGIC/vtimers to reduce the number of world switchesoutweighs the additional world switch overhead fromsaving and restoring VGIC/vtimer state.

5.3 Power Measurements

0.00  

0.50  

1.00  

1.50  

2.00  

2.50  

3.00  

Apache  

MySQL  

memcach

ed  

kernel  co

mpile   unta

r  curl1

k  curl1

g  

hackben

ch  

ARM   ARM  no  vgic/vBmers   x86  Laptop  

Figure 6: SMP VM normalized energy consumption.

Figure 6 shows normalized power consumption of us-ing virtualization versus direct execution for various ap-plication workloads. We only compared KVM/ARM onARM against KVM x86 on x86 laptop. The Intel Corei7 CPU used here is one of Intel’s more power optimizedprocessors, and we expect that server power consump-tion would be even higher. The measurements show thatKVM/ARM using VGIC/vtimers is more power efficientthan KVM x86 virtualization in all cases except kernelcompile. This is most likely explained by the CPU inten-sive operation of compilation compared to the otherwisemore balanced workloads, which indicates that ARM ismore optimized considering an entire SoC, where x86suffers more extreme power consumption, when for ex-ample being memory bound. While a more detailedstudy of energy aspects of virtualization are beyond thescope of this paper, these measurements neverthelessprovide useful data comparing ARM and x86 virtualiza-tion energy costs.

5.4 Implementation Complexity

We compare the code complexity of KVM/ARM to itsKVM x86 counterpart. KVM/ARM is 5,622 lines ofcode (LOC), counting just the architecture-specific code

added to Linux to implement it, of which the lowvisoris a mere 754 LOC. KVM/ARM’s LOC is less than par-tially complete bare-metal microvisors written for Hypmode [24], with the lowvisor LOC almost an order ofmagnitude smaller. As a conservative comparison, KVMx86 is 24,674 LOC, excluding guest performance mon-itoring support, not yet supported by KVM/ARM, and3,305 LOC required for AMD support. These numbersdo not include KVM’s architecture-generic code, 5,978LOC, which is shared by all systems. Table 4 shows abreakdown of the total architecture-specific code into itsmajor components.

Component KVM/ARM KVM x86 (Intel)Core CPU 1,876 15,691Page Fault Handling 673 4,261Interrupts 1,194 2,151Timers 221 630Other 1,658 1,941Architecture-specific 5,622 24,674

Table 4: Code complexity in Lines of Code (LOC).

By inspecting the code we notice that the striking ad-ditional complexity in the x86 implementation is mainlydue to the five following reasons: (1) Since EPT wasnot supported in earlier hardware versions, KVM x86must support shadow page tables. (2) The hardwarevirtualization support have evolved over time, requiringsoftware to conditionally check for support for a largenumber of features such as EPT. (3) A number of oper-ations require software decoding of instructions on thex86 platform. KVM/ARM’s out-of-tree MMIO instruc-tion decode implementation was much simpler, only 462LOC. (4) The various paging mode on x86 requires moresoftware logic to handle page faults. (5) x86 requiresmore software logic to support interrupts and timers thanARM, which provides VGIC/vtimers hardware supportthat reduces software complexity.

6 RecommendationsFrom our experiences building KVM/ARM, we offer afew recommendations for hardware designers to sim-plify and optimize future hypervisor implementations.

Share kernel mode memory model. The hardwaremode to run a hypervisor should use the same memorymodel as the hardware mode to run OS kernels. Softwaredesigners then have greater flexibility in deciding howtightly to integrate a hypervisor with existing OS kernels.ARM Hyp mode unfortunately did not do this, prevent-ing KVM/ARM from simply reusing the kernel’s page

15

tables in Hyp mode. This reuse would have simplifiedthe implementation and allowed for performance criti-cal emulation code to run in Hyp mode, avoiding a com-plete world switch in some cases. Some might argue thatthis recommendation makes for more complicated stan-dalone hypervisor implementations, but this is not reallytrue. For example, ARM kernel mode already has a sim-ple option to use one or two page table base registers tounify or split the address space. Our recommendationis different from the x86 virtualization approach, whichdoes not have a separate and more privileged, hypervi-sor CPU mode. Having a separate CPU mode has thepotential for improved performance for some hypervisordesigns, but without sharing the kernel memory model,makes other common hypervisor designs difficult.

Make VGIC state access fast, or at least infrequent.While VGIC support can improve performance espe-cially on multicore systems, our measurements alsoshow that access to VGIC state adds substantial overheadto world switches. This is caused by slow MMIO accessto the VGIC control interface in the critical path. Im-proving the MMIO access time is likely to improve VMperformance, but if this is not possible or cost-effective,MMIO accesses to the VGIC could at least be made lessfrequent. For example, a summary register could be in-troduced describing the state of each virtual interrupt.This could be read when performing a world switch fromthe VM to the hypervisor to get information which cancurrently only be obtained by reading all the list registers(see Section 3.5) on each world switch.

Completely avoid IPI traps. Hardware support tosend virtual IPIs directly from VMs without the needto trap to the hypervisor would improve performance.Hardware designers underestimate how frequent IPIs areon modern multicore OSes, and our measurements re-veal that sending IPIs adds significant overhead for someworkloads. The current VGIC design requires a trap tothe hypervisor to emulate access to the IPI register inthe distributor, and this emulated access must be syn-chronized between virtual cores using a software lockingmechanism, which adds significant overhead for IPIs.Current hardware supports receiving the virtual IPIs,which can be ACKed and EOIed without traps, but un-fortunately does not address the also important issue ofsending virtual IPIs.

Make virtual timers transparent and directly supportvirtual interrupts. While the virtual counter avoidsthe need to trap to the hypervisor when VMs need toupdate a counter, software written to read the physicalcounter will trap to hypervisors that virtualize time. In-

stead of providing two separate counters with a separateaccess mechanism, the hardware should expose a singleinterface to access a counter, and Hyp mode should de-cide whether such an operation reads the virtual or phys-ical counter.

Since virtual timers are programmed by VMs for useby VMs, it should not be necessary to trap to the hy-pervisor when a virtual timer expires. By instrument-ing KVM/ARM while running the workloads discussesin Section 5, we were able to determine that while pro-gramming a timer is not as frequent as reading a counter,it is still a fairly common operation, and supporting thisfeature would be like to both simplify hypervisor designand improve performance.

Expect hypervisors to swap. When hypervisors swappages to disk the underlying physical mappings of mem-ory pages change, without the knowledge of guest OSes,which can cause cache conflicts. For example, considera standard page containing executable guest code. Whenthis page is swapped to disk, its guest physical to hostphysical mapping is invalidated in the Stage-2 transla-tion tables by the hypervisor so that memory accesses tothat page occurring in the future will trap to the hyper-visor. When this happens, the hypervisor will allocateanother physical memory page and fill that page with thecode that was previously swapped out. However, sincethe page that was just swapped in, may have previouslybeen used by another VM and may have contained codemapped at the same virtual address, an entry with thesame signature may already exist in the instruction cachecontaining invalid code, and the VM may end up execut-ing the wrong instructions.

To address this problem, hardware support should beprovided so that hypervisors can perform cache mainte-nance operations on a page-per-page basis using physicalpage addresses. For example, one solution would be tosimply invalidate the entry for that page in the instruc-tion cache when swapping in the page. However, currenthardware does not necessarily allow this operation, de-pending on the type of hardware cache that is used in theSoC. For example, the ARM architecture allows SoCsto use Virtually-Indexed-Physically-Tagged (VIPT) in-struction caches, and in this case, it is not possible toidentify all potential mappings of guest virtual addressesto the physical address of a given page. As a result, it be-comes necessary to flush the entire cache for correctnessin the presence of swapping, resulting in a very undesir-able performance cost. To avoid this cost, since hypervi-sors are not aware of all possible virtual addresses usedto access that physical page, it must be possible for hy-pervisors to perform cache maintenance operations on apage-per-page basis using physical page addresses.

16

Prepare for memory deduplication. Memory dedu-plication is a commonly used technique to reduce physi-cal memory pressure in virtualized environments [25]. Itworks by letting two or more VMs have a mapping to thesame physical memory page, and marking all such map-pings read-only in the Stage-2 page tables. If a VM triesto write to such a page, a permission fault will trap tothe hypervisor, which can perform copy-on-write to copythe page content to a new page, which can be mappedwritable to the VM. However, the ARM hardware sup-port for virtualization does not report the GPA on per-mission faults; it only reports the GPA on a normal pagefault, where the page is not present. KVM/ARM doessupport memory deduplication by leveraging Linux’sbuilt-in KSM feature, but the software is unnecessarilycomplicated, because the GPA has to be manually re-solved by walking the guest’s Stage-1 page tables whentaking permission faults inside the lowvisor. Further,on multicore VMs, another virtual CPU may modify theStage-1 page tables in the process, and the lowvisor mustdetect such races and handle them accordingly. Sincememory deduplication is likely to be used by any hy-pervisor, hardware designers should consider the typ-ical path of copy-on-write handling and provision thehardware accordingly. In the case of ARM, this shouldinclude populating the GPA fault address register (HP-FAR) on permission faults.

7 Related Work

Virtualization has a long history [20], but has enjoyeda resurgence starting in the late 1990s. Most effortshave almost exclusively focused on virtualizing the x86architecture. While systems such as VMware [1, 8]and Xen [6] were originally based on software-onlyapproaches before the introduction of x86 hardwarevirtualization support, all x86 virtualization platforms,VMware, Xen, and KVM [16], now leverage x86 hard-ware virtualization support. Because x86 hardware vir-tualization support differs substantially from ARM inthe ability to completely run the hypervisor in the samemode as the kernel, x86 virtualization approaches do notlend themselves directly to take advantage of ARM hard-ware virtualization support.

Some x86 approaches also leverage the host kernel toprovide functionality for the hypervisor. VMware Work-station’s hypervisor makes use of host kernel mecha-nisms and device drivers, but cannot reuse host kernelcode since it is not integrated with the kernel, resultingin a more complex hypervisor to build and maintain. Incontrast, KVM benefits from being integrated with theLinux kernel like KVM/ARM, but the x86 design relieson being able to run the kernel and the hypervisor to-

gether in the same hardware hypervisor mode, which isproblematic on ARM.

Full-system virtualization of the ARM architecture isa relatively unexplored research area. Most approachesare software only. A number of bare metal hypervi-sors have been developed [11, 12, 21], but these are notwidespread, are developed specifically for the embed-ded market, and must be modified and ported to everysingle host hardware platform, limiting their adoption.An abandoned port of Xen for ARM [13] requires com-prehensive modifications to the guest kernel, and wasnever fully developed. VMware Horizon Mobile [7] useshosted virtualization to leverage Linux’s support for awide range of hardware platforms, but requires modifi-cations to guest OSes and its performance is unproven.An earlier prototype for KVM on ARM [9, 10] used anautomated lightweight paravirtualization approach to au-tomatically patch kernel source code to run as a guestkernel, but was developed prior to the introduction ofhardware support for virtualization on ARM. None ofthese paravirtualization approaches could run unmodi-fied guest OSes.

With the introduction of ARM hardware virtualiza-tion support, Varanasi and Heiser performed a study toroughly estimate its performance using a software simu-lator and an early prototype of a custom hypervisor lack-ing important features like SMP support and use of stor-age and network devices by multiple VMs [24]. Becauseof the lack of hardware or a cycle-accurate simulator, noreal performance evaluation was possible. In contrast,we present the first evaluation of ARM virtualization ex-tensions using real hardware, provide a direct compari-son with x86, and present the design and implementationof a complete hypervisor using ARM virtualization ex-tensions, including SMP support. Our conclusions basedon a complete hypervisor running on real hardware differfrom this previous study in terms of the simplicity of de-signing a hypervisor using Hyp mode and the real cost ofhypervisor operations. We further show how SMP sup-port introduces performance challenges and the benefitsof hardware support for virtualized timers, issues not ad-dressed by the previous study.

In parallel with our efforts, a newer version of Xenexclusively targeting servers [26] is being developed us-ing ARM hardware virtualization support. Because Xenis a bare metal hypervisor that does not leverage kernelfunctionality, it can be architected to run entirely in Hypmode rather than using split-mode virtualization. At thesame time, this requires a substantial commercial engi-neering effort and Xen ARM still does not have SMPsupport. We would have liked to have done a direct per-formance comparison between Xen and KVM/ARM, butXen still does not work on the popular Arndale boardused for our experiments. Because of Xen’s custom

17

I/O model using hypercalls from VMs for device emula-tion on ARM, Xen unfortunately cannot run guest OSesunless they have been configured to include Xen’s hy-percall layer and include support for XenBus paravirtu-alized drivers. In contrast, KVM/ARM uses standardLinux components to enable faster development, fullSMP support, and the ability to run unmodified OSes.KVM/ARM is easily supported on new devices withLinux support, and we spent almost no effort to sup-port KVM/ARM on ARM’s Versatile Express boards,the Arndale board, and hardware emulators. While Xencan potentially reduce world switch times for operationsthat can be handled inside the Xen hypervisor, switch-ing to Dom0 for I/O support or switching to other VMswould involve a context switching of the same state asKVM/ARM.

Microkernel approaches for hypervisors [23, 11] havebeen used to reduce the hypervisor TCB and run otherhypervisor services in user mode. These approachesdiffer both in design and rationale from split-mode vir-tualization, which splits hypervisor functionality acrossprivileged modes to leverage virtualization hardwaresupport. Split-mode virtualization also provides a dif-ferent split of hypervisor functionality. KVM/ARM’slowvisor is a much smaller code base that implementsonly the lowest level hypervisor mechanisms. It doesnot include higher-level functionality present in the hy-pervisor TCB used in these other approaches.

8 ConclusionsKVM/ARM is the first full system ARM virtualizationsolution that can run unmodified guest operating sys-tems on ARM multicore hardware. By introducing split-mode virtualization, it can leverage ARM virtualizationsupport while leveraging Linux kernel mechanisms andcode to simplify hypervisor development and maintain-ability. Our experimental results show that KVM/ARM(1) incurs minimal performance impact from the ex-tra traps incurred by split-mode virtualization, (2) hasmodest virtualization overhead and power costs, within10% of direct native execution on multicore hardwarefor most application workloads, and (3) can provide sig-nificantly lower virtualization overhead and power costscompared to the widely-used KVM x86 virtualizationon multicore hardware. We have integrated KVM/ARMinto the mainline Linux kernel, ensuring its wide adop-tion as the virtualization platform of choice for ARM.

9 AcknowledgmentsKVM/ARM benefited enormously from the help of theLinux community. Marc Zyngier implemented VGIC

and virtual timers support, provided invaluable insightin understanding their impact on virtualization perfor-mance, and helped with getting KVM/ARM runningon early ARM Cortex-A15 hardware. Peter Maydellprovided QEMU support and useful insights in under-standing the ARM architecture. Rusty Russell imple-mented parts of the world switch and instruction emu-lation mechanisms. Will Deacon provided helpful codereviews. Retired and current KVM maintainers, Avi Kiv-ity, Gleb Natapov, and Marcelo Tosatti, provided helpfulcode reviews and insight in understanding the KVM x86implementation.

Keith Adams, Hani Jamjoom, Oren Laadan, Erran Li,and Emmett Witchel provided helpful comments on ear-lier drafts of this paper. Richard Phelan made it pos-sible for us to experiment with early ARM Cortex-A15hardware. This work was supported in part by ARM, aGoogle Research Award, and NSF grants CNS-1162447,CNS-1018355, and CNS-0905246.

References[1] K. Adams and O. Agesen. A Comparison of Soft-

ware and Hardware Techniques for x86 Virtual-ization. In Proceedings of the 12th InternationalConference on Architectural Support for Program-ming Languages and Operating Systems, pages 2–13, Dec. 2006.

[2] ARM Ltd. ARM Cortex-A15 Technical ReferenceManual ARM DDI 0438C, 2011.

[3] ARM Ltd. ARM Generic Interrupt Controller Ar-chitecture version 2.0 ARM IHI 0048B, 2011.

[4] ARM Ltd. ARM Architecture Reference ManualARMv7-A DDI0406C.b, 2012.

[5] ARM Ltd. ARM Energy Probe, Mar. 2013.http://www.arm.com/products/tools/arm-energy-probe.php.

[6] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Har-ris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield.Xen and the Art of Virtualization. In Proceedingsof the 19th ACM Symposium on Operating SystemsPrinciples, pages 164–177, Dec. 2003.

[7] K. Barr, P. Bungale, S. Deasy, V. Gyuris, P. Hung,C. Newell, H. Tuch, and B. Zoppis. The VMwareMobile Virtualization Platform: is that a hypervi-sor in your pocket? SIGOPS Operating SystemsReview, 44:124–135, Dec. 2010.

[8] E. Bugnion, S. Devine, M. Rosenblum, J. Suger-man, and E. Y. Wang. Bringing Virtualization to the

18

x86 Architecture with the Original VMware Work-station. ACM Transactions on Computer Systems,30:12:1–12:51, Nov. 2012.

[9] C. Dall and J. Nieh. KVM for ARM. In Proceed-ings of the Ottawa Linux Symposium, July 2010.

[10] J.-H. Ding, C.-J. Lin, P.-H. Chang, C.-H. Tsang,W.-C. Hsu, and Y.-C. Chung. ARMvisor: SystemVirtualization for ARM. In Proceedings of the Ot-tawa Linux Symposium, June 2012.

[11] General Dynamics. OKL4 Microvisor, Mar. 2013.http://www.ok-labs.com/products/okl4-microvisor.

[12] Green Hills Software. INTEGRITY Secure Virtu-alization, Mar. 2013. http://www.ghs.com.

[13] J. Hwang, S. Suh, S. Heo, C. Park, J. Ryu, S. Park,and C. Kim. Xen on ARM: System Virtualizationusing Xen Hypervisor for ARM-based Secure Mo-bile Phones. In Proceedings of the 5th ConsumerCommunications and Newtork Conference, pages257–261, Jan. 2008.

[14] InSignal Co.,Ltd. ArndaleBoard.org, Mar. 2013.http://arndaleboard.org.

[15] Intel Corporation. Intel 64 and IA-32 Archi-tectures Software Developers Manual, 325462-044US, 2011.

[16] A. Kivity, Y. Kamay, D. Laor, U. Lublin, andA. Liguori. kvm: The Linux Virtual Machine Mon-itor. In Proceedings of the Ottawa Linux Sympo-sium, June 2007.

[17] Linux ARM Kernel Mailing List. A15 h/w virtual-ization support, Apr. 2011. http://archive.arm.linux.org.uk/lurker/message/20110412.204714.a36702d9.en.html.

[18] L. McVoy and C. Staelin. lmbench: Portable Toolsfor Performance Analysis. In Proceedings of the1996 USENIX Annual Technical Conference, pages23–23, Jan. 1996.

[19] I. Molnar. Hackbench, Feb. 2013.http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c.

[20] G. J. Popek and R. P. Goldberg. Formal Require-ments for Virtualizable Third Generation Architec-tures. Communications of the ACM, 17:412–421,July 1974.

[21] Red Bend Software. vLogix Mobile, Mar.2013. http://www.redbend.com/en/mobile-virtualization.

[22] R. Russell. virtio: Towards a De-Facto StandardFor Virtual I/O Devices. SIGOPS Operating Sys-tems Review, 42:95–103, July 2008.

[23] U. Steinberg and B. Kauer. NOVA: AMicrohypervisor-Based Secure Virtualization Ar-chitecture. In Proceedings of the 5th EuropeanConference on Computer Systems, pages 209–222,Apr. 2010.

[24] P. Varanasi and G. Heiser. Hardware-SupportedVirtualization on ARM. In Proceedings of the 2ndAsia-Pacific Workshop on Systems, pages 11:1–11:5, July 2011.

[25] C. A. Waldspurger. Memory Resource Manage-ment in VMware ESX Server. SIGOPS OperatingSystems Review, 36:181–194, Dec. 2002.

[26] Xen.org. Xen ARM, Mar. 2013. http://xen.org/products/xen_arm.html.

19