The VMware Mobile Virtualization Platform: is that ahypervisor in your pocket?

Ken Barr Prashanth Bungale Stephen Deasy Viktor Gyuris Perry HungCraig Newell Harvey Tuch Bruno Zoppis

{kbarr, prash, sdeasy, gyuris, perry, craign, htuch, bzoppis}@vmware.comVMware, Inc.

ABSTRACTThe virtualization of mobile devices such as smartphones,tablets, netbooks, and MIDs offers significant potential inaddressing the mobile manageability, security, cost, compli-ance, application development and deployment challengesthat exist in the enterprise today. Advances in mobile pro-cessor performance, memory and storage capacities have ledto the availability of many of the virtualization techniquesthat have previously been applied in the desktop and serverdomains. Leveraging these opportunities, VMware’s MobileVirtualization Platform (MVP) makes use of system virtu-alization to deliver an end-to-end solution for facilitatingemployee-owned mobile phones in the enterprise. In thispaper we describe the use case behind MVP, and providean overview of the hypervisor’s design and implementation.We present a novel system architecture for mobile virtual-ization and describe key aspects of both core and platformvirtualization on mobile devices.

1. INTRODUCTIONOver the past decade the mobile phone has evolved froma voice-centric device to a mobile personal computer. Wecan observe multiple progressive generations in the enter-prise adoption of such devices. There has been a shift frompure voice support to email and web browsing, with currenttrends towards the mimicking of desktop functionality. Onthe horizon we expect mobile devices to become true first-class enterprise endpoints with rich applications and coreenterprise connectivity.

As mobile computing advances, it brings some familiar chal-lenges to enterprises; application management and inter-operability, security and manageability, consumerization ofIT and persona management. Independent research firm,Forrester Research, Inc. found in 2008 that firms expecteda 3× increase in enterprise employee smartphone users by2013 [16]. This will lead to an acceleration of the rate atwhich these challenges become IT concerns. In addition, thediversity of usage and maturation of enterprise applicationson mobile platforms will limit the scalability of previous adhoc solutions, for example pushing services like email outsideof enterprise firewalls.

From an end-user perspective there is a healthy desire to con-solidate personal physical device ownership and the dilemmaof carrying a smartphone for work is no different. Ratherthan carrying two devices, allowing enterprise IT stewardshipof a personal device or even sacrificing the personal device for

Figure 1: Memory capacity, internal storage andclock frequency for a sample of smartphones [25].

some enterprise-provided device, we feel that virtualizationoffers an opportunity to provide the best solution by pre-serving isolation of environments without requiring a secondphysical device.

Co-existing virtual phones on a device represents a veryattractive alternative to existing solutions for these problems.The rapid pace of hardware advances in mobile devices overthe past several years, as shown in Figure 1, has led to aclass of devices with resources capable of supporting multiplevirtual phones where the virtualization overhead is small.

The VMware Mobile Virtualization Platform (MVP) is anend-to-end solution for enterprise management of employeeowned phones, encompassing a hypervisor for ARM-baseddevices, an enterprise virtual phone and remote device man-agement. In this paper we focus on the hypervisor layer andprovide the following contributions:

• A system architecture for a Type 2 mobile hypervisor,a novel design point for a mobile hypervisor with signif-icant device compatibility and deployment advantages.

• A lightweight paravirtualization technique for ARMv7cores aimed at minimizing the total system complexity.

124

• Device and platform virtualization approaches for stor-age, networking and telephony, the key devices in en-abling the performance, reliability and security of theMVP solution.

• An application of the MVP hypervisor to the virtualiza-tion of the Android [18] operating system. Android isthe fastest growing mobile operating system today [17]and will be used as a running example for both guestand host.

The following section provides some discussion on the designgoals that fall out of the above motivating use case. TheMVP system architecture and details on the virtualizationapproaches adopted for the core and devices are given inSection 3, the user interface of a virtualized Android-basedsystem is described in Section 4 and we conclude with relatedwork and MVP’s status. We limit the scope of discussionto smartphones but note that the emerging class of ARM-based tablet and netbook devices share with smartphonesmany hardware and software characteristics and could besubstituted in the rest of the paper.

2. DESIGN GOALSThe MVP hypervisor enables an enterprise provisioned andmanaged virtual phone to execute on a personal phone ownedby an employee of the organization, a Bring Your Own Device(BYOD) model. It provides a second completely encapsulatedsmartphone stack by virtualizing the system at the hardwarelevel and providing a virtual machine (VM) platform for theenterprise stack to execute on. We describe below the designgoals that informed the hypervisor’s technical architecture:

• Portability. Since the employee’s personal phone isthe physical platform, it is important to have the MVPhypervisor able to run on a wide variety of possiblemobile devices.

Figure 2 provides a block level diagram of an examplesmartphone. It is a simplified view, since some of thecomponents such as the GSM stack may have addi-tional processor cores and memory devices, but evenat this level of detail it’s evident that a smartphoneplatform consists of a significant number of diversecomponents. Unlike the desktop and server domains,most components are integrated on a phone’s System-on-Chip (SoC). A SoC is made up of a number of IPblocks, including the processor core. For each version ofthe standardized ARM architecture there are a numberof cores available from ARM and architectural licensees,each with implementation defined behaviors within thatallowed by the architecture specification. Beyond thecore there is a plethora of device components, eachwith its own set of interfaces and programming model.

To maximize the number of devices a mobile hypervisorexecutes on it should avoid undue dependency on themany and varied specifics of a phone’s hardware. Thisimplies a rethink of mobile hypervisor system structurewhich we describe in Section 3.

• Compatibility. An effective solution to enterprise ap-plication development and management requirements

Smartphone internals

System-on-Chip (SoC)

ARM core(e.g. Cortex A-8)

MMU

Interrupt controller

Timers

Cache

GPU

DMA engine

DSP

LCD controller

USB controller

SD card controller

UART

GPIO

Camera controller

Power management

Memory controller

SDRAM NAND Flash

GSM modem GPS

Antenna

Accelerometer/compass

WiFi

LCD/touchscreen

Bluetooth

Battery

Camera

Microphone/speaker

Figure 2: Smartphone components.

needs to present a single virtual platform as a targetfor enterprise applications. The role of the virtualiza-tion layer is to map this abstraction to the underlyingphone hardware and system software where the VM isdeployed.

Most applications are written to specific mobile mid-dleware and operating systems, for example Android,iOS or Symbian. It is common for them to have depen-dencies on specific versions of these environments. Bytaking an existing set of components as the softwarebase of the virtual platform, such as the Android li-braries, services and kernel, it’s possible to facilitate thereuse of existing enterprise applications in the virtualphone environment. In addition this allows the enter-prise to tap into the existing ecosystem and developerbase surrounding the platform. The hypervisor is con-cerned with providing a virtual hardware abstractioncapable of supporting efficiently the execution of thiscomplex and demanding virtual phone environment.

• Security. The most significant enterprise security con-cerns introduced by mobility relate to physical threats,for example phone theft or the removal of media con-taining confidential documents. The ability to dictateminimum password strengths, inactivity thresholds andlockout, storage and network encryption and remotewipe are necessary for a mobile platform to be consid-ered secure [22].

In addition, there are traditional software securitythreats that are common to desktop and mobile plat-forms. One of the main attractions of a smartphone is

125

the ability of the user to download and install appli-cations at will. Application store distribution modelspopularized by Apple’s iPhone App Store [4] have ledto an enormous number of applications available onthese markets. We are now facing the same threatsfrom Trojans and spyware that we have seen on PCs.

In a BYOD model, phone owners should not be limitedin the applications that they can download, while cor-porate IT needs the ability to protect corporate assets.We reconcile these opposing views by having the MVPhypervisor secure the execution and data in the enter-prise virtual phone stack from the open environmenton the rest of the phone.

Section 3.4 details the MVP security architecture, ad-dressing these physical and software threats with fea-tures including an encrypted virtual filesystem andvirtual private network (VPN) tunneling of corporateapplication network traffic.

• Low complexity. We strove to minimize the overallsystem complexity, including both that of the hypervi-sor and changes introduced in the guest to increase ourconfidence in the fidelity of the virtualization. Lowercomplexity leads to higher reliability, improved main-tainability and greater security when trusted compo-nents are also kept simple.

• Performance. The performance of an applicationin the virtual phone should be indistinguishable by ahuman user from the performance of that applicationrunning on the native, unvirtualized device. In addition,battery life should not be unreasonably affected by theexecution of the virtual phone. We wanted to go beyondthe limits of microbenchmarking and optimize for theend user experience. Section 3.5 describes some of thetechniques we used to measure performance and totarget optimization efforts.

• Manageability. A key requirement for an enterprisemobile solution is the ability for IT departments toremotely manage devices, or in our case the virtualphone. This requires supporting the provisioning, up-dating, wiping, locking and backup of virtual phonesover mobile networks.

• OEM time-to-market (TTM). Ideally a mobile vir-tualization solution should have minimal additionalengineering effort required by OEMs to support. Inthe next section we describe an architecture capable ofachieving this by supporting the secure distribution ofmost hypervisor components via standard applicationchannels. This differs from existing bare-metal hyper-visor approaches in which the entire mobile hypervisoris embedded on the device by the manufacturer priorto shipping and supports a more flexible approach tohypervisor distribution.

3. SYSTEM ARCHITECTURE

3.1 OverviewFigure 3 provides a block level overview of the MVP systemarchitecture. MVP is a Type 2, or hosted, hypervisor, provid-ing the ability to execute additional guest virtual machinesalongside a smartphone’s existing host operating system.

ARMv7-A based SoC

Host Linux 2.6.29+ Kernel

Existing host Android

applicationsand

middleware MVP VM support services

Virtual phone Android

applicationsand

middleware

MVP Virtual Machine Monitor

mvpkm

MVP user interface proxy

Guest Linux2.6.32 Kernel

Host world Guest/VMM world

pvtcp

mvpd

vmx(Startup, storage,

checkpointing)

Remote management

agent

Telephony support

vpnd

mksck

Figure 3: MVP system architecture.

Similar to the motivation given by Sugerman et al. [30] forthis organization in VMware’s desktop virtualization prod-uct Workstation, a hosted hypervisor provides a powerfulapproach in addressing the hardware diversity present in thenon-core aspects of mobile device SoCs. MVP leverages theexisting device drivers in the host operating system, map-ping guest virtual device operations to standard interfacesexported by the host kernel. Three other key motivationsled to this structure:

• The hosted model is a natural fit for BYOD. A personalphone with an existing operating system, applicationsand settings can be provisioned with an enterprise VMwithout perturbing the existing environment. Appli-cations in the host personal phone are not affected bythe presence of the enterprise VM and do not incur anyvirtualization overhead.

• The hypervisor need not interpose on every devicein the system. For example, hardware accelerated3D graphics necessary for games can be supported inthe home environment without any explicit hypervisorawareness if not required in the enterprise environment.

• A hosted hypervisor does not have to be integratedearly in the smartphone development cycle and mostcomponents can be provisioned over a mobile network.

MVP relies on the ability of certain trusted components toobtain elevated capabilities and execute in privileged modes.A minimal daemon mvpd executes as superuser on the hostand is responsible for granting further MVP-related processesnecessary capabilities. mvpd is the only piece that requiresplacement on the device by an OEM. Section 3.4 gives furtherdetails on the integrity checks that mvpd performs and thechain of trust that enables a verified execution environmentfor the VM.

mvpd is responsible for inserting an authenticated kernel mod-ule mvpkm that supports the transfer of control between thehost kernel and the MVP virtual machine monitor (VMM).When the enterprise VM is launched, mvpd loads the guestimage and VMM into memory and dedicates a thread to theexecution of the VMM. From the host operating system’s

126

point of view, this thread represents the time spent runningthe guest VM.

Once launched, the processor is time multiplexed betweenthe guest/VMM and host worlds — a world being the userand system context on the processor, including both userand privileged register file and coprocessor state. In thehost world, the existing host OS and applications continue toexecute as before. When the guest VM is to be scheduled, theproxy thread in mvpd calls into mvpkm which then orchestratesthe world switch. Control is transferred to the VMM which inturn passes control to the guest. The VMM retains ownershipof the exception vector table and page tables, allowing itto interpose as needed. The VMM returns control to thehost world on interrupts and when necessary to access hostservices, for example host memory allocation or to makesome system call on behalf of a virtualized device.

Core virtualization is performed by the VMM and takesplace entirely in the guest/VMM world — there is no needto world switch on the vast majority of traps related toinstruction set or address space virtualization. The VMMworks in concert with components on the host, such as thevmx process and pvtcp kernel module to present to the guesta set of virtual devices. The general model that we haveadopted is to introduce a paravirtualized guest driver for eachdevice and have the VMM intercept hypercalls for devicespecific behavior, forwarding requests as needed to the hostcomponents. Section 3.3 provides further details on how thisapplies to some of the key devices supported by MVP.

A custom message passing transport between the worlds,which we refer to later as mksck, is introduced to the hostkernel by mvpkm and appears to host processes via a standardsockets abstraction. Support is provided for establishingsecure shared mappings between host processes, the VMMand the guest.

3.2 Core virtualizationAlmost all smartphones today are based on ARM cores,with a recent trend towards ARMv7, the latest version ofthe ARM architecture available in SoCs. Earlier versionsof the ARM architecture featured virtually tagged and/orindexed data caches, which required careful management atthe application, operating system [34] and hypervisor levels.ARMv7 has instead a simpler physically addressed data cacheprogramming model. Given the MVP product time line, andthe availability of ARMv7 support in contemporary mobileoperating systems, it was advantageous to opt to supportonly ARMv7 since this led to a reduction in hypervisorcomplexity.

We describe below the approaches adopted in the MVPVMM to two aspects of core virtualization — instruction setarchitecture (ISA) and memory. While the lightweight par-avirtualization (LPV) techniques described below are used inthe production MVP VMM, the modular architecture in Sec-tion 3.1 enables other VMMs and virtualization techniquesto be employed. VMware has developed internally severalalternative VMM prototypes using binary translation, depriv-ileged user-mode virtualization [12,29] and hardware assistedvirtualization support. The last of these VMMs is based onARM’s upcoming Virtualization Extensions [10] and demon-

strates the potential for eliminating guest paravirtualizationwith respect to the core completely.

3.2.1 ARM Instruction Set Architecture (ISA)ARM is a 32-bit load/store architecture in which data pro-cessing instructions act only on registers. It provides a usermode for application execution and several privileged modesfor kernel execution and exception handling, referred to assystem mode, IRQ mode, etc. Low power variants of the ISAknown as Thumb and Thumb-2 feature a subset of the in-structions with a reduced instruction width and hence higherdensity. The architecture facilitates instruction set extensionwith coprocessors, used for floating-point instructions, single-instruction multiple-data (SIMD) processing instructions,etc.

Similar to other architectures such as x86, ARM has a numberof non-privileged instructions (i.e. instructions that do nottrap when executed from an unprivileged mode) that can po-tentially access or rely on privileged state. Such instructionsare termed sensitive instructions in ISA virtualization termi-nology. Because of the presence of such sensitive instructions,the ARM architecture is not classically virtualizable as perthe original definition proposed by Popek and Goldberg [26].

Two standard ways in which sensitive instructions can bedealt with are by using paravirtualization of the guest ker-nel source to replace sensitive instructions statically or byemploying binary translation to catch and handle sensitiveinstructions dynamically. We opted for a form of paravirtu-alization for two reasons:

• Due to the coupling of kernel source code availabilityand licensing in the mobile space, paravirtualizationposed no limitation on the guest kernels we could sup-port.

• We could reduce overall system complexity, opting forsimple guest kernel changes over the complexity of acomplete and optimized binary translator.

Our goal in virtualizing the ISA for the guest virtual machinewas to avoid any changes in the guest user-mode code, i.e. ap-plications and middleware, and to make minimal changes tothe hardware definition. This implied only those changes tothe guest kernel that were necessary to simplify the monitoras compared to a VMM based on dynamic binary transla-tion. The motivating reasons for minimizing changes beyondreduced complexity were the easing of guest porting to theMVP virtual platform and maintaining a stable interfacebetween the guest and VMM.

Towards this goal, we employ the following approach tovirtualize the ISA for the guest kernel:

• We execute the entire guest, both kernel and user code,in user-mode on the physical processor. We rely onprivileged instruction semantics dictated by the ARMISA to automatically trap and handle most privilegedinstructions encountered in the guest kernel, e.g. privi-leged coprocessor access instructions mcr and mrc. The

127

exception to this, where we replace such instructionswith hypercalls, is described in the next section.

• We paravirtualize all devices, so we do not need toprovide replacements for instructions that might accessregions of the address space responsible for I/O as theydo not exist, see Section 3.3.

• Finally, we rely on paravirtualization to handle the sen-sitive instructions. Most of the instructions in this classrelate either directly or indirectly to state contained inthe Current Processor Status Register (CPSR), whichhas many privileged bits of information such as thecurrent mode of execution, whether interrupts are en-abled or not, etc. However, it also has some innocuousbits of information such as the status flags, includingthe zero flag, the parity flag, etc. Depending on whichmode of execution the CPSR register is accessed from,the semantics governing which bits are overwritten andwhich bits remain unmodified are different.

More specifically, we employ a variant of paravirtualizationthat we call lightweight paravirtualization (LPV). Instead ofmaking changes to entire subsystems and data structures inthe guest kernel, we limit changes to only the architecturedependent branch, e.g. arch/arm in the Linux kernel sourcetree. Further, we limit changes to surface-level changesin leaf functions/macros that can be easily made with asimple search and replace. We replace sensitive instructionseither with hypercalls 1:1 to force a trap (and then emulatesemantics upon trap in the monitor), or with a block ofinstructions that emulate the sensitive instruction’s semanticsin-place, a more performant approach in many cases.

3.2.2 MemoryThere are two components in the memory subsystem thatrequire virtualization, the memory management unit (MMU)and the physical memory resources presented by the SDRAMon a mobile device.

MMU virtualization encompasses how the processor addresstranslation and memory protection hardware is virtualized.The ARMv7 MMU manages a 32-bit virtual address space,represented in a two-level hardware-walked tree page ta-ble. Page sizes range from 4KB–16MB and 8-bit ASIDs areprovided. In addition to the page-level read, write and no-execute permissions, a domain protection mechanism allowsfor page-level protection to be overridden at 1MB granular-ity [5] and modified orthogonally. The architecture supportsmultiple levels of TLBs, unified and split. As no dedicatedhardware virtualization support exists today, i.e. no nestedtranslation, we had the option of modifying the guest viaparavirtualization or providing a virtual MMU faithful tothe original architecture via trap-and-emulate. We opted forthe latter since paravirtualization of page table and memorymanagement involves making changes to extant guest datastructures and algorithms, with greater complexity than asimple search and replace of sensitive instructions. Opti-mizations presented by paravirtualized approaches to MMUvirtualization [7] can still be achieved through a set of op-tional hypercalls that are added to a guest kernel as needed.For example, loops of guest cache and TLB maintenanceCP15 instructions are replaced with a single hypercall.

Shadow page tables

Guest state

VMM state Page faults(VMM guest page tablewalker)

Optional hypercalls

CP15register access

traps

Guest page tables

Virtual CP15 registers

ARM processor state

TLB

Core CP15 registers

VMM CP15register access

Page faults(H/W page table

walker)

Figure 4: ARM MMU virtualization in MVP.

A MMU provides a virtual address space abstraction, map-ping virtual addresses to physical addresses. With virtualiza-tion we introduce an additional level of indirection, with theguest maintaining as before a page table with guest virtualto guest physical mappings and the VMM maintaining amapping from guest physical addresses to machine physicaladdresses. Shadow page tables [2], which cache the derivedguest virtual to machine physical mappings, are maintainedby the MVP VMM and are accessed by the hardware pagetable walker. The shadow page table contents are kept coher-ent with the guest page table and hypervisor physical pagemapping data structure by intercepting traps resulting fromguest page faults and TLB maintenance related Coprocessor15 (CP15) register accesses. Since the guest executes in usermode, any attempt to access these privileged registers resultsin a trap and shadow copies of these register locations aremaintained when they are stateful. Figure 4 illustrates therelationship between guest, VMM and processor state. Theabove traps are sufficient to faithfully virtualize the ARMMMU with shadow page tables. A number of further heuris-tics, caching and prefetch optimizations exist in the MVPVMM to improve performance and the end result is negligibleMMU virtualization overheads for many workloads.

Machine memory is managed by the host operating systemand allocated to the VMM via mvpkm. Since guest workloadsmay be demanding and require a large fraction of the devicememory resources, we overcommit memory between the guest

128

and host, employing a ballooning [33] approach to balancethe available free memory resources and page cache utiliza-tion. The balloon policy avoids the situation in which eitherthe guest or host trigger low memory application lifecyclebehavior while there remains free resources in the comple-mentary world. MVP also supports rapid checkpointing andrestore of VMs, providing an enhanced user experience viavirtual phone persistence and obviating the virtual phoneboot process.

3.3 Platform virtualizationBelow we provide an overview of how a selection of smart-phone components are virtualized in MVP. In the interestof brevity we focus on components where the smartphonehardware and software environment influenced the designchoices.

3.3.1 StorageBoth hypervisor components and VM images are stored onthe existing host filesystems. Smartphones typically havetwo types of storage devices — internal NAND Flash mem-ory and Secure Digital (SD) cards. The internal NAND isfixed and constrained in size due to cost and power consump-tion. On high-end smartphones this typically ranges from256–512MB, while SD card storage, also based on NANDtechnology, but connected via a SD bus and controller, pro-vides removable storage that can extend up to 32GB today.Secure Digital Extended Capacity (SDXC) cards will supportup to 2TB capacities in the future. SD card storage benefitsfrom the economies of semiconductor scaling and supply aftera smartphone has been shipped and purchased.

When the host system image, applications and data are con-sidered, the available internal NAND storage is a fractionof the device capacity. Hypervisor components install onthe internal NAND as with other host applications. SinceVM images are large, containing both a tuned system image,enterprise applications and data, they lend themselves to SDcard storage. The MVP storage virtualization componentsenable higher levels of performance, data integrity and secu-rity on the virtual device than is available on a standard SDcard filesystem.

Figure 5 shows the various layers in the guest and hoststorage stacks. A file read or write by a guest applicationwill require data to be transferred between guest applicationmemory and the physical SD card media. Take for examplea write operation on some file. Once the data has beentransferred to the guest kernel and written out from its pagecache, a virtual I/O operation is started by a hypercall from aparavirtualized device driver. The driver provides a referenceto the monitor which, in turn, provides a shared mappingto a storage thread in the vmx component on the host (fromFigure 3). The block is written to the VM image file usinga write operation on the host, at which point the host I/Ostack is responsible for transferring it to the physical card.

FAT-based formats are standard for SD cards and the flashtranslation layer (FTL) on many cards is optimized for largesequential file transfers — SD cards are primarily used forphotos, music and video storage and document transfers. Weprovide enterprise-level performance, reliability and securitywith:

VFS

libc

GuestApplication

ext3

BlockLayer

NANDLayer

FTL

NAND Chip Driver

NANDCHIP

PV BlockDriver

VFS

Host libc

ext3

VMX(Storage thread,

guest image format processing, encryption)

BlockLayer

NAND FlashFilesystem

VFAT

MMC/SDLayer

MMC/SDDriver

microSDCard

Mksck +Shared Mem

PhysicalHardware

VirtualHardware

Guest LinuxKernel

Host LinuxKernel

Guest "unix"user space

Host "unix"user space

FTL

NAND

Figure 5: MVP storage architecture.

• A high-performance log structured VM image formatmatching the I/O mixture from the guest, containing asignificant amount of small non-sequential operations,to the I/O characteristics of the SD card. Unlike theircousins found in solid state disks (SSDs), the low costFTLs in consumer grade Flash storage devices dictatea significant penalty for non-sequential writes [3, 9]. Itis not uncommon to observe two orders of magnitude ormore difference between sequential and non-sequentialwrite performance for small block sizes.

• A journaling guest filesystem. Barriers issued by thejournaling EXT3 guest filesystem are propagated to thevmx storage thread, allowing it to take steps necessaryto provide the media coherency guarantees expectedby the guest. MVP bypasses the host page cache andperforms direct I/O, maintaining its own buffers thatare flushed in response to barrier requests. This wasfound to be a solution with lower maximum latencythan syncing page cache contents on barriers.

• Block level encryption is used in the guest image formatto mitigate threats posed by SD card removal, phonetheft and malicious applications with access to the SDcard.

• A mostly zero copy stack. With the exception of theencryption/decryption steps, there is no need for theCPU to directly process any guest block data betweenthe guest kernel buffers and SD card controller DMAengine.

The checkpoint images also reside on SD card storage andhave the same log structured and encrypted storage formatas the guest images.

129

Host kernel Guest kernel

SocketAPIs

Guest user mode applicationsHost user mode applications

Mksck / Shared memory

SocketAPIs

PV client module

Offload engine

Host network stack

Host

NIC

driver

Figure 6: MVP networking architecture.

3.3.2 NetworkNetworking capabilities are provided by a high-level paravir-tualization framework called paravirtualized TCP (PVTCP).PVTCP is distinguished from traditional full virtualiza-tion and paravirtualization techniques by operating at thesocket/system call level as opposed to the device interfacelevel. Using this approach allows for benefits such as lowervirtualization overhead and more flexible deployment options.

The architecture of PVTCP is broadly split into two parts,a PV client module that runs in the guest VM kernel con-text, and an offload engine that runs in the host kernelcontext, implemented in the pvtcp module in Figure 3. Inthe PV client, all network services requested by guest user-mode applications are intercepted at runtime just beforethe transport level. These requests are then proxied off tothe offload engine via the mksck high-speed shared-memorycommunication channel.

For example, when a guest VM attempts to perform a socketsystem call to open a socket, a request is made directly tothe offload engine. The offload engine performs a socket

call on behalf of the PV client, creating a proxy socketfor the guest. The net effect is that a guest applicationattempting to execute protocol-level services will delegateprotocol-processing and driver-level processing to the offloadengine.

By interposing in the guest directly at the protocol levelinstead of at the device level, we avoid the necessity oftraversing two network stacks (one for the guest and one forthe host) as well as the complexity of software emulation of ahardware network interface. In fact, because all socket callsare delegated to the offload engine, the guest VM does notneed to have a virtual NIC at all. This architecture allowsfor better CPU utilization and lower virtualization overhead.In internal testing, we have achieved throughput-parity withthe host on many workloads and significantly lower CPU-utilization as opposed to a traditional fully emulated NICapproach.

All PVTCP traffic is tunneled through the vpnd componentin the host as per the security model described in Section3.4.

3.3.3 TelephonyIt is impossible to discuss a smartphone environment withoutreferencing the telephony components involved. For MVP,our vision is to provide isolation of personal and work en-vironments, supporting two distinct phone numbers. MVPsupports a number of approaches to realizing multiple phonenumbers on a single device, with the ability to provide a spe-cific option dependent on some subset of the device manufac-turer, Subscriber Identity Module (SIM) card manufacturer,carrier network as well as enterprise network connectivityand integrated solutions. We review some of the possibleapproaches below and note that the actual product will sup-port one or more of these based on market requirements andcapabilities.

• GSM. Within the GSM space there are number ofviable options. In some geographies devices capable ofsupporting two SIM cards are sold, although these aregenerally not smartphones in nature. Alternatively, asingle SIM card is capable of supporting multiple lineswhich makes the form factor more familiar and inte-grated with lower device design changes. One approachis to provide multiple International Mobile SubscriberIdentities (IMSIs) on a single SIM card, which allowsthe mobile device to appear to the network as multipledevices from a capability and billing perspective. Thesesolutions are also sold and supported in diverse geogra-phies. A separate approach is the alternate line service(ALS) provision in an extension to the GSM specifica-tion [1], which has been activated in some networks,although this appears to be limited in deployment andin some cases decreasing.

• Network. From a network perspective it is possibleto offer a call-forwarding solution where numbers ter-minate at a PBX (e.g. an Asterisk server [6]) and arethen forwarded via a third invisible line to the devicewith some meta-data indicating which of the originalpersonal or work numbers were the target of the in-coming call. Outgoing calls can be similarly tagged toindicate the originating number. Examples of this typeof solution are common, e.g. Line 2 [23].

• Voice-over-IP (VoIP). VoIP continues to be an in-teresting and somewhat controversial addition to thetelephony options available on smartphones. As carri-ers begin to embrace and offer VoIP-based services, theevolution to 4G/LTE will further enhance the abilityof networks to support this as a viable option. Withinthe MVP domain, there are two interesting approaches;firstly connecting to public domain providers like Skypeor Fring. The second option which is particularly in-teresting in the enterprise space is the integration ofUnified Communications (UC) [13] technologies andtheir continued penetration in the corporate data cen-ter. By merging UC and MVP, enterprises have acompelling end user experience while retaining controlover policies, auditing, adding IT-controlled capabilitiesetc. The end-user can benefit with features like inte-grated dialing, easy access to instant messaging (IM)and presence tools as well as access to corporate assetslike address books and accurate person availability.

130

3.4 SecurityBelow we summarize the key security threats and goalsexpressed by enterprises, mobile security researchers andOEMs, followed by details of the MVP approach to securingthe platform.

Enterprise security goals include protecting sensitive informa-tion on the mobile device, data exchanges between the deviceand the corporate intranet and access to data or serviceswithin the corporate intranet. As discussed in Section 2,an additional goal is to counter the physical threat modelposed by devices that may be lost or stolen. To achievethese goals today requires that corporate IT policy placeinconvenient restrictions on the installation of applications,password configurations and device lockout.

A significant threat concerning the security community [8,14, 15] and OEMs is posed by the untrusted downloadableapplications that may exist in the host environment. Suchapplications may have information flows beyond those an-ticipated by the user and/or may be actively malicious, e.g.Trojans, spyware and other malware. On the Android plat-form, the end user is prompted when an application is in-stalled to confirm the permissions granted to access devicessuch as the GPS, SD card and various OS services. It iscommon for applications to request permissions that are notobviously connected with the application’s function and endusers may not have the required technical understanding toeffectively arbitrate, e.g. it may be difficult to understandwhy a game requires access to the address book, networkand GPS location, and whether this request is reasonable.This confusion can lead to Trojans bypassing the manualpermission checks. According to a recent report [31], 20%of applications in the Android market request permissionsto access private or sensitive information and 5% have theability to place a call with no user check. Another illustrativeexample of a malicious host application is the introductionof a proxy application linking an adversarial external hostand a corporate intranet [11].

MVP secures corporate resources accessible by employeeowned devices by creating two separate environments. Thehost environment is open and the device owner can continueto use it without restriction. The enterprise environment ismanaged by corporate IT, which decides exactly what kindof application can be installed in this environment and whichpolicies must be applied. In addition:

• Irrespective of the permissions given to applicationsin the personal environment, they cannot cross thevirtualization barrier, which prohibits host applicationsgaining access to the network, address book, SMS,phone dialing and other data/resources in the enterpriseVM.

• All network traffic from the enterprise VPN is tunneledthrough a VPN to a corporate VPN gateway via adaemon vpnd, shown in Figure 7. Applications in theenterprise VM transparently benefit from the VPNaccess. The host kernel tun device and pvtcp share anamespace isolated from the normal network namespaceon the host (a feature available in the Linux kernelsince version 2.6.24). This prevents untrusted host

Host Linux 2.6.29+ Kernel

Existing host Android

applicationsand

middleware

Virtual phone Android

applicationsand

middleware

MVP Virtual Machine Monitor

Guest Linux2.6.32 Kernel

Host world Guest/VMM world

vmx(Startup, storage,

checkpointing) vpnd

SD card filesystem

Encrypted guest images,

checkpoint files

Internet

Corporate VPN gateway

Network path

Storage path

Figure 7: MVP storage and networking paths.

Android applications from gaining VPN access, rulingout the possibility of a malicious application bridgingan external host to the VPN.

• There are only very limited access controls on the SDcard in Android. All files are accessible from any ap-plication with SD card permission. The card is alsoremovable by anyone with physical access to the device.As described in Section 3.3.1, the MVP solution imple-ments a block level encryption of the guest’s virtual filesystem and checkpoint images to mitigate these risks.Thus, all applications in the enterprise VM benefit fromstorage encryption and are protected from prying hostapplications and SD card removal. This is illustratedin Figure 7.

• A password is required to start the enterprise VM anddepending on corporate policy, to switch from personalto work environment, either each time, or when thework VM has been inactive for some period of time.

From an architecture point of view, the MVP solution hasbeen split into different components, implementing a privilegeseparation pattern.

When performing privileged operations on the host, theroot of trust is always the mvpd process. It is installed ina read-only filesystem by the OEMs, effectively enablingvirtualization on the phone. In conjunction with Android’ssigning and permission model [28], mvpd authenticates theother trusted and, transitively, the untrusted components.The trusted components are kept simple and small. Theother virtualization components on the host, which are used

131

SoC secure boot

Host system image

mvpd Guest VM image

VMM

mvpkm

pvtcp

Other MVP components

vmx

vpnd

Figure 8: MVP chain of trust. Arrows indicate thatthe source authenticates the destination node.

to implement device virtualization or platform management,run with user privileges. For example, the applications re-sponsible for the host side of display or audio virtualizationdon’t need to be run as the superuser. In the guest/VMMworld, only the VMM component executes in a privilegedmode.

At boot time, a chain of trust is created, preventing imagetampering, as shown in Figure 8. The phone boot codeauthenticates the phone system image. mvpd is authenticatedas part of this system image. Before starting the workVM, vmx authenticates the other virtualization componentsinstalled on the platform, as well as the VM system orcheckpoint image.

3.5 Performance methodologyMVP’s BYOD use case focused our performance efforts onend-user experience with no hard real-time constraints. Ourgoal was that the performance of an application in the virtualphone should be indistinguishable by a human user fromthe performance of that application running on the native,unvirtualized device. Our approach to track progress towardthis goal involves continuous benchmarking with workloadsthat include, among others:

• Web browsing: a scripted test that measures the timeto start a browser, load a page with several images,and scroll down that page.

• Bootup: starting an Android-based virtual machine tostress storage, the Android JVM and process creation.

• Multimedia kernels: predominantly unprivileged,easy-to-virtualize computation that serves mostly as apractical sanity check that virtualization imposes littleoverhead with such programs.

• OS and I/O microbenchmarks: used by subsystemdevelopers to isolate and improve a particular aspectof the system.

While we strove to avoid virtualization overhead in all work-loads, it was a guiding philosophy that we would tolerateeven substantial overheads if they could be shown to have noeffect on an interactive user’s experience and battery lifetime.

To a first order, battery life is proportional to execution timeand the active cycles of the display and modem hardware.Hence, a system with low virtualization performance over-head can be expected to have low energy overhead, providingthat host device management is unaffected by the presence

of a hypervisor. However, special care must be taken to allowthe phone to take advantage of any low-power states providedby its hardware. To this end, certain polling-based subsys-tems have been replaced with on-demand implementations.We study power consumption of an idle MVP system usinga battery simulator and proxies such as host OS-reportedscheduler wake-ups per second.

We describe below the performance measurement and anal-ysis methodology for the VMM and host components ofthe system, together with an overview of how we manageperformance regressions.

3.5.1 VMMOur primary tool for diagnosing and correcting performanceproblems is an internal tool called kstats. This tool wasoriginally developed for the VMware x86 VMM to providelightweight, flat execution profiles of VMM services. Callchains involving the VMM do not employ a traditional stack:while the VMM code makes function calls to VMM routines,the VMM is preemptible and may be entered or exited viaexceptions and world switches. This program flow makesfunction-level profiling difficult. Furthermore, a function(like HandleInterrupt) may be called on behalf of variousservices (emulating a device, servicing a page fault, etc...),and the key to improving performance may have more to dowith reducing the number of times such a service is requiredthan speeding up a leaf function it invokes. The trade-offfor achieving such service-oriented profiling is that kstatsrequires developers to annotate the VMM code. As theannotations are simple, the manual effort is an acceptabletradeoff given the importance of VMM performance.

An example of a kstats profile is shown in Figure 9. Theprofile is an example of a workload dominated by easy-to-virtualize unprivileged code with little I/O. The name ofthe service appears at the left. The next column shows thepercentage of total execution time attributed to each service.The Average Cycles column is merely the quotient of TotalCycles divided by Counts. The Overhead column is an esti-mate of virtualization overhead computed with the formula:Total Cycles/(de.User + de.Priv). We assume that theworkload running without virtualization would require ap-proximately the number of cycles spent in direct execution.Thus, the ratio of a VMM service’s cycles to the estimateof native execution cycles represents the overhead caused bythat service. Note that the overheads need not sum to 1.

Looking at both the Overhead and Percent column givesdevelopers two views of the same profile: a view sorted byOverhead shows the most expensive services while one sortedby Percent puts these services in the context of the entireexecution. One might be drawn to optimize a service with alarge overhead, but this may be a low-priority task if thatservice does not contribute much to the overall slowdown.Alternatively, a service may not currently be the largestcomponent by percent of total time, but if it has a highoverhead, its effect may become more pronounced as theimpact of other services is reduced.

In addition to a text-based cumulative profile, it can be usefulto visualize the profile of a workload over time. An interactive,graphical tool has been developed for this purpose, and a

132

----------------------------------------------------------------------------------------------------Name % Total Counts Average Ovrhd

Cycles Cycles----------------------------------------------------------------------------------------------------de.User (Direct Execution of guest user mode code) 91.759% 8809370286 2076 4243434 -host (Host time: vmx and non-MVP processes) 3.749% 359887170 5474 65744 0.04de.Priv (Direct Execution of guest privileged code) 1.438% 138087476 2077 66484 -irq (Handling interrupt) 0.465% 44663350 2266 19710 0.00swi.PrivToUser (Guest mode switch) 0.412% 39565001 2076 19058 0.00...

Figure 9: The first few lines of a kstats profile of an unprivileged multimedia workload

Figure 10: Time series view of a kstats profile. Thelegend has been truncated for brevity.

screen shot is shown in Figure 10. Such a view allows theuser to restrict analysis to the steady state behavior or aparticular program phase. The kstats tool suite includesother visualizations that enable MVP engineers to focusoptimizations.

3.5.2 Hostkstats only provides fine grained information about timespent in the VMM. The rest of execution consists of directexecution of the guest or execution in the host world. Hostworld time appears in a kstats profile, but the asynchronousnature of the host MVP components and batching of tasksthat they perform for the VMM make it difficult to interpretthe information; it is included mostly as a placeholder. To un-derstand how time is spent on the host, we use oprofile [24].oprofile compatibility is achieved via careful transitions ateach world switch. When execution transitions from the hostto the VMM, the MVP world switch code saves the values ofthe performance counters and their status registers. Prior toresuming execution on the host, the values are restored, giv-ing the appearance that no time has elapsed and no counterhas overflowed due to VMM or guest execution.

3.5.3 ARM Performance Monitoring HardwareWe also take advantage of the multiple hardware performancecounters available on our platform. These can be useful forprecise timing of a code routine (e.g., by performing thatroutine in a tight loop surrounded by reads of the cyclecounter). We have measured cache and TLB miss events todetermine whether code that shows up as a kstats hotspotwould benefit from improving the data layout or reducingthe code footprint.

3.5.4 RegressionsIn addition to low-level profiling to reveal and correct par-ticular problems, the MVP team runs continuous tests ofperformance to track progress and catch regressions. Theresults of this testing are recorded in a database and plottedon a website that provides links to the associated log files,profiles and changesets. When the continuous integrationsystem or an interactive user detects a result that is signif-icantly different from the baseline in the database, we canconsult the associated artifacts to determine if the change isdue to inherent variability or the corresponding code change.

4. VIRTUALIZING ANDROID — MVP IN

ACTIONFigure 11 provides a flavor of the user experience when inter-acting with the dual personal/enterprise phone environmentsprovided by MVP. The screenshots were obtained from aNexus One smartphone with both Android 2.1 guest and host.Even though the underlying mobile OS in both environmentsis similar in this example, there is distinct isolation provided,both at the system level as discussed in earlier sections andthrough the user interface (UI).

The home screen of the personal phone is shown in Figure11(a). Various outside-of-work applications are installed,such as Facebook, GMail and a podcast aggregator. Also, anicon exists in the top right corner for the enterprise VM. TheVMmay be running in the background or checkpointed. Oncethis icon is tapped, the UI switches to that of the enterpriseVM, as seen in Figure 11(b). Different, IT provisioned,applications are installed and the Android instance has analternative theme. The browser in Figure 11(c) connects viathe corporate VPN to the network. By selecting the “Home”icon in Figure 11(d), the user can switch control back to thepersonal phone environment.

Further demonstrations of MVP are available at:http://www.vmware.com/mobile.

5. RELATED WORKIn this paper we describe the architecture of a hosted hy-pervisor for mobile devices based on the ARM architecture,aimed at supporting an enterprise BYOD model. Similaritiesexist in the system structure with VMware’s Workstationhosted architecture for x86 PCs, described by Sugerman etal. [30], however there are significant differences in the hostworld componetization, influenced by the mobile networkprovisioning model for the hypervisor.

133

(a) Personal phone home. (b) Enterprise VM home. (c) Browsing inside VM. (d) Return to personal phone

Figure 11: MVP personal/enterprise phone screenshots.

Early commercial use cases for mobile virtualization weretypically oriented around the concerns of silicon vendors, forexample consolidating application and baseband processorsor providing GPL isolation [35]. Type 1, or bare-metal,hypervisor architectures [20] were common, offering real-timeadvantages [32]. The BYOD enterprise use case we addresswith MVP invokes a different set of requirements, describedin Section 2, making an alternative approach to hypervisorstructure and provisioning compelling.

To the best of our knowledge, all previous approaches tosystem virtualization on the ARMv4–7 architectures haveentailed some form of core paravirtualization, for exampleXen on ARM [21]. MVP employs a distinct shallow par-avirtualization approach described in Section 3.2, requiringonly the identification and replacement of sensitive instruc-tions. At the application level, Sehr et al. [27] describetechniques for sandboxing on ARM via rule constraints onbinaries and static validation, allowing for the co-existenceof trusted and untrusted components in the same addressspace. Application performance measurement by dynamicbinary translation on ARM is described by Hazelwood andKlauser [19]. VMware has also developed a dynamic binarytranslation based research VMM for the ARM ISA that elimi-nates the need for any paravirtualization. The close couplingof licensing and source code availability in the mobile do-main allows for an overall system complexity reduction bylightweight paravirtualization.

6. SUMMARYIn this paper we’ve presented the underlying motivation forsmartphone virtualization, the BYOD use case, and provideda tour of the MVP hypervisor’s design goals and systemstructure, including the security model and performancemethodology employed. We believe that the technical archi-tecture underpinning the MVP hypervisor provides a robust,secure and performant approach to provisioning a device

with both personal and enterprise managed environments,while providing a portable solution to OEMs with low time-to-market.

With MVP we feel that we can adequately address the grow-ing market need to support employee-owned smartphonesin the modern enterprise. Finding the right balance of IT-driven control, end-user experience, performance, securityand scalability is a constant struggle in this environment andwith MVP we can meet these challenges across all compo-nents of the solution. We look forward to working with ourpartners across the various industries to bring the product tomarket and are excited by the possibilities it provides hereand in adjacent and orthogonal markets.

Acknowledgments. In addition to the paper authors, thereare many MVP team members who have contributed pro-foundly to the design and implementation of the hypervisorand our overall product solution. We are indebted to ScottDevine, Larry Rudolph and Julia Austin for providing theconception and founding vision for MVP, together with thehosted MVP structure.

7. REFERENCES[1] 3GPP, Orange PCS Ltd, Mercury One-2-One. Common

PCN Handset Specification, 2000. Version 4.2.

[2] Adams, K., and Agesen, O. A comparison of software andhardware techniques for x86 virtualization. In ASPLOS-XII:Proceedings of the 12th international conference onArchitectural support for programming languages andoperating systems (New York, NY, USA, 2006), ACM,pp. 2–13.

[3] Agrawal, N., Prabhakaran, V., Wobber, T., Davis,J. D., Manasse, M., and Panigrahy, R. Design tradeoffsfor SSD performance. In USENIX 2008 Annual TechnicalConference on Annual Technical Conference (Berkeley, CA,USA, 2008), USENIX Association, pp. 57–70.

134

[4] Apple Inc. Apple’s App Store downloads top three billion.http://www.apple.com/pr/library/2010/01/05appstore.html,Jan. 2010.

[5] ARM Limited. ARM Architecture Reference Manual:ARMv7-A and ARMv7-R edition, 2007. ARM DDI 0406A.

[6] Asterisk — the open source telephony projects.http://www.asterisk.org.

[7] Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris,T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A.Xen and the art of virtualization. In SOSP ’03: Proceedingsof the nineteenth ACM symposium on Operating systemsprinciples (New York, NY, USA, 2003), ACM, pp. 164–177.

[8] Bickford, J., O’Hare, R., Baliga, A., Ganapathy, V.,and Iftode, L. Rootkits on smart phones: Attacks,implications and opportunities. In HotMobile’10:Proceedings of the 11th Workshop on Mobile ComputingSystems and Applications (Annapolis, Maryland, USA,February 2010), ACM Press, New York, NY, USA,pp. 49–54. http://doi.acm.org/10.1145/1734583.1734596.

[9] Birrell, A., Isard, M., Thacker, C., and Wobber, T. Adesign for high-performance flash disks. SIGOPS Oper. Syst.Rev. 41, 2 (2007), 88–93.

[10] Brash, D. Extensions to the ARMv7-A architecture. In HotChips 22 (Stanford University, California, Aug. 2010).

[11] D’Aguanno, J. Blackjacking — 0wning the enterprise viathe Blackberry. In Defcon 14 (Las Vegas, NV, August 2006).

[12] Dike, J. A user-mode port of the Linux kernel. In ALS’00:Proceedings of the 4th annual Linux Showcase & Conference(Atlanta, Georgia, 2000), USENIX Association.

[13] Elliot, B., and Blood, S. Magic quadrant for UnifiedCommunications. Gartner RAS Core Research Note,G00201349 (2010).

[14] Enck, W., Gilbert, P., gon Chun, B., Cox, L. P., Jung,J., McDaniel, P., and Sheth, A. N. TaintDroid: Aninformation-flow tracking system for realtime privacymonitoring on smartphones. In USENIX Symposium onOperating Systems Design and Implementation (OSDI) (Oct.2010).

[15] Enck, W., Ongtang, M., and McDaniel, P. Onlightweight mobile phone application certification. In CCS’09: Proceedings of the 16th ACM conference on Computerand communications security (New York, NY, USA, 2009),ACM, pp. 235–245.

[16] Forrester Research, Inc. Collaboration needs will fuel asmartphone surge, Jan. 2010.

[17] Gartner, Inc. Gartner says Android to become no. 2worldwide mobile operating system in 2010 and challengeSymbian for no. 1 position by 2014.http://www.gartner.com/it/page.jsp?id=1434613, Sept.2010.

[18] Google Inc. Android dev guide: What is android?http://developer.android.com/guide/basics/what-is-android.html.

[19] Hazelwood, K., and Klauser, A. A dynamic binaryinstrumentation engine for the ARM architecture. In CASES’06: Proceedings of the 2006 international conference onCompilers, architecture and synthesis for embedded systems(New York, NY, USA, 2006), ACM, pp. 261–270.

[20] Heiser, G., and Leslie, B. The OKL4 Microvisor:Convergence point of microkernels and hypervisors. InProceedings of the 1st Asia-Pacific Workshop on Systems(New Delhi, India, Aug 2010).

[21] Hwang, J., Suh, S., Heo, S., Park, C., Ryu, J., Park, S.,and Kim, C. Xen on ARM: System virtualization using Xenhypervisor for ARM-based secure mobile phones. InProceedings of the 5th Annual IEEE ConsumerCommunications and Networking Conference (Las Vegas,NV, Jan. 2008), pp. 257–261.

[22] Kaneshige, T. 7 steps to stronger enterprise iPhone security.CIO Magazine (Aug. 2010).

[23] Line 2. http://www.line2.com.

[24] OProfile. http://oprofile.sourceforge.net.[25] PDAdb.net. http://pdadb.net.

[26] Popek, G. J., and Goldberg, R. P. Formal requirementsfor virtualizable third generation architectures. In SOSP ’73:Proceedings of the fourth ACM symposium on Operatingsystem principles (New York, NY, USA, October 1973),vol. 7, ACM Press.

[27] Sehr, D., Muth, R., Biffle, C. L., Khimenko, V., Pasko,E., Yee, B., Schimpf, K., and Chen, B. Adapting softwarefault isolation to contemporary CPU architectures. InProceedings of the 19th USENIX Security Symposium(2010), pp. 1–11.

[28] Shabtai, A., Fledel, Y., Kanonov, U., Elovici, Y.,Dolev, S., and Glezer, C. Google Android: Acomprehensive security assessment. IEEE Security andPrivacy 8 (2010), 35–44.

[29] Steinberg, U. Fiasco μ-kernel user-mode port. TU Dresden,Dec. 2002.

[30] Sugerman, J., Venkitachalam, G., and Lim, B.-H.Virtualizing I/O devices on VMware Workstation’s hostedvirtual machine monitor. In Proceedings of the GeneralTrack: 2002 USENIX Annual Technical Conference(Berkeley, CA, USA, 2001), USENIX Association, pp. 1–14.

[31] Vennon, T., and Stroop, D. Threat analysis of theAndroid Market. Tech. rep., SMboile Systems, 2010.

[32] Mobile handset design: Realizing flexible, low cost, highersecurity, device management using OpenOS and real-timevirtualizationTM. Tech. Rep. TR-06-102.1, VirtualLogix Inc.,2006.

[33] Waldspurger, C. Memory resource management inVMware ESX Server. In Proceedings of the Fifth Symposiumon Operating Systems Design and Implementation(December 2002).

[34] Wiggins, A., Tuch, H., Uhlig, V., and Heiser, G.Implementation of fast address-space switching and TLBsharing on the StrongARM processor. In Proceedings of the8th Asia-Pacific Computer Systems Architecture Conference(Aizu-Wakamatsu City, Japan, Sep 2003), Springer Verlag.

[35] Zoppis, B. Using a hypervisor to reconcile GPL andproprietary embedded code. LinuxDevices. com (Aug. 2007).

135