dealing with hardware heterogeneity using embeddedxen, a virtualization framework tailored to arm...
DESCRIPTION
EmbeddedXEN is a particularly efficient virtualization framework tailored to ARM-based core embedded systems. While security and OS isolation are key features of conventional virtualizuation frameworks, the main concerns for EmbeddedXEN are device heterogeneity and realtime aspects, which are particularly important in the embedded world. EmbeddedXEN mainly relies on the original XEN architecture but with major differences in the way guest OS are handled: the hypervisor has been simplified, and only two guest OS (dom0 and domU) can run simultaneously; while dom0 is used to manage the native OS with drivers (original and backend splitted drivers), a paravirtualized OS (domU) can be cross-compiled on a different ARM device, and user applications can run seamlessly on the (virtualized) host device. Another important difference is that no user space tools are required to manage the VMs; the framework produces a compact single binary image containing both dom0 and domU guests, which can be easily deployed. The Xenbus architecture has been adapted to that context. EmbeddedXEN therefore allows the porting of an OS and its applications from an ARM embedded device to last generation ARM hardware, such as HTC Smartphone for example.TRANSCRIPT
Dealing with Hardware Heterogeneity Using a Virtualization Framework Tailored to ARM Based Embedded Systems
Prof. Daniel Rossier, PhD
HEIG-VDInstitut REDS, Reconfigurable & Embedded Digital Systemsrte Cheseaux 1, 1400 Yverdon-les-Bainshttp://www.reds.ch/
REDS@HEIG-VD
Outline• Background
• Overview of EmbeddedXEN
• Protection & Memory Isolation
• Domain Interactions
• Device Heterogeneity
• Conclusions & Future Work
2
REDS@HEIG-VD
Background
• HEIG-VD • University of Applied Sciences in Yverdon, Switzerland (CH)
• Reconfigurable Embedded Digital System Institute
Hardware Design, FPGA
Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc.
• Applied Research & Development• ARM based Microcontrolers (v4, v5, v6, v7)
• Low-level interactions between computing cores (CPU, DSP, FPGA, etc.)
• Towards Cortex-A15 HVM
3
REDS@HEIG-VD4
Background• Embedded Virtualization on ARM
• XEN: free & easy access to a stable & evolving hypervisor source code
• Early port of XEN on ARM in 2007 in the context of a Diploma Project
• Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization framework
• Re-use of Linux file organization & build system (Makefiles, scripts, …)
• Focus on realtime aspects (Linux/Xenomai as RTOS)
• Different sources of inspiration• "Fast Secure Virtualization for the ARM Platform", Daniel
Ferstay, Master Thesis, The University of British Columbia, 2006
• XEN ARM port, George G. Davis, MontaVista, 2007
• Secure Xen on ARM Project, Sang-bum SUH, Samsung, 2007
REDS@HEIG-VD
Background
• Focus on heterogeneity of embedded devices• Idea to re-use OS & applications from old devices to recent
devices
• Time-to-market migration to new hardware generation
• Dealing with various cross-compiled binaries (ARM v5-v7)
• Dealing with different peripherals
• Less emphasis on security aspects
• Publicly available• https://sourceforge.net/projects/embeddedxen
5
REDS@HEIG-VD6
Background• Hardware constraints
• Low latency, reactivity (response time)
• ARM cores do not support virtualization mechanisms
not easy to deal with various levels of execution modes
• Para-virtualization remains attractive• About 30 files to be (slightly) adapted
• Low execution overhead
• Efficient processing of downcalls/upcalls with support of domain interactions
Physical interrupts are quickly processed in dom0.
REDS@HEIG-VD7
Overview of EmbeddedXEN
• Limited use of hypercalls
• ARM execution modes (USR/SVC) – pseudo-user mode with double stack handling / downcalls & upcalls are simple jumps to specific (callback) addresses
• No subtle use of domain access control (ARM DACR) to protect memory
• Memory sharing facilitated between primary and secondary OS
Domain Creation and
Setup
Secondary Guest OS known as DomU
VCPU sched_flipScheduler
Migration
Manager
Hardware
Primary Guest OS known as Dom0
EmbeddedXEN Hypervisor Hypercalls
Handling
Xen-guest(pv)
Xen-guest(pv)
PrioritizedUpcalls
Handling
- Full privilege- DomU original
drivers- Frontend drivers
- Full privilege- Dom0 original drivers- Backend drivers- Native drivers
REDS@HEIG-VD
Overview of EmbeddedXEN• Linux-like source tree and build system (Makefiles)
8
embeddedxen
hypervisor-4.0.2
linux-2.6.32-dom0
arch/arm
linux-2.6.26-domU
xen-guest
xen-guest
tools
xen
arch/arm
environnement.conf
include
xen-guest
include
xen-guest
xen-guest
mach-mx35
arch
arm
mach-msm
mach-msm
xenbus
core includ
earch
arm
mach-mx35
mx35_fab4.c
kernel
mm
console
include
xen-guest
Hypervisor
Commonpart
Primary OS
SecondaryOS
Config & build system
REDS@HEIG-VD9
Overview of EmbeddedXEN
• Single binary (multi-kernel) image
• Automatic parsing & image relocation during hypervisor bootstrap
uImage
Boot Head
EmbeddedXEN
Hypervisor
DOM-0EOD
0 (Dom-0)
DOM-UEOD
1 (Dom-U)
vmlinux.dom0 vmlinux.domUvmlinux
Dom-0 Filesystem
Dom-U Filesystem
/home/root/squeezeos.rootfs.img: • Stored separately in sdcard, for example
REDS@HEIG-VD
Protection & Memory Isolation• Memory isolation between domains relies on different
address space isolation.• No further advanced mechanisms to protect hypervisor and guest memory
• The guest OS kernel runs at the same privilege as the hypervisor.
• Each domain receives its own (contiguous) physical memory region during domain set up.
• No pagination of the kernel linear address space is performed.
• Paravirt of memory management is kept minimal.
• Guest OS kernel has access to the whole memory.• No protection mechanism for strong isolation of VM (but is it really necessary?)
10
REDS@HEIG-VD
Protection & Memory Isolation
• Virtual & Physical Memory Layouts
11
3GiB
PAGE_OFFSET 0xC0000000
0x00000000
Initial Virtual Address Space Physical RAM
boot allocator
XEN heap (2 MiB)
frame table
I/OVMALLOC_END
Interrupt Vectors0xffff0000
0xFF000000
Hypervisor
dom0
Hypervisor code
1st-level page table
domU
dom0domU
Linearmapping
Linearmapping
3GiB
dom0
boot allocator
XEN heap (2 MiB)
frame table
I/O
Interrupt Vectors
Hypervisor code
1st-level page table
dom0 Virtual Address Space
REDS@HEIG-VD
Domain Interactions• Virtualization of peripherals in EmbeddedXEN is quite
similar to the existing mechanisms in XEN.• Driver split with frontend & backend drivers
• Communication with xenbus
• Use of grant tables for sharing/copying pages between domains
• However, a revisited (simplified) implementation of these mechanisms have been achieved in EmbeddedXEN.
• XEN store is dynamically allocated at boot time of guest OSes.
• No user space tools are required to manage XEN store entries or peripherals configs.
• Hotplugs & dynamic configs of peripherals are less relevant to embedded systems.
12
REDS@HEIG-VD
Domain Interactions• domU is passed information during bootstrap under control
of dom0.
13
xen-guestsubsys
Hypervisor
OS Subsys
start_info->store_mfnstart_info->store_evtchn
domU
xenstore
xen-guest subsysxenbus
thread
xenbus thread
xenstore thread
frontend drivers
backend drivers
event_channel C/D
event_channel A/B
Page Dom0
prod
cons
Page DomU
via unpause_domU(store_mfn, store_evtchn)
prod
cons
OS Subsys
REDS@HEIG-VD14
Domain Interactions• Grant tables are used in a different way
• Shared pages are possible only in the vmalloc'd area.
• Kernel linear addresses are not shareable; contents needs to be copied using temporary mappings.
3GiB
0xC0000000
0x00000000
RAM
VMALLOC_END
Hypervisor0xFF000000
Hypervisor
dom0
domU
VMALLOC_START
3GiB
Hypervisor
0xC0000000
0x00000000
VMALLOC_END
0xFF000000
VMALLOC_STARTgnttab(dom0)
gnttab(dom0)
mem_map_foreign(domU)
gnttab(dom0) gnttab_foreign[0]
gnttab
mem_map_foreign mem_map_foreign(dom0)
shared pages
Dom0 DomU
gnttab(domU) gnttabgnttab(domU)gnttab_foreign[1]
gnttab(domU)
shared pages
mem_map_foreign
has references to
has references to
has references to
has references to
REDS@HEIG-VD
Device Heterogeneity• Different levels of heterogeneity
• At CPU level: various instructions sets (locks, cache, etc.), various PTEs (MMU) flags, various co-processors, etc.
Compatibility ensured via hypercalls
• At peripherals level: not the same hardware
Compatibility ensured via backend driver processing
15
hypervisor-4.0.2
arch/arm
xen
arch/arm
mach-mx35
mach-msm
kernel
mm
include
xen-guest
mm
built-in.o cache-v6.S cache-v7.S Kconfig Makefile mm.h proc-macros.S tlb-v6.S tlb-v7.S
REDS@HEIG-VD
Device Heterogeneity• Example of ARMv6 running on ARMv7 CPU (iMX35 -> HTC
Desire HD)
16
linux-2.6.26-domU/arch/arm/mm/cache-v6.S:ENTRY(v6_flush_kern_cache_all) mov r0, #0#ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 0 @ D cache clean+invalidate#ifdef CONFIG_SMP mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate#else b v6_icache_inval_all#endif#else mcr p15, 0, r0, c7, c15, 0 @ Cache clean+invalidate#endif
mov pc, lr
linux-2.6.26-domU/xen-guest/hypervisor.c:void xen_flush_kern_cache_all(void){ struct mmuext_op op;
op.cmd = MMUEXT_FLUSH_CACHE; HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF);}
hypervisor-4.0.2/arch/arm/mm/cache_v7.S:ENTRY(xen_flush_kern_cache_all) stmfd sp!, {r4-r5, r7, r9-r11, lr} bl xen_flush_dcache_all @ much more complex!! mov r0, #0 mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate ldmfd sp!, {r4-r5, r7, r9-r11, lr} mov pc, lrENDPROC(xen_flush_kern_cache_all)
linux-2.6.26-domU/arch/arm/mm/cache-v6.S:.extern xen_flush_kern_cache_allENTRY(v6_flush_kern_cache_all) b xen_flush_kern_cache_all#if 0 /* paravirt */ mov r0, #0
…
mov pc, lr#endif /* 0 */
Original version Paravirt version
REDS@HEIG-VD
Device Heterogeneity• Example of framebuffer device heterogeneity
• Configuration of framebuffer is retrieved from Dom0 via xenbus
17
Dom0 DomUxen-guest
xenfb backend xenfb frontend
User space applications(/dev/fb0)
framebuffercore
User space applications(/dev/fb0)
framebuffercore
dom0 fb memory(allocated by dom0)
fb_event_dom_register fb_event_dom_switch
Query framebuffer params
xenbus thread
domU fb memory(allocated by domU)
FB config is retrieved from dom0
device-specific
framebuffermsmfb_pan_update()
xen-guest
xenstore
xenbus thread
REDS@HEIG-VD
DomUDom0xen-guest
Device Heterogeneity• Example of audio device heterogeneity
• DomU audio buffers are accessed from Dom0 via shared pages.
18
xen-audio backendxen-audio frontend
Sound device (headset, speakers, etc.)
xenstore
User space applications
/dev/msm_pcm_out - or -/dev/pcm/snd/pcm0c0d0p
xenbus thread
User space applications
/dev/pcm/snd/pcm0c0d0p
pcm subsystem
arch-specific audio driver
pcm subsystem
xenvaud_pcm_start xenvaud_pcm_stop
Audio buffers allocated in domU
xen-guest
xenstore
xenbus thread
REDS@HEIG-VD
Conclusions & Future Work• EmbeddedXEN is an embedded virtualization framework
which puts emphasis on efficient and heterogeneous hardware.
• Application environments can be re-used "as such" on modern platforms (Android-based for example) taking advantage of last generation hardware.
• EmbeddedXEN relies on the main principles of XEN, with a revisited lightweight, but less secure architecture.
• A single multikernel binary image, easy to deploy on the target platform without additional tools, makes EmbeddedXEN well tailored to embedded systems.
19
REDS@HEIG-VD
Conclusions & Future Work
• Further investigation projects:
• Elaboration of a domU using a graphical desktop for user applications
• Support of multicore ARM CPUs (cortex-A9, cortex-A15)
• Live migration of domU using remote NFS-filesystem (migration within a cloud)
• Support of hard realtime OS (RTEMS-paravirt)
20