xen on arm - linuxtag
TRANSCRIPT
![Page 1: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/1.jpg)
Xen on ARM
Stefano Stabellini
![Page 2: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/2.jpg)
Virtualization:why it matters
![Page 3: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/3.jpg)
Xen: the gears of the cloud● large user base
more than 10 million individuals users
● power the largest clouds inproduction
● not just for servers
![Page 4: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/4.jpg)
Xen: Open SourceGPLv2 with DCO (like Linux)Diverse contributor community
![Page 5: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/5.jpg)
Xen: Open Sourcesource:Mike Dayhttp://code.ncultra.org
![Page 6: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/6.jpg)
Xen Architecture
Hardware
Xen
Dom0 DomU
HW drivers
PV backends PV Frontends
DomU
PV Frontends
DomU
PV Frontends
![Page 7: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/7.jpg)
Xen Architecture:driver domains
Hardware
Xen
Dom0 DomU
NetFront
Disk Driver Domain
Toolstack Disk Driver
BlockBack
Network Driver Domain
Network Driver
NetBack BlockFront
![Page 8: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/8.jpg)
Xen: advantages● small surface of attack
● isolation
● resilience
● specialized algorithms (scheduler)
![Page 9: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/9.jpg)
Xen Architecture:HVM guests
Hardware
Xen
Dom0 stubdom
HW drivers
PV backends
HVM DomU
PV Frontends
HVM DomU
QEMU IO emulation IO emulation
![Page 10: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/10.jpg)
Xen upstream status● Xen (Dom0 and DomU support, PV
frontends and backends) fully upstream in Linux since v3.0A single 3.0.0 Linux kernel image boots on native, on Xen as domU, as dom0 and PV on HVM guest
● Xen upstream in QEMU since v1.3
● Xen supported by SuSE, Debian, Ubuntu, Fedora, CentOS, NetBSD and more
![Page 11: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/11.jpg)
ARM Serverscoming to market
4GB RAM, 4 cores per node3 x 6 x 4 x 4 = 288 cores
single node virtualization -manageability -
![Page 12: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/12.jpg)
● exploit the hardware as much as possible
● one type of guest
● Rearchitected for the modern age:○ no QEMU○ no compat code○ no shadow pagetables○ no PV MMU hypercalls
Design goals
![Page 13: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/13.jpg)
Xen on ARM architecture
![Page 14: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/14.jpg)
Xen on ARM architecture
![Page 15: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/15.jpg)
Exploit the hardwareExploit the hardware virtualization extensions support as much as possible:
● hypervisor mode● MMU: second stage translation
○ no PV MMU calls○ no shadow pagetables: -10721 lines of code!!
● hypercall: HVC● generic timers
![Page 16: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/16.jpg)
General Interrupt Controlleran interrupt controller with virtualization support
● use the GIC to inject hardware interrupts into dom0
● use the GIC to inject event notifications into any guest domains with Xen support○ use PPI 31○ advertise the IRQ via Device Tree
![Page 17: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/17.jpg)
One type of guest to rule them all
![Page 18: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/18.jpg)
One type of guestLike PV guests do it:● support booting from a supplied kernel● no emulated devices● use PV interfaces for IO
no need for QEMU
![Page 19: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/19.jpg)
Like HVM guests do it:● exploit HW nested paging● same entry point on native and on Xen● use Device Tree to discover Xen presence● no unnecessary devices in the Device Tree● simple device emulation can be done in Xen
no need for QEMU
One type of guest
![Page 20: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/20.jpg)
The hypercall calling conventionthe hypercall interface:● hvc instruction● hypervisor specific imm 0xEA1● hypercall arguments passed in registers
![Page 21: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/21.jpg)
Device TreeUse Device Tree to describe the virtual platform
hypervisor {compatible = "xen,xen", "xen,xen-4.2";reg = <0xb0000000 0x20000>;interrupts = <1 15 0xf08>;
};
![Page 22: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/22.jpg)
Device TreeUse Device Tree to describe the virtual platform
hypervisor {compatible = "xen,xen", "xen,xen-4.2";reg = <0xb0000000 0x20000>;interrupts = <1 15 0xf08>;
};
event notifications IRQ
Grant tablememory area
version of the Xen ABI
![Page 23: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/23.jpg)
a 64 bit "ready" ABI● a single hypercall ABI for 32 bit guests and
64 bit guests
no compat code in Xen○ 2600 lines of code lighter
![Page 24: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/24.jpg)
ARMv8● Builds on foundations laid by ARMv7
○ xen/arch/arm mostly common code
● Initially 32 bit dom0+domU on 64○ Kernels already ready○ 64-bit guest support in progress
![Page 25: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/25.jpg)
Code sizesometimes smaller is better
● Entire hypervisor ~200,000LOC○ X86 (64-bit only) ~100,000LOC (~4,000 ASM)
■ ~22,000: HVM. ~14,000 MMU
Common ARMv7 ARMv8 Total
xen/arch/arm 5,122 1,969 821 7,912
C 5,023 406 344 5,773
ASM 99 1,563 477 2,139
xen/include/asm-arm 2,315 563 666 3,544
TOTAL 7,437 2,532 1,487 11,456
![Page 26: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/26.jpg)
ChallengesFrom the emulator to real hardware:
![Page 27: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/27.jpg)
War Stories ChallengesFrom the emulator to real hardware:● barriers and flushes● cache coherency● GIC and race conditions● virt_timer documentation bugs
![Page 28: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/28.jpg)
Porting Xen to a new board● Xen only relies on GIC and GT
● platform specific code in Xen is reduced to:
○ secondary cpus bring up
○ UART drivers
○ any platform specific bootup quirks (ideally none)
![Page 29: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/29.jpg)
Status of the Project: ARMv7● Xen and Dom0 booting on Versatile Express
Cortex A15 and Arndale
● XL (Xen toolstack) ported to ARM
● PV console, disk and network working
● basic VM lifecycle operations functional
● Xen and Linux ARM patches fully upstream
![Page 30: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/30.jpg)
Status of the Project: ARMv8● Xen booting 64 bit
● Dom0 32 bit boots on Xen 64 bit
● 32 bit guest creation and destruction
● Shared code means most features developed on ARMv7 Just Work
![Page 31: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/31.jpg)
Xen 4.3○ ARMv7 (VExpress and Arndale) fully supported○ ARMv8 64-bit port of the hypervisor
Xen 4.4○ increase HCL○ automated testing○ ARMv8 64-bit virtual machines and tools○ PCI passthrough, live migration
Linux 3.11/3.12○ full ARMv8 64-bit Xen guest support
Roadmap
![Page 32: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/32.jpg)
Demo
![Page 33: Xen on ARM - LinuxTag](https://reader030.vdocuments.site/reader030/viewer/2022032610/623a43b05adea41c8436f89e/html5/thumbnails/33.jpg)
● http://www.xen.org
● Xen on ARM @wiki.xen.org: goo.gl/FKNXe
● http://lists.xen.org/mailman/listinfo/xen-devel
More Information