xpds14 - xen on arm: status and performance - stefano stabellini, citrix
DESCRIPTION
As the first ARM servers and microservers hit the market, Xen on ARM is becoming more mature, stable and reaching feature parity with x86. This talk will present the current status of the project, will describe the latest improvements, the gaps that still need to be filled and the roadmap going forward. ARMv8 silicon is now available for purchase: we can measure how well Xen on ARM 64-bit is performing on real hardware and compare the performance figures with other hypervisors. The presentation will show these results, it will measure the overhead introduced by Xen on ARM and will compare it with the overhead introduced by Xen and KVM on x86. The talk will explain the reasons behind performance shortfalls and present ideas on how to address them in the future. The performance results will be used to determine when it makes sense to use Xen on ARM and what are the best use cases for it.TRANSCRIPT
18 August 2014
Xen on ARMHow fast is it, really?
Stefano Stabellini
Status
© 2013 Citrix | Confidential – Do Not Distribute
Xen Project 4.4 release: statusFeatures:● 64-bit guest support in ARMv8● stable hypercall ABI● basic lifecycle operations● memory ballooning● scheduler configurations, vcpu pinning, cpu pools
HCL (not inclusive):● ARMv8 FVP● Arndale board● Allwinner Sunxi (Cubieboard 2)● TI OMAP5● Applied Micro X-Gene
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Linux v3.13+: status
● dom0 and domU● 32-bit and 64-bit● SMP● swiotlb-xen● PV drivers
○ disk○ network○ console○ framebuffer○ keyboard○ mouse
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● SMMU support● Device Assignment on ARM● Xen UEFI host support● GICv3 support● support for up to 1TB of guest RAM● super pages support● support for sparse memory map● support for 48-bit physical addresses● QEMU PV backend on ARM: framebuffer,
mouse, keyboard, multi-console© 2014 Citrix
● MiniOS on ARM● PSCI v0.2
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● SMMU support● Device Assignment on ARM● Xen UEFI host support● GICv3 support● support for up to 1TB of guest RAM● super pages support● support for sparse memory map● support for 48-bit physical addresses● QEMU PV backend on ARM: framebuffer,
mouse, keyboard, multi-console© 2014 Citrix
● MiniOS on ARM● PSCI v0.2
© 2013 Citrix | Confidential – Do Not Distribute
xen-devel traffic
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Maintainers & reviews
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
irq latency reduction: no maintenance interrupts
DomU
Xen
irq 109
virq 109
DomU
Xen
EOI
DomU
Xen
Maintenance interrupt
GICH_LRWrite
GICH_LRClear
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
irq latency reduction: no maintenance interrupts
DomU
Xen
irq 109
virq 109
DomU
Xen
EOI
DomU
Xen
Maintenance interrupt
GICH_LRWrite
GICH_LRClear
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
irq migration: physical follow virtual
vcpu0 vcpu1
pcpu0 pcpu1
irq 109
virq 109
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
irq migration: physical follow virtual
vcpu0 vcpu1
pcpu0 pcpu1
irq 109
virq 109
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
irq migration: physical follow virtual
vcpu0 vcpu1
pcpu0 pcpu1
irq 109
virq 109
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
multi-grants support
© 2014 Citrix
Backend Frontend
PagePage
Page
Grant
Grant
© 2013 Citrix | Confidential – Do Not Distribute
multi-grants support
© 2014 Citrix
Backend Frontend
PagePage
Page
Grant
Grant
mfn
==
mfn
© 2013 Citrix | Confidential – Do Not Distribute
multi-grants support
© 2014 Citrix
Backend Frontend
PagePage
Page
Grant
Grant
mfn
==
mfn
mfn_to_pfn
??
© 2013 Citrix | Confidential – Do Not Distribute
multi-grants support
© 2014 Citrix
Backend Frontend
PagePage
Page
Grant
Grant
Pagepfn == mfn
mfn
==
mfn
mfn_to_pfn
??
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: performance improvements
● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: hardware and interfaces
● SMMU support● Xen UEFI host support● PSCI v0.2● GICv3 support
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Coming in Xen 4.5: features
● (Non-PCI) Device Assignment on ARM● QEMU PV backend on ARM: framebuffer, mouse, keyboard, multi-console● MiniOS on ARM
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Looking ahead
● save/restore and live-migration● PCI device assignment
● IOMMU emulation for remote processors● GICv2m support● SMMUv2 support● ACPI support
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Embedded and Automotive
● Embedded and Automotive Xen Project initiativehttp://xenproject.org/about/events/viewevent/146-webinar-virtualization-in-the-automotive-industry-2.html
● Embedded and Automotive PV drivers projecthttp://xenproject.org/developers/teams/embedded-and-automotive.htmlhttp://wiki.xenproject.org/wiki/Embedded_and_Automotive_PV_Drivers/Roadmap
© 2014 Citrix
Benchmarks
© 2013 Citrix | Confidential – Do Not Distribute
Execution environment
● Linux v3.15-rc8dom0, domU and native kernel
● Xen v4.5-unstable+
● QEMU-KVM v2.0.0
● OpenSUSE 13.1
© 2014 Citrix
Applied Micro X-Gene Intel Xeon CPU X5650
ARMv8 64-bit 8 cores 2.4 Ghz x86_64 2 sockets x 6 cores 2.67 Ghz
© 2013 Citrix | Confidential – Do Not Distribute
Kernbenchvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
PBZIP2virtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
FIO rand-rwvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
SPECjbb2005virtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
PGBenchvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
PGBenchvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
PGBenchvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
PGBenchvirtualization overhead (percentage) - Lower is better
© 2014 Citrix
120
Benchmarks by
© 2013 Citrix | Confidential – Do Not Distribute
SpecINT 2006Xen on ARM virtualization overhead (percentage) - Lower is better
2%
© 2014 Citrix
Conclusions
© 2013 Citrix | Confidential – Do Not Distribute
● many new features coming● good performance results● still lots of room for improvement
Conclusions
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
● need more reviewers
Conclusions
© 2014 Citrix
Questions?
© 2013 Citrix | Confidential – Do Not Distribute
More information
● http://www.xenproject.org
● Xen on ARM @wiki.xenproject.org http://goo.gl/FKNXe
● Xen on ARM whitepaper http://goo.gl/vWMn7D
● http://lists.xenproject.org/mailman/listinfo/xen-devel
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Work better. Live Better.
Citrix
© 2014 Citrix
Slide parking lot
© 2013 Citrix | Confidential – Do Not Distribute
IPerf
Linerate
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
FIOXen on ARM virtualization overhead (percentage) - Lower is better
© 2014 Citrix
© 2013 Citrix | Confidential – Do Not Distribute
Reading xen-devel
© 2014 Citrix