high-performance hypervisor architectures: …...• self-virtualizing devices (and accelerators)...
TRANSCRIPT
![Page 1: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/1.jpg)
High-Performance Hypervisor Architectures:
Virtualization in HPC Systems
Ada Gavrilovska, Sanjay Kumar, Himanshu Raj, Karsten Schwan, Vishakha Gupta, Ripal Nathuji, Radhika Niranjan, Adit Ranadive, Purav Sarayia
College of ComputingGeorgia Tech
![Page 2: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/2.jpg)
Virtualization and HPC
• Virtualization technology major focus in enterprise settings– power, cost, consolidation; manageability and
portability… • Adoption lags behind in HPC domain
– fear of tapping into scare HPC platforms’ resources– power, cost, consolidation – not critical constraints in
HPC environments• Our objective: understand feasibility and utility
– Is there room and need for virtualization in HPC? – As we move to many-core?– Any new functionality/services?
![Page 3: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/3.jpg)
Overview
• Potential benefits• Sidecore approach to VMM architecture design
– Scalable hypervisor architectures for future many-core platforms
• Self-virtualizing devices (and accelerators)– challenges and opportunities they present
• Platform management in virtualized environments
• Ongoing work targeting general purpose multicore systems, from low-end, personal platforms to high-end data center environments
![Page 4: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/4.jpg)
Potential benefits
• Fault-tolerance: migration• Fault-tolerance: monitoring• Shared I/O and service nodes• New functionality• Portability and manageability• Development, debugging and sharing• Mixed use for capacity & capability computing
• => Worth further investigation…
![Page 5: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/5.jpg)
Sidecore Approach
• VMMs in many-core platforms– coordinating VMM operations across many (80!) cores may
introduce prohibitive noise levels and resource requirements• Decompose VMM functionality
– factor select subsets of VMM operations and assign their execution to a designated core(s)
– eliminate or reduce expensive context/VMentry/VMexitswitches; exploit locality
– improve VMM scalability to number of cores– Sidecore-resident functionality
• factored out from monolithic VMMs (e.g., Xen)• components in future modular/lightweight VMMs
• Architectural considerations– number and location of sidecores– VM core – Sidecore communication channels
![Page 6: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/6.jpg)
Page Table manipulation with Sidecore
Case study:• Modified Xen 3.0 to designate
page-table management to a designated sidecore
• VM entry/exit operation eliminated
• Communication: shared memory with polling
• Basic feasibility and understanding of challenges– Evaluation challenge due to
small core count on current platform
![Page 7: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/7.jpg)
Benefits of eliminating VM-switches
• Up to 41% reduction in page table update latency
![Page 8: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/8.jpg)
Self-Virtualizing Devices
• Device-level virtualization-awareness – Safely mux/demux access
to device resources– Associate self-virtualization
functionality with device-resident or device-near cores
– Challenges: • device-VM notifications ->
interrupts vs. polling• IOMMU operations
![Page 9: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/9.jpg)
Self-virtualized NICs using the IXP2400 NP
IXP2400 NP
PCI Communication
Network
ControllerDomain
Guest 0 Guest 1
VIF0
Hypervisor
Host
Controller VIF1
VIFs
Controller
![Page 10: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/10.jpg)
Interaction in an S-VNIC
![Page 11: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/11.jpg)
Improvements in latency and bandwidth
![Page 12: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/12.jpg)
Architectural limitations
Throughput of the PCI path Effects of virtual interrupt sharing
![Page 13: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/13.jpg)
Insights for future multicore systems
![Page 14: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/14.jpg)
Virtualized interrupts with Sidecore
![Page 15: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/15.jpg)
Importance of S-V I/O
• Performance– Hypervisor acceleration/bypass
• End-to-end QoS• VM Migration & Device Remoting• “Logical” Devices
![Page 16: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/16.jpg)
SV-I/O and QoS support
• VM’s resource requirements need to include device-level resources
• Need for coordinated hypervisor- and device- level scheduling decisions
File Size 256Mb for Dom ratios 768_512_256
0
5000
10000
15000
20000
25000
64 128 256 512 1024 2048 4096 8192 16384Record Size in KB
Thro
ughp
ut in
KB/
sec
Dom1Dom2Dom3
![Page 17: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/17.jpg)
Remote Device Virtualization
Hypervisor
ServiceVM
NICDomain
GuestVM
NICDomain
ServiceVM
Hypervisor
Guest VM
BE FE
NIC vdevice NICdevice
FEBE
device driver
LAN/LAWNLAN/LAWN
- important for VM migration- device-centric S-VIO -> data path through BE
and NIC domain is pushed on device- current numbers: ~11% latency reduction
![Page 18: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/18.jpg)
• Device remoting– feasibility and
utility
![Page 19: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/19.jpg)
“Logical” Devices
• Once a device is virtualized, there is no reason for it to be “real”
• May associate codes with S-V IO processing components to implement upper-level functionality– Data reformating
• e.g., file system issues– Filtering
• e.g., security/privacy issues, threshold comparisons…
– QoS properties
![Page 20: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/20.jpg)
• Case study:– image
reformatting
![Page 21: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/21.jpg)
Other issues• Device sharing among multiple virtualized
platforms– Cannot afford a single device domain to become
hotspot– S-VIO implements coordination and management
functionality• Monitoring and QoS
– Interface and support on device for monitoring and scheduling among virtual device instances (e.g., VIFs)
– Metadata management on S-V I/O– Resource management in virtualized environments:
• system-wide vs. platform-wide vs. VM- management objectives
• current primary consideration in our work power– portable to other domain– power may become relevant factor in HPC too – if the
capabilities to control it are present
![Page 22: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/22.jpg)
Conclusions• Virtualization may have important contributions in HPC
infrastructures• Technical challenges to attain benefits
– efficient hypervisor design to eliminate overheads and noise– scalable hypervisors for multicore platforms– Improved performance on I/O path– Better support for device remoting, needed for efficient VM
migration– Ability to instantiate ‘logical’ devices and better meet
application requirements– Finer grain support for ensuring end-to-end QoS– Grater scalability for shared virtualized devices– Coordinated management (e.g., power?) mechanisms
• Prototype realization of a S-V NIC– Gives us insights into access and control APIs
• Proof of concept concept results– Efficient & scalable Hypervisor for target platforms
![Page 23: High-Performance Hypervisor Architectures: …...• Self-virtualizing devices (and accelerators) – challenges and opportunities they present • Platform management in virtualized](https://reader034.vdocuments.site/reader034/viewer/2022042612/5f3db0cb2091e8547847f326/html5/thumbnails/23.jpg)