pci passthrough with red hat openstack...
TRANSCRIPT
PCI Passthrough with Red Hat
OpenStack Platform
Ken Savich
Sr OpenStack Solution Architect, Red Hat2016-12-6
About Me
- Sr OpenStack Solution Architect
- with Red Hat since January 2016
- previously engineering at Rackspace - 9 years
- I love ramen.
пройти через
Ken Savich
Agenda
- Background
- The story
- Technical details
- Scripting
- Results
- Applications
Ken Savich
Red Hat OpenStack Platform 9
- 100% Community based
- Enterprise proven
- Automated updates and upgrades
- Co-engineered performance
ADD NAME (View > Master > Slide master)
Background
Ken Savich
Why PCI Passthrough?
- Performance
- Exclusive access
- No hypervisor support
- Domain isolation
- Security
Ken Savich
device emulation
Ken Savich
device passthrough
ADD NAME (View > Master > Slide master)
The Story
Ken Savich
- Cisco- Nvidia- Petrel- Red Hat
ADD NAME (View > Master > Slide master)
ADD NAME (View > Master > Slide master)
Requirements
- Red Hat OpenStack
- Self-contained infrastructure
- Windows 7 guests w/Petrel software
- Native guest access to Nvidia Tesla m60 grid cards
- Performance!
ADD NAME (View > Master > Slide master)
Technical Details
ADD NAME (View > Master > Slide master)
Requirements
- RHEL 7.3 - kernel < 4.1
- Virtualization Technology - bios & kernel
- vmx (intel)
- svm (amd)
- Red Hat OpenStack Platform 8 or 9
- A PCI card with native drivers for your guest
$ egrep `(vmx|svm)` /proc/cpuinfo
ADD NAME (View > Master > Slide master)
Procedure
- Turn off hyper-threading- Find your pci device- Configure your Compute node(s)- Configure your Controller node(s)- Create a flavor- Launch and instance- Test test test!
ADD NAME (View > Master > Slide master)
PCI Passthrough - first steps
- Get your packages $ sudo yum install pciutils gcc kernel-devel kernel-debug kernel-headers -y
- blacklist any competing drivers/etc/default/grub: GRUB_CMDLINE_LINUX=... rd.driver.blacklist=nouveau
$ sudo echo "blacklist nouveau" > /etc/modprobe.d/nvidia-blacklist.conf
$ sudo echo "blacklist nvidia" >> /etc/modprobe.d/nvidia-blacklist.conf
- turn on IOMMU/etc/default/grub: GRUB_CMDLINE_LINUX=... intel_iommu=on
ADD NAME (View > Master > Slide master)
PCI Passthrough - now we’re feeling it
- Find your device$ sudo lspci |grep -i nvidia
05:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
- Find the pci id of your device$ sudo lspci -n|grep 05:00.0|awk '{print $3}'
10de:13c2
- Tell grub the PCI stub ID/etc/default/grub: GRUB_CMDLINE_LINUX=... pci-stub.ids=10de:13c2
ADD NAME (View > Master > Slide master)
PCI Passthrough - bounce it
- Update grub$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
- Configure kernel hugepages# mkdir /hugetlbfs
# echo "2048" > /proc/sys/vm/nr_hugepages
# echo "vm.nr_hugepages=2048" > /etc/modprobe.d/hugepages.conf
# echo "hugetlbfs /hugetlbfs hugetlbfs defaults 0 0" >> /etc/fstab
- REBOOT!!!
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Compute Node
- Find your vendor and product codes# lspci -nn|grep -P '^(?=.*nvidia)(?=.*vga)'
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
- Whitelist your device in nova.conf/etc/nova/nova.conf:
pci_passthrough_whitelist = [{"vendor_id": "10de", "product_id": "13c2"}]
- Restart nova service$ sudo systemctl restart openstack-nova-compute.service
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Controller
- Add the filter class to the scheduler/etc/nova/nova.conf:
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
- Make sure PciPassthroughFilter is in the default list/etc/nova/nova.conf:
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Controller
- Add an alias for your device/etc/nova/nova.conf
pci_alias = {"vendor_id":"10de", "product_id":"13c2", "name":"nvidia"}
- Restart nova services# systemctl restart openstack-nova-api.service openstack-nova-scheduler.service \
openstack-nova-conductor.service
ADD NAME (View > Master > Slide master)
Off to the races...
- Create a flavor$ nova flavor-key m1.large set "pci_passthrough:alias"="nvidia:2"
- Launch an instance$ nova boot --image rhel7 --key_name test --flavor m1.large mypci
- Install native drivers in your guest
ADD NAME (View > Master > Slide master)
Common Problems
- PCIe reset
- Virtio drivers
- Display virtualization
- Live migration
ADD NAME (View > Master > Slide master)
Divider SlideScripting
ADD NAME (View > Master > Slide master)
Scripting
- goo.gl/RsceFv
- Requires RHEL 7.x
- Nvidia specific
- Must be run on compute nodes
ADD NAME (View > Master > Slide master)
Per-node configuration with Tripelo
- Interfaces
OS::TripleO::ControllerExtraConfigPre: Controller node additional configuration
OS::TripleO::ComputeExtraConfigPre: Compute node additional configuration
- for examples, see:
/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
ADD NAME (View > Master > Slide master)
Results
ADD NAME (View > Master > Slide master)
Benchmarking
- Windows 7 instance- Red Hat virtIO drivers
- Furmark = 60+ fps
ADD NAME (View > Master > Slide master)
Applications
ADD NAME (View > Master > Slide master)
Possibilities!
- HPC applications- Video encoding- Audio pre/post- NFV
- SR-IOV for VNF- Gaming
ADD NAME (View > Master > Slide master)
Try it for yourself, you’ll like it
- Get Red Hat OpenStack 9 here:- https://access.redhat.com/products/red-hat-openstack-platform/evaluation
● Provides a full-featured 60 day entitlement.● Provides self-service support.● Offers three 2-socket server subscriptions - enough to run a small 3-node test
cluster.● Is not intended for production use.
Let me know how it goes: [email protected]
ADD NAME (View > Master > Slide master)
References
- Red Hat PCI Passthrough Documentation:
- http://goo.gl/Ea9KLp
- OpenStack PCI Passthrough Documentation:
- https://wiki.openstack.org/wiki/Pci_passthrough
ADD NAME (View > Master > Slide master)
Questions
ADD NAME (View > Master > Slide master)
Divider SlideThank you