fast packet processing with kubevirt · 2020. 8. 31. · multus cni linux bridge 1. cni (2. cni)...

Fast Packet Processing with KubeVirt

Petr HoráčekSenior Software EngineerRed Hat

David VosselPrincipal Software EngineerRed Hat

What is KubeVirt?

“KubeVirt is a Kubernetes extension that allows running traditional VM workloads natively side by side with Container workloads.”

KubeVirt Basics

Kubernetes + KubeVirt

KubeVirt VM is a KVM+qemu process running inside a pod

Application POD

App Container

KubeVirt PODKVM + Qemu

Container

Standard CRI Runtime(Docker, CRI-O, Containerd, etc...)

Scope of KubeVirt Project?

● Limited in scope to managing virtual machine lifecycle○ Starting VMs○ Stopping VMs○ Pausing/suspending VMs○ Live Migrating VMs○ Monitoring VMs○ Replicating VMs.○ Etc…

● Utilizes existing cluster resources for everything else○ PVCs for persistent virtual machine disks○ CNI for network devices○ CPU Manager for cpu affinity○ Device Plugins for access to host devices (GPU, SR-IOV)

VM workloadsVM workloads

Addon Functionality

● Enable common workflows and functionality with addons○ CDI: For building VM Image repositories using PVCs○ Multus + CNI Plugins: For tieing VMs into multiple networks. ○ SR-IOV CNI/Device Plugin: for passing making SR-IOV devices available to VMs○ Nvidia GPU Device Plugin: For passing GPU devices to VMs.

Kubernetes

KubeVirt CDI multus SR-IOVCNI+Device Plugin

GPUDevice Plugin

VM workloads

KubernetesControl Plane

KubeVirt + Addons

High Performance VMs

What is an Opinionated Install?

Opinionated Install: KubeVirt + optional addon components + configuration data

VM workloadsVM workloads

Kubernetes

KubeVirt CDI multus SR-IOVCNI+Device Plugin

GPUDevice Plugin

VM workloads

KubernetesControl Plane

KubeVirt + Addons

High Performance VMs

Opinionated Install for Traditional VMs

● Opinionated Install Addons for… ○ Importing VMs from legacy VM management Platforms (Like VMWare and oVirt)

○ Providing feature parity with legacy VM management platforms

● Problem…○ Complex collection of [Kubevirt + Addons + Config data] is difficult to manage.

● Solution?

Hyper Converged Operator (HCO)

● HCO is an operator of operators or meta operator○ Coordinates installing KubeVirt + addons

○ Allows installing/updating entire opinionated install as a single cohesive unit.

HCO Architecture

HCO Manifest

HCO Manifest Deploys HCO operator + Subcomponent operators

HCO Architecture

HCO Manifest HCO Operator

Virt Operator

CDI Operator

CNA Operator

HCO Manifest Deploys HCO operator + Subcomponent operators

HCO Architecture

Virt CR CDI CR CNA CRHCO

Manifest HCO Operator

Virt Operator

CDI Operator

CNA Operator

HCO deploys subcomponents using subcomponent operator CRs

HCO Architecture

Virt Operator

CDI Operator

CNA Operator

Subcomponent Operators deploy the subcomponent using the HCO’s supplied configuration

KubeVirt

Multus

Current HCO Subcomponents

● Virt-operator - KubeVirt Core● Cluster Network Addons (CNA) - multus, mac pool, CNI plugins.● SSP operator - node feature labeller, common vm templates● CDI operator - VM image importing and cloning● Node maintenance operator - server side node maintenance (similar to

kubectl drain but server side)● Hostpath Provisioner Operator - local storage storage class provisioner● VM Import Operator - workflows for importing rhv/ovirt vm to kubevirt

Installing HCO from OLM Marketplace

Virt CR CDI CR CNA CR

Virt Operator

CDI Operator

CNA Operator

KubeVirt

Multus

Operator Lifecycle Manager (OLM)

Marketplace User Subscribes to HCOvia OLM marketplace

Updating from OLM Marketplace

Virt CR CDI CR CNA CR

Virt Operator

CDI Operator

CNA Operator

KubeVirt

Multus

Operator Lifecycle Manager (OLM)

Marketplace

New HCO Manifests are posted on updates

Network Addons

Virt Operator

CDI Operator

CNA Operator

KubeVirt

Multus

Network Addons

CNA CRHCO

CNA OperatorMultus

Network Addons

CNA CRHCO

CNA OperatorMultus

Bridge CNI

Network Addons

CNA CRHCO

CNA OperatorMultus

OVS CNI

Bridge CNI

Network Addons

CNA CRHCO

CNA OperatorMultus

MacPool

OVS CNI

Bridge CNI

Network Addons

CNA CRHCO

CNA OperatorMultus

MacPool

OVS CNI

Bridge CNI

nmstate

Network Addons

CNA CRHCO

CNA OperatorMultus

MacPool

OVS CNI

Bridge CNI

nmstate

SR-IOV Operator

SR-IOVDP

SR-IOVCNI

Regular Packet Processing

● The default CNI plugin of the cluster,● often based on an overlay network,● connected to the VM container,● forwarded to the VM process through NAT or Linux bridge.

● Offers all the benefits of Kubernetes infrastructure,● is not as fast as other solutions.

Regular Packet Processing (Cluster Network)

Node Node

eth0 eth0

Node Node

eth0 eth0

Node Node

eth0 eth0

VMeth0

Node Node

eth0 eth0

VMeth0

Regular Packet Processing (Binding)

eth010.10.0.1

container

eth010.10.0.1

container

a process

eth010.10.0.1

container

virtual machine

eth010.10.0.1

container

virtual machine

eth010.10.0.1

container

virtual machine

eth010.10.0.1

container

virtual machine

eth0br1

eth010.10.0.1

container

virtual machine

DHCP server

offering 192.168.0.1

eth010.10.0.1

container

virtual machine

eth0192.168.0.1

DHCP server

eth010.10.0.1

container

virtual machine

iptables

ifdst.port == 80

forward tobr1 192.168.0.1

DHCP server

eth0192.168.0.1 br1

Regular Packet Processing (Example)

apiVersion: kubevirt.io/v1alpha3kind: VirtualMachineInstancemetadata: name: vmi-test ...

spec: networks: - name: default pod: {} domain: devices: ... interfaces: - name: default masquerade: {} ......

Faster Packet Processing

● Bridge configured on the host,● with host NIC as its port,● extending L2 access to the network to containers,● as an additional network,● forwarded to the VM through another bridge.

Faster Packet Processing

● Bridge configured on the host,● with host NIC as its port,● extending L2 access to the network to containers,● as an additional network,● forwarded to the VM through another bridge.

● May be faster thanks to direct L2 access,● is not 100% integrated to Kubernetes networking components.

Faster Packet Processing (Cluster Network)

Node Node

eth0 eth0

Node Node

eth0 eth1 eth0 eth1

Switch

Node Node

eth0 eth1 eth0 eth1

br1 br1

Switch

Node Node

eth0 eth1 eth0 eth1

br1 br1

Switch

VMeth0 eth1

Faster Packet Processing (Multus)

kubelet

calico

kubelet

multus

kubelet

calico

multus

1. CNI

kubelet

calico

multus

linux bridge

1. CNI

(2. CNI)

kubelet

calico

multus

linux bridge …

1. CNI (3. CNI)

(2. CNI)

Faster Packet Processing (Binding)

container

virtual machine

container

virtual machine

container

virtual machine

br0 enp3

container

virtual machine

br0 enp3

container

virtual machine

br0 enp3

container

virtual machine

eth1 br1

br0 enp3

container

virtual machine

eth1 br1

br0 enp3

Faster Packet Processing (Example)

apiVersion: nmstate.io/v1alpha1kind: NodeNetworkConfigurationPolicymetadata: name: br1-eth0

spec: desiredState:

interfaces:- name: br1

type: linux-bridge state: up ipv4: dhcp: true enabled: true bridge: port: - name: eth0

apiVersion: k8s.cni.cncf.io/v1kind: NetworkAttachmentDefinitionmetadata: name: blue-network annotations: k8s.v1.cni.cncf.io/resourceName: ovs-cni.network.kubevirt.io/br1spec: config: '{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "br1", "vlan": 100 }'

apiVersion: kubevirt.io/v1alpha3kind: VirtualMachineInstancemetadata: name: vmi-test ...

spec: networks: - name: default pod: {} - name: blue multus: networkName: blue-network domain: devices: ... interfaces: - name: default masquerade: {} - name: blue bridge: {} ......

Really Fast Packet Processing

Fast Packet Processing

● SR-IOV NIC configured on the host,● exposed as a node resource,● plugged as a device into a container,● mounted into the virtual machine through as a passthrough.

Fast Packet Processing

● SR-IOV NIC configured on the host,● exposed as a node resource,● plugged as a device into a container,● mounted into the virtual machine through as a passthrough.

● As fast as it can get,● requires special hardware and can service only so many containers.

Fast Packet Processing (SR-IOV)

● Dedicated NICs for each VM would be a clunky solution.

● SR-IOV exposes a single NIC as many,● isolation of PCIe resources,● Physical Functions (PFs) and Virtual Functions (VFs),● multiple containers and VMs can utilize the hardware NIC at the same time.

Fast Packet Processing (not SR-IOV)

The slow part

Fast Packet Processing (SR-IOV)

Fast Packet Processing (SR-IOV Operator)

● Optional, but so useful,● reports available NICs,● configures them,● deploys other needed components,● and all of that through Kubernetes API.

Fast Packet Processing (Device Plugin)

● Allocates countable resources and plugs them into containers,● gRPC service cooperating with kubelet,● discovery,● advertising,● allocation,● health checking.

Fast Packet Processing (CNI)

● SR-IOV CNI complements the device plugin,● it configures allocated VF,● both VF parameters and configuration of container's netlink interface.

Fast Packet Processing (Configuration)

Node Node

eth0 eth1 eth0

Switch"fast net"

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

Fast Packet Processing (Advertisement)

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

device-plugin

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

device-plugin

capacity: cpu: 2 memory: 2049008Ki mlx/sriov_net: 3

Node Node

eth0 eth1 eth0

Switch"fast net"

conf-daemon

device-plugin

capacity: cpu: 2 memory: 2049008Ki mlx/sriov_net: 3

Fast Packet Processing (Scheduling)

Node Node

eth0 eth1 eth0

Switch"fast net"

Node Node

eth0 eth1 eth0

Switch"fast net"

Node Node

eth0 eth1 eth0

nets: fast-netVM

Switch"fast net"

Node Node

eth0 eth1 eth0

nets: fast-netVM

Switch"fast net"

Node Node

eth0 eth1 eth0

nets: fast-netVM

Switch"fast net"

Node Node

eth0 eth1 eth0

nets: fast-netVM

Switch"fast net"

Fast Packet Processing (Attachment)

● After a Pod requesting SR-IOV gets scheduled● device plugin would tell kubelet how to pass VF into a container,● kubelet plugs in the VF,● Multus fetches ID of the VF and passes it to SR-IOV CNI,● SR-IOV CNI then configures the VF.

● Difference between netlink and vfio-pci method.

Fast Packet Processing (Binding)

container

virtual machine

enp3VF

container

virtual machine

VFeth1

container

virtual machine

VFeth1

container

virtual machine

VFeth1

Fast Packet Processing (Tweaking)

● SR-IOV can be made even faster with NUMA awareness and CPU management enabled on the cluster,

● the goal is to align all resources of a VM as close together as possible,● KubeVirt leverages CPU manager when dedicated CPU placement is enabled.

Fast Packet Processing (DPDK)

● Very fast user-space networking,● requires hugepages,● hugepages can be propagated through Kubernetes and KubeVirt to the guest.

Fast Packet Processing (Example)

apiVersion: sriovnetwork.openshift.io/v1kind: SriovNetworkNodePolicymetadata: name: policy-mlx namespace: sriov-network-operatorspec: deviceType: vfio-pci mtu: 9000 nicSelector:

deviceID: "1017"vendor: "15b3"pfNames:- ens801f0

nodeSelector:feature.node.kubernetes.io/network-sriov.capable: "true"

numVfs: 6 resourceName: fast-network

apiVersion: sriovnetwork.openshift.io/v1kind: SriovNetworkmetadata: name: fast-network-10 namespace: sriov-network-operatorspec: ipam: |

{} vlan: 10 spoofChk: "off" resourceName: fast-network networkNamespace: default

apiVersion: kubevirt.io/v1alpha3kind: VirtualMachineInstancemetadata: name: vmi-test ...spec: domain: cpu: sockets: 3 cores: 1 threads: 1 model: host-model dedicatedCpuPlacement: true isolateEmulatorThread: true memory: hugepages: pageSize: "1Gi"

resources: limits: cpu: 3 memory: 4Gi devices: interfaces: - name: fast sriov: {} networkInterfaceMultiqueue: true networks: - name: fast multus: networkName: fast-network-10 ...

Questions?

References

Hyperconverged Cluster OperatorKubeVirtCluster Network Addons OperatorSR-IOV OperatorSR-IOV Device PluginSR-IOV CNI

fast packet processing with kubevirt · 2020. 8. 31. · multus cni linux bridge 1. cni (2. cni)...

Documents

cni dec 2007 copyright and mass dig for cni

pesquisa cni ibope

textos latinos 2º bachillerato. tema 1: jÚpiter romani...

kubevirt: how to run vms on kubernetes

cni young intelmax

cni offerta

· spanungstechnik die erste multus 13400 für den...

doppelmayr/garaventa setzt auf eine multus u 4000 …...die...

okuma multus operation & maintenance manual le11-165-r4

pesquisa cni/ibope

multus b300 multus b400 - logismarket b300 multus b400...

ambiental - cni

caixa2 cni

intelligent multitasking machines€¦ · multus...

multi-tasking okuma multus b300w. multi-tasking mori-seiki...

a.01562-3-21222 3 4 5 0.7mmq 3.0mm 3kg/cni fj20kg/cni...

cni junho 2009

multitasking dreh-/frÄszentrum. okuma multus b300ii...

cni mc donough_preservation

osp-p200l macturn+multus series operation manual le32-114-r4