vfio-pci for VMs, and the DRA SR-IOV driver

1One VF, two completely different handovers

Everything in this module so far has produced the same shape: a VF that stays a kernel network device on the host, and a pod that is allowed to use it. That is deviceType: netDevice, and it is the default when the device plugin’s ConfigMap says nothing.^[3] The plugin also accepts accelerator and auxNetDevice, and the selector list differs between them.^[3]

vfio-pci is not a fourth flavour of the same thing. It changes who owns the hardware. NVIDIA’s KubeVirt page defines it as passing “a VF’s PCI device directly into the guest VM via the VFIO userspace interface”.^[1] The host stops driving the function: there is no mlx5_core binding, no host netdevice, nothing for ip link to show. The guest gets the PCI function and must bring its own driver.

Two consequences follow immediately, and both surprise people.

First, the RDMA flag goes off. NVIDIA’s example policy sets isRdma: false and the page is explicit that RDMA is incompatible with VFIO passthrough.^[1] That is not a performance compromise. isRdma asks the host-side plumbing to expose RDMA resources for a host netdevice, and with vfio-pci there is no host netdevice; the guest runs its own RDMA stack over its own driver.

Second, the guest image becomes part of the network design. “NVIDIA VFs passed via VFIO require the mlx5_core driver inside the guest VM.”^[1] Without it the device is present in the guest’s lspci and no interface appears, which reads on a ticket as a passthrough failure and is a guest-image problem.

The resource-name rules from lesson 2 do not change here. The device plugin still advertises <prefix>/<resourceName>, and the pod - in this case the virt-launcher pod KubeVirt creates for the VM - still has to request that exact string.^[3][8]

# NicClusterPolicy .spec.sriovDevicePlugin.config
{ "resourceList": [{
    "resourcePrefix": "nvidia.com",
    "resourceName": "hostdev",
    "selectors": { "vendors": ["15b3"], "isRdma": true }
}]}

2The three objects, in the order they must exist

NVIDIA documents the KubeVirt path as three objects plus a set of host prerequisites.^[1]

One - the node policy. It creates the functions and advertises them, and it is the only place vfio-pci appears.

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: kubevirt-policy
  namespace: nvidia-network-operator
spec:
  resourceName: kubevirt_sriov
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  numVfs: 8
  nicSelector:
    vendor: "15b3"
    pfNames:
      - ens1f0
  deviceType: vfio-pci
  isRdma: false

Note the API group: the CRDs are sriovnetwork.openshift.io/v1 even on vanilla Kubernetes, because that is the group the upstream SR-IOV Network Operator uses.^[1][4] The node selector is an NFD label, so a node whose NFD never ran matches nothing and the policy applies to zero nodes without error.^[1]

Two - the network. SriovNetwork is what turns the advertised resource into a NetworkAttachmentDefinition in the workload namespace, which is why you do not hand-write the NAD on this path.^[1][4]

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-kubevirt-net
  namespace: nvidia-network-operator
spec:
  resourceName: kubevirt_sriov
  networkNamespace: default
  spoofChk: "off"
  trust: "on"

networkNamespace: default is the namespace the generated NAD lands in - and on OpenShift, where Multus enforces namespace isolation by default, that value is the whole ballgame.^[5]

Three - the VM. The interface type is sriov and the network is a Multus reference to the generated NAD.^[1]

spec:
  domain:
    devices:
      interfaces:
        - name: sriov-net
          sriov: {}
  networks:
    - name: sriov-net
      multus:
        networkName: sriov-kubevirt-net

IP configuration happens inside the guest, through cloud-init or DHCP - there is no CNI IPAM step for a device the host no longer owns.^[1] KubeVirt support on Red Hat OpenShift is itself new in Network Operator 26.7.0, which is worth saying out loud to a customer who is quoting a 26.1 or 26.4 experience.^[7]

3IOMMU is a BIOS conversation before it is a Kubernetes one

The prerequisites are three, and only one of them is inside Kubernetes.^[1]

IOMMU must be enabled: intel_iommu=on iommu=pt on Intel hosts, amd_iommu=on iommu=pt on AMD.^[1] On a PowerEdge that is two settings and a kernel argument: the processor’s virtualisation option under System BIOS > Processor Settings, and “SR-IOV Global Enable” under System BIOS > Integrated Devices, which “Enables or disables the BIOS configuration of Single Root I/O Virtualization (SR-IOV) devices” and is disabled by default^[13] - plus intel_iommu=on iommu=pt or the AMD pair on the command line^[1], and customers routinely set one without the other. The vfio-pci kernel module must be available on the worker nodes.^[1] And mlx5_core must be inside the guest.^[1]

The reason IOMMU is non-negotiable rather than a tuning flag: a device a guest controls can issue DMA. Without an IOMMU translating and constraining those addresses, a guest that owns a NIC owns host memory. This is why the failure is a refusal to start rather than a degraded mode.

What you can check in five minutes on any node, before anyone touches a CRD:

cat /proc/cmdline                          # look for intel_iommu=on iommu=pt or the AMD pair
dmesg | grep -i -e DMAR -e IOMMU | head    # firmware and kernel agreeing that it is on
ls /sys/kernel/iommu_groups/ | wc -l       # zero groups means it is off, whatever the BIOS says
lsmod | grep vfio                          # vfio_pci present or loadable

The third command is the one that settles arguments. An empty /sys/kernel/iommu_groups/ is the ground truth regardless of what the BIOS screen claims, and the group listing is also what tells you whether a VF is isolated on its own group or shares one with functions you did not intend to hand over.

4DRA: asking for a VF by its properties

Dynamic Resource Allocation is the second path that is not a plain netdevice VF, and it is a different question from vfio-pci. vfio-pci changes who consumes the VF; DRA changes how you ask for one.

The device-plugin model can express exactly two things: a name and a count.^[8] Everything else - which NUMA node, which PCIe root, which PF - has to be inferred afterwards. TopologyInfo lets a plugin publish a NUMA hint, and Topology Manager then decides whether the resulting placement is acceptable, rejecting the pod with TopologyAffinityError if not.^[8][9] The request itself never said what it needed.

The DRA SR-IOV driver inverts that. It publishes each VF as a device in a ResourceSlice, carrying attributes including dra.net/numaNode, resource.kubernetes.io/pciBusID and resource.kubernetes.io/pcieRoot, plus vendor, PCI address, VF id, PF name and eswitch mode.^[2] A workload then allocates through a ResourceClaimTemplate that selects with a CEL expression:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: sriov-vf
spec:
  spec:
    devices:
      requests:
      - name: vf
        exactly:
          deviceClassName: sriovnetwork.k8snetworkplumbingwg.io
          count: 1
          selectors:
          - cel:
              expression: >
                device.attributes["k8s.cni.cncf.io"].resourceName == "nvidia.com/sriov_resource"

That expression matches the same resource name you have been chasing all module, but now as a device attribute rather than as a scheduling counter.^[2] The interesting part is what else is in that attribute namespace: because PCIe bus ID and NUMA node are attributes, a claim can in principle express NIC-to-GPU alignment as a selection criterion rather than leaving it to Topology Manager to approve or reject after the fact.^[2][9]

Two limits decide whether any of this is usable. The driver is “a Tech Preview feature: it has limited testing and is not recommended for production deployments”, and it “is supported only for Vanilla Kubernetes deployments with SR-IOV Network Operator”.^[2] It requires the dynamicResourceAllocation: true feature gate on the SR-IOV Network Operator.^[2] The page does not state a minimum Kubernetes version, so do not quote one - pin it against the operator’s own supported range of 1.32 to 1.36 instead and say that is what you are doing.^[2][6]

5Choosing between the three, out loud

Three questions decide the path, and they are worth memorising in this order.

Who consumes the VF? A container gets netDevice; a virtual machine gets vfio-pci.^[1][3] This is not a preference. It follows from whether the consumer has its own kernel.

Does the consumer need RDMA on the host side? If yes, vfio-pci is out - the flag is documented as incompatible and the example sets isRdma: false.^[1] A VM that needs RDMA does it inside the guest with the guest’s driver, and the host-side chained rdma plugin from lesson 3 has no role.

Is this cluster vanilla Kubernetes, and is Tech Preview acceptable? Both must be true for DRA.^[2] If either is false, the answer is the device-plugin path with Topology Manager, which is what the rest of this course teaches.^[8][9]

One certification note, phrased honestly. NCA-AIIO objective 3.4 is “Identify the key considerations for virtualizing accelerated infrastructure”, and this lesson over-serves it: the exam blueprint never names SR-IOV, VFIO, KubeVirt or DRA.^[12] Treat the mapping as “this content is more than sufficient for that objective”, not as exam coverage.