SR-IOV plumbing: device plugin, CNI and operator

1Three programs, three jobs

SR-IOV in Kubernetes is not one component. It is three, and every escalation gets easier once you can say which one owns the symptom.

The device plugin counts. “The SR-IOV Network Device Plugin is Kubernetes device plugin for discovering and advertising networking resources”, covering SR-IOV VFs, PCI PFs and auxiliary network devices.^[1] It turns hardware into a schedulable integer under a name it chooses.

The CNI attaches. “This plugin enables the configuration and usage of SR-IOV VF networks in containers and orchestrators like Kubernetes.”^[2] Crucially it does not choose a VF: “A metaplugin such as Multus gets the allocated VF’s deviceID (PCI address) and is responsible for invoking the SR-IOV CNI plugin with that deviceID.”^[2]

The operator configures the node. “The SR-IOV Network Operator simplifies the deployment and management of SR-IOV networking in Kubernetes and OpenShift clusters”, and it is what creates VFs and generates the network definitions rather than leaving you to hand-write them.^[4]

The sequence is fixed: advertise, schedule, delegate, attach. Multus only reads the k8s.v1.cni.cncf.io/networks annotation after the pod is placed, and the delegate only runs after Multus resolves the NetworkAttachmentDefinition.^[8] Everything downstream of scheduling is invisible while a pod is Pending - which is the whole reason the last segment of this lesson exists.

Without the operator the same path still works; you just write more YAML. NVIDIA’s older bare-metal Ethernet page shows exactly that shape: resource name nvidia.com/mlnx_sriov_netdevice with a hand-written NAD whose plugins array holds {"type":"sriov", ...}.^[9] Keep that example in mind as the “what the operator is doing for you” reference.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: nvidia.com/mlnx_sriov_netdevice
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "sriov-net",
      "type": "sriov",
      "vlan": 100,
      "spoofchk": "off",
      "trust": "on",
      "link_state": "enable",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.2.225/28",
        "exclude": [
          "192.168.2.229/30",
          "192.168.2.236/32"
        ]
      }
    }

2The device-plugin ConfigMap, field by field

The config is a resourceList of pools. Each pool has resourceName (required), resourcePrefix (optional), deviceType (optional), selectors (an object or a list) and additionalInfo (which adds environment variables).^[1]

resourcePrefix defaults to intel.com.^[1] That single default is why NVIDIA’s own configuration always sets it explicitly: the operator’s sriovDevicePlugin block uses resourcePrefix: nvidia.com with resourceName: hostdev and selectors: {vendors: ["15b3"], isRdma: true}, producing nvidia.com/hostdev.^[5]

deviceType takes netDevice (the default), accelerator or auxNetDevice.^[1] The netDevice selectors are vendors, devices, drivers, pciAddresses, acpiIndexes, pfNames, rootDevices, linkTypes, ddpProfiles, pKeys, isRdma, needVhostNet and vdpaType.^[1] An auxNetDevice pool takes a narrower set that adds auxTypes.^[1]

Two of those selectors carry most of the meaning for an NVIDIA fleet. vendors: ["15b3"] is the NVIDIA/Mellanox PCI vendor ID, and isRdma: true restricts the pool to devices that can carry RDMA.^[5]

What the pod sees is the other half of the contract. The plugin injects PCIDEVICE_<RESOURCE_NAME> with the comma-separated device IDs and PCIDEVICE_<RESOURCE_NAME>_INFO with JSON holding mount points and metadata; the resource name is upper-cased with . and / replaced by underscores.^[1] That is your in-pod proof of which VF you got.

{ "resourceList": [{
    "resourcePrefix": "nvidia.com",
    "resourceName": "hostdev",
    "selectors": { "vendors": ["15b3"], "isRdma": true }
}]}

3The SR-IOV CNI delegate and its ranges

The delegate config is small and almost entirely optional. name and type ("sriov") are required, ipam is optional, and deviceID is required - “A valid pci address of an SRIOV NIC’s VF”, for example "0000:03:02.3".^[3]

The optional fields are where a hand-written NAD goes wrong, so learn the ranges rather than the list:

One documented incompatibility is worth memorising because it produces a pod with an interface and no address: “DHCP IPAM plugin can not be used for VF bound to a dpdk driver (uio/vfio).”^[2] If the customer moved a VF to vfio-pci for a VM and left a DHCP IPAM block in the NAD, that is the bug, and lesson 5 covers why the VF was moved in the first place.

kubectl describe pod sriov-test | tail -n 5
Warning  FailedScheduling  …  0/3 nodes are available: 3 Insufficient intel.com/hostdev.

kubectl describe node dell-r760-01 | grep -A20 Allocatable
  nvidia.com/hostdev:  8      # intel.com/hostdev is not in this list

ls -l /var/lib/kubelet/device-plugins/
kubelet.sock   sriovdp.sock

# NicClusterPolicy .spec.sriovDevicePlugin.config
{ "resourceList": [{
    "resourcePrefix": "nvidia.com",
    "resourceName": "hostdev",
    "selectors": { "vendors": ["15b3"], "isRdma": true }
}]}

4The operator: four CRDs and one privileged namespace

The SR-IOV Network Operator exposes four primary CRDs, and each answers a different question.^[4]

• SriovNetworkNodePolicy - what hardware should this node group present? Its example fields are deviceType (for example netdevice), nicSelector.pfName, nodeSelector, numVfs and resourceName.^[4]
• SriovNetwork - what network do workloads attach to? It generates the NetworkAttachmentDefinition for you.^[4]
• SriovNetworkNodeState - what did the operator actually find? Read-only discovered hardware state.^[4]
• SriovOperatorConfig - global settings.^[4]

OVSNetwork and SriovIBNetwork exist alongside them for OVS-offloaded and InfiniBand attachments.^[4]

Install is a Helm chart from oci://ghcr.io/k8snetworkplumbingwg/sriov-network-operator-chart, and the namespace must carry pod-security.kubernetes.io/enforce=privileged.^[4] The repo states a base requirement of “Kubernetes 1.30+ or OpenShift 4.16+”.^[4] In an NVIDIA deployment you normally do not install it separately: the Network Operator chart enables it with --set sriovNetworkOperator.enabled=true, and you verify with kubectl -n nvidia-network-operator get pods.^[7]

Two platform-specific adjustments are documented. On OpenShift the config daemon must be kept out of the DOCA driver’s way with a configDaemonNodeSelector patch that includes network.nvidia.com/operator.mofed.wait: "false", and SR-IOV resources use the openshift.io prefix rather than a custom one.^[6] In Spectrum-X deployments the operator is passed disablePlugins: [mellanox] plus featureGates: {manageSoftwareBridges: true}, because otherwise its built-in mellanox plugin and the NIC Configuration Operator fight over the same firmware settings.^[11]

One honest gap: the repo’s landing page does not state the operator’s node-draining default, and it defers its feature-gate list to separate documents.^[4] In Network Operator deployments the Maintenance Operator is the documented drain path, so do not promise a customer standalone drain behaviour you have not read.

5Where numVfs really lives

numVfs appears in two different places and they are not the same knob. In SriovNetworkNodePolicy it is a request the operator applies to a node group.^[4] In the NIC Configuration Operator’s NicConfigurationTemplate it is a firmware setting, alongside linkType ("Ethernet" or "Infiniband"), pciPerformanceOptimized, roceOptimized and gpuDirectOptimized.^[10] The documentation is blunt about the cost: “A configuration reset triggers a node reboot. Ensure that workloads are drained or that the Maintenance Operator is configured to handle the node maintenance automatically.”^[10]

So when a customer says “we set numVfs: 8 and nothing happened”, the answer is a two-part check: what does SriovNetworkNodeState say was discovered, and has the firmware level actually been applied and rebooted.^[4][10]

{ "resourceList": [{
    "resourcePrefix": "nvidia.com",
    "resourceName": "sriov_netdevice",
    "deviceType": "netDevice",
    "selectors": { "vendors": ["15b3"], "pfNames": ["ens1f0"], "isRdma": true }
}]}

{"cniVersion":"0.3.1","name":"sriov-network","plugins":[
  {"type":"sriov","vlan":101,"ipam":{"type":"whereabouts","range":"192.168.2.225/28"}}]}

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-network
spec:
  containers:
  - name: test
    resources:
      requests: { nvidia.com/sriov_netdevice: "1" }
      limits:   { nvidia.com/sriov_netdevice: "1" }

{ "resourceList": [{
    "resourcePrefix": "________",
    "resourceName": "storage_vf",
    "selectors": { "vendors": ["____"], "pfNames": ["________"], "isRdma": ____ }
}]}

{"cniVersion":"0.3.1","name":"storage-net","plugins":[
  {"type":"________","vlan":____,"spoofchk":"____",
   "ipam":{"type":"whereabouts","range":"10.20.30.0/24"}}]}

    resources:
      requests: { ____________________: "1" }