RDMA in Kubernetes: the NVIDIA Network Operator

1What the operator is, and what it is made of

The NVIDIA Network Operator is the Kubernetes side of everything the previous four lessons configured by hand. The current documented release is v26.7.0, and the version index runs from v26.7.0 back through v26.4.1, v26.1.x, v25.10.0, v25.7.0 and earlier to v23.5.0.^[1][2] It is described as software for enabling AI networking in Kubernetes, and it is designed to work alongside the GPU Operator to give “high-throughput, low-latency networking for scale-out, GPU computing clusters”.^[1]

It installs a set of components rather than one thing: the containerized NVIDIA DOCA-OFED networking driver, the Kubernetes RDMA shared device plugin, the SR-IOV Network Device Plugin, Multus, the container networking plugins, Whereabouts, the NVIDIA IPAM plugin, NVIDIA NIC Feature Discovery and the Maintenance Operator.^[1] Between them they cover driver installation, resource advertisement, secondary interfaces and address management, which is exactly the list of things a bare-metal RDMA host solves with packages and static configuration.^[1]

The control surface is custom resources. The primary CRDs are NicClusterPolicy for cluster-wide NIC configuration and MacvlanNetwork, HostDeviceNetwork and IPoIBNetwork for the networks themselves; newer API groups add NIC configuration and firmware templates under configuration.net.nvidia.com and a Spectrum-X rail pool configuration under spectrumx.nvidia.com.^[1] One operational detail circulates widely and cannot be confirmed on the pages fetched: that the operator acts only on a NicClusterPolicy named nic-cluster-policy and ignores instances with other names. Treat that as unverified and simply use the documented name, which costs nothing.^[1]

This lesson is also the whole of NCP-AIN Domain 4, worth 5 percent of the exam, whose two objectives are deploying the Network Operator to manage RDMA interfaces in Kubernetes clusters and verifying its functionality.^[9] The study guide’s reading list for that domain still names Network Operator v25.7.0 while v26.7.0 ships, which is a useful thing to know before the exam and a useful thing to say to a customer comparing documents.^[9][2]

flowchart LR; A["NicClusterPolicy: rdmaSharedDevicePlugin"] --> B["Device plugin advertises rdma/rdma_shared_device_a"]; C["IPPool: subnet + perNodeBlockSize + gateway"] --> D["MacvlanNetwork: master = PF netdev - ipam nv-ipam"]; D --> E["Pod annotation k8s.v1.cni.cncf.io/networks"]; B --> F["Pod requests + limits rdma/rdma_shared_device_a: 1"]; E --> G["Pod: second interface with an address"]; F --> G; H["securityContext capabilities add IPC_LOCK"] --> G; G --> I["kubectl exec -- ibv_devinfo"]

2The shared-device path, object by object

NVIDIA’s quick start gives a working NicClusterPolicy for the shared path.^[3]

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  rdmaSharedDevicePlugin:
    image: k8s-rdma-shared-dev-plugin
    repository: nvcr.io/nvidia/mellanox
    version: network-operator-v25.7.0
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 63,
            "selectors": { "ifNames": ["ens1f0"] }
          }
        ]
      }

Two fields carry all the meaning. rdmaHcaMax: 63 is the sharing factor — the plugin README defines it as the “Maximum number of RDMA resources that can be provided by the device plugin resource”, and each pod requesting the resource consumes one unit, so 63 pods may share the HCA. The quick start prints the field without defining it.^[3][12] selectors.ifNames is the netdev, not the RDMA device name.^[3] That second point is a real operational hazard on a RoCE host: bonding into RoCE LAG changes the RDMA device name to mlx5_bond_0 while the selector keeps naming Ethernet interfaces, and any renaming scheme that moves ens1f0 silently stops the resource from being advertised.^[3][7]

The network side is a MacvlanNetwork whose master is the PF netdev, with mode: "bridge", an mtu, a networkNamespace, and an ipam block of {"type":"nv-ipam","poolName":"macvlan-pool"}; the pool itself is an IPPool in nv-ipam.nvidia.com/v1alpha1 with subnet, perNodeBlockSize and gateway.^[3]

The pod then has to ask for three things at once: the annotation k8s.v1.cni.cncf.io/networks: macvlan-network, the capability IPC_LOCK in securityContext.capabilities.add, and both requests and limits of rdma/rdma_shared_device_a: 1.^[3] Missing the annotation gives a pod with no second interface; missing IPC_LOCK gives a pod that has the device and cannot register memory; missing the resource gives a pod that may be scheduled where the device does not exist.^[3][5]

Verification is two commands, both documented: kubectl exec -it macvlan-test-pod -- ip addr show and kubectl exec -it macvlan-test-pod -- ibv_devinfo.^[3]

3Shared or SR-IOV, and what GPUDirect adds

The design question a customer asks is which path to use, and it decomposes cleanly.^[1][3]

The shared device plugin puts many pods on one physical function. Everything shares the PF’s queues, its GID table, and its QoS context, and the sharing factor is rdmaHcaMax. That is the right shape for storage clients and inference services, where the pods are numerous, individually modest, and not adversarial.^[3]

SR-IOV gives each pod a virtual function through the SR-IOV Network Device Plugin.^[1] A VF per pod is what buys isolation and a per-tenant QoS story, which is what a training job with a service-level expectation needs. It also inherits the bare-metal rule that NVIDIA states directly — “Setting the Trust State mode shall be done before enabling SR-IOV in order to propagate the Trust State to the VFs” — so a trust state set after the VFs exist does not reach them.^[11]

GPUDirect is a separate axis that the GPU Operator owns. Its helm surface is small: current releases enable DMA-BUF by default, --set driver.rdma.useHostMofed=true covers host-installed drivers, --set driver.rdma.enabled=true selects the legacy nvidia-peermem path, and --set gds.enabled=true adds GPUDirect Storage.^[4] The legacy path carries its bare-metal ordering trap into the cluster: if the GPU driver was installed before the network drivers, it must be uninstalled and installed again so nvidia-peermem builds against the right RDMA APIs.^[6] Inspecting the rollout is kubectl describe ds -n gpu-operator nvidia-driver-daemonset, where the mofed-validation init container should be present.^[4]

The cross-pod proof NVIDIA documents is a perftest pair: on the server kubectl exec -it demo-pod-1 -- ib_write_bw --use_cuda=0 --use_cuda_dmabuf -d mlx5_0 -a -F --report_gbits -q 1, with the client adding -n 5000 and the server pod’s IP.^[4]

4Kubernetes does not create a fabric

The single most useful sentence to say on a Dell AI Factory call: Kubernetes only plumbs a RoCE fabric that already works.^[10][7] The node still needs the same GID selection, trust state, PFC and ECN configuration as bare metal, and the switches still need their half of the policy — RoCE data on priority 3, CNP on priority 6, an ECN threshold, and PFC on every hop if the design is lossless.^[10]

So the failures split into two piles, and telling them apart is the analysis skill this lesson is for.^[3][5]

The practical consequence is a sequencing rule for any cluster bring-up: prove the fabric between two plain perftest pods before anyone schedules a training job.^[4] A failing all-reduce is an expensive way to discover that priority 3 was never trusted on the host.^[7]