Network Operator and the NicClusterPolicy

1What the chart installs, and what it does not

The NVIDIA Network Operator “leverages Kubernetes CRDs and Operator SDK to manage Networking related Components in order to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster.”^[2] Read that as a scope statement. It does not replace the primary CNI. Calico or Cilium keeps eth0, Services, DNS and NetworkPolicy; the operator builds the second path that NCCL traffic rides.^[2]

What it can deploy: the OFED/DOCA driver container, the RDMA Shared Device Plugin, the SR-IOV Network Device Plugin, InfiniBand Kubernetes, Multus-CNI, the container networking plugins, the IPoIB CNI plugin, NVIDIA Kubernetes IPAM, and the NVIDIA NIC Configuration Operator.^[2] The word “can” is load-bearing. The chart install alone gives you a controller and a set of CRDs. Nothing appears on a node until you apply a NicClusterPolicy.^[1]

And here is the rule that costs the most support time in this whole module: the operator acts only on a NicClusterPolicy named exactly nic-cluster-policy, and instances with any other name are silently ignored.^[2] No admission error, no event, no status. kubectl get nicclusterpolicy shows the object. It just never becomes anything.

flowchart LR; H["helm install network-operator"] --> O["operator controller + CRDs"]; O --> P["NicClusterPolicy named nic-cluster-policy"]; P --> D["doca-driver DaemonSet"]; P --> R["rdmaSharedDevicePlugin"]; P --> S["sriovDevicePlugin"]; P --> N["secondaryNetwork: multus + cniPlugins + ipoib"]; N --> C["MacvlanNetwork / HostDeviceNetwork / IPoIBNetwork"]; C --> A["generated NetworkAttachmentDefinition"]; A --> W["pod annotation + resource request"]

2Install and prerequisites

The install is three commands and the version pin matters, because the doc set and the chart are versioned together.^[1]

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install network-operator nvidia/network-operator \
  -n nvidia-network-operator \
  --create-namespace \
  --version v26.7.0 \
  --wait

Prerequisites the deployment guide names: kubectl and helm CLIs; Pod Security Admission enforcement set to privileged on the operator namespace; Node Feature Discovery present, or disabled with nfd.enabled=false if the cluster already runs it; and a container engine, CRI-O or containerd, on all nodes.^[1] Verify with kubectl -n nvidia-network-operator get pods.^[3]

Two optional switches change the shape of the deployment. --set sriovNetworkOperator.enabled=true brings in the SR-IOV Network Operator so the cluster can create and manage VFs and policies.^[3] Do not confuse that with the sriovDevicePlugin section of the policy, which only advertises VFs that already exist. On OpenShift the flow is different again: create the namespace with oc create namespace nvidia-network-operator, find the channel with oc get packagemanifest nvidia-network-operator -n openshift-marketplace -o jsonpath='{.status.defaultChannel}', then subscribe against source: certified-operators in openshift-marketplace.^[9]

For GPUDirect the guide states DOCA-OFED v3.1.0 or newer and GPU Operator v25.3.x or newer.^[1] Hold that number - lesson 2 shows it contradicting the GPU Operator’s own matrix, and both are official.

3The policy, section by section

NicClusterPolicy has seven spec sections: ofedDriver, rdmaSharedDevicePlugin, sriovDevicePlugin, ibKubernetes, secondaryNetwork, nvIpam and nicConfigurationOperator.^[2] You rarely want all seven. Pick by what the node must end up advertising.

ofedDriver names the driver container:

ofedDriver:
  image: doca-driver
  repository: nvcr.io/nvidia/mellanox
  version: doca3.5.0-26.07-0.7.7.0-0
  forcePrecompiled: false

^[1] That version string is the single most useful field on an escalation - it is what actually determines whether the RDMA stack the GPU side expects will bind. forcePrecompiled is a real field, shown as false in the guide.^[1] The page that documents precompiled image tag naming and when to prefer it did not resolve when this lesson was written, so treat the field as confirmed and the surrounding workflow as unverified until you fetch it for the customer’s version.

rdmaSharedDevicePlugin carries inline JSON in config: with configList[].resourceName, rdmaHcaMax and selectors.ifNames.^[1] That path advertises names like rdma/rdma_shared_device_a.^[1]

sriovDevicePlugin is where the naming trap lives:

sriovDevicePlugin:
  config: |
    {
      "resourceList": [{
        "resourcePrefix": "nvidia.com",
        "resourceName": "hostdev",
        "selectors": { "vendors": ["15b3"], "isRdma": true }
      }]
    }

^[1] Prefix plus name gives nvidia.com/hostdev, and that exact string has to appear in three places: the advertised allocatable on the node, the resourceName in the network CR, and the pod’s requests and limits.^[1] The older bare-metal guide used nvidia.com/mlnx_sriov_netdevice with a hand-written NAD, which is worth keeping as the before picture: {"cniVersion":"0.3.1","name":"sriov-network","plugins":[{"type":"sriov","ipam":{"type":"host-local","subnet":"10.56.217.0/24"}},{"type":"rdma"}]}.^[8]

secondaryNetwork has three sub-sections - cniPlugins (image plugins), multus (image multus-cni) and ipoib (image ipoib-cni) - all pulled from nvcr.io/nvidia/mellanox at tag network-operator-v26.7.0.^[1][5]

4Operator-owned network CRs, and the pods that use them

In the operator flow you do not hand-write the NAD. Three small CRs generate it.^[1]

MacvlanNetwork (apiVersion: mellanox.com/v1alpha1) takes networkNamespace, master such as ens2f0, mode: bridge, mtu, and ipam as an inline JSON string.^[1] HostDeviceNetwork takes networkNamespace, resourceName such as hostdev, and ipam - no master, because the device itself comes from the device plugin.^[1] IPoIBNetwork takes networkNamespace, master such as ibs1f0, and ipam.^[1] The documented exception is InfiniBand SR-IOV, where you still write the NAD by hand with "type":"ib-sriov", "pkey", "link_state" and a chained {"type":"rdma"} plugin.^[1]

The pod side is three lines and they are always the same shape. The annotation is k8s.v1.cni.cncf.io/networks naming the network, the container adds capability IPC_LOCK, and requests and limits carry the resource.^[1] For the RDMA shared path that resource is rdma/rdma_shared_device_a: 1; for host device it is nvidia.com/hostdev: 1.^[1] Multus resolves the annotation to a NAD in the pod namespace unless you namespace-qualify it as ns/name.^[7]

The GPUDirect variant is the host-device pod plus nvidia.com/gpu: 1 in both requests and limits, in the same container.^[1] That co-request is not cosmetic - it is the only reason both the NIC hint provider and the GPU hint provider vote, which is what makes NUMA alignment possible at all. Lessons 4 and 5 are built on it.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: host-device-net
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: nvidia.com/hostdev
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "host-device-net",
      "type": "host-device",
      "pciBusID": "0000:03:00.0",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.2.225/28",
        "exclude": [
          "192.168.2.229/30",
          "192.168.2.236/32"
        ]
      }
    }

5Reading status, and what nothing happened means

NicClusterPolicy exposes status.conditions[] with a per-component condition for each section you enabled and an aggregate Ready. The set includes OFEDDriverReady, RDMASharedDevicePluginReady, SRIOVDevicePluginReady, IBKubernetesReady, MultusCNIReady, CNIPluginsReady, IPoIBCNIReady, NVIPAMReady, NICFeatureDiscoveryReady, DOCATelemetryServiceReady, NICConfigurationOperatorReady, SpectrumXOperatorReady and Ready.^[4] Those names are your triage index: the first one that is not True names the stage that stalled.

So “we installed it and nothing happened” resolves in three checks, in this order. One: is the CR named nic-cluster-policy.^[2] Two: does status.conditions show OFEDDriverReady - a driver container that cannot compile against the running kernel is the most common stall.^[4][1] Three: does the advertised resource name match what the pod requests, prefix included.^[1] Only after those three does it become a fabric question.

Before you change anything on a customer cluster, collect the evidence. kubectl netop-sosreport --verbose --log-lines 10000 gathers the custom resources, the daemonset and deployment specs, current and previous container logs, node labels and allocatable resources, and runs lsmod, ibstat, ibv_devinfo, mst status and dmesg on the node.^[10] The previous-container logs are the part you destroy by restarting a pod to see if that fixes it.