RDMA in pods: shared plugin vs exclusive netns

1A fork in the road, not a setting

There are two supported ways to get RDMA inside a pod, and a node can only be in one of them. This is the single most useful thing in the module, so state it the way you would to a customer: it is a design fork, not a bug.

Shared. k8s-rdma-shared-dev-plugin “provisions RDMA (InfiniBand and RoCE) HCA resources” and runs as a DaemonSet.^[1] Every pod that lands on the node sees the same HCA. The kernel is in shared netns mode, where “RDMA device is accessible in all network namespaces”.^[4]

Exclusive. rdma-cni is a “CNI compliant plugin for network namespace aware RDMA interfaces” that provides “network namespace isolation for RDMA workloads in a containerized environment”.^[3] It is chained after an SR-IOV delegate and moves “the associated RDMA interfaces of the provided network interface to the container’s network namespace path”.^[3] The kernel is in exclusive mode, where the manual page says exclusive “should be set before creating any network namespace” so that a dedicated RDMA device can be assigned to a particular namespace.^[4]

The two are incompatible because they need opposite kernel states. Shared needs devices visible everywhere; exclusive needs them visible in exactly one place.^[4] Deploy the shared plugin on an exclusive node and its pod rdma-shared-dp-ds sits in CrashLoopBackOff.^[2]

Choosing is a workload question. Several small jobs that trust each other on one card, or an IPoIB/macvlan secondary path? Shared. Multi-tenant, one VF per pod, per-pod isolation and Spectrum-X style rails? Exclusive.^[11]

rdma system show
rdma system set netns exclusive
rdma system set netns shared
echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf

2Shared: a counter with a required number

The shared config is one JSON document with a configList of pools:

{"periodicUpdateInterval": 300,
 "configList": [{
    "resourceName": "hca_shared_devices_a",
    "resourcePrefix": "example_prefix",
    "rdmaHcaMax": 1000,
    "devices": ["ib0","ib1"]
 }]}

rdmaHcaMax is the “maximum number of RDMA resources that can be provided by the device plugin resource”; it is required and has no default.^[1] resourcePrefix defaults to rdma, which is where the familiar rdma/rdma_shared_device_a comes from.^[1][5] periodicUpdateInterval defaults to 60 seconds if unset and is disabled at 0.^[1]

Selectors are vendors (hex, such as "15b3"), deviceIDs (hex), drivers ("mlx5_core"), ifNames ("ib0") and linkTypes ("ether").^[1] Matching is “logical OR between elements of a specific selector and logical AND … between selectors”, and unspecified selectors are ignored.^[1] That last clause is why a config carrying only ifNames can capture more devices than the author expected.

In the Network Operator flow this is a block of NicClusterPolicy, with rdmaHcaMax: 63 and selectors: {ifNames: ["ens1f0"]} producing rdma/rdma_shared_device_a.^[5] The pod then requests rdma/rdma_shared_device_a: 1 in both requests and limits and adds IPC_LOCK.^[5]

Now the honesty note. The README does not document how devices are exposed to the container - whether /dev/infiniband is mounted, and whether rdmaHcaMax maps to an rdma cgroup limit. Treat the internal mechanism as unverified: confirm it on the node rather than describing it to a customer.^[1]

Because extended resources are integer and non-shareable, rdmaHcaMax is the only lever you have, and what it limits is how many pods land here, not what any of them may do.^[9]

3Exclusive: chain rdma after sriov

Exclusive mode is set one of two ways. Persistently, echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf and reboot.^[3] At runtime, rdma system set netns exclusive.^[4]

The ordering rule is the part people learn the hard way: “If there are active network namespaces and if one or more RDMA devices exist, changing mode from shared to exclusive returns error code EBUSY.”^[4] The manual page also advises against changing mode while RDMA traffic is running, even though it is supported.^[4] On a node that has already joined a cluster, that makes the modprobe-plus-reboot path the realistic one.

Prerequisites are explicit: kernel 5.3.0 or newer (or an out-of-tree driver from MLNX_OFED 4.7+), an iproute2 built against 5.3.0 or newer, Multus CNI v3.4.1 or newer, SR-IOV capable hardware and an SR-IOV supporting CNI; Kubernetes 1.16+ for the DaemonSet deployment.^[3]

The CNI config itself is trivially small:

{"cniVersion":"0.3.1","type":"rdma","args":{"cni":{"debug":true}}}

In a NAD it is simply the second plugin in the array:

{"cniVersion":"0.3.1","name":"sriov-network","plugins":[
  {"type":"sriov","ipam":{"type":"host-local","subnet":"10.56.217.0/24"}},
  {"type":"rdma"}]}

Order is not stylistic. rdma moves the RDMA interfaces associated with a network interface that already exists, so the delegate that creates that interface has to run first.^[3] The InfiniBand variant is the same shape with ib-sriov and a PKey: {"type":"ib-sriov","pkey":"0x6","link_state":"enable","ibKubernetesEnabled":true,...} followed by {"type":"rdma"}.^[5]

One documented limitation to carry into a design review: “RDMA workloads utilizing RDMA Connection Manager on Mellanox hardware require pre-allocated MAC addresses for all VFs due to kernel constraints.”^[3]

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: nvidia.com/mlnx_sriov_netdevice
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "sriov-net",
      "plugins": [
        {
          "type": "sriov",
          "vlan": 100,
          "spoofchk": "off",
          "trust": "on",
          "link_state": "enable",
          "ipam": {
            "type": "nv-ipam",
            "poolName": "pool1",
            "poolType": "ippool"
          }
        },
        {
          "type": "rdma"
        }
      ]
    }

4The collision, in the customer's words

Two error signatures resolve most of this lesson’s field cases in under ten minutes.

Signature one - wrong pairing. The shared plugin started on a node whose RDMA subsystem is exclusive. The pod rdma-shared-dp-ds is in CrashLoopBackOff and the log line is verbatim:

Exiting.. can not change : incorrect RDMA subsystem network namespace

The issue has been open since 2023 with no maintainer resolution captured on the page, so present it as an observed incompatibility rather than as vendor guidance.^[2]

Signature two - too late. Someone tried to fix signature one by flipping the mode on a running node and got EBUSY from rdma system set netns exclusive.^[4] Active namespaces plus RDMA devices means the window has closed; the fix is the modprobe line and a reboot.^[3]

The inverse case appears in NVIDIA’s own Spectrum-X troubleshooting: the symptom “RDMA visibility across Pods” is fixed by enabling RDMA subsystem namespace awareness with options ib_core netns_mode=0 and a reboot.^[6] Read those two pages together and the picture is consistent: NVIDIA’s isolated designs assume exclusive, the shared plugin assumes shared, and neither document is wrong.

Spectrum-X shows what exclusive looks like at scale. Each rail gets the RDMA CNI with rdmaQoS: {tos: 96, tc: 96}, and the pod’s RDMA device is named rdma_<rail topology name>.^[11]

5Choosing, and defending the choice