GPUDirect RDMA: dma-buf, nvidia-peermem and GDS
S3·E4lsmod says nothing, so GPUDirect must be broken · Dell customer lab, the morning after the firmware window
Builds on: Network Operator and the NicClusterPolicy, Version matrices, upgrades and heterogeneous clusters
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish the DMA-BUF and legacy nvidia-peermem module paths by their documented prerequisites.
- Analyse a driver daemonset and a host inventory to decide which module path a cluster is actually on.
- Map the GPU Operator helm flags to who owns the OFED driver on that node.
- Explain why the pod must co-request GPU and NIC in one container for alignment to be possible at all.
Episode 4 — lsmod says nothing, so GPUDirect must be broken
By breakfast the 02:10 ticket has an owner, a severity and a theory. The customer’s ML engineer ran lsmod | grep nvidia on the GPU nodes, saw the driver modules and no nvidia_peermem line, read a post saying that module is what makes GPUDirect RDMA work, and concluded the firmware window broke it. Six days to the acceptance run. The network lead wants a counter. The Dell SE wants one sentence he can put in the spreadsheet.
Nothing is broken. There are two module paths, and the check only means something once you know which one this stack is on. The DMA-BUF path requires the Open Kernel module driver, CUDA 11.7 or higher and Linux kernel 5.12 or higher, and loads no extra module at all; the legacy path takes any supported driver and no minimum CUDA, but requires MLNX_OFED or DOCA-OFED to be present - and NVIDIA recommends DMA-BUF over the legacy module.[1] On a modern stack, an absent nvidia_peermem is the expected state.
The technology exists to keep a payload out of host memory entirely. GPUDirect RDMA is direct data exchange between GPUs and a third-party peer device over PCI Express, and in this cluster that peer is the NIC[1] - so a training step stops paying for a bounce through the CPU. Which mechanism grants that access is a driver-generation detail, and a day and a half just went into it.
Absence of a module is evidence only once you know which path the stack is on.
1A path, not a switch
GPUDirect RDMA “enables direct data exchange between GPUs and a third-party peer device using PCI Express” - in an AI cluster the peer is the NIC.[1] Its storage sibling, GPUDirect Storage, is “a direct data path between local or remote storage…and GPU memory” that does DMA transfers without the CPU bounce buffer.[1] Both are paths through the kernel that have to be enabled, matched to a driver, and then actually chosen at runtime by the library doing the transfer.
That is why “is GPUDirect RDMA on?” has no yes-or-no answer. Three separate things must all be true: the kernel has a peer-memory mechanism available, the GPU driver flavour and OFED ownership match the mechanism you picked, and the workload’s pod requested the NIC and the GPU in a way that lets the scheduler place them together.[1][2] Each is checked with a different command, and each has a failure that looks like the others.
The version floor sits underneath all three. Network Operator 26.7.0 states “NVIDIA GPU Operator v25.3.x or newer - required for workloads that use NVIDIA GPUs and GPUDirect RDMA”.[3] The GPU Operator’s own matrix approaches the same pairing from the other end and names specific Network Operator releases instead.[4] Neither page names the pair the other one names; that is a release-train artefact, not a mistake, and the working answer is to quote both.
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2Two module paths with mirror-image prerequisites
There are two kernel-module approaches, and they are almost exact opposites in what they demand.[1]
| DMA-BUF (recommended) | legacy nvidia-peermem |
|
|---|---|---|
| GPU driver | Open Kernel module required | any supported driver |
| CUDA | 11.7 or higher | no minimum |
| Linux kernel | 5.12 or higher | no minimum |
| MLNX_OFED / DOCA-OFED | optional | required |
NVIDIA states plainly that it recommends DMA-BUF over the legacy nvidia-peermem module from the GPU driver.[1] Read the table as a fork rather than a ranking: DMA-BUF is strict about the driver flavour and the kernel and relaxed about OFED; the legacy module is relaxed about the driver and strict about OFED. A customer who is pinned to the proprietary driver for an unrelated reason has already chosen the legacy path, and with it the requirement that MLNX_OFED or DOCA-OFED be present.[1]
The helm flags are how you tell the GPU Operator which world it is in. Where the Network Operator supplies the driver, GPUDirect RDMA with DMA-BUF is selected with --set driver.rdma.useHostMofed=false; where the host already has MOFED or DOCA installed, --set driver.rdma.useHostMofed=true; the legacy peermem module is --set driver.rdma.enabled=true; and GPUDirect Storage is --set gds.enabled=true.[1] A re-read of that page on 2026-09-09 adds two details worth carrying: GDS “is only supported with the NVIDIA Open GPU Kernel module driver” and requires nvidia-fs “version 2.17.5 or newer”, deployed as an nvidia-fs-ctr container inside the driver pods, and pre-R570 drivers can be pushed to the open flavour with --set driver.kernelModuleType=open.[1]
3Verifying without fooling yourself
The documented verification path is three commands, and each answers a different question.[1] kubectl describe ds -n gpu-operator nvidia-driver-daemonset tells you what the operator intended - which containers the daemonset carries and with what settings. kubectl logs -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-peermem-ctr reads the legacy module’s own container, which exists only on the legacy path. lsmod | grep nvidia tells you what is loaded right now on the node.[1]
The trap is in the third one. On a DMA-BUF stack nothing extra appears in lsmod, so the absence of nvidia_peermem is the expected state and is not evidence of failure.[1] An engineer who learned this stack in the peermem era will read a clean lsmod as a broken cluster and start reinstalling drivers on a node that was working.
The host-side inventory that resolves it is short: the driver flavour and version from /proc/driver/nvidia/version and nvidia-smi, the kernel from uname -r, the CUDA version, and whether an OFED stack is installed. Those four map directly onto the rows of the prerequisite table.[1] The Network Operator’s own collector runs lsmod, ibstat, ibv_devinfo, mst status and dmesg on the node, which is also a fair manual checklist for the NIC half of the same question.[9]
4The pod contract, and why it is a NUMA statement
Everything above is host and cluster setup. The workload still has to ask for the hardware in a way that lets the scheduler do the right thing. The Network Operator’s GPUDirect example is the host-device RDMA pod plus nvidia.com/gpu: 1 in both requests and limits, in the same container, with IPC_LOCK.[2] That single co-request is what makes NUMA alignment possible at all.[2]
The mechanism is in the device plugin API: TopologyInfo in a plugin’s response carries NUMA node information, which is what lets Topology Manager align anything.[6] Extended resources are integer-only and not overcommittable, and devices cannot be shared between containers - so a GPU and a NIC requested by two different containers are two independent allocations.[6] Topology Manager’s default scope is container, which aligns each container on its own with no grouping.[7] Two containers, two decisions, and nothing in the system objects when they land on opposite sockets.
The Network Operator GPUDirect example requests nvidia.com/hostdev: 1 and nvidia.com/gpu: 1 in the same container, with securityContext.capabilities.add: ["IPC_LOCK"]. That single co-request is what makes NUMA alignment possible at all.
Selecting a device explains it and lets you move it to the other socket — the physical change an FAE actually recommends. Devices are keyboard reachable: Tab, then Enter.
Set the policy, the placement and the request, then submit the pod to see the admission decision.
All resources must come from one NUMA node or the pod is rejected.
The strongest guarantee and the one that produces TopologyAffinityError most often. Pair it with topologyManagerScope: pod and a pod that requests GPU and NIC in the same container.
All containers in the pod land on a common set of NUMA nodes. Pod-scope resource math uses the effective-request formula: the max of the sum of the app containers, or the max of the init containers.
Exclusive CPUs for Guaranteed pods with integer CPU requests; BestEffort and Burstable pods stay in the shared pool. Only then does CPU become a Hint Provider and vote with the GPU and the NIC.
Both options sit behind the TopologyManagerPolicyOptions feature gate. prefer-closest-numa-nodes defaults to false and applies to best-effort and restricted only. ⚠ max-allowable-numa-nodes is recorded in the course notes as “default unlimited, caps how many NUMA nodes a pod's resources may span”; the upstream page could not be re-read to confirm whether it caps the pod span or the machine's NUMA count — verify before quoting it to a customer.
full-pcpus-only=true— GA 1.33+distribute-cpus-across-numa=true— Beta 1.33+align-by-socket=true— Alpha 1.25+distribute-cpus-across-cores=true— Alpha 1.31+strict-cpu-reservation=true— GA 1.35+prefer-align-cpus-by-uncorecache=true— GA 1.36+
Static-policy reservation precedence, first match wins: --reserved-cpus (explicit list, highest), then --kube-reserved, then --system-reserved. The reservation must be greater than zero.
Switching CPU Manager from none to static: drain the node, stop kubelet, rm /var/lib/kubelet/cpu_manager_state, update the config, start kubelet. Skipping the state-file delete is a classic silent failure — the new policy simply does not take effect.
# /var/lib/kubelet/config.yaml topologyManagerPolicy: "single-numa-node" topologyManagerScope: "pod" cpuManagerPolicy: "static"
spec:
containers:
- name: trainer
securityContext:
capabilities:
add: ["IPC_LOCK"]
resources:
limits: # requests must match limits → Guaranteed QoS
nvidia.com/gpu: "1"
nvidia.com/hostdev: "1"
cpu: "8"On a Spectrum-X rail deployment the NIC side is requested as nvidia.com/rail0 / nvidia.com/rail1 instead of nvidia.com/hostdev; on the RDMA shared plugin it is rdma/rdma_shared_device_a. Same alignment question, different resource name.
“The job runs at half speed” on a dual-socket PowerEdge is usually a NIC on socket 0 feeding a GPU on socket 1. The Kubernetes-side fix is single-numa-node + scope: pod + CPU Manager static, with nvidia.com/gpu and nvidia.com/hostdev in the same container so both hint providers vote. Then the failure inverts into TopologyAffinityError — and that rejection is the system working.
Map the box before you argue about policy: lstopo, numactl --hardware, nvidia-smi topo -m, cat /sys/class/net/<pf>/device/numa_node.
Documented limitation: a hard limit of 8 NUMA nodes per system; systems with more are unsupported, with no workaround. Check it before recommending an NPS4 BIOS setting on a high-core-count AMD node — dual socket × NPS4 is already 8.
Dell publishes the 2-8-9-400 configuration on PowerEdge XE9680: up to 12 worker nodes, "each equipped with two high-performance processors and eight NVIDIA H200 SXM GPUs". Two sockets, eight GPUs — that is the board this planner draws.
Topology Manager aligns pods of all QoS classes, but only for resources whose Hint Providers actually supply topology hints. Topology Manager requires Kubernetes v1.18 or later.
GPUDirect RDMA moves data NIC↔GPU without touching host memory, which is exactly why the NIC and the GPU want to hang off the same NUMA node. Verify the module path with lsmod | grep nvidia and the transport with NCCL_DEBUG=INFO.
⚠ The per-node NIC count and the 24 allocatable CPUs per NUMA node in this diagram are illustrative: the Dell brief confirms two sockets and eight H200 SXM GPUs, but the “9 NICs” reading of 2-8-9-400 is unverified.
5Measuring it: NCCL as the instrument
Configuration says what should happen. NCCL says what did. NCCL_DEBUG prints the version at VERSION, warnings at WARN, and debug information at INFO, and is not set by default; NCCL_DEBUG_SUBSYS defaults to INIT,BOOTSTRAP,ENV and takes a comma-separated subsystem list where a ^ prefix excludes.[5] An INFO run names the NIC chosen per rank and the transport in use.
Two more knobs turn that into a decision. NCCL_NET_GDR_LEVEL controls GPUDirect RDMA use between NIC and GPU, taking the topological cutoffs LOC, PIX, PXB, PHB and SYS, and is auto-selected by default.[5] NCCL_IB_DISABLE set to 1 disables InfiniBand verbs and falls back to IP sockets - the fastest A/B test in the stack, because if the job’s throughput does not change when you disable RDMA, RDMA was never being used.[5]
Two more are worth carrying for RoCE clusters even though they are not GPUDirect settings: NCCL_IB_GID_INDEX defaults to -1 and defines the Global ID index used in RoCE mode, and NCCL_IB_TC defaults to 0 and is the traffic-class - the same lever a rail’s rdmaQoS sets at the CNI level, which is why the two must not contradict each other.[5]
A customer sends a node inventory and asks why GPUDirect RDMA “is not enabled”.
$ uname -r
5.15.0-118-generic
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 570.86.15
$ lsmod | grep nvidia
nvidia_uvm 1560576 0
nvidia_drm 94208 0
nvidia_modeset 1355776 1 nvidia_drm
nvidia 8560640 3 nvidia_uvm,nvidia_modeset
$ ofed_info -s
(command not found)GPU Operator was installed with --set driver.rdma.useHostMofed=true.
- Which path is this? The driver identifies itself as the Open Kernel Module and the kernel is 5.15, so both DMA-BUF prerequisites that concern the host are met.[1] Confirm CUDA is 11.7 or higher before you commit to that reading.[1]
- Is the missing
nvidia_peermema fault? No. On DMA-BUF nothing extra is loaded, so thislsmodis exactly what a healthy DMA-BUF node looks like.[1] - Where is the real problem?
ofed_infois not found, so no OFED stack is installed on the host - yet the operator was tolduseHostMofed=true, meaning “the host owns the OFED driver”.[1] That is the mismatch: the setting claims a driver that does not exist, so nothing supplies the NIC-side stack the Network Operator would otherwise have provided. - What is the fix? Either let the Network Operator supply DOCA-OFED and set
--set driver.rdma.useHostMofed=false, or install the OFED stack on the host and keeptrue.[1] Pick the first on a cluster that already runs the Network Operator, because it keeps the driver tag versioned insideNicClusterPolicyrather than in a host build.[2] - How do you prove it afterwards?
kubectl describe ds -n gpu-operator nvidia-driver-daemonsetfor intent, then one job withNCCL_DEBUG=INFOand one withNCCL_IB_DISABLE=1.[1][5] Different throughput between the two runs is your evidence that the fabric is now in the path.
Second node, same customer. Fill the blanks.
$ uname -r
4.18.0-553.el8_10.x86_64
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.183.06
$ lsmod | grep nvidia
nvidia_peermem 16384 0
nvidia 8560640 3
$ ofed_info -s
MLNX_OFED_LINUX-24.10-1.1.4.0- The driver flavour here is ____ rather than the Open Kernel module, which rules out the ____ path.[1]
- The kernel is 4.18, which fails the ____ minimum of ____ for that same path.[1]
- Therefore this node is on the ____ path, and the presence of
nvidia_peermemis ____ rather than a warning sign.[1] - The helm flag that matches this node is ____ and the OFED requirement for it is ____[1]
- If the customer wants to move this node to DMA-BUF, the two host-side changes are ____ and ____[1]
- The pod-side contract does not change: request ____ and ____ in the same container plus the ____ capability.[2]
A Dell AI Factory customer has twelve nodes. Eight report nvidia_peermem loaded with MLNX_OFED installed on the host; four report no nvidia_peermem, the Open Kernel Module driver, kernel 5.15 and no OFED on the host. All twelve run the same GPU Operator release installed once with --set driver.rdma.useHostMofed=true, and the Network Operator is deployed cluster-wide with a NicClusterPolicy that includes ofedDriver. Training jobs are slow on four nodes and fine on eight.
Produce: (a) which module path each group is on and the evidence line that decides it; (b) which group the single useHostMofed value is wrong for and what that produces; (c) the two configuration answers available - one that changes the four nodes and one that changes the cluster setting - with the trade-off of each; (d) the measurement you would ask for to confirm the diagnosis before changing anything; (e) the one sentence you would put in the case notes about lsmod so the next engineer does not undo your fix.
Acceptance criteria: your evidence line for the four-node group is the driver flavour and kernel rather than the missing module; you name the useHostMofed mismatch as claiming a driver that is not installed; your measurement is an NCCL_DEBUG=INFO run plus an NCCL_IB_DISABLE=1 A/B on one node from each group; and your case note says absence of nvidia_peermem is expected on a DMA-BUF stack.[1][2][5]
One flag, one A/B
Twenty minutes of capture settles it: Open Kernel module driver, a modern kernel, no host OFED - the DMA-BUF row, so no module was ever going to appear in lsmod.[1] The real finding is one line away. The GPU Operator was installed with driver.rdma.useHostMofed=true, claiming a host OFED stack nobody installed.[1] One helm flag, one A/B pair of runs with and without NCCL_IB_DISABLE=1 to prove the fabric is in the path[5], and the ticket closes.
Then the first full training run reports back. Six nodes finish. Six finish forty minutes later. Same image, same container, same pod spec.
Lab
Goal: classify the real stack on a GPU-equipped Dell node with a ConnectX NIC, then read what NCCL chose. Read-only: every command below observes. Nothing here changes a driver, a firmware setting or a node.
Pre-flight inventory - capture all of it before you interpret any of it:
uname -r
cat /proc/driver/nvidia/version
nvidia-smi
nvcc --version 2>/dev/null || cat /usr/local/cuda/version.json 2>/dev/null
lsmod | grep nvidia
ofed_info -s 2>/dev/null || echo "no host OFED"
mst status
ibv_devinfoExpected: a driver line that either says Open Kernel Module or does not, a kernel version, a CUDA version, and a clear answer on host OFED. If mst status or ibv_devinfo finds nothing, stop and fix the NIC half first - the GPU half cannot be assessed against a card the host cannot see.[9]
- Classify the node against the prerequisite table using only the capture. Write down the path and the one line that decided it.[1]
- Check the operator’s intent:
kubectl describe ds -n gpu-operator nvidia-driver-daemonsetand, if it exists,kubectl logs -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-peermem-ctr.[1] Expected: the containers present agree with the path you classified. A disagreement here is the finding. - Confirm the NIC resource is advertised:
kubectl describe node <node> | grep -A20 Allocatable.[6] Expected: the resource your pod will request appears with a non-zero count. - Run a pod that co-requests both devices -
nvidia.com/hostdev: 1andnvidia.com/gpu: 1in the same container, withIPC_LOCK, and the network annotation for your host-device network.[2] Expected: the pod reaches Running. If it stays Pending, read the scheduler event verbatim - a resource-name mismatch and a topology rejection look nothing alike, and one of them namesTopologyAffinityError.[7] - Read the transport: run a small NCCL workload inside the pod with
NCCL_DEBUG=INFOand record the lines naming the NIC per rank and whether GDR is in use.[5] - A/B it: repeat the identical run with
NCCL_IB_DISABLE=1.[5] Expected: measurably worse throughput. If it is unchanged, RDMA was not in the path in run 5 either, and your finding is in the pod plumbing rather than the fabric. - Record
NCCL_NET_GDR_LEVELas reported and note that it is auto-selected by default - do not pin it as an experiment on a shared node.[5] - Deliverable: a one-page record - path, evidence line, operator intent, whether they agree, and the two NCCL numbers. Keep it; it is the template for the next GPUDirect ticket. Nothing here changed a driver, a firmware setting or kubelet configuration. Do delete the test pod when you are done -
kubectl delete pod <pod>- it holds a NIC and a GPU exclusively until it goes, and extended resources are integer and non-shareable, so a forgotten pod quietly removes an alignable pair from the node.[6]
Goal: become able to classify a stack from evidence alone, without a GPU. Read-only; nothing here changes a cluster.
- Write the prerequisite table from memory, both columns, then check it against the source page.[1] Expected: four rows - driver flavour, CUDA, kernel, OFED - and the two columns should be mirror images. If you wrote the same requirement in both columns, you have merged the paths, which is the error this lesson exists to prevent.
- Take the two captured inventories from the worked example and the faded block and classify each one before reading the answers. For each, write the single line that decides it. Expected: driver flavour plus kernel for the first, driver flavour alone for the second.
- Read a driver-daemonset description. Use
kubectl describe ds -n gpu-operator nvidia-driver-daemonsetoutput from any cluster you have, or the structure of it if you have none, and list which containers would exist on each path.[1] Expected: annvidia-peermem-ctrcontainer on the legacy path and no such container on DMA-BUF. If not: you are looking at a GPU Operator release older than the one your notes describe - record what you actually saw. - Build the decision tree as a one-page customer answer. Top question: which driver flavour. Second: kernel and CUDA. Third: who owns OFED. Leaf: which helm flag, and what to expect from
lsmod.[1] Expected: a page a Dell SE could use without you on the call. - Add the measurement section. Name the two NCCL runs you would ask for and what each result means, including the case where both runs are identical.[5] Expected: identical throughput reads as RDMA never engaged, not as a fabric fault.
- Add the version paragraph. State the Network Operator requirement for GPU Operator and note that the GPU Operator matrix is anchored from the other end, so both get quoted.[3][4] Expected: two sentences, no claim that one page overrides the other.
- Deliverable: the one-page answer plus a three-line summary of what you would ask a customer for in the first reply to a GPUDirect ticket.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE in four sentences why is GPUDirect RDMA on is not a yes or no question, and what you would ask for instead.
Sources
Facts in this lesson were checked against Re-fetched 2026-09-09: GPU Operator gpu-operator-rdma (DMA-BUF vs nvidia-peermem prerequisite table, helm flags, GDS notes) and Network Operator 26.7.0 platform-support. Dates are when each page was fetched.
- NVIDIA GPU Operator - GPUDirect RDMA and GPUDirect Storage · fetched 2026-09-09
- NVIDIA Network Operator v26.7.0 - Deployment Guide with Kubernetes · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Platform Support · fetched 2026-09-09
- NVIDIA GPU Operator - Platform Support · fetched 2026-09-07
- NCCL User Guide - Environment Variables · fetched 2026-09-07
- Kubernetes - Device Plugins · fetched 2026-09-07
- Kubernetes - Control Topology Management Policies on a node · fetched 2026-09-09
- NVIDIA GPU Operator - Overview · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - SOS-Report Collection Script · fetched 2026-09-07
- Dell AI Factory with NVIDIA - NVIDIA 2-8-9-400 configuration (Solution Brief) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- GPUDirect RDMA: proving it is actually onRoCE course · Same ground: nvidia-peermem, gpudirect and buf
- Proving the fabric: perftest and nccl-testsInfiniBand course · Same ground: gpudirect, numa and NCCL_IB_TC
- RDMA in Kubernetes: the NVIDIA Network OperatorRoCE course · Same ground: gpudirect, pod and versions