GPUDirect RDMA: proving it is actually on

1Distance decides whether GPUDirect can happen at all

GPUDirect RDMA is “a technology introduced in Kepler-class GPUs and CUDA 5.0 that enables a direct path for data exchange between the GPU and a third-party peer device using standard features of PCI Express”.^[2] The hard topology rule is one sentence: “The two devices must share the same upstream PCI Express root complex.”^[2] NVIDIA describes three cases — PCIe switches only (optimal), a single CPU or IOH (works but degraded), and CPU/IOH connected via QPI/HT, which “may be extremely limited or unreliable”.^[2]

You read that off one command. nvidia-smi topo -m “Displays a matrix of connections between all GPUs and NICs (including their data-direct devices if applicable) in the system along with CPU/memory affinities for the GPUs”.^[4] The legend, verbatim: X is Self; PIX is “Connection traversing a single PCIe switch”; PXB is “Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)”; PHB is “Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)”; NODE is “Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node”; SYS is “Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)”; NV# is “Connection traversing a bonded set of # NVLinks”.^[4]

That is not just a diagnostic vocabulary — it is the same vocabulary NCCL_NET_GDR_LEVEL uses. The variable “Defines maximum distance between NIC and GPU” for GDR to be used, accepting LOC, PIX, PXB, PHB, SYS or the legacy integers 0–4, with the default auto-selected by architecture.^[9] So the matrix and the environment variable are read together: the matrix tells you the distance, the variable tells you the threshold NCCL will accept.

On Dell hardware the distance is a slot decision. The XE9680’s dedicated NIC/SmartNIC/DPU slots are 31 on Processor 2 and 40 on Processor 1, while slots 32/33 and 34/35 hang off Processor 2 and 36/37 and 38/39 off Processor 1.^[11] A customer who filled 31 and 40 first with ConnectX-7s has spread two cards across two sockets before any software was installed.

flowchart TD; A["nvidia-smi topo -m: NIC to GPU distance"] -->|SYS or NODE| Z1["Move the card or repair the pairing. Same root complex is a hard requirement."]; A -->|PIX PXB or PHB| B["Peer-memory path present? lsmod plus driver install order"]; B -->|neither DMA-BUF nor peermem| Z2["Install path wrong. GPU driver before OFED means reinstall the GPU driver."]; B -->|present| C["ulimit -l unlimited, or IPC_LOCK in the pod?"]; C -->|limited| Z3["ibv_create_qp or ibv_reg_mr will fail. Fix limits.conf then re-login."]; C -->|unlimited| D["IOMMU 1:1 or off? /proc/cmdline and dmesg"]; D -->|Translated| Z4["Non 1:1 translation is incompatible with GPUDirect RDMA."]; D -->|off or passthrough| E["PCIe ACS on the bridges? lspci -vvv ACSCtl"]; E -->|enabled on bare metal| Z5["Disable with setpci, or enable ATS if this is a VM."]; E -->|clear| F["Prove it: ib_write_bw host memory versus --use_cuda --use_cuda_dmabuf"]

2Two paths to the same thing: DMA-BUF and nvidia-peermem

There are two mechanisms and NVIDIA’s GPU Operator documentation states their requirements as a table.^[1] For DMA-BUF: “An Open Kernel module driver is required”, “CUDA 11.7 or higher”, “Turing architecture data center, Quadro RTX, and RTX GPU or higher”, “MLNX_OFED or DOCA-OFED are optional”, and Linux kernel “5.12 or higher”.^[1] For the legacy nvidia-peermem: “Any supported driver”, “No minimum version” of CUDA, “All data center, Quadro RTX, and RTX GPU or higher”, “MLNX_OFED or DOCA-OFED are required”, and no minimum kernel.^[1] NVIDIA’s recommendation is explicit: “NVIDIA recommends using the DMA-BUF rather than using the nvidia-peermem kernel module from the GPU Driver.”^[1]

The peer-memory module itself “provides direct peer-to-peer read and write access to the NVIDIA GPU’s video memory” for NVIDIA InfiniBand-based HCAs, so RDMA applications avoid a bounce through host memory.^[5] It is “a drop-in replacement for nv_peer_mem”, and only one of the two may be loaded at a time; the documented migration order is service nv_peer_mem stop, lsmod | grep nv_peer_mem, rmmod nv_peer_mem, then uninstall the package.^[5][6]

Two traps live here. First, ordering: “If the NVIDIA GPU driver is installed before MOFED, the GPU driver must be uninstalled and installed again” so that nvidia-peermem compiles against MLNX_OFED’s RDMA APIs.^[5] On a freshly imaged Dell node this is the single most common reason GPUDirect is silently unused. Second, persistence: it is loaded with modprobe nvidia-peermem, and “there is no service to automatically load nvidia-peermem.ko”.^[5] A /etc/modules-load.d/ entry is the usual workaround, but the NVIDIA README does not document it — say so rather than presenting it as vendor guidance.

Verification is lsmod | grep nvidia, where the expected modules include nvidia_fs and nvidia_peermem.^[1] Do not confuse gdrcopy with GPUDirect RDMA: gdrcopy is a separate library checked with lsmod | grep gdrdrv.^[6] And on a modern DMA-BUF system there may be no module to find at all — NCCL “auto-detects and enables DMA-BUF support” so “modern systems using DMA-BUF with open-source drivers don’t require this module”.^[3]

3The three silent killers on a healthy-looking host

Memory lock. The symptoms are NCCL WARN Call to ibv_create_qp failed and NCCL WARN Call to ibv_reg_mr failed.^[8] The fix is * soft memlock unlimited and * hard memlock unlimited in /etc/security/limits.conf; “Users must log out/reboot or update Slurm job launch settings. Verify with ulimit -l.”^[8] In Kubernetes the equivalent is securityContext.capabilities.add: ["IPC_LOCK"], which appears in both of NVIDIA’s sample pods.^[10][1]

IOMMU. The constraint is verbatim: “GPUDirect RDMA currently relies upon all physical addresses being the same from the different PCI devices’ point of view. This makes it incompatible with IOMMUs performing any form of translation other than 1:1.”^[2] NCCL’s own troubleshooting is blunter, quoting the CUDA Programming Guide that “the IOMMU must be disabled on Linux bare-metal systems to prevent silent device memory corruption”.^[3] Check with cat /proc/cmdline and dmesg | grep -i -E "iommu|dmar|default domain"; the concerning strings are Default domain type: Translated and DMAR: IOMMU enabled.^[3]

PCIe ACS. Access Control Services forces peer transactions up to the root complex, which is the opposite of what GPUDirect wants. Detect with sudo lspci -vvv | grep ACSCtl; disable one device with sudo setpci -s 03:00.0 ECAP_ACS+0x6.w=0000; NVIDIA publishes a loop for every bridge in the box.^[3]

for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do
  sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
  if [ $? -ne 0 ]; then continue; fi
  sudo setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done

For virtual machines the answer is different: “For virtual machines, ACS cannot be disabled; instead, enable ATS in network adapters for optimal performance.”^[3] One more container-and-VM rule from the same page: ensure /sys is properly mounted, because that is what provides PCI topology discovery, and a virtual PCI topology can degrade performance significantly.^[3]

4The proof: the host-memory versus GPU-memory A/B

Everything above is a prerequisite. The only thing that proves GPUDirect is in use is a pair of measurements. NVIDIA’s own GPU Operator documentation runs it as a server line ib_write_bw --use_cuda=0 --use_cuda_dmabuf -d mlx5_0 -a -F --report_gbits -q 1 with the client adding -n 5000 and the server address.^[1] Run the identical line without the CUDA flags to get the host-memory baseline: if the GPU number approximately equals the host number, GPUDirect is on; if it is a fraction of it, the transfer is staging through host memory.^[1][7]

perftest’s CUDA support has moved recently. “As of perftest release 25.07 the build system automatically detects the location of cuda.h. Passing CUDA_H_PATH to the configure script is therefore no longer required… The variable will depracted in the 25.10 release.”^[7] DMA-BUF has its own build requirements, quoted verbatim: “1) CUDA Toolkit 11.7 or later. 2) NVIDIA Open-Source GPU Kernel Modules version 515 or later.”^[7] Usage is ib_send_bw -d mlx5_0 --use_cuda=0 --use_cuda_dmabuf.^[7]

Two flags make the run reproducible on a rail-optimised box. --use_cuda_bus_id=<cuda full BUS id> selects the GPU by full PCIe address instead of by index, which matters because CUDA ordinals move; --cuda_mem_type=<value> chooses 0(device,default), 1(managed) or 4(malloc).^[7]

And the known issue worth memorising: “If GPUDirect is not working, (e.g. you see Couldn't allocate MR error message), consider disabling Scatter to CQE feature. Set the environmental variable MLX5_SCATTER_TO_CQE=0”, as in MLX5_SCATTER_TO_CQE=0 ./ib_write_bw -d ib_dev --use_cuda=<gpu index> -a.^[7]