Proving the fabric: perftest and nccl-tests

1Nothing is measured until the fabric is up

Benchmarks on InfiniBand have a precondition that Ethernet people forget: “When running the benchmarks over an Infiniband fabric, a Subnet Manager must run on the switch or on one of the nodes in your fabric, prior to starting the benchmarks.”^[1] NVIDIA’s own NCCL troubleshooting page opens with the same check - run sudo sminfo, and if it fails “start the opensm service on at least one node”.^[2] A host SM confirms success by writing SUBNET UP into both /var/log/messages and /var/log/opensm.log.^[5]

The second precondition is the port itself. Read ibstat, which “displays basic information obtained from the local IB driver” including LID, SMLID, port state, link width active and port physical state.^[6] NVIDIA tells you exactly which four fields to read before benchmarking: port state is Active and not Down, Init or Armed; physical state is LinkUp; the link layer matches the fabric; and the rate “matches your fabric specification without unexpected downshift”.^[2] A port stuck in Initializing trained physically but no SM has programmed it - that is an SM problem, not a cable problem, and no benchmark will run through it.^[6][5] For the whole fabric at once, iblinkinfo “reports link info for each port in an IB fabric, node by node” and -d prints only nodes with a port in the Down state.^[8]

flowchart TD
  A[Slow training job reported] --> B{sminfo returns one master}
  B -- no --> B1[SM problem - start opensm or enable the switch SM]
  B -- yes --> C{ibstat Active LinkUp expected rate}
  C -- no --> C1[Port or link problem - iblinkinfo and mlxlink]
  C -- yes --> D{ib_write_bw host to host near line rate}
  D -- no --> D1[Fabric problem - stop touching NCCL]
  D -- yes --> E{same run with use_cuda near line rate}
  E -- no --> E1[GPUDirect or PCIe topology]
  E -- yes --> F[Application layer - read NCCL_DEBUG INFO first]

2perftest, exactly as documented

The suite is ib_send_bw/ib_send_lat, ib_read_bw/ib_read_lat, ib_write_bw/ib_write_lat and ib_atomic_*, built from source with ./autogen.sh, ./configure, make, make install.^[1] You start the server with no address and the client with the server’s address, “IPv4 or IPv6 address. You can use the IPoIB address if IPoIB is configured.”^[1]

# server
ib_write_bw -d mlx5_0 -a
# client
ib_write_bw -d mlx5_0 -a <server_ip>

The defaults you must be able to quote: -p/--port is 18515, -c/--connection is RC, -d is the first device found, -i/--ib-port is 1, -s/--size is 1, -n/--iters is 1000 with a minimum of 100, and -a/--all runs “sizes from 2 till 2^23”.^[1] Bandwidth tests add -b bidirectional, -t/--tx-depth default 128, -q/--qp default 1 and --run_infinitely, which prints results every five seconds.^[1]

Three methodology facts decide whether a number is defensible. “The SAME OPTIONS must be passed to both server and client.”^[1] “On all unidirectional bandwidth benchmarks, the client measures the bandwidth.”^[1] Latency benchmarks “measure round-trip time but report half of that as one-way latency” and print minimum, median and maximum, where “Typically, the first value measured is the maximum value, due to warmup effects”.^[1] Raising iterations far above the default “may have negative impact on the measured performance”, so use -N (No Peak) when you do.^[1]

Two more traps. Perftest “automatically detects the NUMA node of the IB device and binds the benchmark thread and memory allocations to that node”, and “The active NUMA node is reported in the test output header”; --numa_node is mutually exclusive with --pin_cores and --disable_numa and the conflict is “rejected at parse time”.^[1] And versions must match: “Different versions of perftest may not be compatible with each other. Please use the same perftest version on both sides.”^[1]

3Adding the GPU without losing the plot

GPUDirect is a second measurement, not a replacement for the first. NVIDIA prescribes the exact pair: run ib_write_bw -d <device> -a on the server and ib_write_bw -d <device> <server> -a on the client, then “repeat with --use_cuda=<gpu_id> on both sides” to isolate GPUDirect RDMA.^[2][1] A gap that appears only in the second run is a GPU-to-NIC path problem - PCIe topology, GPUDirect support, or the NIC and GPU sitting on different roots - and not a fabric problem.

MLX5_SCATTER_TO_CQE=0 ./ib_write_bw -d ib_dev --use_cuda=<gpu index> -a

That prefix is the documented recipe for the most common GPUDirect failure: “If GPUDirect is not working, (e.g. you see ‘Couldn’t allocate MR’ error message), consider disabling Scatter to CQE feature. Set the environmental variable MLX5_SCATTER_TO_CQE=0.”^[1] DMA-BUF support adds --use_cuda_dmabuf and requires CUDA Toolkit 11.7 or later with NVIDIA open GPU kernel modules 515 or later.^[1] Build-side detail worth knowing before a customer’s build fails: “As of perftest release 25.07 the build system automatically detects the location of cuda.h”, so CUDA_H_PATH is no longer required and is deprecated in the 25.10 release.^[1]

The other class of failure is pinned memory. NCCL WARN Call to ibv_reg_mr failed and NCCL WARN Call to ibv_create_qp failed both mean the process cannot lock enough memory.^[2] The fix is two lines in /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited

then log out and back in - or restart the scheduler job with the new limits - and verify with ulimit -l.^[2] This is worth checking before a customer’s first run rather than after, because a Slurm job inherits the limits of the daemon that launched it, not of your login shell.

4NCCL over InfiniBand: the few variables that matter

nccl-tests builds with make, or make CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl, and with MPI=1 MPI_HOME=... for multi-node runs.^[4] The single-node eight-GPU sweep is the one to memorise:

./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
# multi-node, 64 ranks over MPI
mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

The flag defaults explain why an unqualified run tells you nothing: -b/--minbytes and -e/--maxbytes both default to 32M, -f/--stepfactor is disabled by default, -g/--ngpus is 1, -n/--iters is 20 and -w/--warmup_iters is 1.^[4] The bus-bandwidth column is documented separately in the repository’s own doc/PERFORMANCE.md.^[4]

On the environment side, four variables carry most of the weight on InfiniBand. NCCL_IB_TIMEOUT is “computed as 4.096 µs × 2^timeout” and defaults to 20 since NCCL 2.23, having been 18 since 2.14.^[3] NCCL_IB_ADAPTIVE_ROUTING is “Enabled (1) by default on IB networks. Disabled (0) by default on RoCE networks”.^[3] NCCL_IB_HCA selects which adapters to use, with the format <hca>[:<port>[:<rail>[:<plane>]]] and comma-separated lists.^[3] NCCL_IB_QPS_PER_CONNECTION is “Number of IB queue pairs to use for each connection between two ranks”, default 1 in the range 1 to 128.^[3]

The anti-pattern to name explicitly: do not set NCCL_IB_GID_INDEX. It “defines the Global ID index used in RoCE mode” with a default of -1, and NVIDIA states that for NCCL 2.21 and later the GID index is selected automatically and should not be set by hand.^[3][2] It is a reflex carried over from RoCE deployments and it is the most common way a working InfiniBand job is broken by tuning.

When you do need to look inside, NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,NET prints which adapter each rank selected - the answer to “are our eight rails actually being used” - and sudo ibping -S on the remote node followed by sudo ibping <lid> locally is the minimal connectivity proof underneath it.^[3][2]

5The proof pack

What you send back after a performance escalation should be reproducible by the customer without you. Four artefacts do it: the sminfo output showing exactly one master; ibstat from both hosts showing Active, LinkUp and the expected rate; the ib_write_bw -a client output with its NUMA header line and the exact command line used on both sides; and the same run again with --use_cuda if GPUs are in scope.^[2][6][1]

Add two numbers that make the result auditable rather than anecdotal. ulimit -l on the host that ran the test, because a pinned-memory limit silently caps large-message performance and produces ibv_reg_mr failures under load.^[2] And a counter baseline: perfquery reads PortCounters and the extended counters from the PMA, and its unit trap must be in your note - “components that represent Data (e.g. PortXmitData and PortRcvData) indicate octets divided by 4 rather than just octets”.^[10] A customer who multiplies by four and gets line rate has learned something; one who forgets has opened a second ticket.

This is also the shape the certification asks for. The NCP-AIN Troubleshooting Tools domain is 20% of the exam and names ib_write_lat, ib_write_bw, ibping, ibstat, ibdiagnet, ibnodes and iblinkinfo directly, alongside the objective “Verify low-latency interconnects”.^[9] The drill and the deliverable are the same work.