algbw, busbw, and reading a nccl-tests result

1The arguments that decide what you measured

Before the arithmetic, the run. nccl-tests builds with plain make, or with paths as make CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl, and with MPI as make MPI=1 MPI_HOME=/path/to/mpi, where NAME_SUFFIX= adds a suffix to the binaries.^[2] The README excerpt confirms all_reduce_perf, comm_ops_perf (requires MPI) and device_api/gin (requires MPI, NCCL 2.30.7 or later, exactly two ranks); the other binaries built from src/ are the conventional ones but were not enumerated on the page fetched, so confirm the list with ls build/ after a build rather than quoting it.^[2]

The defaults are where results go wrong quietly. -b, --minbytes and -e, --maxbytes both default to 32M, -i, --stepbytes to 1M, and -f, --stepfactor is disabled by default — so a bare all_reduce_perf is a single point at 32 MiB, not a sweep.^[2] -n, --iters defaults to 20 timed iterations, -w, --warmup_iters to 1, -m, --agg_iters to 1.^[2] -c, --check defaults to 1 correctness-check iteration; set -c 0 for clean timing runs.^[2] -a, --average defaults to 1, where 0=Rank0, 1=Avg, 2=Min, 3=Max.^[2] -g, --ngpus is GPUs per thread and defaults to 1, -t, --nthreads to 1, -o, --op to Sum, -d, --datatype to Float, -r, --root to 0.^[2]

Reporting and control flags worth knowing: -J, --output_file writes a JSON report, -M, --memory reports memory usage, -U, --tuning reports NCCL tuning information, -I, --per_iter_timing gives CUDA-event per-iteration timings with -K, --per_iter_skip to drop leading samples, -T, --timeout in seconds is disabled by default, and -N, --run_cycles defaults to 1 with 0 meaning infinite.^[2] NCCL_TESTS_SPLIT controls GPU partitioning for parallel operations, and when several operations run in parallel the reported bandwidth is per group.^[2]

The canonical sweep for a fabric question is therefore explicit about all of it: all_reduce_perf -b 8 -e 8G -f 2 -g 8 -c 0.^[2]

mpirun -np 2 --map-by ppr:1:node  ./build/all_reduce_perf_mpi -g 8 -b 8 -e 8G -f 2 -c 0

2algbw is arithmetic; busbw is a model

“NCCL tests report the average operation time in ms, and two bandwidths in GB/s: algorithm bandwidth and bus bandwidth.”^[1] Algorithm bandwidth is the simple one: “Algorithm bandwidth is using the most commonly used formula for bandwidth : size (S) / time (t).”^[1] Its use is equally simple — “It is useful to compute how much time any large operation would take by simply dividing the size of the operation by the algorithm bandwidth.”^[1] If a customer asks how long their all-reduce will take, algbw answers it.

Bus bandwidth answers a different question: how well is the hardware being used. It exists because “the theoretical peak algorithm bandwidth is not equal to the hardware peak bandwidth, usually depending on the number of ranks”, and NCCL tests introduce it “to provide a number which reflects how optimally the hardware is used”.^[1]

For AllReduce the derivation gives t = (S/B) * (2*(n-1)/n) and therefore B = algbw * (2*(n-1)/n).^[1] Crucially, that derivation is “independent of the algorithm used (ring, tree, or other) as long as they use point-to-point operations (send/receive)” — so a busbw number does not depend on which algorithm NCCL chose that day.^[1]

The full set of correction factors: AllReduce 2*(n-1)/n; ReduceScatter (n-1)/n where “S is the size in bytes of the total array, which for NCCL is equal to recvcount*sizeof(datatype)*n”; AllGather (n-1)/n with S = sendcount*sizeof(datatype)*n; AllToAll (n-1)/n; Broadcast and Reduce 1, because “all data has to get out of the root rank, hence the bottleneck is on the root rank”.^[1]

Two consequences to state out loud in any report. A busbw figure without n and without the collective is uninterpretable, because both are inputs to the factor. And AllReduce is the only collective whose factor exceeds 1, which is why an 8-rank AllReduce busbw is 1.75× its algbw and a naive reader thinks the tool is inflating the number.

3Turning a number into a bottleneck claim

The sentence that makes busbw diagnostic: “The bus bandwidth should reflect the speed of the hardware bottleneck: NVLink, PCI, QPI, or network.”^[1] That is the whole method. Compute busbw, then say which of those four it most nearly matches, then defend the choice.

The reading changes with the shape of the run. Within a single node, an 8-GPU AllReduce should approach the NVLink bandwidth, because the collective never leaves the node. Add a second node and the bottleneck moves to the per-GPU network bandwidth — so a drop at exactly two nodes is expected, and the question is whether the size of the drop matches the per-GPU link speed you were sold. Do the arithmetic openly: a 400 Gb/s rail per GPU is 50 GB/s nominal before protocol overhead, so a per-GPU busbw in that neighbourhood at large sizes is consistent with the fabric being the bottleneck and nothing being broken. Any tighter rule of thumb you have heard for the achievable fraction is field practice, not a published NVIDIA figure; label it as such.

Where you read the number also matters. “Time matters at small sizes (‘to measure the constant overhead (or latency)’), bandwidth at large sizes” — so a busbw figure taken at 8 KiB is measuring launch overhead, not hardware.^[1] This is why the sweep exists and why the knee in the sweep is more informative than any single row.

Rails are the reason a two-node number can be worse than the link speed suggests. NVIDIA’s rail definition is that “NIC-0 from each DGX system is connected to the same leaf switch (L0), NIC-1s are connected to the same leaf switch (L1), and so on”, which “helps maximize all-reduce performance while minimizing network interference between flows”.^[4] PXN — “PCI × NVLink” — has GPUs reach a NIC “through NVLink and then PCI” rather than through the CPU, which “couldn’t deliver full bandwidth”, and it “leverages NVIDIA NVSwitch connectivity between GPUs within the node to first move data on a GPU on the same rail as the destination, then send it to the destination without crossing rails”.^[4] That is the mechanism NCCL_CROSS_NIC=0 is protecting on a rail-optimized fabric.^[3]

4Cross-checking with a tool that is not NCCL

A busbw claim gets much stronger when a second, unrelated tool measures the same path. The natural cross-check is point-to-point GPU-to-GPU RDMA between the same two GPUs: NVIDIA’s own reference run is ib_write_bw --use_cuda=0 --use_cuda_dmabuf -d mlx5_0 -a -F --report_gbits -q 1 on the server, with the client adding -n 5000 and the server address.^[6] Convert once and compare: perftest reports Gb/s with --report_gbits, nccl-tests reports GB/s, so divide the perftest figure by 8 before putting the two side by side.^[5][1]

The logic of the comparison is what matters. If perftest between the two GPUs is healthy and NCCL busbw is not, the fabric carried full-rate RDMA on that path and the fault is in configuration or topology — marking, GID selection, cross-NIC policy, rank placement. If both are low, the fabric is implicated and you move to counters. If perftest cannot run in CUDA mode at all, you are back in the GPUDirect gates rather than in a performance question.^[6]

Keep the two tools honest about their differences. perftest is a synthetic micro-benchmark whose README says the benchmarks “are not designed to emulate any real application traffic”, and its latency figures are half a round trip.^[5] NCCL busbw is a modelled figure derived from a collective. They agree about a link, not about a workload, and saying so in the report is what stops a customer treating either number as a service-level guarantee.

When the pair disagrees and you need NCCL’s own account of what it did, the debug pair from the previous lesson is the evidence: NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH, written per rank with NCCL_DEBUG_FILE.^[3] And before either tool, rping proves basic RoCE connectivity so that a slow number is not actually a broken path.^[7]