Scenario: "the 400G NIC only does 180 Gb/s"

1The ticket, and what it does not say

The mail says: “ib_write_bw between two nodes gives 180 on a 400G link, and our NCCL busbw halves when we add the second node.” Two numbers, no units, no command lines, no node names.

Before asking for anything, fix what perftest actually reports. “On all unidirectional bandwidth benchmarks, the client measures the bandwidth”, so the server-side terminal is not the number.^[1] Latency tests “measure round-trip time but report half of that as one-way latency”, which “may not be accurate for asymmetrical configurations”.^[1] And the sentence to keep in a snippet file for customers who treat a micro-benchmark as an application SLA: “The benchmarks generate a synthetic stream of operations… The benchmarks are not designed to emulate any real application traffic.”^[1]

Two constraints decide whether the evidence is admissible at all. “The same options must be passed to both server and client” - stated in both the README and the man page.^[1][2] And “Different versions of perftest may not be compatible with each other. Please use the same perftest version on both sides”, with the specific note that “Test version 5.3 and above won’t work with previous versions of perftest. As well as 5.70 and above.”^[1] If the customer used --dont_xchg_versions to get past a mismatch, the run skipped the version and MTU exchange and the result describes two differently configured endpoints.^[2]

2The checklist, in the order that costs least

Six checks explain most 400G bandwidth complaints, and they are ordered by what the customer already has in hand.

One - units. --report_gbits reports “Max/Average BW of test in Gbit/sec (instead of MiB/sec). Relevant only for bandwidth.”^[2] A run printing 22,400 MiB/s is about 188 Gb/s. That is the same “180” the ticket reported, and the fabric is fine.

Two - queue pairs. -q, --qp=<num of qp's> is “Num of qp’s (default 1). Relevant only for bandwidth.”^[2] One QP is one flow, one entropy value on the ECMP hash, and one hardware scheduling context; ask for -q 4 to -q 8 before drawing conclusions.

Three - message size. -s, --size defaults to “65536 for bw, for lat 2” in the man page while the README’s common-options block says the default is 1 - a genuine conflict worth quoting both ways.^[2][1] -a “Run sizes from 2 till 2^23” and gives a curve instead of a point.^[2] A small-message run is a message-rate test wearing a bandwidth test’s name.

Four - the NUMA line in the header. “Perftest automatically detects the NUMA node of the IB device and binds the benchmark thread and memory allocations to that node… The active NUMA node is reported in the test output header”, and “If the NIC’s NUMA node cannot be determined, perftest continues without binding.”^[1] It also “respects pre-existing CPU affinity set by external tools (e.g. taskset or numactl)”, in which case automatic binding is skipped.^[1] That gives a clean A/B for a suspected NUMA regression: --numa_node=<id> against --disable_numa, which restores “the original behavior (OS scheduler decides placement)”.^[1]

Five - GID and traffic class. -x, --gid-index picks the GID, and --tclass=<value> sets “the Traffic Class in GRH (if GRH is in use)” - the non--R path to a DSCP marking.^[2] With -R, -T, --tos applies instead and is “available only with -R flag. values 0-256 (default off)”.^[2] A run on the wrong GID or with no traffic class is a run through the lossy queue, which is a QoS finding, not a fabric-speed finding.^[13]

Six - versions on both sides, as above.^[1]

Two flags make the re-run defensible rather than repeatable-by-luck: -D, --duration with -f, --margin (default 2 s excluded at each end) instead of an iteration count, and -W, --report-counters=<list> which diffs sysfs counters across the run, for example counters/port_xmit_data,hw_counters/out_of_buffer.^[2] Ask for one command line, run identically on both hosts, with those two flags.

3Counters second, and only on both hosts

When the command line is clean and the number is still low, the counters decide where to look, and the read order matters more than the counter list.

Start on the two hosts with the per-priority Ethernet counters. rx_prio[p]_pause is “The number of pause packets received with priority p on a physical port. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from the adapter”, and it is “available only if PFC was enabled on priority p”.^[5] Its mirror, tx_prio[p]_pause, means “the adapter is congested and cannot absorb the traffic coming from the network”.^[5] Direction identifies the victim, which is why you read both hosts before either switch.

Three counters change the diagnosis outright. rx_pause_ctrl_phy and tx_pause_ctrl_phy count link-layer, non-PFC pause; movement there while the customer believes PFC is configured means global pause is on and PFC is not.^[5] tx_pause_storm_error_events counts the times pause transmission was disabled by timeout, “on the period where pause frames were disabled, drop could have been” - a lossless fabric that became lossy.^[5] And rx_pci_signal_integrity is the one that ends the QoS conversation: “If this counter is raising, try moving the adapter card to a different slot to rule out a bad PCI slot. Validate that you are running with the latest firmware available and latest server BIOS version.”^[5] That is the Dell escalation script, printed by the driver.

On the RDMA side, packet_seq_err and out_of_buffer come from the in-tree mlx5 driver along with the DCQCN four - np_ecn_marked_roce_packets, np_cnp_sent, rp_cnp_handled, rp_cnp_ignored.^[10] The wording usually quoted for np_cnp_sent and rp_cnp_handled comes from a blog whose page returns 403 to automated fetch rather than from an NVIDIA page, so quote the behaviour, not the sentence.^[14] Two mechanical traps: hw_counters refresh is capped by lifespan, “in milliseconds and defaults to 10”, with values 0 to 10000 writable, so a fast polling loop can report a frozen counter that is only cached; and port_xmit_data is “Total number of data octets, divided by 4 (lanes)”, so it must be multiplied by four before it is compared with a perftest figure.^[6]

4The second node: is it the fabric or is it GPUDirect?

The other half of the ticket is NCCL. The arithmetic is small and settles the argument: “algbw = S/t”, and bus bandwidth applies a per-collective correction because “the theoretical peak algorithm bandwidth is not equal to the hardware peak bandwidth, usually depending on the number of ranks”.^[3] For AllReduce, t = (S/B) * (2*(n-1)/n) so B = algbw * (2*(n-1)/n); ReduceScatter, AllGather and AllToAll use (n-1)/n; Broadcast and Reduce use 1 “because all data has to get out of the root rank”.^[3] The sentence that turns the number into a diagnosis: “The bus bandwidth should reflect the speed of the hardware bottleneck: NVLink, PCI, QPI, or network.”^[3]

So a drop at exactly two nodes is expected in kind - the bottleneck moved from NVLink to the fabric - and the question is only whether the new busbw matches the per-GPU rail bandwidth. Ask for n, the size sweep (-b 8 -e 8G -f 2), and whether correctness checking was on: -c defaults to 1 and -c 0 is the clean timing run.^[12]

If the two-node number is far below the rail, split the fault in one step: run ib_write_bw host-memory between the same two hosts, then the same run with --use_cuda. NVIDIA’s own reference line is ib_write_bw --use_cuda=0 --use_cuda_dmabuf -d mlx5_0 -a -F --report_gbits -q 1 on the server with -n 5000 and the server IP on the client.^[11] If the GPU number is roughly the host number, GPUDirect is on; if it is a fraction, it is bouncing through host memory and the fault is not the fabric.

The GPUDirect list, in order: NIC-to-GPU distance from nvidia-smi topo -m;^[9] the peer-memory path (lsmod | grep nvidia, expecting modules such as nvidia_fs and nvidia_peermem, and the driver-install order trap where a GPU driver installed before the network drivers must be reinstalled);^[11] memlock, where NCCL WARN Call to ibv_create_qp failed or ibv_reg_mr failed means memlock is not unlimited and the container equivalent is securityContext.capabilities.add: ["IPC_LOCK"];^[7][11] IOMMU, checked with cat /proc/cmdline and dmesg | grep -i -E "iommu|dmar|default domain", where “Default domain type: Translated” is the concerning output;^[8] and PCIe ACS, found with sudo lspci -vvv | grep ACSCtl.^[8] If perftest itself fails with “Couldn’t allocate MR” in CUDA mode, retry with MLX5_SCATTER_TO_CQE=0.^[1]

Four NCCL variables account for most of the rest. NCCL_IB_DISABLE “Prevents IB/RoCE transport; forces fallback to IP sockets” and defaults to 0.^[4] NCCL_IB_TC defaults to 0, so unmarked traffic lands in the default queue no matter what priority 3 was configured for.^[4] NCCL_IB_GID_INDEX defaults to -1, and current guidance is explicit: “NCCL 2.21+: GID index selected dynamically; do not set NCCL_IB_GID_INDEX.”^[7] NCCL_IB_QPS_PER_CONNECTION defaults to 1 and is “Useful for multi-level fabrics requiring routing entropy” - the same single-flow problem as -q 1, one layer up.^[4] For evidence, NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH is the right first shot, written to NCCL_DEBUG_FILE with %h and %p so ranks do not interleave.^[4]

5The deliverable: measured, inferred, one change, one test

A finding is not a diagnosis - it is a document the customer can act on without you in the room. Four sections, under 300 words.

Measured. The exact command lines from both sides, the exact output including the NUMA header line, the counter deltas with the commands that read them, and the conversion arithmetic if units were involved. Anything you did not see with your own eyes goes in the next section instead.

Inferred. The chain of reasoning from the measurements to the cause, with each step attributable to a documented behaviour: -q 1 is documented as the default and is documented as bandwidth-only; rx_prio3_pause is documented as the network pausing the adapter.^[2][5]

The one change. One. Not a tuning list. If the evidence supports two changes, the finding is not finished; run the A/B that separates them.

How the customer verifies it. The exact re-run, the number that means success, and the counter that must stay flat. This is what makes the finding survive contact with a team you will never meet.