NCCL environment variables on a RoCE fabric
S3·E4The design was correct; the traffic never entered it · NVIDIA briefing room, joint design review, the day before the report
Builds on: What ib_write_bw actually measures, GPUDirect RDMA: proving it is actually on
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Separate the variables that choose the RDMA data path from the one that only chooses the bootstrap interface.
- Compute NCCL_IB_TC from a fabric's DSCP and justify why the default of 0 is usually wrong.
- Apply the current NVIDIA guidance on NCCL_IB_GID_INDEX for NCCL 2.21 and later.
- Build the debug invocation that answers which HCA which GID which traffic class and whether GPUDirect was used.
Episode 4 — The design was correct; the traffic never entered it
Whiteboard, two teams: the customer’s network lead, notebook two-thirds full now, and their ML platform people. The fabric is finished and it is good work: RoCE data on switch priority 3, DSCP 26, PFC and buffers configured on every hop.[5] The training job still stalls, and both halves of the room have spent three weeks proving the problem is on the far side of the NIC. The NVIDIA PM answers every forward-looking question with “not announced”.
You ask what NCCL_IB_TC is set to. Nobody in the room knows. It defines the InfiniBand traffic class field and its default is 0[1], and on RoCE that field is the ToS byte of the outer IP header — so every collective packet is marked DSCP 0 and rides a queue with none of the configuration just presented. The design is correct. The traffic never enters it.
That is why the variable exists. NCCL hides the fabric from the application so a training job need not know what a queue pair is, which means the fabric’s policy must be handed to NCCL explicitly. The arithmetic is NVIDIA’s own: the ToS field is 8 bits, DSCP is 6, so multiply the DSCP value by 4.[6] DSCP 26 gives 104. Someone mentions NCCL_SOCKET_IFNAME was set to the RoCE interface months ago; that one specifies which IP interfaces to use, while NCCL_IB_HCA specifies the RDMA interfaces.[1]
A lossless fabric carries only the traffic class the job was told to ask for.
Segment 1 separates those two: that confusion costs more days than any tuning mistake.
1Two different questions: which NIC, and which network
NCCL has one variable that chooses the RDMA data path and a different one that chooses the IP interface for everything else, and confusing them is the most common configuration error on a new cluster. NCCL_IB_HCA “Specifies which Host Channel Adapter (RDMA) interfaces to use for communication”; the format is <hca>[:<port>[:<rail>[:<plane>]]], comma-separated, where ^ excludes and = requires an exact match, with a documented maximum of 32 HCAs and a default of all available HCA ports.[1] NCCL_SOCKET_IFNAME “Specifies which IP interfaces to use for communication”, with the same ^ and = semantics, where =eth0 matches only eth0 while eth matches eth0 and eth1.[1]
The practical consequence follows directly from the two definitions: NCCL_SOCKET_IFNAME selects the bootstrap and socket-transport interface, while NCCL_IB_HCA selects the RDMA data path — so pointing the former at the RoCE interface does not move RDMA traffic anywhere.[1] It is a useful experiment to run once in a lab, because watching nothing change is what makes the distinction stick.
Two more selection variables matter. NCCL_IB_DISABLE “Prevents the IB/RoCE transport from being used by NCCL. Instead, NCCL will fall back to IP sockets” — an order-of-magnitude slowdown hiding in one environment variable someone set while debugging.[1] NCCL_NET forces the network module by name: "IB" for generic IB verbs, "Socket" for TCP/IP, or an external plugin name.[1]
Before any of this, run the connectivity smoke test NCCL’s own troubleshooting recommends for RoCE: rping -s -a <server_ip> -V -C 10 on the server and rping -c -a <server_ip> -S <client_ip> -V -C 10 on the client.[2] If rping fails there is no point reading NCCL logs.
mpirun -np 2 --map-by ppr:1:node -x NCCL_IB_HCA=mlx5_0 -x NCCL_IB_TC=104 -x NCCL_DEBUG=INFO -x NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH ./build/all_reduce_perf_mpi -g 8 -b 8 -e 8G -f 2 -c 0
-g follows the rank scrubber (8 GPUs per node assumed, XE9680-class). n in the busbw formula is the total rank count, not the GPUs on one node.
perftest "is a collection of tests written over uverbs intended for use as a performance micro-benchmark". Quote this when a customer treats a result as an application SLA: "The benchmarks generate a synthetic stream of operations… The benchmarks are not designed to emulate any real application traffic."
They are "usually installed to the /usr/bin/" and "may need to be ran with sudo when running from non root". A cryptic syndrome string on stderr comes from rdma-core, not from perftest.
perftest README (master)- noteToS = DSCP x 4, so NCCL_IB_TC=104 is DSCP 26. Prove the move with ethtool -S <if> | grep prio3. NCCL 2.31.2 environment variables
- noteNCCL_IB_ADAPTIVE_ROUTING defaults to 1 on InfiniBand but 0 on RoCE, and needs a system-defined AR-enabled SL. NCCL 2.31.2 environment variables
t = (S/B) * (2*(n-1)/n), so B = algbw * 2*(n-1)/n — "independent of the algorithm used (ring, tree, or other) as long as they use point-to-point operations".
"The bus bandwidth should reflect the speed of the hardware bottleneck: NVLink, PCI, QPI, or network."
nccl-tests doc/PERFORMANCE.md2Marking: the default that undoes your whole QoS design
NCCL_IB_TC “Defines the InfiniBand traffic class field” and its default is 0.[1] On RoCE that field is the ToS byte of the outer IP header, so a default of 0 means DSCP 0 — the default, usually lossy queue — regardless of how carefully priority 3 was configured on every switch and host in the path. NCCL_IB_SL sets the Service Level and also defaults to 0.[1] NCCL_IB_FIFO_TC, added in 2.22.3, sets the “Traffic class for control messages (RDMA writes controlling credit return)” and defaults to whatever NCCL_IB_TC is, i.e. 0 unless you set it.[1]
The arithmetic is NVIDIA’s, stated on the RoCE page: “The ToS field is 8 bits, while the DSCP field is 6 bits. To set a DSCP value of X, you need to multiply this value by 4 (SHIFT 2). For example, to set DSCP value of 24, set the ToS bit to 96.”[6] A Cumulus-default fabric puts RoCE data on switch priority 3 with DSCP 26 and CNPs on priority 6 with DSCP 48, so DSCP 26 gives NCCL_IB_TC=104.[5][6]
Two cautions. NCCL’s own RoCE advice is to “Set NCCL_IB_TC per vendor documentation for traffic class tuning” — NVIDIA expects a fabric-specific value, not a universal one.[2] And the value NCCL_IB_TC=106 that circulates in cloud-provider runbooks does not appear on any NVIDIA page; treat it as unverified and vendor-specific rather than as guidance.[1]
Proving the marking took effect does not require a packet capture. The mlx5 driver exposes per-priority volume counters — rx_prio[p]_bytes, rx_prio[p]_packets, tx_prio[p]_bytes, tx_prio[p]_packets — so ethtool -S <if> | grep prio3 before and after a run says whether the traffic actually landed on priority 3.[8] That is the same evidence you would use for a perftest run marked with --tclass.[9]
3Behaviour: GIDs, timeouts, queue pairs, routing
NCCL_IB_GID_INDEX “Defines the Global ID index used in RoCE mode” and defaults to −1.[1] The current guidance is explicit and it reverses older runbooks: on NCCL 2.21 and later the GID index is selected dynamically and should not be set; only on earlier versions do you set it manually after running show_gids.[2] A stale NCCL_IB_GID_INDEX in an inherited job script is itself a documented cause of NCCL WARN Call to ibv_modify_qp failed with error Invalid argument on RoCE.[2][1] Related: NCCL_IB_PKEY defaults to 0 (the first PKey table entry) and NCCL_IB_PKEY_VALUE, added in 2.31, selects by value and takes precedence.[1]
Timeouts: NCCL_IB_TIMEOUT is “computed as 4.096 µs × 2^timeout”, defaults to 20 since 2.23, and accepts 0–31, where 0 or a value of 32 or more yields an infinite timeout.[1] NCCL_IB_RETRY_CNT defaults to 7 with a range of 0–7, and the documentation gives roughly 30 seconds of total wait before a network error with those defaults.[1]
Queue pairs and paths: NCCL_IB_QPS_PER_CONNECTION defaults to 1 with a range of 1–128 and is “Useful for multi-level fabrics requiring routing entropy” — on a Clos it is the knob that spreads flows across ECMP paths.[1] NCCL_IB_SPLIT_DATA_ON_QPS defaults to 0, meaning round-robin, since 2.20; setting 1 splits data evenly, and split mode “may cause latency degradation with many QPs”.[1] NCCL_IB_ADAPTIVE_ROUTING is enabled by default on IB networks and disabled by default on RoCE, and it “requires system-defined AR-enabled SL” — the RoCE default of 0 is the fact people get wrong when they assume Spectrum-X adaptive routing is a NCCL setting.[1] NCCL_IB_ECE_ENABLE enables Enhanced Connection Establishment on IB and RoCE verbs networks, default 1 since 2.19, with a warning that incorrect configuration can harm performance.[1] NCCL_IB_PCI_RELAXED_ORDERING defaults to 2, auto-detect, and is “Beneficial in virtualized environments”.[1]
GPUDirect gating and topology policy: NCCL_NET_GDR_LEVEL defines the maximum NIC-to-GPU distance for GDR using the nvidia-smi topo vocabulary; NCCL_NET_GDR_READ is 0 on most platforms and 1 on NVLink platforms since 2.4.2; NCCL_NET_GDR_C2C was 0 in 2.26 and is 1 since 2.27 for Grace-class systems.[1] NCCL_CROSS_NIC defaults to 2 (try the same NIC, allow a different one if it performs better), and NVIDIA’s guidance is 0 for rail-optimized networks and 1 for flat topologies.[1] On DMA-BUF systems there is nothing extra to configure: “modern systems using DMA-BUF with open-source drivers don’t require this module; NCCL auto-detects and enables DMA-BUF support”.[3]
4Debugging: making NCCL tell you what it chose
NCCL_DEBUG is unset by default and accepts VERSION, WARN, INFO and TRACE.[1] NCCL_DEBUG_SUBSYS filters the INFO output and defaults to INIT,BOOTSTRAP,ENV; supported subsystems include INIT, COLL, P2P, SHM, NET, GRAPH, TUNING, ENV, ALLOC, CALL, PROXY, NVLS, BOOTSTRAP, REG, PROFILE, RAS, DESTROY, ALL, and ^ excludes.[1] For a fabric problem the right first shot is NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH: init shows device and GID selection, net shows the transport, graph shows the topology NCCL derived.[1]
Under MPI or Slurm the ranks interleave, so send each rank’s output to its own file: NCCL_DEBUG_FILE defaults to stdout and supports %h for hostname and %p for PID, as in filename.%h.%p; it also accepts /dev/stdout and /dev/stderr.[1] The documentation adds a rule worth repeating to customers: network-related debugging variables “should generally not be retained in production configurations”.[1]
Two failure signatures close most tickets. NCCL WARN Call to ibv_create_qp failed and NCCL WARN Call to ibv_reg_mr failed point at memlock, fixed with unlimited soft and hard memlock in /etc/security/limits.conf and verified with ulimit -l.[2] NCCL WARN Call to ibv_modify_qp failed with error Invalid argument on RoCE points at GID selection — check show_gids and check whether NCCL_IB_GID_INDEX is set at all on a 2.21-or-later build.[2]
Finally, on a Spectrum-X fabric there is a plugin rather than a variable set. It lives at $HPCX_DIR/nccl_spectrum-x_plugin/lib and is loaded either by prepending that directory to LD_LIBRARY_PATH or by setting NCCL_NET_PLUGIN=spcx; it adds network-failure recovery, dynamic load balancing across multiple NICs, topology awareness, profiler integration, SHARP support and GPU-Initiated Networking.[7] Its own variables include NCCL_COLLNET_ENABLE=1 for SHARP and NCCL_IB_NIC_BW_LOSS_LIMITS for bandwidth-degradation thresholds.[7]
Dev / environment: build, capabilities, performance
Target: two nodes, eight GPUs each, one ConnectX port per GPU, a Cumulus-default fabric with RoCE on DSCP 26 and priority 3, rail-optimized cabling, NCCL 2.31.2.
- Devices. Set
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7, or use the exclusion form to drop the management HCA. The default is all available HCA ports, so name them explicitly when the box has an HCA you do not want used.[1] - Bootstrap. Set
NCCL_SOCKET_IFNAME=to the management interface, not the RoCE one, and say in the runbook that this is bootstrap only.[1] - Marking. DSCP 26 on this fabric, and ToS = DSCP × 4, so
NCCL_IB_TC=104.[5][6] LeaveNCCL_IB_FIFO_TCunset so control messages inherit the same class, or set it explicitly if the fabric gives control traffic its own priority.[1] - GID. Do not set
NCCL_IB_GID_INDEX: on 2.21 and later selection is dynamic.[2] - Entropy and topology.
NCCL_IB_QPS_PER_CONNECTION=4for ECMP spread on a multi-level fabric, andNCCL_CROSS_NIC=0because the cabling is rail-optimized.[1] - Do not set
NCCL_IB_ADAPTIVE_ROUTING=1unless the fabric has an AR-enabled SL defined; the RoCE default is 0 for a reason.[1] - Verify. Run
all_reduce_perf -b 8 -e 8G -f 2 -g 8 -c 0withNCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH NCCL_DEBUG_FILE=/tmp/nccl.%h.%p, and read back which HCA, which GID and whether GDR was used.[4][1] - Prove the queue.
ethtool -S <if> | grep prio3before and after;tx_prio3_bytesmust move.[8]
Acceptance: the env block has one line per decision with a reason, the log confirms the device and transport, and a per-priority counter delta proves the marking.
- RDMA devices are chosen with ________; the bootstrap interface with ________.
- DSCP 26 becomes
NCCL_IB_TC=________ because ToS = DSCP × ________. - On NCCL 2.21 and later you ________ set
NCCL_IB_GID_INDEX. - ECMP entropy comes from ; a rail-optimized fabric wants
NCCL_CROSS_NIC=. NCCL_IB_ADAPTIVE_ROUTINGdefaults to ________ on RoCE and requires ________.- Verification pair:
NCCL_DEBUG=________ withNCCL_DEBUG_SUBSYS=, plusethtool -S <if> | grepbefore and after.
A customer’s job script contains NCCL_SOCKET_IFNAME=ens1f0np0, NCCL_IB_GID_INDEX=3, NCCL_IB_TC=26 and NCCL_IB_DISABLE=0, on NCCL 2.31.2 over a fabric whose switches mark RoCE with DSCP 26. Two-node all-reduce busbw is a third of the single-node figure. Write the correction with a reason per line, and name the two pieces of evidence you would collect to prove each correction worked. Acceptance: you identify the variable that does nothing useful here, the one that is actively wrong arithmetic, the one that current guidance says to remove, and the one that is harmless; and you do not claim any performance number you have not measured.
The counter that ended the argument
The job reruns with NCCL_IB_HCA naming the rail NICs and NCCL_IB_TC=104[1], and ethtool -S <if> | grep prio3 before and after shows the bytes arriving in the queue the lead built.[8] He asks to see the counter twice, then writes it down. The stale NCCL_IB_GID_INDEX comes out of the inherited job script, because on NCCL 2.21 and later the GID index is selected dynamically and should not be set.[2] What you say to the room: nothing on your switches was wrong, the job was never told which queue to use. That night the SE sends the nccl-tests output ahead to the customer’s lead engineer. By morning it is back, with one row circled.
Lab
Dell-lab GPU pair with ConnectX. Read-only: no switch, firmware or QoS changes; environment variables and benchmark runs only.
- Pre-flight inventory. On both hosts:
nvidia-smi topo -m,show_gids,ibdev2netdev,ulimit -l,nvidia-smi | head -12, and the NCCL version fromNCCL_DEBUG=VERSIONon a trivial run. Record everything before changing anything. - Smoke test first.
rping -s -a <server_ip> -V -C 10andrping -c -a <server_ip> -S <client_ip> -V -C 10. Expected: clean exchanges. If rping fails, stop — the problem is below NCCL. - Baseline. Run
all_reduce_perf -b 8 -e 8G -f 2 -g <n> -c 0across the pair with no NCCL variables set beyond the debug pair, capturingNCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH NCCL_DEBUG_FILE=/tmp/nccl-base.%h.%p. Record busbw at the largest size and, fromethtool -S <if> | grep prio, which priority carried the bytes. Expected: priority 0. - Marking run. Repeat with
NCCL_IB_TC=104. Expected: the per-priority counters now move on priority 3. If they do not, the host trust state ispcprather thandscpand this becomes a module 2 problem, not a NCCL one. - Entropy run. Repeat with
NCCL_IB_TC=104 NCCL_IB_QPS_PER_CONNECTION=4. Expected: at large sizes the number moves only if there were alternative paths to spread across; on a single-hop lab pair expect little or no change and record that as the result. - Deliverable. Three busbw figures with their full env blocks, three
ethtool -S ... priodeltas, and one paragraph naming what each run proved and what it did not. - Optional, customer lab only: repeat step 4 on a Spectrum-X fabric with the HPC-X plugin loaded via
NCCL_NET_PLUGIN=spcxand compare. Note in the report that the plugin, not a variable, is what changes behaviour there.
No GPUs required for steps 1–4; step 5 needs the Soft-RoCE containerlab pair from module 1.
- Transcript drill. Read the supplied
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPHlog and answer four questions in writing: which HCA was selected, which GID index was used, which traffic class was applied, and whether GPUDirect was used. Expected: every answer is a quoted line from the log, not an inference. If a question cannot be answered from the log: say which subsystem you would add. - Find the fallback. In the same log, locate the line proving the run fell back to IP sockets rather than the IB/RoCE transport, and name the variable that would produce that. Expected:
NCCL_IB_DISABLEorNCCL_NET=Socket. If neither, the fallback was automatic and you record what preceded it. - Marking arithmetic. Compute
NCCL_IB_TCfor DSCP 24, 26 and 48 and write the ToS byte for each. Expected: 96, 104 and 192. Then state which of the three matches the Cumulus RoCE data default and which matches the CNP default. - Build the env block from first principles for a rail-optimized DSCP-26 fabric with 16 ranks over two nodes, one line per decision with a one-sentence reason. Expected: device selection, marking, entropy, cross-NIC policy, debug pair. Do not include
NCCL_IB_GID_INDEXand be able to say why. - Bootstrap versus data path in containerlab. With Soft-RoCE up on both nodes, set
NCCL_SOCKET_IFNAMEto the management interface and run any two-rank test you can build; then set it to a nonexistent interface and observe the run fail to bootstrap. Expected: the failure is at rank discovery, which is the point — it demonstrates what that variable owns. Rollback: unset the variable. - Write the two failure signatures and their first checks from memory: the
ibv_create_qp/ibv_reg_mrpair, and theibv_modify_qp ... Invalid argumentcase. Expected: memlock andulimit -lfor the first;show_gidsand whetherNCCL_IB_GID_INDEXis set at all for the second.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in four sentences, why a correctly configured lossless RoCE fabric can still carry a customer's training traffic in the wrong queue.
Sources
Facts in this lesson were checked against NCCL 2.31.2 environment-variable reference and troubleshooting pages (re-fetched 2026-09-07); nccl-tests master README; Cumulus Linux 5.18 RoCE defaults; HPC-X 2.51 Spectrum-X NCCL plugin. Dates are when each page was fetched.
- Environment Variables — NCCL 2.31.2 documentation · fetched 2026-09-07
- Networking Troubleshooting — NCCL 2.31.2 documentation · fetched 2026-09-07
- GPU Troubleshooting — NCCL 2.31.2 documentation · fetched 2026-09-07
- NVIDIA/nccl-tests README (master) · fetched 2026-09-07
- RDMA over Converged Ethernet — RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- RDMA over Converged Ethernet (DOCA-Host) · fetched 2026-09-07
- Spectrum-X NCCL Plugin | NVIDIA HPC-X Software Toolkit 2.51 · fetched 2026-09-07
- Linux kernel: mlx5 ethtool counters documentation · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.