RoCE, GID tables and the NCCL knobs that read them
S2·E4The slow fabric that was never on the fabric · An escalation bridge, two days before the acceptance run
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish RoCEv1 and RoCEv2 by the header each one puts on the wire and by what a router can do with it.
- Read a GID table from sysfs and explain why every IP address produces two entries.
- Select the NCCL variables that decide NIC choice traffic class and transport and state each default.
- Analyse a slow-fabric report and decide whether RDMA was ever engaged before blaming the switch.
Episode 4 — The slow fabric that was never on the fabric
Two teams are on the bridge and one of them did not ask to be. The customer’s mail says the fabric is slow, the switch vendor has been asked to explain their buffers, and procurement has joined to ask whether any of this moves the lead time. The network lead says the sentence he says once per engagement: show me the counter.
You ask for one run instead of one graph. Same job, same twelve nodes, one variable added - NCCL_IB_DISABLE=1, which disables the InfiniBand and RoCE transport and pushes NCCL back onto IP sockets.[2] The throughput does not move. Whatever the job was doing, it was not doing it over RDMA, so this is not yet a conversation about buffers.
The test works because of how the addressing is built. RoCEv2 carries RDMA inside IP and UDP destination port 4791, which is what makes it routable across a leaf-spine fabric at all - and countable at the leaf, for a network lead who wants a counter.[1] The queue pair that carries it is bound to one GID index, in a table the driver builds from the IP addresses on the port, and nothing promises that table looks the same inside a pod as on the host - which is why NCCL_IB_GID_INDEX defaults to -1 rather than to somebody’s favourite number.[1][2]
Prove RDMA is on the path before you tune anything on it.
Segment 1 starts with the two wire formats that index chooses between.
1Two wire formats, one name
RoCE is two protocols wearing one acronym, and the difference decides whether your fabric can have more than one subnet.
RoCEv1 uses ethertype 0x8915 and encapsulates the InfiniBand transport in Ethernet headers - a Layer 2 protocol.[1] It cannot cross a router.
RoCEv2 encapsulates RDMA in IP plus a UDP destination port of 4791, “so it routes”.[1] That single change is why every rail-optimised AI fabric is v2: leaf-spine designs put GPUs in different subnets, and a Layer 2-only transport cannot reach across them.
Both are supported at the same time. “Both RoCEv1 and RoCEv2 are supported by default; the driver associates all GID indexes to RoCEv1 and RoCEv2.”[1] Nothing forces a choice at the port level - the choice is made per queue pair, by which GID index the application uses.
That is also the fact that makes UDP 4791 a practical handle. If you can count packets on that port at the switch, you can prove RDMA traffic exists without reading a single host counter - which is the containerlab half of this lesson’s no-hardware path.
On the NIC side the same choice appears as configuration. The NIC Configuration Operator’s roceOptimized block carries a roceMode field where 1 is v1 and 2 is v2, alongside a qos object with trust, pfc, tos, cableLen, ecn and pauseFrames.[7] RoCE can also be disabled per device entirely: echo 0 > /sys/devices/{pci-bus-address}/roce_enable, and read back with cat on the same path.[1]
For the NCA-AIIO blueprint this segment is the part that maps cleanly to objective 2.8, “Identify and describe data center networking protocols and key concepts” - though note the blueprint never names RoCE, CNI or RDMA explicitly, so treat this as over-serving the domain rather than as exam coverage.[12]
Diagram source (Mermaid)
flowchart TD; A["Application posts a send on a queue pair"] --> B["QP is bound to one GID index"]; B --> C1["GID type RoCEv1"]; B --> C2["GID type RoCEv2"]; C1 --> D1["Ethernet header + ethertype 0x8915 + IB transport"]; C2 --> D2["Ethernet + IP + UDP dst 4791 + IB transport"]; D1 --> E1["Stops at the first router - Layer 2 only"]; D2 --> E2["Routable across leaf and spine"]; E2 --> F["Switch can classify by UDP 4791 and by DSCP from rdmaQoS tos"]
2Reading the GID table
A GID table entry holds three things: the GID value, the GID type - the RoCE version the entry speaks, shown as v1 or v2 - and the associated network device.[1] The GID format is a separate property: “The GID format can be of 2 types; IPv4 and IPv6”, where an IPv4 GID is an IPv4-mapped IPv6 address.[1] Three sysfs paths expose the three fields, all under one port directory:[1]
# value
cat /sys/class/infiniband/mlx5_0/ports/1/gids/3
# type
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/3
# net device
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/3Entries are created when IP addresses are configured on Ethernet devices.[1] Because the driver registers every index for both RoCE versions, one IP address yields two entries with identical values and different types.[1] So an index by itself never tells you which version you are about to speak - you have to read the type file next to it.
VLANs add the other twist. If the source IP sits on a VLAN interface, “Packets from a QP that is associated with these GID indexes will have a VLAN header (VID=100)”.[1] A GID index picked from the wrong interface therefore silently changes the tag on the wire, and the switch drops or misclassifies traffic that looked correct from the host.
Two documentation notes worth carrying. The kernel’s own Documentation/infiniband/sysfs now states that “The sysfs interface has moved to ABI file stable/sysfs-class-infiniband”, so the authoritative per-attribute list is the ABI file rather than the InfiniBand document.[3] And the kernel InfiniBand index covers Midlayer Locking, IP over InfiniBand, Sysfs files, tag matching, userspace capabilities, MAD access and verbs access - with no coverage of network-namespace behaviour at all.[4]
One more caveat about tooling: the widely used show_gids helper script could not be verified for this lesson - the NVIDIA enterprise-support page returned a client-side error rather than content.[1] Everything in the lab is written against the three sysfs paths instead, which are documented.
3What nobody documents: the container case
Here is the honest state of the world, and it is a genuine documentation gap rather than a lesson-writing shortcut.
- The MLNX_OFED RoCE page describes the GID table and its sysfs paths and makes no statement about GID index selection inside containers.[1]
- The kernel InfiniBand documentation index has no network-namespace coverage.[4]
- The per-attribute sysfs document has been emptied into an ABI file.[3]
So the honest teaching is: measure the table from inside the pod, and do not assume it matches the host. In exclusive netns mode the RDMA device has been moved into the pod’s namespace by rdma-cni, and the addresses configured inside the pod are not the addresses configured on the host.[5] In shared mode the device is “accessible in all network namespaces”, which is a statement about visibility and not a promise about index ordering.[11]
That is why NVIDIA’s own verification step for a Spectrum-X workload is an exec into the pod - kubectl exec -it spectrum-x-test -- rdma link - rather than a host command.[8] Verify where the application lives.
exclusive mode · SR-IOV + rdma-cni · fresh boot — no pod netns yet. Does the node come up ok, does a pod CrashLoopBackOff, or does the command itself fail?
Current: “RDMA devices are visible only in single network namespace.”
One switch, node-wide, two values. `rdma system show` prints the netns mode, privileged-qkey state and monitoring support.
“If there are active network namespaces and if one or more RDMA devices exist, changing mode from shared to exclusive returns error code EBUSY.”
Persist it with `options ib_core netns_mode=0` in /etc/modprobe.d/ib_core.conf rather than setting it at runtime; the runtime path is the one that hits EBUSY.
rdma system show rdma system set netns exclusive rdma system set netns shared echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf
FAE angle: ask for this one line of output before any other question about RDMA in pods. It tells you which half of the fork the customer is standing on.
4The NCCL variables that read the table
NCCL is the consumer. These are the variables a networking FAE actually needs, with the documented defaults.[2]
| Variable | Default | What it does |
|---|---|---|
NCCL_DEBUG |
not set | VERSION, WARN, INFO or TRACE output[2] |
NCCL_DEBUG_SUBSYS |
INIT,BOOTSTRAP,ENV |
subsystem filter; ^ excludes[2] |
NCCL_IB_HCA |
not set | <hca>[:<port>[:<rail>[:<plane>]]]; prefix match by default, = exact, ^ excludes[2] |
NCCL_IB_GID_INDEX |
-1 |
“Defines the Global ID index used in RoCE mode”[2] |
NCCL_IB_TC |
0 |
the InfiniBand traffic class field - the DSCP lever on RoCE[2] |
NCCL_IB_SL |
0 |
InfiniBand service level[2] |
NCCL_IB_TIMEOUT |
20 since NCCL 2.23 |
timeout is 4.096 microseconds times 2 to the power of the value; range 0-31[2] |
NCCL_IB_QPS_PER_CONNECTION |
1 |
range 1-128; more queue pairs give a fabric more routing entropy[2] |
NCCL_NET_GDR_LEVEL |
auto | LOC, PIX, PXB, PHB, SYS - the topological cutoff for GPUDirect RDMA[2] |
NCCL_IB_DISABLE |
not set | set to 1 to disable IB and RoCE transport and fall back to IP sockets[2] |
NCCL_SOCKET_IFNAME |
not set | prefix matching, comma separated, ^ excludes[2] |
Two of these are diagnostics rather than tuning. NCCL_DEBUG=INFO names the chosen NIC per rank, whether GDR is in use and which transport was selected.[2] NCCL_IB_DISABLE=1 is the A/B test: if throughput does not change when you disable RDMA, RDMA was never engaged and the fault is in the pod plumbing from lesson 3, not in the fabric.[2]
The rest are levers with fabric-side twins. NCCL_NET_GDR_LEVEL interacts with which GPUDirect module path is in use - DMA-BUF or the legacy nvidia-peermem - which the GPU Operator decides.[10] And NCCL_IB_TC has a direct counterpart in the CNI, covered next.
5Making the two halves agree
The lesson’s payoff is a class of bug that lives between two teams. Spectrum-X configures each rail’s RDMA CNI with rdmaQoS: {tos: 96, tc: 96}, and names the pod’s RDMA device rdma_<rail topology name>.[6] That is the same knob NCCL exposes as NCCL_IB_TC, whose default is 0.[2] If the CNI marks a rail as traffic class 96 and the job runs with the default, the job’s traffic is not in the queue the fabric provisioned for it. Nothing errors. The switch’s lossless queue sits idle while the job’s packets take the default one.
The same pattern repeats one layer down. The NIC’s own roceOptimized.qos carries trust, pfc, tos and ecn, and roceMode selects v1 or v2.[7] Three places, one policy: the switch DSCP-to-queue map, the NIC’s trust and PFC settings, and the application’s traffic class.
Report: an eight-node training job on ConnectX-7 nodes runs at roughly a third of expected bandwidth. The customer has already opened a case with the switch vendor.
Step 1 - is RDMA even in use. Ask for one run with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS left at its default of INIT,BOOTSTRAP,ENV.[2]
Result: the log names mlx5_2 per rank and reports the IB transport. RDMA is engaged, so this is not a lesson 3 problem.
Step 2 - confirm with the A/B. Re-run with NCCL_IB_DISABLE=1.[2]
Result: throughput drops by half. Good - the first run really was using RDMA. Had it been unchanged, the investigation would stop here and move to rdma system show and the NAD.[11]
Step 3 - the wire format. Confirm RoCEv2 by counting UDP destination port 4791 at the leaf.[1] Non-zero: v2 is in use and the traffic is routable.
Step 4 - the queue. The rails were built by the Spectrum-X flow with rdmaQoS: {tos: 96, tc: 96}.[6] The job’s environment does not set NCCL_IB_TC, so it is 0.[2]
Cause found: the traffic is not landing in the lossless class the fabric reserves. The switch counters looked fine because the packets were never in that queue.
Step 5 - the fix and the proof. Set NCCL_IB_TC to match the rail’s tc, re-run step 1, and confirm at the switch that the class counters now move.[6][2]
Step 6 - what you did not change. NCCL_IB_GID_INDEX stays at -1, because nothing in the evidence pointed at GID selection and the container case is undocumented.[2][1]
Same eight nodes, different report: bandwidth is fine on two nodes and collapses when the job spans more than one leaf, and one uplink is saturated while three sit idle.
- First run:
NCCL_DEBUG=________to learn________and________per rank. - A/B run:
________________=1. If the number is unchanged the conclusion is________________. - The symptom - traffic polarised onto few paths - points at
________________________, whose default is____and whose range is____. - Before changing it, check whether the rail’s
rdmaQoStcand the job’s____________agree, because a mismatch produces________________________________. - The variable you must not touch on this evidence is
____________________, because________________________________. - State the one measurement that would falsify your hypothesis.
A Dell customer sends you a runbook their previous integrator left behind. It sets NCCL_IB_GID_INDEX=3, NCCL_IB_TIMEOUT=25 and NCCL_IB_HCA=mlx5. They report intermittent stalls on a new cluster with a different NIC layout.
Write the review. Acceptance criteria: for each of the three variables you state the documented default, say what the setting assumes about the host, and say what evidence would justify keeping it; you identify which of the three is exponential and quantify the change; you explain why NCCL_IB_HCA=mlx5 is a prefix match and what it would capture on a node with several adapters; and you propose the two-run diagnostic that decides whether any of them should stay, naming the exact variables for each run.[2]
The number lands
With RDMA proven absent the ladder is short - the resource on the node, the netns mode, the NAD - and then the last mismatch. The rails were built with rdmaQoS tc 96 while the job ran with NCCL_IB_TC at its default of 0, so the queue the fabric reserved sat idle and the traffic took the default one.[6][2] Set to match, the run lands on the number, and the switch team goes back to work with the counter they asked for.
Procurement is happy for roughly ten minutes. Then the last open item surfaces: a legacy appliance that is a virtual machine, and needs one of the same cards.
Lab
Read-only on the ConnectX or BlueField-3 host in the Dell lab. No step changes host state; there is nothing to roll back.
-
Pre-flight inventory.
ibstat,ibv_devinfo,rdma system show,ip -br addr show, andls /sys/class/infiniband/. Record the device name and port you will use, and the netns mode - it decides how to read step 5.[9][11] -
List the GID indexes for one port.
ls /sys/class/infiniband/<dev>/ports/<port>/gids/Expected: a numbered list of index files.[1] If not: a missinggidsdirectory means you are on the wrong device name - checkls /sys/class/infiniband/. -
Walk the table. For each index, print value, type and net device:
d=/sys/class/infiniband/mlx5_0/ports/1 for i in $(ls $d/gids); do printf "%s\t%s\t%s\t%s\n" "$i" "$(cat $d/gids/$i)" \ "$(cat $d/gid_attrs/types/$i 2>/dev/null)" \ "$(cat $d/gid_attrs/ndevs/$i 2>/dev/null)" doneExpected: a four-column table - index, value, type, ndev - in which at least one value appears twice with two different types.[1] If not: empty type files usually mean no IP address is configured on the associated Ethernet device, since entries are created when addresses are configured.[1]
-
Explain the duplication out loud before reading on: the driver associates all GID indexes to both RoCEv1 and RoCEv2, so one address yields two entries with identical values and different types.[1]
-
Check for VLAN effects. If any
ndevis a VLAN interface, note it - queue pairs on those indexes carry a VLAN header.[1] -
Repeat the walk from inside a pod. Start a pod with an RDMA device attached (either design from lesson 3) and run the same loop under
kubectl exec. Diff the two tables. Expected: you can state the difference concretely rather than assuming. In exclusive mode the pod’s device was moved into its namespace byrdma-cniand its addresses are the pod’s.[5] If not:kubectl exec -it <pod> -- rdma linkfirst - if that lists nothing, you have a lesson 3 problem, not a GID problem.[8] -
Record the gap, do not paper over it. Write down in your notes that no fetched NVIDIA or kernel page documents which GID index a container sees; the RoCE page is silent on containers, the kernel InfiniBand docs have no netns coverage, and the sysfs document has moved to an ABI file.[1][4][3] Your measured diff is now your own evidence for this host - and only for this host.
-
Optional read-only check.
cat /sys/devices/<pci-bus-address>/roce_enableto confirm RoCE is enabled on the device. Do not write to it in this lab.[1]
Two halves: a log-reading exercise and a containerlab classification exercise on Dell Enterprise SONiC 4.5.1. Nothing here needs an NVIDIA NIC and nothing mutates a host.
-
Read the pair of captured NCCL logs. Open the two provided runs - one plain, one with
NCCL_IB_DISABLE=1. Expected: from the plain run you can extract, per rank, the NIC that was chosen, the transport, and whether GDR is in use, becauseNCCL_DEBUG=INFOprints all three.[2] If not: check thatNCCL_DEBUG_SUBSYSin the capture was left at its default ofINIT,BOOTSTRAP,ENV- a narrowed subsystem list hides the lines you need.[2] -
Compare the two runs and write the conclusion in one sentence. State whether RDMA was engaged in the first run and what the second run proves.[2]
-
Tabulate the environment. From the logs list every
NCCL_IB_*variable that appears, and next to each write its documented default. Flag every one that differs. Expected: a two-column table where the differences are the only things worth discussing. -
Bring up the containerlab topology with two Dell Enterprise SONiC 4.5.1 nodes and two Linux hosts. Expected:
containerlab inspectshows all four running. If not: the SONiC image tag is the usual cause; check thekindand image lines in the topology file. -
Prove UDP 4791 is the routable handle. Configure an ACL or classifier counter on the SONiC interface matching UDP destination port 4791, generate traffic on that port between the two Linux hosts (
iperf3 -u -p 4791is enough - you are counting the port, not doing real RDMA), and read the counter. Expected: the counter increments only for that port.[1] If not: confirm the classifier is bound to the right direction on the interface. Note honestly what this does and does not show: it proves the switch can classify RoCEv2 by port, and it does not prove any RDMA occurred. -
Extend to DSCP. Add a second counter matching the DSCP value corresponding to the
tosa Spectrum-X rail would set, and observe that port and DSCP are independent matches.[6] Expected: you can state which of the two a misconfiguredNCCL_IB_TCwould change. -
Tear down.
containerlab destroyand confirm the lab bridges are gone.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in five sentences, what a GID index is, why the same address appears twice in the table, and why you refuse to copy a GID index out of somebody else's runbook.
Sources
Facts in this lesson were checked against NCCL User Guide environment variables and rdma-system(8) re-fetched 2026-09-07; MLNX_OFED RoCE page, kernel InfiniBand docs and Network Operator 26.7.0 Spectrum-X pages as fetched 2026-09-07. Dates are when each page was fetched.
- MLNX_OFED - RDMA over Converged Ethernet (RoCE) · fetched 2026-09-07
- NCCL User Guide - Environment Variables · fetched 2026-09-07
- Linux kernel - InfiniBand sysfs files · fetched 2026-09-07
- Linux kernel - InfiniBand and remote DMA (RDMA) Interfaces · fetched 2026-09-07
- GitHub - k8snetworkplumbingwg/rdma-cni · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Spectrum-X Architecture and Components · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - NIC Configuration Operator CRD API Reference · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Spectrum-X Verify and Troubleshoot · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - SOS-Report Collection Script · fetched 2026-09-07
- NVIDIA GPU Operator - GPUDirect RDMA and GPUDirect Storage · fetched 2026-09-07
- rdma-system(8) - Linux manual page · fetched 2026-09-07
- NVIDIA-Certified Associate AI Infrastructure and Operations Exam Study Guide (doc 4694224 Jan26) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.