Rail-optimized design and what breaks it
S2·E1The pod that passed every test · A crop insurer's data hall, eleven weeks before the rate filing
Builds on: The networks in an AI factory, and what the RA refuses to specify
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- State what the NVIDIA RAs actually define as rail-optimized and derive the rail count for a given node from its GPU count.
- Predict which traffic stays one hop and which traverses the spine for a given cabling scheme.
- Diagnose a below-line-rate all-reduce by separating a cabling fault from an NCCL pinning fault from a traffic-class fault.
- Name the three host-side knobs that keep NCCL traffic on its rail and state each one's documented default.
Episode 1 — The pod that passed every test
The customer underwrites hail and drought risk, and once a year they publish new rates on a fixed filing date. Their case for phase two — another 32 nodes, doubling the fabric — rests on the 32-node HGX B300 pod accepted six weeks ago. This morning their ML team reported the hail model’s all-reduce at roughly 60 percent of budget, and the network lead opened a case asking for a switch RMA. The Dell SE is on the bridge with a coffee he has reheated twice and a spreadsheet of every promise he has made about this pod.
Nothing in the hall is broken. The trouble is that the design promised something narrower than “it works”. The HGX B300 RA connects GPUs “using a rail-optimized network topology through their respective NVIDIA ConnectX-8 SuperNICs”, one adapter per GPU on a baseboard that carries eight of them.[1][7] The SuperPOD RA states the payoff in one line: traffic per rail “is always one hop away from the other 64 nodes in a SU”.[2] That is why the wiring rule exists — collectives spend most of their bytes talking to the same-index peer on every other node, so the design buys that dominant pattern a single switch hop and leaves the spine for traffic that genuinely has to cross rails.
A pod can forward every packet correctly and still not be the pod that was designed.
The network lead taps his notebook and says the thing he always says: show me the counter. Segment 1 starts with what the RAs actually say.
1What the RAs actually say
Rail-optimized is a wiring rule, not a protocol. The HGX B300 Enterprise RA states it in one sentence: “GPUs are connected using a rail-optimized network topology through their respective NVIDIA ConnectX-8 SuperNICs.”[1] The load-bearing word is respective. Each GPU owns its own SuperNIC — the HGX B300 baseboard carries “Eight NVIDIA ConnectX-8 SuperNICs per NVIDIA HGX B300 baseboard” at “Up to 800 Gbps per adapter”, a 1:1 GPU-to-NIC ratio.[7] Rail R is then the set of all the rail-R NICs across every node in the scalable unit, and the rule is that they all land on the same leaf switch.
The consequence is what the SuperPOD RAs publish. In the Spectrum-4 Ethernet and DC-busbar DGX B300 RA, “Each group of 64 nodes is rail-aligned” and “Traffic per rail of the DGX B300 systems is always one hop away from the other 64 nodes in a SU.”[2] The Quantum-X800 InfiniBand edition of the same server says the same thing with a different number: “Traffic per rail of the DGX B300 systems is always one hop away from the other 72 nodes in a SU.”[3] Same server, two RAs, two SU sizes — which matters the moment someone quotes leaf counts.
Rail count is not a design choice. The GB200 NVL72 SuperPOD RA fixes it: “Each SU contains 4 SLGs to match with the number of IB rails (which equals the number of GPUs per compute tray).”[4] Four GPUs per tray, four rails. Eight GPUs per HGX node, eight rails. You do not get to round it.
Rail-optimized
GPUs are connected "using a rail-optimized network topology through their respective NVIDIA ConnectX-8 SuperNICs" — the same-index NIC on every node lands on the same leaf. Rail count equals GPUs per node.
Click any GPU cell to follow its SuperNIC to a leaf and a fabric.
2 × 400 Gb/s per GPU (800 Gb/s ConnectX-8 port, broken out)
"Each GPU generating 800 Gb/s bandwidth through the ConnectX-8 SuperNICs, dual plane topology involves breaking the interface to 2x400 Gb/s interfaces" — each interface goes "to a different leaf switch", and "Each leaf switch is part of an independent fabric that scales to 1024 interfaces of 400 Gb/s". Load balancing is done "by the ConnectX-8 SuperNIC at hardware level".
Transceiver: twin-port transceiver — MMA4Z00-NS, 800 Gb/s twin-port OSFP, 2 × 400 Gb/s, multimode, 50 m max reach. The RA states the fork in the optics: "dual-plane using twin-port transceivers or single-plane using single-port OSFP transceivers".
Host side: NUM_OF_PLANES_P1=2 (hwplb; 0 for swplb/uniplane) · LAG_RESOURCE_ALLOCATION=1 for hwplb · CNP DSCP 48 for hwplb vs 0 · multiplaneMode: "swplb" | "hwplb"
Network Operator 26.4.0 matrix: Dual-Plane on ConnectX-8 with B300 / GB300 — "swplb" GA, "hwplb" tech preview. "hwplb multiplane mode is only supported on ConnectX-8 (device ID 1023)".
- It is not a bond and not redundancy pairing: two independent fabrics with no links between them.
- BOM consequence: leaf count doubles versus single plane at the same per-GPU bandwidth, and the customer must operate two IP fabrics.
- Twin-planar resilience claim on the DGX B300 SuperPOD RA: "A single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion".
8 nodes × 8 rails = 64 server-facing 400 Gb/s interfaces per fabric × 2 fabrics = 128 total.
Rail-aligned needs at least one leaf per rail per fabric: 8 rails × 1 leaf per rail = 8 leaves per fabric, 16 leaves total. Pure port arithmetic would only need 1 per fabric — the difference is what rail alignment costs. derived
Leaf radix: SN5610 / SN5600 presents 64 OSFP cages = 128 × 400 Gb/s logical; a non-blocking leaf faces 64 of them at the servers and 64 at the spines. derived — the RAs do not print a per-leaf split
Transceiver at the NIC: twin-port OSFP (MMA4Z00-NS class) — this is a BOM fork, not a field option.
8 GPUs / node: HGX B300 node: "Eight NVIDIA B300 GPUs on an HGX B300 baseboard" with "Eight NVIDIA ConnectX-8 SuperNICs per NVIDIA HGX B300 baseboard" — a 1:1 GPU-to-SuperNIC ratio at "800 Gb/s (2 x 400Gb/s Ethernet) per GPU". 8 GPUs per node means 8 rails.
8 nodes: Below the 64-node line where the RA starts recommending dual plane.
Keeping traffic on its rail from the host
- NCCL_IB_HCA selects the HCAs, format <hca>[:<port>[:<rail>[:<plane>]]]; ^ excludes, = forces an exact match. Default is "All available IB interfaces".
- NCCL_IB_TC "Defines the InfiniBand traffic class field", default 0, range 0–15 — set it to the traffic class the switch marks as RoCE.
- NCCL_IB_SL "Defines the InfiniBand Service Level", default 0, range 0–15. NCCL_IB_GID_INDEX "Defines the Global ID index used in RoCE mode", default -1.
- NCCL_NET_GDR_LEVEL controls GPUDirect RDMA between a NIC and a GPU: LOC (disabled), PIX, PXB, PHB, SYS. NCCL_SOCKET_IFNAME auto-selects, "favoring interfaces starting with ib".
- NicInterfaceNameTemplate gives "rail/plane-based network device naming via udev rules" — that is how a rail survives a reboot with a stable interface name.
- SpectrumXRailPoolConfig carries "rail topology, PF selection, IPAM binding, resource exposure"; CIDRPool allocates IPs per rail or per rail/plane.
FAE angle. "We bought the RA and we get 60% of line rate on all-reduce" is the escalation. Check in order: (1) is the cabling actually rail-aligned, or did the installer cable node-sequentially? (2) is NCCL_IB_HCA pinned so rank N uses rail N? (3) is NCCL_IB_TC set to the traffic class the switch marks as RoCE? Two of those three are free to fix.
Where the RA stops. The RA names six logical networks over one physical fabric — GPU Compute, CPU Converged, Storage, Customer, Support Server Networking and OOB — with "VLAN isolation ... to provide logical separation". It publishes no BGP/EVPN configuration, no IP addressing, no MTU, no DSCP/PFC/ECN values and no adaptive-routing or congestion-control parameters. Those live in the Cumulus and Spectrum-X software docs.
2One hop per rail, and what pays for it
The one-hop property applies per rail and nowhere else. Traffic that must cross rails traverses the spine layer, and the RAs treat that as the accepted cost of the design rather than a fault.[2] So a rail-optimized pod has two very different path lengths inside one fabric, and the job’s communication pattern decides which one it gets.
That is also why the design is described as resilient rather than fragile. In the twin-planar Ethernet B300 design “Each GPU has 2x 400GbE connectivity through two different planes”, and the RA claims that “A single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion” — the job degrades instead of dying.[2] Rail alignment and plane splitting are separate ideas that ship together: rails decide which leaf, planes decide which fabric. The next lesson takes planes apart on their own.
For a Dell conversation the number to carry is the leaf radix that the rail has to fit into. The SN5600 and SN5610 present 64 OSFP cages that become 128 logical interfaces of 400 GbE in 2U.[8] A rail of 64 nodes is 64 server-facing 400 GbE interfaces on one leaf — comfortably inside that radix, with room left for uplinks. A rail of 128 nodes is not, and that is where the design stops being a single leaf per rail.
3Failure one: the installer cabled it node-sequentially
The first thing that breaks rail optimization is a cable map that nobody checked. Rail-aligned means node N interface R goes to leaf R. Node-sequential means node N’s eight interfaces all go to leaf N, because that is the tidy way to cable a rack and it uses less patch cord.
Node-sequential wiring is not a failure in any operational sense. Every link trains, every counter is clean, BGP comes up, RoCE is configured, and the fabric forwards everything correctly. The only thing that changed is that the same-index peers a collective talks to are now scattered across eight leaves instead of sitting behind one, so the exchange that the RA promises at one hop takes three.[2][1]
Go back to the interactive above and switch the cabling toggle to node-sequential. Watch the same-rail hop count move from one to three while nothing else on the panel changes. That visual is the whole argument to make in front of a customer who wants to “simplify the wiring”.
Rail-optimized
GPUs are connected "using a rail-optimized network topology through their respective NVIDIA ConnectX-8 SuperNICs" — the same-index NIC on every node lands on the same leaf. Rail count equals GPUs per node.
Click any GPU cell to follow its SuperNIC to a leaf and a fabric.
2 × 400 Gb/s per GPU (800 Gb/s ConnectX-8 port, broken out)
"Each GPU generating 800 Gb/s bandwidth through the ConnectX-8 SuperNICs, dual plane topology involves breaking the interface to 2x400 Gb/s interfaces" — each interface goes "to a different leaf switch", and "Each leaf switch is part of an independent fabric that scales to 1024 interfaces of 400 Gb/s". Load balancing is done "by the ConnectX-8 SuperNIC at hardware level".
Transceiver: twin-port transceiver — MMA4Z00-NS, 800 Gb/s twin-port OSFP, 2 × 400 Gb/s, multimode, 50 m max reach. The RA states the fork in the optics: "dual-plane using twin-port transceivers or single-plane using single-port OSFP transceivers".
Host side: NUM_OF_PLANES_P1=2 (hwplb; 0 for swplb/uniplane) · LAG_RESOURCE_ALLOCATION=1 for hwplb · CNP DSCP 48 for hwplb vs 0 · multiplaneMode: "swplb" | "hwplb"
Network Operator 26.4.0 matrix: Dual-Plane on ConnectX-8 with B300 / GB300 — "swplb" GA, "hwplb" tech preview. "hwplb multiplane mode is only supported on ConnectX-8 (device ID 1023)".
- It is not a bond and not redundancy pairing: two independent fabrics with no links between them.
- BOM consequence: leaf count doubles versus single plane at the same per-GPU bandwidth, and the customer must operate two IP fabrics.
- Twin-planar resilience claim on the DGX B300 SuperPOD RA: "A single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion".
8 nodes × 8 rails = 64 server-facing 400 Gb/s interfaces per fabric × 2 fabrics = 128 total.
Node-sequential only has to fill ports: 64 ÷ 64 = 1 leaves per fabric, 2 leaves total — fewer leaves than the 8 per fabric rail alignment needs, and no one-hop rail. derived
Leaf radix: SN5610 / SN5600 presents 64 OSFP cages = 128 × 400 Gb/s logical; a non-blocking leaf faces 64 of them at the servers and 64 at the spines. derived — the RAs do not print a per-leaf split
Transceiver at the NIC: twin-port OSFP (MMA4Z00-NS class) — this is a BOM fork, not a field option.
8 GPUs / node: HGX B300 node: "Eight NVIDIA B300 GPUs on an HGX B300 baseboard" with "Eight NVIDIA ConnectX-8 SuperNICs per NVIDIA HGX B300 baseboard" — a 1:1 GPU-to-SuperNIC ratio at "800 Gb/s (2 x 400Gb/s Ethernet) per GPU". 8 GPUs per node means 8 rails.
8 nodes: Below the 64-node line where the RA starts recommending dual plane.
Keeping traffic on its rail from the host
- NCCL_IB_HCA selects the HCAs, format <hca>[:<port>[:<rail>[:<plane>]]]; ^ excludes, = forces an exact match. Default is "All available IB interfaces".
- NCCL_IB_TC "Defines the InfiniBand traffic class field", default 0, range 0–15 — set it to the traffic class the switch marks as RoCE.
- NCCL_IB_SL "Defines the InfiniBand Service Level", default 0, range 0–15. NCCL_IB_GID_INDEX "Defines the Global ID index used in RoCE mode", default -1.
- NCCL_NET_GDR_LEVEL controls GPUDirect RDMA between a NIC and a GPU: LOC (disabled), PIX, PXB, PHB, SYS. NCCL_SOCKET_IFNAME auto-selects, "favoring interfaces starting with ib".
- NicInterfaceNameTemplate gives "rail/plane-based network device naming via udev rules" — that is how a rail survives a reboot with a stable interface name.
- SpectrumXRailPoolConfig carries "rail topology, PF selection, IPAM binding, resource exposure"; CIDRPool allocates IPs per rail or per rail/plane.
FAE angle. "We bought the RA and we get 60% of line rate on all-reduce" is the escalation. Check in order: (1) is the cabling actually rail-aligned, or did the installer cable node-sequentially? (2) is NCCL_IB_HCA pinned so rank N uses rail N? (3) is NCCL_IB_TC set to the traffic class the switch marks as RoCE? Two of those three are free to fix.
Where the RA stops. The RA names six logical networks over one physical fabric — GPU Compute, CPU Converged, Storage, Customer, Support Server Networking and OOB — with "VLAN isolation ... to provide logical separation". It publishes no BGP/EVPN configuration, no IP addressing, no MTU, no DSCP/PFC/ECN values and no adaptive-routing or congestion-control parameters. Those live in the Cumulus and Spectrum-X software docs.
4Failures two and three: NCCL is not pinned, and lands in the wrong class
Even on perfect cabling, the host has to cooperate. NCCL_IB_HCA selects which HCAs NCCL may use, with the format <hca>[:<port>[:<rail>[:<plane>]]]; a ^ prefix excludes and = forces an exact match, and when the port is omitted all ports on the HCA are used.[5] The page documents the filter, not a default: with the variable unset NCCL uses every IB Verbs interface it finds, so nothing is rail-pinned until somebody pins it.[5] A rank that picks the wrong device is doing rail-crossing traffic by accident on correctly cabled hardware.
The second host knob is the traffic class. NCCL_IB_TC “Defines the InfiniBand traffic class field”; the page states only that the default value is 0 and refers you to the InfiniBand specification Volume 1 or vendor documentation for the legal values.[5] If the switch marks a different class as the RoCE class, default NCCL traffic is riding in the wrong queue — plenty fast when the fabric is idle and badly behaved the moment it is not. Its siblings matter for the same reason: NCCL_IB_SL (Service Level, default 0, range likewise deferred to the IB spec) and NCCL_IB_GID_INDEX, which “Defines the Global ID index used in RoCE mode”, default -1.[5]
Two more variables belong on the same checklist. NCCL_NET_GDR_LEVEL controls when GPUDirect RDMA is used between a NIC and a GPU, with values LOC (never), PIX, PXB, PHB and SYS (even across the SMP interconnect between NUMA nodes).[5] And NCCL_SOCKET_IFNAME auto-selects by default, favouring interfaces whose names start with ib and skipping lo and docker* unless nothing else exists.[5] On a Kubernetes cluster the naming itself is managed: the Spectrum-X Network Operator supplies NicInterfaceNameTemplate for “predictable rail/plane-based netdev names driven by udev rules” and SpectrumXRailPoolConfig for “rail topology, PF selection, IPAM binding, and DRA / SR-IOV resource exposure”.[6]
5The escalation, in the order that costs least
The classic ticket is “we bought the reference architecture and we get 60% of line rate on all-reduce.” Three causes account for most of them and they have very different costs to check, so check them in cost order: cabling, then NCCL_IB_HCA pinning, then NCCL_IB_TC.[5][1] Two of the three are free.
Walk the tree below on the throughput branch before you open a real case. It is worth internalising that the first question is never “which switch is bad” — it is “is the traffic where the design assumed it would be”.
Sample runs, exits cleanly, but zero throughput
A Dell-installed 32-node HGX B300 pod (256 GPUs, dual plane). Pair bandwidth tests pass at line rate. A 32-node all-reduce runs at roughly 60% of expected. The customer wants a switch RMA.
- Establish what the design promised. 8 GPUs per node, 1 ConnectX-8 SuperNIC per GPU, so eight rails.[7] The RA’s promise is that rail-R traffic is one hop inside the SU.[2] Anything above one hop for same-index peers is a deviation from the design, not a switch defect.
- Read the as-built port map, not the design drawing. Look for the mechanical signature: every node’s port 1 on the same leaf. If instead every leaf holds all eight ports of four nodes, the build is node-sequential and you have your answer.[1]
Expected: eight device-to-netdev pairs, one per SuperNIC. If not: fix the inventory question before the topology question.# on one compute node, list the RDMA devices and their netdevs ibdev2netdev - Read the job environment.
Expected on a tuned pod: an explicitenv | grep -E '^NCCL_(IB_HCA|IB_TC|IB_SL|IB_GID_INDEX|NET_GDR_LEVEL|SOCKET_IFNAME)='NCCL_IB_HCAlist and an explicitNCCL_IB_TC. Empty output means NCCL is using every available IB interface and traffic class0, both defaults.[5] - Compare
NCCL_IB_TCagainst the switch’s RoCE class. Traffic class0is the NCCL default; if the fabric marks a different class as RoCE, the traffic is outside the lossless queue.[5] - Change one thing and re-measure. Pin
NCCL_IB_HCAso rank N uses its rail-matched device, re-run the same collective, record the number. Then setNCCL_IB_TCand re-run again. Two runs, two deltas, one attributable cause each. - Conclusion to write in the case. “Fabric forwards correctly. Deviation is in cabling/host configuration, not hardware.” That sentence is why no switch was returned.
Same pod, same symptom, but the customer has already pinned NCCL_IB_HCA and still sees 60%.
- The design promise is that rail traffic is ____ hop inside the SU, and rail count equals ____ per node.[2][4]
- If
NCCL_IB_HCAis pinned but the cable map is node-sequential, pinning rank N to rail N sends it to leaf ____, which is the ____ leaf for its peers. - The command to list RDMA devices against their netdevs is
________. - The variable that decides which queue the RDMA traffic lands in is
________, default____.[5] - The variable that decides whether GPUDirect RDMA is used between the NIC and the GPU is
________, and the value that disables it entirely is____.[5] - State in one sentence why a correct pin on an incorrect cable map makes the measurement worse.
A GB200 NVL72 customer (not B300) reports the same symptom on a two-SU pod. Their compute trays integrate four ConnectX-7 NICs for the compute fabric and two BlueField-3 NICs for in-band management and storage.[10]
Produce, without further hints:
- The rail count for this platform and the sentence in the RA that fixes it.
- Whether the “one hop to the other 64 nodes” phrasing from the B300 Ethernet RA applies here, and what the GB200 RA uses instead of a flat leaf-per-rail layout.
- A three-item escalation checklist in cost order with the exact command or variable for each item.
Acceptance criteria: every factual claim carries the page it came from; anything you computed yourself is labelled as your arithmetic; the checklist names no switch replacement.
Episode 1 — Case note: no switch goes back
The as-built port map settles it in two minutes, and it exists only because the night-shift operator labelled every cable during the install. Every node’s eight cables land on one leaf, so the same-index peers the collective talks to are scattered across eight leaves and pay a spine crossing the design never sized for.[2][1] The job environment is empty as well, so NCCL was never pinned.[5] What you say: “The fabric forwards correctly — the deviation is cabling plus host configuration, and two of the three fixes cost nothing but a re-run.” Phase two unfreezes that afternoon. Then the network lead mails one condition ahead of the design review: bond both 400 Gb/s halves of every GPU link into a single LACP aggregate.
Lab
Read-only on the Dell lab’s ConnectX / BlueField-3 hosts. No firmware, no mode changes, no persistent configuration.
- Pre-flight inventory. On both hosts, capture what you are about to measure:
Expected: at least one device withibstat | head -40 ibdev2netdev ibv_devinfo | grep -E 'hca_id|link_layer|state'PORT_ACTIVE. Save the output — it is your baseline and your rollback reference. - Record the NCCL-relevant defaults on this host. Nothing is changed, only read:
Expected: empty environment (defaults:env | grep -E '^NCCL_IB_(HCA|TC|SL|GID_INDEX)=' show_gids 2>/dev/null | headNCCL_IB_TC=0,NCCL_IB_SL=0,NCCL_IB_GID_INDEX=-1).[5] Write the three defaults into your notes as facts you can quote. - Baseline bandwidth pinned to an explicit device. Server side, then client side:
Expected: a stable BW average near line rate for the link. If it errors on the GID index, drop# host A ib_write_bw -d mlx5_0 -x 3 -F --report_gbits # host B ib_write_bw -d mlx5_0 -x 3 -F --report_gbits <hostA-ip>-x 3and re-readshow_gids. - Repeat with the device chosen for you. Run the same test with a second device (
-d mlx5_1) and compare. On a two-port host this is the smallest possible model of “the rank picked a different rail”. - Exercise the exclude modifier without changing anything persistent. In a shell only:
Expected: the exact-match and exclude forms echo back. This is a shell-scoped variable — nothing persists past the process, so there is no rollback to name.NCCL_IB_HCA='=mlx5_0' env | grep NCCL_IB_HCA NCCL_IB_HCA='^mlx5_0' env | grep NCCL_IB_HCA - Optional, customer lab only. Repeat step 3 across a real rail-aligned leaf and record the hop count from the switch’s own forwarding view. Read-only commands only; do not enable or disable adaptive routing on a live cluster.
Build both cable maps in containerlab and measure the difference in hop count. Nothing here needs an NVIDIA switch.
- Write the topology and fix the addressing plan. One /24 per (rail, host): host N on rail R is
10.R.N.1/24and leaf R holds10.R.N.254/24, so the traceroute targets later in this lab are meaningful.
Expected: the file exists andmkdir -p ~/containerlab && cd ~/containerlab cat > rail-aligned.clab.yml <<'YML' name: rail topology: kinds: linux: { image: frrouting/frr:latest } nodes: spine1: { kind: linux } leaf1: { kind: linux } leaf2: { kind: linux } leaf3: { kind: linux } leaf4: { kind: linux } h1: { kind: linux } h2: { kind: linux } h3: { kind: linux } h4: { kind: linux } links: # rail-aligned: host N interface R -> leaf R - endpoints: ["h1:eth1", "leaf1:eth1"] - endpoints: ["h1:eth2", "leaf2:eth1"] - endpoints: ["h1:eth3", "leaf3:eth1"] - endpoints: ["h1:eth4", "leaf4:eth1"] # ...repeat for h2 (leafR:eth2), h3 (leafR:eth3), h4 (leafR:eth4) - endpoints: ["leaf1:eth9", "spine1:eth1"] - endpoints: ["leaf2:eth9", "spine1:eth2"] - endpoints: ["leaf3:eth9", "spine1:eth3"] - endpoints: ["leaf4:eth9", "spine1:eth4"] YMLsudo containerlab inspect -t rail-aligned.clab.ymlparses it. If not: check that every host’s interface R lands on leafR — that mapping is the whole experiment, and getting it wrong here silently turns step 2 into step 4. - Deploy it and apply the addresses. The topology file only cables the lab; the /24s from step 0 have to be put on the interfaces before any traceroute means anything.
Expected: 9 containers running and every address accepted. If not:sudo containerlab deploy -t rail-aligned.clab.yml # host N, rail R: 10.R.N.1/24 on ethR ; leaf R side: 10.R.N.254/24 for N in 1 2 3 4; do for R in 1 2 3 4; do sudo docker exec clab-rail-h$N ip addr add 10.$R.$N.1/24 dev eth$R sudo docker exec clab-rail-leaf$R ip addr add 10.$R.$N.254/24 dev eth$N done; donesudo containerlab inspect -t rail-aligned.clab.ymland check for address conflicts with an existing lab. - Prove the one-hop property. From host 1’s rail-2 address to host 3’s rail-2 address:
Expected: one intermediate hop (leaf 2). If you see two, your addressing is putting the pair on different leaves — re-read the link list.sudo ip netns exec clab-rail-h1 traceroute -n 10.2.3.1 - Prove the cross-rail cost. From host 1’s rail-2 address to host 3’s rail-3 address:
Expected: three intermediate hops (leaf 2, spine, leaf 3). This is the designed cost of crossing a rail.[2]sudo ip netns exec clab-rail-h1 traceroute -n 10.3.3.1 - Rebuild it node-sequentially. Same file with one change: every host’s four links point at a single leaf N instead of at leaf R, the addressing plan stays identical, and the lab is renamed so both can run side by side.
Expected: three hops for what was one hop before. Nothing failed, nothing logged, the path just got longer — which is the entire teaching point.sed -e 's/^name: rail$/name: nodeseq/' rail-aligned.clab.yml > node-sequential.clab.yml # then edit the links block: h1:eth1-4 -> leaf1, h2:eth1-4 -> leaf2, h3 -> leaf3, h4 -> leaf4 sudo containerlab deploy -t node-sequential.clab.yml # re-apply the same 10.R.N.x plan against the leaf each interface now lands on sudo ip netns exec clab-nodeseq-h1 traceroute -n 10.2.3.1 - Record the defaults you would ask a customer for. On any host with an RDMA stack:
Expected: usually nothing, i.e. all NCCL defaults are in force.[5]env | grep '^NCCL_' ; echo "exit=$?" - Write the three-step escalation checklist in your own words: cabling,
NCCL_IB_HCApinning,NCCL_IB_TC. Keep it to one line each — this is the thing you will say out loud on a bridge call.[5] - Tear down both labs:
sudo containerlab destroy -t rail-aligned.clab.yml --cleanupand the same for the node-sequential file.
Retrieval check
9 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A Dell SE asks why the install guide insists on a wiring map instead of just plugging eight cables into the nearest leaf. Explain in four sentences what rail-optimized means what it buys and what happens if the installer ignores it.
Sources
Facts in this lesson were checked against HGX AI Factory RA (Networking Physical Topologies, Components, Networking Logical Architecture) re-fetched 2026-09-07; DGX SuperPOD B300 Spectrum-4/DC-busbar Network Fabrics re-fetched 2026-09-07; NCCL Environment Variables re-fetched 2026-09-07 (NCCL_IB_HCA format, NCCL_IB_TC/SL default 0, NCCL_IB_GID_INDEX default -1); Network Operator 26.4.0 Spectrum-X page re-fetched 2026-09-07; GB200 NVL72 Key Components page fetched 2026-09-09 (four ConnectX-7 and two BlueField-3 per compute tray). Dates are when each page was fetched.
- Networking Physical Topologies — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
- Network Fabrics — DGX SuperPOD with DGX B300, Spectrum-4 Ethernet and DC Busbar Power RA · fetched 2026-09-07
- Network Fabrics — DGX SuperPOD with DGX B300, Quantum-X800 InfiniBand and AC Power RA · fetched 2026-09-07
- Network Fabrics — DGX GB200 NVL72 SuperPOD Reference Architecture · fetched 2026-09-07
- Environment Variables — NCCL user guide · fetched 2026-09-07
- NVIDIA Spectrum-X Ethernet Networking Platform — Network Operator 26.4.0 · fetched 2026-09-07
- Components — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
- NVIDIA Spectrum SN5600 Series Switches Datasheet (Dell-branded) · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) exam blueprint · fetched 2026-09-07
- Key Components of the DGX SuperPOD — DGX GB200 NVL72 Reference Architecture · fetched 2026-09-09
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.