Rail-optimized fabrics on Dell hardware

1A rail is a lane, not a link

NVIDIA states the rail rule in one sentence: “NIC-0 from each DGX system is connected to the same leaf switch (L0), NIC-1s are connected to the same leaf switch (L1), and so on”, and the reason is that this “helps maximize all-reduce performance while minimizing network interference between flows”.^[1] The DGX SuperPOD B200 reference architecture applies the same rule at scale: “each group of 32 nodes is rail-aligned”, so node-to-node traffic inside a rail is one hop and only cross-rail traffic reaches the spine.^[3] The same document adds the design rule that keeps a partial deployment honest: the fabric “should be designed to support the full SU, including leaf switches and leaf-spine cables”, even when only part of it is bought.^[3]

Rails create a second problem that PXN solves. PXN expands to “PCI x NVLink”: GPUs reach a NIC “through NVLink and then PCI” instead of through the CPU, which “couldn’t deliver full bandwidth”.^[1] Functionally, PXN “leverages NVIDIA NVSwitch connectivity between GPUs within the node to first move data on a GPU on the same rail as the destination, then send it to the destination without crossing rails”, which removes those flows from the spine entirely.^[1] It also aggregates: PXN can “wait until all eight sends are ready, then send all eight messages at one time”, which is where NVIDIA’s Figure 3 shows “more than 2X improvement” in all2all latency.^[1]

The knob that tells NCCL your fabric is rail-optimized is NCCL_CROSS_NIC. Its default is 2 - “Try to use the same NIC for the same ring/tree, but still allow for the use of different NICs if it would result in a better performance.” Value 0 is “Always use the same NIC for the same ring/tree, to avoid crossing network rails. Suited for networks with per NIC switches (rails)”, and value 1 is for “networks where all NICs from a node are connected to the same switch”.^[5] A customer who built rails and left NCCL_CROSS_NIC alone has not told the library what they built.

2On an XE9680 the rail is decided in the slot

The Dell PowerEdge XE9680 “supports up to eight PCI express (PCIe) Gen 5 expansion cards installed on PCIe Switch Board (PSB), and two PCI express (PCIe) Gen 5 expansion cards installed on PCIe Base Board (PBB)”.^[2] Those ten slots are not interchangeable, because each one is wired to a specific socket.^[2]

Three placement rules sit on top of that map. First, power: “SmartNIC/DPUs with high power consumption (greater than 75 W), should be installed in slots 31 and 40. SmartNIC/DPUs with normal power consumption (less than 75 W), can also be installed in slots 32-39.”^[2] Second, mixing: “When a system is configured with two different types of high-speed NIC/DPU cards (for example, 2x BF3 and 8x CX-7)… The card type with the smaller quantity (2 cards or fewer) must be installed in Slot 31 and Slot 40 first”, and Dell says the rule “applies to all high-speed NIC/DPU card types, including but not limited to BF3, CX-7, and CX-6”.^[2] Third, order: the published slot priority for a Mellanox (SmartNIC/DPU: 400G, 2P) card alongside NVIDIA GPUs is 33, 37, 35, 39, 32, 36, 34, 38, 31, 40, maximum ten cards, and Dell adds the general note “Install the PCIe card in the odd-numbered slots first, followed by the even-numbered slots”.^[2] A 200G 2P SmartNIC/DPU is restricted to slots 31 and 40 with a maximum of two.^[2]

Two chassis-level exclusions belong in the same conversation: the Gaudi3 variant blocks slots 33 and 38 “to improve airflow and address thermal concerns”, and the MI300X variant “does not support SmartNIC/DPUs” at all.^[2]

3Proving the pairing on the machine

nvidia-smi topo -m “Displays a matrix of connections between all GPUs and NICs (including their data-direct devices if applicable) in the system along with CPU/memory affinities for the GPUs”.^[4] The legend is the whole diagnosis: X is Self, PIX is “Connection traversing a single PCIe switch”, PXB is “Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)”, PHB traverses “a PCIe Host Bridge (typically the CPU)”, NODE traverses “the interconnect between PCIe Host Bridges within a NUMA node”, and SYS is “Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)”.^[4] Best to worst for GPUDirect RDMA runs PIX, PXB, PHB, NODE, SYS - a reading of that legend, not a published ranking.^[4] NCCL_NET_GDR_LEVEL accepts most of the same tokens, LOC, PIX, PXB, PHB and SYS but not NODE, so the matrix and the variable are read together.^[5]

Two more reads finish the picture. cat /sys/class/net/<if>/device/numa_node follows the netdev’s link to its PCI device and reads the kernel’s numa_node attribute - “the NUMA node to which the PCI device is attached, or -1 if the node is unknown” - which must match where the ranks run.^[13] And NCCL_IB_HCA is what pins the RDMA data path to specific HCAs, in the format <hca>[:<port>[:<rail>[:<plane>]]], comma separated, with ^ to exclude and = to require an exact match, up to a maximum of 32 HCAs.^[5] Setting NCCL_SOCKET_IFNAME instead only moves the bootstrap interface, not the RDMA traffic.^[5]

Inside containers and VMs the matrix can lie: NVIDIA warns to “Ensure /sys is properly mounted in containers and virtual machines, as it provides PCI topology discovery. Virtual PCI topology exposure can degrade performance significantly.”^[7] NVIDIA’s GPUDirect RDMA documentation states the constraint that kills GPUDirect regardless of slot: it “currently relies upon all physical addresses being the same from the different PCI devices’ point of view”, and “This makes it incompatible with IOMMUs performing any form of translation other than 1:1, hence they must be disabled or configured for pass-through translation for GPUDirect RDMA to work.”^[12]

4What a rail design does not prove

Three things a Dell-account FAE will be asked about are not sourceable today, and saying so is part of the deliverable.

The first is automated rail validation. ibdiagnet is documented as having a “Rail Optimized Topology” validation that “checks that all HCAs nodes connected to the same Top-of-Rack switch have the same PCIe address (BDF) in the corresponding server”, but the page 404s on both docs.nvidia.com and its networking-docs redirect target, so that wording is a search snippet and nothing more; it is also an InfiniBand tool, so it does not cover a RoCE leaf.^[8] Treat it as a lead to check, not a capability to promise.

The second is the Kubernetes-side rail model. The Network Operator v26.7.0 index confirms that CRDs named SpectrumXRailPoolConfig, NicConfigurationTemplate and NicInterfaceNameTemplate exist, but the per-field description of what the rail-pool CRD configures came from a search snippet rather than the fetched page.^[9] Name the CRD; do not describe its fields from memory.

The third is Dell BIOS. The 16G “PowerEdge BIOS Performance and Workload Tuning Guide” is behind a reCAPTCHA interstitial for automated fetch, which means every BIOS value in a rail recommendation - System Profile, NUMA Nodes Per Socket, memory interleaving, IOMMU, PCI ASPM - has to be read from iDRAC on the machine before it goes on a page.^[11] The XE9680 slot map and the 75 W rule are the NUMA-relevant facts that are on a fetched Dell page, which is exactly why this lesson is built on them.^[2]

For the exam, this material sits in Domain 1 (AI Data Center Design and Optimization), weighted at 5%, whose objectives are to “Describe an AI factory networking architecture and its components”, “Describe rail-optimized topologies for high-performance AI workloads” and “Describe GPU-to-GPU communications”.^[10] The Spectrum-X platform page supplies the component vocabulary that objective 1.1 expects: BlueField-3 SuperNICs in the Hopper era, ConnectX-8 at 800 Gb/s with Blackwell, and ConnectX-9 at 1,600 Gb/s per GPU with Vera Rubin.^[6]