Topologies: fat-tree, rail-optimized, twin-plane

1Rail optimization is a cabling contract

NVIDIA describes the H200 SuperPOD compute fabric as a rail-optimized, non-blocking, full fat-tree network with eight NDR400 connections per system.^[2] The behaviour that phrase buys is stated exactly: each group of 32 nodes is rail-aligned, traffic per rail of the DGX H200 systems is always one hop away from the other 31 nodes in a SU, and traffic between nodes, or between rails, traverses the spine layer.^[1]

The same sentence reappears at every generation with different numbers. At B300 and XDR scale: each group of 72 nodes is rail-aligned, traffic per rail of the DGX B300 systems is always one hop away from the other 72 nodes in a SU, and traffic between SUs, or between rails, traverses the spine layer.^[3] At GB200 rack scale: each compute rack is rail-aligned, traffic per rail of each compute tray is always one hop away from other compute trays in the same scalable unit, and traffic between different compute racks, or between different rails, traverses the spine layer.^[6]

Nothing in those sentences is a switch setting. Rail alignment is produced by where the technician plugs each cable: rail 0 of every node to leaf 0, rail 1 to leaf 1, and so on. To OpenSM the result is an ordinary fat tree, and no SM parameter reports it. That is why it belongs in the cabling map and the acceptance test, not in a configuration review.^[1][9]

2The published sizing tables

Reproducing the tables is the exercise, because a customer conversation about scale is a conversation about switch and cable counts.

DGX H200, NDR, QM9700. The compute fabric per SU is 31 nodes and 248 GPUs on 8 leaf and 4 spine switches, with 252 compute-and-UFM cables and 256 spine-leaf cables. Two SU is 63 nodes and 504 GPUs on 16 leaf and 8 spine, with 508 compute-and-UFM cables and 512 spine-leaf; three SU is 95 nodes and 760 GPUs on 24 leaf and 16 spine, with 764 and 768; four SU is 127 nodes and 1,016 GPUs on 32 leaf and 16 spine with 1,020 plus 1,024 cables.^[1] The footnote is the part people miss: this is a 32 node per SU design, however a DGX system must be removed to accommodate for UFM connectivity.^[1]

DGX B300, XDR, Quantum-X800 Q3400-RA. The SU is 72 nodes and 576 GPUs at 4 DGX B300 per rack and roughly 56 kW per rack.^[5] The component table reads: 1 SU, 8 leaf, 4 spine, 576 node-leaf and 576 leaf-spine cables; 2 SU, 16 and 8, 1,152 plus 1,152; 4 SU, 32 and 18, 2,304 plus 2,304; 8 SU, 64 and 36, 4,608 plus 4,608; 18 SU, 144 leaf, 72 spine, 10,368 node-leaf and 10,368 leaf-spine cables for 1,296 nodes and 9,216 GPUs.^[5] Two things to quote carefully. The 4 SU spine column reads 18, not 16 - copy the published row rather than “fixing” it. And 1,296 nodes at eight GPUs each would be 10,368 GPUs while the table’s GPU column reads 9,216; quote the RA and flag the arithmetic instead of silently choosing one.^[5] The RA itself is focused on 8 SU with 576 DGX nodes and states that DGX SuperPOD can scale to much larger configurations, up to and beyond 72 SU with 2,000 plus DGX B300 nodes.^[5]

DGX GB200, rack scale. The unit of scale is the spine-leaf-group: each SU contains 4 SLGs to match the number of IB rails, which equals the number of GPUs per compute tray, and there are 8 leaf switches, one for each compute rack, and 6 spine switches in each SLG, allowing a fully non-blocking fat tree for each SU to be attached to 6 core groups, scalable up to and including 16 SUs.^[6] The published scale-out rows: 1,152 GPUs at 2 SU gives 3 switches per core group, 18 core switches, 64 leaf and 48 spine; 2,304 GPUs at 4 SU gives 6, 36, 128 and 96; 4,608 GPUs at 8 SU gives 12, 72, 256 and 192; 9,216 GPUs at 16 SU gives 24 per core group, 144 core switches, 512 leaf and 384 spine.^[6]

3Partial builds, and what asymmetry breaks

The design rule for a customer who cannot fill a scalable unit is published and unambiguous: if a different node count is required due to budgetary constraints, data center constraints, or other needs, the fabric should be designed to support the full SU, including leaf switches and leaf-spine cables, and leave the portion of the fabric unused where these nodes would be located; this will ensure optimal traffic routing and ensure that performance is consistent across all portions of the fabric.^[6][3]

The cost of ignoring it is a routing story, not a bandwidth story. A shrunken fabric is an asymmetric tree, and the fat-tree engine’s rules are stated in terms of symmetry: same-rank switches must have identical numbers of up-going and down-going port groups and identical port counts per group, and all the CAs have to be at the same tree level.^[9] Break those and ftree declines, OpenSM moves to the next engine in the -R list, and without no_fallback that is eventually MinHop.^[9] The customer sees no error - they see job-to-job timing that varies with which part of the cluster the scheduler picked.

NVIDIA does ship parameters for the asymmetric case - ar_tree_asymmetric_flow = 1, ar_tree_asymmetric_flow_threshold = 0 and ar_tree_asymmetric_flow_threshold_limit = 2 - and their existence is a hint about how common the situation is.^[10] They are not a licence to design an asymmetric tree on purpose; they help adaptive routing cope with one.^[10]

Two commands make this checkable rather than arguable. ibdiagnet --rail_validation checks topology being rail optimized and writes ibdiagnet2.rails; --ft produces a fat-tree analysis and --dfp a DragonFly+ analysis, with --ft_roots_regex_opt and --dfp_roots_regex to name the roots.^[11][12]

ibdiagnet --rail_validation      # is the cabling actually rail optimized
ibdiagnet --ft                   # fat-tree analysis
ibdiagnet -r                     # routing report, credit loops

4Twin-plane, and where the published record stops

Twin-plane is the XDR-era shape. The B300 compute fabric is described as rail-optimized, non-blocking, twin-plane, fat tree topology for next-generation extreme-scale AI factory, built on the NVIDIA Quantum-X800 Q3400-RA at 800 Gbps.^[4] The design requirements section adds that the compute fabric is a rail-optimized, full-fat tree topology and that managed Quantum-X800 switches are used throughout the design to provide better management of the fabric.^[4]

What the fetched pages do not publish is how the two planes are wired. The common reading is two independent planes, each with its own spines, so a port on one plane cannot reach the other through the compute fabric - but that model is an inference, not an NVIDIA statement. Say “as drawn” when you present it, and do not put plane wiring in a customer design document as if it were quoted.^[4]

Two ceilings anchor the scale conversation. On Quantum-X800, the switches’ high radix supports a two-level fat-tree topology capable of connecting up to 10,368 network interface cards with minimal latency and optimal job locality.^[8] On Quantum-2, the platform claims over one million 400 Gb/s nodes in a four-switch-tier (three hops) DragonFly+ network, 6.5 times higher than the previous generation.^[13] Between them sits the QM9700 positioning: the family enables small to medium-sized deployments to scale with a two-level fat tree topology while reducing power, latency and space requirements.^[7] Quantum-2 switches also enable a variety of topologies including Fat Tree, SlimFly, DragonFly+, multi-dimensional Torus and more, which is the sentence to quote when a customer asks whether a shape is supported at all.^[7]

Fabric management changes shape here too, and it changes a rack elevation. On NDR the RA gives up a whole compute node for UFM connectivity; on Quantum-X800, UFM 3.5 nodes are connected to four FNM ports on the Q3400 switches, the dedicated in-band management port each switch carries so that the full set of standard ports stays available for data.^[1][3][8]

mlxlink -d <mst_dev> -p 1 -m       # vendor, PN, length, temperature
ibdiagnet --get_cable_info         # fabric-wide → ibdiagnet2.cables

5Doing the arithmetic in front of a customer