Skip to content
nodeza.learn
InfiniBand course/5. FAE & cert/Lesson 5.2

Design review: an eight-rail IB pod on Dell XE9680

S5·E2Sixty-four nodes and a quote nobody derived · Dell partner briefing room, two days before the purchase order

S5·E2Create~35 minsources checked todayverified against DGX SuperPOD H200 / B300 / GB200 reference architectures, QM97XX and Q32xx-Q34xx user manuals, Dell XE9680 technical guide and ISM, LinkX ordering pages - source set as fetched 2026-09-07

Builds on: Topologies: fat-tree, rail-optimized, twin-plane, Quantum switches and the LinkX BOM

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Compose a rail-optimized NDR pod for 64 Dell XE9680 nodes with leaf and spine counts derived from NVIDIA's own scalable-unit tables.
  • Assign the eight InfiniBand adapters to XE9680 slots using Dell's published priority order and its DPU and GPU restrictions.
  • Build a cable BOM by length class and justify each copper-to-optics boundary from NVIDIA's ordering tables.
  • Decide where the subnet manager or UFM attaches and state the cost of that decision on NDR versus XDR hardware.
  • Separate design claims that rest on a Dell document from those transferred from an NVIDIA DGX reference architecture.

Episode 2 - Sixty-four nodes and a quote nobody derived

The situation · Dell partner briefing room, two days before the purchase order

The quote is up before anyone sits: 64 PowerEdge XE9680 for the imaging-diagnostics customer, NDR compute fabric, one storage array, growth to two scalable units. Eight adapters per node in slots 32 through 39, no spine line item, every link priced as one 3 m passive cable. The Dell SE has the promise spreadsheet on the second monitor. Procurement asks the only question nobody has answered: how many cables, of which kind, and how soon. At the far end an NVIDIA PM has a roadmap slide and one phrase, “not announced.” The network lead opens a fresh page.

You argue from tables, not taste. Dell publishes the slot priority for a single-port NDR400 card in the NVIDIA-GPU configuration and the rule that goes with it: install the PCIe card in the odd-numbered slots first, followed by the even-numbered slots.[6][7] NVIDIA publishes the shape: eight NDR400 connections per eight-GPU system, rail-aligned so that traffic per rail is always one hop away from the other nodes in a scalable unit, with traffic between nodes or between rails traversing the spine layer.[1][2]

Rail optimization exists because collective traffic is predictable. Land every rail-N port on leaf N and the common case never leaves the leaf, so the spine carries only what must cross it. It is a contract signed by the patch panel months before anyone starts a subnet manager, which is why an under-cabled pod degrades permanently.

Ratios travel between platforms; bills of materials do not.

So you start where the money is: count the node links, then count the cages.[11]

1What transfers from a SuperPOD and what does not

NVIDIA describes the H200 SuperPOD compute fabric as a “Rail-optimized, non-blocking, full fat-tree network with eight NDR400 connections per system”, with a separate NDR storage fabric “optimized to match peak performance of the configured storage array” and fabric management by the UFM Appliance, Enterprise Edition.[1] Rail alignment is defined behaviourally: “Each group of 32 nodes is rail-aligned. Traffic per rail of the DGX H200 systems is always one hop away from the other 31 nodes in a SU. Traffic between nodes, or between rails, traverses the spine layer.”[2] The XDR-era B300 design repeats the shape at 72 nodes per rail-aligned group and adds the twin-plane variant.[3][4]

Four of those ideas are ratios, and ratios transfer to a PowerEdge: eight InfiniBand rails per eight-GPU server; every rail-N port landing on leaf switch N; a storage fabric that “is independent of the compute fabric to maximize performance of both storage and application performance”; and high-speed storage “connected at a 1:1 port to uplink ratio” while node connections are “slightly oversubscribed with a ratio near 4:3”.[4][3]

What does not transfer is the bill of materials. The DGX BOM assumes DGX internals and DGX thermals, and in the NDR design it assumes a sacrificed compute node - the H200 SU table reads 31 nodes because “This is a 32 node per SU design, however a DGX system must be removed to accommodate for UFM connectivity.”[2] On an XE9680 the slot rules, the card types and the DPU restrictions are Dell’s, not NVIDIA’s.

One design rule is worth quoting to a customer verbatim before the purchase order, because it costs money and is always challenged: “if a different node count is required due to budgetary constraints, data center constraints, or other needs, the fabric should be designed to support the full SU, including leaf switches and leaf-spine cables, and leave the portion of the fabric unused where these nodes would be located. This will ensure optimal traffic routing and ensure that performance is consistent across all portions of the fabric.”[5][3]

2The XE9680 slot map, from Dell's own tables

Dell publishes the expansion-card layout precisely. Slots 32 and 33 sit on Riser 4, 34 and 35 on Riser 3, 36 and 37 on Riser 2, 38 and 39 on Riser 1; slots 32 to 35 connect to Processor 2 and 36 to 39 to Processor 1, all full height, half length, x16.[6] Slots 31 and 40 are the NIC, SmartNIC and DPU slots, on Processor 2 and Processor 1 respectively.[6] The technical guide frames the chassis as “10 Gen5 PCIe slots” - “8 x16 Gen5 (x16 PCIe) Full-height, Half-length” plus “2 x16 Gen5 (x16 PCIe) Full-height, Half-length for SmartNIC/DPU” - and notes “8 PCIe Gen5 slots with Intel Gaudi3. Slots 33 and 38 are unavailable due to thermal concerns.”[6]

For a single-port NDR400 InfiniBand card in the NVIDIA-GPU configuration, the published slot priority is 33, 37, 35, 39, 32, 36, 34, 38, 31, 40 with a maximum of ten cards, and the general population rule is “Install the PCIe card in the odd-numbered slots first, followed by the even-numbered slots.”[6][7] Three restrictions turn a quote into a support case if they are missed:

Rule Consequence
“SmartNIC/DPUs with high power consumption (> 75 W), should be installed in slots 31 and 40” Dell’s recommended home for a >75 W DPU is a base-board slot - but Table 20 still lists the 400G 2P SmartNIC/DPU across all ten slots, so quote this as Dell’s recommendation, not as a hard block
“PowerEdge XE9680 system with MI300X GPUs does not support SmartNIC/DPUs” No DPU line item at all in that configuration
“PowerEdge XE9680 system with Gaudi3 GPUs supports 8 PCIe slots, as slots 33 and 38 are blocked” The eight-rail map has to be redrawn

Dell also states the mixing rule: “The card type with the smaller quantity (≤2 cards) must be installed in Slot 31 and Slot 40 first. The remaining card type then fills the other available slots following the standard SPM order.”[6][7] For the classic eight-rail build - eight ConnectX-7 NDR400 plus two BlueField-3 - that rule and the power rule agree: the DPUs take 31 and 40, and the eight IB cards take the riser slots in priority order.

Here is the gap you must declare. The technical guide maps risers to processors; it does not publish which GPU each slot is affine to.[6] Rail affinity to a specific GPU is asserted in NVIDIA’s DGX documents, not in Dell’s. Any rail map you hand a customer must label GPU-to-slot affinity as unverified for the XE9680 or, better, measure it in the lab with lspci -tv and each device’s numa_node.

a cabling contract, not a switch feature
Rail-optimized fat tree — one slice: 3 nodes × 4 rails, 4 leaf, 2 spineleaf→spine: optics across the rownode→leaf: DAC ≤ 2 m / ACC 3–5 mS0spine (root)OSFP cagesS1spine (root)OSFP cagesL0leaf — rail 0OSFP cagesL1leaf — rail 1OSFP cagesL2leaf — rail 2OSFP cagesL3leaf — rail 3OSFP cagesnode n0 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112node n1 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112node n2 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112
2 SU
Path

Click one HCA for the source, another for the destination. Hops are counted as switch ASICs traversed — a link count is hops + 1.

Rail-optimized fat tree

  • Every GPU's rail-N port lands on leaf N: "Traffic per rail of the DGX H200 systems is always one hop away from the other 31 nodes in a SU. Traffic between nodes, or between rails, traverses the spine layer."
  • Same rail, different node = 1 switch hop. Different rail = 3 hops through a spine.
  • H200 SuperPOD compute fabric: QM9700 NDR, "Rail-optimized, non-blocking, full fat-tree network with eight NDR400 connections per system".
  • Nothing is enabled on the switch to get this. It is decided by where the technician plugs the cable.

FAE angle. On an eight-rail PowerEdge XE9680 the rail map is the deliverable. Dell's slot priority for NDR400 1P cards is 33, 37, 35, 39, 32, 36, 34, 38, leaving 31 and 40 for the storage NIC or a BlueField-3 — DPUs over 75 W are legal only in 31/40. Check it against ibdev2netdev and each card's numa_node, then validate with ibdiagnet --rail_validation (→ ibdiagnet2.rails).

DGX H200 SuperPOD RA — Network Fabrics

What each link class costs in LinkX parts
  • Node-to-leaf runs stay inside or beside the rack; leaf-to-spine runs cross the row. That distinction is the whole cable BOM.
  • NDR copper reach in NVIDIA's own catalogue: passive DAC MCP4Y10-Nxxx up to 2 m (0.5 / 1 / 1.5 / 2 m), active copper MCA4J80-Nxxx at 3, 4 and 5 m. Anything longer is optics.
  • Twin-port OSFP is the arithmetic trap: one cage carries "two transceiver engines… creating 800Gb/s electrical to the switch and 2x400G optics". One MCP4Y10 is one 800G switch-to-switch link; one MCP7Y00 1:2 splitter is two 400G node links. Count OSFP cages, not ports.
  • Top type is thermal, not preference: finned-top at the air-cooled switch, flat-top (-FLT) at DGX and liquid-cooled ends, -FTF ("flat to finned") for an asymmetric run.
  • Fiber rules: multimode "up to 100-meters straight and 50-meters for splitters"; both fibers of a twin-port link same type and roughly the same length — 4.5 ns/m of delay.
  • A QM9700 gives 64 NDR 400 Gb/s ports over 32 OSFP cages (or 128 × NDR200 split); a Q3400-RA gives 144 × 800 Gb/s over 72 OSFP cages.
mlxlink -d <mst_dev> -p 1 -m       # vendor, PN, length, temperature
ibdiagnet --get_cable_info         # fabric-wide → ibdiagnet2.cables

LinkX twin-port OSFP overview

Scalable-unit arithmetic — 2 SU

DGX H200 SuperPOD compute fabric — QM9700 NDR

Sources: opensm(8) · UFM SM defaults · MLNX-OS Subnet Manager · SHARP environment · Quantum-X800 switches · MCA4J80 ACC · ibdiagnet · ibdiagnet dump files · Dell XE9680 technical guide

Two scalable units, rail-optimized, with the cable overlay on. Pick a source and destination to see which pairs are one hop and which cross the spine layer.

3Sizing: from 64 nodes to leaf and spine counts

The customer ask: 64 XE9680, NDR, one storage array, growth to two scalable units. Start from the node, because the rail count is a property of the server - eight NDR400 connections per eight-GPU system.[1] That fixes 64 times 8, or 512 node-to-leaf links.

Now the switch. A QM9700 presents “64 ports of NDR 400Gb/s InfiniBand per port in a 1U standard chassis” over 32 OSFP cages.[8] A non-blocking leaf splits its radix in half: 32 ports down to nodes and 32 up to spines. So one leaf serves 32 nodes on one rail, eight rails need eight leaves per 32 nodes, and 64 nodes need 16 leaves. Total uplinks are 16 times 32, or 512, and at 64 ports per spine that is 8 spines.

Check the arithmetic against NVIDIA’s published component table rather than trusting it: the H200 SU table gives 1 SU as 31 nodes with 8 leaf and 4 spine, 2 SU as 63 nodes with 16 leaf and 8 spine, and 4 SU as 127 nodes with 32 leaf and 16 spine.[2] Sixteen leaves and eight spines is exactly the two-SU row, which is the confirmation you want before quoting.

The row also carries the awkward number: 63, not 64. The RA gives up exactly one compute node, at any size - the table reads 31, 63, 95 and 127 - so that UFM has fabric connectivity.[2] On a Dell pod you have three honest options: accept 63 compute nodes plus a UFM host, add a sixty-fifth server for UFM, or move to hardware where the problem does not exist, which is the next segment.

Storage is a separate fabric, not a partition of this one.[4] Size it against the array at 1:1 port-to-uplink, and expect the node side to sit near 4:3 rather than 1:1.[3] The B300 RA runs its InfiniBand storage fabric on QM9700 NDR switches even when compute is XDR, with parts MQM9700-NS2R or MQM9701-NS2R, which is a useful precedent when a customer asks why their storage fabric is a generation behind compute.[4][3]

4The cable BOM, and the three ways it goes wrong

Count cages, not ports. Quantum-2 and the matching optics use “Twin port OSFP cages supporting two transceiver engines in a single OSFP form-factor plug creating 800Gb/s electrical to the switch and 2x400G optics using two MPO/-12/APC optical connectors.”[11] One twin-port switch cage therefore serves two 400G node ports through a 1:2 splitter, or one 800G switch-to-switch link. For the pod above that means 512 node links land on 256 leaf-side cages, and 512 leaf-spine links land on 256 twin-port runs.

Copper reach sets the boundaries. The twin-port NDR passive DAC family MCP4Y10-Nxxx is published at 0.5 m, 1 m, 1.5 m and 2 m, with flat-top variants for liquid-cooled and DGX ends.[12] The active copper family MCA4J80-Nxxx is published at 3 m, 4 m and 5 m, including a flat-top part and an -FTF “flat to finned” part for an asymmetric run.[13] So: intra-rack node-to-leaf is usually passive DAC, adjacent-rack runs are active copper, and anything crossing a row is optics with fibre and MPO patching in the same quote.[11]

The thermal rule is not a preference. “This requires the use of flat-top transceivers, ACCs, and DACs in the DGX H100. The 400G IB/EN switches require finned-top 2x400G transceivers for additional cooling due to the reduced air flow inlets in the switches.”[11] A flat-top part in a switch cage fits mechanically and then runs hot.

Two fibre rules for the optical runs: both fibres of a twin-port transceiver must be the same type, “both straight or both splitters, and not mixed”, and should be approximately the same length “to avoid inducing different latency delays in the fibers (4.5ns/meter)”; NVIDIA supplies multimode fibre “up to 100-meters straight and 50-meters for splitters”.[11]

Declare what you cannot source. The XDR-generation LinkX ordering tables for InfiniBand-protocol 1600G optics were not retrievable in this research; the only 1600G optic captured is Ethernet-only.[11] If the customer’s growth path is XDR, the NDR BOM above is sound and the XDR BOM is a line item to confirm with NVIDIA rather than a table to copy.

5Where the SM lives, and what you write in the risks section

The management decision comes before the switch model. QM9700 and QM9701 are internally managed with an on-board subnet manager under MLNX-OS; QM9790 is externally managed and “can utilize the advanced NVIDIA Unified Fabric Manager (UFM) feature sets”.[8] QM9790 also has no console, no management Ethernet port and no USB - front I2C only - so choosing it commits the customer to UFM or a host SM and to out-of-band access by another route.[8]

On NDR the UFM attachment costs a node, as the H200 SU table shows.[2] On Quantum-X800 it does not: all those switches “include a dedicated OSFP InfiniBand in-band management port specifically for NVIDIA UFM (Unified Fabric Manager), separated on the front panel from the other ports. This separation allows the full set of standard ports to be used for data network connectivity”, and the B300 RA connects “UFM 3.5 nodes … to four (4) FNM ports on the Q3400 switches.”[10][3] That is a concrete, quotable reason for a customer weighing NDR against XDR beyond bandwidth.

Then write the risks section, and make it specific. For this design it contains at least five entries. GPU-to-slot rail affinity on the XE9680 is not published by Dell and is asserted only for DGX systems.[6] No Dell InfiniBand reference architecture or validated design was retrievable, so every ratio in segments 1 and 3 is transferred from an NVIDIA DGX RA and should be confirmed with a Dell-opened document before it appears in a customer deliverable. Port-to-port latency figures for Quantum-2 and Quantum-X800 appear on no fetched NVIDIA page, so quote bandwidth and radix and refuse to quote nanoseconds.[8][10] The XDR LinkX ordering tables are unconfirmed.[11] And the aggregate-throughput number has two official forms - “51.2 terabits per second (Tb/s)” bidirectional in the introduction against “Switching: 25.6Tbps” in the specification table - so state which one you mean.[8][9]

Finally, pre-load the answer to the ticket this pod will generate on day one: iDRAC will show ConnectX-7 InfiniBand link speed as “Single Data Rate (SDR)” because the UEFI and PXE drivers configure the link to SDR when the cable supports it, and Dell’s own KB says this “is expected behavior and is cosmetic in the iDRAC … the card functions at and reflects Next Data Rate (NDR) speeds within the operating system.”[14] Put the KB number and the ibstat proof in the handover pack, not in the first escalation.

Qualification request · decision 1/6A 0 · P 0 · S 0

Brief — Email, Tuesday: "We want to offer the BlueField-3 B3140H SuperNIC in the XE9780 for Spectrum-X customers. Can NVIDIA qualify it? I need a plan by Friday."

Dell platform PM (17G AI servers): So — can you get it qualified? What do you need from us to start?

Scoping drill. The same discipline the design review needs - pin the versions, the slots and the power envelope before committing to a plan - scored on accuracy, positioning and safety.
The design package

Ask: 64 XE9680 with NVIDIA GPUs, NDR compute fabric, one storage array, growth to two scalable units.

Rail map. Eight rails per node, one single-port ConnectX-7 NDR400 per rail, rail-N to leaf-N.[1][2] Slots in Dell’s priority order: rail 1 to slot 33, rail 2 to 37, rail 3 to 35, rail 4 to 39, rail 5 to 32, rail 6 to 36, rail 7 to 34, rail 8 to 38.[6] Slots 31 and 40 reserved for the storage or management NIC, or for BlueField-3 if the customer wants one; Dell says DPUs above 75 W should be installed there.[6] Label the map: GPU-to-rail affinity unverified for this platform, to be measured.[6]

Switch count. 512 node links; a non-blocking QM9700 leaf is 32 down and 32 up out of 64 ports; 16 leaves; 512 uplinks; 8 spines.[8] Cross-check: NVIDIA’s 2 SU row is 16 leaf and 8 spine.[2]

Cable BOM by length class. Node to leaf inside the rack: a 1:2 splitter from one leaf twin-port cage to two single-port ConnectX-7 node ports - MCP7Y00-Nxxx passive DAC, published at 1 m, 1.5 m, 2 m, 2.5 m and 3 m.[16] Leaf to spine: twin-port switch-to-switch parts - MCP4Y10 passive DAC up to 2 m, MCA4J80 active copper at 3 m to 5 m.[12][13] Leaf to spine across the row: optics, finned-top on both switch ends, matched fibre lengths.[11] Count 256 leaf-side cages for the node links and 256 twin-port leaf-spine runs.[11]

Storage. Separate QM9700 fabric, 1:1 at the array, node side near 4:3.[4][3]

SM and UFM. QM9700 managed switches give an on-board SM option; if the customer wants UFM, budget either a 65th server or accept 63 compute nodes as the RA does.[8][2] Note the XDR alternative: Q3400 FNM ports remove this cost entirely.[10][3]

Risks. Rail affinity unverified; all ratios transferred from DGX RAs; no latency numbers quoted; XDR cable tables unconfirmed; Quantum-2 throughput quoted as 51.2 Tb/s bidirectional and 25.6 Tb/s unidirectional.[6][9] Plus the day-one iDRAC SDR KB in the handover pack.[14]

What went into the risks section

How it ended

The review ends with 16 leaves and 8 spines, derived from the rail count and the switch radix, then cross-checked against NVIDIA’s two-SU row, which reads 16 leaf and 8 spine.[8][2] The cable lines are split by reach - passive DAC inside the rack, active copper out to 5 m, optics with finned-top parts at the switch ends for anything crossing a row.[12][13][11] The risks section names the GPU-to-slot rail affinity Dell does not publish.[6] What you tell the network lead: “Every number in here comes from a table, and the rest are risks with an owner.” The order signs. Six weeks later the pod is racked, every cable labelled by hand, and your phone rings at 02:10 on the third night of a training run that will not move.

Lab

Read-only verification on a Dell-lab XE9680, or the closest available PowerEdge. Nothing here changes firmware, mode or configuration - every command is a query.

Pre-flight inventory: record the service tag, the GPU family, the installed card list and the current slot population from iDRAC before you start, so your annotations reference a fixed machine state.

  1. lspci -tv and note the PCIe tree: which cards hang off which root complex, and how that maps to the two processors.[6]
  2. ibdev2netdev and ibstat -l to list the IB devices present and their names in the order the OS enumerates them - which is not the order Dell’s slot table uses.
  3. For each adapter, read cat /sys/class/infiniband/<dev>/device/numa_node. Expected: values that group the cards into two sets matching the Processor 1 and Processor 2 riser split.[6] If they do not group cleanly, record it - this is exactly the rail-affinity gap the design must declare.
  4. Compare iDRAC’s inventory view of each slot against the physical card in it, and against Dell’s published slot-to-riser table. Expected: agreement on riser and processor; the guide publishes nothing about GPU affinity, so make no claim there.[6]
  5. If iDRAC reports an InfiniBand link speed of SDR, confirm the real rate in the OS with ibstat and ibstatus and record both. Expected: SDR in iDRAC and the negotiated NDR rate in the OS, exactly as Dell KB 000221452 describes.[14]
  6. Annotate your paper design with every place it disagrees with the physical machine, and mark each disagreement as either a design error or an undocumented platform behaviour. Optional, in a customer lab: walk the design against the rack elevation, confirm airflow direction matches the switch SKUs ordered, and confirm the power input type - a QM9701 takes a DC busbar and has no replaceable PSUs.[9]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell account team, in five sentences, why a PowerEdge InfiniBand pod can copy a SuperPOD's ratios but not its bill of materials.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DGX SuperPOD H200 / B300 / GB200 reference architectures, QM97XX and Q32xx-Q34xx user manuals, Dell XE9680 technical guide and ISM, LinkX ordering pages - source set as fetched 2026-09-07. Dates are when each page was fetched.

  1. DGX SuperPOD RA featuring DGX H200: Key Components · fetched 2026-09-07
  2. DGX SuperPOD RA featuring DGX H200: Network Fabrics · fetched 2026-09-07
  3. SuperPOD with DGX B300 RA: Network Fabrics · fetched 2026-09-07
  4. SuperPOD with DGX B300 RA: Key Components · fetched 2026-09-07
  5. DGX SuperPOD RA featuring DGX GB200: Network Fabrics · fetched 2026-09-07
  6. Dell PowerEdge XE9680 Technical Guide (Regulatory Model E90S) · fetched 2026-09-07
  7. Dell PowerEdge XE9680 Installation and Service Manual: Expansion card installation guidelines · fetched 2026-09-07
  8. QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
  9. QM97XX User Manual: Specifications · fetched 2026-09-07
  10. NVIDIA Q32xx and Q34xx XDR 800Gb/s InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
  11. MMA4Z00-NS 800Gb/s Twin-port OSFP transceiver: Overview · fetched 2026-09-07
  12. MCP4Y10-Nxxx passive DAC: Ordering Information · fetched 2026-09-07
  13. MCA4J80-Nxxx active copper cable: Ordering Information · fetched 2026-09-07
  14. Dell KB 000221452: ConnectX-7 InfiniBand link speed shows as SDR in iDRAC · fetched 2026-09-07
  15. SuperPOD with DGX B300 RA: DGX SuperPOD Architecture · fetched 2026-09-07
  16. MCP7Y00-Nxxx twin-port 2x400G OSFP to 2x400G OSFP passive DAC splitter: Ordering Information · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.