Skip to content
nodeza.learn
Spectrum-X course/6. FAE + cert/Lesson 6.3

Scenario: sizing, quoting and defending a Dell AI pod fabric

S6·E3The airflow question that stopped the room · Customer design review, a windowless conference room, Monday, day of the BOM sign-off

S6·E3Create~35 minsources checked todayverified against Dell AI switch catalogue, NVIDIA and Dell SN5000/SN6000/SN2201 datasheets, SN5000 ordering information, Spectrum-X validated stack v2.3.1, Dell Enterprise SONiC 4.4.0 matrix, 2026-09-07

Builds on: Choosing the switch: SN2201 through SN6800-LD, The RA row: version pinning across switch, NIC and host, Dell PowerSwitch SN-series: who supports what

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Design the three fabrics of a Dell AI pod and assign a switch model to each with the reason it was chosen.
  • Build a bill of materials that survives facility review: airflow direction, power feed, cooling and NOS per switch.
  • Pin every component of the design to one Spectrum-X reference-architecture row and say what happens when a customer leaves it.
  • Defend the design against the three standard objections and state the support boundary between Dell and NVIDIA.

Episode 3 — The airflow question that stopped the room

The situation · Customer design review, a windowless conference room, Monday, day of the BOM sign-off

The second pod is on the screen and the network lead is happy with it, notebook open at Friday’s three reasons. Procurement wants price and lead time, and is waiting for line items. Then the facilities engineer, silent all morning, asks the two questions that stop the meeting: which way does the air move through these, and is that a busbar rack or a PDU rack. The Dell SE looks at you. The quote survives this or goes back a week.

The answers came from the ordering table, not the datasheet headline. The SN5400 ships in two airflow forms, 920-9N42C-00RB-7C0 as C2P and 920-9N42C-00FB-7C0 as P2C, and the wrong one is a return rather than a setting.[4] The SN5600D takes a DC busbar at 40 to 60 VDC, which is a rack-integration commitment.[6][2] And the AC SN5600 OPN 920-9N42F-00RI-7C0 is flagged End of Life, which is why the design names the SN5610 instead[4] — procurement, who only asks two things, is right that lead time cannot be quoted on a part that is leaving.

Reference architectures exist for this reason: the features being bought are negotiated end to end across switch OS, NIC firmware, host software and the collective stack, so NVIDIA publishes one validated row per release and tells customers to deploy on it rather than on a list of latest versions.[8]

A line item without an airflow direction and a power feed is not a line item — it is a return. Segment 1 starts above the boxes, with the three fabrics the pod needs.

1Three fabrics, three jobs

A Dell AI pod is not one network. Dell’s own AI switch catalogue sells the Spectrum family alongside the Broadcom PowerSwitch line and the InfiniBand SKUs, which is your first clue that the customer is choosing per fabric, not per vendor: the Spectrum entries are SN6800-LD and SN6810-LD and SN6600-LD on Spectrum-6, SN5610 and SN5600D and SN5400 on Spectrum-4, SN4700 on Spectrum-3 and SN2201.[1]

The split that matters commercially is backend, frontend and out-of-band. The backend or east-west GPU fabric is the one that justifies Spectrum-X at all, and it lands on 800G Spectrum-4 (SN5600 or SN5610 at 51.2 Tb/s over 64 OSFP ports) or on the Spectrum-6 line for a customer refreshing to 102.4 Tb/s per ASIC.[2][6] The frontend and storage fabric usually does not need 800G per port, which is where the SN5400 sits with 64 QSFP-DD 400GbE plus 2 SFP28 at 25.6 Tb/s.[3] Out-of-band management is the SN2201: 48 RJ45 plus 4 QSFP28 at 448 Gb/s, positioned by NVIDIA as an out-of-band management switch or a ToR for up to 48 1G Base-T host ports with non-blocking 100GbE spine uplinks, drawing around 100 W typical.[5]

A word on the reference split you will hear quoted. A Dell Info Hub description of a PowerEdge XE9680 cluster puts the frontend on two Spectrum-4 SN5600, the backend on one or more SN5600 and out-of-band on an SN2201, but that page returned HTTP 403 when fetched and the claim survives only as a search snippet, so treat it as an unverified shape rather than a Dell-published reference you can cite in a quote.[1] The three-fabric structure itself is safe; the specific model counts are the customer’s design decision.

ProductSpeedPCIeRoleGPU generation
NIC
SuperNIC (no Arm)
SuperNIC (no Arm)
DPU
DPU
SuperNIC (Arm inactive)
DPU / storage processor
Ethernet switch
Ethernet switch
InfiniBand switch
InfiniBand switch

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

Use the products view to keep the roles straight while you draft the fabric list.

2The constraints that kill a bill of materials

Performance rarely eliminates a switch. Facilities do. Four constraints kill more BOMs than any benchmark.

Airflow direction. The SN5000 ordering table lists the SN5400 in two forms, 920-9N42C-00RB-7C0 with C2P airflow and 920-9N42C-00FB-7C0 with P2C, while the 800G members appear with C2P: 920-9N42F-00RI-7C0 for the AC SN5600, 920-9N42F-00RI-KC0 for the SN5600D and 920-9N42F-00RI-3C1 for the SN5610.[4] C2P is connector-to-power and P2C is power-to-connector; the datasheets describe the SN5600 airflow as reverse and the SN5400 as reverse or forward.[2][3] Get this wrong and the remedy is a return, not a setting.

End of life. That same table flags the AC SN5600 OPN 920-9N42F-00RI-7C0 as End of Life.[4] NVIDIA describes the SN5610 as introducing a modified hardware specification that optimises system power consumption and enhances efficiency while sharing identical scale and connectivity with the SN5600, which is exactly the sentence to put in the quote when the customer asks why the model changed.[2]

Power feed. The SN5600D takes a DC busbar at 40 to 60 VDC and is described as ideal for DGX SuperPOD offerings; the SN6000 line lists 48 VDC or 54 VDC busbar with a permissible 40 to 60 VDC range.[2][6] A busbar is a rack-integration commitment, not a networking preference.

Cooling. The SN6800-LD is 5RU and liquid-cooled with ten UQD8 v2 connectors, the SN6810-LD is 2U with four, and the SN6600-LD is 2U with two, all with a coolant temperature range of 17 to 45 degrees C.[6] Dell’s spec sheet adds an air-cooled SN6600 at 3RU with a 256 MB packet buffer, 220 to 240 VAC and an operational ambient of 0 to 30 degrees C, ASHRAE A3 — tighter than the SN5600’s 0 to 35 degrees C.[7][2] A warm room can eliminate the air-cooled option by itself.

Two honesty notes for the BOM. The Cumulus Linux 5.18 release notes reference an SN5640 that has no fetched datasheet, so if a customer asks about it, say the model appears in release notes and its specifications are not something you can quote yet.[13] And no Dell page fetched carries a Dell SKU for any SN-series switch; the only part numbers available are NVIDIA OPNs, so Dell part numbers are a question for the account team.[1][4]

3One RA row, one NOS, one support lifetime

A Spectrum-X quote that names versions loosely is a support incident waiting to happen. The Spectrum-X Validated Solution Stack publishes one row per RA release across three hardware tracks, and NVIDIA’s own guidance is to deploy clusters in a configuration where all components are on the most up-to-date validated configuration version.[8] The September 2026 row, v2.3.1, pins Cumulus Linux 5.18.1, ConnectX-8 firmware 40.50.1002, DOCA-Host 3.5.0-082, NetQ 5.1.0, NCCL 2.30.7, HPC-X 2.51, Network Operator 26.7.0 and DTS 1.26.5, with the H200 track carrying BlueField-3 firmware 32.50.1002 instead of the ConnectX-8 line.[8] Put the row number in the quote, not a list of latest versions.

The switch-OS column of that page lists only Cumulus Linux; there is no SONiC row in any RA.[8] That is a factual boundary, not a preference. NVIDIA does support Pure SONiC on Spectrum with an ASIC-to-protocol support model covering ASIC, firmware, SDK, SAI and SONiC, and lists RoCE, ISSU and WJH among the supported features, so SONiC on Spectrum is a real option — it is simply not the option the Spectrum-X reference architecture validates.[10][8] Community SONiC’s supported-platform list includes the SN5600 on Spectrum-4 and no Spectrum-6 platform at all.[14]

Software lifetime belongs in the quote too. Cumulus Linux ships a mainline branch with 12 months of NVIDIA support from the date of the LTS release, and LTS lines with three years from release carrying security updates and critical bug fixes; 5.9.z and 5.11.z are the named LTS lines.[12] A customer buying a five-year pod needs at least one planned upgrade in the plan, and the RA row is what they upgrade to.

◐ Level 2 — limited (same annual cycle)
  • Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
  • This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
  • ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).

Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

Pick a track and read the whole row. A quote that names one component's version and leaves the rest to chance is the quote that becomes an escalation.

4The support boundary and the three objections

“Who do I call?” is the question the Dell rep actually needs answered, and a clean table beats a paragraph.

Layer Owner Evidence
Hardware, RMA, entitlement Dell, sold as PowerSwitch SN-series with Dell’s service catalogue Dell’s AI switch catalogue and the Dell-branded datasheets[1][2]
Cumulus Linux software NVIDIA: release lifetime and support policy are NVIDIA’s, and Dell’s own install KB reproduces the NVIDIA procedure Dell KB 000376999 and the NVIDIA support policy[11][12]
Enterprise SONiC on Dell PowerSwitch Dell Dell’s Enterprise SONiC compatibility matrix[9]
SN6000 purchased from NVIDIA Minimum one year NVIDIA Enterprise Business-Standard Support required NVIDIA SN6000 datasheet[6]

Now the three objections, each answered with a citable fact rather than a position.

“We already run Dell Enterprise SONiC.” Every platform in the Dell Enterprise SONiC 4.4.0 compatibility matrix uses a Broadcom NPU — Tomahawk 5, 4, 3 and 2 and Trident 4 and 3 — and no NVIDIA Spectrum platform is listed.[9] The bundles also gate features: RoCEv2 is supported on Cloud Standard, Cloud Premium, Enterprise Standard and Enterprise Premium and not on Lite, and VXLAN EVPN appears on Enterprise and Lite bundles rather than the Cloud ones.[9] Dell’s SN6000 spec sheet does substitute Enterprise SONiC Distribution by Dell Technologies for NVIDIA’s Pure SONiC entry, so the direction of travel is real — but the 4.4.0 platform list is the fact to quote today.[7]

“We will just buy Tomahawk 5.” Z9864F-ON at 51.2 Tbps is a legitimate box and Dell sells it.[1] The honest answer is not that it is slower; it is that adaptive routing is documented as an NVIDIA Spectrum-X platform capability on Spectrum-4 at 400G and 200G, on layer-3 default-VRF interfaces, for RoCEv2 and VXLAN-encapsulated RoCE, and it is armed end to end with the SuperNIC.[15] Those features do not move to a different ASIC by changing the NOS.

“We will upgrade Cumulus for that bug fix.” That is how a pod leaves the RA row. Answer with the row and with the lifetime: the RA row is the tested combination, and LTS lines exist precisely so a customer can take critical bug fixes without leaving a supported line.[8][12]

Positioning: three customers, one afternoon · decision 1/7A 0 · P 0 · S 0

Brief — Back-to-back calls: a VMware farm (200× R760), an AI training pod (64× XE9780 on Spectrum-X) and a storage-heavy multi-tenant inference cluster. Each has objections.

Dell account team + three end customers: VMware farm: "200 R760s on vSphere 8; NSX eats about 20% of our cores. Should we go DPU?"

Run the positioning scenario. Each customer objection scores on technical accuracy, positioning and safety - over-promising costs you as much as being wrong.

5Produce the deliverable

The output of this lesson is a document you could hand to a Dell account team tomorrow.

Fabric design and BOM: worked, faded, problem

Requirement. 32 PowerEdge GPU nodes, 8 east-west ports per node at 400G, plus frontend and storage, plus management. Facility answers received: hot aisle requires reverse (C2P) airflow, racks are AC 200-240 V, inlet air 24 degrees C, no facility water.

Fabric design.

Fabric Requirement Choice Why
Backend GPU east-west 256 x 400G host ports plus spine uplinks SN5610 leaves, 800G ports broken out to 400G 64 OSFP 800GbE at 51.2 Tb/s; the SN5610 is the current AC 800G SKU after the SN5600 AC OPN went End of Life[2][4]
Frontend and storage 400G, moderate port count SN5400 64 QSFP-DD 400GbE plus 2 SFP28 at 25.6 Tb/s, and it is the model with a documented P2C option if a second room disagrees on airflow[3][4]
Out-of-band 1G Base-T to every node BMC, 100G uplinks SN2201 48 RJ45 plus 4 QSFP28, positioned by NVIDIA as an OOB management switch, about 100 W typical[5]

Topology and arithmetic. Rail-optimised: one leaf per GPU rail, so 8 leaves. Each leaf takes one 400G port from all 32 nodes - 16 of its 64 OSFP cages broken out 2x400G, since the SN5610 yields 128x400G from 64 ports[4][2] - which is 12.8 Tb/s down, matched by 16 native 800G uplinks. Two spines absorb 8x800G from each of the 8 leaves, filling 64 ports each. That is 10 SN5610 in total, non-blocking, with 32 leaf cages still free for growth.

Bill of materials.

Qty Model OPN Airflow Power Cooling NOS
8 SN5610 (leaf) 920-9N42F-00RI-3C1 C2P 4 AC PSUs (2+2), 200-240 VAC Air, 0-40 C Cumulus Linux 5.18.1
2 SN5610 (spine) 920-9N42F-00RI-3C1 C2P 4 AC PSUs (2+2), 200-240 VAC Air, 0-40 C Cumulus Linux 5.18.1
2 SN5400 920-9N42C-00RB-7C0 C2P AC PSUs Air Cumulus Linux 5.18.1
2 SN2201 see Dell account team Forward or reverse per SKU 100-264 VAC Air Cumulus Linux

Dell part numbers are deliberately absent: no Dell page fetched publishes a Dell SKU for any SN-series switch.[1]

Version pin. Every component lands on Spectrum-X RA v2.3.1: Cumulus Linux 5.18.1, NetQ 5.1.0, DOCA-Host 3.5.0-082, and the NIC firmware line for the track the customer’s GPU platform sits on.[8]

Support ownership. Hardware and RMA to Dell; Cumulus Linux software and release lifetime to NVIDIA; the LTS-versus-mainline decision belongs in the operations plan because mainline carries 12 months of support and LTS three years.[11][12]

What would change my recommendation.

  1. If facility water at 17 to 45 degrees C with UQD8 v2 exists and the roadmap is 800G-per-GPU, the SN6000-LD line becomes viable and the whole power design changes to busbar.[6]
  2. If the room’s inlet temperature can exceed 30 degrees C, the air-cooled SN6600 is out on its ASHRAE A3 ceiling.[7]
  3. If the customer requires Dell Enterprise SONiC as the NOS, the design moves to Dell Broadcom-based PowerSwitch hardware and the Spectrum-X feature set is off the table.[9][15]

Episode 3 — What goes in the quote

How it ended

The revised BOM leaves the room with an airflow direction and a power feed per line item, an RA row number instead of a version list, and a one-page support table naming who is called for hardware, for Cumulus Linux and for SONiC.[8][11][9] The Tomahawk objection gets an eligibility answer rather than a benchmark, because adaptive routing is a Spectrum platform capability armed end to end with the SuperNIC.[15] What you say to the facilities engineer: “Every line here has a direction, a feed and a cooling method written next to it — check me.”

In the corridor the network lead asks the one question you cannot cite a page for: the third hall they are planning is InfiniBand, so who signs off that you know that side too?

Lab

Pre-flight inventory. In the Dell lab, record for the rack you have access to: measured airflow direction relative to the hot aisle, power feed type and voltage, inlet air temperature at the top and bottom of the rack, available rack units, and whether any facility water connection exists. Photograph the PDU label and the rack layout. All steps below are observation only; nothing is installed, moved or rewired.

  1. Airflow. Determine the direction air actually travels through the installed equipment and which side faces the hot aisle. Expected: a single consistent direction per row. If not: a mixed row is itself a finding and it decides which OPN suffix you can order.[4]
  2. Power. Record the feed: AC voltage and phase, or busbar voltage if present. Compare against the SN5600’s 200-240 VAC, the SN5600D’s 40-60 VDC and the SN6000 line’s 48/54 VDC busbar.[2][6]
  3. Thermal. Log inlet temperature over a working day. Compare against the SN5600’s 0 to 35 C, the SN5610’s 0 to 40 C and the air-cooled SN6600’s 0 to 30 C ASHRAE A3 ceiling.[2][7]
  4. Verdict. Write one line per SKU in your BOM: could it be installed in this rack today, yes or no, with the constraint that decides it. Expected: at least one model is eliminated by the room rather than by price.
  5. Optional (customer lab, read-only). Validate two BOM assumptions on live hardware: the installed NOS (nv show system version on Cumulus, or show version on a SONiC box) and the support entitlement on the service tag. Rollback: none needed; both are read-only. Record the answers as facts with a date, because both change without anyone telling the FAE.[11]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell account executive, in five sentences, why three facility questions can change your switch recommendation more than any performance number.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against Dell AI switch catalogue, NVIDIA and Dell SN5000/SN6000/SN2201 datasheets, SN5000 ordering information, Spectrum-X validated stack v2.3.1, Dell Enterprise SONiC 4.4.0 matrix, 2026-09-07. Dates are when each page was fetched.

  1. AI Networking Switches | Dell USA · fetched 2026-09-07
  2. NVIDIA Spectrum SN5600 Series Switches Datasheet (Dell-hosted, doc 3546804, JUL25) · fetched 2026-09-07
  3. NVIDIA Spectrum SN5000 Series Switches Datasheet (NVIDIA document, distributor mirror) · fetched 2026-09-07
  4. Ordering Information | NVIDIA Spectrum-4 SN5000 2U Switch Systems Hardware User Manual · fetched 2026-09-07
  5. NVIDIA Spectrum SN2201 Switch Datasheet (Dell-hosted) · fetched 2026-09-07
  6. NVIDIA Spectrum SN6000 Ethernet Switch Series Datasheet (doc 4664025, JAN26) · fetched 2026-09-07
  7. Dell PowerSwitch SN6000 Ethernet Switch Series spec sheet · fetched 2026-09-07
  8. NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
  9. Dell Enterprise SONiC Distribution Compatibility Matrix Release 4.4.0 · fetched 2026-09-07
  10. NVIDIA and SONiC · fetched 2026-09-07
  11. Dell KB 000376999: Dell Networking Spectrum-X Cumulus Linux, installing a new image with ONIE · fetched 2026-09-07
  12. Cumulus Linux Release Versioning and Support Policy · fetched 2026-09-07
  13. Cumulus Linux 5.18 Release Notes · fetched 2026-09-07
  14. Supported Devices and Platforms | SONiC (sonic-net) · fetched 2026-09-07
  15. Equal Cost Multipath Load Sharing (including Adaptive Routing) | Cumulus Linux 5.18 · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.