PowerScale back-end networking and the version-pin trap
S3·E4The two switches procurement wanted back · a design review, screen shared, the storage architect dialled in
Builds on: Dell's switch catalog: SN, Q and Z in one price list, Reading a real Dell AI Factory BOM
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Explain why the PowerScale back-end network is a cluster backplane rather than a data network and what Dell forbids on it.
- Read the supported back-end switch matrix and decide whether a given node and speed combination is allowed.
- Analyse the Cumulus Linux version-pin conflict between the PowerScale back end and the Spectrum-X validated stack.
- Produce the discovery evidence that proves which pin a customer is actually running.
Episode 4 - The two switches procurement wanted back
The drawing on the screen is elegant and it cannot ship. Procurement’s saving is on it: one SN5600 pair carrying the GPU fabric and, on its own VLANs, the PowerScale back end. One SKU, one spare, one NOS image. The storage architect asks a single question: which Cumulus version.
There is no answer that works. The PowerScale document’s footnote for that switch reads “please ensure the Spectrum 4 switches are running the version Cumulus Linux 5.9.1. This is the version that Dell Technologies tested and qualified.”[1] The Spectrum-X validated solution stack pins the switch NOS at 5.18.1 for the AI fabric.[2] One switch runs one version, so whichever number goes on the box, one role is outside the statement published for it.
And the version was never the first blocker. Dell writes it in one flat sentence: “Dell does not support connecting any other devices to the back-end switches.”[1] That rule exists because this is not a storage network in the sense a GPU customer means. It is the cluster’s own backplane - the thing that makes several appliances behave as one file system - carrying “node-to-node communication with a private, high-speed, low-latency network”.[1] A neighbour on that switch shares its buffers, its congestion events and its maintenance windows.
The night-shift operator has already labelled two spare ports int-a and int-b, which is more documentation than the drawing carries. A backplane is not a network you may borrow. The elegant design dies on a sentence, not on a benchmark, so start with the sentence.
1A backplane that happens to use Ethernet
The PowerScale back-end network is not a storage network in the sense a GPU customer means. Dell describes it as the network that “acts as the backplane” for the cluster, “configured with redundant switches for high availability”, providing “node-to-node communication with a private, high-speed, low-latency network”.[1] Every design rule that follows is a consequence of that one sentence.
The rules are unusually blunt for a vendor document. “Dell does not support connecting any other devices to the back-end switches.”[1] The Ethernet back-end option “is only supported in clusters that consist entirely of Ethernet back-end nodes”, and in those configurations “only Ethernet back-end switches that are provided and managed by Dell are supported”.[1] On configuration: “Customers should not attempt to alter the back-end network configurations provided by Dell. Any attempt to do so can result in a cluster-wide outage.”[1]
The addressing is fixed too. The back end is built from an int-a network for node-to-node communication plus an optional int-b and failover pair for redundancy; all of them must be IPv4 and must sit on separate, distinct subnets.[1] Data transport on Ethernet is RBM over TCP, replacing the legacy InfiniBand pairing of SDP for data and IPoIB for management, with the TCP stack modified for cluster traffic while conforming to the standard.[1] From OneFS 9.10 an InfiniBand back end is supported again on F210, F710 and F910, so “back end” no longer implies Ethernet on its own.[1]
Two operational facts matter on a bridge call. Back-end switch monitoring covers field-replaceable units - power supply, fan and similar - and “Protocol and performance monitoring capability is not provided”; alerts reach the customer by SNMP through CELOG.[1] And there is no subnet-manager topology file to read, so the way to see what is attached to the fabric is isi_dump_fabric int-a or isi_dump_fabric int-b.[1]
2The support matrix, read as a constraint
The document publishes a thirteen-row table of switches supported for back-end connectivity, with a native switch speed and a list of supported speeds per model.[1] Read it as a fence, not a menu - and read the whole fence, because the last three rows continue after a page break and a switch that is EOL-but-listed is not the same thing as a switch that was never supported.
| Vendor | Model | Native speed | Supported speeds | EOL |
|---|---|---|---|---|
| Dell | Z9664 | 100 GbE | 25, 100 GbE | - |
| Arista | 7308X3 | 100 GbE | 25, 100 GbE | - |
| NVIDIA | Spectrum-4 SN5600 | 800 GbE | 200 GbE | - |
| Dell | S5232-ON | 100 GbE | 10, 25, 40, 100 GbE | - |
| Dell | Z9264-ON | 100 GbE | 10, 25, 40, 100 GbE | - |
| Dell | Z9100-ON | 100 GbE | 10, 25, 40, 100 GbE | 1/31/2023 |
| Dell | S4112F-ON | 10/100 GbE | 10, 25, 100 GbE | - |
| Dell | S4148F-ON | 10 GbE | 10/100 GbE | 5/5/2023 |
| Celestica | D4040 | 40 GbE | 40 GbE | 3/31/2021 |
| Arista | DCS-7308 | 40 GbE | 25, 40, 100 GbE | 5/5/2023 |
| Celestica | D2024 | 10 GbE | 10, 40 GbE | 5/5/2023 |
| Celestica | D2060 | 10 GbE | 10, 40 GbE | 3/31/2021 |
| Arista | DCS-7304 | 10 GbE | 10, 40 GbE | 5/5/2023 |
Note the shape of the SN5600 row: native 800 GbE, supported 200 GbE.[1] Native speed describes the silicon - the same 64-cage, 51.2 Tb/s Spectrum-4 platform Dell sells for AI fabrics - while supported speed describes the attachment Dell qualified for PowerScale nodes.[1][4]
The node side has its own rules. Back-end card options are 200 GbE QSFP56 on F910 and F710; 40/100 GbE QSFP56 on F910, F900, F710, F600, F210, F200, P100 and B100; 40/100 GbE QSFP28 on F900, F600, H700, H7000, A300 and A3000; and 10/25 GbE SFP28 on F210, F200, F600, H700, H7000, A300, A3000, P100 and B100.[1] Two constraints bite in real quotes: “The F910, F710, F210, F200, P100, and B100 nodes do not support a 25 GbE back-end connection if they are configured with 100 GbE front-end connectivity”, and for an all-performance cluster “the back end must be 100/40 GbE (10 GbE and 25 GbE are not supported)”.[1] Mixed clusters run a 100 GbE infrastructure with 8x10 GbE or 4x25 GbE breakout cables to the archive nodes.[1]
3The version pin, and why it cannot be waved away
Here is the footnote that runs the rest of this lesson, verbatim from the table: “Spectrum-4 SN5600 switch is supported via the Dell Technologies ETC program, please consult Dell Technologies account team for more details. You need to configure Spectrum-4 manually to set up for a PowerScale cluster. Refer to NVIDIA documentation for details about the installation instruction. And please ensure the Spectrum 4 switches are running the version Cumulus Linux 5.9.1. This is the version that Dell Technologies tested and qualified.”[1]
Four separate facts hide in that footnote: the support route is the ETC program and not the ordinary catalog, the configuration is manual rather than delivered by Dell, the installation instructions belong to NVIDIA, and the software version is pinned to 5.9.1.[1]
Now put it next to the other pin. The NVIDIA Spectrum-X validated solution stack publishes rows of matched component versions; as of v2.3.1 in September 2026 the row names Cumulus or SONiC 5.18.1, ConnectX-8 firmware 40.50.1002, BlueField-3 firmware 32.50.1002, DOCA-Host 3.5.0-082, NetQ 5.1.0 and NCCL 2.30.7.[2] That is the switch NOS version an AI fabric is expected to run.[2]
One switch runs one NOS version. If a customer proposes a single SN5600 pair serving both the GPU fabric and the PowerScale back end, then at 5.9.1 the AI fabric is off the validated stack and at 5.18.1 the storage back end is off Dell’s qualified version - and the isolation rule already ruled the arrangement out before software entered the conversation.[1][2]
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
4Proving the pin, and the three ways out
You cannot resolve this from a slide. You resolve it by producing three strings and one sentence: the switch NOS version, the role that switch serves, the node and back-end card model, and a statement of which published pin governs.[1][2]
Firmware: Dell DUP vs NVIDIA mlxfwmanager / bf-fwbundle — which one wins?
The ways out are three, and only three.
Separate switch pairs per role. This is what Dell’s own published AI Factory solution does: the AI fabric is 3x SN5600 carrying a 400G back end plus a 200G front end, while the PowerScale storage cluster network is a separate pair of PowerSwitch S5232F-ON at 100G.[5] S5232-ON sits on the supported back-end list at 100 GbE native with 10, 25, 40 and 100 GbE supported, so the storage pair is inside the matrix without touching the AI fabric’s version.[1] It costs two switches and removes the entire argument.
Re-qualification through the account team. The footnote routes SN5600 back-end support through the ETC program and tells you to consult the Dell account team; that is where a request to qualify a different version belongs.[1] It is not a field decision and not an NVIDIA decision.
An explicit, recorded customer decision. If a customer chooses to run outside one of the pins, the design note says which pin, which role it affects and who accepted it. What is not acceptable is a silent upgrade: bringing an SN5600 to 5.18.1 during an AI-fabric maintenance window when that switch also serves PowerScale changes a support status without anybody deciding to.[1][2]
Two adjacent facts keep the conversation honest. Ethernet storage fabrics on Cumulus default to an MTU of 9216 bytes, set with nv set interface <interface-list> link mtu <mtu>, which is the AI-side storage network and not the PowerScale backplane.[6] And the Enterprise RA requires storage from the NVIDIA-Certified Storage program, described as “a comprehensive validation framework that ensures storage platforms meet stringent performance, quality, and interoperability standards for AI workloads” - the fetched page names no partners or protocols, so asking which certification a quoted platform carries is the correct move rather than asserting one.[7]
After the review
The design gains two switches and loses an argument. The PowerScale back end moves onto its own qualified pair, which is exactly what Dell’s own published AI Factory solution does - 2x PowerSwitch S5232F-ON at 100G for the storage cluster while the SN5600s carry the fabric, with S5232-ON sitting on the supported back-end list at 100 GbE native.[5][1] What you actually say: “You cannot converge these two, and the isolation rule ends it before the version pin even comes up.”
At 02:10 the operator calls. A leaf has been dropping links for three days, Dell is telling him to call NVIDIA, NVIDIA is telling him to call Dell, and the freeze is eleven days out.
Given. A Dell SE proposes: 4x XE9780 with B300, one SN5600 pair for everything, 4x PowerScale F710 with 100 GbE front end and 25 GbE back end, upgrading the switches to the current Spectrum-X stack.
Step 1 - separate the roles. Two roles are being asked of one switch pair: AI compute and converged fabric for the GPU nodes, and the PowerScale cluster backplane.[1]
Step 2 - test the isolation rule first. “Dell does not support connecting any other devices to the back-end switches.”[1] That alone ends the shared-switch proposal, before any version is discussed. Record it as blocker 1.
Step 3 - test the node build. F710 with 100 GbE front-end connectivity cannot take a 25 GbE back end.[1] Blocker 2, and it is in the storage quote rather than the network quote - which is why it is usually found late.
Step 4 - test the version pin. SN5600 for PowerScale is pinned at Cumulus Linux 5.9.1 through the ETC program.[1] The Spectrum-X validated stack v2.3.1 pins 5.18.1 for the AI fabric.[2] Upgrading to the AI stack would take the storage role outside Dell’s qualified version. Blocker 3.
Step 5 - propose the resolution Dell itself uses. Split the roles: SN5600 for the AI fabric at the validated-stack version, and a dedicated pair for the PowerScale back end from the supported list - S5232-ON at 100 GbE is what the published Dell AI Factory solution uses in that role.[5][1] Fix the node build to a supported back-end speed.[1]
Step 6 - name the owner of what remains. Back-end SN5600 support runs through the Dell Technologies ETC program and the account team, so any request to qualify a different version goes there, not to a case and not to NVIDIA.[1]
Deliverable. A one-page note: two boxes, the version each requires with its source, three blockers, and one recommended split with the alternative routes named.
Same method, new proposal. A customer has an existing SN5600 pair running Cumulus 5.9.1 serving a PowerScale cluster and wants to attach 8 new GPU nodes to spare ports on those switches. Fill the blanks.
- Roles requested of the switch pair: ____ and ____.[1]
- Does the isolation rule permit the new attachment? ____ Quote the sentence: ____.[1]
- If the isolation rule were somehow satisfied, what version would the AI fabric require? ____ Source: ____.[2]
- What happens to the storage role at that version? ____.[1]
- Which support path owns the decision to change the back-end version? ____.[1]
- Write the recommendation in one sentence, naming the hardware you would add: ____.[5]
A Dell account team forwards you this: “Customer has 6x PowerScale F910 with 100 GbE front end, wants the back end at 25 GbE to reuse existing optics, and wants us to reuse the SN5600 pair from their AI pilot which is currently on the Spectrum-X validated stack. They need an answer today.”
Produce a written response containing: (a) every published rule the request violates, each with the verbatim sentence and its source; (b) the two version strings in play and which document pins each; (c) the back-end speed that is actually supported for an all-performance F910 cluster, with the source; (d) a recommended configuration naming a specific switch model from the supported list; (e) the single question you would send to the Dell account team, phrased so that it can be answered without a lab; and (f) an explicit list of anything you could not verify from a published page.
Acceptance criteria: no claim about a specific storage platform’s NVIDIA certification status; every version number carries the document that pins it; the isolation rule appears before any performance argument; and the response distinguishes what is unsupported from what is merely unqualified.[1][2][7]
Lab
Not reproducible on the Dell lab hardware - there is no PowerScale cluster and no Spectrum switch here. What is reproducible is the discovery you would run on site, so rehearse it read-only on the equipment you do have. No configuration changes, no firmware updates.
- Pre-flight inventory. Record host model, service tag and the adapters present, because a version conversation is worthless without knowing which box it describes.
Expected: service tag, part numbers, PSIDs and firmware versions. If not:racadm getsysinfo sudo mst start && sudo mst status -v sudo mlxfwmanager --queryracadmmay not be installed - read the service tag from the iDRAC dashboard orsudo dmidecode -s system-serial-number;mst status -vprints nothing when the driver is not loaded, so runsudo mst startagain and checklsmod | grep mst. Keep this output - lesson 5 attaches it to a support case. - Write down, without running them, the three commands you would use on a customer site to prove the pin: the switch NOS version, the PowerScale node and back-end card model, and the fabric membership listing.[1] Expected: a switch version command such as
nv show systemon Cumulus, an OneFS node listing, andisi_dump_fabric int-a. Mark which of the three you are allowed to run yourself and which belong to the storage team - that boundary is the lesson. - Practise the read on your own kit: capture the link speed and transceiver identity per port and note how the transceiver alone changes the negotiated speed.
Expected: negotiated speed plus the module identifier. If not:sudo ethtool ens1f0np0 | head -12 sudo ethtool -m ens1f0np0 | head -12ethtool -mreturns “Cannot get module EEPROM information” on an empty cage or on some DAC cables - record what is physically attached instead, since a copper cable identifier makes the same point about speed following the media. Connect this to the document’s statement that the same NIC covers two speeds and the change is achieved with different transceivers or cables.[1] - Assemble a version table for the lab as if it were a customer: host firmware, adapter firmware, driver or DOCA version, and the validated-stack row you would compare it to.[2] Expected: a four-row table with one clearly marked gap or none, and an explicit note where a version could not be read. If not: if
ofed_infoand the DOCA packages are both absent, the host is running an inbox driver - write “inbox, version frommodinfo mlx5_core” rather than leaving the row empty, because “no DOCA installed” is itself a comparison against the validated row. - Deliverable: the rehearsed discovery script as a text file, the lab version table, and one sentence naming which support path would own each gap you found.[1][3]
Goal: build the design note and the support matrix so that both are ready before a customer asks.
- Draw the version-pin conflict as a one-page note: two boxes labelled AI compute fabric and PowerScale back end, the NOS version each requires, the exact source sentence under each, and the three ways out.[1][2] Expected: 5.18.1 under the fabric box, 5.9.1 under the storage box, and the three routes named as separate switch pairs, re-qualification through ETC, or a recorded customer decision. If not: if you cannot find a verbatim sentence under one of the boxes, do not paraphrase - a version pin with no quotable sentence behind it will not survive the first challenge in a design review.
- Rebuild the supported back-end switch matrix as a table with vendor, model, native speed, supported speeds and EOL date, and mark every EOL row.[1] Expected: thirteen rows, of which five carry a 2023 EOL date (Z9100-ON 1/31/2023, and S4148F-ON, DCS-7308, D2024 and DCS-7304 all 5/5/2023) and two carry 3/31/2021 (Celestica D4040 and D2060), plus the SN5600 row carrying a footnote rather than a plain entry. If your table stops at ten rows you missed the three that continue on the following page.
- Write the node-side rules as three one-line tests you can apply to a quote: the 25 GbE with 100 GbE front-end exclusion, the all-performance 100/40 GbE rule, and the mixed-cluster breakout pattern.[1] Expected: three sentences, each traceable to a printed sentence in the document. If not: if a test of yours depends on a card datasheet rather than on the PowerScale document, drop it - what will link and what Dell will support are different questions, and only the second one is on trial here.
- Open the stack checker above and record the reference row for September 2026.[2] Expected: the ten component versions, with the switch NOS entry circled - that is the only line that collides with PowerScale. If not: if the stack checker shows a different reference row than the one in this lesson, NVIDIA has published a newer stack - record the new row and its date rather than forcing it to match, because the pin moving is the normal case.
- Take Dell Solution 19845005.1 and mark which BOM line resolves the conflict.[5] Expected: the S5232F-ON pair, and a sentence explaining that its presence is a design decision rather than an accident. If not: if you marked the SN2201 line instead, re-read the roles - out-of-band management is a third network again, and it is not what isolates the storage backplane.
- In containerlab, model the shape rather than the product: bring up two Linux or SONiC nodes as a redundant pair and two more as cluster members on separate
int-aandint-bsubnets, and confirm that traffic on one subnet cannot reach the other.
Expected: two isolated IPv4 subnets and no route between them. If they can reach each other, your topology has a shared bridge - fix it before drawing conclusions, because the isolation is the whole point.sudo containerlab deploy -t backend.clab.yml sudo containerlab inspect -t backend.clab.yml - Deliverable: the one-page design note, the annotated support matrix, and the three quote tests.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell storage SE in four sentences why a customer cannot run their PowerScale back end and their GPU fabric on the same pair of SN5600 switches.
Sources
Facts in this lesson were checked against Dell PowerScale Ethernet Back-End Network Overview H16346.8 (March 2025) re-fetched and read in full 2026-09-09; NVIDIA Spectrum-X Validated Solution Stack and Dell AI Networking Switches page, 2026-09-07. Dates are when each page was fetched.
- Dell PowerScale: Ethernet Back-End Network Overview (H16346.8, March 2025) · fetched 2026-09-09
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- AI Networking Switches | Dell USA · fetched 2026-09-07
- NVIDIA Spectrum SN5600 Series Switches Datasheet (Dell-hosted) · fetched 2026-09-07
- Dell AI Factory with NVIDIA Solution ID 19845005.1 · fetched 2026-09-07
- Cumulus Linux Configuration Guide for Ethernet Storage Fabrics · fetched 2026-09-07
- NVIDIA-Certified Storage - NVIDIA HGX AI Factory · fetched 2026-09-07
- Dell-NVIDIA Partnership Powers High-Performance AI Fabric Solutions · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Who's who in an InfiniBand subnetInfiniBand course · Same ground: addressing, tools and design
- GIDs: how a connection picks its RoCE versionRoCE course · Same ground: addressing, tools and version-pin
- Scenario: a 256-GPU Dell AI Factory design reviewElsewhere in this course · Same ground: powerscale, version-pin and design