The RA row: version pinning across switch, NIC and host
S1·E4The bridge call about one bug fix · Bridge call, 16:00, the customer, the Dell SE and you
Builds on: What Spectrum-X is, and what it is not
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Read a Spectrum-X Validated Solution Stack row and name every component it pins.
- Select the correct hardware track for a given cluster from its GPU platform and SuperNIC.
- Judge whether a proposed upgrade keeps a cluster on-row and say what a Cumulus-only bump leaves untested.
- Defend a version recommendation to a customer using only what the published pages state.
Episode 4 — The bridge call about one bug fix
Twenty people on the line and one question: can we upgrade only the switches. The customer wants a fix that landed in Cumulus Linux 5.18.1, their leaves run 5.16.5, and the change window is Saturday. Procurement asks whether the answer costs anything, then mutes.
The rest of the inventory arrives in pieces while you listen: BlueField-3 firmware 32.47.x on the hosts, DOCA-Host 3.3.0-088826, Network Operator 26.1.0. That is the H200 track, sitting around RA 2.1.4 and already inconsistent with itself.[1] The row they would be moving toward, RA v2.3.1, pins Cumulus 5.18.1 together with BlueField-3 firmware 32.50.1002, DOCA-Host 3.5.0-082, Network Operator 26.7.0 and DTS 1.26.5 - one bundle, not one column.[1]
The row exists because the behaviour they bought is not implemented in one device. Adaptive routing depends on the switch choosing a port and the NIC reordering the result, and congestion control depends on both ends agreeing what a probe means, so a switch-only bump leaves the fabric in a combination nobody tested.[10][1] Then the two sentences nobody wants before a Saturday window: being on-row means tested together, not defect-free, and 5.18.1 is a mainline release rather than an LTS one.[3][5]
On Spectrum-X you do not upgrade a component, you move a row.
Segment 1 is what that row actually is.
1What a validated stack is
NVIDIA publishes a Spectrum-X Validated Solution Stack page that is the single compatibility matrix for the platform, split into three hardware tracks: GB300, B300 and H200.[1] Its own guidance is explicit: “We strongly advise deploying clusters in a configuration where all components are operating on the most up-to-date Validated Configuration version for optimal performance and compatibility.”[1]
The important mental move is that the RA number is not a switch-OS version. It is a bundle number, and the components carry it in their own release notes - ConnectX-8 firmware v40.48.1000 states that “this ConnectX-8 firmware version is tested as part of Spectrum-X reference architecture release version 2.1”.[6] On the Kubernetes side it is machine-readable: the NicConfigurationTemplate CRD carries an explicit RA version field.[4]
Two absences on the page matter as much as its contents. There is no NVIDIA Air or DSX Air row - simulation is not a validated component - and there is no separate SuperNIC firmware row, because SuperNIC firmware is the ConnectX-8 or BlueField-3 column.[1]
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2The current row, and the one before it
| Component | RA 2.1.6 (Aug 2026) | RA v2.3.1 (Sep 2026) |
|---|---|---|
| Cumulus Linux | 5.16.6[1] | 5.18.1[1] |
| ConnectX-8 firmware | 40.48.1146[1] | 40.50.1002[1] |
| BlueField-3 firmware (H200) | 32.48.1000[1] | 32.50.1002[1] |
| DOCA-Host | 3.3.0-088826[1] | 3.5.0-082[1] |
| NetQ | 5.1.0[1] | 5.1.0[1] |
| NCCL | 2.29.2[1] | 2.30.7[1] |
| HPC-X | 2.26[1] | 2.51[1] |
| Network Operator | 26.1.0[1] | 26.7.0[1] |
| DTS | 1.24.3[1] | 1.26.5[1] |
A re-fetch of the page on 2026-09-07 confirms the same v2.3.1 values on all three tracks, with the GB300 and B300 tracks carrying ConnectX-8 firmware and the H200 track carrying BlueField-3 firmware 32.50.1002.[1] Reading the two rows side by side is the whole lesson: between August and September 2026 NVIDIA moved Cumulus, DOCA, NIC firmware, Network Operator and NCCL together.[1] That is a whole-stack step, not a patch, and it is the shape of every RA transition.
The mapping back through time is equally blunt about the independence of the two numbering schemes: RA 1.0.1 with Cumulus 5.9.1 (April 2024), RA 1.1 with 5.10.1, RA 1.2 with 5.11.0, RA 1.3 with 5.12.1, RA 2.0 with 5.14, RA 2.1 with 5.16.0, RA 2.1.4 with 5.16.5, RA 2.1.5 and 2.1.6 with 5.16.6, and RA 2.3 and v2.3.1 with 5.18.1.[1][7][8] Nothing in the RA number lets you predict the Cumulus number or the reverse.
3Reading the row for a real customer
Pick the track from the hardware, not from the newest number. The H200 track is the BlueField-3 SuperNIC track; GB300 and B300 are ConnectX-8 tracks.[1] Then read the whole row - and check that the row exists: the B300 and H200 tracks have no v2.1.3 row, so a track and version pair cannot be assumed.[1]
Collect the customer’s actual state with three reads. nv show system version on at least two leaves gives the switch OS; ethtool -i <ifname> and flint -d /dev/mst/<dev> q on at least two hosts give driver and firmware; the installed DOCA-Host version completes the host side.[1][12] Two leaves and two hosts rather than one of each, because the most common real finding is not “we are behind” but “we are inconsistent”.
Then disclose the two documented mismatches rather than waiting to be caught by them. The Network Operator documentation maps Spectrum-X RA 2.1 to Network Operator 26.1.0 and RA 2.2 to 26.4.0, while the validated-stack row for v2.3.1 lists 26.7.0 - the Network Operator page simply stops earlier.[4][1] And the research for this course recorded a NetQ gap, where the RA pairs NetQ 5.1.0 with Cumulus 5.18.1 while the NetQ 5.1 supported-OS list stops at Cumulus 5.16; that gap was not re-confirmed at write time, so check the current NetQ release notes before repeating it, and state both facts with their dates if it persists.[9][1]
Finally, the lifecycle tension nobody raises until an audit. Cumulus Linux mainline releases receive “12 months of NVIDIA support from date of LTS release”, while LTS versions get “3 years from release date of version”, with 5.9.z running to April 2027 and 5.11.z to November 2027.[5] The current RA row pins 5.18.1, a mainline release. An RA-compliant Spectrum-X fabric is therefore not running an LTS line, and a customer whose standards mandate LTS has a real decision to make: follow the RA and accept mainline support windows, or run an LTS release and accept being off-row.[5][1]
Scenario: a Dell customer runs H200 nodes with BlueField-3 SuperNICs and two SN5600 leaves. The inventory comes back as Cumulus Linux 5.16.5 on both leaves, BlueField-3 firmware 32.47.x on the hosts, DOCA-Host 3.3.0-088826, NetQ 5.1.0, Network Operator 26.1.0. They want the 5.18.1 ECMP fix and ask whether they can “just upgrade the switches”.
- Identify the track. BlueField-3 SuperNICs on H200 nodes means the H200 track. Read that track’s rows only.[1]
- Locate their current row. Cumulus 5.16.5 with DOCA-Host 3.3.0-088826 is the RA 2.1.4 to 2.1.6 neighbourhood; RA 2.1.4 pairs with Cumulus 5.16.5 and RA 2.1.5 and 2.1.6 with 5.16.6, so they are on or just below RA 2.1.4 and already inconsistent with their own DOCA version.[1] Record that as finding one, before any upgrade discussion.
- Locate the target row. RA v2.3.1 requires Cumulus 5.18.1, BlueField-3 firmware 32.50.1002, DOCA-Host 3.5.0-082, NetQ 5.1.0, NCCL 2.30.7, HPC-X 2.51, Network Operator 26.7.0, DTS 1.26.5.[1]
- Answer the actual question. A switch-only move to 5.18.1 puts the fabric in a combination that appears in no row: 5.18.1 was validated with 32.50.1002 and DOCA 3.5.0-082, not with 32.47.x and DOCA 3.3.[1] Since adaptive routing and congestion control are negotiated between switch and NIC, this is precisely the class of change where a mismatch shows up as intermittent performance rather than a clean failure.[10]
- State what must move together: Cumulus to 5.18.1, BlueField-3 firmware to 32.50.1002, DOCA-Host to 3.5.0-082, Network Operator to 26.7.0 and DTS to 1.26.5; NetQ 5.1.0 already matches.[1]
- Add the caveats you owe them. Being on-row is not defect-free - 5.18.1’s own release notes carry open issues including BGP peer sessions repeatedly flapping and traffic not forwarded on some interfaces in an ECMP group.[3] And 5.18.1 is a mainline release, so it does not inherit the three-year LTS window.[5]
- Deliverable: a delta sheet with three columns - component, measured, RA v2.3.1 target - and a verdict line per component, plus a short paragraph saying which components are mandatory to move and why the fix they want cannot be taken alone.
Same job, different cluster: B300 nodes with ConnectX-8 SuperNICs. Inventory: Cumulus Linux 5.16.6 on three of four leaves and 5.16.5 on the fourth, ConnectX-8 firmware 40.48.1146, DOCA-Host 3.3.0-088826, Network Operator 26.4.0, NCCL 2.29.2.
- Track to read: ____.[1]
- The row that matches most of the inventory is RA ____ (Cumulus ____).[1]
- Finding one, before any upgrade: ____ (hint - look at the fourth leaf).
- Target row v2.3.1 requires Cumulus ____, ConnectX-8 firmware ____, DOCA-Host ____, Network Operator ____ and NCCL ____.[1]
- The customer’s Kubernetes team objects that the Network Operator page says RA 2.2 maps to 26.4.0. Your one-sentence answer: ____.[4]
- Caveat you must add about the pinned Cumulus release and NVIDIA’s support policy: ____.[5]
A Dell SE forwards this from an account: “We standardise on LTS software. Our AI pod runs Spectrum-X with ConnectX-8 on B300. Put us on Cumulus Linux 5.11.z and keep everything else current.”
Write the reply. Acceptance criteria: (a) you state what the RA row for the B300 track actually pins today and cite it; (b) you state the support difference between mainline and LTS in NVIDIA’s own terms; (c) you say explicitly what running 5.11.z would mean for their Spectrum-X status without overclaiming - the row names a version, it does not say another version is forbidden; (d) you name at least two components that would then be off-row and why that matters for adaptive routing and congestion control specifically; (e) you close with the two inventory commands you want run before the next call. No sentence in the reply may assert anything that is not on a page you can cite.[1][5][10]
A delta sheet instead of a number
You send a table rather than an answer: component, measured, RA v2.3.1 target, verdict per line - sampled on two leaves and two hosts, because drift is more common than being uniformly behind.[1][12] Five components have to move together and NetQ already matches.[1] The Saturday switch window becomes a stack window, which answers procurement: the window is free, the drift is not. What you say on the call: “You can have 5.18.1 on Saturday, but only with the firmware and DOCA it was validated against - otherwise the fix arrives and the fabric leaves the row.”[1] Then the customer’s architect unmutes after an hour of silence: his team runs one NOS everywhere, and could the thirty-two leaves not simply run SONiC?
Lab
Read-only inventory on the Dell lab hosts. Nothing here changes firmware or software.
- Identify the adapters and their drivers:
Expected: asudo mst start sudo mst status -v sudo ethtool -i ens1f0np0/dev/mst/...device path plus driver and firmware-version strings. Record the exact firmware string; the leading number identifies the family (32.x BlueField-3, 40.x ConnectX-8).[1] - Read the firmware from the device itself and compare the two answers:
Expected: ansudo flint -d /dev/mst/mt41692_pciconf0 qFW Versionthat agrees withethtool -i. A disagreement usually means a firmware was flashed but the host was never power cycled - note it, do not fix it here. - Record the host software version. On a DOCA-Host installation:
Expected: a DOCA-Host or OFED version string. Write it next to the RA target ofofed_info -s3.5.0-082.[1] - Repeat steps 1 to 3 on a second host and put both results in one table. Expected: identical strings. Any difference is the finding - fleet drift is more common than being uniformly behind.
- Produce the delta sheet in the form you would send a customer: component, measured on host A, measured on host B, RA v2.3.1 target, verdict.[1]
- Optional, customer or partner lab with a live switch: add the switch row with
nv show system versionon two leaves, read-only, and note whether the two leaves match each other before comparing either to the row.[1][12]
Goal: reconstruct the RA table from primary sources and rehearse the delta sheet you will actually send.
- Open the Validated Solution Stack page and write out the RA to Cumulus mapping from RA 1.0.1 to the current row, one line each with the release month.[1] Expected: ten or eleven lines ending at v2.3.1 with Cumulus 5.18.1 in September 2026. If your table has a Cumulus version repeated across two RA numbers, that is correct - 5.16.6 covers 2.1.5 and 2.1.6.
- Switch tracks and check for holes: list which of GB300, B300 and H200 have a v2.1.3 row.[1] Expected: not all of them. Write one sentence you would say to a customer whose track lacks the row they asked for.
- Take the worked scenario’s inventory - Cumulus 5.16.5, BlueField-3 firmware 32.47.x, DOCA-Host 3.3.0-088826, Network Operator 26.1.0 - and produce the delta sheet against v2.3.1.[1] Expected: five components moving, one already matching. If you marked NetQ as needing a move, re-read the row.
- Read the 5.18.1 release notes and add a “known issues at the target version” section to the sheet.[3] Expected: at least the BGP session flapping and the ECMP forwarding issue. This section is what separates a version list from advice.
- Open the CompatMatrix stack checker, set the components to a deliberately mixed combination - for example DOCA-Host from one row with firmware from another - and record the verdict text.[1] Then set the true v2.3.1 combination and confirm the verdict changes.
- Deliverable: one page you could send a Dell account team, containing the target row, the measured values, the delta, the mandatory moves, and the two caveats (known issues, mainline versus LTS).[3][5]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell account team in five sentences why 'what version should we run?' has no single-number answer on Spectrum-X and what you send them instead.
Sources
Facts in this lesson were checked against Spectrum-X Validated Solution Stack (re-fetched 2026-09-07, latest row v2.3.1 September 2026), Cumulus Linux release versioning policy, Cumulus Linux 5.18 What's New and release notes. Dates are when each page was fetched.
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- Cumulus Linux 5.18 Release Notes · fetched 2026-09-07
- NVIDIA Spectrum-X Ethernet Networking Platform (Network Operator 26.4.0) · fetched 2026-09-07
- Cumulus Linux Release Versioning and Support Policy · fetched 2026-09-07
- NVIDIA ConnectX-8 SuperNIC Firmware Release Notes v40.48.1000 · fetched 2026-09-07
- What's New | Cumulus Linux 5.16 · fetched 2026-09-07
- What's New | Cumulus Linux 5.14 · fetched 2026-09-07
- What's New | Cumulus NetQ 5.1 · fetched 2026-09-07
- Equal Cost Multipath Load Sharing incl. Adaptive Routing | Cumulus Linux 5.18 · fetched 2026-09-07
- Upgrading Cumulus Linux | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference: Show Commands - System · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Version matrices, upgrades and heterogeneous clustersK8sNet course · Same ground: upgrade, lifecycle and kubernetes
- The validated stack and version pinningDOCA course · Same ground: firmware, versions and pinning
- Drivers and firmware: DOCA-OFED, mlxfwmanager, mlxconfig, NVOSInfiniBand course · Same ground: lifecycle, firmware and E-switch