Skip to content
nodeza.learn
DOCA course/7. AI factories/Lesson 7.3

The validated stack and version pinning

S7·E301:10, and the change window closes at four · Bridge call from the customer's operations center, mid change window

S7·E3Evaluate~35 minsources checked todayverified against Spectrum-X Validated Solution Stack (v2.3.1 row, Sep 2026), DOCA Dependency Compatibility Policy 3.5.0, DOCA Backward Compatibility Policy PDF, DOCA 3.5.0 release notes and General Support, Network Operator v26.1.1 and NIC Configuration Operator docs — fetched 2026-09-06

Builds on: HGX reference architecture roles, The DOCA framework map, versions and LTS

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Read the Spectrum-X Validated Solution Stack page and extract the exact v2.3.1 pin for a given platform column, noting what the table does not cover.
  • Classify each host-to-card dependency as Level 1 or Level 2 using the DOCA Dependency Compatibility Policy and the installed DOCA-Host profile.
  • Derive and defend an upgrade order for a mixed BlueField-3 plus ConnectX-8 node that respects the Level 2 lock-step rule.
  • Evaluate a customer's proposed partial upgrade and decide whether the policy supports it.
  • Produce a gap analysis from an inventory with the checker script and the template in this lesson.

Episode 3 — 01:10, and the change window closes at four

The situation · Bridge call from the customer's operations center, mid change window

The window opened at midnight, it closes at four, and it is the last one before the storm-season freeze. You are on the bridge with the carrier’s platform lead, a Dell services engineer, a very tired change manager and the night operator, who has already labelled the new DPU tray NOT A SUPERNIC — ASK THE FAE and is quietly pleased with himself. The plan has shrunk since the review: they want to move DOCA-Host to 3.5.0-082 tonight because a fix they need is in it, and leave the BlueField bundle at 3.3.0 until next quarter, because the DPU upgrade needs a full power cycle and there is no second outage to spend. The platform lead asks the only question that matters at this hour: is that allowed?

Nothing about the answer is opinion, which is why NVIDIA publishes two documents instead of a slogan. The Spectrum-X Validated Solution Stack pins exactly which component versions were tested together per platform, release by release.[1] The Dependency Compatibility Policy says what happens when you deviate: a host running only the driver-only DOCA-OFED profile may lead or lag the card indefinitely, Level 1, while a host on doca-all or doca-networking, and every DOCA service, forms a Level 2 group whose compatibility only holds inside one October-through-July cycle.[2][9]

The profile installed on the host decides whether a version gap is supported or merely survivable. Nobody on the bridge has said which profile these hosts run, so the first thing you ask for is not an upgrade. It is an inventory.

1What the validated stack is, and what it is not

NVIDIA publishes the Spectrum-X Validated Solution Stack as one page with three tables, one per GPU platform: GB300, B300 and H200. Columns are releases, rows are components.[1] The release history runs April 2024 (v1.0.1), July 2024 (v1.1), November 2024 (v1.2), February 2025 (v1.3), September 2025 (v2.0), February 2026 (v2.1), March 2026 (v2.1.1), April 2026 (v2.1.2, plus v2.1.3 in the GB300 table only), June 2026 (v2.1.4), July 2026 (v2.1.5), August 2026 (v2.1.6) and September 2026 (v2.3.1). There is no v2.2 or v2.3.0 column.[1]

The rows are: the platform FW/SW package (GB300 1.0.10, B300 1.5.10), Cumulus (Spectrum-4), ConnectX-8 firmware for the Blackwell tables or BlueField FW Bundle plus BlueField-3 firmware for the H200 table, DOCA-Host, NetQ, NCCL, HPC-X, Network Operator and DTS.[1] Not on the page: SONiC, UFM, the GPU driver, CUDA, Base Command Manager, and switch or adapter part numbers. Those come from other documents, so a “validated stack” sign-off that stops at this page is incomplete.[1]

The DOCA side confirms the pairing from its end: the DOCA 3.5.0 release notes state “DOCA v3.5.0 is tested as part of Spectrum-X reference architecture release version 2.3”, last updated September 02, 2026.[4] The September 2026 row, v2.3.1, is:

Component GB300 B300 H200
Platform FW/SW package 1.0.10 1.5.10 (no row)
Cumulus (Spectrum-4) 5.18.1 5.18.1 5.18.1
BlueField FW Bundle (including FW) (no row) (no row) 3.5.0
BlueField-3 firmware (no row) (no row) 32.50.1002
ConnectX-8 firmware 40.50.1002 40.50.1002 (no row)
DOCA-Host 3.5.0-082 3.5.0-082 3.5.0-082
NetQ 5.1.0 5.1.0 5.1.0
NCCL 2.30.7 2.30.7 2.30.7
HPC-X 2.51 2.51 2.51
Network Operator 26.7.0 26.7.0 26.7.0
DTS 1.26.5 1.26.5 1.26.5

All values are quoted from the page as fetched on 2026-09-06.[1]

◐ Level 2 — limited (same annual cycle)
  • Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
  • This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
  • ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).

Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

The stack tab reproduces the validated-stack rows; select a release to see which components moved.

2Reading across columns: the B300 gap and the v2.1.6 delta

A B300 node in the HGX RA carries eight ConnectX-8 and one BlueField-3 DPU for north-south.[15] The B300 table pins the ConnectX-8 side (40.50.1002) and the platform package (1.5.10) but has no BlueField row at all.[1] The pin for that DPU has to be assembled: the same release’s H200 column gives BlueField FW Bundle 3.5.0 and BlueField-3 firmware 32.50.1002, and the DOCA 3.5.0 General Support firmware table gives the same 32.50.1002 for BlueField-3, with 40.50.1002 for ConnectX-8 and 82.50.1002 for ConnectX-9.[1][6] Write that pin down as derived from the same release, not as a B300-table value; the customer’s NVIDIA support contact may ask where it came from.

The previous row shows how much a release moves. From v2.1.6 (August 2026) to v2.3.1, every component moved except NetQ (5.1.0 in both) and the platform FW/SW packages (GB300 1.0.10, B300 1.5.10, unchanged): Cumulus 5.16.6 to 5.18.1, BlueField FW Bundle 3.3.0 to 3.5.0, BlueField-3 firmware 32.48.1000 to 32.50.1002, ConnectX-8 firmware 40.48.1146 to 40.50.1002, DOCA-Host 3.3.0-088826 to 3.5.0-082 (the H200 v2.1.6 cell reads just 3.3.0), NCCL 2.29.2 to 2.30.7, HPC-X 2.26 to 2.51, Network Operator 26.1.0 to 26.7.0, DTS 1.24.3 to 1.26.5.[1] Two details matter for reading older rows: Network Operator first appears in the February 2026 (v2.1) column as 26.01, and from v2.1.4 back to v2.1 the H200 BlueField-3 firmware cell reads “N/A (Part of the BlueField FW Bundle)”, so the firmware pin is implicit in the bundle version for those releases.[1]

The BlueField bundle itself has two forms. DOCA defines the BF-Bundle as the package “installed on the BlueField Arm cores”: the Arm OS, the DOCA runtime, and the platform software (firmware and UEFI).[10] The BF-FW-Bundle is the firmware-only variant: “the bf-fwbundle upgrades ATF-UEFI, NIC firmware, and BMC firmware only (without OS overwrite)”.[11] The stack row “BlueField FW Bundle (including FW) 3.5.0” therefore pins the bundle release; the firmware row makes the NIC firmware inside it explicit.[1]

3The Dependency Compatibility Policy: Level 1 versus Level 2

DOCA ships on a quarterly release cadence.[3] The dependency policy anchors the year on the October GA release and guarantees compatibility “for all releases within a single annual cycle (October through July)”, within a three-year LTS support window.[2] The policy does not publish the months of the intervening updates or an LTS naming convention; the documentation version list shows LTS update releases under names of the form 3.2.3-LTS25-U3, so read the release notes rather than inferring a date from a version string — DOCA 3.5.0 itself is dated September 02, 2026.[4] Versioning is semantic X.Y.Z: a major may introduce incompatible API changes, a minor adds backward-compatible features, a patch fixes bugs. APIs move through DOCA_EXPERIMENTAL, DOCA_STABLE (supported for the major), DOCA_DEPRECATED and Removed; the C ABI is versioned so binaries stay compatible “across both minor and major versions”.[3]

The dependency policy defines two levels. Level 1, Perpetual Dependency Compatibility: “Either the upper or lower layer may be upgraded independently, and all prior functionality is guaranteed to work seamlessly”, within the three-year LTS window.[2] Level 2, Limited Dependency Compatibility: “Breaking changes may be introduced annually, specifically with the October GA release”; compatibility holds across releases within one annual cycle, October through July, and the July release may not be compatible with the next October.[2]

The table has seven rows.[2]

Upper layer Lower layer Level
DOCA-Host, DOCA-OFED profile FW / BF-FW-Bundle Level 1
DOCA-Host, other profiles FW / BF-FW-Bundle Level 2
DOCA-Host, DOCA-OFED profile BF-Bundle Level 1
DOCA-Host, other profiles BF-Bundle Level 2
DOCA-DPU BF-FW-Bundle Level 2
DOCA Services BF-Bundle Level 2
DOCA Services FW / BF-FW-Bundle Level 2

The profile is the pivot. DOCA-Host 3.5.0 offers five profiles: doca-all, doca-networking, doca-ofed, doca-roce and doca-host-basic; doca-ofed is “Driver-only installation equivalent to MLNX_OFED”, while doca-all and doca-networking add DOCA Core, DOCA Flow, OVS-DOCA and more.[9] So a host with only doca-ofed may lead or lag the card across cycles (Level 1); a host with doca-all or doca-networking, and every DOCA service such as DTS, must move with the BF-Bundle inside the same cycle (Level 2).[2][9]

Two more rules bound the plan. DOCA ships as a meta-package (BFB for Arm, DOCA-for-host rpm or deb) and “There is no guarantee that DOCA SDK would work correctly if any of these components is upgraded independently”.[3] Host-to-BlueField and node-to-node DOCA components stay protocol-compatible across SDK versions so a cluster can do “a rolling upgrade”.[3] DPA device libraries are statically linked, so library-to-application binary compatibility is “not applicable” and DPA follows the limited policy.[16]

4Applying the policy: an upgrade order for an XE9780 node

The policy does not publish a step list; this lesson derives one and states the reasoning so you can defend or adjust it.

  1. Inventory and classify. Record adapter firmware (mlxfwmanager --query), the Arm image (/etc/mlnx-release), the DOCA-Host package and its profile, the switch OS and NetQ, and the Kubernetes operator versions. Mark each pair Level 1 or Level 2 from the table using the actual profile.[2][9]
  2. Fabric first or last, but not in the middle of the node work. Cumulus 5.18.1 and NetQ 5.1.0 are outside the DOCA dependency policy; schedule them as their own change.[1]
  3. DPU: BF-Bundle 3.5.0, which carries BlueField-3 firmware 32.50.1002. Dell requires a full power cycle after a DPU firmware update on the XE9680L, and NVIDIA’s mode changes carry the same power-cycle rule; plan the outage.[1][13] Dell’s own fix history shows where the bits come from: KB 000379421 resolves a BlueField-3 PCIe initialization failure with firmware 32.46.3048 and an aligned BFB downloaded from NVIDIA DOCA downloads, not from a Dell update package.[12]
  4. Host, same cycle: DOCA-Host 3.5.0-082. For doca-all or doca-networking this is Level 2 with the BFB, so it lands in the same window as step 3; for doca-ofed it could be staged separately.[2][9]
  5. ConnectX-8: firmware 40.50.1002. No Arm side, no BFB; a NIC firmware update from the host or from the NIC Configuration Operator.[1][8]
  6. Kubernetes layer: Network Operator 26.7.0 (drivers, device plugin, CNI, IPAM) and DTS 1.26.5, which as a DOCA service is Level 2 against the BF-Bundle and belongs in the same cycle.[1][7][2]
  7. Libraries: NCCL 2.30.7 and HPC-X 2.51, then re-run the acceptance test the customer used to sign off the previous release.[1]

The non-negotiable constraint is the lock-step of Level 2 pairs. The rest of the order is judgement: do the DPU before the host so the Arm side is at the new cycle when the host’s DOCA libraries come up, and do adapters that have no OS last because they have the fewest interactions.

5Operators, and what an FAE checks before a customer upgrade

The NVIDIA Network Operator (v26.1.1 docs, updated April 29, 2026) installs the “NVIDIA networking driver, Kubernetes device plugin, CNI plugins, IP address management (IPAM) plugin and others” and “works in conjunction with the NVIDIA GPU Operator” for GPUDirect workloads.[7] Its docs tree lists the DOCA-OFED driver container, SR-IOV Network Operator, Multus, secondary-network CNIs, the NIC Configuration Operator, Node Feature Discovery, a Maintenance Operator, and a Spectrum-X Operator under advanced configurations.[7] The validated stack pins it at 26.7.0 for September 2026.[1]

The NIC Configuration Operator turns firmware and mlxconfig settings into Kubernetes objects. Its CRDs are NicConfigurationTemplate (firmware and network configuration applied to matched devices), NicDevice (discovered NIC hardware), NicFirmwareTemplate (firmware versions to deploy), NicFirmwareSource (where images come from) and NicInterfaceNameTemplate (interface naming); configurable areas include link type, PCI performance, VF count, RoCE optimizations, GPUDirect optimizations and QoS; it is enabled through NicClusterPolicy.[8] The page fetched for this lesson does not spell out how the reboot after a firmware change is coordinated; the docs tree ships a Maintenance Operator beside it, so confirm the hand-off in the release you deploy before promising it.[7][8]

What an FAE checks before a customer upgrade, in order:

  • The target row: which validated-stack release and which platform column, with derived pins labelled.[1]
  • The profile on every host, because it decides Level 1 versus Level 2 for the host-to-card pair.[2][9]
  • The BlueField OS baseline: DOCA 3.5.0’s default BlueField OS is Ubuntu 24.04 64k, and the release notes carry the supported-adapter list; check both against the customer’s image.[6][4]
  • Deprecation horizon: DOCA 3.6.0 (October 2026) is announced as the last release supporting ConnectX-4 Lx and ConnectX-5, with DOCA 4.0 (January 2027) dropping them; irrelevant to a B300 pod, decisive for a mixed estate.[5]
  • Dell specifics: aux power on the DPU slot, a full power cycle in the change window, and the Dell part on the quote (352-BBFH for the XE9780) so the firmware family is the one you pinned.[13][14]
Pin a stack for a BOM: XE9780 + SN5600 + BlueField-3 + ConnectX-8

BOM: PowerEdge XE9780 (HGX B300) with 8 x ConnectX-8 on the baseboard and 1 x Dell 352-BBFH (BlueField-3 dual-port 400GbE) for north-south; SN5600 leaf and spine on Cumulus; Kubernetes with Network Operator; DOCA-Host profile doca-all.[14][15]

  1. Choose the column: B300, September 2026 (v2.3.1).[1]
  2. Copy the B300 values: B300 FW/SW package 1.5.10, Cumulus 5.18.1, ConnectX-8 firmware 40.50.1002, DOCA-Host 3.5.0-082, NetQ 5.1.0, NCCL 2.30.7, HPC-X 2.51, Network Operator 26.7.0, DTS 1.26.5.[1]
  3. Derive the DPU pin: BlueField FW Bundle 3.5.0 and BlueField-3 firmware 32.50.1002 from the H200 column of the same release, cross-checked with the DOCA 3.5.0 General Support table. Label: derived.[1][6]
  4. Classify: DOCA-Host (doca-all) to BF-Bundle is Level 2; DTS to BF-Bundle is Level 2; ConnectX-8 firmware to host driver follows the doca-all row against FW, Level 2.[2]
  5. Upgrade order: inventory; BF-Bundle 3.5.0 with power cycle; DOCA-Host 3.5.0-082; ConnectX-8 40.50.1002; Network Operator 26.7.0 and DTS 1.26.5; NCCL and HPC-X; fabric as its own change.[13]
  6. Open items: SN2201 role, GPU driver and CUDA (not on the stack page), Dell aux-power cable for the DPU slot.[1]

Episode 3 closes — 03:40, one deviation, documented

How it ended

The hosts came back doca-networking, which makes the pair Level 2 — but 3.3.0 and 3.5.0 sit inside the same annual cycle, so the policy window is not crossed.[2] The combination was still never validated together, and the meta-package rule guarantees nothing for components upgraded independently.[3] They take the DOCA-Host fix, the deviation goes into the change record with its level and its expiry at the next October GA, and the BF-Bundle gets a booked window. Your sentence for the change manager: “It is not unsupported tonight — it is a Level 2 deviation with a deadline, and here is the date it expires.”[1] By 07:00 that expiry date has travelled: the CTO’s overnight slide plans two years past it, with BlueField-4 in the storage tier and a purchase-order date.

Lab

Read-only. Compare the Dell-lab BlueField-3 with the v2.3.1 pins and write a gap report. Do not update anything.

  1. Adapter firmware, from the host: sudo mst start && sudo mlxfwmanager --query. Record FW version, PSID and Part Number. Pin: 32.50.1002.[1][6]
  2. Arm image: ssh ubuntu@192.168.100.2 cat /etc/mlnx-release over the tmfifo link (the default BlueField address on the rshim network), or run it on the Arm console. Record the full build string. Pin: BF-Bundle 3.5.0.[17][1] If not reachable: ls /dev/rshim* and ip -br addr show tmfifo_net0 to see whether the host-side rshim path is up; if it is not, record that and stop.
  3. Host DOCA-Host package and profile: dpkg -l 'doca*' | grep '^ii' (Ubuntu) or rpm -qa 'doca*' (RHEL). Record the version string as printed and which profile meta-package is installed. Pin: 3.5.0-082.[1][9]
  4. Write the three values into an inventory.txt (leave the ConnectX-8, switch and Kubernetes keys as MISSING) and run python3 stackcheck.py inventory.txt from the no-hardware lab. Expected: the ConnectX-8 and fabric rows show MISSING gaps because the lab has none; the three BlueField and host rows show the real state.
  5. Write the gap report with the template: for each of the three rows, from, to, Level with the lab’s profile, and whether the change would need a power cycle. Acceptance: the report names the exact strings printed by each tool and the stack pin next to them, and it states that no change was made.[2][13]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why a customer cannot upgrade only the host DOCA package on their HGX pod and call it done.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against Spectrum-X Validated Solution Stack (v2.3.1 row, Sep 2026), DOCA Dependency Compatibility Policy 3.5.0, DOCA Backward Compatibility Policy PDF, DOCA 3.5.0 release notes and General Support, Network Operator v26.1.1 and NIC Configuration Operator docs — fetched 2026-09-06. Dates are when each page was fetched.

  1. NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-06 · DOCA 3.5.0
  2. DOCA Dependency Compatibility Policy (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  3. DOCA Backward Compatibility Policy (PDF) · fetched 2026-09-06 · DOCA 3.5.0
  4. DOCA Release Notes v3.5.0 · fetched 2026-09-06 · DOCA 3.5.0
  5. DOCA 3.5.0 Changes and New Features · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA General Support (OS matrix, firmware table) · fetched 2026-09-06 · DOCA 3.5.0
  7. NVIDIA Network Operator v26.1.1 · fetched 2026-09-06
  8. NIC Configuration Operator (Network Operator v26.1.0 docs) · fetched 2026-09-06
  9. DOCA Profiles (DOCA-Host installation profiles) · fetched 2026-09-06 · DOCA 3.5.0
  10. DOCA Overview · fetched 2026-09-06 · DOCA 3.5.0
  11. BF-Bundle Installation and Upgrade (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  12. Dell KB 000379421 — PowerEdge BlueField-3 DPU PCIe Initialization Failure · fetched 2026-09-06
  13. Dell KB 000300192 — XE9680L: power cycle required after updating DPU firmware · fetched 2026-09-06
  14. Dell Partner DPU, NVIDIA BlueField-3 Dual Port 400GbE (352-BBFH) · fetched 2026-09-06
  15. NVIDIA HGX AI Factory Enterprise Reference Architecture — Components · fetched 2026-09-06
  16. DOCA SDK Compatibility Policy · fetched 2026-09-06 · DOCA 3.5.0
  17. BF-Bundle Installation and Upgrade (rshim, bfb-install, tmfifo addresses) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.