Skip to content
nodeza.learn
K8sNet course/3. Operators/Lesson 3.2

Version matrices, upgrades and heterogeneous clusters

S3·E2The bookmark that says 1.31 is fine · Dell customer bridge call, Friday night, eleven days from the acceptance run

S3·E2Evaluate~30 minsources checked todayverified against Re-fetched 2026-09-07: Network Operator 26.7.0 platform-support and life-cycle-management; GPU Operator platform-support (latest)

Builds on: Network Operator and the NicClusterPolicy

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Derive the Kubernetes version intersection two NVIDIA operators can both support and judge whether a customer cluster sits inside it.
  • Critique an upgrade plan against the three documented steps and name the step customers skip.
  • Justify NicNodePolicy over NicClusterPolicy for a heterogeneous cluster and state the three constraints that come with it.
  • Detect a versioned documentation URL that is silently serving an older release.

Episode 2 — The bookmark that says 1.31 is fine

The situation · Dell customer bridge call, Friday night, eleven days from the acceptance run

Six people on the bridge, all with the same page open. The network lead reads the supported Kubernetes range off his bookmark and it includes the 1.31 the cluster is running. You read yours and it does not.

Nobody is misreading. His bookmark points at kubernetes2570, which serves the 25.7.0 doc set; yours is kubernetes2670, the 26.7.0 release they actually installed last month.[7] The two pages are identical furniture. Only yours says the minimum Kubernetes version is now 1.32, because 26.7.0 raised it.[3]

This is why version matrices exist and why they are published per release. An operator does not merely run on a cluster, it consumes interfaces that move underneath it, so each release states the window NVIDIA tested and will fix. Network Operator 26.7.0 states Kubernetes 1.32 through 1.36; the GPU Operator, on its own release train, states 1.33 through 1.37, and the customer has to live in the overlap of both rather than in either one.[1][2]

The procurement person, silent for forty minutes, asks the only question she came for: does this move the delivery date. It might. The Dell SE opens the spreadsheet and does not say anything, which for him is remarkable.

A page that describes a release nobody is running is not a support statement.

Before you can answer “is this supported”, you have to establish which four numbers are actually in front of you.

1The numbers, and why they age

Network Operator uses calendar versioning YY.MM.PP. Right now 26.7.x is supported, 26.4.x is deprecated, and 26.1.x and lower are end-of-life.[3] That is three states, not two, and customers plan maintenance windows off the difference between deprecated and end-of-life.

Version 26.7.0 released 1 September 2026 and raised the minimum Kubernetes version to 1.32.[3] The 26.7.0 matrix, re-read today: Kubernetes >=1.32 and <=1.36; Red Hat OpenShift 4.17 to 4.22; Rancher RKE2 1.32 to 1.36; Canonical MicroK8s 1.32 to 1.36; Nutanix NKP 2.12 to 2.15.[1] Host OS: Ubuntu 26.04, 24.04 and 22.04 LTS, RHEL 10.2 through 8.8, RHCOS for the supported OpenShift range, and SLES 15 SP7.[1] Supported adapters: ConnectX-6, ConnectX-6 Dx, ConnectX-7, ConnectX-8, ConnectX-9, and BlueField-3 and BlueField-3 SuperNIC in NIC mode only, up to 800 Gb/s.[1] Node Feature Discovery >=0.15.6 and <=0.17.0, Helm v3.5 or newer.[1] Driver container tags: GA doca3.5.0-26.07-0.7.7.0-0 and LTS doca3.2.2-25.10-2.4.1.0-4.[1]

Two of those lines matter more than the rest on a Dell account. BlueField-3 is supported in NIC mode only - a customer who wants DPU mode is asking for the DOCA Platform Framework, not this operator.[1] And RKE2 is explicitly in the list, which matters because RKE2 is a common Dell AI Factory choice alongside OpenShift.[1][10]

2The intersection is the whole skill

The Network Operator is not the only operator on that node. GPU Operator 26.7.x is supported, 26.3.x deprecated, 25.10.x and lower end of support, and its Kubernetes range is 1.33 to 1.37.[2] Put the two ranges side by side and the arithmetic answers the customer question before any configuration does: 1.32 to 1.36 intersected with 1.33 to 1.37 is 1.33 to 1.36.[1][2] A cluster on 1.32 has the GPU Operator outside its window. A cluster on 1.37 has the Network Operator outside its window. Neither will necessarily fail; both are outside what NVIDIA agreed to fix.

Now the harder half. The two matrices also try to name each other, and they do not agree. The Network Operator deployment guide says GPUDirect needs DOCA-OFED v3.1.0 or newer and GPU Operator v25.3.x or newer.[8] The GPU Operator’s own platform page names Network Operator 25.7.0, 25.10.0 and 26.1.0 as the versions validated for GPUDirect RDMA depending on operating system, and Network Operator 25.7.0 for GPUDirect Storage.[2] Those are anchored from opposite ends and never name the same pair. The honest answer to a customer is to quote both and land the cluster where both statements are satisfied - not to pick the page that suits the argument.

◐ Level 2 — limited (same annual cycle)
  • Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
  • This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
  • ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).

Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

Read the stack tab first, then switch to timeline. The question to hold in your head: for a given Kubernetes minor, which components are inside their published windows and which are merely working.

3Upgrades, and the step that gets skipped

The documented upgrade is three stages: fetch the new helm chart, run helm upgrade with your values file, then manually update the component versions inside the NicClusterPolicy - because “Helm upgrade does not update components version in the NicClusterPolicy”.[4] A fourth follow-on applies if the device-plugin config changed: restart the affected pods.[4]

The middle step is the one customers skip, and the resulting cluster is worse than an un-upgraded one because it looks correct. helm list -A reports the new chart. kubectl get nicclusterpolicy nic-cluster-policy -o yaml still names last quarter’s ofedDriver.version. That combination is not a supported configuration and nobody’s dashboard says so.[4]

Driver rollout progress is visible on the node itself. The label nvidia.com/ofed-driver-upgrade-state walks through upgrade-required, cordon-required, drain-required, pod-restart-required, uncordon-required and upgrade-done, with node-maintenance-required and wait-for-jobs-required appearing when the requestor and node-maintenance paths are in use, and upgrade-failed when it needs a human.[4] A node parked on one value tells you the blocked stage without reading a single log. ofedDriver.upgradePolicy.safeLoad cordons and drains a node before the driver container is loaded on it, so pods already using the NICs are moved rather than surprised.[4]

One platform caveat to carry: “Automatic DOCA-OFED Driver Upgrade doesn’t work on Single Node OpenShift (SNO) deployments.”[11] Single-node OpenShift shows up in Dell edge conversations often enough that it is worth saying before the customer designs around it.

4Heterogeneous clusters: NicNodePolicy

A single NicClusterPolicy describes one intent for the whole cluster. Real AI clusters are not uniform - an older ConnectX-6 group next to a new ConnectX-8 group, or a GPU group and a storage group that want different device-plugin configs. NicNodePolicy allows multiple per-node-group policies, each targeting specific nodes via nodeSelector labels.[5] The documented example targets node-role.kubernetes.io/gpu: "" with its own ofedDriver.version and rdmaSharedDevicePlugin block.[5]

Three constraints come with it and all three bite in production. Names are at most 30 characters.[5] Two NicNodePolicy instances must not select overlapping sets of nodes, validated at admission and continuously at runtime - so a later node relabel can invalidate a policy that was valid when applied.[5] And section exclusivity: a given section such as ofedDriver, rdmaSharedDevicePlugin or sriovDevicePlugin can exist in either NicClusterPolicy or NicNodePolicy instances, but not both simultaneously, enforced by an admission webhook.[5]

Then the caveat that decides the recommendation: “There is no automated migration path to transition an existing NicClusterPolicy-only deployment”, the admission controller must be enabled, and the docs recommend it for new deployments.[5] So for a greenfield Dell AI Factory build with mixed node groups, propose it. For a running cluster, price the migration honestly rather than selling it as a flag.

5Launch Kit, and the URL that lies

The Kubernetes Launch Kit (l8k) is “a CLI tool that automates the deployment of NVIDIA cloud-native networking on Kubernetes”: it discovers hardware, selects a profile, generates manifests and can deploy them.[6] Five commands - l8k discover, l8k generate, l8k deploy, l8k validate, l8k preset.[6] Its profiles cover SR-IOV Ethernet and InfiniBand, host device, IPoIB with RDMA shared, macvlan, and Spectrum-X in hwplb, swplb and single-plane forms, across vanilla Kubernetes, OpenShift, KubeVirt SR-IOV, disconnected environments and mixed NIC/GPU heterogeneous clusters.[6] For an FAE the useful mode is discover plus generate: it turns a customer’s hardware into a manifest you can read and argue with, without deploying anything.

Last trap, and it is cheap to fall into. The doc set is versioned in the URL path: docs.nvidia.com/networking/display/kubernetes2670/... is 26.7.0 and kubernetes2570 is 25.7.0.[7] The pages look identical. A customer citing a support statement from a bookmark is often citing a release they are not running - and since 26.7.0 raised the Kubernetes minimum to 1.32, the older page will happily tell them 1.31 is fine.[3] Check the digits in the URL before you check the claim.

Three customer clusters, one support answer each

Cluster A: upstream Kubernetes 1.34, Ubuntu 24.04, ConnectX-7, Network Operator 26.7.0, GPU Operator 26.7.1, ofedDriver.version: doca3.5.0-26.07-0.7.7.0-0, NFD 0.16.2.

  1. Kubernetes. 1.34 is inside Network Operator 1.32-1.36 and inside GPU Operator 1.33-1.37, so it is inside the 1.33-1.36 intersection.[1][2]
  2. Host OS. Ubuntu 24.04 LTS is listed.[1]
  3. Hardware. ConnectX-7 is listed.[1]
  4. Operator versions. 26.7.0 is supported; GPU Operator 26.7.x is supported.[3][2]
  5. Driver tag. Matches the GA tag shipped with 26.7.0.[1]
  6. NFD. 0.16.2 is inside 0.15.6-0.17.0.[1]
  7. The GPUDirect pairing. Network Operator says GPU Operator 25.3.x or newer - satisfied. The GPU Operator matrix names Network Operator 25.7.0, 25.10.0 and 26.1.0 for GPUDirect RDMA - not satisfied by 26.7.0 literally.[8][2]
  8. Answer: every published window is satisfied except the GPU Operator matrix’s literal list of Network Operator versions, which has not yet been updated for the 26.7 line. Say exactly that, quote both pages, and note that the mismatch is a documentation lag rather than a known incompatibility - and that if the customer needs a written support statement, it has to come from the account team, not from you reading two tables.

Four numbers, ten minutes

How it ended

The four numbers land fast: Kubernetes 1.31, Network Operator 26.7.0, GPU Operator 26.7.1, and an ofedDriver.version still carrying a 26.4-era tag - because helm upgrade does not update component versions in the NicClusterPolicy, and the documented third step was never run.[4] Two findings, one self-inflicted, and neither is a networking fault. The customer leaves with a version plan and a maintenance window rather than a lab session.

Which is when the platform lead files the next change: enable lossless RoCE on all twelve nodes, before the acceptance run, service impact none.

Lab

Goal: capture the real four numbers from the Dell-lab BlueField-3 host and turn them into the template you will actually send. Read-only - every command below reads state. No rollback is needed because nothing is mutated.

  1. Kubernetes. kubectl version on the single-node cluster. Record the server minor. Expected: a minor between 1.32 and 1.36 if you built it against this course.[1]
  2. Helm releases. helm list -A. Record chart and app version for network-operator and, if present, gpu-operator. Expected: network-operator-v26.7.0.[4]
  3. Driver tag from the CR. 
    kubectl get nicclusterpolicy nic-cluster-policy \
  -o jsonpath='{.spec.ofedDriver.version}{"\n"}'
    Expected: doca3.5.0-26.07-0.7.7.0-0 if you followed lesson 1.[1] If this returns nothing, check the CR name before anything else.
  4. Upgrade state label. 
    kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/ofed-driver-upgrade-state}{"\n"}{end}'
    Expected: upgrade-done on a settled node. Any other value names the stage that is blocked.[4]
  5. Host reality check. uname -r and cat /etc/os-release, then compare against the supported host OS list. Expected: your kernel and distro are inside the published set - if not, write down exactly which line they violate.[1]
  6. Adapter check. mst status and ibv_devinfo to confirm the model, then compare against the supported adapter list, remembering BlueField-3 is supported in NIC mode only.[1]
  7. Deliverable: the filled template from the no-hardware lab, populated with your lab’s real numbers, plus one sentence stating whether this cluster sits inside the 1.33-1.36 intersection and what you would change if it did not.[1][2]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell account SE in five sentences why you cannot answer is this cluster supported with a single version number and what four numbers you ask for instead.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against Re-fetched 2026-09-07: Network Operator 26.7.0 platform-support and life-cycle-management; GPU Operator platform-support (latest). Dates are when each page was fetched.

  1. NVIDIA Network Operator v26.7.0 - Platform Support · fetched 2026-09-07
  2. NVIDIA GPU Operator - Platform Support · fetched 2026-09-07
  3. NVIDIA Network Operator v26.7.0 - Release Notes · fetched 2026-09-07
  4. NVIDIA Network Operator v26.7.0 - Life Cycle Management · fetched 2026-09-07
  5. NVIDIA Network Operator v26.7.0 - Heterogeneous Clusters with NicNodePolicy · fetched 2026-09-07
  6. NVIDIA Network Operator v26.7.0 - Kubernetes Launch Kit · fetched 2026-09-07
  7. NVIDIA Network Operator v26.7.0 - documentation index · fetched 2026-09-07
  8. NVIDIA Network Operator v26.7.0 - Deployment Guide with Kubernetes · fetched 2026-09-07
  9. NVIDIA GPU Operator - GPUDirect RDMA and GPUDirect Storage · fetched 2026-09-07
  10. Dell AI Factory with NVIDIA - NVIDIA 2-8-9-400 configuration (Solution Brief) · fetched 2026-09-07
  11. NVIDIA Network Operator v26.7.0 - Deployment Guide with OpenShift · fetched 2026-09-09

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.