Skip to content
nodeza.learn
Spectrum-X course/5. Operate/Lesson 5.5

Upgrades that do not lose the fabric

S5·E5Fifty minutes before the change board · Hotel lobby across from the customer's campus, Thursday, 07:40

S5·E5Analyze~35 minsources checked todaylab mutates hardwareverified against Cumulus Linux 5.18 Upgrading, NVUE CLI, What's New and NVUE Config Commands reference; Cumulus Linux 5.16 What's New; DGX SuperPOD update guide; Spectrum-X Validated Solution Stack — research fetched 2026-09-07

Builds on: Automating NVUE: REST API, config templates and Ansible, The RA row: version pinning across switch, NIC and host

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Choose between optimized image upgrade, package upgrade and ONIE reinstall for a stated starting release and target.
  • Analyse an ISSU request against the documented source-release lists, the resource-mode cost and the ASIC exclusions.
  • Build the pre-flight and rollback set that makes a Spectrum-X switch upgrade recoverable.
  • Explain what the 5.18 automatic configuration backup does and does not protect.

Episode 5 — Fifty minutes before the change board

The situation · Hotel lobby across from the customer's campus, Thursday, 07:40

The Dell SE has a laptop balanced on one knee and a coffee he bought forty minutes ago and has not touched. On screen is the change request for Pod B: most of the leaves on 5.16.6, four added last quarter on a newer build and a newer ASIC generation. One maintenance weekend, with Friday’s acceptance re-test on the far side of it. The words in the request are “no data-plane outage” and the plan behind them is ISSU on everything. The board meets in fifty minutes, and procurement has already asked what a stopped weekend of GPU hours costs — the one question nobody disputes.

ISSU is real: the warm-boot path is hitless, with sub-second data plane disruption, because BGP graceful restart keeps the switch forwarding on known routes while protocol state is restored.[1] That is why the feature exists — a fabric that cannot be upgraded without an outage is a fabric that never gets upgraded.

The bill arrives before the window, not during it. ISSU requires graceful-restart mode full and forwarding resource-mode half; half-resource mode reduces forwarding resources by 50 percent, and changing resource mode requires a switchd restart, which is itself disruptive.[1] And one hardware fact deletes a piece of their plan: Spectrum-6 switches do not support ISSU.[1][4] There is a version answer too, and it belongs on the same slide.[4]

Judge an upgrade plan by whether it can be undone, not by whether it is expected to work.

Start with the three upgrade paths and the pre-flight common to all of them.

1Three paths, and the pre-flight that is common to all of them

Cumulus Linux 5.18 documents three ways to move a switch forward.[1]

  1. Optimized image upgrade — installs the new image into the second partition, one reboot, configuration retained.[1]
  2. Package upgrade — updates only changed packages from the APT repository, all settings intact.[1]
  3. ONIE image upgrade — a full fresh install; configuration must be restored manually.[1]

Before any of them, the same pre-flight:

nv action generate system tech-support        # NVUE
sudo cl-support                               # Linux equivalent
nv config apply
nv config save                                # writes /etc/nvue.d/startup.yaml
nv show system
nv show platform hardware

[1][6] Then two artefacts leave the switch: a copy of /etc/nvue.d/startup.yaml by scp or sftp, and a command-form export, nv config show -o commands > backup.config.[6] Keep both. The YAML is what the switch reads at boot and what the documented restore procedure uses; the command form is the same configuration in NVUE verbs, reviewable in a change record.[2][6] NVIDIA does not state that either form survives a schema change better, so treat the command export as insurance rather than a guaranteed migration path.

nv configapplynv config save(auto in 5.9+)nv configdetachboot: nvue-startup.service replays startup.yamlrendered on applyoverwrites hand editsPATCH ?rev=raw textpendingnv set / nv unsetnv set interface lo ip addres…nv set qos roce state enablednv set system hostname leaf01appliedswitchd + FRR are running thisnv set interface lo ip addres…nv set qos roce state enablednv set system hostname leaf01startup/etc/nvue.d/startup.yamlnv set interface lo ip addres…nv set qos roce state enablednv set system hostname leaf01NVUE REST API :8765/nvue_v1/<resource> ?rev=/etc/network/interfaces/etc/frr/frr.conf · switchd.confsnippetsystem config snippet <file>
startup out of syncNothing has been applied yet.
session log
  • nothing run yet — start with `nv set`.
change-control drill · step 2 / 6
Read the diff before you touch anything
`nv config diff startup pending`

Pending revision

  • Where `nv set` and `nv unset` land. Nothing is running yet — the switch behaves exactly as it did before.
  • Read it with `nv show --pending`. Before every apply, read `nv config diff startup pending`.
  • `nv config detach` throws away every pending change made since the last apply.

NVUE CLI (5.13)

Snippet destination

Blocked destinations: /bin, /sbin, /boot, /root, /sys, /proc, /dev. One snippet per file, set/unset only, 1 MB cap on flexible snippets.

Before an upgrade (the config half)
nv action generate system tech-support   # or: sudo cl-support
nv config apply
nv config save                           # writes /etc/nvue.d/startup.yaml
nv config show -o commands > backup.config
scp cumulus@leaf01:/etc/nvue.d/startup.yaml .
nv config diff startup applied           # must be empty before the reboot

Destructive ONIE install keeps the config only with -t /etc/nvue.d/startup.yaml. 5.18.0 adds automatic snapshots in /var/lib/config-backup/ (512 MB cap) restored with nv action restore system config backup <path>.

Upgrading Cumulus Linux · DGX SuperPOD switch update guide · 5.18 What's New · NVUE object model

Walk pending, applied and startup — including an apply that was never saved, which is the classic 'config disappeared after reboot' ticket.

2Choosing the path

Optimized image upgrade. Fetch, install, point the next boot at the other partition, reboot:

nv action fetch system image <url>
nv action install system image files <filename>
nv action boot-next system image other
nv action reboot system mode warm      # ISSU
nv action reboot system                # normal

Rollback is nv action boot-next system image other rollback.[1] It requires a 30 GB or larger SSD — verify with sudo blockdev --getsize64 /dev/sda before you plan the window.[1] Binary image installation through NVUE is itself new in 5.18.0, so an older switch may only offer the Linux forms.[4]

Package upgrade.

sudo nv action upgrade system packages to latest use-vrf default dry-run
sudo nv action upgrade system packages to latest use-vrf default
nv show system reboot required

The Linux equivalent is sudo -E apt-get update then sudo -E apt-get upgrade — not dist-upgrade.[1] It cannot cross major versions (4.x to 5.x) and needs 0.8 GB free.[1] For an air-gapped pod NVIDIA ships an APT mirror container: docker load -i cumulus-linux-apt-mirror-5.18.0.tar, run it with certificate mounts, repoint the switch, then do a normal package upgrade.[1] One 5.18 caveat to read out loud: upgrading to 5.18 from 5.16.7 by packages does not upgrade the TACACS and RADIUS packages — you must run the package upgrade a second time, with apt-get update && apt-get upgrade or nv action upgrade system packages, to pick them up.[4] On a fabric that authenticates against either, that second run is a planned step, not a retry.

ONIE reinstall.

sudo onie-install -a -i cumulus-linux-5.11.1-mlx-amd64.bin -t /etc/nvue.d/startup.yaml && sudo reboot
sudo onie-install -h
nv show system

[1][6] The -t flag carries the NVUE configuration through the destructive install.[1] Back up the rest of /etc (frr, network, cumulus, ssh) first, reimport all certificates afterwards, reinstall third-party applications, and do not migrate /etc/mlx, /etc/init, /etc/fstab or other hardware-specific paths across an image install.[1] ONIE older than 5.3.0013 must be upgraded before the Cumulus Linux upgrade.[1] The DGX SuperPOD guidance sets the practical boundary: switches on 5.9 or earlier, including 4.x, require a new binary image to reach 5.10 or later; a new binary install works in every scenario, and package update is recommended only within the same major release.[6]

3ISSU: what it costs before it saves you

ISSU is the warm-boot path and it is “hitless” with sub-second data plane disruption.[1] The prerequisites are configuration, not flags:

nv set router bgp graceful-restart mode full
nv set system forwarding resource-mode half
nv config apply
nv action reboot system mode warm

[1] Graceful restart is RFC 4724 behaviour: the switch keeps forwarding on known routes while BGP state is restored, which is exactly why ISSU works at all.[1]

The cost is resource-mode half. It reduces forwarding resources by 50 percent, and changing resource mode requires a switchd restart, which is disruptive.[1] Two consequences an FAE must state before a customer commits: the resource-mode change belongs in its own maintenance window well ahead of the upgrade, and a pod sized to fit its route, MAC and ACL tables in full-resource mode has to be re-checked against the halved maxima before ISSU becomes the standing plan.[1]

The eligibility lists are exact and worth carrying:[4]

Target ISSU-supported source releases
5.18.1 5.16.0, 5.16.1, 5.16.3, 5.16.5, 5.16.6, 5.16.7, 5.17.0, 5.18.0
5.18.0 5.16.0, 5.16.1, 5.16.5, 5.16.6, 5.17.0

Note that 5.16.3 and 5.16.7 appear on the 5.18.1 list and not on the 5.18.0 list.[4] ISSU or warm-boot package upgrade to 5.16 is supported from 5.14.0, 5.15.0 and 5.15.1.[5] And the dead end: “The switch does not support upgrade from Cumulus Linux 5.16.7 to Cumulus Linux 5.17.0 or 5.18.0” — from 5.16.7 the package path lands only on 5.18.1.[5][4]

One hardware exclusion overrides all of it: Spectrum-6 switches do not support ISSU, and the SN6600_LD explicitly has no warm-boot support.[1][4]

For an MLAG pair the order is documented: shut down core uplinks and the peer link on the secondary; upgrade the secondary by any method; restore configuration if needed; verify STP and MLAG convergence; make the secondary the primary; repeat on the new secondary; restore the original priorities.[1][6]

◐ Level 2 — limited (same annual cycle)
  • Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
  • This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
  • ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).

Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

The validated-stack view: the switch OS is one row of a bundle, so an upgrade is a row move, not a version bump.

4Configuration backup, restore, and what cannot be restored

NVUE writes saved configuration to /etc/nvue.d/startup.yaml, which persists across reboots when nvue-startup.service is enabled.[2] Cumulus Linux 5.18.0 adds automatic configuration backup and restore as an NVUE enhancement.[4] Applied snapshots land in /var/lib/config-backup/auto-snapshots/applied/ and weekly snapshots in /var/lib/config-backup/auto-snapshots/weekly/.[2] Enable the feature with nv set system config backup state; snapshots live under /var/lib/config-backup/ with a 512 MB limit, oldest deleted when it is exceeded.[3] Restore with:

nv action restore system config backup <snapshot-path> [force]

which “creates a pre-restore rollback snapshot, restores files, rebuilds NVUE state, and reboots the system”.[2]

Three limits on all of that, and they are the analysis that separates a plan from a wish.

It is on the switch. Snapshots protect against a bad change, not against a lost switch or a destructive reinstall. The off-box copies from the pre-flight are the ones that survive an RMA.[1][6]

It does not port across ASIC generations. RoCE configuration is ASIC-specific and is not transferable between Spectrum generations — copying a 5.x startup.yaml from an SN4700 to an SN5600 is not a supported migration.[8] A Spectrum-3 to Spectrum-4 refresh is a re-derivation of QoS, not a restore.

The release itself can rewrite your assumptions. In 5.18.0, NVUE auto-save overwrites manual Linux configuration changes on reboot, or on password changes made with passwd; switches with NVUE enabled ignore hostnames received via DHCP lease; and post-upgrade NVUE commands may override features that only became NVUE-configurable in the new release.[4] Also plan the lab: NVIDIA no longer releases Cumulus VX as a standalone image, so upgrade rehearsals move to DSX Air.[4][1]

Pick the path for five starting releases

Target: Cumulus Linux 5.18.1 on Spectrum-4 leaves in a Spectrum-X fabric. For each starting release, decide the path, whether ISSU is available, and what it costs.

  1. 5.14. Not on any 5.18 ISSU source list.[4] Path: optimized image upgrade to an intermediate release that is on the list, or straight to 5.18.1 with a normal reboot. ISSU: not available directly from 5.14 to 5.18.x. Cost: a real reboot, so the MLAG pair order applies.[1]
  2. 5.16.1. On both the 5.18.1 and 5.18.0 ISSU source lists.[4] Path: optimized image upgrade with nv action reboot system mode warm, or the package path. ISSU: available, but only after graceful-restart mode full and resource-mode half are already in place — and the resource-mode change is disruptive, so it needs an earlier window.[1]
  3. 5.16.6. Same as 5.16.1: on both lists.[4] Same plan.
  4. 5.16.7. On the 5.18.1 ISSU list and not on the 5.18.0 list, and the switch does not support upgrading from 5.16.7 to 5.17.0 or 5.18.0 at all.[4][5] Path: go to 5.18.1. Also flag that a package upgrade from 5.16.7 to 5.18 does not upgrade TACACS and RADIUS packages, which matters if the customer authenticates against either — the documented fix is a second package-upgrade run.[4]
  5. 5.17.0. On both ISSU lists.[4] Straightforward.
  6. Overlay the hardware answer on all five: if any of these switches are Spectrum-6, ISSU is off the table entirely and every one becomes a cold reboot planned around the MLAG order.[1][4]
  7. Overlay the version answer: the target is 5.18.1, not 5.18.0, because 5.18.0 is qualified for non-Spectrum-X only and RA 2.3 pins 5.18.1.[4][7]

Case closed — Friday, 06:00

How it ended

Two answers and one push-back. The older leaves can take the warm-boot path, provided resource-mode half lands in its own earlier window and the pod’s forwarding tables are re-measured against the halved maxima first.[1] The four newest cannot: no ISSU on that generation, so they are cold reboots planned around the MLAG order.[1] Before anything, startup.yaml and a command-form export leave every switch.[6] The target build is 5.18.1 — 5.18.0 is qualified for non-Spectrum-X only.[4][7]

Friday the run finishes clean. The network lead closes the notebook on page nineteen, procurement releases the rest of the order, and the operator labels the board printout, because the tape was there.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

There is no switch in the Dell lab, so rehearse the same discipline where it is safe to fail: BlueField-3 firmware, with an explicit rollback per step.

Pre-flight inventory, and do not skip it — these files are the rollback:

sudo mst start
flint -d /dev/mst/<dev> q                     > fw-before.txt
mlxconfig -d /dev/mst/<dev> q                 > mlxconfig-before.txt
mlxconfig -d /dev/mst/<dev> -e q              >> mlxconfig-before.txt
ethtool -i <ifname>                           > driver-before.txt
  1. Read-only. Compare what you captured against the RA row your fabric targets — for RA v2.3.1 the H200 track pins BlueField-3 firmware 32.50.1002 and DOCA-Host 3.5.0-082.[7] Write the delta sheet before you plan anything.
  2. Read-only. Confirm you have the image you intend to flash and that it matches the pinned version. An upgrade whose target version you cannot state is not a plan.[7]
  3. Mutating. Stage the firmware update with flint on one host only, following the burn procedure for your adapter, then power-cycle the host as the firmware change requires and re-read flint -d /dev/mst/<dev> q. Expected: the new version string. Rollback: re-flash the version recorded in fw-before.txt from the image you kept for it, then power cycle and verify against fw-before.txt. Do not start step 3 without that image on disk.
  4. Mutating. Re-apply any mlxconfig settings the update reset, comparing against mlxconfig-before.txt line by line. Rollback: mlxconfig -d /dev/mst/<dev> s <field>=<old value> for each field that differs, then power cycle and re-diff.
  5. Write the maintenance-window script as the deliverable: for each step, the command, the expected output, the verification command, and the rollback line. A step without a rollback line does not go in the script.
  6. Optional, customer lab in a real window — mutating: a full optimized-image upgrade on the secondary of an MLAG pair following the documented seven-step order, with nv config diff startup applied checked before the reboot and nv action boot-next system image other rollback as the named rollback.[1][2] Verify STP and MLAG convergence before promoting the secondary, and do not touch the second switch until the first is proven.[1]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell customer's change board, in five sentences, why the risky part of a switch upgrade is the configuration rather than the image.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against Cumulus Linux 5.18 Upgrading, NVUE CLI, What's New and NVUE Config Commands reference; Cumulus Linux 5.16 What's New; DGX SuperPOD update guide; Spectrum-X Validated Solution Stack — research fetched 2026-09-07. Dates are when each page was fetched.

  1. Upgrading Cumulus Linux | Cumulus Linux 5.18 · fetched 2026-09-07
  2. NVUE CLI | Cumulus Linux 5.18 · fetched 2026-09-07
  3. NVUE Reference - Config Commands · fetched 2026-09-07
  4. What's New | Cumulus Linux 5.18 · fetched 2026-09-07
  5. What's New | Cumulus Linux 5.16 · fetched 2026-09-07
  6. NVIDIA Spectrum Ethernet Switches | DGX SuperPOD Update Guide · fetched 2026-09-07
  7. NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
  8. RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.