Day-0: ONIE install, first boot and ZTP
S2·E2Twenty-nine of thirty-two · Customer staging row, 08:40, eight days from acceptance
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Choose between the four documented ONIE install paths for a given site constraint.
- Execute a first boot to a reachable, named switch using the documented NVUE commands.
- Build a ZTP flow from DHCP option 239 to a script that reports success, and read its state and logs.
- Diagnose a failed provisioning run from the ZTP state string and the autoprovision log.
Episode 2 — Twenty-nine of thirty-two
You are standing between a rack of freshly cabled leaves and a Dell integrator holding a console cable he would rather not use. Behind him the night-shift operator works down the row with a label maker, tagging cables he has already traced twice. Thirty-two went in yesterday; twenty-nine came up named, addressed and configured with nobody touching them. Three did not: one sits at an ONIE prompt retrying discovery, one reports ZTP state Script Failure, and one reports disabled, which the integrator is certain means broken. The SE has already promised procurement that day 0 costs no labour.
None of it is mysterious once you know what each stage consumes. ONIE takes the image URL from DHCP option 114, so a switch with no lease loops instead of installing.[1] The running OS takes the script URL from option 239, and runs that script only if it carries the flag CUMULUS-AUTOPROVISIONING and exits 0.[4]
Zero-touch provisioning exists because racking is the one part of a fabric that scales linearly with people. Someone has to name each box, address its management port and give it a base configuration; by hand at thirty-two that is a week of console cables and typos, and at three hundred it is not a plan. ZTP replaces the operator with a contract between DHCP, a web server and a script — cheap, unauthenticated, and therefore built to fail closed rather than half-succeed.[4]
Read the state string before you touch the switch.
Day 0 is this whole lesson: image, first boot, script, and the three ways it stops.
1Four ways to put an image on a switch
ONIE is the installer environment already on the box; every install path is a way of pointing it at an image. NVIDIA and Dell document four.[1][2]
| Path | Command | When you use it |
|---|---|---|
| From a running Cumulus Linux, through NVUE | nv action install system image onie <URL> force activate reboot |
The switch is up and reachable; scripted refresh |
| From the ONIE shell | onie-nos-install http://<server>/<image> |
Bare metal, or after wiping; ftp:// and TFTP forms also documented |
| From a running OS, legacy | sudo onie-install -a -i <URL> |
Same as above but from Linux, and the only form that takes -t |
| USB | Image renamed onie-installer-x86_64 on a FAT32/vFAT/EXT2 stick, boot ONIE install mode |
No network to the management port at all |
Two mechanics make the difference between a smooth rack-and-stack and an afternoon with a console cable. DHCP option 114 carries “the entire URL to the web server” so ONIE discovers the image with no operator at all.[1][2] And when there is no DHCP, onie-discovery-stop halts the retry loop so you get a usable prompt.[1] To reach ONIE from a switch that already boots an OS, press Esc or F4 during the five-second GRUB countdown.[1]
The rule that costs people a maintenance window is stated plainly by Dell’s own KB: “Installing a Cumulus Linux Image using ONIE does not retain the configuration files. The configuration files must be saved to a different location prior to the upgrade process.”[2] The one exception is the legacy form’s -t flag, which carries the NVUE configuration through an otherwise destructive install:[8]
sudo onie-install -a -i cumulus-linux-5.11.1-mlx-amd64.bin -t /etc/nvue.d/startup.yaml && sudo reboot
sudo onie-install -h # installer optionsCumulus Linux 5.18 also exposes an ONIE action set through NVUE: nv action install system image onie, nv action activate system image onie, nv action boot-next system image onie (install, rescue, uninstall) and nv action cancel system image onie.[5]
2First boot: from a console prompt to a reachable, named switch
The factory state is minimal and deliberately annoying. root is locked; cumulus has the password cumulus and sudo rights; the first login forces a password change and then drops the SSH session, so plan for a reconnect in any automation.[3] Dell’s KB records the historical variants for older code - cumulus/cumulus from 4.2 onward, cumulus/CumulusLinux! on 4.1.x and earlier - which is worth knowing when a customer hands you a switch that has been in a cupboard.[2]
eth0 runs DHCPv4 by default. Pinning it and naming the box is four commands:[3]
nv unset interface eth0 ipv4 dhcp-client
nv set interface eth0 ipv4 address 192.0.2.42/24
nv set interface eth0 ipv4 gateway 192.0.2.1
nv set system hostname leaf01
nv set system date-time timezone US/Central
nv set system dns domain example.com
nv config applyThen verify the install itself, not just reachability: nv show system image and nv show system version.[2] Images follow the naming convention cumulus-linux-5.x-mlx-amd64.bin, so the version you see should match the file you served.[1]
One release caveat belongs in every day-0 runbook. Cumulus Linux 5.18.0 is qualified for non-Spectrum-X deployments; the Spectrum-X-qualified build is 5.18.1.[9] Also on the 5.18 list: switches with NVUE enabled ignore hostnames received via DHCP lease, so a ZTP flow that expects DHCP to name the box must set the hostname in the script instead.[9] Support lifetime matters when the customer asks what to standardise on: mainline releases get 12 months of support from the date of the LTS release, LTS lines get three years, and the named LTS lines are 5.9.z (EOL April 2027) and 5.11.z (EOL November 2027).[10]
3ZTP: the contract between a script and a switch
Once Cumulus Linux is running, ZTP looks for a script. The DHCP side is one option:[4]
option cumulus-provision-url code 239 = text;
option cumulus-provision-url "http://192.0.2.10/ztp-leaf.sh";The script side is two rules, both of which are pass/fail. It must carry the flag CUMULUS-AUTOPROVISIONING - “You can include this flag in a comment or remark” - and it “must return an exit code of 0 upon success”.[4]
#!/bin/bash
# CUMULUS-AUTOPROVISIONING
set -euo pipefail
nv set system hostname "leaf$(printf '%02d' "${RACK_SLOT:-1}")"
nv set interface lo ip address 10.10.10.101/32
nv config apply
exit 0Operate it with five verbs: ztp -s shows state, ztp -R resets so the next boot provisions again, ztp -v -r <URL> runs a script by hand with verbose output, ztp -e enables it for the next boot and ztp -d disables it.[4] When it goes wrong the evidence is in three places: /var/log/autoprovision, /var/log/syslog, and journalctl -l -u ztp.service.[4] The state strings you will actually read back are disabled, enabled, success and Script Failure.[4]
1. Host prerequisites: DOCA-Host, MFT, rshim
sudo apt install -y doca-all # DEB; RPM: sudo yum install -y doca-all (or doca-ofed / doca-networking) sudo mst start systemctl status rshim
● rshim.service - rshim driver for BlueField SoC
Active: active (running)If it does not match · rshim and MFT ship with DOCA-Host. "failed to setup CUSE rshim" → sudo dnf install kernel-modules-extra; sudo modprobe cuse; systemctl restart rshim. mst status -v should list /dev/mst/mt41692_pciconf0 (BlueField-3 device ID 41692). Keep the apt/yum repo channel and the installed DOCA version on the same release.
sourceSources: BF-Bundle Installation and Upgrade, BlueField Platform Software Troubleshooting (RShim), BSP 4.15.0 bf.cfg guide, rshim-user-space (rshim.8, rshim.conf, bfb-install), BlueField BMC 26.04 (Deploying Software Using BFB, Modes), NVIDIA/skills doca-bf4-deployment.
4Where day-0 actually goes wrong
Situation. A Dell integrator racks 32 SN-series leaves. Images are served over HTTP; one DHCP server hands out both option 114 and option 239. Twenty-nine come up configured. Three do not.
Leaf A: sits at the ONIE prompt. Diagnose by asking what discovery returned. If the switch never got a lease, ONIE loops; onie-discovery-stop gives you a prompt and onie-nos-install http://192.0.2.10/cumulus-linux-5.18.1-mlx-amd64.bin finishes the job by hand.[1] Root cause is almost always the management VLAN, not ONIE.
Leaf B: OS installed, ZTP state is Script Failure. Read /var/log/autoprovision and journalctl -l -u ztp.service for the failing line.[4] Re-run the script by hand with sudo ztp -v -r http://192.0.2.10/ztp-leaf.sh so you see it fail live, fix it, then ztp -R and reboot to prove the unattended path works.[4]
Leaf C: OS installed, ZTP state is disabled. This is the one people misread as a bug. ZTP does not re-run on a box that has already provisioned; ztp -e enables it for the next boot and ztp -R resets state so a re-provision actually happens.[4]
Verification for all three. nv show system version and nv show system image for the build, then ztp -s for the state, then nv config diff startup applied to prove the script’s work is persisted rather than only applied.[2][12]
Fill in the command for each symptom.
Symptom Command
ONIE loops on discovery, no DHCP ____________________
Install an image from the ONIE shell ____________________ http://server/image.bin
Read why the provisioning script failed cat ____________________
Run the provisioning script by hand sudo ztp ____ ____ <URL>
Make the next boot provision again ztp ____ then reboot
Confirm the installed build nv show system ____Then write the one-line reason a switch in state disabled is not a fault.
A customer reinstalls a leaf with nv action install system image onie <URL> force activate reboot to fix an unrelated problem. The switch comes back on the right version, ZTP reports success, and the fabric still does not work: the leaf has its loopback and hostname but none of the interface, bridge or BGP configuration it had yesterday.
Produce: (1) the sentence from the documentation that explains this outcome; (2) the two backups that should have been taken and the exact commands; (3) the one install form that could have preserved the configuration and its flag; (4) a decision rule, in one sentence, for when an operator should choose an ONIE install over the optimized image upgrade path.
Acceptance: your rule names a condition under which the destructive path is the correct choice, not merely the fastest.
Console cable still in the bag
Leaf A gets onie-discovery-stop and a manual onie-nos-install; its real fault was the management VLAN, not ONIE.[1] Leaf B’s autoprovision log names the failing line, so you fix the script, ztp -R, reboot, and watch it reach success unattended.[4] Leaf C was never broken: ZTP does not re-run on a switch that has already provisioned.[4]
What you tell the integrator before he reinstalls anything: “Take both backups first. An ONIE install does not retain the configuration, and only the onie-install -t form carries it through.”[2][8]
The operator labels all three leaves and goes home. GPU hosts land dual-attached on Friday, the second MLAG pair is cabled on Saturday, and your phone goes at 02:40 on Sunday.
Lab
Goal. Build the provisioning infrastructure in ~/containerlab, since the Dell lab has no Spectrum switch to install. You are proving the DHCP and HTTP side, which is the half that actually breaks on customer sites.
Pre-flight inventory: record the containerlab topology file, the bridge names, and ip -br addr on the host before you start, so you can put the host networking back.
- Stand up a Linux container as the provisioning server with dnsmasq or ISC DHCP, handing out option 239 (
option cumulus-provision-url code 239 = text;) and option 114 with an image URL.[1][4] Expected: the daemon starts and logs a listening interface. - Serve a directory over HTTP from the same container holding a dummy image file and a ZTP script carrying
CUMULUS-AUTOPROVISIONING. Expected:curlfrom a second container returns both. - Bring up a plain Linux container as the client and request a lease. Expected: a lease with both options present.
- Prove it on the wire, not in the log:
tcpdump -i <link> -nv port 67 or port 68and read the DHCP options in the offer. Expected: options 114 and 239 with your URLs. If not: the option definition is missing from the server config, which is the single most common cause of a rack that installs but never provisions. - Write the two-line mapping you would hand a Dell deployment engineer: which option ONIE consumes and which option the running OS consumes.
Rollback. containerlab destroy -t <topology> removes the lab bridges and containers; restore any host DHCP or firewall settings you changed from the inventory you took in pre-flight.
Optional, customer lab, mutating and destructive. On a spare switch only, reinstall with onie-nos-install http://<server>/<image> and confirm that configuration is not retained.[1][2] Rollback for that step is not a command: it is the backup you took first - nv config save, scp /etc/nvue.d/startup.yaml, and nv config show -o commands > backup.config - plus, if you want the configuration to survive the install itself, the onie-install -t /etc/nvue.d/startup.yaml form instead.[8] Never run this on a switch carrying traffic.
Goal. Drive a real ZTP run to success, then force it to Script Failure and read the evidence. ZTP is control plane, so DSX Air is faithful for all of it.[11]
- Create a DSX Air simulation and enable OOB. Expected: Air “automatically adds an
oob-mgmt-switchandoob-mgmt-serverto your simulation”.[6] If not: you disabled OOB at creation - and note that “You cannot add, remove, or edit nodes after the simulation starts for the first time”, so recreate rather than fight it.[6] - On
oob-mgmt-server, place a script containing theCUMULUS-AUTOPROVISIONINGcomment that sets a hostname and a loopback and endsexit 0. Air distributes ZTP scripts from this server over DHCP plus a web server, and ships a default bash script you can start from.[6] Expected: the file is reachable over HTTP from a leaf. - On
leaf01, run it by hand first:sudo ztp -v -r http://<oob-mgmt-server>/ztp-leaf.sh. Expected: verbose output ending in success. If not: fix the script now, before you involve DHCP.[4] - Reset and reboot:
sudo ztp -Rthensudo reboot. Expected: after boot,ztp -sreportssuccessand the hostname and loopback are set.[4] - Read the trail:
cat /var/log/autoprovisionandjournalctl -l -u ztp.service. Expected: a start line, the script URL, and a completion line. Record the exact wording - you will quote it to a customer one day.[4] - Now break it on purpose. Edit the script so a command fails (for example an
nv setwith an invalid value) withset -ein force, thensudo ztp -Rand reboot. Expected:ztp -sreportsScript Failure, and the log names the failing line.[4] - Finally, prove the misread: with the switch in
success, reboot again withoutztp -R. Expected: state stays as it is and the script does not re-run. Write down why that is correct behaviour.[4]
Rollback. The simulation is disposable - delete it. Nothing here touches the Dell lab.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell deployment engineer, in five sentences, the difference between ONIE discovery and ZTP, and why a rack of 32 leaves can come up configured without anyone touching a console.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 documentation, Dell KB 000376999 and the DSX Air user guide, fetched 2026-09-07. Dates are when each page was fetched.
- Installing a New Cumulus Linux Image with ONIE | Cumulus Linux 5.18 · fetched 2026-09-07
- Dell Networking Spectrum-X Cumulus Linux: Installing a New Image with ONIE (KB 000376999) · fetched 2026-09-07
- Quick Start Guide | Cumulus Linux 5.18 · fetched 2026-09-07
- Zero Touch Provisioning (ZTP) | Cumulus Linux 5.18 · fetched 2026-09-07
- New and Removed NVUE Commands | Cumulus Linux 5.18 · fetched 2026-09-07
- Custom Topology | NVIDIA DSX Air · fetched 2026-09-07
- NVIDIA DSX Air User Guide · fetched 2026-09-07
- Upgrading Cumulus Linux | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- Cumulus Linux Release Versioning and Support Policy | Knowledge Base · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE CLI | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.