Drivers and firmware: DOCA-OFED, mlxfwmanager, mlxconfig, NVOS
S4·E5The migration plan with one line missing · Hall 2, Friday, five hours before the acceptance signature
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- State the MLNX_OFED lifecycle dates and what DOCA-OFED does and does not replace.
- Prove which InfiniBand packages are present on a host and install the ones the profiles leave out.
- Read PSID and firmware state with mlxfwmanager and set a port to InfiniBand with mlxconfig plus its rollback.
- Choose the correct firmware path for a managed switch an externally managed switch and an adapter.
Episode 5 — The migration plan with one line missing
Forty hosts, one managed switch, two externally managed switches, and a slide that says migrate to DOCA-Host over the weekend. The architect walks the room through it; the network lead’s notebook is open at Wednesday’s page. The Dell SE is here because the PowerEdge platform firmware falls in the same window; his spreadsheet has one row open: fabric accepted Friday. You are here to say yes, or what is missing.
What is missing is one line, and it cost eleven hours on Wednesday. The plan assumes the driver package brings the fabric tooling with it.
The transition is not a matter of opinion, and the dates are published: October 2024 was “The last standalone release of MLNX_OFED”, after which it receives no new features or enhancements; until October 2027 that final release sits in long-term support with “critical bug fixes and security updates”; and in October 2027 “MLNX_OFED will no longer receive support or updates by NVIDIA.”[1] The replacement claim is equally explicit: “DOCA-OFED is a 1-to-1 substitute for MLNX_OFED”.[1] For the driver stack that holds. For an InfiniBand fabric it is incomplete, because “The installation procedure does not install proprietary packages. Those packages are installed upon request”, and that list includes ibutils2 and opensm.[2] A host can install a profile cleanly, load every driver, pass ibstat, and still have neither a subnet manager nor the diagnostic tool of the last two lessons.[2][10]
A 1-to-1 substitute for the drivers is not one for the fabric.
So you start at the beginning: what the transition replaces, and what it leaves behind.
1The transition, and the surprise it causes
The lifecycle is published and worth quoting to a customer verbatim. October 2024 was “The last standalone release of MLNX_OFED. Following this release, MLNX_OFED will no longer receive support for new features or enhancements.”[1] From then until October 2027 the final release sits in long-term support, receiving “critical bug fixes and security updates for MLNX_OFED users as part of its long-term support (LTS) plan”.[1] In October 2027 “MLNX_OFED will no longer receive support or updates by NVIDIA.”[1]
The replacement claim is equally explicit: “DOCA-OFED is a 1-to-1 substitute for MLNX_OFED”, offering the same functionality and user experience.[1] For the driver stack that is true. For an InfiniBand fabric it is incomplete, and the gap is where the tickets come from.[2]
The DOCA-Host installation page states it plainly: “The installation procedure does not install proprietary packages. Those packages are installed upon request”, and the list of those packages includes hcoll, sharp, ibutils2 and opensm alongside Clusterkit and DPCP.[2] ibdiagnet lives in ibutils2; the subnet manager lives in opensm.[2] So a host can install doca-all perfectly, load every driver, pass ibstat, and have neither a subnet manager nor the diagnostic tool from the previous two lessons.[2][10]
The install itself is short, and the two post-install commands are not optional: after the profile package, run sudo /etc/init.d/openibd restart to load the drivers and sudo mst restart to initialise MST.[2] The transition guide adds one prerequisite that catches RPM-based hosts: libibumad3, libibumad-dev and librdmacm-dev come from the distribution, and “if any other rdma tools (e.g. ibstat via infiniband-diags package) are needed they need to be installed from upstream before installing doca-host package repo.”[1]
The profile lists in the two documents do not match. The transition guide names doca-ofed, doca-networking and doca-all; the DOCA-Host 3.5.0 installation page names doca-all, doca-ofed, doca-libvma and doca-libxlio.[1][2] Quote the page for the release the customer is actually installing, and say which page you are quoting.[1][2]
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2mlxfwmanager: what is on the flash right now
“The mlxfwmanager is a firmware update and query utility which scans the system for available NVIDIA devices (only mst PCI devices) and performs the necessary firmware updates.”[3] The words “only mst PCI devices” explain the first failure mode: without mst start the tool finds nothing and the customer concludes the card is dead.[3][2]
The three commands that matter:[3]
mlxfwmanager --query # every device
mlxfwmanager -d 09:00.0 --query # one device by BDF
mlxfwmanager --online -u -d 0000:09:00.0 -y # fetch from NVIDIA and update, unattended--query reports device type, Part Number, PSID, PCI device name, base MAC or GUID, the current versus available firmware and a status such as “Up to date” or “Update required”.[3] --online fetches the required image from NVIDIA’s servers, -u applies it, -i uses a local image instead, --list-content lists the images inside a package, --nofs burns non-failsafe and --force overrides standard update restrictions.[3] The PSID is the identity that makes the whole thing safe: a 16-ASCII-character string embedded in the image that uniquely identifies the configuration of that firmware, which is what stops a generic image landing on an OEM-specific card.[7]
Two cautions. --clear-semaphore carries its own warning: it “May result in system instability or flash corruption if the device or another application is currently using the flash. Exercise caution.”[3] And --force, by definition, disables the checks that would otherwise stop you.[3]
For the fabric rather than the host, do not iterate over hosts: ibdiagnet2.nodes_info carries “Nodes Information (FW version, etc)” for every node one sweep saw, switches included.[9]
3mlxconfig: turning a port into an InfiniBand port
mlxconfig -d <device> [command] [parameters] reads and writes the card’s non-volatile configuration; the device is either a BDF with the pciconf- prefix or a /dev/mst/ path.[4] Three forms cover everything in this lesson: mlxconfig -d <device> q [PARAMETER] to query, mlxconfig -d <device> set [PARAMETER=VALUE] to set, and mlxconfig -f ./tlv_file.conf -d <device> set_raw for a raw TLV file.[4]
The port-type parameter is the one that matters on a lab card that arrived in Ethernet mode: mlxconfig -d <device> set [LINK_TYPE_P1=<link_type>] [LINK_TYPE_P2=<link_type>], where IB is 1 and ETH is 2.[4] After a set, the tool tells you “Please reboot machine to load new configurations”, and the documented alternative is flint -d <device> swreset.[4] Nothing changes until one of those happens.[4]
Two honest limits on what this course can claim. First, the widely repeated statement that both VPI ports default to InfiniBand and that ConnectX-4 and later can be set per port comes from a search summary of a page that no longer resolves — treat it as unverified and read the actual values with q instead of assuming them.[4] Second, mlxconfig has no documented -y flag; that flag belongs to mlxfwmanager, so do not paste it into a mlxconfig command in a runbook.[3][4]
The rollback for a port-type change is entirely procedural, and it is the reason step one is a query: record the value you found, and if the port was LINK_TYPE_P1=2 before your change, restoring it means setting 2 again and repeating the reset.[4] Verification is ibstat, which reports the link layer along with LID, SMLID, port state and physical state.[10]
4Switches: two firmware paths, and one that reboots
The first question about switch firmware is not which version but which kind of switch.[7]
For managed systems, “The firmware of managed switch systems is automatically performed by management software - MLNX-OS.”[7] For externally managed systems, they “require the use of a Mellanox firmware burning tool like flint or mlxburn, which are part of the MFT package.”[7] The prescribed procedure on the firmware portal is download the MFT package and its documentation, install MFT per its manual, get the switch’s user manual and follow the burning instructions there.[7]
On NVOS the image workflow is three commands and a prompt:[5]
nv action fetch system image scp://<username>:<password>@<ip-address>/var/www/html/<new_image>
nv show system image files
nv action install system image files nvos-amd64-25.01.0003.binThe prompt is explicit that this is a reboot: “Operation will reboot the system. If you choose ‘y’, the system will install the image and reboot.”[5] Two facts to put in the change record before anyone types y. The image model is dual partition — after the reboot nv show system image --view detail returns rows current, next, partition1 and partition2, which is your rollback path.[5] And “Configuration files on the switch are not saved” during installation, so backing up the configuration is a prerequisite and not a nicety.[5]
ASIC firmware is a separate flow: nv action fetch platform firmware ASIC /path/to/fw-image.mfa, nv action install platform firmware ASIC files fw-image.mfa, nv show platform firmware ASIC, then nv action reboot system.[6] NVOS also does some of this for you: “NVOS software package version has a default switch firmware version. When updating the operating system software to a new version, an automatic firmware update process will be attempted by NVOS.”[6] The caveats are the ones that decide the maintenance window: a power cycle is required at the end, each CLI command must complete before the next begins, “Upgrading one CPLD automatically upgrades all CPLDs”, and updates are not performed in every release, so the release notes decide whether there is anything to do.[6][8]
Diagram source (Mermaid)
flowchart TD; A["What am I updating?"] --> B["Adapter: ConnectX or BlueField"]; A --> C["Managed switch: MLNX-OS or NVOS"]; A --> D["Externally managed switch"]; B --> B1["mst start then mlxfwmanager --query"]; B1 --> B2["Read PSID and current versus available"]; B2 --> B3["mlxfwmanager --online -u -d bdf -y"]; C --> C1["nv action fetch system image"]; C1 --> C2["Back up config first: install does not save it"]; C2 --> C3["nv action install system image files"]; C3 --> C4["Reboot then nv show system image --view detail"]; D --> D1["MFT flint or mlxburn per the switch manual"]; B3 --> E["Verify fabric-wide with ibdiagnet2.nodes_info"]; C4 --> E; D1 --> E
A Dell customer asks: “What firmware should our ConnectX-7 cards be on?” Answer it with reads, not with a number from memory.
# 1. Make the devices visible to the MFT tools.
sudo mst start && mst status
# 2. Ground truth per card: PSID, current and available.
sudo mlxfwmanager --query | tee /var/tmp/fw-$(hostname).txt
# 3. Ground truth for the whole fabric in one sweep.
ibdiagnet -o /var/tmp/fwsweep
grep -i fw /var/tmp/fwsweep/ibdiagnet2.nodes_info | head -40
# 4. What driver stack are they on, and do they have the IB tooling at all?
ofed_info -s ; rpm -q ibutils2 opensm || dpkg -l | grep -E 'ibutils2|opensm'Reasoning: step 1 exists because mlxfwmanager scans only mst PCI devices.[3] Step 2 gives the PSID plus the current versus available pair and the “Up to date” or “Update required” status.[3] Step 3 replaces a host-by-host survey: ibdiagnet2.nodes_info carries firmware for every node the sweep saw.[9] Step 4 is the packaging check from segment 1, and its absence answers a different ticket before it is opened.[2]
The answer you send is a shape, not a number: the firmware has to satisfy the DOCA-Host release they run, the Dell-qualified level for that PowerEdge platform, and the switch firmware in the same fabric — and here is the current state of all three, attached.[3][9][8]
Same customer, but one card came from stock in Ethernet mode and has to join the IB fabric.
sudo mlxconfig -d /dev/mst/<device> q | grep ________ # record what it is now
sudo mlxconfig -d /dev/mst/<device> set LINK_TYPE_P1=____ # 1 is ____ and 2 is ____
sudo ________ # make it take effect
ibstat # prove the link layerFill the blanks, then write the exact rollback command you would run if the customer decides tomorrow that the port must go back to Ethernet, and say what has to happen after it before ibstat will agree.[4][10]
A customer is moving 40 hosts from MLNX_OFED LTS to DOCA-Host, and their fabric has one managed QM9700 and two externally managed switches.
Produce the pre-upgrade checklist. Acceptance criteria: it names the packages that must be installed separately and the exact command that proves each one is present afterwards; it names the two mandatory post-install commands; it states which of the two published profile lists you are quoting and why; it gives a different firmware path for the adapters, the managed switch and the externally managed switches, with the tool named for each; it requires a configuration backup where the documentation says configuration is not preserved; and it ends with one fabric-wide verification command rather than 40 host checks.[1][2][7][5][9]
Case closed — an intersection, not a number
The plan leaves the room with three additions: the packages that are only installed on request plus the query that proves each one is present, the two mandatory post-install commands, and a different firmware path per device class — mlxfwmanager for the adapters, the NVOS image flow for the managed switch, MFT flint or mlxburn for the externally managed pair.[2][3][5][7] Plus a configuration backup, because the install does not save it.[5] The certificate is signed at 17:40. What you tell the architect: I cannot give you a firmware number, only an intersection. And on the management host, where Monday’s run begins, the operator has label-printed four words: TAKE THE CACHE FIRST.
Lab
On the Dell-lab BlueField-3 or ConnectX. Steps 1-4 are read-only. Step 5 writes non-volatile configuration and names its rollback.
- Pre-flight inventory, read-only:
sudo mst start,mst status,ibstat,ofed_info -s. Expected: at least one/dev/mst/device, andibstatreporting the port’s current link layer and state.[3][10] Ifmst statusis empty, nothing else in this lab will find the card.[3] - Firmware ground truth:
sudo mlxfwmanager --query | tee /var/tmp/fw-before.txt. Expected: device type, Part Number, PSID, current firmware, available firmware and an up-to-date or update-required status.[3] Record the PSID — it is the identity you quote in any support case.[3] - Configuration ground truth:
sudo mlxconfig -d /dev/mst/<device> q | tee /var/tmp/mlxconfig-before.txtandgrep LINK_TYPE /var/tmp/mlxconfig-before.txt. Expected:LINK_TYPE_P1and, on a two-port card,LINK_TYPE_P2, each showing IB as 1 or ETH as 2.[4] This file is your rollback; do not skip it. - Fabric-wide firmware, read-only:
ibdiagnet -o /tmp/fwsweepthen read/tmp/fwsweep/ibdiagnet2.nodes_info. Expected: FW version per node. Compare against step 2 for the local card; they must agree.[9] - Mutating with rollback: move a port to InfiniBand with
sudo mlxconfig -d /dev/mst/<device> set LINK_TYPE_P1=1, then apply it with a reboot orsudo flint -d /dev/mst/<device> swreset, then confirm withibstatthat the link layer is InfiniBand.[4][10] Rollback: if/var/tmp/mlxconfig-before.txtrecordedLINK_TYPE_P1=2, restore it withsudo mlxconfig -d /dev/mst/<device> set LINK_TYPE_P1=2and repeat the reset, then re-checkibstat.[4] Do not add-ytomlxconfig; that flag is documented formlxfwmanager, not for this tool.[3][4] - Optional, read-only, on a customer lab: compare every firmware version in
ibdiagnet2.nodes_infoagainst the release notes for the release the customer runs, and write the one-line answer to “are we consistent” rather than “are we newest”.[9][8] - Optional, on a lab switch only and inside a maintenance window: read
nv show system image --view detailand note thecurrent,next,partition1andpartition2rows without installing anything. Expected: you can state the rollback image by name before any upgrade is attempted, and you have a configuration backup because installation does not preserve it.[5]
Documentation and checklist work. No adapter required.
- Write the pre-upgrade checklist for a customer moving from MLNX_OFED LTS to DOCA-Host. It must name every package the installer leaves out and, for each, the command that proves it is present afterwards.[2] Expected: your list contains
opensmandibutils2at minimum, withrpm -q ibutils2 opensmordpkg -l | grep -E 'ibutils2|opensm'as the proof.[2] - Add the post-install verification:
sudo /etc/init.d/openibd restart,sudo mst restart, thenibstat.[2][10] Expected: three commands in that order, with the reason for each written next to it. - Reconcile the two published profile lists — the transition guide’s three names against the 3.5.0 install page’s four — and write one sentence stating which you would quote to a customer and why.[1][2] Expected: your answer is tied to the release being installed, not to a preference.
- Write the lifecycle paragraph you would send to a customer still standardised on MLNX_OFED, using the three published dates and the 1-to-1 substitute wording.[1]
- Draw the firmware decision as three branches — adapter, managed switch, externally managed switch — and name the tool and the disruptive step on each branch.[7][5][6] Expected:
flintormlxburnappears only on the externally managed branch, and the power-cycle requirement appears on the ASIC firmware path.[6] - Open the CompatMatrix above on the profiles tab and check your answer to step 3 against it; then note the one thing the matrix does not show, which is that
opensmandibutils2sit outside every profile.[2]
Nothing here changes a host.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell customer, in five sentences, why upgrading from MLNX_OFED to DOCA-Host made ibdiagnet vanish and what the fix is.
Sources
Facts in this lesson were checked against MLNX_OFED to DOCA-OFED transition guide (DOCA 3.4.0) and DOCA-Host 3.5.0 installation page, both re-fetched 2026-09-07; MFT 4.36.0 mlxfwmanager page re-fetched 2026-09-07; MFT 4.30.18 LTS mlxconfig; NVOS image install v25.02.3000 and system firmware v25.02.2002; Quantum-2 firmware portal and release notes 31.2014.2084. Dates are when each page was fetched.
- MLNX_OFED to DOCA-OFED Transition Guide (DOCA 3.4.0) · fetched 2026-09-07
- DOCA-Host Installation and Upgrade (DOCA 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
- mlxfwmanager Firmware Update and Query Tool (MFT 4.36.0) · fetched 2026-09-07
- Using mlxconfig (MFT 4.30.18 LTS) · fetched 2026-09-07
- Installing a New NVOS Image (NVOS UM v25.02.3000) · fetched 2026-09-07
- Upgrading System Firmware (NVOS UM v25.02.2002) · fetched 2026-09-07
- Updating Firmware for NVIDIA Quantum-2 Based InfiniBand Switch Platforms · fetched 2026-09-07
- NVIDIA Quantum-2 Firmware Release Notes v31.2014.2084 · fetched 2026-09-07
- ibdiagnet Dump Files (IBUtils2 2.24.0) · fetched 2026-09-07
- ibstat(8) man page source (rdma-core master) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.