Skip to content
nodeza.learn
DOCA course/2. Bring-up/Lesson 2.2

The MFT toolbox: read-only vs mutating

S2·E2The card that vanished at 02:40 · Bridge call from the hotel, Tuesday 02:40, three days to the demo

S2·E2Apply~30 minsources checked todayverified against MFT 4.34.1-18 LTS General Information, DOCA 3.5.0 BlueField Modes of Operation, Dell KB 000300192, 2026-09-06

Builds on: DOCA-Host install and profiles

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Start MFT, enumerate BlueField devices and map an mst device to its PCIe address and rshim instance.
  • Classify every common MFT invocation as read-only or mutating before running it.
  • Read firmware version, configuration and link state from a BlueField-3 without changing anything.
  • State the cold power-cycle rule and the Dell evidence for it.

Episode 2 — The card that vanished at 02:40

The situation · Bridge call from the hotel, Tuesday 02:40, three days to the demo

The night-shift operator is already talking when you join. Four hours ago he updated BlueField-3 firmware on one of the eight XE9680L nodes, rebooted, and now lspci shows nothing at the card’s bus address. He has printed a label, “RMA — DO NOT RACK”, and stuck it on the rail; it is the most decisive artifact anyone has produced tonight. The procurement person, awake for reasons nobody asks about, wants the lead time on a replacement. Someone else proposes running flint and mlxfwreset from two windows at once.

You ask one question: was that a power cycle, or a warm reboot. It was reboot. Dell KB 000300192 records exactly this on the XE9680L, where a DPU firmware update without a power cycle leaves a fatal error on the component’s bus in the Lifecycle Log, and states, per NVIDIA, that a system power cycle is required; a warm reboot is not enough.[8]

MFT exists because a NIC is a computer with its own firmware, and something has to read and write that firmware below the OS driver, through the mst register-access path.[1] Every tool in the box has a reading form and a writing form, and the discipline of a bring-up is knowing which one you just typed. The documentation is also explicit that running two of these tools on the same device at once is not supported and might cause unexpected behavior, so the two-window plan is cancelled.[1]

Read before you write, and never from two windows.

Segment 1 opens the safe half of the toolbox.

1The toolbox and its one hard rule

MFT (Mellanox Firmware Tools) is the low-level toolbox for ConnectX and BlueField: it talks to the device through the mst register-access driver, below DOCA and below the OS driver. The reference used here is “MFT 4.34.1-18 LTS (2025 LTS U2)”.[1] Start with sudo mst start and list devices with sudo mst status -v; the install page has you run sudo mst restart after a DOCA-Host install.[1][10]

A BlueField-3 shows up as /dev/mst/mt41692_pciconf0. The number is the device ID: the BlueField-3 user guide states that “the Device ID of all DPUs is 41692”, so the SKU does not change the prefix and the suffix counts devices on the host.[7] The 3.5.0 modes page writes /dev/mst/<device> generically; you fill in what mst status printed.[2]

Before MFT, use the bus. lspci -d 15b3: (vendor 15b3 is Mellanox/NVIDIA; add -nn for numeric IDs) lists every function the host sees.[5] On a host with more than one DPU, cross-match each /dev/rshim<N>/misc DEV_NAME (for example pcie-04:00.2) with those addresses before any bfb-install; the deployment skill calls skipping that step “the #1 cause of ‘I flashed the wrong DPU’ incidents”.[5] The RShim PF is the .2 function of the BlueField PCIe device, which is why the mapping works.[12]

The one hard rule: “Running MFT tools in parallel or simultaneously on the same device is not supported and might cause unexpected behavior.”[1] That includes a monitoring script that polls mlxlink while you run flint. Serialise everything that touches one device.

SymptomBlueField-3 on a Dell Pow…SymptomBring-up: card, rshim, BF…SymptomHost does not see the Blu…CheckIs the card enumerated o…
Symptom

Host does not see the BlueField

Start at the device node: the card is missing or misbehaving. Each branch names the read-only MFT command that decides it.

2Read-only versus mutating

Every MFT tool has a read form and most have a write form. Classify before you type. The official one-liners come from the MFT General Information page; the class column is what those descriptions imply.[1]

Tool Official description Read-only form Mutating form
mst Lists the available mst devices; starts and stops the register access driver mst status -v mst start, mst stop only load a driver on the host
mlxconfig Change some of the device configurations without creating and burning a new firmware mlxconfig -d <dev> q mlxconfig -d <dev> s PARAM=value (needs a reset)
flint Burns a firmware binary image or an expansion ROM image to the flash flint -d <dev> q flint -d <dev> -i <image> burn
mlxfwmanager A firmware update and query utility with single-click update mlxfwmanager --query update run (burns firmware)
mlxfwreset Loads the firmware after firmware update on ISFU-capable devices none always mutating: resets the device
mlxprivhost Enables the user to restrict the hosts from configuring the NIC mlxprivhost -d <dev> q r (restrict) and p (privilege); run on the Arm
mlxlink Displays and configures port related data at the physical layer mlxlink -d <dev> any configure option
mlxdump Dumps device internal configuration registers always none
mlxreg Exposes supported access registers get set
mlxburn Generates firmware images, burns, queries versions and VPD query burn

The hardware-safety skill draws the same line from the other side. Its list of operations that need the full safety workflow is: “mlxconfig firmware-parameter write, NIC firmware burn, BFB reflash, NIC ↔ DPU mode flip, SR-IOV or device-emulation slot enable, kernel boot-parameter change (IOMMU, hugepages, VFIO), PCIe rebind/rescan/link-state flip, or BlueField cold reboot”; the firmware-burn tools it names are flint, mft, mlxfwmanager and “mlxconfig with -y reset”.[3] If a command is on that list, it needs a window, an out-of-band path and a rollback. If it is not, run it whenever you need the answer.

The mode parameter is the most common write. mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE returns ENABLED(0) in DPU mode, DISABLED(1) in NIC mode, and shows an RO flag when Zero Trust has locked it.[2] Setting it is mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1 (NIC) or =0 (DPU).[2]

3The cold power-cycle rule

The modes page is explicit: “For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended.”[2] The hardware-safety skill hardens that into “the mlxconfig cold-power-cycle rule”: a warm reboot is not treated as sufficient.[3] NVIDIA’s own vSphere procedure ends the same way: “Power cycle the system after switching to NIC mode”, then verify with mlxconfig -d mt41692_pciconf0 q | grep -i offload showing DISABLED.[9]

Dell has the evidence for firmware, not just configuration. KB 000300192 (XE9680L) records that after updating the DPU firmware without a power cycle the Lifecycle Log reports “A fatal error was detected on a component at bus [number], device 0, function 0”, and states, per NVIDIA, that “a system power cycle is required after updating the DPU firmware”; a warm reboot is insufficient.[8] mlxfwreset exists to load new firmware on ISFU-capable devices, but the BlueField guidance in both the NVIDIA and Dell documents is the power cycle; treat mlxfwreset as an option you have read about, not the default you reach for on a PowerEdge.[1][8]

What the rule means in practice on a Dell host: the change window for any mlxconfig s, flint burn or mlxfwmanager update includes a full power cycle from iDRAC (or the front panel), and the ticket is not closed until a read-only query afterwards shows the intended value. For firmware, compare FW Version: against the release’s General Support table (32.50.1002 for BlueField-3 in 3.5.0).[4][11]

Read a BlueField-3 completely without changing it

Goal: a full read-only picture of one card on a Dell-lab host, to attach to a ticket or to serve as the BEFORE state for a later change.

  1. Bus view, no driver needed: lspci -d 15b3: -nn. Expect one bus address per BlueField-3 carrying its network PFs (one per port) plus the .2 RShim function.[5][12] Only the RShim function is fixed by the documentation; the number of network functions follows the SKU’s port count, so a single-port part such as the B3140H shows fewer of them.[7][12]
  2. Start MFT: sudo mst start && sudo mst status -v. Expect /dev/mst/mt41692_pciconf0 with the matching PCIe address.[1][7]
  3. Tie the rshim to the card: grep DEV_NAME /dev/rshim0/misc should print the same address as step 1 with function .2.[12]
  4. Firmware: sudo flint -d /dev/mst/mt41692_pciconf0 q. Record FW Version: and FW Release Date:; compare with 32.50.1002 for DOCA 3.5.0.[4][11]
  5. Second opinion on firmware: sudo mlxfwmanager --query.[1]
  6. Mode: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE. ENABLED(0) is DPU mode, DISABLED(1) is NIC mode, RO means Zero Trust.[2]
  7. Link: sudo mlxlink -d /dev/mst/mt41692_pciconf0 for physical-layer state, display only.[1]
  8. Every command above is a read; nothing needs a window, and nothing needs a power cycle afterwards.

End of Episode 2 — After the power cycle

How it ended

iDRAC power-cycles the chassis. After POST, lspci -d 15b3: lists the functions again, mst status -v prints /dev/mst/mt41692_pciconf0, and flint q shows the firmware version the operator believed he had installed.[1][5] Nobody quotes a replacement. The label goes in his pocket, for later. Every read-only output is saved as the night’s record, and the mode value becomes the BEFORE state for the change request to come.

What you say on the call: “On BlueField a reboot is not a reset. The new value loads when the card comes out of reset, so every firmware or mlxconfig change ends with a power cycle and a read that confirms it.”[2][8]

By Wednesday noon the SE reports twenty-four BF-Bundle pushes, exit code zero every time, and not one SSH session to an Arm.

Lab

All steps are read-only; nothing here changes firmware, mode or link state.

  1. lspci -d 15b3: -nn. Expected: one bus address per BlueField-3 carrying its network PFs (one per port, so two on a dual-port B3220 and one on a single-port B3140H) plus the .2 RShim function; only the .2 function is fixed by the documentation, so do not grade this step on a fixed count.[12][7] If nothing prints, the card is not enumerated; stop and go to lesson 2.6 (aux power, PCIe init).[5]
  2. sudo mst start && sudo mst status -v. Expected: /dev/mst/mt41692_pciconf0 and the PCIe address from step 1. If mst start fails, DOCA-Host or MFT is not installed correctly; do not continue.[1][7]
  3. grep -E 'DEV_NAME|DEV_INFO' /dev/rshim0/misc. Expected: DEV_NAME pcie-<bus>:00.2 matching step 1. If there is no /dev/rshim0, the rshim service is not running or the BMC owns rshim (lesson 2.4).[12]
  4. sudo flint -d /dev/mst/mt41692_pciconf0 q | grep -E 'FW Version|FW Release Date|PSID'. Record both version and date; compare with 32.50.1002 for 3.5.0.[4][11]
  5. sudo mlxfwmanager --query. Expected: the same firmware version as step 4; a difference means one of the tools is reading a different device.[1]
  6. sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE. Record ENABLED(0), DISABLED(1) or RO.[2]
  7. sudo mlxlink -d /dev/mst/mt41692_pciconf0. Record link state and speed per port.[1]
  8. Save all outputs as ~/bf3-readonly-$(date +%F).txt. This file is the BEFORE state required by the mode-switching lesson.[6]

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, how you tell a safe MFT command from a dangerous one and why a reboot is not always a reboot.

12 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against MFT 4.34.1-18 LTS General Information, DOCA 3.5.0 BlueField Modes of Operation, Dell KB 000300192, 2026-09-06. Dates are when each page was fetched.

  1. MFT 4.34.1-18 LTS: General Information (tool list) · fetched 2026-09-06
  2. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  3. NVIDIA/skills: doca-hardware-safety SKILL (mutating operations list, cold power-cycle rule) · fetched 2026-09-06
  4. NVIDIA/skills: doca-version CAPABILITIES (flint q anchors) · fetched 2026-09-06
  5. NVIDIA/skills: doca-bf3-deployment CAPABILITIES (lspci 15b3, DEV_NAME cross-match) · fetched 2026-09-06
  6. NVIDIA/skills: doca-bf3-deployment TASKS (pre-flight capture) · fetched 2026-09-06
  7. NVIDIA BlueField-3 Networking Platform User Guide (device ID 41692) · fetched 2026-09-06
  8. Dell KB 000300192: XE9680L, a power cycle is required after updating DPU firmware · fetched 2026-09-06
  9. HowTo Configure NVIDIA BlueField-3 to NIC Mode on VMware vSphere 8.0 · fetched 2026-09-06
  10. DOCA-Host Installation and Upgrade (mst restart) · fetched 2026-09-06 · DOCA 3.5.0
  11. DOCA General Support (firmware table) · fetched 2026-09-06 · DOCA 3.5.0
  12. BlueField Platform Software Troubleshooting Guide: SoC Management Interface (RShim) · fetched 2026-09-06

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.