The MFT toolbox: read-only vs mutating

Episode 2 — The card that vanished at 02:40

The situation · Bridge call from the hotel, Tuesday 02:40, three days to the demo

Bridge call from the hotelTuesday 02:40, three days to the demo

The night-shift operator is already talking when you join. Four hours ago he updated BlueField-3 firmware on one of the eight XE9680L nodes, rebooted, and now lspci shows nothing at the card’s bus address. He has printed a label, “RMA — DO NOT RACK”, and stuck it on the rail; it is the most decisive artifact anyone has produced tonight. The procurement person, awake for reasons nobody asks about, wants the lead time on a replacement. Someone else proposes running flint and mlxfwreset from two windows at once.

You ask one question: was that a power cycle, or a warm reboot. It was reboot. Dell KB 000300192 records exactly this on the XE9680L, where a DPU firmware update without a power cycle leaves a fatal error on the component’s bus in the Lifecycle Log, and states, per NVIDIA, that a system power cycle is required; a warm reboot is not enough.^⁠8

MFT exists because a NIC is a computer with its own firmware, and something has to read and write that firmware below the OS driver, through the mst register-access path.^⁠1 Every tool in the box has a reading form and a writing form, and the discipline of a bring-up is knowing which one you just typed. The documentation is also explicit that running two of these tools on the same device at once is not supported and might cause unexpected behavior, so the two-window plan is cancelled.^⁠1

Read before you write, and never from two windows.

Segment 1 opens the safe half of the toolbox.

1The toolbox and its one hard rule

MFT (Mellanox Firmware Tools) is the low-level toolbox for ConnectX and BlueField: it talks to the device through the mst register-access driver, below DOCA and below the OS driver. The reference used here is “MFT 4.34.1-18 LTS (2025 LTS U2)”.^⁠1 Start with sudo mst start and list devices with sudo mst status -v; the install page has you run sudo mst restart after a DOCA-Host install.^⁠1^⁠10

A BlueField-3 shows up as /dev/mst/mt41692_pciconf0. The number is the device ID: the BlueField-3 user guide states that “the Device ID of all DPUs is 41692”, so the SKU does not change the prefix and the suffix counts devices on the host.^⁠7 The 3.5.0 modes page writes /dev/mst/<device> generically; you fill in what mst status printed.^⁠2

Before MFT, use the bus. lspci -d 15b3: (vendor 15b3 is Mellanox/NVIDIA; add -nn for numeric IDs) lists every function the host sees.^⁠5 On a host with more than one DPU, cross-match each /dev/rshim<N>/misc DEV_NAME (for example pcie-04:00.2) with those addresses before any bfb-install; the deployment skill calls skipping that step “the #1 cause of ‘I flashed the wrong DPU’ incidents”.^⁠5 The RShim PF is the .2 function of the BlueField PCIe device, which is why the mapping works.^⁠12

The one hard rule: “Running MFT tools in parallel or simultaneously on the same device is not supported and might cause unexpected behavior.”^⁠1 That includes a monitoring script that polls mlxlink while you run flint. Serialise everything that touches one device.

Troubleshooting tree

Symptom

Host does not see the BlueField

Start at the device node: the card is missing or misbehaving. Each branch names the read-only MFT command that decides it.

Why does the deployment skill make you cross-match DEV_NAME against lspci before pushing a BFB, when the host has only one rshim device today?

Question 1 of 3

2Read-only versus mutating

Every MFT tool has a read form and most have a write form. Classify before you type. The official one-liners come from the MFT General Information page; the class column is what those descriptions imply.^⁠1

Tool	Official description	Read-only form	Mutating form
`mst`	Lists the available mst devices; starts and stops the register access driver	`mst status -v`	`mst start`, `mst stop` only load a driver on the host
`mlxconfig`	Change some of the device configurations without creating and burning a new firmware	`mlxconfig -d <dev> q`	`mlxconfig -d <dev> s PARAM=value` (needs a reset)
`flint`	Burns a firmware binary image or an expansion ROM image to the flash	`flint -d <dev> q`	`flint -d <dev> -i <image> burn`
`mlxfwmanager`	A firmware update and query utility with single-click update	`mlxfwmanager --query`	update run (burns firmware)
`mlxfwreset`	Loads the firmware after firmware update on ISFU-capable devices	none	always mutating: resets the device
`mlxprivhost`	Enables the user to restrict the hosts from configuring the NIC	`mlxprivhost -d <dev> q`	`r` (restrict) and `p` (privilege); run on the Arm
`mlxlink`	Displays and configures port related data at the physical layer	`mlxlink -d <dev>`	any configure option
`mlxdump`	Dumps device internal configuration registers	always	none
`mlxreg`	Exposes supported access registers	`get`	`set`
`mlxburn`	Generates firmware images, burns, queries versions and VPD	query	burn

The hardware-safety skill draws the same line from the other side. Its list of operations that need the full safety workflow is: “mlxconfig firmware-parameter write, NIC firmware burn, BFB reflash, NIC ↔ DPU mode flip, SR-IOV or device-emulation slot enable, kernel boot-parameter change (IOMMU, hugepages, VFIO), PCIe rebind/rescan/link-state flip, or BlueField cold reboot”; the firmware-burn tools it names are flint, mft, mlxfwmanager and “mlxconfig with -y reset”.^⁠3 If a command is on that list, it needs a window, an out-of-band path and a rollback. If it is not, run it whenever you need the answer.

The mode parameter is the most common write. mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE returns ENABLED(0) in DPU mode, DISABLED(1) in NIC mode, and shows an RO flag when Zero Trust has locked it.^⁠2 Setting it is mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1 (NIC) or =0 (DPU).^⁠2

Why is mlxfwreset classified as mutating when it does not write any bytes to flash?

Question 2 of 3

3The cold power-cycle rule

The modes page is explicit: “For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended.”^⁠2 The hardware-safety skill hardens that into “the mlxconfig cold-power-cycle rule”: a warm reboot is not treated as sufficient.^⁠3 NVIDIA’s own vSphere procedure ends the same way: “Power cycle the system after switching to NIC mode”, then verify with mlxconfig -d mt41692_pciconf0 q | grep -i offload showing DISABLED.^⁠9

Dell has the evidence for firmware, not just configuration. KB 000300192 (XE9680L) records that after updating the DPU firmware without a power cycle the Lifecycle Log reports “A fatal error was detected on a component at bus [number], device 0, function 0”, and states, per NVIDIA, that “a system power cycle is required after updating the DPU firmware”; a warm reboot is insufficient.^⁠8 mlxfwreset exists to load new firmware on ISFU-capable devices, but the BlueField guidance in both the NVIDIA and Dell documents is the power cycle; treat mlxfwreset as an option you have read about, not the default you reach for on a PowerEdge.^⁠1^⁠8

What the rule means in practice on a Dell host: the change window for any mlxconfig s, flint burn or mlxfwmanager update includes a full power cycle from iDRAC (or the front panel), and the ticket is not closed until a read-only query afterwards shows the intended value. For firmware, compare FW Version: against the release’s General Support table (32.50.1002 for BlueField-3 in 3.5.0).^⁠4^⁠11

Read a BlueField-3 completely without changing it

Every step is worked for you. Read it once, then try the faded version.

Goal: a full read-only picture of one card on a Dell-lab host, to attach to a ticket or to serve as the BEFORE state for a later change.

Bus view, no driver needed: lspci -d 15b3: -nn. Expect one bus address per BlueField-3 carrying its network PFs (one per port) plus the .2 RShim function.^⁠5^⁠12 Only the RShim function is fixed by the documentation; the number of network functions follows the SKU’s port count, so a single-port part such as the B3140H shows fewer of them.^⁠7^⁠12
Start MFT: sudo mst start && sudo mst status -v. Expect /dev/mst/mt41692_pciconf0 with the matching PCIe address.^⁠1^⁠7
Tie the rshim to the card: grep DEV_NAME /dev/rshim0/misc should print the same address as step 1 with function .2.^⁠12
Firmware: sudo flint -d /dev/mst/mt41692_pciconf0 q. Record FW Version: and FW Release Date:; compare with 32.50.1002 for DOCA 3.5.0.^⁠4^⁠11
Second opinion on firmware: sudo mlxfwmanager --query.^⁠1
Mode: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE. ENABLED(0) is DPU mode, DISABLED(1) is NIC mode, RO means Zero Trust.^⁠2
Link: sudo mlxlink -d /dev/mst/mt41692_pciconf0 for physical-layer state, display only.^⁠1
Every command above is a read; nothing needs a window, and nothing needs a power cycle afterwards.

lspci -d ____: -nn — expect one network function per ____ plus the .____ RShim function.
sudo mst ____ && sudo mst ____ -v.
grep ____ /dev/rshim0/misc matches the PCIe address with function .____.
sudo flint -d /dev/mst/____ q — record FW ____: and FW ____ ____:.
sudo mlxfwmanager --____.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q ____ — ENABLED(0) means ____ mode.
sudo mlxlink -d /dev/mst/mt41692_pciconf0 is ____-only.

A host has two BlueField-3 cards. lspci -d 15b3: -nn shows functions at 21:00.0, 21:00.1, 21:00.2, 98:00.0, 98:00.1, 98:00.2. mst status -v lists mt41692_pciconf0 and mt41692_pciconf1. /dev/rshim0/misc reports DEV_NAME pcie-98:00.2. Write the exact read-only sequence that (a) tells you which mst device is the card behind rshim0, (b) records firmware and mode for both cards, and (c) flags which of these commands would have required a window if you had typed them: mlxconfig s, mlxconfig q, flint q, mlxfwreset. Acceptance: your join uses the PCIe address, not the index; every command you list is a read; the classification matches the table.

Why does a warm reboot not apply an mlxconfig change on a BlueField, when it does apply a driver change on the host?

Question 3 of 3

End of Episode 2 — After the power cycle

How it ended

iDRAC power-cycles the chassis. After POST, lspci -d 15b3: lists the functions again, mst status -v prints /dev/mst/mt41692_pciconf0, and flint q shows the firmware version the operator believed he had installed.^⁠1^⁠5 Nobody quotes a replacement. The label goes in his pocket, for later. Every read-only output is saved as the night’s record, and the mode value becomes the BEFORE state for the change request to come.

What you say on the call: “On BlueField a reboot is not a reset. The new value loads when the card comes out of reset, so every firmware or mlxconfig change ends with a power cycle and a read that confirms it.”^⁠2^⁠8

By Wednesday noon the SE reports twenty-four BF-Bundle pushes, exit code zero every time, and not one SSH session to an Arm.

Lab

Read-only0 of 8 steps done

All steps are read-only; nothing here changes firmware, mode or link state.

lspci -d 15b3: -nn.
Expected: one bus address per BlueField-3 carrying its network PFs (one per port, so two on a dual-port B3220 and one on a single-port B3140H) plus the .2 RShim function; only the .2 function is fixed by the documentation, so do not grade this step on a fixed count.^⁠12^⁠7 If nothing prints, the card is not enumerated; stop and go to lesson 2.6 (aux power, PCIe init).^⁠5
sudo mst start && sudo mst status -v.
Expected: /dev/mst/mt41692_pciconf0 and the PCIe address from step 1. If mst start fails, DOCA-Host or MFT is not installed correctly; do not continue.^⁠1^⁠7
grep -E 'DEV_NAME|DEV_INFO' /dev/rshim0/misc.
Expected: DEV_NAME pcie-<bus>:00.2 matching step 1. If there is no /dev/rshim0, the rshim service is not running or the BMC owns rshim (lesson 2.4).^⁠12
sudo flint -d /dev/mst/mt41692_pciconf0 q | grep -E 'FW Version|FW Release Date|PSID'. Record both version and date; compare with 32.50.1002 for 3.5.0.^⁠4^⁠11
sudo mlxfwmanager --query.
Expected: the same firmware version as step 4; a difference means one of the tools is reading a different device.^⁠1
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE. Record ENABLED(0), DISABLED(1) or RO.^⁠2
sudo mlxlink -d /dev/mst/mt41692_pciconf0. Record link state and speed per port.^⁠1
Save all outputs as ~/bf3-readonly-$(date +%F).txt. This file is the BEFORE state required by the mode-switching lesson.^⁠6

Evidence / notes (commands run, outputs, what surprised you)

Next: Retrieval check

The MFT toolbox: read-only vs mutating

Before you read: what do you already know?

After this lesson you can

Episode 2 — The card that vanished at 02:40

1The toolbox and its one hard rule

Host does not see the BlueField

2Read-only versus mutating

3The cold power-cycle rule

End of Episode 2 — After the power cycle

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere