Skip to content
nodeza.learn
DOCA course/2. Bring-up/Lesson 2.5

Switching modes safely

S2·E5It is one parameter, how long can it take · Change-advisory call, Thursday afternoon, then the caged row

S2·E5Analyze~40 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 BlueField Modes of Operation, BlueField BMC 26.04 Modes Configuration, NVIDIA/skills doca-hardware-safety and doca-bf3-deployment, Dell KB 000227031, Broadcom KB 379391, 2026-09-06

Builds on: Modes of operation: DPU, NIC, Zero-Trust, The MFT toolbox: read-only vs mutating, Out-of-band: BMC, Redfish, and the BlueField-4 path

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Run a mode change as a safety procedure: pre-flight inventory, out-of-band access, window, apply, verify, rollback.
  • Issue the exact commands for DPU to NIC, NIC to DPU, DPU to Zero-Trust and back on BlueField-3, and state the reset each needs.
  • Analyse a failed or half-applied switch from its symptoms (host PFs, tmfifo, mlxconfig output) and choose the rollback.
  • Apply the four hard refusals of the hardware-safety workflow to a real change request.

Episode 5 — It is one parameter, how long can it take

The situation · Change-advisory call, Thursday afternoon, then the caged row

The change request says: flip the four storage nodes from DPU mode to NIC mode, fifteen minutes, no window required, so Friday’s throughput number has no Arm in the path. The network lead wrote it himself: he wants a counter with nothing on the card left to argue about. The change manager reads your estimate back and asks, not unreasonably, why one mlxconfig line needs a maintenance window, a rollback plan and an out-of-band path. The SE, whose spreadsheet still says fifteen minutes, studies his coffee.

Because of what that parameter does. In DPU mode the embedded Arm system runs services that manage the NIC resources and the data path; in NIC mode on BlueField-3 the Arm cores are inactive and the device functions as a ConnectX network adapter.[1] The flip does not tune the card, it removes a computer from the server, and with it the Arm-side tools you would use to undo the change. The configuration takes effect only after the Arm and NIC components reset, and a power cycle is what NVIDIA recommends.[1]

The modes exist because one piece of silicon serves two customers: one that wants infrastructure services offloaded onto the card, one that wants a fast NIC and nothing else. Zero Trust is the third position, where the host administrator is deliberately locked out of the card in his own server.[1]

A mode change is not a setting; it is a hardware change you make with a keyboard.

You ask for one thing before agreeing to anything: their BEFORE state.[3] Segment 1 recaps the knob.

1Three modes, one knob, one privilege

Lesson 1.4 defined the modes; this lesson is about changing them without losing the server. Quick recap from the 3.5.0 modes page: in DPU mode “the embedded Arm system runs services that manage the NIC resources and data path”; Zero Trust is “a specialized variation of DPU Mode” that stops the host administrator from reaching the card; in NIC mode on BlueField-3 “the Arm cores of BlueField are inactive, and the device functions as an NVIDIA ConnectX network adapter”.[1] DPU SKUs ship in DPU mode, SuperNIC SKUs in NIC mode.[1]

On BlueField-3 the mode is one firmware parameter. sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE prints ENABLED(0) for DPU mode and DISABLED(1) for NIC mode; an RO flag on the line means Zero Trust has made it read-only from the host.[1] Zero Trust is a separate privilege, set from the Arm with mlxprivhost, whose official purpose is to “restrict the hosts from configuring the NIC”; it can individually deny port ownership, hardware counters, the tracer, the RShim interface and firmware flashing from the host.[1][9]

Three restrictions shape every plan: “Multi-host is not supported when BlueField is operating in NIC Mode”; “Socket Direct is not supported when BlueField is operating in DPU Mode”; and “Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior”.[1] The last one gives the order rule: “Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode”, with “a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode”.[1]

Arm active — embedded Arm owns the NIC (ECPF)
Host (x86 / Arm server) — mlx5 netdevsArm subsystem — representors + bridgeUplinks (Arm netdevs p0 / p1)PF netdevhost pf0 (mlx5)VF netdevsSR-IOV pf0vf0…SF netdevsen3f0pf0sf4 …PCIe Gen5 x16e-switch (embedded switch, switchdev)inside the NIC ASIC — flow tables programmed by the e-switch managerp0uplink 0p1uplink 1Wire — QSFP112 400G / 200Gpf0hpfpf0vf0pf0sf0OVS-DOCA bridge ovsbr1 (p0 + pf0hpf)or HBN br-hbnArm appsovs-vswitchd · DOCA Flow · tcpdump -i pf0hpf
hop 1 / 4
DPU mode · Host → Arm

1. Host sends on the PF netdev

Same start as any transmit. Nothing on the host says "this goes to the Arm".

Mode configuration
sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE   # ENABLED(0) = DPU mode
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

NIC → DPU on BlueField-3 (host side). BlueField-2 additionally needs INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0.

  • "For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended."
  • Default for DPU SKUs B3240 / B3220 / B3210 / B3210E.
  • Socket Direct is not supported in DPU mode.
BlueField Modes of Operation · Virtual Switch on BlueField · OVS-DOCA
Switch the mode and step a packet from a host PF. Watch what disappears in NIC mode: the Arm, the representors, and the tmfifo path you use to manage the card.

2Pre-flight: the BEFORE state and the out-of-band path

The hardware-safety skill’s workflow is “pre-flight inventory → out-of-band path → window → apply → verify → rollback”, and it refuses to proceed in four cases: no documented rollback, a link-breaking change with no out-of-band access, no explicit time-boxed maintenance window, and a change that was not rehearsed on a representative non-production replica.[2] A mode flip is on its list of operations that trigger the full workflow.[2]

The BEFORE state is a specific set of reads, run from the host before anything changes:[3]

lspci -d 15b3: -nn                          > before-lspci.txt
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q > before-mlxconfig.txt   # query only
sudo flint -d /dev/mst/mt41692_pciconf0 q     > before-flint.txt
cat /dev/rshim*/misc                          > before-rshim.txt
dmesg                                         > before-dmesg.txt

The skill’s phrasing: “without a BEFORE state, ‘rollback’ is a phrase, not a thing the operator can do”.[3] Add the Arm release (ssh ubuntu@192.168.100.2 cat /etc/mlnx-release) while the Arm is still reachable.[3]

Out-of-band means a path to the server that does not depend on the card. On a PowerEdge that is the iDRAC virtual console for the host and the power-cycle control, plus the card’s BMC (Redfish) or the rshim console for the Arm side.[5][11] Going to NIC mode removes the Arm, so tmfifo SSH and Arm-side mlxprivhost disappear; the way back is host mlxconfig or BMC Redfish.[1][5] Going to Zero Trust removes the host’s ability to change firmware configuration, so the way back is the Arm or the BMC.[1] Confirm the path you will need afterwards works before you start.

3Apply: exact commands for each transition

All commands verbatim from the 3.5.0 modes page; host side unless marked.[1]

# DPU -> NIC (BlueField-3), host
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1
# NIC -> DPU (BlueField-3), host
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

# DPU -> NIC (BlueField-2), host: four parameters
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=1 INTERNAL_CPU_ESWITCH_MANAGER=1 INTERNAL_CPU_IB_VPORT0=1 INTERNAL_CPU_OFFLOAD_ENGINE=1
# NIC -> DPU (BlueField-2), host
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0 INTERNAL_CPU_OFFLOAD_ENGINE=0

# DPU -> Zero Trust, run on the ARM OS
sudo mlxprivhost -d /dev/mst/<device> r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner
# Zero Trust -> DPU, run on the ARM OS
sudo mlxprivhost -d /dev/mst/<device> p

Then the reset: “For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended.”[1] The safety skill’s “mlxconfig cold-power-cycle rule” treats a warm reboot as insufficient; Dell’s KB 000300192 documents the fatal PCIe error you get on a PowerEdge when a DPU firmware change is followed by a warm reboot instead of a power cycle.[2][10]

Two alternatives to the host path. From the BMC: POST /redfish/v1/Systems/Bluefield/Oem/Nvidia/Actions/Mode.Set with NicMode or DpuMode (“Two consecutive Arm reboots are required”), and PATCH .../HostPrivilegeConfig/Settings with PrivilegeMode Restricted or Privileged (“Power cycle is required”).[5] From UEFI, the menu depends on which side you are on. External host UEFI: select BlueField Internal Cpu Configuration and set Internal Cpu Offload Engine to Disabled for NIC mode or Enabled for DPU mode. Arm UEFI (press Esc twice on the console): Device Manager -> System Configuration -> BlueField Modes, set the NIC Mode option to NicMode or DpuMode, exit and save, then “Issue power cycle to apply new configuration”.[1]

VMware adds a step. Broadcom KB 379391 states that “When you switch BlueField-3 from NIC mode to VMware Distributed Services Engine enabled DPU mode, you must re-install ESXi”, in both directions, on vSphere 8.0 U3b and later; the card leaves the factory in DPU mode with DSE disabled.[7] NVIDIA’s own ESXi procedure is /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=1, then “Power cycle the system after switching to NIC mode”, then verify with mlxconfig -d mt41692_pciconf0 q | grep -i offload showing DISABLED.[8]

SymptomBlueField-3 on a Dell Pow…SymptomBring-up: card, rshim, BF…SymptomDPU stuck / not back afte…CheckWhat mode does the firmw…
Symptom

DPU stuck / not back after a mode change (DPU ↔ NIC ↔ Zero-Trust)

Start at the mode-switch node: the card came back different from what you expected. Each branch is a read from the verification set.

4Verify, then roll back if you must

Verification is the same set of reads as the BEFORE state, compared line by line. The decisive one: sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE must show the value you intended, ENABLED(0) for DPU, DISABLED(1) for NIC, and no RO unless you meant Zero Trust.[1] Then lspci -d 15b3: against ip link to confirm the host PFs are bound and up; then, in DPU mode, cat /etc/mlnx-release on the Arm after ip route get 192.168.100.2 reports dev tmfifo_net0 or dev tm-br.[3][4] In NIC mode the Arm checks are skipped by design; the host and BMC reads are the whole verification.[1]

Rollback is the inverse write plus the same power cycle: mlxconfig s INTERNAL_CPU_OFFLOAD_ENGINE=0 returns a BlueField-3 to DPU mode; mlxprivhost -d <device> p from the Arm, or the Redfish Privileged PATCH from the BMC, undoes Zero Trust.[1][5] The BEFORE files tell you when you are back.

One Dell-specific symptom belongs in the plan. On PowerEdge R660, R760, R760XA and XE9680 with CPLD 1.1.5 or 1.1.7, iDRAC 7.10.50.00 and BlueField-3 firmware 32.40.1000, a card in DPU mode can make POST show “No Memory Found”; the server waits about a minute and reboots on its own, after which memory is seen normally. The B3140H “comes set in Super NIC mode. If it is changed to DPU Mode, it can also encounter this event.”[6] If you switch a card to DPU mode on that combination, budget the extra reboot and do not treat it as a failed change.

DPU to NIC on the Dell-lab R760, with rollback

Card: BlueField-3 B3220 at /dev/mst/mt41692_pciconf0, currently ENABLED(0). Host management is on the LOM through iDRAC; the card’s ports carry only test traffic. Window: 30 minutes, agreed. Replica: the same flip was run last week on the second lab R760.

  1. BEFORE state: the five reads from Segment 2 into before-*.txt, plus ssh ubuntu@192.168.100.2 cat /etc/mlnx-release.[3]
  2. OOB check: open the iDRAC virtual console and confirm you can issue a power cycle from it; open screen /dev/rshim0/console 115200 and confirm you see the Arm login prompt.[11]
  3. Refusal check: rollback documented (step 8), OOB works (step 2), window agreed, replica done. Proceed.[2]
  4. Apply: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1. Answer the confirmation prompt.[1]
  5. Power cycle from iDRAC. Not reboot.[1][2]
  6. Verify: sudo mst start; sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE shows DISABLED(1); lspci -d 15b3: -nn still lists the functions; ip link shows the two PFs.[1][3]
  7. Expected losses: no tmfifo SSH, no Arm console login. Record them as expected, not as faults.[1]
  8. Rollback (if step 6 fails or the customer changes their mind): sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=0, power cycle from iDRAC, re-run step 1’s reads and diff against before-*.txt.[1]

End of Episode 5 — Thirty minutes, and the way back

How it ended

The window is agreed at thirty minutes. The BEFORE files are captured, iDRAC proves it can power-cycle the chassis over a path that does not ride the card’s ports, and the flip goes: write, cold power cycle, then mlxconfig q INTERNAL_CPU_OFFLOAD_ENGINE reading DISABLED(1) on all four.[1][3] One node takes the extra automatic reboot the Dell KB describes, and nobody panics, because it was in the plan.[6]

What you say to the change manager: “The command takes a second. The window is for the power cycle, the verification and the way back.”[2]

At 06:00 the operator calls from the far end of the row: three cards, three different symptoms, one replacement request covering all three, three hours before the demo.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Do this only on the Dell-lab R760 during a booked window, with the iDRAC virtual console open. Every mutating step names its rollback.

  1. Pre-flight (read-only): the five reads into before-*.txt and ssh ubuntu@192.168.100.2 cat /etc/mlnx-release.[3] Confirm the card is ENABLED(0); if it shows RO, stop and plan a Zero Trust exit first.[1]
  2. OOB check (read-only): power-cycle control visible in iDRAC; screen /dev/rshim0/console 115200 shows the Arm prompt. If either fails, refuse the change.[2]
  3. Apply: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1. Rollback: s INTERNAL_CPU_OFFLOAD_ENGINE=0 before the power cycle leaves the card as it was, since nothing applies until the reset.[1]
  4. Power cycle from iDRAC (not reboot). Rollback: if the host does not POST cleanly within a few minutes, check for the “No Memory Found” auto-reboot described in Segment 4 before acting; after POST, run step 3’s rollback and power-cycle again.[1][6]
  5. Verify (read-only): sudo mst start; sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE reads DISABLED(1); lspci -d 15b3: -nn and ip link show the PFs; note that ssh 192.168.100.2 fails by design.[1][3]
  6. Diff: diff before-lspci.txt <(lspci -d 15b3: -nn) and diff before-flint.txt <(sudo flint -d /dev/mst/mt41692_pciconf0 q). Expected: no firmware change, same functions. Anything else is a finding.[3]
  7. Return the lab card to DPU mode: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=0, power cycle from iDRAC. Rollback of the rollback: repeat step 3 and 4. Verify ENABLED(0), then ip route get 192.168.100.2 shows dev tmfifo_net0 or dev tm-br and ssh ubuntu@192.168.100.2 cat /etc/mlnx-release matches your pre-flight value.[1][4]
  8. Save before-*.txt, the after reads and the timestamps of both power cycles in the lab notes.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in five sentences, how you would switch a customer's BlueField-3 from DPU mode to NIC mode without risking a truck roll.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 BlueField Modes of Operation, BlueField BMC 26.04 Modes Configuration, NVIDIA/skills doca-hardware-safety and doca-bf3-deployment, Dell KB 000227031, Broadcom KB 379391, 2026-09-06. Dates are when each page was fetched.

  1. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  2. NVIDIA/skills: doca-hardware-safety SKILL (workflow, hard refusals, cold power-cycle rule) · fetched 2026-09-06
  3. NVIDIA/skills: doca-bf3-deployment TASKS (pre-flight capture, verification) · fetched 2026-09-06
  4. NVIDIA/skills: doca-bf3-deployment CAPABILITIES (tmfifo route check) · fetched 2026-09-06
  5. BlueField BMC 26.04: Modes of Operation Configuration (Redfish) · fetched 2026-09-06
  6. Dell KB 000227031: 'No Memory Found' event on BlueField-3 enabled PowerEdge during POST · fetched 2026-09-06
  7. Broadcom KB 379391: Configuring BlueField-3 into DPU mode for vSphere DSE or NIC mode · fetched 2026-09-06
  8. HowTo Configure NVIDIA BlueField-3 to NIC Mode on VMware vSphere 8.0 · fetched 2026-09-06
  9. MFT 4.34.1-18 LTS: General Information (mlxprivhost) · fetched 2026-09-06
  10. Dell KB 000300192: XE9680L, a power cycle is required after updating DPU firmware · fetched 2026-09-06
  11. BlueField Platform Software Troubleshooting Guide: SoC Management Interface (console, SW_RESET) · fetched 2026-09-06

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.