Skip to content
nodeza.learn
DOCA course/1. Foundations/Lesson 1.4

Modes of operation: DPU, NIC, Zero-Trust

S1·E402:10, twelve nodes with no NIC · Bridge call from home, the customer's night shift on the other end

S1·E4Analyze~30 minsources checked todayverified against DOCA 3.5.0 BlueField Modes of Operation page, BlueField BMC 26.04 Redfish mode configuration, MFT 4.34 tool list, NVIDIA/skills hardware-safety, Dell KB 000227031, 2026-09-06

Builds on: BlueField-3 under the hood

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Distinguish DPU, NIC and Zero-Trust mode by what the host can see and do, using the DOCA 3.5.0 definitions.
  • Read a mlxconfig query and decide which mode a BlueField-3 is in, including the read-only flag that signals Zero Trust.
  • Reconstruct the exact command sequence, and where each command runs, to move between any two modes without entering an unsupported state.
  • Analyze a mode-change request against the restrictions (multi-host, Socket Direct, Zero Trust plus NIC mode, reset rule) and the Dell POST caveat before approving it.

Episode 4 — 02:10, twelve nodes with no NIC

The situation · Bridge call from home, the customer's night shift on the other end

Your phone goes at 02:10 and it is the operator with the label maker. Twelve BlueField-3 servers have stopped presenting a NIC to their hosts. The change window closed two hours ago: the push set INTERNAL_CPU_OFFLOAD_ENGINE=1 across the fleet to “free up power” before the cutover, everything was warm-rebooted, and now part of the fleet is in one state and part in another. He wants to know: roll back, or press on. The network lead is on the bridge too, notebook open.

Modes exist because one piece of silicon has to honor three different contracts about who owns the card. In DPU mode the Arm cores are active and the embedded Arm system runs services that manage the NIC resources and data path; in NIC mode the Arm cores are inactive and the device functions as an NVIDIA ConnectX network adapter; Zero Trust is DPU mode plus enforcement, preventing the host system administrator from accessing BlueField from the host side[1]. An operator who sells two tenants per node lives on the third contract. That is what this fleet was running until two hours ago.

Two details decide the call. A mode change takes effect only after the Arm and NIC components reset, and NVIDIA recommends a power cycle — a warm reboot is not that[1]. And Zero Trust cannot be moved straight to NIC mode: you return to DPU mode first[1]. A mode change is a hardware event; it is not done until the power has actually gone away. You ask for one mlxconfig query before anyone types anything else.

1One card, three operating contracts

The DOCA 3.5.0 modes page defines the three modes by what the Arm side does. DPU mode: “The Arm cores of BlueField are active, and the embedded Arm system runs services that manage the NIC resources and data path”.[1] Zero Trust (Restricted) mode: “The Arm cores of BlueField are active, and the embedded Arm system runs services to manage the NIC resources and data path while enforcing restrictions on the external host (host isolation)”.[1] NIC mode: “The Arm cores of BlueField are inactive, and the device functions as an NVIDIA ConnectX network adapter”.[1]

Read them as contracts with the host. In DPU mode the host gets network functions but not control: the NIC resources and the data path are managed by the Arm system, so the host driver is a tenant of a switch it does not own. In NIC mode “BlueField operates as a ConnectX network adapter for the external host” — the host owns everything, and on BlueField-3 that “reduces power consumption, improves network performance, and minimizes the host memory footprint”.[1] Zero Trust is DPU mode with the host’s remaining privileges removed one by one (segment 4).

One generational detail changes the power story: “For BlueField-3, the Arm cores are inactive, while for BlueField-2, the Arm cores are active but non-functional” in NIC mode.[1] A BlueField-2 in NIC mode still runs its Arm complex idle; a BlueField-3 powers it down.

Arm active — embedded Arm owns the NIC (ECPF)
Host (x86 / Arm server) — mlx5 netdevsArm subsystem — representors + bridgeUplinks (Arm netdevs p0 / p1)PF netdevhost pf0 (mlx5)VF netdevsSR-IOV pf0vf0…SF netdevsen3f0pf0sf4 …PCIe Gen5 x16e-switch (embedded switch, switchdev)inside the NIC ASIC — flow tables programmed by the e-switch managerp0uplink 0p1uplink 1Wire — QSFP112 400G / 200Gpf0hpfpf0vf0pf0sf0OVS-DOCA bridge ovsbr1 (p0 + pf0hpf)or HBN br-hbnArm appsovs-vswitchd · DOCA Flow · tcpdump -i pf0hpf
hop 1 / 6
DPU mode · Host → wire

1. Host app transmits on the PF netdev

The host PF is an ordinary mlx5 netdev (VFs and SFs likewise). In DPU mode the host does not own the e-switch — the embedded Arm system does.

Mode configuration
sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE   # ENABLED(0) = DPU mode
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

NIC → DPU on BlueField-3 (host side). BlueField-2 additionally needs INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0.

  • "For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended."
  • Default for DPU SKUs B3240 / B3220 / B3210 / B3210E.
  • Socket Direct is not supported in DPU mode.
BlueField Modes of Operation · Virtual Switch on BlueField · OVS-DOCA
Step one packet from a host function to the wire. Switch the mode: in DPU mode the packet crosses an e-switch the Arm side owns; in NIC mode the host driver owns the same silicon; in Zero Trust the path is the DPU-mode path with host management calls refused.

2Defaults per SKU and the one query that tells you everything

Defaults follow the SKU family: BlueField-2 and BlueField-3 DPU SKUs are “shipped in DPU Mode by default”; for BlueField-3 SuperNIC SKUs the default is NIC mode.[1] In BlueField-3 terms that means B3240, B3220, B3210 and B3210E arrive in DPU mode and B3140H, B3140L, B3220L and B3210L arrive in NIC mode.[6][8] Dell’s own KB confirms the field reality: “The NVIDIA Bluefield-3 B3140H comes set in Super NIC mode”.[5]

From the host, one read-only command answers the mode question for BlueField-3 and BlueField-2 alike:[1]

sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE

The page gives the decoding: “ENABLED(0) means BlueField is running in DPU Mode”, “DISABLED(1) means BlueField is running in NIC Mode”, and “RO is marked for read only, and indicates DPU Mode with Zero Trust”.[1] The parameter name reads backwards to most people: it is the offload engine — the Arm side — that is enabled or disabled, so ENABLED means DPU mode. On a BlueField-3 the device is /dev/mst/mt41692_pciconf0, from the universal device ID 41692.[6] An abridged transcript from a DPU SKU (header lines vary with MFT version):

Device #1:
Device type:    BlueField3
Device:         /dev/mst/mt41692_pciconf0
Configurations:                              Next Boot
     INTERNAL_CPU_OFFLOAD_ENGINE             ENABLED(0)

The same answer is available without any host at all. In the external host’s UEFI, under “BlueField Internal Cpu Configuration”, the “Internal Cpu Offload Engine” field reads “Disabled – BlueField is operating in NIC Mode; Enabled – BlueField is operating in DPU Mode”; in the Arm UEFI, under Device Manager, System Configuration, BlueField Modes, the “NIC Mode” option shows NicMode or DpuMode.[1] Through the BlueField BMC, a Redfish GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/Oem/Nvidia returns "Mode": "DpuMode" or "NicMode".[2]

3Flipping DPU and NIC mode: BlueField-3, BlueField-2, and the reset rule

On BlueField-3 the switch is one parameter, and the page shows it run from either side — the external host prompt or the Arm OS prompt:[1]

# DPU -> NIC mode (BlueField-3), host or Arm OS
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1
# NIC -> DPU mode (BlueField-3), host or Arm OS
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

BlueField-2 needs four parameters moved together:[1]

# DPU -> NIC mode (BlueField-2)
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=1 INTERNAL_CPU_ESWITCH_MANAGER=1 INTERNAL_CPU_IB_VPORT0=1 INTERNAL_CPU_OFFLOAD_ENGINE=1
# NIC -> DPU mode (BlueField-2)
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0 INTERNAL_CPU_OFFLOAD_ENGINE=0

What changed in 3.x is worth saying out loud: the 3.5.0 modes page does not use INTERNAL_CPU_MODEL, INTERNAL_CPU_RSHIM or EXP_ROM_UEFI_ARM_ENABLE anywhere in the BlueField-3 mode-switch flow, so a mode-switch runbook that still sets them is out of date.[1] Those parameters are not dead on BlueField-3: the 3.5.0 SNAP-4 deployment page sets INTERNAL_CPU_MODEL=1 PF_BAR2_ENABLE=0, the Firefly guide sets it to 1 for embedded or 0 for separated mode, and the DPF production flavor sets it to 1 (lessons 5.2, 5.4 and 5.6).[12][13][14] Its presence in a storage, PTP or fleet recipe is not an error; its presence in a mode-switch procedure is.

Nothing takes effect until the silicon resets: “For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended.”[1] The BMC path says the same in its own units — after POST .../redfish/v1/Systems/Bluefield/Oem/Nvidia/Actions/Mode.Set with {"Mode": "NicMode"} or {"Mode": "DpuMode"}, “Two consecutive Arm reboots are required to apply configuration”.[2] mlxconfig itself is defined as the tool that lets you “change some of the device configurations without having to create and burn a new firmware”; it stages a next-boot value, which is why the query output has a Next Boot column.[3]

4Zero Trust: mlxprivhost, not mlxconfig

Zero Trust is applied from the card, never from the host. The page’s commands carry the bf> prompt:[1]

# DPU mode -> DPU mode with Zero Trust (run on the Arm OS)
sudo mlxprivhost -d /dev/mst/<device> r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner
# Zero Trust -> DPU mode (run on the Arm OS)
sudo mlxprivhost -d /dev/mst/<device> p

mlxprivhost “enables the user to restrict the hosts from configuring the NIC”; r restricts, p returns the host to privileged.[3][1] The flags map onto the five things the page says can be restricted individually: “Port ownership – The host cannot assign itself as the port owner”; “Hardware counters – The host is denied access to hardware counters”; “Tracer functionality – The tracer functionality is blocked”; “RShim interface – The RShim interface is disabled”; and “Firmware flash – firmware flashing from the host is restricted”.[1] Disabling RShim has a concrete consequence from lesson 1.3: the host loses /dev/rshim0/boot and /dev/rshim0/console, so BFB installation and the serial console move to the BMC.[9]

The BMC offers the same knob over Redfish: PATCH https://<bmc_ip>/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/Oem/Nvidia/HostPrivilegeConfig/Settings with {"PrivilegeMode": "Restricted"} or {"PrivilegeMode": "Privileged"}, and “Power cycle is required to apply configuration”.[2]

Two rules make Zero Trust a one-way door from the host’s perspective. Zero Trust to NIC mode is “Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode”, with the added instruction to “Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode”.[1] And “Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior”.[1]

5Restrictions that bite in a PowerEdge

Combination Status on the 3.5.0 modes page
Multi-host while in NIC mode “Multi-host is not supported when BlueField is operating in NIC Mode.”[1]
Socket Direct while in DPU mode “Socket Direct is not supported when BlueField is operating in DPU Mode.”[1]
NIC mode with Zero Trust configured “not supported and may lead to undefined behavior”[1]
Zero Trust straight to NIC mode “Not supported” — go through DPU mode with a system-level reset[1]
Any mode change without a reset Does not take effect; “Power cycle is recommended”[1]

The Socket Direct row has a second layer on BlueField-3: the aux-card-kit page states “The socket-direct functionality is currently not supported by firmware”, so on this generation the only working aux-kit mode is the Down Stream Port extension regardless of DPU or NIC mode.[7]

Query, flip, verify: DPU mode to NIC mode on a BlueField-3 DPU SKU

Context: a B3220 in a lab R760, host has MFT, BMC reachable, maintenance window agreed, rollback is the inverse command. Every step below is from the 3.5.0 modes page unless marked.[1]

  1. Pre-flight inventory (read-only). Record the OPN, firmware and current mode so the rollback target is written down before anything changes:[4] 
    sudo mst start && sudo mst status -v
sudo mlxfwmanager --query
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE
    Expected: ENABLED(0), no RO. If RO is present, stop: this is Zero Trust and the host cannot proceed (see the problem tab).
  2. Check the four safety conditions: rollback documented (yes, =0), out-of-band path (BMC or iDRAC console), time-boxed window, rehearsed on a replica.[4] Check KB 000227031 for this server’s CPLD and iDRAC levels, because the target mode’s counterpart (DPU mode) is the affected one on the way back.[5]
  3. Stage the change from the host: 
    sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1
    Expected: mlxconfig shows the new value under Next Boot and asks for confirmation. Nothing has changed yet.
  4. Reset both components: “Arm and NIC components must undergo a reset. Power cycle is recommended.” Perform a full power cycle through the BMC or iDRAC, not a warm reboot.[1][4]
  5. Verify from two places. Host: the query now reads DISABLED(1), and lspci shows the device presenting as a ConnectX-class adapter. BMC: GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/Oem/Nvidia returns "Mode": "NicMode".[1][2]
  6. Rollback if needed: s INTERNAL_CPU_OFFLOAD_ENGINE=0, power cycle, verify ENABLED(0).[1]

Episode 4 — Sunrise, one power cycle later

How it ended

The query told the story. On the untouched nodes the parameter line carried the read-only flag: Zero Trust, where the host cannot configure the card at all[1]. The nodes that did flip needed a cold power cycle, not the warm reboot they got[1]. You sequence the recovery the way the modes page requires: Zero Trust back to DPU mode with a system-level reset, then DPU mode to NIC mode, then a power cycle[1]. What you tell the night shift at breakfast: schedule mode changes with the people who can pull power. What the Dell host-image lead says over the same breakfast: the March build keeps MLNX_OFED.

Lab

Read-only. Query the mode from every path you have; change nothing.

  1. Host query and evidence capture:[1][3] 
    sudo mst start
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_MODEL INTERNAL_CPU_OFFLOAD_ENGINE
sudo mlxfwmanager --query
    Expected on the Dell-lab DPU SKU: INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0), no RO. Record the exact line, plus firmware and OPN; note INTERNAL_CPU_MODEL as informational — it is not used by the 3.5.0 BlueField-3 procedure.
  2. Cross-check with what the host sees: 
    lspci | grep -i -E 'mellanox|bluefield'
ls /dev/rshim0/ 2>/dev/null || echo "no rshim node"
    Expected in DPU mode with rshim installed: the BlueField functions and boot console misc rshim. A missing rshim node with ENABLED(0) and no RO means the host driver is not installed; a missing node with RO means Zero Trust disabled RShim.[9][1]
  3. If the card’s BMC is reachable, read the mode out of band (GET only):[2] 
    curl -k -u root:'<PASSWORD>' -X GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/Oem/Nvidia
    Expected: "Mode": "DpuMode" matching step 1. A mismatch between host and BMC readings means a staged change has not been applied by a reset; report it, do not reset.
  4. Record the server’s CPLD and iDRAC versions from iDRAC and compare with Dell KB 000227031’s affected set (CPLD 1.1.5/1.1.7, iDRAC 7.10.50.00, BlueField-3 firmware 32.40.1000). Write one sentence on whether a future mode change on this host would fall inside the KB.[5]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in five sentences, what changes for the host when a BlueField-3 moves from DPU mode to NIC mode to Zero Trust, and why the last transition cannot be undone from the host.

15 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 BlueField Modes of Operation page, BlueField BMC 26.04 Redfish mode configuration, MFT 4.34 tool list, NVIDIA/skills hardware-safety, Dell KB 000227031, 2026-09-06. Dates are when each page was fetched.

  1. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  2. BlueField BMC 26.04: Modes of Operation Configuration (Redfish) · fetched 2026-09-06
  3. MFT 4.34.1-18 LTS: General Information (tool list) · fetched 2026-09-06
  4. NVIDIA/skills: skills/doca-hardware-safety/SKILL.md · fetched 2026-09-06
  5. Dell KB 000227031: 'No Memory Found' event on BlueField-3 enabled PowerEdge during POST · fetched 2026-09-06
  6. NVIDIA BlueField-3 Networking Platform User Guide (index) · fetched 2026-09-06
  7. BlueField-3 User Guide: PCIe Auxiliary Card Kit · fetched 2026-09-06
  8. BlueField-3 User Guide: Specifications · fetched 2026-09-06
  9. BlueField Platform Software Troubleshooting Guide: SoC Management Interface (RShim) · fetched 2026-09-06
  10. NGC catalog: nvidia/doca/doca container · fetched 2026-09-06 · DOCA 3.5.0
  11. BlueField-3 User Guide: Introduction · fetched 2026-09-06
  12. SNAP-4 Service Deployment (INTERNAL_CPU_MODEL=1 PF_BAR2_ENABLE=0) · fetched 2026-09-06 · DOCA 3.5.0
  13. DOCA Firefly Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  14. doca-platform v26.4.1: dpuflavor.yaml (production nvconfig) · fetched 2026-09-06

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.