Skip to content
nodeza.learn
DOCA course/9. FAE capstone/Lesson 9.2

Scenario: escalations

S9·E2The card that answers rshim and nothing else · Bridge call, 01:40, the customer's cage two time zones away, day nine

S9·E2Evaluate~50 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs, BlueField troubleshooting guide, Dell KBs 000379421/000227031/000300192, NVIDIA forum threads 312378/271327/304754/380253/348988/373806, 2026-09-06

Builds on: rshim and the BFB path, Dell-specific bring-up: iDRAC, BIOS, CPLD, aux power, DSE, DOCA Ethernet and OVS-DOCA

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Triage the six recurring BlueField-3 field failures with an ask-run-do-not-first runbook each.
  • Judge from the first evidence bundle whether a case is configuration, firmware alignment, power, ownership, or a support-entitled hardware recovery.
  • Decide when to stop local debugging and open an NVIDIA Enterprise Support case or a Dell service request, and say why in writing.
  • Separate a documented product limit from a bug, and verify the claim against the current DOCA page before repeating it.

Episode 2 — The card that answers rshim and nothing else

The situation · Bridge call, 01:40, the customer's cage two time zones away, day nine

You join sixteen minutes after the page. On the call: the customer’s network lead, who has started a notebook for everything about this pod he cannot explain; the Dell SE, still awake, still logging promises; and the night-shift operator, the only person with hands on the rack and a label maker in her pocket.

The BlueField-3 came back from a maintenance window with mlx5_core reporting firmware over 120000 MS in pre-initializing state, mlx5_init_one failing with error code -110, and an empty ibv_devices, while rshim attached normally.[7] Someone has already downloaded a BFB and wants permission to reflash.

Your first job is to stop that. A reflash needs a power cycle nobody has a window for, and it erases the state that identifies the fault; without a before-state, rollback is a phrase and not a plan.[17][18] The duty at this hour is issue replication, and replication starts with evidence.[22]

RShim exists because this card has its own computer inside it. When the host sees nothing useful you still need to watch the Arm boot and talk to it out of band, so the interface rides a PCIe management function and hands you a console and a misc file, single-owner by design, one backend at a time.[10] That console is why this call can end tonight.

The network lead says the thing he says once a week: show me the counter. The evidence you destroy at 01:40 is the evidence you need at 09:00. You ask the operator to open the console and read the last ten lines aloud.

1Triage discipline: ask, run, do not

An escalation reaches you as a paragraph and a screenshot. The FAE job description names issue replication as a core duty, and replication starts with evidence, not with a fix.[22] Every runbook in this lesson has three columns: what to ask, what to run, and what not to do first.

What to ask is the same list every time: server model and riser; SKU and whether it is a Dell catalogue part; mode (mlxconfig q INTERNAL_CPU_OFFLOAD_ENGINE); BlueField firmware; BFB and DOCA-Host versions; BIOS, iDRAC and CPLD levels; what changed last.[15][2] What to run is read-only: lspci -d 15b3: -nn, mlxconfig -d <dev> q, flint -d <dev> q or mlxfwmanager --query, cat /dev/rshim0/misc, a console capture with minicom -D /dev/rshim0/console -C rshim_console.txt, and dmesg filtered for mlx5_core.[19][20][10] Do not trust a single tool’s version string: the four-way match is pkg-config --modversion doca-common, /opt/mellanox/doca/applications/VERSION, /opt/mellanox/doca/tools/doca_caps --version on the host, and cat /etc/mlnx-release plus bfver on the Arm; disagreement means a partial install.[23]

What not to do first is the column that saves cards. No firmware burn, no BFB reflash, no mode flip and no RMA until the evidence bundle exists, because each of those destroys the state you need and each requires a power cycle you may not have a window for.[17][3] The hardware-safety rule is explicit: without a before-state, rollback is a phrase, not a plan.[18]

Two boundaries end local debugging. A BlueField-3 looping at PSC BL1 START with a stalled BFB transfer, flint reporting DRAM not available and the BMC golden-image script aborting on “PCIE is not in reset” got one answer from NVIDIA: open an Enterprise Support case, handled according to entitlement.[9] A card still undetectable after a full power cycle and a firmware update gets Dell’s answer: open a service request for a replacement.[1]

SymptomBlueField-3 on a Dell Pow…SymptomBring-up: card, rshim, BF…SymptomHost does not see the Blu…CheckIs the card enumerated o…
Symptom

Host does not see the BlueField

Start at the symptom the customer reported and follow the branches; each leaf names the source it rests on.

2Card not detected: PCIe and power

Pattern 1: DPU not detected after a BIOS update. Dell’s KB 000379421 describes the Lifecycle Log repeating “Device not detected: Nvidia Network Adapter” (PR8): a PCIe training failure caused by BlueField-3 firmware below 32.46.3048, released 14 August 2025.[1] KB 000227031 covers the neighbouring symptom, “No Memory Found” at POST after BIOS updates, then an automatic reboot after about a minute, caused by unsynchronised state machines between BlueField-3, iDRAC and CPLD at CPLD 1.1.5/1.1.7, iDRAC 7.10.50.00 and firmware 32.40.1000, in DPU mode.[2] The community version, a B3220SH absent from lspci and shown as N/A by the BIOS across every slot on a consumer board, stayed unresolved for five months; the advice was slot incompatibility.[4]

  • Ask: exact firmware; BIOS, iDRAC, CPLD levels; which slot and riser; whether a power cycle, not a reboot, has happened since the change.[1][3]
  • Run: read the LC log for PR8; lspci -d 15b3:; if the card appears, mlxfwmanager --query for the version.[1][20]
  • Do not first: reseat or RMA; flash firmware on a card the host cannot see; loop warm reboots. The KB order is power cycle, then firmware 32.46.3048 or later with a matching BFB, then replacement.[1]

Pattern 6: the auxiliary-power halt. A B3210 in DPU mode on Ubuntu 24.04 with DOCA-OFED 3.4.1 logged mlx5_core “Firmware over 120000 MS in pre-initializing state, aborting”, mlx5_init_one failed with error code -110, and an empty ibv_devices, while rshim attached normally.[7] The console showed the bootloader halting: “ATX missing - initiating halt sequence” and “CRITICAL ERROR: ATX power not detected! Halting system!!”; the card draws up to 150 W and the slot supplies roughly 66 W.[7] NVIDIA’s specifications page requires the supplementary 8-pin ATX connector on the four DPU SKUs and 75 W minimum through the slot for every SKU; SuperNICs have no connector; the HGX B300 RA states the same requirement for the B3240.[16][21]

  • Ask: is an 8-pin cable fitted, and is it a PCIe (GPU-style) cable? The thread is explicit: not a CPU/EPS12V cable, the pinout differs and can damage the card.[7]
  • Run: cat /dev/rshim0/misc, then the console; the halt message is unambiguous.[7][10]
  • Do not first: swap BFBs, reinstall DOCA-Host, or blame the kernel. The fix needed three things in order: power, a BFB install, and only then the mode the customer wanted.[7]

3RShim and tmfifo

Pattern 4: rshim missing. The card is in lspci, including the SoC Management Interface function, rshim.service is active, but there is no /dev/rshim* and tmfifo_net0 reports “Device not found”.[5] The answer was ownership: cards with an onboard BMC own the RShim by default; disable RShim on the BMC side so the host can claim it.[5] The troubleshooting guide generalises it: RShim is single-owner, the error “Another backend already attached” means someone else holds it, and the transfer is systemctl stop rshim; systemctl disable rshim on the current owner and systemctl enable rshim; systemctl start rshim on the desired one.[10] A different message, “failed to setup CUSE rshim”, means cuse.ko is missing: install kernel-modules-extra, modprobe cuse, restart the service.[10]

  • Ask: does the card have a BMC (every BlueField-3 DPU and SuperNIC does); is anything attached to the BMC’s USB; what does the rshim log say exactly.[16][10]
  • Run: journalctl -u rshim, ls -la /dev/ | grep rshim, and on a multi-card host cat /dev/rshim<N>/misc | grep DEV_NAME to map each device to its PCIe address before touching anything.[10][19]
  • Do not first: reinstall the rshim package in a loop, or reflash. Ownership does not change by reinstalling.[10]

Pattern 5: tmfifo not coming up. If tmfifo_net0 is absent, go back to Pattern 4. If it exists on both sides but nothing answers, check the addressing: the guide uses host 192.168.100.1 and Arm 192.168.100.2 on a /30, and some DOCA-Host installs bridge tmfifo_net0 into tm-br.[11][10][19] A ping 192.168.100.2 that works can be answered locally; run ip route get 192.168.100.2 and accept only dev tmfifo_net0 or dev tm-br.[18][19] Raise the log level with echo 'DISPLAY_LEVEL 2' > /dev/rshim0/misc and look for INFO[MISC]: Linux up and INFO[MISC]: DPU is ready; login is not possible before that marker even if a prompt appears.[10][11]

  • Do not first: assign new addresses on both sides at once, or SW_RESET a DPU that is still installing (the installer marker sequence tells you).[11]

4bfb-install says done; the DPU says nothing

Pattern 2. After bfb-install --bfb <image> --rshim rshim0 the host could not ssh ubuntu@192.168.100.2, although tmfifo_net0 existed on both sides with the /30 addresses.[6] The console (screen /dev/rshim0/console 115200) printed nothing until echo "SW_RESET 1" > /dev/rshim0/misc, after which BL2, BL31 and UEFI boot logs appeared.[6] mlxfwmanager --query showed B3220 firmware 32.42.1000 while bfvcheck on the Arm recommended 32.37.1306: the firmware and the BFB were not aligned.[6] The suggested fixes were the host-side tmfifo interface configuration and DHCP versus static on the DPU out-of-band interface.[6]

The deployment skill states the rule bluntly: do not trust bfb-install exit code 0 alone; require the console markers, the tmfifo route, SSH, then cat /etc/mlnx-release and bfver.[18] It also classifies a not-yet-healthy card into six states: installer still running, UEFI only, Linux up but tmfifo down, tmfifo up but SSH down, Arm fine but host PFs unbound, host and BlueField version mismatch.[18][19] Each state has a different owner.

  • Ask: the exact BFB file name and the firmware it carries; whether INFO[MISC]: DPU is ready was ever seen; what cat /dev/rshim0/misc shows now; whether a bf.cfg set ubuntu_PASSWORD or a network hook.[11]
  • Run, in this order: console; SW_RESET 1 only if the console is silent; mlxfwmanager --query against the BFB’s expected firmware; ip route get 192.168.100.2; then the DPU side over the console.[6][10][18]
  • Do not first: re-run bfb-install repeatedly, change both tmfifo addresses, or push a different BFB before reading the console. Dell’s KB adds the alignment rule from the other direction: keep the BFB aligned with the updated firmware.[1]

5Offload went to software after an upgrade

Pattern 3. A DPU running OVS-DOCA lost hardware offload entirely after the operator added an OVS mirror (P1 to pf1hpf for Flow Inspector).[8] NVIDIA’s answer called it expected: “the current OVS-DOCA offload provider cannot offload multi-output flows. This is a limitation of OVS-DOCA, not BF2/BF3 hardware”, and pointed to tc-flower based mirroring under OVS-kernel hardware acceleration instead.[8]

Hold that answer next to the current page before you repeat it. The DOCA 3.5.0 OVS-DOCA page lists a mirror feature (ovs-vsctl create mirror, 128 mirror configurations by default, raised with other_config:doca-mirror-max — the page’s own example sets 2048) among offloaded capabilities, and separately names ct-ct-nat as unsupported.[12] So the honest statement is: “for your DOCA version, the page says X”; the forum answer is evidence of how NVIDIA classifies such cases (software provider limit, not silicon), not a permanent fact.

The verification command is ovs-appctl dpctl/offload-stats-show; ovs-appctl coverage/show exposes counters such as doca_async_queue_full, doca_async_add_failed and doca_pipe_resize.[12] A search snippet suggests flows shown with dp:ovs were handled in software, but that indicator was not found on a fetched page, so use the offload-stats output as the evidence.[12]

After an upgrade, two 3.5.0 changes are the first places to look when offload behaviour shifts: OVS-DOCA gained IPv6 connection tracking and B3220SH support, and the driver replaced the MR cache with FRMR pools.[13] Then check lock-step: OVS-DOCA is a DOCA service, therefore Level 2 against firmware and BFB, and a July service with an October firmware is outside the guaranteed window.[14]

  • Ask: what changed (mirror, CT knob, DOCA-Host or BFB version); which DOCA version on the Arm; whether the bridge is datapath_type=doca and hw-offload=true.[12]
  • Run: ovs-appctl dpctl/offload-stats-show, ovs-appctl coverage/show, and ovs-appctl vlog/set at debug for the offload module if needed.[12]
  • Do not first: file a hardware bug, downgrade firmware alone, or remove OVS-DOCA. Multi-output, ct-ct-nat and Geneve option limits are documented boundaries first.[12][8]
Escalation: DPU disappears after BIOS update · decision 1/6A 0 · P 0 · S 0

Brief — Customer's R760 with a BlueField-3 B3220 (DPU mode) lost the DPU after a BIOS update. Host sees nothing. They are about to reflash the DPU firmware "to be safe".

Dell L2 support engineer, bridge call with the customer: We are about to reflash the DPU firmware from the host — go ahead?

Run the escalation scenario: choose what to ask, what to run and what to refuse; the score penalises destructive first moves.

6A full triage, worked, faded, and a new case

Triage of the ATX-missing case: worked, faded, problem

Report received: B3210 (DPU mode), firmware 32.41.0084, Ubuntu 24.04, DOCA-OFED 3.4.1. dmesg: mlx5_core ... Firmware over 120000 MS in pre-initializing state, aborting; mlx5_init_one failed with error code -110; ibv_devices empty; /dev/rshim0 present.[7]

  1. Classify the evidence. rshim present means PCIe enumeration and the SoC management function work; the NIC function never reported firmware ready. That excludes DOCA-Host, kernel and BFB as first causes.[7][10]
  2. Ask for the console: minicom -D /dev/rshim0/console -C rshim_console.txt, then echo "SW_RESET 1" > /dev/rshim0/misc to re-run the bootloader while capturing.[10]
  3. Read the halt: “ATX missing - initiating halt sequence” and “CRITICAL ERROR: ATX power not detected! Halting system!!”.[7]
  4. Explain the physics in one sentence: the card draws up to 150 W and the slot supplies roughly 66 W; the specifications page requires the 8-pin connector on DPU SKUs.[7][16]
  5. Prescribe the cable: PCIe GPU-style 8-pin; never a CPU/EPS12V cable, whose pinout differs and can damage the card.[7]
  6. Sequence the recovery: fit power, power-cycle, confirm INFO[MISC]: DPU is ready, then install the BFB the customer wants, then apply the mode change they asked for with its own power cycle.[7][11][15]
  7. Record firmware alignment afterwards with mlxfwmanager --query against the BFB, because the reporter’s version was old.[20][1]
  8. Do not: swap BFBs, reinstall DOCA-Host, or RMA. None addresses a missing cable.[7]

The last ten lines

How it ended

The console has printed the same halt for hours: ATX missing, initiating halt sequence, then ATX power not detected, halting system.[7] The card draws up to 150 W and the slot supplies roughly 66 W, so the supplementary 8-pin connector is not optional, and it must be a PCIe cable, not an EPS12V one, because the pinout differs and can damage the card.[7][16] The operator finds it coiled behind the riser, fits it, and labels the free end “PCIE, NOT EPS”. No reflash, no replacement request, no second bridge call.

What you say: “Fit the PCIe 8-pin, cold power cycle, and send me the console capture before anyone touches firmware.”[3]

Day twenty-two is the architecture review, where the same network lead will ask why this pod needs DPUs.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: a five-minute narrated demo for a Dell audience of a mode flip and an OVS-DOCA offload check on the lab BlueField-3, narrated as the triage checks you would run at each step. Mode changes require a power cycle; work inside a written maintenance window with rollback.[15][17]

  1. Pre-flight evidence bundle, on camera: lspci -d 15b3: -nn; sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q; sudo flint -d /dev/mst/mt41692_pciconf0 q; cat /dev/rshim0/misc (expect INFO[MISC]: DPU is ready); journalctl -u rshim -n 20.[10][20][11] If any command fails: stop; you now have a real escalation, not a demo.
  2. Console proof (read-only): sudo minicom -D /dev/rshim0/console -C rshim_console.txt, show a login prompt, exit. Explain to the audience that this capture is what you ask for first in Patterns 2 and 6.[6][7]
  3. Offload check in DPU mode on the Arm: sudo ovs-appctl dpctl/offload-stats-show and sudo ovs-appctl coverage/show; point out the counters you would compare before and after a customer’s change.[12] If no OVS-DOCA bridge exists: say so and skip; do not configure OVS inside a mode-flip window.
  4. Flip to NIC mode (mutating): sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1, full power cycle. Rollback is the inverse write plus a power cycle.[15][3]
  5. Verify DISABLED(1), show that the rshim console is now silent because the Arm is inactive, and explain why this silence is expected in NIC mode but a fault in DPU mode.[15][6]
  6. Roll back to DPU mode, power cycle, wait for INFO[MISC]: DPU is ready, re-run step 3 and show the offload statistics return.[15][11][12] If the marker does not return within the installer’s normal window: capture the console and follow Pattern 2.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell support engineer, in four sentences, why you ask for the rshim console capture before you let anyone reflash a BlueField-3 that will not come up.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs, BlueField troubleshooting guide, Dell KBs 000379421/000227031/000300192, NVIDIA forum threads 312378/271327/304754/380253/348988/373806, 2026-09-06. Dates are when each page was fetched.

  1. Dell KB 000379421: BlueField-3 DPU PCIe Initialization Failure · fetched 2026-09-06
  2. Dell KB 000227031: No Memory Found event on NVIDIA BlueField-3 enabled PowerEdge servers · fetched 2026-09-06
  3. Dell KB 000300192: XE9680L power cycle required after updating DPU firmware · fetched 2026-09-06
  4. NVIDIA Developer Forums: BlueField-3 DPU not found in the server · fetched 2026-09-06
  5. NVIDIA Developer Forums: BlueField visible in lspci, rshim running, but no /dev/rshim and no tmfifo_net0 · fetched 2026-09-06
  6. NVIDIA Developer Forums: BF3 RShim not working (bfb-install done, no ssh to 192.168.100.2) · fetched 2026-09-06
  7. NVIDIA Developer Forums: BlueField-3 (DPU mode) stuck in FW pre-initializing, no host netdevs · fetched 2026-09-06
  8. NVIDIA Developer Forums: Hardware offload lost when applying mirror in DOCA · fetched 2026-09-06
  9. NVIDIA Developer Forums: BlueField-3 stuck at PSC BL1 START, RShim BFB transfer stalls · fetched 2026-09-06
  10. BlueField Platform Software Troubleshooting Guide: SoC Management Interface (RShim) · fetched 2026-09-06
  11. BF-Bundle Installation and Upgrade · fetched 2026-09-06 · DOCA 3.5.0
  12. OVS-DOCA Hardware Acceleration · fetched 2026-09-06 · DOCA 3.5.0
  13. DOCA 3.5.0 Changes and New Features · fetched 2026-09-06 · DOCA 3.5.0
  14. DOCA Dependency Compatibility Policy · fetched 2026-09-06 · DOCA 3.5.0
  15. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  16. NVIDIA BlueField-3 Networking Platform User Guide: Specifications · fetched 2026-09-06
  17. NVIDIA/skills: doca-hardware-safety · fetched 2026-09-06
  18. NVIDIA/skills: doca-bf3-deployment TASKS · fetched 2026-09-06
  19. NVIDIA/skills: doca-bf3-deployment CAPABILITIES · fetched 2026-09-06
  20. MFT 4.34.1-18 LTS: General Information (tool list) · fetched 2026-09-06
  21. NVIDIA HGX AI Factory Enterprise Reference Architecture: Components · fetched 2026-09-06
  22. NVIDIA: Senior Field Application Engineer, Networking System Hardware Design (posting) · fetched 2026-09-06
  23. NVIDIA/skills: doca-version CAPABILITIES · fetched 2026-09-06
  24. NGC catalog: nvidia/doca/doca container · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.