Scenario: escalations
S9·E2 The card that answers rshim and nothing else · Bridge call, 01:40, the customer's cage two time zones away, day nine
Builds on: rshim and the BFB path, Dell-specific bring-up: iDRAC, BIOS, CPLD, aux power, DSE, DOCA Ethernet and OVS-DOCA
Before you read: what do you already know?
Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Triage the six recurring BlueField-3 field failures with an ask-run-do-not-first runbook each.
- Judge from the first evidence bundle whether a case is configuration, firmware alignment, power, ownership, or a support-entitled hardware recovery.
- Decide when to stop local debugging and open an NVIDIA Enterprise Support case or a Dell service request, and say why in writing.
- Separate a documented product limit from a bug, and verify the claim against the current DOCA page before repeating it.
Episode 2 — The card that answers rshim and nothing else
You join sixteen minutes after the page. On the call: the customer’s network lead, who has started a notebook for everything about this pod he cannot explain; the Dell SE, still awake, still logging promises; and the night-shift operator, the only person with hands on the rack and a label maker in her pocket.
The BlueField-3 came back from a maintenance window with mlx5_core reporting firmware over 120000 MS in pre-initializing state, mlx5_init_one failing with error code -110, and an empty ibv_devices, while rshim attached normally.7 Someone has already downloaded a BFB and wants permission to reflash.
Your first job is to stop that. A reflash needs a power cycle nobody has a window for, and it erases the state that identifies the fault; without a before-state, rollback is a phrase and not a plan.1718 The duty at this hour is issue replication, and replication starts with evidence.22
RShim exists because this card has its own computer inside it. When the host sees nothing useful you still need to watch the Arm boot and talk to it out of band, so the interface rides a PCIe management function and hands you a console and a misc file, single-owner by design, one backend at a time.10 That console is why this call can end tonight.
The network lead says the thing he says once a week: show me the counter. The evidence you destroy at 01:40 is the evidence you need at 09:00. You ask the operator to open the console and read the last ten lines aloud.
1Triage discipline: ask, run, do not
An escalation reaches you as a paragraph and a screenshot. The FAE job description names issue replication as a core duty, and replication starts with evidence, not with a fix.22 Every runbook in this lesson has three columns: what to ask, what to run, and what not to do first.
What to ask is the same list every time: server model and riser; SKU and whether it is a Dell catalogue part; mode (mlxconfig q INTERNAL_CPU_OFFLOAD_ENGINE); BlueField firmware; BFB and DOCA-Host versions; BIOS, iDRAC and CPLD levels; what changed last.152 What to run is read-only: lspci -d 15b3: -nn, mlxconfig -d <dev> q, flint -d <dev> q or mlxfwmanager --query, cat /dev/rshim0/misc, a console capture with minicom -D /dev/rshim0/console -C rshim_console.txt, and dmesg filtered for mlx5_core.192010 Do not trust a single tool’s version string: the four-way match is pkg-config --modversion doca-common, /opt/mellanox/doca/applications/VERSION, /opt/mellanox/doca/tools/doca_caps --version on the host, and cat /etc/mlnx-release plus bfver on the Arm; disagreement means a partial install.23
What not to do first is the column that saves cards. No firmware burn, no BFB reflash, no mode flip and no RMA until the evidence bundle exists, because each of those destroys the state you need and each requires a power cycle you may not have a window for.173 The hardware-safety rule is explicit: without a before-state, rollback is a phrase, not a plan.18
Two boundaries end local debugging. A BlueField-3 looping at PSC BL1 START with a stalled BFB transfer, flint reporting DRAM not available and the BMC golden-image script aborting on “PCIE is not in reset” got one answer from NVIDIA: open an Enterprise Support case, handled according to entitlement.9 A card still undetectable after a full power cycle and a firmware update gets Dell’s answer: open a service request for a replacement.1
Host does not see the BlueField
Why does the runbook forbid a BFB reflash as the first move even when the symptom is a boot problem?
2Card not detected: PCIe and power
Pattern 1: DPU not detected after a BIOS update. Dell’s KB 000379421 describes the Lifecycle Log repeating “Device not detected: Nvidia Network Adapter” (PR8): a PCIe training failure caused by BlueField-3 firmware below 32.46.3048, released 14 August 2025.1 KB 000227031 covers the neighbouring symptom, “No Memory Found” at POST after BIOS updates, then an automatic reboot after about a minute, caused by unsynchronised state machines between BlueField-3, iDRAC and CPLD at CPLD 1.1.5/1.1.7, iDRAC 7.10.50.00 and firmware 32.40.1000, in DPU mode.2 The community version, a B3220SH absent from lspci and shown as N/A by the BIOS across every slot on a consumer board, stayed unresolved for five months; the advice was slot incompatibility.4
- Ask: exact firmware; BIOS, iDRAC, CPLD levels; which slot and riser; whether a power cycle, not a reboot, has happened since the change.13
- Run: read the LC log for PR8;
lspci -d 15b3:; if the card appears,mlxfwmanager --queryfor the version.120 - Do not first: reseat or RMA; flash firmware on a card the host cannot see; loop warm reboots. The KB order is power cycle, then firmware 32.46.3048 or later with a matching BFB, then replacement.1
Pattern 6: the auxiliary-power halt. A B3210 in DPU mode on Ubuntu 24.04 with DOCA-OFED 3.4.1 logged mlx5_core “Firmware over 120000 MS in pre-initializing state, aborting”, mlx5_init_one failed with error code -110, and an empty ibv_devices, while rshim attached normally.7 The console showed the bootloader halting: “ATX missing - initiating halt sequence” and “CRITICAL ERROR: ATX power not detected! Halting system!!”; the card draws up to 150 W and the slot supplies roughly 66 W.7 NVIDIA’s specifications page requires the supplementary 8-pin ATX connector on the four DPU SKUs and 75 W minimum through the slot for every SKU; SuperNICs have no connector; the HGX B300 RA states the same requirement for the B3240.1621
- Ask: is an 8-pin cable fitted, and is it a PCIe (GPU-style) cable? The thread is explicit: not a CPU/EPS12V cable, the pinout differs and can damage the card.7
- Run:
cat /dev/rshim0/misc, then the console; the halt message is unambiguous.710 - Do not first: swap BFBs, reinstall DOCA-Host, or blame the kernel. The fix needed three things in order: power, a BFB install, and only then the mode the customer wanted.7
Why does rshim attaching while ibv_devices is empty point at power rather than at the host driver?
3RShim and tmfifo
Pattern 4: rshim missing. The card is in lspci, including the SoC Management Interface function, rshim.service is active, but there is no /dev/rshim* and tmfifo_net0 reports “Device not found”.5 The answer was ownership: cards with an onboard BMC own the RShim by default; disable RShim on the BMC side so the host can claim it.5 The troubleshooting guide generalises it: RShim is single-owner, the error “Another backend already attached” means someone else holds it, and the transfer is systemctl stop rshim; systemctl disable rshim on the current owner and systemctl enable rshim; systemctl start rshim on the desired one.10 A different message, “failed to setup CUSE rshim”, means cuse.ko is missing: install kernel-modules-extra, modprobe cuse, restart the service.10
- Ask: does the card have a BMC (every BlueField-3 DPU and SuperNIC does); is anything attached to the BMC’s USB; what does the rshim log say exactly.1610
- Run:
journalctl -u rshim,ls -la /dev/ | grep rshim, and on a multi-card hostcat /dev/rshim<N>/misc | grep DEV_NAMEto map each device to its PCIe address before touching anything.1019 - Do not first: reinstall the rshim package in a loop, or reflash. Ownership does not change by reinstalling.10
Pattern 5: tmfifo not coming up. If tmfifo_net0 is absent, go back to Pattern 4. If it exists on both sides but nothing answers, check the addressing: the guide uses host 192.168.100.1 and Arm 192.168.100.2 on a /30, and some DOCA-Host installs bridge tmfifo_net0 into tm-br.111019 A ping 192.168.100.2 that works can be answered locally; run ip route get 192.168.100.2 and accept only dev tmfifo_net0 or dev tm-br.1819 Raise the log level with echo 'DISPLAY_LEVEL 2' > /dev/rshim0/misc and look for INFO[MISC]: Linux up and INFO[MISC]: DPU is ready; login is not possible before that marker even if a prompt appears.1011
- Do not first: assign new addresses on both sides at once, or SW_RESET a DPU that is still installing (the installer marker sequence tells you).11
Why is rshim missing on a BMC-equipped B3220 usually not a driver bug?
4bfb-install says done; the DPU says nothing
Pattern 2. After bfb-install --bfb <image> --rshim rshim0 the host could not ssh ubuntu@192.168.100.2, although tmfifo_net0 existed on both sides with the /30 addresses.6 The console (screen /dev/rshim0/console 115200) printed nothing until echo "SW_RESET 1" > /dev/rshim0/misc, after which BL2, BL31 and UEFI boot logs appeared.6 mlxfwmanager --query showed B3220 firmware 32.42.1000 while bfvcheck on the Arm recommended 32.37.1306: the firmware and the BFB were not aligned.6 The suggested fixes were the host-side tmfifo interface configuration and DHCP versus static on the DPU out-of-band interface.6
The deployment skill states the rule bluntly: do not trust bfb-install exit code 0 alone; require the console markers, the tmfifo route, SSH, then cat /etc/mlnx-release and bfver.18 It also classifies a not-yet-healthy card into six states: installer still running, UEFI only, Linux up but tmfifo down, tmfifo up but SSH down, Arm fine but host PFs unbound, host and BlueField version mismatch.1819 Each state has a different owner.
- Ask: the exact BFB file name and the firmware it carries; whether
INFO[MISC]: DPU is readywas ever seen; whatcat /dev/rshim0/miscshows now; whether abf.cfgsetubuntu_PASSWORDor a network hook.11 - Run, in this order: console;
SW_RESET 1only if the console is silent;mlxfwmanager --queryagainst the BFB’s expected firmware;ip route get 192.168.100.2; then the DPU side over the console.61018 - Do not first: re-run
bfb-installrepeatedly, change both tmfifo addresses, or push a different BFB before reading the console. Dell’s KB adds the alignment rule from the other direction: keep the BFB aligned with the updated firmware.1
Why is a silent rshim console after bfb-install more informative than the exit code?
5Offload went to software after an upgrade
Pattern 3. A DPU running OVS-DOCA lost hardware offload entirely after the operator added an OVS mirror (P1 to pf1hpf for Flow Inspector).8 NVIDIA’s answer called it expected: “the current OVS-DOCA offload provider cannot offload multi-output flows. This is a limitation of OVS-DOCA, not BF2/BF3 hardware”, and pointed to tc-flower based mirroring under OVS-kernel hardware acceleration instead.8
Hold that answer next to the current page before you repeat it. The DOCA 3.5.0 OVS-DOCA page lists a mirror feature (ovs-vsctl create mirror, 128 mirror configurations by default, raised with other_config:doca-mirror-max — the page’s own example sets 2048) among offloaded capabilities, and separately names ct-ct-nat as unsupported.12 So the honest statement is: “for your DOCA version, the page says X”; the forum answer is evidence of how NVIDIA classifies such cases (software provider limit, not silicon), not a permanent fact.
The verification command is ovs-appctl dpctl/offload-stats-show; ovs-appctl coverage/show exposes counters such as doca_async_queue_full, doca_async_add_failed and doca_pipe_resize.12 A search snippet suggests flows shown with dp:ovs were handled in software, but that indicator was not found on a fetched page, so use the offload-stats output as the evidence.12
After an upgrade, two 3.5.0 changes are the first places to look when offload behaviour shifts: OVS-DOCA gained IPv6 connection tracking and B3220SH support, and the driver replaced the MR cache with FRMR pools.13 Then check lock-step: OVS-DOCA is a DOCA service, therefore Level 2 against firmware and BFB, and a July service with an October firmware is outside the guaranteed window.14
- Ask: what changed (mirror, CT knob, DOCA-Host or BFB version); which DOCA version on the Arm; whether the bridge is
datapath_type=docaandhw-offload=true.12 - Run:
ovs-appctl dpctl/offload-stats-show,ovs-appctl coverage/show, andovs-appctl vlog/setat debug for the offload module if needed.12 - Do not first: file a hardware bug, downgrade firmware alone, or remove OVS-DOCA. Multi-output,
ct-ct-natand Geneve option limits are documented boundaries first.128
Brief — Customer's R760 with a BlueField-3 B3220 (DPU mode) lost the DPU after a BIOS update. Host sees nothing. They are about to reflash the DPU firmware "to be safe".
Dell L2 support engineer, bridge call with the customer: We are about to reflash the DPU firmware from the host — go ahead?
Why must the FAE re-check the current OVS-DOCA page before repeating the forum's expected-behaviour answer?
6A full triage, worked, faded, and a new case
Every step is worked for you. Read it once, then try the faded version.The same method with steps blanked out (____). Fill them in from memory.A fresh case with no steps shown. Solve it on your own.
Report received: B3210 (DPU mode), firmware 32.41.0084, Ubuntu 24.04, DOCA-OFED 3.4.1. dmesg: mlx5_core ... Firmware over 120000 MS in pre-initializing state, aborting; mlx5_init_one failed with error code -110; ibv_devices empty; /dev/rshim0 present.7
- Classify the evidence. rshim present means PCIe enumeration and the SoC management function work; the NIC function never reported firmware ready. That excludes DOCA-Host, kernel and BFB as first causes.710
- Ask for the console:
minicom -D /dev/rshim0/console -C rshim_console.txt, thenecho "SW_RESET 1" > /dev/rshim0/miscto re-run the bootloader while capturing.10 - Read the halt: “ATX missing - initiating halt sequence” and “CRITICAL ERROR: ATX power not detected! Halting system!!”.7
- Explain the physics in one sentence: the card draws up to 150 W and the slot supplies roughly 66 W; the specifications page requires the 8-pin connector on DPU SKUs.716
- Prescribe the cable: PCIe GPU-style 8-pin; never a CPU/EPS12V cable, whose pinout differs and can damage the card.7
- Sequence the recovery: fit power, power-cycle, confirm
INFO[MISC]: DPU is ready, then install the BFB the customer wants, then apply the mode change they asked for with its own power cycle.71115 - Record firmware alignment afterwards with
mlxfwmanager --queryagainst the BFB, because the reporter’s version was old.201 - Do not: swap BFBs, reinstall DOCA-Host, or RMA. None addresses a missing cable.7
Report received: B3210 (DPU mode), mlx5_core pre-initializing for ____ ms, ibv_devices ____, /dev/rshim0 ____.
- rshim present means ____ works; the ____ function never reported firmware ready.7
- Capture the console with
minicom -D ____ -C rshim_console.txt; write____to/dev/rshim0/misc.10 - Look for the halt string: “____ power not detected”.7
- The card draws up to ____ W; the slot supplies about ____ W.7
- Use a ____-style 8-pin cable; never a ____ cable.7
- Recovery order: ____, power cycle, wait for
INFO[MISC]: ____, then ____, then ____.11 - Verify alignment:
____ --queryagainst the BFB.20
New case. A Dell lab reports a B3220 in an R770: lspci shows both network functions and the SoC Management Interface, rshim.service is active, but /dev/rshim0 does not exist and tmfifo_net0 says Device not found. The lab has already reinstalled the rshim package twice.
Write the runbook (ask, run, do not) and the customer-facing explanation. Acceptance criteria:
- You name the default owner (the card’s own BMC) and the transfer procedure on both sides.510
- You cite the exact log string that would confirm it (“Another backend already attached”) and the different string for a missing
cusemodule.10 - You include the multi-card safety step: map
DEV_NAMEin/dev/rshim<N>/miscto the PCIe address before any push.19 - You state what the lab should stop doing and why reinstalling cannot change ownership.10
- You do not mention iDRAC as the owner; no fetched Dell page documents an iDRAC-to-Arm path.10
Why does the worked case end with a firmware-alignment check even though the root cause was a cable?
The last ten lines
The console has printed the same halt for hours: ATX missing, initiating halt sequence, then ATX power not detected, halting system.7 The card draws up to 150 W and the slot supplies roughly 66 W, so the supplementary 8-pin connector is not optional, and it must be a PCIe cable, not an EPS12V one, because the pinout differs and can damage the card.716 The operator finds it coiled behind the riser, fits it, and labels the free end “PCIE, NOT EPS”. No reflash, no replacement request, no second bridge call.
What you say: “Fit the PCIe 8-pin, cold power cycle, and send me the console capture before anyone touches firmware.”3
Day twenty-two is the architecture review, where the same network lead will ask why this pod needs DPUs.
Lab
Changes the system0 of 6 steps doneMutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.
Goal: a five-minute narrated demo for a Dell audience of a mode flip and an OVS-DOCA offload check on the lab BlueField-3, narrated as the triage checks you would run at each step. Mode changes require a power cycle; work inside a written maintenance window with rollback.1517
- Pre-flight evidence bundle, on camera:
lspci -d 15b3: -nn;sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q;sudo flint -d /dev/mst/mt41692_pciconf0 q;cat /dev/rshim0/misc(expectINFO[MISC]: DPU is ready);journalctl -u rshim -n 20.102011 If any command fails: stop; you now have a real escalation, not a demo. - Console proof (read-only):
sudo minicom -D /dev/rshim0/console -C rshim_console.txt, show a login prompt, exit. Explain to the audience that this capture is what you ask for first in Patterns 2 and 6.67 - Offload check in DPU mode on the Arm:
sudo ovs-appctl dpctl/offload-stats-showandsudo ovs-appctl coverage/show; point out the counters you would compare before and after a customer’s change.12 If no OVS-DOCA bridge exists: say so and skip; do not configure OVS inside a mode-flip window. - Flip to NIC mode (mutating):
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1, full power cycle. Rollback is the inverse write plus a power cycle.153 - Verify
DISABLED(1), show that the rshim console is now silent because the Arm is inactive, and explain why this silence is expected in NIC mode but a fault in DPU mode.156 - Roll back to DPU mode, power cycle, wait for
INFO[MISC]: DPU is ready, re-run step 3 and show the offload statistics return.151112 If the marker does not return within the installer’s normal window: capture the console and follow Pattern 2.
- Run the FaeScenarioSim above; it opens on the escalation scenario, an R760 with a B3220 that vanished after a BIOS update.
Expected: six scored decisions with the customer’s reaction to each. If not: replay and request the evidence bundle before proposing any action.
- Write the runbook for the assigned pattern as three lists (ask, run, do not first) plus one “escalate when” line, each item ending with the source it rests on.110
Expected: no destructive action before the evidence bundle; the escalate line names Enterprise Support or a Dell service request.91 If not: your runbook would reflash or RMA a card whose console you have not read.
- Open the forum thread for your pattern (Segment 2 to 5 sources) and check your runbook against what actually resolved it.
Expected: your first three “run” items appear in the thread. If not: note the gap; the thread is the ground truth.
- Optional container step:
docker pull nvcr.io/nvidia/doca/doca:devel-3.5.0-hostand inside itls /opt/mellanox/doca/tools/to confirmdoca_capsand the version files you would ask a customer for.2423Expected:
doca_capspresent and/opt/mellanox/doca/applications/VERSIONreadable. If not: you pulled a non-3.5.0 tag; copy the tag from the NGC page verbatim.
Retrieval check
Answer from memory before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell support engineer, in four sentences, why you ask for the rshim console capture before you let anyone reflash a BlueField-3 that will not come up.
Sources
24 sourcesFacts in this lesson were checked against DOCA 3.5.0 docs, BlueField troubleshooting guide, Dell KBs 000379421/000227031/000300192, NVIDIA forum threads 312378/271327/304754/380253/348988/373806, 2026-09-06. Dates are when each page was fetched. Sources fetched 2026-09-06, checked 16 days ago.
- Dell KB 000379421: BlueField-3 DPU PCIe Initialization Failure
- Dell KB 000227031: No Memory Found event on NVIDIA BlueField-3 enabled PowerEdge servers
- Dell KB 000300192: XE9680L power cycle required after updating DPU firmware
- NVIDIA Developer Forums: BlueField-3 DPU not found in the server
- NVIDIA Developer Forums: BlueField visible in lspci, rshim running, but no /dev/rshim and no tmfifo_net0
- NVIDIA Developer Forums: BF3 RShim not working (bfb-install done, no ssh to 192.168.100.2)
- NVIDIA Developer Forums: BlueField-3 (DPU mode) stuck in FW pre-initializing, no host netdevs
- NVIDIA Developer Forums: Hardware offload lost when applying mirror in DOCA
- NVIDIA Developer Forums: BlueField-3 stuck at PSC BL1 START, RShim BFB transfer stalls
- BlueField Platform Software Troubleshooting Guide: SoC Management Interface (RShim)
- BF-Bundle Installation and Upgrade
- OVS-DOCA Hardware Acceleration
- DOCA 3.5.0 Changes and New Features
- DOCA Dependency Compatibility Policy
- BlueField Modes of Operation
- NVIDIA BlueField-3 Networking Platform User Guide: Specifications
- NVIDIA/skills: doca-hardware-safety
- NVIDIA/skills: doca-bf3-deployment TASKS
- NVIDIA/skills: doca-bf3-deployment CAPABILITIES
- MFT 4.34.1-18 LTS: General Information (tool list)
- NVIDIA HGX AI Factory Enterprise Reference Architecture: Components
- NVIDIA: Senior Field Application Engineer, Networking System Hardware Design (posting)
- NVIDIA/skills: doca-version CAPABILITIES
- NGC catalog: nvidia/doca/doca container
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- RA course Who do I call: three support paths, one asset tag RA course · Same ground: Evidence bundle, support and case
- Elsewhere in this course rshim and the BFB path Elsewhere in this course · Same ground: RShim, BFB and tmfifo
- Elsewhere in this course Out-of-band: BMC, Redfish, and the BlueField-4 path Elsewhere in this course · Same ground: bmc, RShim and BFB