Skip to content
nodeza.learn
DOCA course/1. Foundations/Lesson 1.3

BlueField-3 under the hood

S1·E3The card that boots and the card that does not · Dell lab, Round Rock, day two of the BlueField-3 bring-up

S1·E3Understand~30 minsources checked todayverified against BlueField-3 HW user guide (introduction, specifications, aux card kit, index), BlueField-3 datasheet, DOCA 3.5.0 DPA subsystem / libraries / release-notes pages, Dell R770 and R7725 ISM, 2026-09-06

Builds on: The NVIDIA networking portfolio

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Describe the BlueField-3 compute complex (Arm cores, caches, DDR5 channels, eMMC and SSD) and how DPU and SuperNIC SKUs differ.
  • Explain how the 32 PCIe Gen5 lanes are split between the host x16 and the auxiliary card kit, and what the kit does and does not support.
  • Locate the crypto, storage and DPA accelerators and state which have a DOCA 3.5.0 library behind them and at what quality level.
  • Identify a BlueField-3 SKU from its OPN, including crypto variant, power connector and form factor.
  • Name the management-plane paths (BMC, out-of-band port, RShim) and the device identity used by MFT.

Episode 3 — The card that boots and the card that does not

The situation · Dell lab, Round Rock, day two of the BlueField-3 bring-up

Two PowerEdge servers on the rails, one card from each of the BOM’s two part numbers. The B3140H SuperNIC came up on the first try. The B3240 does not: no link, no Arm console, and on the third attempt the display prints a “No Memory Found” event at POST before the server reboots itself about a minute later. The customer’s night-shift operator came in early to watch, and is already labelling both risers with the label maker he carries like a sidearm. The demo is at two.

You start where the difference is physical. A DPU SKU is not a NIC with software bolted on: 16 Armv8.2+ A78 Hercules cores, 32GB of DDR5, 40GB of eMMC plus a 128GB onboard SSD, and 32 lanes of PCIe Gen 5.0 split between the host x16 and an optional auxiliary card[2][3]. That is a computer, and a computer has to be fed: DPU SKUs require the supplementary 8-pin ATX power connector, while the SuperNIC SKUs run on 75W of slot power with no external connector[2]. The card that “just worked” this morning is the one that asked for less.

The POST message has a name, too. Dell documents it for BlueField-3 in DPU mode on specific firmware, iDRAC and CPLD combinations, and the server recovers on its own[13]. Before firmware, before drivers: account for every watt the SKU asks for. Now open the card up.

1The compute complex

BlueField-3 carries “x8 / x16 Armv8.2+ A78 Hercules cores (64-bit)”.[1] The datasheet adds the cache structure: “8MB L2 cache” and a “16MB LLC system cache”.[3] DPU SKUs get all 16 cores, SuperNIC SKUs 8.[4] Clocks depend on series, not core count: the 16-core P-Series parts (B3240, B3220, B3210) run “560MHz/2133MHz”, while every E-Series part — the 16-core B3210E and all four 8-core SuperNICs — runs “505MHz/2000MHz”.[2]

Memory is where DPU and SuperNIC diverge most. The introduction’s summary line is “16GB/32GB @5200MT/s or @5600MT/s single/dual-channel DDR5 SDRAM memory”.[1] The specification tables unpack “single/dual”: DPU SKUs use “Dual-channel with 20 DDR5 + ECC (128bit + 16bit ECC) for a total of 32GB”, SuperNIC SKUs “Single-channel with 10 DDR5 + ECC (64bit + 8bit ECC) for a total of 16GB @ 5200MT/s”.[2] Speed by SKU: B3240 and B3210 at 5600MT/s; B3220 at 5200 (5600 on hardware revision AC and later), B3210E at 5200, all SuperNICs at 5200.[2] The APR21 datasheet’s “16GB on-board DDR5” predates this split and is stale for DPU SKUs.[3][1]

Storage is identical on every SKU and has two roles: “40GB with 30K Write Cycles eMMC for SoC BIOS” and an SSD of “128GByte for user SoC OS, logs and application SW”; each spec row reads “40GB pSLC eMMC memory + 128GB SSD”.[1][2] When lesson 2.3 streams a BFB to the card, the Arm operating system lands on that SSD.

The root of trust is on the same die: “Secure boot (RSA authenticated)”, “Secure boot with Public key accelerator”, a true random number generator and EC-DSA/EC-DH support.[3]

Ships in DPU mode (Arm owns the NIC — ECPF)
B3240 · B3220 · B3210 · B3210E · Needs the supplementary 8-pin ATX power connector. Dell R7725 ISM lists B3220/B3240 at 150 W.
PCIe Gen5 x16 → host32 lanes total · switch bifurcation up …16× Arm A78 coresArmv8.2+ Hercules · 560/2133 MHz (B…32 GB DDR5dual controllers · 5600 MT/s (B3220…40 GB eMMC pSLC + 128 GB SSDon every SKU · boot/OS storageBMC (integrated, all models)Redfish · IPMI 2.0 · NC-SI · owns r…1GbE OOBRJ45 "MGMT"rshim / tmfifoPCIe fn .2 or …ConnectX-7-generation NIC core + e-switchup to 400 Gb/s Ethernet or NDR IB · ASA…DPA — 16 cores / 256 threadsdatapath accelerator …2× QSFP112 — 400G / 200G / 100G2× QSFP112 — 400G/NDR (B3240) · 200G/ND…Crypto — IPsec / TLS / MACsec inlineAES-XTS 256/512 · PKA · TRNGCompression enginedatasheet: "Decompression engine"Erasure coding / RAID"Erasure coding for RAID implementa…SHA enginehashing / integrityRegEx (datasheet only)no DOCA library in 3.xArm subsystem · memory · managementNetwork coreHardware accelerators

Select a block. Every block explains what it is, which DOCA library touches it, and one thing an FAE gets asked.

Facts: BlueField-3 user guide (introduction, specifications), DOCA 3.5.0 libraries table, DPA subsystem, BMC 26.04 docs, Dell KB 000227031 / R7725 ISM.

Click a block to see its spec line and source. Toggle the SKU to SuperNIC and watch the Arm cluster halve, one DDR5 channel disappear and the 8-pin ATX connector vanish.

2The network core and the 32 PCIe lanes

The network side offers “1, 2, 4 ports with up to 400Gb/s” of Ethernet, or InfiniBand as a “Single port of NDR (400Gb/s), or dual ports of NDR200 / HDR (200Gb/s)”.[3] Every shipping SKU uses QSFP112 cages; the port count and top speed vary by SKU (segment 5).[2] Fixed-function offloads in the NIC core include “ASAP² - Accelerated Switch and Packet Processing”, VXLAN/GENEVE/NVGRE overlays, RoCE with Zero Touch RoCE, and a “Programmable flexible parser: user defined”.[3] The fetched BlueField-3 pages do not say “ConnectX-7” verbatim (the BlueField-2 guide does say “ConnectX-6 Dx”), so describe it as a ConnectX-7-generation 400G core rather than quoting a name.[1]

On the host side, “32 lanes of PCIe Gen 5.0” with “PCIe switch bi-furcation of up to 16”.[3] Sixteen of those lanes reach the server through “PCIe Gen 5.0 (32GT/s) through x16 edge connector”.[1] The other sixteen exist on DPU SKUs as an option: the PCIe Auxiliary Card Kit splits the bus into “two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16 Auxiliary PCIe Connection card”, on B3220, B3240, B3210 and B3210E.[8] The kit page names two modes. Socket-Direct “enables direct access from each CPU to the network through its dedicated PCIe interface” — but “The socket-direct functionality is currently not supported by firmware”.[8] Down Stream Port extension is the working mode: “The BlueField-3 DPUs with downstream port extension option integrate a single network adapter silicon on a primary board”.[8] Harnesses come in 150mm, and 250mm or 350mm for Gen 4.0 only.[8]

The 3.5.0 release notes list the eight DPU OPNs that take the kit and the 8-pin ATX cable: B3220 900-9D3B6-00CV-AA0 and -00SV-AA0; B3240 900-9D3B6-00CN-AB0 and -00SN-AB0; B3210 900-9D3B6-00CC-AA0 and -00SC-AA0; B3210E 900-9D3B6-00CC-EA0 and -00SC-EA0.[10]

3Accelerators and the DPA

The datasheet’s accelerator list is the DPU’s value proposition in hardware. Crypto: “MACsec/IPsec/TLS data-in-motion”, “AES-XTS 256/512-bit data-at-rest encryption”, a public key accelerator and a TRNG.[3] Every SKU is sold in two variants, and the specifications page labels each OPN pair as Crypto Enabled or Crypto Disabled — the B3220’s 900-9D3B6-00CV-AA0 versus -00SV-AA0, for example.[2] Storage: “BlueField SNAP - Elastic block storage - NVMe and VirtIO-blk”, “NVMe-oF and NVMe/TCP acceleration”, a “Decompression engine”, “Erasure coding for RAID implementation”.[3] Time: a “PTP hardware clock (PHC)” with “Configurable PPS In and PPS Out”.[3]

One line on that list is a trap. “Regular expression (RegEx) matching” is on the APR21 datasheet, but the DOCA 3.5.0 libraries page lists no RegEx library.[3][6] A hardware block being present does not mean the software supports it; the libraries page is the contract.

The block that distinguishes BlueField-3 from BlueField-2 is the datapath accelerator: “Programmable Datapath Accelerator: 16 cores, 256 threads; Programmability through DOCA”.[3] NVIDIA defines it as “an embedded subsystem designed to accelerate workloads that require high-performance access to the NIC engines”.[5] Its execution model: a process has multiple threads and “Each thread can be mapped to a different execution unit to achieve parallel execution”.[5] Its memory: “Each execution unit has a private L1 data cache. The L1 code cache is shared among all the execution units in a DPA core. The L2 cache is shared among all the DPA cores”, and “The DPA caches are backed by NIC private memory, which is located in the DPU’s DDR memory banks”.[5] Applications reach it “through the DOCA library (DOCA DPA) or the DOCA driver layer (FlexIO SDK)”.[5]

Two facts to keep straight. First, maturity: in 3.5.0 DOCA DPA is Beta, Flex IO is GA and DOCA PCC (congestion control, which runs on the DPA) is Beta.[6] Second, reach: the samples’ dpacc_mcpu build option defaults to nv-dpa-bf3,nv-dpa-cx7,nv-dpa-cx8,nv-dpa-cx9,nv-dpa-cx10, so the DPA is a ConnectX-family capability, not a BlueField-3 exclusive.[7] The compiler, dpacc, is installed under /opt/mellanox/doca/tools.[15] The DPA’s instruction set and clock frequency are not stated on the official page; figures you may have seen (RISC-V, 1.8 GHz) come from third parties.[5]

Ships in DPU mode (Arm owns the NIC — ECPF)
B3240 · B3220 · B3210 · B3210E · Needs the supplementary 8-pin ATX power connector. Dell R7725 ISM lists B3220/B3240 at 150 W.
PCIe Gen5 x16 → host32 lanes total · switch bifurcation up …16× Arm A78 coresArmv8.2+ Hercules · 560/2133 MHz (B…32 GB DDR5dual controllers · 5600 MT/s (B3220…40 GB eMMC pSLC + 128 GB SSDon every SKU · boot/OS storageBMC (integrated, all models)Redfish · IPMI 2.0 · NC-SI · owns r…1GbE OOBRJ45 "MGMT"rshim / tmfifoPCIe fn .2 or …ConnectX-7-generation NIC core + e-switchup to 400 Gb/s Ethernet or NDR IB · ASA…DPA — 16 cores / 256 threadsdatapath accelerator …2× QSFP112 — 400G / 200G / 100G2× QSFP112 — 400G/NDR (B3240) · 200G/ND…Crypto — IPsec / TLS / MACsec inlineAES-XTS 256/512 · PKA · TRNGCompression enginedatasheet: "Decompression engine"Erasure coding / RAID"Erasure coding for RAID implementa…SHA enginehashing / integrityRegEx (datasheet only)no DOCA library in 3.xArm subsystem · memory · managementNetwork coreHardware accelerators
DPA overlay — datapath accelerator to NIC

16 cores / 256 threads sit next to the NIC engines. Memory is NIC private memory in DDR. A host or Arm DOCA app compiles kernels with dpacc (DOCA DPA, Beta) or Flex IO (GA) and launches them through a DPA context; DOCA PCC (congestion control) and SNAP-4 are the shipped users.

source

Select a block. Every block explains what it is, which DOCA library touches it, and one thing an FAE gets asked.

Facts: BlueField-3 user guide (introduction, specifications), DOCA 3.5.0 libraries table, DPA subsystem, BMC 26.04 docs, Dell KB 000227031 / R7725 ISM.

DPA overlay: execution units, the per-EU L1 data cache, the per-core L1 code cache, the shared L2, and the path to NIC private memory in DDR.

4Management plane, power and identity

Three paths reach the card without the host OS. The integrated BMC is “Present on all models” and, in the introduction’s words, a “local NIC BMC (Baseboard Management Controller)” enabling “remote power cycling, board environment monitoring” and “individual interface resets”.[4][1] It is reached through the “1GbE RJ45 out-of-band port”, and the datasheet lists “NC-SI, MCTP over SMBus, and MCTP over PCIe” for platform-BMC integration.[1][3]

The third path is RShim: “RShim, the SoC management interface in the BlueField System-on-Chip (SoC), enables management, monitoring, and debugging of the device”.[9] On a host with the rshim driver it appears as /dev/rshim0/boot, /dev/rshim0/console and /dev/rshim0/misc, plus a network interface, tmfifo_net0; it runs over PCIe or, on boards with the connector, over USB to an external host.[9] The misc file shows DISPLAY_LEVEL and DEV_NAME (for example pcie-04:00.2), which tells you which function carries it.[9] Lesson 2.3 uses boot to push a BFB; lesson 1.4 explains when Zero Trust turns this path off.

Identity: “The Device ID of all DPUs is 41692”, so MFT names the first BlueField-3 /dev/mst/mt41692_pciconf0 after mst start, whatever the SKU.[4][16]

Power is the fact most often missed in a Dell configuration. Every SKU needs a “Minimum 75W or greater system power supply” through the slot; the four DPU SKUs (B3240, B3220, B3210, B3210E) additionally require “supplementary 8-pin ATX power supply connectivity”, while the four SuperNIC SKUs run on “75W system power supply through the PCIe x16 interface” with no external connector.[1][2] Dell’s manuals translate that into wattage: the R770 lists the B3220 at 150 W and the B3140H at 75 W; the R7725 lists B3220 and B3240 at 150 W and B3140H at 75 W, with specific riser configurations for each.[11][12]

5SKU tables: DPU versus SuperNIC

DPU SKUs: 16 Arm cores, 32GB dual-channel DDR5, 8-pin ATX required, shipped in DPU mode.[4][2][17]

SKU Series / clocks Ports Ethernet ; InfiniBand DDR5 Form factor OPNs (Crypto Enabled ; Crypto Disabled)
B3240 P ; 560/2133 MHz Dual QSFP112 400/200/100/50/25/10 GbE ; NDR/NDR200/HDR/HDR100/EDR/FDR/SDR 32GB @5600 Dual-slot FHHL ; Cold-Aisle single-slot FHHL 900-9D3B6-00CN-AB0 ; 900-9D3B6-00SN-AB0 ; Cold-Aisle 900-9D3B6-00CN-PA0 ; also 900-9D3L6-00CN-AA0
B3220 P ; 560/2133 MHz Dual-port QSFP112 200/100/50/25/10 GbE ; NDR200/HDR/HDR100/EDR/FDR/SDR 32GB @5200 Single-slot FHHL 900-9D3B6-00CV-AA0 ; 900-9D3B6-00SV-AA0
B3210 P ; 560/2133 MHz Dual-port QSFP112 100/50/25/10 GbE ; HDR100/EDR/FDR/SDR 32GB @5600 Single-slot FHHL 900-9D3B6-00CC-AA0 ; 900-9D3B6-00SC-AA0
B3210E E ; 505/2000 MHz Dual-port QSFP112 100/50/25/10 GbE ; HDR100/EDR/FDR/SDR 32GB @5200 Single-slot FHHL 900-9D3B6-00CC-EA0 ; 900-9D3B6-00SC-EA0

SuperNIC SKUs: 8 Arm cores at 505/2000 MHz, 16GB single-channel DDR5 at 5200MT/s, 75W slot power only, shipped in NIC mode.[2][17]

SKU Ports Ethernet ; InfiniBand Form factor OPNs (Crypto Enabled ; Crypto Disabled)
B3140H Single-port QSFP112 400/200/100/50/25/10 GbE ; NDR/NDR200/HDR/HDR100/EDR/FDR/SDR Single-slot HHHL 900-9D3D4-00EN-HA0 ; 900-9D3D4-00NN-HA0
B3140L Single-port QSFP112 400/200/100/50/25/10 GbE ; NDR/NDR200/HDR/HDR100/EDR/FDR/SDR Single-slot FHHL 900-9D3B4-00EN-EA0 ; 900-9D3B4-00PN-EA0
B3220L Dual-port QSFP112 200/100/50/25/10 GbE ; NDR200/HDR/HDR100/EDR/FDR/SDR Single-slot FHHL 900-9D3B4-00CV-EA0 ; 900-9D3B4-00SV-EA0
B3210L Dual-port QSFP112 100/50/25/10 GbE ; HDR100/EDR/FDR/SDR Single-slot FHHL 900-9D3B4-00CC-EA0 ; 900-9D3B4-00SC-EA0

All eight share PCIe Gen 5.0 at 32.0GT/s x16, 40GB eMMC plus 128GB SSD, and a 0°C to 55°C operating range.[2] Read the OPN from right to left: the -HA0, -EA0, -AA0, -AB0 tails follow the board family, and the middle group changes between the crypto-enabled and crypto-disabled member of each pair; the spec page is the decoder, not a naming rule.[2]

Episode 3 — Two o'clock

How it ended

The 8-pin harness was still in the accessory box[2]. With power, the B3240 boots, the BMC answers, and mst status shows mt41692_pciconf0 — the device ID every BlueField-3 shares, DPU or SuperNIC[4]. The POST event you leave alone: documented, self-clearing, not today’s problem[13]. What the operator will remember: the SuperNIC is a 75W card and the DPU is not, so check the ATX connector before you go looking at firmware. Then he peels off his label and asks the better question: when the other sixty-two arrive, which mode do they ship in? Tonight’s change window already has a config push written for it.

Lab

Read-only inventory of the Dell-lab BlueField-3. Record, do not change.

  1. Start MFT and identify the card:[16] 
    sudo mst start && sudo mst status -v
sudo mlxfwmanager --query
    Expected: /dev/mst/mt41692_pciconf0 and a query block with the part number (OPN) and PSID. Look the OPN up in the SKU tables and write down: SKU, crypto variant, DPU or SuperNIC, and whether the 8-pin ATX cable must be present.[4][2]
  2. Confirm the host link matches the spec:[2] 
    lspci | grep -i -E 'mellanox|bluefield'
sudo lspci -vv -s <bus:dev.fn> | grep -E 'LnkCap|LnkSta'
    Expected: Speed 32GT/s, Width x16 in LnkSta — PCIe Gen 5.0 SERDES at 32.0GT/s, 16 lanes. A lower width or speed means a riser or seating problem; note it, do not reseat during this lesson.
  3. Check the RShim path from the host side:[9] 
    ls /dev/rshim0/
cat /dev/rshim0/misc
ip link show tmfifo_net0
    Expected: boot, console, misc (and rshim); a DEV_NAME pcie-... line naming the PCIe function; a tmfifo_net0 interface. If /dev/rshim0 is absent, either the rshim driver is not installed on the host or the card is in Zero Trust with RShim disabled — lesson 1.4 tells you how to tell which.
  4. Record the mode parameters without setting anything:[17] 
    sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_MODEL INTERNAL_CPU_OFFLOAD_ENGINE
    Expected: ENABLED(0) on a DPU SKU, DISABLED(1) on a SuperNIC SKU; a value that contradicts the SKU you identified in step 1 is the first thing to raise with the lab owner.
  5. If the card’s BMC is on the lab management network, note its address and confirm the OOB port link light; do not log in during this lesson.[1]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in five sentences, what is physically different between a BlueField-3 B3220 DPU and a B3140H SuperNIC, and why that dictates the slot, cable and default mode.

15 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against BlueField-3 HW user guide (introduction, specifications, aux card kit, index), BlueField-3 datasheet, DOCA 3.5.0 DPA subsystem / libraries / release-notes pages, Dell R770 and R7725 ISM, 2026-09-06. Dates are when each page was fetched.

  1. BlueField-3 User Guide: Introduction · fetched 2026-09-06
  2. BlueField-3 User Guide: Specifications · fetched 2026-09-06
  3. NVIDIA BlueField-3 DPU Datasheet (PDF, APR21 revision) · fetched 2026-09-06
  4. NVIDIA BlueField-3 Networking Platform User Guide (index) · fetched 2026-09-06
  5. DPA Subsystem · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA Libraries (3.5.0) with quality levels · fetched 2026-09-06 · DOCA 3.5.0
  7. doca-samples applications/meson_options.txt (build options incl. dpacc_mcpu) · fetched 2026-09-06 · DOCA 3.5.0
  8. BlueField-3 User Guide: PCIe Auxiliary Card Kit · fetched 2026-09-06
  9. BlueField Platform Software Troubleshooting Guide: SoC Management Interface (RShim) · fetched 2026-09-06
  10. DOCA Release Notes v3.5.0 · fetched 2026-09-06 · DOCA 3.5.0
  11. Dell PowerEdge R770 Installation and Service Manual: DPU Specifications · fetched 2026-09-06
  12. Dell PowerEdge R7725 Installation and Service Manual: DPU Specifications · fetched 2026-09-06
  13. Dell KB 000227031: 'No Memory Found' event on BlueField-3 enabled PowerEdge during POST · fetched 2026-09-06
  14. NGC catalog: nvidia/doca/doca container · fetched 2026-09-06 · DOCA 3.5.0
  15. doca-samples applications/meson.build (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  16. MFT 4.34.1-18 LTS: General Information (tool list) · fetched 2026-09-06
  17. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.