Skip to content
nodeza.learn
DOCA course/1. Foundations/Lesson 1.1

Why infrastructure moved onto the NIC

S1·E1Show me the counter · Dell customer lab, Round Rock, two days before the PoC report

S1·E1Understand~25 minsources checked todayverified against NVIDIA BlueField product page, BlueField-3 HW user guide, DOCA 3.5.0 modes/overview/general-support pages, 2026-09-06

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Name, in NVIDIA's own wording, the host workloads a BlueField DPU is built to take over.
  • Distinguish SmartNIC, DPU and SuperNIC by whether an OS runs on the card and who owns the NIC data path.
  • Answer the Dell buyer's question 'why not just a faster NIC?' along the three axes of bandwidth, ownership and isolation.
  • Separate the CPU-offload figures NVIDIA has published from the ones that circulate without an NVIDIA page behind them.

Episode 1 — Show me the counter

The situation · Dell customer lab, Round Rock, two days before the PoC report

The customer’s network lead has her notebook open at page three and a question she has already asked twice. Two identical PowerEdge servers face each other on the bench: one with ConnectX-7, one with a BlueField-3 DPU. Sixty-four nodes of a bare-metal GPU cloud ride on which one goes into Friday’s PoC report. The Dell SE, holding coffee he has let go cold, has a row in his promise spreadsheet offering the customer thirty percent of their CPU back. She taps that row and says: show me the counter.

You cannot. The only quantitative claim NVIDIA owns is the 2021 launch line, the equivalent data center services of up to 300 CPU cores[2]; the current product page and the datasheet publish no percentage at all[1][3]. So you answer with work instead of arithmetic. Every item on the BlueField-3 datasheet’s list — cloud overlay and SDN acceleration, NAT and load balancing, NVMe-oF and NVMe/TCP, distributed next-generation firewall and micro-segmentation[3] — runs today as software on the host cores of both servers, and none of it is the customer’s application. That is the whole reason the DPU exists: NVIDIA positions BlueField as offloading networking, storage, and security services from host CPUs, and BlueField-3 as a 400 Gb/s infrastructure compute platform[1] — a second computer on the card whose job is the plumbing, so the host’s job can be the workload.

Never quote a number you cannot source; quote the workload list, because the workload list is sourced. That list is where this lesson starts.

1The work that never shows up on the invoice

NVIDIA’s one-line positioning for BlueField is “offloading networking, storage, and security services from host CPUs”; BlueField-3 is “a 400 Gb/s infrastructure compute platform with line-rate processing of software-defined networking, storage, and cybersecurity”.[1] The datasheet spells out what that means in workload terms: cloud overlay and SDN acceleration, NAT, load balancing and NFV; NVMe-oF, NVMe/TCP, elastic storage, compression and erasure coding; distributed next-generation firewall, IDS/IPS, root of trust and micro-segmentation; multi-tenant cloud-native supercomputing; Cloud RAN and virtualized edge gateways.[3] Every item on that list runs today as software on host cores in a conventional server, and not one of them is the customer’s application.

How large is the bill? Be exact about what is sourced. The BlueField-3 launch release of 12 April 2021 says “One BlueField-3 DPU delivers the equivalent data center services of up to 300 CPU cores, freeing up valuable CPU cycles to run business-critical applications”.[2] That is a positioning statement in a press release, not a benchmark with a method. NVIDIA’s current product page and the datasheet publish no percentage at all.[1][3] The “up to 30% of CPU” figure that circulates in trade press could not be traced to any NVIDIA page fetched for this lesson; treat it as unverified and do not present it to a customer as NVIDIA’s number.

The defensible version of the argument is therefore qualitative and workload-specific: the infrastructure stack consumes host cores, memory bandwidth and PCIe slots, and how much depends on the mix — OVS with many flows, encrypted storage, IPsec tunnels, a host security agent. That is exactly what the calculator below lets you vary. Use it to build intuition about which knobs matter, not to produce a number for a proposal.

Scenario:

vSphere 8 + NSX. Enter the per-core licence to see the lever that flips the verdict. Remember Dell KB 000225111: DSE only on Dell-sourced DPUs; NVIDIA-channel cards run NIC mode. ⚠ 25% infra share is an estimate.

Features needed
Assumptions and sources (defaults are estimates — every field is editable)
  • Nodes = 200 — Fleet size.
  • Host cores per node = 64 cores — Physical cores across both sockets (e.g. 2× 32c = 64).
  • Cores consumed by infrastructure = 20 %estimate — Share of host cores spent on vSwitch/SDN agents, storage initiator, encryption, telemetry. NVIDIA marketing says "up to 30%"; hyperscaler blogs quote 25–40% — both secondary. ESTIMATE, measure your own with perf/top. source ↗
  • Server cost per core-year = $120estimate — e.g. $30k 2-socket server ÷ 64 cores ÷ 4-year life ≈ $117. ESTIMATE.
  • Per-core software licence per year = $0estimate — Hypervisor / SDN stacks licensed per core (VMware VCF is per-core). Default 0 — enter the customer's quote; this lever dominates in virtualization farms. ESTIMATE.
  • Watts per core = 6 Westimate — ≈ 350–400 W TDP ÷ 64 cores ≈ 6 W at load. ESTIMATE.
  • Electricity price = 0.12 $/kWhestimate — US commercial average ≈ $0.12–0.13/kWh (EIA). ESTIMATE — use the site tariff. source ↗
  • PUE = 1.3estimate — Facility overhead multiplier on IT watts. 1.3 is a typical enterprise value. ESTIMATE.
  • DPU price delta vs ConnectX = $2500estimate — Dell configurator snippets: BF-3 B3140H SuperNIC $6,213.80 (R7725), Partner DPU dual-port 400GbE $8,347.83 (R670) → ≈ $2.1k delta; ConnectX-7 400G list not fetched. UNVERIFIED ESTIMATE. source ↗
  • Extra watts per DPU vs NIC = 60 Westimate — B3220/B3240 are 150 W cards vs 75 W for the B3140H SuperNIC (Dell R7725 ISM). Real delta depends on load; 60 W is an ESTIMATE. source ↗
  • DPUs per node = 1 — HGX B300 RA uses one north-south BF-3 per node; VMware hosts typically one. source ↗
  • OVS / vSwitch datapath offload: frees 20% of infra cores in NIC mode, 35% in DPU mode estimate — ASAP² (tc-flower / OVS-DOCA) offloads the datapath on ConnectX too; DPU mode additionally moves ovs-vswitchd and the SDN agents to the Arm. source ↗
  • Connection tracking (CT) offload: frees 10% of infra cores in NIC mode, 15% in DPU mode estimate — CT offload is an ASIC feature (ConnectX-6 Dx and later); OVS-DOCA 3.5.0 adds IPv6 CT. source ↗
  • IPsec / TLS inline crypto: frees 15% of infra cores in NIC mode, 20% in DPU mode estimate — BF-3 datasheet: "MACsec/IPsec/TLS data-in-motion" inline. Crypto is SKU-dependent ("selective on SuperNICs") — check the OPN. source ↗
  • NVMe-oF / SNAP storage emulation: frees 0% of infra cores in NIC mode, 20% in DPU mode estimate — BlueField SNAP (NVMe / virtio-blk emulation) and the storage initiator run on the Arm cores → DPU mode only. source ↗
  • Tenant isolation / zero-trust: frees 0% of infra cores in NIC mode, 5% in DPU mode estimate — Zero Trust (Restricted) is a DPU-mode variant: the host admin loses port ownership, HW counters, tracer, RShim and FW flashing. source ↗
  • HBN (BGP/EVPN router on the DPU): frees 0% of infra cores in NIC mode, 5% in DPU mode estimate — HBN runs on the Arm side; not supported on 8 GB DDR SKUs. DPU mode only. source ↗
  • Model: incremental cores = infra cores × (DPU share − NIC share); savings = cores × ($/core-year + licence) + (cores × W/core − DPUs × extra W) × PUE × 8.76 kWh × $/kWh; payback = premium ÷ monthly savings. Recommendation rules: any Arm-only feature → DPU mode; no features + east-west AI → SuperNIC/ConnectX-8 in NIC mode; no features → plain ConnectX; else DPU mode only if payback ≤ 36 months.
Intro mode: move the OVS, storage and crypto sliders and watch which one dominates the host-core estimate for a virtualization farm.

2SmartNIC, DPU, SuperNIC: three words that are not synonyms

NVIDIA’s own 2020 definition: a DPU is “a system on a chip, or SoC, that combines” three elements — “an industry-standard, high-performance, software-programmable, multi-core CPU”, “a high-performance network interface capable of parsing, processing and efficiently transferring data”, and “a rich set of flexible and programmable acceleration engines”.[4] The same post warns that “other devices that claim to be DPUs miss significant elements of these three critical capabilities”.[4] The element a SmartNIC usually lacks is the first: a general-purpose CPU running its own operating system. A SmartNIC has flow tables, RoCE and crypto engines, but the host driver still owns the data path.

The DOCA modes page turns that into an operational test. In DPU mode “the Arm cores of BlueField are active, and the embedded Arm system runs services that manage the NIC resources and data path”.[5] In NIC mode “the Arm cores of BlueField are inactive, and the device functions as an NVIDIA ConnectX network adapter” — DPU silicon behaving as a SmartNIC.[5] BlueField-2 and BlueField-3 DPU SKUs are shipped in DPU mode by default; BlueField-3 SuperNIC SKUs are shipped in NIC mode by default.[5]

So “SuperNIC” names a product role, not a single chip. A BlueField-3 SuperNIC (B3140H, B3140L, B3220L, B3210L) is E-Series BlueField-3 silicon with 8 Arm cores and 16GB DDR5, sold in NIC mode; a BlueField-3 DPU (B3240, B3220, B3210, B3210E) carries 16 Arm cores and 32GB.[6] The ConnectX-8 SuperNIC is a different device altogether: “up to 800 gigabits per second (Gb/s)”, “PCIe Gen6 @ 64GT/s through x16 edge connector”, InfiniBand and Ethernet in one adapter — and its manual describes no Arm cores.[7] Historically the DPU has always been a NIC plus a computer: BlueField-2 integrates a ConnectX-6 Dx alongside “eight 64-bit Armv8 A72 cores” and tops out at “up to 200 Gigabit Ethernet”.[14]

Term Own OS on the card Who owns the NIC data path NVIDIA examples
SmartNIC No Host driver ConnectX-7 / -8 / -9; any BlueField in NIC mode
DPU Yes (Arm Linux) Embedded Arm subsystem (DPU mode) BlueField-2; BlueField-3 B3240 / B3220 / B3210 / B3210E
SuperNIC Depends on the part Host driver (NIC mode) BlueField-3 B3140H (Arm present, inactive); ConnectX-8 (no Arm)

The table’s last column is the point: the same word “SuperNIC” covers a card with dormant Arm cores and a card with none.[5][7]

3Why not just a faster NIC? The Dell buyer's question

An enterprise buyer with ConnectX-7 in the rack asks the only question that matters: “Is BlueField-3 just the faster one?” The answer has three axes, and NVIDIA’s own documents supply each.

Bandwidth. If the requirement is east-west throughput — RoCE between GPUs, storage fabric — the modes page is blunt: “Operating in NIC Mode on BlueField-3 reduces power consumption, improves network performance, and minimizes the host memory footprint”.[5] That sentence is NVIDIA explaining why its own SuperNIC ships with the Arm cores off. For pure bandwidth, a ConnectX or a SuperNIC in NIC mode is the right part, and active Arm cores are idle cost.

Ownership. In DPU mode the NIC resources and data path are managed by the embedded Arm system, so the virtual switch, the storage initiator or the firewall run on the card and are patched there, independent of the host OS image.[5] A faster NIC cannot substitute for this: it puts the same infrastructure software back on the tenant’s cores under the tenant’s OS.

Isolation. Zero Trust mode: “The Arm cores of BlueField are active, and the embedded Arm system runs services to manage the NIC resources and data path while enforcing restrictions on the external host (host isolation)”.[5] For bare-metal-as-a-service or any design where the host administrator is not trusted, no ConnectX offers an equivalent.

What the customer gets for the extra cores is the datasheet’s accelerator list: “BlueField SNAP - Elastic block storage - NVMe and VirtIO-blk”, “NVMe-oF and NVMe/TCP acceleration”, “MACsec/IPsec/TLS data-in-motion”, “AES-XTS 256/512-bit data-at-rest encryption”, erasure coding and a decompression engine.[3] Each one is a service the host was running in software or not running at all.

4Where DOCA fits

DOCA is the software that makes the previous segment true. It ships as two packages: “DOCA-Host: the software package installed on the host server which includes different DOCA installation profiles” and the “BlueField Software Bundle (BF-Bundle): the software package installed on the BlueField Arm cores”.[9] DOCA provides “all the required libraries and drivers for hosts that include NVIDIA Networking platforms (i.e., BlueField and ConnectX)” — one framework for both device classes, so a ConnectX-only server and a BlueField server share a host stack.[9]

The DOCA component vocabulary you will meet in every later lesson: libraries and drivers; samples, “an implementation example of a single feature of a given DOCA library”; applications, “higher-level reference code than the samples”; tools, “helper tools for both runtime and development”; and services, “containerized DOCA-based programs that provide an end-to-end solution for a given use case”.[9]

The boundary that matters for a Dell host image is this sentence from the general-support page: “NVIDIA BlueField-2 and NVIDIA ConnectX-7 are the last generation of BlueField and ConnectX families that are supported by MLNX_OFED”.[10] A BlueField-3 or ConnectX-8 host therefore runs DOCA-Host, not MLNX_OFED, and the 3.5.0 bundle pairs BlueField-3 with firmware 32.50.1002 and ConnectX-7 with 28.50.1002.[10] Lesson 1.5 maps the framework; lesson 1.2 places every device in the portfolio first.

Episode 1 — The BOM at 23:40

How it ended

You give her three axes and she writes down all three. Bandwidth: for east-west RoCE, NVIDIA itself says NIC mode reduces power consumption, improves network performance, and minimizes the host memory footprint[5]. Ownership: her virtual switch and NVMe-oF initiator move onto the card and are patched there, not in the host image[5]. Isolation: only an active Arm subsystem can enforce restrictions on the external host[5], and she sells two tenants per node. Then the sentence you would actually say: the GPU nodes ship with SuperNICs, the multi-tenant nodes do not. At 23:40 the SE mails the bill of materials for all sixty-four. Every line carries the same part number.

Lab

Read-only inventory of the Dell-lab BlueField-3. Nothing here changes firmware or mode.

  1. Start the MFT register-access service and list devices. mst “lists the available mst devices” and starts or stops “the register access driver”.[13] 
    sudo mst start
sudo mst status -v
    Expected: a /dev/mst/mt41692_pciconf0 device — 41692 is the device ID of all BlueField-3 DPUs and SuperNICs.[6] If mst is not found, MFT is not installed on this host; stop and note it.
  2. Query the two mode-related parameters and record them verbatim (do not set anything): 
    sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_MODEL INTERNAL_CPU_OFFLOAD_ENGINE
    Expected: INTERNAL_CPU_OFFLOAD_ENGINE reads ENABLED(0) on a DPU SKU (DPU mode) or DISABLED(1) on a SuperNIC SKU (NIC mode).[5] INTERNAL_CPU_MODEL is a legacy parameter that the 3.5.0 modes page no longer uses for BlueField-3; record it, do not act on it.[5]
  3. Find the card on the PCIe bus and record vendor, device and link lines: 
    lspci | grep -i -E 'mellanox|bluefield'
lspci -vv -s <bus:dev.fn from the line above> | grep -E 'LnkCap|LnkSta'
    Expected: the BlueField-3 functions, with a Gen5 x16 link on a properly seated card.
  4. Query firmware without updating it. mlxfwmanager “is a firmware update and query utility”; --query only reads.[13] 
    sudo mlxfwmanager --query
    Expected: a BlueField-3 firmware in the 32.x line; DOCA 3.5.0 bundles 32.50.1002.[10] Record the FW version, PSID and part number. If it differs from 32.50.1002, note the gap — do not update in this lesson.

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why a customer would buy a BlueField-3 DPU instead of a faster ConnectX, and when they should not.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against NVIDIA BlueField product page, BlueField-3 HW user guide, DOCA 3.5.0 modes/overview/general-support pages, 2026-09-06. Dates are when each page was fetched.

  1. NVIDIA BlueField DPU product page · fetched 2026-09-06
  2. NVIDIA press release: NVIDIA Extends Data Center Infrastructure Processing Roadmap with BlueField-3 (2021-04-12) · fetched 2026-09-06
  3. NVIDIA BlueField-3 DPU Datasheet (PDF, APR21 revision) · fetched 2026-09-06
  4. NVIDIA blog: What Is a DPU? (Kevin Deierling, 2020-05-20) · fetched 2026-09-06
  5. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  6. NVIDIA BlueField-3 Networking Platform User Guide (index) · fetched 2026-09-06
  7. ConnectX-8 SuperNIC User Manual: Introduction · fetched 2026-09-06
  8. NVIDIA Spectrum-X Ethernet platform · fetched 2026-09-06
  9. DOCA Overview · fetched 2026-09-06 · DOCA 3.5.0
  10. DOCA General Support (OS matrix, firmware table) · fetched 2026-09-06 · DOCA 3.5.0
  11. NGC catalog: nvidia/doca/doca container · fetched 2026-09-06 · DOCA 3.5.0
  12. doca-samples applications/meson.build (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  13. MFT 4.34.1-18 LTS: General Information (tool list) · fetched 2026-09-06
  14. BlueField-2 DPU User Guide: Introduction · fetched 2026-09-06

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.