BlueField in Kubernetes: DPF versus Network Operator
S4·E2Two cards on the loading dock · A scoping call the next morning, ten days before acceptance
Builds on: Network Operator and the NicClusterPolicy, Spectrum-X rails inside Kubernetes
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Draw the line between the Network Operator's host-side NIC story and DPF's second-computer story, using the platform matrix as the deciding fact.
- Classify the DPF custom resources into provisioning, service and chaining families and say which cluster each one acts on.
- Explain the two-cluster model and where the DPU cluster's control plane actually runs.
- Choose Zero-Trust or Host-Trusted mode for a stated customer environment and defend the choice.
Episode 2 — Two cards on the loading dock
Nobody in the room can say who ordered the two BlueField-3 cards, only that they arrived in the same shipment as the rail NICs. The procurement person joins for four minutes to ask whether they can go back and what the lead time is if they cannot. The platform architect wants them running the customer’s east-west firewall, managed from the same cluster as the GPUs, and the Dell SE has already typed “DPU offload, phase two” into row 19.
You cannot do that with the operator already on the cluster, and the reason is a published line rather than an opinion. The Network Operator manages networking components so that pods on the host get fast networking, RDMA and GPUDirect, and everything it does happens on the host side of the card.[7][8] Its platform matrix supports BlueField-3 and BlueField-3 SuperNIC in NIC mode only.[6] The moment someone says “and our own service runs on the card”, they have described a second computer inside the server — which is the problem DPF exists to solve: a system that provisions and orchestrates NVIDIA BlueField DPUs in cloud environments, on dual-port cards whose arm64 CPUs run Linux and DOCA components.[1][5]
Ask which mode the card is in and who is allowed to change it; every other answer follows from that one.
Saying it well takes a map: two products, two clusters, and a CRD list you must never improvise. That map is this lesson.
1The line customers keep blurring
Both stories involve a BlueField-3, Kubernetes and NVIDIA operators, so they get conflated in nearly every scoping call. The distinction is not subtle once you have the right fact.
Network Operator “leverages Kubernetes CRDs and Operator SDK to manage Networking related Components in order to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster”.[7] Everything it does happens on the host: it deploys a DOCA driver container, device plugins, Multus and IPAM, and it makes a NIC attachable by pods on that host.[8] The card’s Arm cores are not part of the story.
DPF is “a system that provisions and orchestrates NVIDIA BlueField DPUs in cloud environments”; the repository puts it as “DOCA Platform manages provisioning and service orchestration for Bluefield DPUs”.[1][5] It states three capabilities: Kubernetes integration, DPU service orchestration and cluster management.[5][1] Hardware: “DPF supports dual port NVIDIA BlueField-3 DPUs” installed as PCI devices, whose “arm64 CPUs running Linux OS and NVIDIA DOCA Software components” are the point.[1][5]
The deciding fact for a scoping conversation is in the Network Operator’s own platform matrix: supported NICs are ConnectX-6 through ConnectX-9, BlueField-3 and BlueField-3 SuperNIC in NIC mode only.[6] A customer who wants DPU-mode offload and Kubernetes management has, by that sentence, left Network Operator territory.
1. BFB CR created
The BFB controller downloads spec.url ("Download the BFB from a remote server") and records status.versions {bsp, doca, uefi, atf} read from the image. v26.4.1 matrix: BFB 3.4 fully supported, 3.2 LTS allowed, non-LTS BFBs (3.3, 3.5) only with their matching DPF release.
DPU phases: BFB.status.phase: Initializing → Downloading → Ready
DOCA Platform Framework (DPF) v26.4.1
DPF "is a system that provisions and orchestrates NVIDIA BlueField DPUs" from Kubernetes: the operator installs controllers in the host cluster, CRs describe the desired image (BFB), sizing (DPUFlavor) and node set (DPUSet), each card becomes a DPU object, and the DPUs join a separate tenant control plane where DPUServices (Helm charts) run.
Why it matters. GA in DOCA 3.5.0. Latest tag v26.4.1 (2026-08-31); v26.4.0 shipped 2026-06-29. Requirements (v26.4.1): Kubernetes 1.33–1.36, Helm 3.5+, DPU firmware ≥ 32.38.1002, MFT 4.33.0-169, Ubuntu 24.04 + containerd; B3240/B3220/B3210; Socket Direct unsupported.
FAE note. Click any box for a YAML skeleton and what it owns. Switch trust mode to see who provisions, then walk the Reconcile stepper.
Source: networking-docs.nvidia.com · fields marked UNVERIFIED were not confirmed against the Go types.
2Two clusters, and where the second control plane lives
DPF’s architecture is a two-cluster model. The Host Cluster provisions and manages DPUs and hosts the DPU cluster’s control-plane components; the DPU Cluster manages the lifecycle of services deployed on DPUs.[2]
The part that surprises people is where the second control plane runs. DPUCluster “creates underlying Kamaji TenantControlPlane for DPU cluster management”, so the DPU cluster’s API server and friends are pods scheduled in the host cluster, not processes on the DPUs.[2] The documentation notes the flow “is based on the Kamaji Cluster Manager—but other Cluster managers may be used”, so treat Kamaji as the documented implementation rather than an architectural requirement.[2]
Diagram source (Mermaid)
flowchart LR; HC["Host Cluster - provisions DPUs"] --> DPUC["DPUCluster CR"]; DPUC --> KAM["Kamaji TenantControlPlane - runs as pods in the host cluster"]; KAM --> DC["DPU Cluster - control plane for services on DPUs"]; HC --> SET["DPUSet"]; SET --> D["DPU - flashes the BFB"]; BFB["BFB - firmware bundle downloaded from a remote server"] --> D; FLAV["DPUFlavor - configuration template"] --> D; D --> DC; DC --> SVC["DPUService - services running on the DPU"]; SVC --> SFC["ServiceFunctionChain CNI - OVS ports and flows on the DPU"]
3The CRD families, and which cluster each one acts on
DPF’s API is large, but it sorts into three families plus the objects that appear on the DPU side.[2]
| Family | CRDs | What they do |
|---|---|---|
| Provisioning | DPUSet, DPU, BFB, DPUFlavor |
DPUSet creates DPU objects and manages their lifecycle; DPU flashes the BFB to one device; BFB downloads the firmware bundle from a remote server; DPUFlavor is the configuration template referenced at DPU creation |
| Service | DPUService, DPUDeployment, DPUServiceCredentialRequest |
DPUService manages the lifecycle of services deployed on DPUs; DPUDeployment manages the lifecycle of a group of DPUServices and DPUSets; DPUServiceCredentialRequest creates a secret containing either a kubeconfig or a token, which is what authorises host-to-DPU control-plane communication |
| Chaining (host side) | DPUServiceInterface, DPUServiceIPAM, DPUServiceChain |
Manage DPU service network interfaces, IP allocation configuration, and ServiceFunctionChains on the DPUs |
| DPU cluster side | ServiceInterfaceSet, ServiceChainSet, ServiceInterface, ServiceChain |
ServiceInterface creates OVS ports on the DPU; ServiceChain creates OVS flows |
The datapath is the last row. On DPU nodes the ServiceFunctionChain CNI adds OVS network interfaces to pods: the ServiceInterface controller creates the OVS ports and the ServiceChain controller creates the flows between them.[2] If you already know OVS from the DOCA side, that is the same mental model arriving through a Kubernetes API.
Two honesty notes for the field. A DPUDeployment is widely described as “a set of DPUServices and a DPUServiceChain that run on a set of DPUs with a given BFB and DPUFlavor”, but that exact sentence comes from a v25.4 search summary rather than a page fetched for 26.4.1 — the 26.4.1 component page states only that DPUDeployment manages the lifecycle of a group of DPUServices and DPUSets.[2] And DPFOperatorConfig, DPUNode and DPUServiceConfiguration do not appear on the fetched 26.4.1 component page; they are unverified for this release and should not be named to a customer as current API.[2]
4Zero-Trust or Host-Trusted, and the install you can actually run
DPF documents two deployment modes. Zero Trust Mode is “recommended for environments requiring maximum security with BMC access and secure boot capabilities”. Host Trusted Mode is “suitable for controlled environments with host trusted infrastructure”.[3] That is the whole published decision rule at this level, and it maps cleanly onto the customer’s own security posture: a shared or hostile-host environment picks Zero Trust and pays for it in BMC dependencies; a single-tenant AI cluster the customer owns end to end picks Host Trusted and gets a simpler install.
The host-trusted install is short enough to memorise:[4]
helm repo add --force-update dpf-repository https://helm.ngc.nvidia.com/nvidia/doca
helm repo update
helm upgrade --install -n dpf-operator-system dpf-operator \
dpf-repository/dpf-operator --version=v26.4.1
# verify
kubectl rollout status deployment --namespace dpf-operator-system \
dpf-operator-controller-managerThe prerequisites stated at that level are only three: “A Kubernetes cluster with administrative access for DPF Operator deployment”; “Bare-metal infrastructure with NVIDIA DPUs”; “Network access to the NVIDIA NGC Catalog”.[4] The detailed prerequisite pages — exact Kubernetes version, required CNI, BFB version, BIOS and SR-IOV settings, DHCP and MTU requirements, node labels — returned 404 at the URLs tried, so those requirements are unverified here.[4][9] Say so to a customer rather than inventing a version: “the published quick start pins the chart at v26.4.1 and does not state a Kubernetes range on that page; I will get you the prerequisite doc” is a better answer than a number you cannot source.
A Dell OEM account asks: “The customer has 24 PowerEdge nodes with BlueField-3 dual-port cards. They want a distributed firewall running on the DPUs, managed like the rest of their Kubernetes estate. What do we sell and what do we tell them?”
- Identify which product owns the request. A firewall running on the DPU means the Arm cores execute customer software. The Network Operator’s platform matrix supports BlueField-3 in NIC mode only, and NIC mode does not run customer services on the Arm.[6] Therefore this is DPF.[1]
- Check the hardware claim. DPF states support for dual port BlueField-3 DPUs installed as PCI devices — the customer’s cards match.[1]
- Name the architecture consequence, because it changes their operations. They are not adding a feature to their cluster; they are adding a second cluster whose control plane runs as Kamaji tenant pods inside the existing one.[2] That is a new backup, upgrade and RBAC surface.
- Pick the mode from their environment. Single tenant, they own the hosts, no secure-boot mandate → Host Trusted Mode.[3] Say the words “we can revisit Zero Trust if the security review asks for BMC-mediated provisioning”.
- Give them the concrete next step, pinned.
helm upgrade --install -n dpf-operator-system dpf-operator dpf-repository/dpf-operator --version=v26.4.1, verified with the controller-manager rollout.[4] - Flag what you do not know. The exact Kubernetes version, CNI, BFB and BIOS prerequisites are on pages that did not resolve; commit to sending them rather than guessing.[4]
The written answer is one paragraph: what it is, what it costs operationally, which mode, which command, and the one open item.
Now the same account asks the opposite question: “The customer bought BlueField-3 SuperNICs for a Spectrum-X training cluster. Do they need DPF?”
- Determine what the card is being used for: line-rate RDMA to ____, not services on the Arm.
- Check the platform matrix — SuperNIC is supported in ____ mode, which is what a rail NIC runs in.
- Conclude which operator owns it: ____.
- Name the one thing that would change the answer: if the customer later wants ____ running on the card.
- Write the answer in three sentences, and state explicitly what you are not selling them.
A customer has DPF running in Host Trusted Mode and now wants the same DPUs to also serve as rail NICs for a Spectrum-X training workload on the host.
Write the scoping answer. Acceptance criteria: your answer must name the mode constraint from the Network Operator platform matrix, state which of the two products would own each half, identify at least one open question you cannot answer from published documentation, and avoid asserting any DPF prerequisite that is marked unverified in this lesson.
Case notes, the whiteboard
The whiteboard ends with two boxes, not one. Host side: Network Operator, driver container, device plugins, NIC mode.[8][6] Card side: DPF, a DPU cluster whose control plane runs as Kamaji tenant pods back in the host cluster, with services placed through DPUService and DPUDeployment.[2] Phase two gets its own quarter, and procurement keeps the cards.
The ML platform team stays behind to move the case forward. The rails are proven, the scope is drawn, and their training job still finishes at a number nobody likes — so tomorrow’s design review is about which scheduler to standardise on, and everyone already has a favourite.
Lab
On the Dell-lab BlueField-3 host. Read-only: this lesson does not change the card’s mode, and you should not change it here — a mode change is a firmware operation covered in the DOCA course, with its own rollback.
- Pre-flight inventory. Record the host OS, kernel (
uname -r), the DOCA-Host version andlspci -d 15b3: -nnoutput before anything else. sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep -i internal_cpu. Expected:INTERNAL_CPU_OFFLOAD_ENGINEreadsENABLED(0)for DPU mode orDISABLED(1)for NIC mode. Ifmstis not started, runsudo mst startfirst — that is a service start, not a card change.ip -br linkandlspci -d 15b3: -nn. Expected: in NIC mode the host sees the network PFs and no separate DPU-side story; note exactly what the host can and cannot see.- Write the verdict. In one sentence: which of the two operator stories applies to this card as it is configured right now, and cite the platform-matrix fact that decides it.
- Write the counterfactual. In one sentence: what would have to change for the other story to apply, and who would have to authorise it.
- Rollback: none. No step here mutates the card. If you ran
mst start,sudo mst stopreturns the host to its previous state.
Goal: see DPF’s API surface without owning a DPU, then write the scoping paragraph.
- Create a throwaway cluster.
kind create cluster --name dpf-lab. Expected:kubectl get nodesshows one control-plane node. If not: check that the container runtime is running. - Add the chart repo.
helm repo add --force-update dpf-repository https://helm.ngc.nvidia.com/nvidia/doca && helm repo update. Expected: the repo appears inhelm repo list. If not: you have no route to NGC — note it and continue with step 5 from the published CRD list instead. - Install the operator chart only.
helm upgrade --install -n dpf-operator-system --create-namespace dpf-operator dpf-repository/dpf-operator --version=v26.4.1. Expected: the release installs. No DPU will ever be provisioned — there is no hardware, and that is fine; you are here for the CRDs. - Inventory the API.
kubectl get crds | grep -i dpuandkubectl api-resources | grep -i dpf. Expected: the provisioning, service and chaining kinds from segment 3. If a kind you expected is missing, that is data: record it against the unverified list rather than assuming a bad install. - Draw the two-cluster diagram yourself. On paper or in Mermaid, place every CRD you found into one of two boxes: acts on the host cluster, or is realised in the DPU cluster. Check
DPUServiceChainagainstServiceChain. Expected: your diagram matches the interactive in segment 2. - Write the scoping paragraph. One paragraph answering “we want DPU-mode offload plus Kubernetes”, covering: which product, the NIC-mode-only fact, the second-cluster operational cost, the mode choice, the pinned install command, and one honest unknown. Expected: under 150 words.
- Clean up.
kind delete cluster --name dpf-lab.
Retrieval check
9 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in five sentences, the difference between running the NVIDIA Network Operator with a BlueField-3 and running DPF with the same card.
Sources
Facts in this lesson were checked against DPF v26.4.1 docs (component-description, getting-started, dpf-host-trusted) re-fetched 2026-09-07; NVIDIA Network Operator v26.7.0 platform support re-fetched 2026-09-07. Dates are when each page was fetched.
- DOCA Platform Framework (DPF) Documentation v26.4.1 · fetched 2026-09-07
- DPF v26.4.1 — Component interactions · fetched 2026-09-07
- DPF v26.4.1 — Getting Started · fetched 2026-09-07
- DPF v26.4.1 — Host Trusted Mode · fetched 2026-09-07
- GitHub — NVIDIA/doca-platform · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 — Platform Support · fetched 2026-09-07
- GitHub — Mellanox/network-operator (NVIDIA Network Operator) · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 — Deployment Guide with Kubernetes · fetched 2026-09-07
- DPF v26.4.1 — Architecture (section index) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.