Skip to content
nodeza.learn
DOCA course/5. Services/Lesson 5.4

Storage offload: SNAP, Virtio, DevEmu, STA

S5·E4The drive the host cannot find · The staging lab beside Hall 2, the Friday of pilot week, twelve days to move-in

S5·E4Analyze~40 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs (SNAP-4, Virtio-net, DevEmu, STA), 2026-09-06

Builds on: BlueField-3 under the hood, The MFT toolbox: read-only vs mutating, DMS: one interface for every device

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain how SNAP-4 presents networked storage to the host as local NVMe or virtio-blk PCIe devices and which firmware keys make that possible.
  • Sequence the mlxconfig, hugepage, static-pod and snap_rpc.py/spdk_rpc.py steps that bring an emulated NVMe controller online.
  • Compare SNAP-4, the Virtio-net service, the DevEmu library and STA by what each emulates and where it runs.
  • Analyze a customer storage design for the documented restrictions (hotplug, SR-IOV, PF_BAR2, provider choice) before committing.

Episode 4 — The drive the host cannot find

The situation · The staging lab beside Hall 2, the Friday of pilot week, twelve days to move-in

The pilot exit criteria say the host boots and runs from what it believes is a local NVMe drive while the data lives on the tenant’s NVMe-oF array. The firmware keys went in on Tuesday, the card took its power cycle, the SNAP container is running, and lsblk on the host shows nothing at all. The tenant’s storage engineer flies home tonight. The network lead is standing behind you with the notebook that is now three pages of open items, not saying anything, which is worse than if she were. The Dell SE already demoed this on a slide.

SNAP exists for exactly the promise on that exit-criteria sheet. A customer who disaggregates storage does not want initiator software, fabric drivers or fabric knowledge in every host image, so the card emulates the drive: the host sees an ordinary NVMe or virtio-blk PCIe device while the data travels over the network.[2] Nothing in the host has to change, which is the whole point, and it is also why this failure is invisible from the host side.

The order of the RPCs is the thing. The SPDK backend is attached first, then the emulated subsystem and namespace, then a controller created suspended, the namespace attached to it, and only then a resume that lets the host see a complete device.[4] A device the host cannot see is usually a controller nobody resumed. Start with what is being emulated, and where.

1The idea: emulate the drive, move the data elsewhere

The SNAP family lets BlueField “present networked storage as local block or file system devices to the host, emulating local drives on the PCIe bus”. Three products: SNAP Virtio-fs (file-system emulation on BlueField-3, Beta in 3.5.0), SNAP-4 (“NVMe and virtio-block for block device emulation on BlueField-3”), and SNAP-3 for BlueField-2.[1][12] A commercial detail that ends many pre-sales questions: “BlueField-3 and BlueField-4 DPU does not have license requirements to run BlueField SNAP”.[2]

SNAP-4 offers two data-path providers, a DPA provider (default for NVMe and virtio-blk) that runs the IO path on the datapath accelerator, and a DPU provider (hybrid, virtio-blk); it ships as an NGC container or as a package with service code plus library headers.[2] The backend is SPDK: NVMe over RDMA or TCP, malloc, null, AIO, or any other SPDK bdev plugin.[4]

Ships in DPU mode (Arm owns the NIC — ECPF)
B3240 · B3220 · B3210 · B3210E · Needs the supplementary 8-pin ATX power connector. Dell R7725 ISM lists B3220/B3240 at 150 W.
PCIe Gen5 x16 → host32 lanes total · switch bifurcation up …16× Arm A78 coresArmv8.2+ Hercules · 560/2133 MHz (B…32 GB DDR5dual controllers · 5600 MT/s (B3220…40 GB eMMC pSLC + 128 GB SSDon every SKU · boot/OS storageBMC (integrated, all models)Redfish · IPMI 2.0 · NC-SI · owns r…1GbE OOBRJ45 "MGMT"rshim / tmfifoPCIe fn .2 or …ConnectX-7-generation NIC core + e-switchup to 400 Gb/s Ethernet or NDR IB · ASA…DPA — 16 cores / 256 threadsdatapath accelerator …2× QSFP112 — 400G / 200G / 100G2× QSFP112 — 400G/NDR (B3240) · 200G/ND…Crypto — IPsec / TLS / MACsec inlineAES-XTS 256/512 · PKA · TRNGCompression enginedatasheet: "Decompression engine"Erasure coding / RAID"Erasure coding for RAID implementa…SHA enginehashing / integrityRegEx (datasheet only)no DOCA library in 3.xArm subsystem · memory · managementNetwork coreHardware accelerators
Storage overlay — SNAP-4 / DevEmu / NVMe-oF

The host sees an NVMe or virtio-blk PF on PCIe (DevEmu-emulated; mlxconfig NVME_EMULATION_ENABLE=1). SNAP-4 on the Arm (SPDK) attaches a backend — spdk_rpc.py bdev_nvme_attach_controller -t rdma|tcp — and creates the controller with snap_rpc.py. The default "DPA provider" runs the data path on the DPA; queues live in DDR; NVMe-oF exits through the NIC core and uplinks. No license needed on BlueField-3.

source

Select a block. Every block explains what it is, which DOCA library touches it, and one thing an FAE gets asked.

Facts: BlueField-3 user guide (introduction, specifications), DOCA 3.5.0 libraries table, DPA subsystem, BMC 26.04 docs, Dell KB 000227031 / R7725 ISM.

Overlay storage: select the PCIe emulation block, the DPA and the network core to see how an emulated NVMe PF on the host side maps to an SPDK bdev and a RoCE or TCP flow on the wire side.

2Firmware, hugepages and the SNAP-4 pod

Emulation is a firmware capability, so it starts with mlxconfig on the BlueField-3 (mt41692_pciconf0), followed by a power cycle:[3]

mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_MODEL=1 PF_BAR2_ENABLE=0
mlxconfig -d /dev/mst/mt41692_pciconf0 s VIRTIO_BLK_EMULATION_ENABLE=1 VIRTIO_BLK_EMULATION_NUM_PF=1
mlxconfig -d /dev/mst/mt41692_pciconf0 s NVME_EMULATION_ENABLE=1 NVME_EMULATION_NUM_PF=1
# SR-IOV, 512 NVMe VFs:
mlxconfig -d /dev/mst/mt41692_pciconf0 s NVME_EMULATION_ENABLE=1 NVME_EMULATION_NUM_VF=256 NVME_EMULATION_NUM_PF=2 NVME_EMULATION_NUM_MSIX=2 NVME_EMULATION_NUM_VF_MSIX=2 NVME_EMULATION_MAX_QUEUE_DEPTH=12
# hotplug:
mlxconfig -d /dev/mst/mt41692_pciconf0 s PCI_SWITCH_EMULATION_ENABLE=1 PCI_SWITCH_EMULATION_NUM_PORT=32

The container needs memory: doca-hugepages config --app snap --size 2048 --num 2048; doca-hugepages reload allocates 4 GiB of hugepages.[3] Deployment is the static-pod pattern: cp doca_snap.yaml /etc/kubelet.d/, with environment variables SPDK_RPC_INIT_CONF: "/etc/nvda_snap/spdk_rpc_init.conf", SNAP_RPC_INIT_CONF: "/etc/nvda_snap/snap_rpc_init.conf" and APP_ARGS: "-m 0xffff". Success is crictl logs <id> showing “SNAP Service running successfully” and spdk_rpc.py spdk_get_version answering.[3] The image tag was not captured on the fetched deployment page;[3] the DPF reference sets SNAP_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_vfs for its SNAP Virtio-fs service.[14]

Restrictions, verbatim: “Hotplug is not guaranteed to work on AMD machines”; “Enabling PCI_SWITCH_EMULATION_ENABLE could potentially impact SR-IOV capabilities on Intel and AMD machines”; “hotplug PFs do not support SR-IOV”; and on BlueField-4 “GGA architecture is no longer supported”.[3]

A production fleet uses the same keys. The DPF hbn-ovnk-snap-nvme flavor carries PF_BAR2_ENABLE=0 … INTERNAL_CPU_MODEL=1 INTERNAL_CPU_OFFLOAD_ENGINE=0 SRIOV_EN=1 NUM_OF_VFS=46 … NVME_EMULATION_ENABLE=1 NVME_EMULATION_NUM_PF=1 NVME_EMULATION_NUM_VF=125 NVME_EMULATION_NUM_MSIX=2 in its nvconfig, plus hugepagesz=2048kB hugepages=2298 in GRUB.[11]

3The RPC layer: SPDK backends, SNAP front ends

Two RPC tools divide the work. spdk_rpc.py builds the backend block device; snap_rpc.py builds the emulated subsystem, namespace and controller the host sees.[4] Useful helpers: snap_rpc.py emulation_function_list (which emulated PFs and VFs exist), snap_rpc.py bdev_list, snap_rpc.py snap_log_level_set 3 (0 Critical to 5 Trace), snap_rpc.py snap_global_param_list, and for VF resources nvme_controller_vfs_msix_reclaim --ctrl NVMeCtrl1 or virtio_blk_controller_vfs_dynamic_resources_reclaim.[4]

Worked → faded → problem: a remote NVMe-oF namespace as a local NVMe drive

Ask. Expose namespace 1 of a remote NVMe-oF RDMA target at 1.1.1.1:4420 (NQN nqn.2022-10.io.nvda.nvme:swx-storage) to the host as an NVMe drive on emulated PF 0.

  1. Firmware and pod are already in place (Segment 2). Confirm the function exists: snap_rpc.py emulation_function_list.[4]
  2. Backend: spdk_rpc.py bdev_nvme_attach_controller -b nvme0 -t rdma -a 1.1.1.1 -f ipv4 -s 4420 -n nqn.2022-10.io.nvda.nvme:swx-storage (use -t tcp for NVMe/TCP). This yields bdev nvme0n1.[4]
  3. Front-end subsystem: snap_rpc.py nvme_subsystem_create --nqn nqn.2022-10.io.nvda.nvme:0 (optional --serial_number, --model_number, --nn 0xFFFFFFFE, --mnan 1024).[4]
  4. Namespace: snap_rpc.py nvme_namespace_create -b nvme0n1 -n 1 --nqn nqn.2022-10.io.nvda.nvme:0 --uuid 263826ad-19a3-4feb-bc25-4bc81ee7749e.[4]
  5. Controller, created suspended so the host does not see a half-built device: snap_rpc.py nvme_controller_create --nqn nqn.2022-10.io.nvda.nvme:0 --ctrl NVMeCtrl1 --pf_id 0 --vf_id 0 --num_queues 8 --mdts 7 --quirks 0 --suspended.[4]
  6. Attach and go live: snap_rpc.py nvme_controller_attach_ns -c NVMeCtrl1 -n 1 then snap_rpc.py nvme_controller_resume -c NVMeCtrl1.[4]
  7. Verify: snap_rpc.py nvme_controller_list --ctrl NVMeCtrl1 on the DPU; on the host a new nvme device appears in lsblk.[4]

Reasoning: backend before front end; suspended create plus resume is the documented way to make the device appear complete in one step.

For virtio-blk the equivalent create line is snap_rpc.py virtio_blk_controller_create --pf_id 0 --vf_id 0 --bdev nvme0n1 --num_queues 1 --queue_depth 256 --force_in_order 0 --admin_q 0 (or address the function by --vuid MT2114X12200VBLKS1D0F0), with virtio_blk_controller_list and virtio_blk_controller_destroy -c VblkCtrl1 [--force].[4] Hotplug is a two-step flow: virtio_blk_function_create or nvme_function_create, then virtio_blk_controller_hotplug -c VblkCtrl1 --wait_for_done or nvme_controller_hotplug -c NVMeCtrl1 --wait_for_done; the legacy emulation_device_detach_prepare/detach pair is deprecated. Other backends: spdk_rpc.py bdev_malloc_create 64 512, bdev_null_create null1 64 512, bdev_aio_create /path/to/file blockdev_name.[4]

4Virtio-net: the same trick for network devices

The Virtio-net service (GA) makes “virtio-net PCIe devices offload both data and control plane to the BlueField networking device” without QEMU. It runs as the systemd unit virtio-net-controller with processes virtio_net_emu and virtio_net_ha.[5][12] Firmware keys: VIRTIO_NET_EMULATION_ENABLE=TRUE, VIRTIO_NET_EMULATION_NUM_PF, VIRTIO_NET_EMULATION_NUM_VF, VIRTIO_NET_EMULATION_NUM_MSIX (minimum 4), and PCI_SWITCH_EMULATION_ENABLE for hotplug.[5]

Operations use virtnet list|hotplug|unplug|query|modify, configured by /opt/mellanox/mlnx_virtnet/virtnet.conf (JSON: ib_dev_p0/ib_dev_p1, static_pf with mac_base, features, mtu, is_lag, single_port).[5] Each emulated device is backed by an SF whose representor you attach to OVS: ovs-vsctl add-port <bridge> <sf_rep_net_device>; ip link set dev <sf_rep_net_device> up. SF numbering is fixed: static PFs 1000 to 1999, hotplug PFs 2000 to 2999, VFs from 3000. Host-side VFs are created the usual way, echo 1 > /sys/bus/pci/drivers/virtio-pci/<bdf>/sriov_numvfs.[5] Scale: 31 hotplug PFs, 31 static PFs, 1008 VFs in total; features include multi-queue, TSO and checksum, dynamic MSI-X, live migration through admin commands, and host-aware attention-button hotplug.[5]

5DevEmu, STA and the B3220SH

DevEmu is the library underneath the emulation services. It lets you “implement a hardware PCIe function using software, such that the host is not aware that the PCIe function is emulated”, covering “Discovery, configuration, hot plugging/unplugging, management, and IO path handling”, with sub-libraries DOCA DevEmu PCI and DevEmu Virtio. Two limits: “Virtual Functions (VFs) unsupported” and backward compatibility is not guaranteed.[6] The 3.5.0 samples tree lists devemu_pci_device_db, _dma, _hotplug, _list, _msix, _stateful_region, _tlp_bridge_handler, _tlp_handler, devemu_vfs_device_hotplug_unplug, devemu_vfs_list_devices, the helper devemu_pci_vfio_bind.py and build_dpacc_samples.sh; the exact API names were not captured on the landing page.[10][6]

STA (Alpha) works on the other side of the wire: it “simplifies integration and offloading of storage target applications, such as SPDK, onto NVIDIA BlueField-3 and newer”, serving NVMe-oF over “RoCEv1/IB RDMA and NVMeTCP” with DPA-based acceleration. Constraints: “When running on the BlueField platform, NVMe drives must be directly PCIe-attached”; a host-side target needs a P2P-patched kernel (three patches, manual build) or the P2P BFB; Ubuntu 22.04 and DOCA 2.10.0 or later.[7] The API surface starts with doca_sta_cap_is_supported and doca_sta_add_dev, then queue pairs (doca_sta_io_qp_alloc/accept/connect) and tasks such as doca_sta_io_task_non_offload_rdma_read_alloc_init, …_rdma_write_send_alloc_init, …_rdma_send_alloc_init, doca_sta_io_task_disconnect_alloc_init, doca_sta_subsystem_task_rm_ns_alloc_init and doca_sta_be_destroy_queue_task_alloc_init; one STA context per system or VM, and QP operations stay on the creating thread.[7]

The B3220SH Storage Controller SKU appears in the 3.5.0 change log as “Support for B3220SH Storage Controller” for OVS-DOCA, and HBN 3.5.0 “Added support for B3220-SH Storage Controller”. SNAP-specific notes for that SKU were not on the fetched SNAP pages; when a customer quotes it, confirm the SNAP-4 release notes before sizing.[8][9]

Component What it emulates or accelerates Where 3.5.0 level
SNAP-4 NVMe and virtio-blk PFs/VFs toward the host Service on Arm plus DPA (see SNAP guides)[2]
SNAP Virtio-fs virtio-fs file system toward the host Service Beta[12]
Virtio-net virtio-net PFs/VFs toward the host systemd service GA[12]
DevEmu any PCIe function, your code Library GA in the libraries index[15], alpha-level wording on the page[6]
STA NVMe-oF target offload toward the network Library on Arm plus DPA Alpha[7]

End of Episode 4 — Ten minutes before the airport

How it ended

The controller had been created and never resumed, so the host was looking at an incomplete device; the resume brings the namespace up and lsblk finally lists a drive that is not in the chassis.[4] You hand the storage engineer the two things he needs on the plane: the mlxconfig line with its power cycle and the hugepage allocation, and the documented hotplug and SR-IOV limits to check against his server model.[3] One promise you do not make. Monday is the tenant’s security review, they want Argus watching this same card, and PF_BAR2 is set opposite ways by the two guides. That is a lab test, not a yes.[3][13]

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Pre-flight on the Dell-lab BlueField-3 (read-only):

  1. Host: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_MODEL PF_BAR2_ENABLE NVME_EMULATION_ENABLE NVME_EMULATION_NUM_PF VIRTIO_BLK_EMULATION_ENABLE PCI_SWITCH_EMULATION_ENABLE. Record every value; this is your rollback baseline.[3] DPU: sudo crictl pods | grep -i snap and ls /etc/kubelet.d/. If a SNAP pod already runs, snap_rpc.py emulation_function_list, spdk_rpc.py spdk_get_version and snap_rpc.py snap_global_param_list are safe reads.[3][4]

Mutating steps (each requires a power cycle or changes what the host sees; skip them if the card is shared with the Argus lab in the next lesson, because PF_BAR2_ENABLE conflicts):

  1. Enable NVMe emulation. sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_MODEL=1 PF_BAR2_ENABLE=0 then sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s NVME_EMULATION_ENABLE=1 NVME_EMULATION_NUM_PF=1, then power-cycle the server. Expected: after boot, lspci | grep -i "non-volatile" on the host shows one more NVMe function. Rollback: set NVME_EMULATION_ENABLE=0 and restore the baseline values from step 1, then power-cycle.[3]

  2. Hugepages and pod. On the DPU: sudo doca-hugepages config --app snap --size 2048 --num 2048 && sudo doca-hugepages reload, then sudo cp doca_snap.yaml /etc/kubelet.d/. Expected: sudo crictl logs $(crictl ps | grep snap | awk '{print $1}') contains “SNAP Service running successfully”; spdk_rpc.py spdk_get_version answers. If not: sudo journalctl -u kubelet --since -5m; check the hugepage count matches the YAML request. Rollback: sudo rm /etc/kubelet.d/doca_snap.yaml.[3]

  3. A local-only device. With no remote target in the lab, use a malloc backend: spdk_rpc.py bdev_malloc_create 64 512 (returns a bdev name, typically Malloc0), then the subsystem, namespace (-b Malloc0), suspended controller, attach and resume lines from the Worked example. Expected: snap_rpc.py nvme_controller_list shows NVMeCtrl1; on the host lsblk and nvme list show a 64 MiB NVMe device. Rollback: remove the pod YAML (step 3), which withdraws the emulated controller; then step 2’s rollback.[4]

  4. Write-test carefully. Identify the device first: lsblk -o NAME,SIZE,MODEL and nvme list — the target must be the 64 MiB device backed by Malloc0, never a BOSS pair or a data disk. Then, on the host, sudo dd if=/dev/zero of=/dev/nvmeXn1 bs=1M count=8 oflag=direct on that device only. Expected: 8 MiB written; the DPU’s snap_rpc.py nvme_controller_list counters move. If not: check the controller was resumed. Rollback: the data lives only in the malloc bdev, so step 6’s rollback of step 4 (removing the pod YAML) discards it and nothing on host storage is touched — but a write to the wrong device has no rollback, which is why the lsblk/nvme list check is not optional.[4]

  5. Restore. Roll back steps 4, 3 and 2 in that order and re-run the step 1 query to prove the baseline is back.

Retrieval check

11 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, what SNAP does for a customer with a disaggregated NVMe-oF storage array and why the host OS team likes it.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs (SNAP-4, Virtio-net, DevEmu, STA), 2026-09-06. Dates are when each page was fetched.

  1. DOCA SNAP Services (landing) · fetched 2026-09-06 · DOCA 3.5.0
  2. DOCA SNAP-4 Service Guide (landing) · fetched 2026-09-06 · DOCA 3.5.0
  3. SNAP-4 Service Deployment · fetched 2026-09-06 · DOCA 3.5.0
  4. SNAP-4 Service RPC Commands · fetched 2026-09-06 · DOCA 3.5.0
  5. DOCA Virtio-net Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA Device Emulation (landing) · fetched 2026-09-06 · DOCA 3.5.0
  7. DOCA STA · fetched 2026-09-06 · DOCA 3.5.0
  8. Changes and New Features (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  9. HBN Service Release Notes (HBN 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  10. doca-samples 3.5.0 full git tree (API) · fetched 2026-09-06 · DOCA 3.5.0
  11. doca-platform v26.4.1: dpuflavor.yaml (production nvconfig with NVMe emulation) · fetched 2026-09-06
  12. DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  13. DOCA Argus Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  14. doca-platform v26.4.1: hbn-ovnk-snap use case README · fetched 2026-09-06
  15. DOCA Libraries (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.