Skip to content
nodeza.learn
DOCA course/6. Acceleration/Lesson 6.3

DPA, Flex IO, and PCC

S6·E302:40, the kernel that would not bind · The archive's data center, Austin, a maintenance window that ends at 05:00

S6·E3Analyze~50 minsources checked todayverified against DOCA 3.5.0 docs and doca-samples tag 3.5.0, 2026-09-06

Builds on: DMA, RDMA, and Verbs, BlueField-3 under the hood

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Describe the DPA as hardware (cores, threads, execution units, partitions) and as two software layers (DOCA DPA vs Flex IO) with their 3.5.0 quality levels.
  • Read a DPA kernel and its host launcher and identify `__dpa_global__`, `__dpa_rpc__`, sync events and the exact DPACC build line that binds them.
  • Analyze where a DPA program can fail silently (coherency, stack, thread limits, EU partitions) and name the tool that exposes each.
  • Explain how DPA Comms, DPA Verbs and EU-group affinity in 3.5.0 change what runs on the data path, and where ConnectX-8 fits.
  • Position PCC as a DPA application: RP vs NP, prerequisites, and the mlxconfig knobs that must be set.

Episode 3 — 02:40, the kernel that would not bind

The situation · The archive's data center, Austin, a maintenance window that ends at 05:00

The night operator has already labeled the two spare cables RAIL-A and RAIL-B with the label maker he carries everywhere, and he is the only person here who knows which riser they land in. The archive wants its own RTT-based congestion-control algorithm loaded before the morning ingest test; their developer has been fighting the toolchain since midnight. The kernel compiles. The host program then dies in doca_dpa_set_app. You ask for the dpacc line and the extern declaration, put them side by side, and there it is: the --app-name token and the struct doca_dpa_app symbol the host passes must be identical, and theirs differ by one underscore.[4] The next failure is on a non-ECPF function with no execution-unit partition, which is dpaeumgmt work rather than code.[1]

While the rebuild runs you answer the question nobody asked: why this processor exists at all. Congestion control must react within a round trip, in microseconds, to notifications arriving at the NIC. The DPA is an embedded subsystem placed beside the NIC engines for exactly that kind of work, and the datasheet counts 16 cores and 256 threads in it.[9][10] So the rate is adjusted where the events land instead of a wake-up away on the host. That is what DOCA PCC is: a DPA application, Beta in 3.5.0.[7][8]

Code that runs beside the NIC is bound by name: one underscore is an outage.

Segment 1 starts with the hardware and its two software layers.

1The DPA as hardware and as two software layers

The BlueField-3 datasheet lists a “Programmable Datapath Accelerator: 16 cores, 256 threads; Programmability through DOCA”.[10] NVIDIA’s DPA subsystem page describes it as “an embedded subsystem designed to accelerate workloads that require high-performance access to the NIC engines”, where “Each thread can be mapped to a different execution unit”, each execution unit has a private L1 data cache, the L1 code cache is shared per DPA core and the L2 is shared by all cores, with memory backed by NIC-private memory in the DPU’s DDR.[9] The DOCA DPA guide turns the thread count into a rule: “DPA threads are an actual hardware resource and are, therefore, limited in number to 256 (including internal allocations and allocations explicitly requested by the user as part of the kernel launch API)”, and “DOCA DPA does not check these limits”.[1]

Execution units are grouped into partitions; running an application on a non-ECPF function without a partition fails, and running from the host requires EU pre-configuration with the dpaeumgmt tool.[1] Supported devices in the 3.5.0 DPA Development guide are “BlueField-3 based DPUs” and “ConnectX-8 based NICs”, with the note that only ECPF/PF may load DPA applications.[2] The sample build infrastructure already knows more targets: the applications meson_options.txt default for dpacc_mcpu is nv-dpa-bf3,nv-dpa-cx7,nv-dpa-cx8,nv-dpa-cx9,nv-dpa-cx10.[11] Quote the guide for support and the meson string for what the compiler accepts; they are not the same claim.

Ships in DPU mode (Arm owns the NIC — ECPF)
B3240 · B3220 · B3210 · B3210E · Needs the supplementary 8-pin ATX power connector. Dell R7725 ISM lists B3220/B3240 at 150 W.
PCIe Gen5 x16 → host32 lanes total · switch bifurcation up …16× Arm A78 coresArmv8.2+ Hercules · 560/2133 MHz (B…32 GB DDR5dual controllers · 5600 MT/s (B3220…40 GB eMMC pSLC + 128 GB SSDon every SKU · boot/OS storageBMC (integrated, all models)Redfish · IPMI 2.0 · NC-SI · owns r…1GbE OOBRJ45 "MGMT"rshim / tmfifoPCIe fn .2 or …ConnectX-7-generation NIC core + e-switchup to 400 Gb/s Ethernet or NDR IB · ASA…DPA — 16 cores / 256 threadsdatapath accelerator …2× QSFP112 — 400G / 200G / 100G2× QSFP112 — 400G/NDR (B3240) · 200G/ND…Crypto — IPsec / TLS / MACsec inlineAES-XTS 256/512 · PKA · TRNGCompression enginedatasheet: "Decompression engine"Erasure coding / RAID"Erasure coding for RAID implementa…SHA enginehashing / integrityRegEx (datasheet only)no DOCA library in 3.xArm subsystem · memory · managementNetwork coreHardware accelerators
DPA overlay — datapath accelerator to NIC

16 cores / 256 threads sit next to the NIC engines. Memory is NIC private memory in DDR. A host or Arm DOCA app compiles kernels with dpacc (DOCA DPA, Beta) or Flex IO (GA) and launches them through a DPA context; DOCA PCC (congestion control) and SNAP-4 are the shipped users.

source

Select a block. Every block explains what it is, which DOCA library touches it, and one thing an FAE gets asked.

Facts: BlueField-3 user guide (introduction, specifications), DOCA 3.5.0 libraries table, DPA subsystem, BMC 26.04 docs, Dell KB 000227031 / R7725 ISM.

DPA overlay: select the DPA block, its execution units and the NIC engines it sits beside; then select the Arm complex and compare the path a completion takes to each.

Two software layers exist. DOCA DPA offers “high-level primitives for DPA work offloading, synchronization, and communication”; Flex IO is the “low-level event-driven library”, and “The implementation of DOCA DPA is based on the Flex IO API”.[2] In the 3.5.0 libraries index DOCA DPA is Beta, Flex IO is GA, and the DPA Development guide’s supported-SDK line reads “DOCA Flex IO at GA level” — Flex IO is not deprecated in 3.5.0.[8][2] Flex IO APIs are, however, marked experimental: host code needs -DFLEXIO_ALLOW_EXPERIMENTAL_API and device code -DFLEXIO_DEV_ALLOW_EXPERIMENTAL_API, and even the DOCA DPA sample build passes both.[2][4]

2Kernels, RPCs, threads and the memory model

DPA device code is C with two annotations: __dpa_global__ before a kernel or event-handler function and __dpa_rpc__ before an RPC function, which is “a blocking one-time call from the host application to execute a kernel on DPA”.[1][2] The smallest complete kernel in the tree is the whole of dpa_kernel_launch_kernels_dev.c: #include <doca_dpa_dev.h> and __dpa_global__ void hello_world(void) { DOCA_DPA_DEV_LOG_INFO("Hello from kernel\n"); }.[5] The RPC form is __dpa_rpc__ uint64_t hello_rpc(int arg) on the device and extern doca_dpa_func_t hello_rpc; doca_dpa_rpc(dpa, &hello_rpc, &retval, 10); on the host.[1]

The host side of kernel_launch is small and shows the synchronization idiom: declare extern doca_dpa_func_t hello_world;, create a wait sync event and a completion sync event, call doca_dpa_kernel_launch_update_set(dpa_ctx, wait_event, wait_thresh, comp_event, comp_event_val, num_dpa_threads, &hello_world) with num_dpa_threads = 1, let a helper thread doca_sync_event_update_set(wait_event, 5) after five seconds, and block on doca_sync_event_wait_gt(comp_event, comp_event_val - 1, SYNC_EVENT_MASK_FFS).[6] Beyond one-shot kernels, DOCA DPA has persistent threads (doca_dpa_thread_create/set_func_arg/start/run), completion contexts that wake a thread on NIC completions (doca_dpa_completion_create/set_thread), notification completions for thread-to-thread wakeups, async ops, thread groups, memory (doca_dpa_mem_alloc, doca_dpa_h2d_memcpy, doca_dpa_d2h_memcpy) and hash tables; device-side threads call doca_dpa_dev_get_completion, _completion_ack, _completion_request_notification and end with doca_dpa_dev_thread_reschedule or _thread_finish.[1][16]

The memory model is where silent bugs live. Address spaces are __DPA_HEAP, __DPA_MEMORY, __DPA_MMIO and __DPA_SYSTEM; the coherency table says a DPA-thread producer to a NIC-engine observer is “Not coherent” — “Data to be read by the NIC must be written back using the appropriate intrinsic” (__dpa_thread_memory_writeback() or __dpa_thread_fence(space, pred, succ)), while NIC to DPA thread is coherent.[2] The user stack is “slightly less than 8 KB (8184 bytes)” and overrunning it is the program’s problem.[2] None of these is checked at run time, which is why the tools in the next segment exist.

3Building with DPACC and debugging what you built

The 3.5.0 sample script build_dpacc_samples.sh is the canonical build line. It first converts an attributes YAML into a blob, then invokes the compiler:[4]

/opt/mellanox/doca/tools/dpa-app-attributes2blob <sample>_attributes.yaml <sample>_attributes.blob
/opt/mellanox/doca/tools/dpacc <device_srcs> -o <out>.a -mcpu=<nv-dpa-bf3|nv-dpa-cx8> -hostcc=gcc \
  -hostcc-options="-Wno-deprecated-declarations -Werror -Wall -Wextra -DFLEXIO_ALLOW_EXPERIMENTAL_API" \
  --devicecc-options="-Wno-deprecated-declarations -Werror -Wall -Wextra -DFLEXIO_DEV_ALLOW_EXPERIMENTAL_API -O2" \
  --app-name="dpa_sample_app" \
  -device-libs="-L<doca_lib> -ldoca_dpa_dev -ldoca_dpa_dev_comm -ldoca_dpa_dev_verbs" \
  -flto -I/opt/mellanox/doca/include --dpa-proc-attr="<sample>_attributes.blob"

The script’s own comment states the binding rule: the --app-name token “must be identical to the struct doca_dpa_app parameter passed to doca_dpa_set_app()”.[4] The output .a is a host library that embeds the device ELF; the kernel_launch attributes file is essentially empty (“This sample requires no attributes”).[2][4] The -device-libs line is also the answer to “what are DPA Comms and DPA Verbs”: separate device libraries, libdoca_dpa_dev_comm for RDMA and inter-DPA messaging from kernels and libdoca_dpa_dev_verbs for the Verbs datapath on the DPA.[4][1]

Debugging tools, all documented in the DPA guides: dpaeumgmt manages EU partitions; the dpa-gdbserver package provides host and device binaries used as a gdbserver for DPA applications; for core dumps, open the device-side ELF in gdb-multiarch after recovering it with dpacc-extract; Flex IO’s dTrace captures code flow, and flexio_msg_stream provides device printf-style output.[2][1] The DOCA DPA guide additionally exposes doca_dpa_dev_log (DOCA_DPA_DEV_LOG_INFO), doca_dpa_dev_trace/_flush and doca_dpa_peek_at_last_error for host-side error retrieval.[1] DPA application authentication (code signing of the device image) exists for BlueField-3 and is delivered through firmware tooling; treat its exact key and certificate requirements as something to confirm on the live page before quoting.[2]

Worked → faded → problem: from a kernel source to a running DPA kernel
  1. Device file hello_dev.c: #include <doca_dpa_dev.h> then __dpa_global__ void hello_world(void) { DOCA_DPA_DEV_LOG_INFO("Hello from kernel\n"); }.[5]
  2. Attributes: an almost-empty YAML with Feature_list: and a comment, converted with dpa-app-attributes2blob hello_attributes.yaml hello_attributes.blob.[4]
  3. Compile for BlueField-3: the dpacc line above with -mcpu=nv-dpa-bf3, --app-name="dpa_sample_app", output hello_dev.a.[4]
  4. Host: extern struct doca_dpa_app *dpa_sample_app; extern doca_dpa_func_t hello_world; → doca_dpa_create(dev, &dpa) → doca_dpa_set_app(dpa, dpa_sample_app) → doca_dpa_start(dpa); then the two sync events and doca_dpa_kernel_launch_update_set(dpa, wait_ev, 4, comp_ev, 10, 1, &hello_world); raise the wait event with doca_sync_event_update_set(wait_ev, 5); block on doca_sync_event_wait_gt(comp_ev, 9, SYNC_EVENT_MASK_FFS).[6][16]
  5. Link the host program against hello_dev.a and doca-dpa; run on the Arm side (PF) or on the host after dpaeumgmt has provisioned EUs.[1]
  6. Expected: “Hello from kernel” in the DPA log stream; the host returns after the completion event passes 9.[3]
  7. If the host fails in doca_dpa_set_app, the --app-name token and the extern symbol differ; if it fails at start on a non-ECPF function, no EU partition exists.[4][1]

4DPA on the data path: RDMA, Verbs, and EU-group affinity in 3.5.0

Once kernels exist, the interesting question is what wakes them. DOCA RDMA can hand its datapath to the DPA: obtain the context with doca_rdma_as_ctx(), call doca_ctx_set_datapath_on_dpa(), start the context and use doca_rdma_get_dpa_handle() from kernels via doca_dpa_dev_rdma_post_send/receive.[14][1] The DPA samples README fixes the device rule for the DPU: “the DOCA DPA context is created on the PF device, an Extended DPA context is created on the SF device, and the DOCA RDMA context is created on the SF device”, and all DPA resources tied to that RDMA instance must come from the extended context (doca_dpa_device_extend in dpa_common.c).[3][16] The dpa_basic_initiator_target, dpa_initiator_target and dpa_ping_pong samples all follow the pattern: attach a DOCA RDMA to a DPA completion context, attach that to a DPA thread, use an RPC to post the first operations, and let receive completions wake the thread until a sync event releases the host.[3]

DOCA Verbs offers the same handoff at a lower level: doca_verbs_qp_init_attr_set_external_datapath_en() lets doca_dpa own the QP, and the Verbs samples verbs_receive_packets_on_dpa and verbs_send_ethernet_frames_on_dpa run Ethernet queues from DPA kernels; dpa_verbs_initiator_target exchanges buffer addresses, MKEYs, QP numbers and GIDs over TCP, then runs the RDMA data path entirely in DPA threads with a DPA completion context instead of a CQ.[13][17][3]

3.5.0 adds placement control. The guide’s changes section lists “DPA EU group affinity APIs: doca_dpa_eu_group_affinity_create(), _destroy(), _set(), _clear(), _get()”, “DPA thread group-affinity APIs” (doca_dpa_thread_set_group_affinity) and resource queries doca_dpa_resources_get_num_eu_groups() / doca_dpa_resources_get_eu_groups(); thread affinity “now supports relaxed, fixed (single EU), and group (EU group) modes”, with relaxed the default, fixed via doca_dpa_thread_set_affinity(), and validation resources that must be set before doca_dpa_start().[1] The 3.5.0 changes page frames ConnectX-8 support and the nv-dpa-cx8 target as the other half of this release for DPA.[12][4] DOCA Bench has no doca_dpa pipeline step, so DPA performance is measured through the application it accelerates (RDMA, Ethernet), not directly.[15]

5PCC: congestion control as a DPA application

DOCA PCC (Beta in 3.5.0) is programmable congestion control that runs on the DPA.[8][7] Two roles: the reaction point (RP) “Monitors network conditions actively, dynamically adjusting data transmission rates”, and the notification point (NP) “Passively receives congestion notifications … processing them intelligently”; reference algorithms are RTT-based Zero Touch RoCE and DCQCN.[7] Prerequisites: BlueField-3 or later, “supported only for the ETHERNET link type”, firmware 32.38.1000 or higher, and two mlxconfig switches — RP: mlxconfig -d <mlx_device> -y s USER_PROGRAMMABLE_CC=1; NP: mlxconfig -d <mlx_device> -y s PCC_INT_EN=0 — each followed by a graceful shutdown and power cycle.[7]

Host-side lifecycle: doca_pcc_create (or doca_pcc_np_create), doca_pcc_set_app, doca_pcc_set_thread_affinity, doca_pcc_start, then doca_pcc_get_process_state — the first process becomes ACTIVE and later ones STANDBY and auto-promote, which is the high-availability model — plus a mailbox (doca_pcc_set_mailbox, doca_pcc_mailbox_send) for host-to-device parameters.[7] Device-side entry points are doca_pcc_dev_user_init, doca_pcc_dev_user_algo (the primary entry point), doca_pcc_dev_user_set_algo_params, and for NP doca_pcc_dev_np_user_packet_handler; helpers include doca_pcc_dev_default_internal_algo, doca_pcc_dev_init_algo_slot, doca_pcc_dev_get_rtt_req_recv_timestamp, doca_pcc_dev_histogram_update and doca_pcc_dev_nic_counters_sample.[7] In 3.5.0 the NP side gains doca_pcc_np_set_ts_source() (with doca_pcc_np_cap_is_ts_source_supported()) to pick the hardware clock used for injected response timestamps, which requires flexio-sdk 25.10 or newer.[7] The build is the same DPACC flow as any DPA app, and the doca_pcc_counter tool reads the counters.[7]

04:20, before the window closes

How it ended

The token matches, dpaeumgmt has provisioned the partition, and their algorithm loads on the reaction point.[1][7] You leave one warning in writing: the moment a kernel writes descriptors the NIC will read, the coherency table applies and the writeback intrinsic is not optional.[2] At handover: “PCC is Beta in 3.5.0 — build it, measure it, and keep a reference algorithm configured as the thing you fall back to.” The operator prints one more label for the tray. On the drive out, the network lead forwards the imaging team’s plan: their CUDA kernels want the receive queues themselves, and the design review is in six hours.

Lab

Pre-flight (read-only): on the Arm side dpaeumgmt -h and, if permitted, the partition listing subcommand; mlxconfig -d /dev/mst/<dev> q USER_PROGRAMMABLE_CC PCC_INT_EN to record current PCC settings; flint -d /dev/mst/<dev> q | grep -i fw to confirm firmware is at or above 32.38.1000. Do not set anything.[1][7]

  1. Build and run dpa_kernel_launch on the Arm side (PF): cd /opt/mellanox/doca/samples/doca_dpa/dpa_kernel_launch && meson /tmp/b_kl && ninja -C /tmp/b_kl && /tmp/b_kl/doca_dpa_kernel_launch. Expected: “Hello from kernel” in the DPA log and a clean exit after the completion event. If it fails at doca_dpa_start, note the EU partition state from pre-flight.[3]
  2. Run dpa_basic_initiator_target passing the SF as the RDMA device (-rdma_dev <sf_name>). Expected: the target thread prints completion info and the host is released by the sync event. Record which device each context was created on.[3]
  3. Measure the effect indirectly: doca_bench --device <pci> --pipeline-steps doca_rdma::send --sweep core-count,1,4,*2 --csv-output-file /tmp/rdma_send_sweep.csv with the Arm companion (there is no doca_dpa step). Keep the CSV for M6.5.[15]
  4. Do NOT change USER_PROGRAMMABLE_CC or PCC_INT_EN in this lab; both require a power cycle and are out of scope for a read-only session. Record the values and the firmware version as the PCC readiness line for the customer report.[7]

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell platform engineer, in five sentences, what the DPA is, why a customer would put code there instead of on the Arm cores, and what the first three things are that go wrong.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs and doca-samples tag 3.5.0, 2026-09-06. Dates are when each page was fetched.

  1. DOCA DPA (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  2. DPA Development: DOCA DPA vs Flex IO (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  3. doca-samples 3.5.0: samples/doca_dpa/README.md · fetched 2026-09-06 · DOCA 3.5.0
  4. doca-samples 3.5.0: build_dpacc_samples.sh · fetched 2026-09-06 · DOCA 3.5.0
  5. doca-samples 3.5.0: dpa_kernel_launch_kernels_dev.c · fetched 2026-09-06 · DOCA 3.5.0
  6. doca-samples 3.5.0: dpa_kernel_launch_sample.c · fetched 2026-09-06 · DOCA 3.5.0
  7. DOCA PCC (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  8. DOCA Libraries index with quality levels (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  9. DPA Subsystem · fetched 2026-09-06 · DOCA 3.5.0
  10. NVIDIA BlueField-3 DPU datasheet (PDF) · fetched 2026-09-06
  11. doca-samples 3.5.0: applications/meson_options.txt · fetched 2026-09-06 · DOCA 3.5.0
  12. DOCA 3.5.0 Changes and New Features · fetched 2026-09-06 · DOCA 3.5.0
  13. DOCA RDMA Verbs (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  14. DOCA RDMA (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  15. DOCA Bench (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  16. doca-samples 3.5.0: dpa_common.c · fetched 2026-09-06 · DOCA 3.5.0
  17. doca-samples 3.5.0: samples/doca_verbs/README.md · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.