DMA, RDMA, and Verbs

1One object graph for every engine

Every DOCA acceleration library in this module — DMA, RDMA, Compress, SHA, Erasure Coding, AES-GCM — is driven through the same Core objects you met in M3: a doca_dev opened from doca_devinfo, one or more doca_mmap objects, a doca_buf_inventory, a progress engine doca_pe, and a library context reached through doca_<lib>_as_ctx().^[15] The DMA sequence is the reference shape: doca_dma_create(dev, &dma), doca_dma_as_ctx(dma), doca_dma_task_memcpy_set_conf(dma, ok_cb, err_cb, n), doca_pe_connect_ctx(pe, ctx), doca_ctx_start(ctx), then at run time doca_dma_task_memcpy_alloc_init(dma, src_buf, dst_buf, user_data, &task), doca_task_submit(doca_dma_task_memcpy_as_task(task)), and a doca_pe_progress(pe) loop until the completion callback fires.^[1][17]

Before you open a device, ask it what it can do. The DMA capability calls are doca_dma_cap_task_memcpy_is_supported(devinfo), doca_dma_cap_task_memcpy_get_max_buf_size(devinfo, &max), doca_dma_cap_task_memcpy_get_max_buf_list_len() and doca_dma_cap_get_max_num_tasks(); the samples accept --num-src-buf and --num-dst-buf so you can exercise linked-list buffers.^[1][17] The 3.5.0 sample flags are -p/--pci-addr, -t/--text, -d/--descriptor-path, -b/--buffer-path, -ns/--num-src-buf, -nd/--num-dst-buf, -nt/--num-tasks (local copy only) and -mnr/--max-num-range-profiling-displayed for NVTX ranges.^[17]

Quality levels matter when you quote this to a customer: in the 3.5.0 libraries index DMA and RDMA are GA, and the DMA guide lists no changes since 2.9.0.^[14][1] The 3.5.0 release adds, on the RDMA side of the house, the DOCA Multi-Rail API in DOCA Verbs “to support NIC resiliency” and replaces the host MR cache with generic Fast Registration Memory Region (FRMR) pools.^[13]

2Local vs remote DMA: the export/import handshake

The DMA guide draws three pictures: copying local memory to local memory, using the DPU to copy between host and DPU, and using the host to copy between host and DPU.^[1] For a local copy any device on the same BlueField works. For memory that is not local, “the DPU side of the application must select a device with an appropriate representor”, and the device must stay valid for as long as the DMA instance exists.^[1]

The shipped pair dma_copy_host and dma_copy_dpu shows the remote case, and the README is explicit that the host sample runs first and “it is user responsibility to transfer the two configuration files (descriptor and buffer) to the DPU”.^[4] On the host, dma_copy_host() does, in order: doca_mmap_set_permissions(src_mmap, DOCA_ACCESS_FLAG_PCI_READ_ONLY), doca_mmap_set_memrange(src_mmap, src_buffer, size), doca_mmap_start(src_mmap), doca_mmap_export_pci(src_mmap, dev, &export_desc, &export_desc_len), then writes the descriptor bytes to one file and the buffer address and length to a second file, and blocks on getchar() until you press Enter.^[2] The host does not create a DMA context at all; it only registers and exports.^[2]

On the DPU, dma_copy_dpu() connects and starts the DMA context first, reads doca_dma_cap_task_memcpy_get_max_buf_size() and rejects a remote buffer larger than that, allocates the destination, then doca_mmap_set_memrange/doca_mmap_start on the destination mmap and doca_mmap_create_from_export(NULL, export_desc, export_desc_len, dev, &remote_mmap) for the source.^[3] Two doca_buf objects are acquired (remote source, local destination), the task is allocated with doca_dma_task_memcpy_alloc_init, submitted, and doca_pe_progress is polled until the callback stores the status.^[3] Teardown order is the reverse: doca_buf_dec_refcount on both buffers, doca_mmap_destroy(remote_mmap), doca_ctx_stop, then the resource destroy.^[3] Note that this sample opens its device with -p/--pci-addr only, through open_doca_device_with_pci(); the representor rule is a property of which device you pick, not an extra argument.^[17][1]

The mirror-image API for network peers is doca_mmap_export_rdma(), consumed by the same doca_mmap_create_from_export(), which is how the RDMA read and write samples share memory across the fabric.^[15][7]

3DOCA RDMA: contexts, connections, tasks

A DOCA RDMA context is created and configured before start: doca_rdma_create(dev, &rdma), doca_rdma_as_ctx, doca_rdma_set_permissions(rdma, DOCA_ACCESS_FLAG_LOCAL_READ_WRITE | …), doca_rdma_set_gid_index(rdma, idx) (sample flag -g/--gid-index), doca_rdma_set_transport_type(rdma, DOCA_RDMA_TRANSPORT_TYPE_RC) or _DC, doca_rdma_set_max_num_connections, plus *_task_*_set_conf for each task type you will use.^[5][7] DC is real but “current support for this transport type is in alpha level only”, and the sample’s -tt/--transport-type help says it is only useful for single-connection out-of-band RDMA for now.^[5][7] Device rule: “On the host, any doca_dev is supported; On the BlueField Platform, applications must provide the library with SFs as a doca_dev”.^[5]

doca_rdma_task_send_set_conf(rdma, send_completed_cb, send_error_cb, num_tasks);
doca_rdma_task_receive_set_conf(rdma, recv_completed_cb, recv_error_cb, num_tasks);
doca_pe_connect_ctx(pe, doca_rdma_as_ctx(rdma));
doca_ctx_start(doca_rdma_as_ctx(rdma));

doca_rdma_export(rdma, &local_desc, &local_desc_len, &connection);
/* exchange descriptors out-of-band (file, TCP, gRPC), then: */
doca_rdma_connect(rdma, remote_desc, remote_desc_len, connection);

/* one-sided read/write: the peer runs doca_mmap_export_rdma(mmap, dev, &desc, &len);
   this side imports it: doca_mmap_create_from_export(NULL, desc, len, dev, &remote_mmap) */

Connection flow one, out-of-band: doca_rdma_export(rdma, &blob, &len, &connection) returns a descriptor and a connection object; you move the blob to the peer by any means (the samples write /tmp/local_connection_desc_path.txt and read the remote one), and the peer calls doca_rdma_connect(rdma, remote_blob, len, connection).^[5][6][18] The guide warns: “The exported data contains sensitive information. Make sure to pass this data through a secure channel!”^[5] Connection flow two, RDMA-CM: the server calls doca_rdma_start_listen_to_port(rdma, port); the client builds an address with doca_rdma_addr_create(addr_type, addr, port, &cm_addr) (DOCA_RDMA_ADDR_TYPE_IP or _GID) and calls doca_rdma_connect_to_addr(rdma, cm_addr, user_data); both register doca_rdma_set_connection_state_callbacks(rdma, request_cb, established_cb, failure_cb, disconnect_cb), and the server accepts inside the request callback with doca_rdma_connection_accept(conn, NULL, 0).^[5][7] The samples default DEFAULT_RDMA_CM_PORT to 13579 and cap MAX_NUM_CONNECTIONS at 8.^[18]

Tasks are configured before start and allocated at run time: send, receive, write, read, write_imm, send_imm, atomic_cmp_swp (8-byte aligned), atomic_fetch_add, and the remote_net_sync_event get/notify_set/notify_add family.^[5] For send: doca_rdma_task_send_set_conf(rdma, ok_cb, err_cb, n) then doca_rdma_task_send_allocate_init(rdma, conn, src_buf, imm, user_data, &task) and doca_task_submit(doca_rdma_task_send_as_task(task)).^[6] Remote memory follows the DMA pattern with doca_mmap_export_rdma(mmap, dev, &desc, &len) on the owner and doca_mmap_create_from_export on the peer; the permission table says read needs the peer mmap to carry local R/W plus RDMA read and be exported, write needs RDMA write plus export, and send needs no export at all.^[5][7] Two guide sentences to memorize: “Most DOCA RDMA operations are not atomic”, and a completed write “does not guarantee data has been fully written” — the application synchronizes.^[5]

The 3.5.0 rdma_send_sample.c is state-driven: the state-changed callback logs on DOCA_CTX_STATE_STARTING, and on DOCA_CTX_STATE_RUNNING it calls the export-and-connect helper and then, unless RDMA-CM is in use, prepares and submits the send task; on STOPPING it notes that in-flight tasks are flushed, and on IDLE it releases the main loop.^[6] The README adds the rule that costs newcomers hours: “Failing to submit a receive task prior to the send task results in a fatal failure.”^[8] RDMA can also hand its datapath to other engines: doca_ctx_set_datapath_on_dpa() with doca_rdma_get_dpa_handle(), or doca_ctx_set_datapath_on_gpu() with doca_rdma_get_gpu_handle(), which is how M6.3 and M6.4 reuse this lesson.^[5]

4DOCA Verbs and the 3.5.0 Multi-Rail API

DOCA RDMA Verbs is “a Verbs-like API that exposes the low-level features of RDMA”: you create doca_verbs_context, doca_verbs_pd, doca_verbs_cq (via doca_verbs_cq_attr), doca_verbs_qp (via doca_verbs_qp_init_attr) and doca_verbs_ah_attr, register memory with rdma-core’s ibv_reg_mr() for lkey/rkey, and drive the QP with doca_verbs_qp_modify() through RST→INIT→RTR→RTS.^[9] The guide names the datapath options — CPU, GPU or DPA — and an external datapath mode enabled with doca_verbs_qp_init_attr_set_external_datapath_en() so that doca_dpa or doca_gpunetio can own the queue.^[9] NVIDIA’s 3.5.0 Verbs guide lists BlueField-3, ConnectX-7 and ConnectX-8 as supported devices, with BlueField-2 unsupported; the copy of the page fetched for this lesson rendered without that table, so re-check the live page before quoting the device list to a customer.^[9]

The samples make the split concrete. verbs_server_client builds context/PD/CQ/AH/QP, exchanges “local buffer addresses, MKEYs, QP numbers and GID addresses” over a TCP socket, modifies the QP states, then runs send/receive, write or read.^[10] verbs_wait_cq_wr posts a wait_cq_pi work request so QP3’s send executes only after QP1’s send completes — hardware-ordered chaining with no CPU in the loop.^[10] Two more samples, verbs_receive_packets_on_dpa and verbs_send_ethernet_frames_on_dpa, run the Verbs datapath on the DPA.^[10]

Multi-Rail is new in 3.5.0 and lives in applications/verbs_multi_rail_server_client (C++).^[13][11] The objects are doca_verbs_multi_rail_cfg and doca_verbs_multi_rail; the configuration setters are doca_verbs_multi_rail_cfg_set_bw_decay_factor, _set_max_bw_drop_factor, _set_min_chunk_size, _set_chunk_alignment, _set_min_splitable_message_size, _set_lb_mode, _set_probe_count and _set_probe_timeout_s; the run-time calls are doca_verbs_multi_rail_create, _apply_cfg, _set_probe_qp, _update_qp, _probe_start, _probe_get_status, _get_post_split, _update, _error_is_recoverable, _destroy and _cfg_destroy.^[11] The application takes --num-qps (RC QPs, mandatory, ceiling 4096), optional --qp-json, --dev-json and --multi-rail-json, --chunks-total (default 131072 bytes), --traffic send|write, --server, --server-ip, --tcp-port (default 5000) and --use-multi-rail, whose help text reads “Default behavior when omitted: disabled.”^[12][11] Under the hood the app brings up per-device UC probe QPs and a recovery thread after the RC QPs reach RTS, which is the link-failure detection path; num_qps, chunks_total, traffic and use_multi_rail are handshake fields that must match on both sides or the run aborts with DOCA_ERROR_BAD_CONFIG.^[12] The public Verbs page did not detail the Multi-Rail API when fetched, so anything beyond what the source shows (exact load-balancing semantics, failover timing) is unverified and should be quoted as such.^[9]