First programs: dma_copy and secure_channel

Episode 4 — The rehearsal that hangs

The situation · Customer data center, cold aisle, 02:10 the night before the PoC demo

The operator called because he could see link lights and nothing else, and he does not like being the only person awake who knows where the cables go. Nineteen hours before the demo the aisle is loud and the terminal is silent: the archive’s engineer has secure_channel running on the host and nothing is printing — no Producer sent, no Consumer received, just a cursor. She has run it four times. The Dell SE is awake in a hotel room deciding whether Thursday becomes a slide deck instead of a live run, which would put a question mark on the purchase order.

NVIDIA ships these two programs precisely so a first DOCA run is not written from scratch: the samples are “simplistic code snippets that demonstrate the API usage”, and the applications implement logic that crosses several SDK libraries.^[6] The local DMA copy proves one process’s own data path and “should be run on the DPU”; secure_channel proves the path between the two sides of the PCIe bus.^[1] That second one carries an order requirement its arguments do not mention: the server “must be run on the BlueField Arm side and started before” the client on the host.^[11] It also carries an asymmetry — -r/--rep-pci is “needed only on DPU”, because the Arm side speaks for a specific host function.^[8] The quick start prints both invocations side by side.^[10]

When a first run hangs, prove the smallest thing first.

So you do not touch the hanging application. You go back to the smallest program that proves anything.

1dma_local_copy: what main() does before any DOCA object exists

The README summarizes the sample in ten steps: locate the device, initialize core structures, populate the mmap with source and destination, allocate an inventory element per buffer, optionally set NVTX ranges, initialize the memcpy task, submit it (with DOCA_TASK_SUBMIT_FLAG_RANGE_PROFILING_REQUESTED if profiling), handle completion, check the result, destroy everything; it “should be run on the DPU”.^[1]

main() in dma_local_copy_main.c first sets defaults: pci_address "03:00.0", cpy_txt "This is a sample piece of text", one source and one destination doca_buf, num_tasks = DEFAULT_NUM_TASKS, profiling off.^[2] DEFAULT_NUM_TASKS is 1 and MAX_NUM_TASKS is 128 in dma_common.h.^[14] It then creates two log backends, doca_log_backend_create_standard() for the application and doca_log_backend_create_with_file_sdk(stderr, &sdk_log) with doca_log_backend_set_sdk_level(sdk_log, DOCA_LOG_LEVEL_WARNING) for SDK internals, and parses arguments with doca_argp_init(NULL, &dma_conf), register_dma_params(false, true) and doca_argp_start(argc, argv).^[2] A guard follows: under #ifndef DOCA_ARCH_DPU it logs “Local DMA copy can run only on the DPU” and exits unless NVTX profiling was requested.^[2] Finally it computes length = strlen(dma_conf.cpy_txt) + 1, allocates dst_buffer with calloc(1, length * num_tasks) and src_buffer with malloc, copies the text num_tasks times into the source, calls dma_local_copy(&dma_conf, dst_buffer, src_buffer, length), frees, and ends with doca_argp_destroy() and “Sample finished successfully”.^[2]

The parameters come from register_dma_params in dma_common.c: -p/--pci-addr, -t/--text, and for the local copy sample -nt/--num-tasks (“only valid for dma_local_copy sample”) and -mnr/--max-num-range-profiling-displayed (0 disables NVTX, upper bound 128); DPU builds additionally register -ns/--num-src-buf and -nd/--num-dst-buf.^[4]

2dma_local_copy: the sample function, phase by phase

dma_local_copy() validates first: null pointers or zero length return DOCA_ERROR_INVALID_VALUE, and memory_ranges_overlap rejects overlapping ranges with “Memory ranges must not overlap”.^[3] It then calls allocate_dma_resources(dma_conf->pci_address, num_src_buf, num_dst_buf, num_tasks, &resources), which opens the device with the DMA capability filter, checks doca_dma_cap_task_memcpy_get_max_buf_list_len, creates the core objects, creates and configures the DMA context and stores the resources pointer as context user data.^[4] Optional NVTX: doca_ctx_profiler_set_max_nranges(state->ctx, n), tolerating DOCA_ERROR_NOT_SUPPORTED with a warning.^[3]

The data-path setup is then doca_pe_connect_ctx(state->pe, state->ctx), doca_ctx_start(state->ctx), register_memory_range_and_start_mmap for destination and source (each doca_mmap_set_memrange then doca_mmap_start), memset(dst_buffer, 0, total_length), and allocate_doca_buf_for_task for source then destination.^[3] That helper calls allocate_doca_buf_list(state->buf_inv, mmap, buffer + i * length, length, num_buf_elem, is_source, &doca_buf[i]), which splits each task’s range into num_buf chained segments through doca_buf_inventory_buf_get_by_addr and doca_buf_chain_list, setting data with doca_buf_set_data only for sources.^[3]^[13] Per task: task_user_data[i].ptr = &task_results[i], doca_dma_task_memcpy_alloc_init(resources.dma_ctx, src_doca_buf[i], dst_doca_buf[i], task_user_data[i], &dma_tasks[i]), then doca_task_submit (or doca_task_submit_ex with the profiling flag), counting num_submitted and num_remaining_tasks.^[3]

The wait is resources.run_pe_progress = true; while (resources.run_pe_progress) if (doca_pe_progress(state->pe) == 0) nanosleep(&ts, &ts); with a ten-microsecond sleep.^[3] The callbacks in dma_common.c stop the context when the last task completes, and the state callback clears run_pe_progress on Idle.^[4] Afterwards each task_results[i] is checked and logged as “Success, DMA memory task %d copied and verified as correct” or propagated as an error; the cleanup labels destroy_dst_buf, destroy_src_buf, stop_dma and destroy_resources unwind in reverse.^[3]

dma_local_copy: worked → faded → problem

Build and run on the Arm side, then read the output against the code.

cd doca-samples/samples/doca_dma/dma_local_copy
meson /tmp/build-dma && ninja -C /tmp/build-dma
/tmp/build-dma/doca_dma_local_copy -p 03:00.0 -t "hello dell lab" -nt 4

Expected log sequence and the code that produces each line: Starting the sample (main), DMA context is running (state callback on Running after doca_ctx_start), four times DMA task was completed successfully (success callback), DMA context entered into stopping state. Any inflight tasks will be flushed (the fourth callback called doca_ctx_stop), DMA context has been stopped (state callback on Idle, ends the loop), four times Success, DMA memory task N copied and verified as correct (result check in the sample function), Sample finished successfully (main).

Reasoning: -nt 4 makes length * 4 bytes of source and destination; each task copies one length slice because allocate_doca_buf_for_task offsets by i * length. The count in the callbacks is what triggers the stop, not the main loop.

3secure_channel: the Comch control path

secure_channel.c is the smallest possible main() for a two-sided application. Under #ifdef DOCA_ARCH_DPU it sets app_cfg.mode = SC_MODE_DPU; it creates the two log backends, calls doca_argp_init(NULL, &app_cfg), register_secure_channel_params() and doca_argp_start, then comch_utils_fast_path_init(SERVER_NAME, app_cfg.cc_dev_pci_addr, app_cfg.cc_dev_rep_pci_addr, &ctx, comch_recv_event_cb, comch_recv_event_cb, new_consumer_callback, expired_consumer_callback, &comch_cfg), sc_start(comch_cfg, &app_cfg, &ctx), comch_utils_destroy(comch_cfg) and doca_argp_destroy(); SERVER_NAME is "secure_channel_server".^[7] The parameters are -s/--msg-size, -n/--num-msgs and -p/--pci-addr, all doca_argp_param_set_mandatory, plus -r/--rep-pci described as “needed only on DPU”, and a version callback registered with doca_argp_register_version_callback.^[8] The quick start runs the server on the BlueField as /opt/mellanox/doca/applications/secure_channel/bin/doca_secure_channel -s 256 -n 10 -p 03:00.0 -r 3b:00.0 and the client on the host as ... -s 256 -n 10 -p 3b:00.0.^[10] The Comch README fixes the order: the server “must be run on the BlueField Arm side and started before” the client on the host.^[11]

comch_utils_fast_path_init is the shared glue. On the DPU it runs doca_pe_create, open_doca_device_with_pci, doca_comch_cap_get_max_msg_size, doca_comch_cap_server_is_supported, open_doca_device_rep_with_pci, doca_comch_server_create(dev, rep, server_name, &server), doca_comch_server_set_max_msg_size, doca_comch_server_as_ctx, doca_pe_connect_ctx, doca_comch_server_task_send_set_conf, doca_comch_server_event_msg_recv_register, doca_comch_server_event_connection_status_changed_register, doca_comch_server_event_consumer_register, doca_ctx_set_user_data, doca_ctx_start.^[9] On the host it runs doca_comch_cap_client_is_supported, doca_comch_client_create, doca_comch_client_set_max_msg_size, doca_comch_client_as_ctx, doca_pe_connect_ctx, doca_comch_client_task_send_set_conf, doca_comch_client_event_msg_recv_register, doca_comch_client_event_consumer_register, doca_ctx_set_user_data, doca_ctx_start, then polls doca_ctx_get_state through doca_pe_progress until Running and calls doca_comch_client_get_connection and doca_comch_connection_set_user_data.^[9]

sc_start then sends one struct metadata_msg of type START_MSG with htonl(cfg->send_msg_nb) and htonl(cfg->send_msg_size) via comch_utils_send, and progresses the connection until ctx->expected_msgs is set by comch_recv_event_cb, which rejects any message whose length differs from sizeof(struct metadata_msg) and resets expected_msgs to zero on END_MSG.^[8]

4secure_channel: the fast path

After the metadata exchange, start_threads spawns two detached pthreads, run_producer and run_consumer, and the main thread keeps calling comch_utils_progress_connection because “Comch handles producer and consumer control messages so must continue to run”.^[8] The producer registers one cache-aligned buffer with prepare_local_memory(&local_mem, pci_addr, msg_len, 1, DOCA_ACCESS_FLAG_PCI_READ_ONLY), a helper that runs doca_mmap_create, doca_mmap_set_permissions, doca_mmap_add_dev, doca_mmap_set_memrange, doca_mmap_start, doca_buf_inventory_create and doca_buf_inventory_start.^[8] It checks doca_comch_producer_cap_get_max_buf_size, failing with “Producer does not support message size. Requested: %u, max: %u” and DOCA_ERROR_INVALID_VALUE when -s is too large, creates its own PE with doca_pe_create(&producer_pe), then doca_comch_producer_create(ctx->comch_connection, &producer), doca_pe_connect_ctx, doca_comch_producer_task_send_set_conf(producer, send_task_completed_callback, send_task_fail_callback, total_tasks), doca_ctx_set_user_data, doca_ctx_start, and takes the buffer with doca_buf_inventory_buf_get_by_data.^[8]

It then waits until ctx->consumer_id is non-zero, which new_consumer_callback sets from the peer’s consumer event, and submits total_tasks tasks built with doca_comch_producer_task_send_alloc_init(producer, doca_buf, NULL, 0, ctx->consumer_id, &task[i]), retrying doca_task_submit while it returns DOCA_ERROR_AGAIN.^[8] The consumer mirrors this with DOCA_ACCESS_FLAG_PCI_READ_WRITE, doca_comch_consumer_cap_get_max_buf_size, doca_comch_consumer_create(ctx->comch_connection, local_mem.mmap, &consumer), doca_comch_consumer_task_post_recv_set_conf, a wait for Running, and one doca_buf_inventory_buf_get_by_addr plus doca_comch_consumer_task_post_recv_alloc_init(consumer, doca_buf[i], &task[i]) per task; its completion callback resets the buffer and resubmits.^[8] Both threads stop their context, poll to Idle, destroy the producer or consumer, then the PE, then the local memory, and sc_start prints “Producer sent %u messages in approximately %0.4f milliseconds” and “Consumer received %u messages in approximately %0.4f milliseconds”.^[8] The DPU sends END_MSG so the host knows both sides are done before the client disconnects.^[8] The build pulls in secure_channel_core.c, common/comch_utils.c, common/utils.c and samples/common.c and names the binary doca_secure_channel.^[12]

Task lifecycle · DOCA Comch · control-path sendengine

Step 1 / 7owner: pool

Zero-allocation data path

Every task comes from a pool sized by *_task_*_set_conf(…, num_tasks) and every doca_buf from doca_buf_inventory_create(max_bufs), both fixed before doca_ctx_start. alloc_init / doca_task_free only move items in and out of those pools. DOCA_ERROR_NO_MEMORY from a pool is back-pressure: release finished work, do not retry.

Ownership handoff. doca_task_submit hands the task (and its buffers) to the context; the completion callback hands it back. In between: no read, modify, free or dec_refcount.

1. Prerequisites (context Running)

/* host side = client. BlueField Arm side = server:
   doca_comch_server_create(hw_dev, rep_dev, server_name, &server) */
doca_comch_client_create(hw_dev, server_name, &client);
struct doca_ctx *ctx = doca_comch_client_as_ctx(client);
doca_pe_connect_ctx(pe, ctx);
doca_comch_client_task_send_set_conf(client, send_completion_cb, send_error_cb, num_tasks);
doca_comch_client_event_msg_recv_register(client, msg_recv_cb);
doca_ctx_start(ctx);                       /* RUNNING == connected to the server */
doca_comch_client_get_connection(client, &connection);

Why: Events (msg_recv; connection status on the server via doca_comch_server_event_connection_status_changed_register) are registered before start. The client reaching RUNNING means the connection exists.

source

Predict: what is the next call for COMCH, and who owns the task after it?

Comch engine: the consumer posts receive tasks first, the producer sends once it knows the consumer id, and each completed post_recv is reset and resubmitted.

5Change the message, change the size

The second program is where a developer learns the limits are real. -s is bounded on each side by doca_comch_producer_cap_get_max_buf_size and doca_comch_consumer_cap_get_max_buf_size, both read from doca_dev_as_devinfo(local_mem.dev), and the control-path metadata is bounded separately by doca_comch_cap_get_max_msg_size.^[8]^[9] -n above MAX_FASTPATH_TASKS (1024) is legal and simply reuses tasks through resubmission.^[8] Because both sides exchange num_msgs and msg_size in START_MSG, each consumer expects exactly what the opposite producer will send; the two invocations must agree on -s and -n.^[8]

secure_channel: worked → faded → problem

On the BlueField Arm side (server, started first), then on the host (client):

# Arm side
/opt/mellanox/doca/applications/secure_channel/bin/doca_secure_channel -s 256 -n 10 -p 03:00.0 -r 3b:00.0
# host side
/opt/mellanox/doca/applications/secure_channel/bin/doca_secure_channel -s 256 -n 10 -p 3b:00.0

Expected on both sides: Producer sent 10 messages in approximately X milliseconds and Consumer received 10 messages in approximately Y milliseconds. Reasoning: -p on the Arm is the local PF; -r is the representor of the host function the client will use; -p on the host is that same function seen from the host. Both sides agree on 256 bytes and 10 messages, so each consumer expects ten 256-byte messages and finishes when its completion counter reaches ten.

Episode 4 — Server first, then the demo

How it ended

The local DMA copy runs clean on the Arm side, which clears the card and the DOCA install in under a minute. Then the sequence: server on the BlueField first, client on the host second, both agreeing on -s and -n, since each side is told what to expect in the metadata message.^[8]^[11] Both consumers print their counts. Thursday stays live. By 02:40 the label maker has produced one more sticker, START ME FIRST, now under the Arm-side node’s service tag.

What you say to her lead: “It was start order, not hardware. Start the Arm side first, and match -s and -n on both invocations.”

The demo runs. Then the perf lead measures DMA throughput on his own, and comes back thirty percent under the number in our material.

Lab

Pre-flight (read-only): on the Arm side /opt/mellanox/doca/tools/doca_caps --list-devs to get the local PF; on the host lspci | grep -i mellanox to get the host function address; /opt/mellanox/doca/tools/doca_caps --version on both must match. The representor address for -r is the host function’s address as the quick start shows (-r 3b:00.0 paired with the host’s -p 3b:00.0).

Arm side, in the devel container or on the BlueField OS: /opt/mellanox/doca/applications/secure_channel/bin/doca_secure_channel -s 256 -n 10 -p <arm pf> -r <host fn> — expect it to block waiting for the client. If it exits with Failed to open Comm Channel DOCA device representor, the -r address is wrong.
Host side: /opt/mellanox/doca/applications/secure_channel/bin/doca_secure_channel -s 256 -n 10 -p <host fn> — expect both sides to print Producer sent 10 messages and Consumer received 10 messages. If the client exits with a device error, the -p address is wrong; if it hangs, the server was not running first.
Repeat with -s 4096 -n 1000 on both sides — expect matching counts of 1000. Save both terminals’ output to a file for the module report.
Run the rebuilt dma_local_copy from the no-hardware lab on the Arm side with -p <arm pf> -nt 4 -nd 2 — expect four success lines and no Copy mismatch.
Rollback: none required. Both programs are user-space and leave no device configuration behind; stop them with Ctrl-C if a side hangs.

Evidence / notes (commands run, outputs, what surprised you)

First programs: dma_copy and secure_channel

Before you read: what do you already know?

After this lesson you can

Episode 4 — The rehearsal that hangs

1dma_local_copy: what main() does before any DOCA object exists

2dma_local_copy: the sample function, phase by phase

3secure_channel: the Comch control path

4secure_channel: the fast path

5Change the message, change the size

Episode 4 — Server first, then the demo

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere