Tasks, the progress engine, and completion

1A task changes hands twice

The Core guide defines a task’s life “by a clear ownership model” bounded by the context: “An application can only allocate a task when the owning CTX enters the RUNNING state” and can no longer allocate once it leaves it.^[1] Phase 1: allocation with doca_<T>_<J>_task_alloc(ctx, &task) gives the application ownership, configuration with doca_<T>_<J>_task_set_<param>(task, param) keeps it, and doca_task_submit(task) “passes ownership of the task object to the CTX”.^[1] Phase 2: the PE executes; the application “must periodically call doca_pe_progress(pe)”, and when the PE detects completion it invokes the handler, at which point “ownership of the task is passed back to the application” and result fields are safe to read.^[1] Phase 3: doca_task_free(task) or re-submit.^[1]

The DMA sample compresses phase 1 into one call: doca_dma_task_memcpy_alloc_init(resources.dma_ctx, src_doca_buf[i], dst_doca_buf[i], task_user_data[i], &dma_tasks[i]), then doca_dma_task_memcpy_as_task(dma_tasks[i]) for the generic handle and doca_task_submit(tasks); with NVTX profiling it uses doca_task_submit_ex(tasks, DOCA_TASK_SUBMIT_FLAG_FLUSH | DOCA_TASK_SUBMIT_FLAG_RANGE_PROFILING_REQUESTED) and notes the default flag is DOCA_TASK_SUBMIT_FLAG_FLUSH.^[5] Two kinds of user data travel with the work: the task’s union doca_data points at a per-task doca_error_t slot, and the context’s user data points at the dma_resources struct, so callbacks can update both without globals.^[5][4]

One caution from the PE README: “doca_task_submit does not validate task inputs (to increase performance)”; doca_task_try_submit validates during development and “should not be used in production”.^[2]

doca_dma_task_memcpy_set_conf(dma, memcpy_completed_cb, memcpy_error_cb, num_tasks);
doca_pe_connect_ctx(pe, doca_dma_as_ctx(dma));
doca_ctx_start(doca_dma_as_ctx(dma));

2Polling or waiting: two ways to drive the PE

After a PE is created “an application can define an event loop using one of these modes: Polling mode, Blocking (notification-driven) mode”, and in both “All completion handlers for both tasks and events are strictly executed within the context of the doca_pe_progress() function”.^[1] One call iterates every scheduled unit and stops on a task completion or positive event probe (returning “made some progress”), on considerable partial progress, or falls through with “no progress”.^[1] Polling code is therefore a loop; the DMA sample sleeps ten microseconds whenever progress returns zero: while (resources.run_pe_progress) if (doca_pe_progress(state->pe) == 0) nanosleep(&ts, &ts).^[5]

Blocking mode replaces the sleep with a file descriptor. The guide: the application gets a notification handle “representing a Linux file descriptor”, arms the PE with doca_pe_request_notification() “every time” it wants a notification, and “After doca_pe_request_notification(), no calls to doca_pe_progress() are allowed” until doca_pe_clear_notification.^[1] The 3.5.0 event sample is the reference implementation:

/* doca_event_handle_t is a file descriptor that can be added to an epoll */
EXIT_ON_FAILURE(doca_pe_get_notification_handle(state->base.pe, &event_handle));
epoll_ctl(state->epoll_fd, EPOLL_CTL_ADD, event_handle, &events_in);
...
do {
	while (doca_pe_progress(state->base.pe) != 0) {
		if (state->base.num_completed_tasks == NUM_TASKS)
			return DOCA_SUCCESS;
	}
	EXIT_ON_FAILURE(doca_pe_request_notification(state->base.pe));
	epoll_status = epoll_wait(state->epoll_fd, &ep_event, 1, no_timeout);
	/* handle parameter is not used in Linux */
	EXIT_ON_FAILURE(doca_pe_clear_notification(state->base.pe, 0));
} while (1);

The sample’s own comment states the trade: arming “implies enabling an interrupt, but it also reduces CPU utilization because the program can sleep until the event is fired”.^[3] The README is more direct: “Event-driven mode reduces CPU utilization … but may increase latency or reduce performance.”^[2] A single PE may host several contexts of one or many types; pe_multi_context connects four DMA contexts to one PE.^[1][2]

3Zero allocations on the data path

The Core guide’s reason for the create, configure, start flow is that “All core objects adhere to same flow that later helps in doing no allocations in the fast path”; after start an object “adheres to zero allocations and can be used safely in the data path”.^[1] Everything a data path needs is sized while Idle. allocate_dma_resources reads doca_dma_cap_get_max_num_tasks(resources->dma_ctx, &max_tasks_num), rejects a larger request with DOCA_ERROR_INVALID_VALUE and the message “Number of tasks [%d] exceed the memcpy task max_tasks_num capability”, then fixes the pool with doca_dma_task_memcpy_set_conf(resources->dma_ctx, dma_memcpy_completed_callback, dma_memcpy_error_callback, num_tasks).^[4] The buffer pool is fixed the same way: total_num_buf is source plus destination segments per task times tasks, and it becomes doca_buf_inventory_create(max_bufs, ...).^[4]

The Comch fast path shows what to do when the work exceeds the pool. The producer thread caps its pool at MAX_FASTPATH_TASKS, noting “If requested messages exceeds maximum tasks, tasks will be resubmitted in their completion callback”, and calls doca_comch_producer_task_send_set_conf(producer, send_task_completed_callback, send_task_fail_callback, total_tasks).^[6] The consumer’s completion callback resubmits in place: it takes the buffer with doca_comch_consumer_task_post_recv_get_buf(task), resets it with doca_buf_reset_data_len(buf) “so that it can be fully repopulated”, and calls doca_task_submit on the same task.^[6] The PE README lists the benefits of that pattern: no free and allocate per iteration, a smaller task pool, and the ability to set new source or destination buffers before resubmitting.^[2]

Backpressure is the other side of fixed pools. The producer’s submit loop reads “May need to wait for a post_recv message before being able to send” and spins on while (result == DOCA_ERROR_AGAIN) result = doca_task_submit(...).^[6] DOCA_ERROR_AGAIN is the one code a retry loop is designed for.

4Completion, error and flush

“A DOCA task can invoke a success or error callback. Both callbacks share the same structure”, and DOCA recommends two: the success callback “does not need to check the task status, thereby improving performance”, the error callback “may need to run a different flow”.^[2] In dma_common.c the success callback writes *result = DOCA_SUCCESS, frees the task with doca_task_free(doca_dma_task_memcpy_as_task(dma_task)), decrements num_remaining_tasks, and calls doca_ctx_stop(resources->state.ctx) when the count reaches zero; the error callback does the same after *result = doca_task_get_status(task) and a log of doca_error_get_descr(*result).^[4]

Stopping delivers the rest. The state callback’s comment for DOCA_CTX_STATE_STOPPING says doca_pe_progress() “will cause any inflight task to be flushed” and will “eventually transition the context to idle state”, where the sample sets run_pe_progress = false.^[4] The pe_async_stop sample makes the point that a stop “is asynchronous because the context must complete/abort all tasks” and that the error callback must “check if this is a real error or if the task is flushed”.^[2] secure_channel codes that check explicitly: its consumer failure callback comments “Task fail errors may occur if context is in stopping state - this is expect” and returns without marking an error once the transfer is complete.^[6]

A genuine failure can start the same sequence from the other end. The Core guide: “Once a task fails, the context may transition to stopping state, in this state, the application has to progress all in-flight tasks until completion before destroying or restarting the context.”^[1] The pe_task_error sample submits 255 tasks with one deliberately invalid and shows the mitigation: progress until every submitted task has been flushed.^[2] Clean teardown then follows request_stop_ctx, which loops doca_pe_progress and doca_ctx_get_state until DOCA_CTX_STATE_IDLE when doca_ctx_stop returns DOCA_ERROR_IN_PROGRESS, and the rule that all contexts are destroyed before the PE.^[7][2]