DOCA Core objects and lifecycle

1Devices: devinfo, dev, representor

The Core guide describes device handling as querying device information and capabilities, then opening the selected device. On the BlueField it “Gets local BlueField devices” and “Gets representors list (representing host local devices)”; on the host it gets local devices and their representors, which matter for ConnectX or a DPU in NIC mode.^[1] The entities are doca_devinfo, doca_devinfo_rep, doca_dev and doca_dev_rep.^[1]

The samples wrap this in common.c. open_doca_device_with_pci_and_callback calls doca_devinfo_create_list(&dev_list, &nb_devs), loops with doca_devinfo_is_equal_pci_addr(dev_list[i], pci_addr, &is_addr_equal), skips any device whose capability callback does not return DOCA_SUCCESS, opens the first match with doca_dev_open(dev_list[i], retval), frees the list with doca_devinfo_destroy_list, and otherwise logs “Matching device not found” and returns DOCA_ERROR_NOT_FOUND.^[2] The DMA code passes dma_task_is_supported, which is nothing more than doca_dma_cap_task_memcpy_is_supported(devinfo); the capability is checked on the doca_devinfo before any open.^[3] After the open, capabilities are read from doca_dev_as_devinfo(dev), for example doca_dma_cap_task_memcpy_get_max_buf_list_len(doca_dev_as_devinfo(state->dev), &max_buf_list_len).^[3]

Representors follow the same shape one level down. open_doca_device_rep_with_pci(local, filter, pci_addr, &rep) calls doca_devinfo_rep_create_list(local, filter, &rep_dev_list, &nb_rdevs), matches with doca_devinfo_rep_is_equal_pci_addr, opens with doca_dev_rep_open, and cleans up with doca_devinfo_rep_destroy_list.^[2] The Comch glue on the DPU side uses it as open_doca_device_rep_with_pci(cfg->dev, DOCA_DEVINFO_REP_FILTER_NET, rep_pci_addr, &cfg->dev_rep) and then creates the server with doca_comch_server_create(cfg->dev, cfg->dev_rep, server_name, &cfg->server); teardown closes the representor with doca_dev_rep_close before doca_dev_close.^[8]

2Memory: mmap and buffers

The Core guide gives one flow for every core object: “Create the object instance (e.g., doca_mmap_create). Configure the instance (e.g., doca_mmap_set_memory_range). Start the instance (e.g., doca_mmap_start).” After start “it adheres to zero allocations and can be used safely in the data path”, and it “must be stopped and destroyed (doca_mmap_stop, doca_mmap_destroy)”; some objects can go create, configure, start, stop, configure, start.^[1] Note a naming drift: the guide’s prose says doca_mmap_set_memory_range, but the function the 3.5.0 samples call is doca_mmap_set_memrange.^[4]

In code the configuration is doca_mmap_create(&state->src_mmap) and doca_mmap_add_dev(state->src_mmap, state->dev) inside create_core_objects, then later doca_mmap_set_memrange(mmap, buffer, length) followed by doca_mmap_start(mmap) in register_memory_range_and_start_mmap.^[2][4] When the memory must be reachable from the other side of the PCIe bus, permissions come first: dma_copy_host calls doca_mmap_set_permissions(state.src_mmap, DOCA_ACCESS_FLAG_PCI_READ_ONLY), then doca_mmap_set_memrange, doca_mmap_start, and doca_mmap_export_pci(state.src_mmap, state.dev, &export_desc, &export_desc_len).^[5] The guide warns that “If appropriate access has not been provided, the export fails” and that the exported descriptor “contains sensitive information”.^[1] The guide’s page prints an enum spelled DOCA_ACCESS_LOCAL_READ_ONLY, DOCA_ACCESS_LOCAL_READ_WRITE, DOCA_ACCESS_RDMA_READ, DOCA_ACCESS_RDMA_WRITE, DOCA_ACCESS_RDMA_ATOMIC, DOCA_ACCESS_DPU_READ_ONLY and DOCA_ACCESS_DPU_READ_WRITE; the 3.5.0 samples compile with the DOCA_ACCESS_FLAG_ names, so trust the sample when the two disagree.^[1][5]

A doca_buf is a view onto mapped memory with three regions: headroom “starting from the buffer’s address up to the buffer’s data address”, dataroom “starting from the buffer’s data address with a length indicated by the buffer’s data length”, and tailroom “starting from the end of the dataroom to the end of the buffer”; the tailroom is free writing space for libraries, and the buffer “does not own nor manage the data it references”.^[1] Buffers come from an inventory: doca_buf_inventory_create(max_bufs, &inv) then doca_buf_inventory_start(inv); allocate_doca_buf_list takes each one with doca_buf_inventory_buf_get_by_addr(inv, mmap, addr, len, &buf), marks source data with doca_buf_set_data(buf, addr, len), chains with doca_buf_chain_list(head, next), and releases with doca_buf_dec_refcount(buf, NULL), where releasing the head releases every chained buffer.^[2] Once handed to a task, “ownership of the buffer moves to the library until that task is complete”.^[1]

3Context state machine and the progress engine

A doca_ctx has four states. Idle: “0 in-flight tasks”, all configuration APIs enabled right after doca_<T>_create. Starting: “mandatory for CTXs where transition to running state is conditioned by one or more async op completions/external events”, the guide’s example being a client connecting to a comm channel. Running: “Task allocation/submission enabled (disabled in all other states)” and “All configuration APIs are disabled”. Stopping: “Clean all in-flight tasks that may not complete in near future”.^[1] An internal error in Starting or Running causes an involuntary transition to Stopping.^[1]

The samples name them DOCA_CTX_STATE_IDLE, DOCA_CTX_STATE_STARTING, DOCA_CTX_STATE_RUNNING and DOCA_CTX_STATE_STOPPING, and observe them through a callback registered with doca_ctx_set_state_changed_cb: dma_state_changed_callback(const union doca_data user_data, struct doca_ctx *ctx, enum doca_ctx_states prev_state, enum doca_ctx_states next_state).^[3] Because DMA has no asynchronous start, that callback logs “DMA context entered into starting state. Unexpected transition” if it ever sees Starting.^[3] Creation is doca_dma_create(state->dev, &resources->dma_ctx), the generic handle is doca_dma_as_ctx(resources->dma_ctx), and configuration includes doca_dma_cap_get_max_num_tasks, doca_dma_task_memcpy_set_conf(dma_ctx, completed_cb, error_cb, num_tasks) and doca_ctx_set_user_data.^[3]

The progress engine is created with doca_pe_create(&state->pe) in create_core_objects; the context is attached with doca_pe_connect_ctx(state->pe, state->ctx) and only then started with doca_ctx_start(state->ctx).^[2][4] A PE “can be connected to multiple contexts”, of the same or different types, so one thread can wait on all of them.^[1] Stopping is asynchronous: request_stop_ctx calls doca_ctx_stop, and on DOCA_ERROR_IN_PROGRESS loops doca_pe_progress(pe) and doca_ctx_get_state(ctx, &st) until DOCA_CTX_STATE_IDLE.^[2] The PE README adds the destruction rule: “all contexts must be destroyed before the PE”.^[6]

struct doca_devinfo **dev_list;
uint32_t nb_devs;
doca_devinfo_create_list(&dev_list, &nb_devs);

4doca_error_t: what each family means

Every DOCA API returns doca_error_t. The Core guide lists DOCA_SUCCESS, DOCA_ERROR_UNKNOWN, DOCA_ERROR_NOT_PERMITTED, DOCA_ERROR_IN_USE, DOCA_ERROR_NOT_SUPPORTED, DOCA_ERROR_AGAIN, DOCA_ERROR_INVALID_VALUE, DOCA_ERROR_NO_MEMORY, DOCA_ERROR_INITIALIZATION, DOCA_ERROR_TIME_OUT, DOCA_ERROR_SHUTDOWN, DOCA_ERROR_CONNECTION_RESET, DOCA_ERROR_CONNECTION_ABORTED, DOCA_ERROR_CONNECTION_INPROGRESS, DOCA_ERROR_NOT_CONNECTED, DOCA_ERROR_NO_LOCK, DOCA_ERROR_NOT_FOUND, DOCA_ERROR_IO_FAILED, DOCA_ERROR_BAD_STATE, DOCA_ERROR_UNSUPPORTED_VERSION, DOCA_ERROR_OPERATING_SYSTEM, DOCA_ERROR_DRIVER, DOCA_ERROR_UNEXPECTED, DOCA_ERROR_ALREADY_EXIST, DOCA_ERROR_FULL, DOCA_ERROR_EMPTY, DOCA_ERROR_IN_PROGRESS and DOCA_ERROR_TOO_BIG.^[1] The same headers serve BlueField and host, “but specific API calls may return DOCA_ERROR_NOT_SUPPORTED if the API is not implemented for that processor”.^[1]

The samples decode with doca_error_get_descr(result) in every log line and keep the first failure through teardown with DOCA_ERROR_PROPAGATE(result, tmp_result).^[3][4] The debugging notes turn the enum into a diagnosis map, reproduced here with the sample evidence:

The rule from the notes: never retry-loop INVALID_VALUE, BAD_STATE, NOT_SUPPORTED, INITIALIZATION or DRIVER, and retry TIME_OUT only with a documented bound.^[7]

5The canonical order, as executed by dma_common.c and common.c

Read together, allocate_dma_resources and dma_local_copy execute the lifecycle in this order. Open the device with a capability filter: open_doca_device_with_pci(pcie_addr, &dma_task_is_supported, &state->dev).^[3] Query limits: doca_dma_cap_task_memcpy_get_max_buf_list_len, and reject requests above them with DOCA_ERROR_INVALID_VALUE.^[3] Create the core objects: two mmaps with doca_mmap_create and doca_mmap_add_dev, an inventory with doca_buf_inventory_create and doca_buf_inventory_start, and the PE with doca_pe_create.^[2] Create and configure the context: doca_dma_create, doca_dma_as_ctx, doca_ctx_set_state_changed_cb, doca_dma_cap_get_max_num_tasks, doca_dma_task_memcpy_set_conf, doca_ctx_set_user_data.^[3] Attach and start: doca_pe_connect_ctx, doca_ctx_start.^[4] Finish the mmaps: doca_mmap_set_memrange and doca_mmap_start for destination then source.^[4] Use: buffers from the inventory, doca_dma_task_memcpy_alloc_init, doca_task_submit, and a doca_pe_progress loop.^[4] Tear down: release buffers, doca_ctx_stop, doca_dma_destroy, then destroy_core_objects, which destroys the PE, the inventory, both mmaps and closes the device.^[2][3] The README summarises the same run as locate device, initialize core structures, populate the mmap, allocate inventory elements, initialize the task, submit, handle completion, check the result, destroy.^[9]

/* 1. device: filtered open */
open_doca_device_with_pci(pcie_addr, &dma_task_is_supported, &state->dev);
doca_dma_cap_task_memcpy_get_max_buf_list_len(doca_dev_as_devinfo(state->dev), &max_buf_list_len);
/* 2. core objects (create_core_objects) */
doca_mmap_create(&state->src_mmap);  doca_mmap_add_dev(state->src_mmap, state->dev);
doca_mmap_create(&state->dst_mmap);  doca_mmap_add_dev(state->dst_mmap, state->dev);
doca_buf_inventory_create(max_bufs, &state->buf_inv);  doca_buf_inventory_start(state->buf_inv);
doca_pe_create(&state->pe);
/* 3. context: create + configure (Idle) */
doca_dma_create(state->dev, &resources->dma_ctx);
state->ctx = doca_dma_as_ctx(resources->dma_ctx);
doca_ctx_set_state_changed_cb(state->ctx, dma_state_changed_callback);
doca_dma_cap_get_max_num_tasks(resources->dma_ctx, &max_tasks_num);
doca_dma_task_memcpy_set_conf(resources->dma_ctx, dma_memcpy_completed_callback, dma_memcpy_error_callback, num_tasks);
doca_ctx_set_user_data(state->ctx, ctx_user_data);
/* 4. attach + start (Idle -> Running) */
doca_pe_connect_ctx(state->pe, state->ctx);
doca_ctx_start(state->ctx);
/* 5. mmaps: configure + start */
doca_mmap_set_memrange(state->dst_mmap, dst_buffer, total_length);  doca_mmap_start(state->dst_mmap);
doca_mmap_set_memrange(state->src_mmap, src_buffer, total_length);  doca_mmap_start(state->src_mmap);
/* 6. use (Running) */
doca_buf_inventory_buf_get_by_addr(state->buf_inv, mmap, addr, len, &buf);  doca_buf_set_data(buf, addr, len); /* source only */
doca_dma_task_memcpy_alloc_init(resources->dma_ctx, src_buf, dst_buf, task_user_data, &task);
doca_task_submit(doca_dma_task_memcpy_as_task(task));
while (resources->run_pe_progress) if (doca_pe_progress(state->pe) == 0) nanosleep(&ts, &ts);
/* 7. teardown (Running -> Stopping -> Idle, then destroy) */
doca_buf_dec_refcount(buf, NULL);
doca_ctx_stop(state->ctx);            /* completion callback already called it when the last task finished */
doca_dma_destroy(resources->dma_ctx);
destroy_core_objects(state);          /* pe -> buf_inv -> dst_mmap -> src_mmap -> dev */