Development environments

1Decide where it runs before you decide where to build

A DOCA binary is compiled for one side of the PCIe bus and one DOCA version. The Developer Guide’s rule is blunt: “Make sure that the DOCA version used for compilation is the same as the version installed on BlueField used for testing.”^[2] The samples discover their side at configure time: applications/meson.build asks the compiler whether doca_build_config.h defines DOCA_ARCH_DPU and sets is_dpu or is_host from the answer, and source files guard roles with #ifdef DOCA_ARCH_DPU.^[9] The consequence is visible in dma_local_copy_main.c, which refuses to run on a host build (“Local DMA copy can run only on the DPU”) unless NVTX profiling is requested.^[12]

So the first question for any new program is: will it execute as an x86 process on the host, or as an Arm process on the BlueField? The answer selects the container tag, the toolchain and the test plan. The chart below is the decision as an FAE would draw it on a whiteboard.

flowchart TD; S(["New DOCA program"]) --> Q1{"Where will the binary execute?"}; Q1 -->|"x86 process on the host"| Q2{"DOCA-Host installed on a dev box?"}; Q1 -->|"Arm process on BlueField"| Q3{"BlueField reachable now?"}; Q2 -->|"yes"| H1["Build natively: meson against /opt/mellanox/doca"]; Q2 -->|"no"| H2["Build in nvcr.io/nvidia/doca/doca:devel-3.5.0-host"]; Q3 -->|"yes"| A1["Build on the card: devel-3.5.0 with --privileged --net=host and hugepages mounted"]; Q3 -->|"no"| Q4{"x86 workstation only?"}; Q4 -->|"QEMU"| A2["docker pull --platform=linux/arm64 devel-3.5.0: build only, never run"]; Q4 -->|"cross toolchain"| A3["meson -Denable_cross_compilation_to_dpu=true with sysroot under /root/doca-cross"]; H1 --> R["Run where the device is, same DOCA version both sides"]; H2 --> R; A1 --> R; A2 --> R; A3 --> R

2The NGC container: flavor, side, architecture

The image is nvcr.io/nvidia/doca/doca. It ships in three flavors: base-rt includes the DOCA runtime, full-rt builds on it as a full runtime “similar to the BlueField image”, and devel builds on that again and adds the development tools.^[1][2] For 3.5.0 the BlueField-side tags are base-rt-3.5.0, full-rt-3.5.0 and devel-3.5.0; the host-side tags add a suffix: base-rt-3.5.0-host, full-rt-3.5.0-host, devel-3.5.0-host; CUDA variants exist such as devel-cuda13.0.0-3.5.0-devel-host.^[1] NGC marks the image multi-arch, and older listings show explicit linux-amd64 and linux-arm64 tags.^[1]

The NVIDIA skills repository gives the operational rule: open the tags page, pick by uname -m (x86_64 maps to amd64, aarch64 or Apple Silicon to arm64), copy the tag verbatim and “never invent one”.^[3] The public images pull anonymously; if the registry demands a login, use printf '%s' "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin.^[3]

On a BlueField the Developer Guide runs the container with device access: sudo docker run -v <source-code-folder>:/doca_devel -v /dev/hugepages:/dev/hugepages --privileged --net=host -it <image>, where --net=host “ensures the container has network access, including visibility to SFs and VFs” and the hugepages mount exposes pages reserved on the host OS.^[2] The same page still prints 1.5.1-devel in its examples; substitute a 3.5.0 tag.^[2]

3arm64 on x86: QEMU builds, QEMU does not run

An x86 workstation can host the Arm image through user-mode emulation. On Ubuntu the Developer Guide installs sudo apt-get install qemu binfmt-support qemu-user-static and registers the handlers with sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; CentOS, RHEL and Fedora use qemu-system-arm, qemu-kvm or qemu-system-aarch64 plus a /etc/binfmt.d/qemu-aarch64.conf entry followed by sudo systemctl restart systemd-binfmt.^[2] Then pull the Arm image explicitly: sudo docker pull --platform=linux/arm64 nvcr.io/nvidia/doca/doca:devel-3.5.0.^[2]

The limit is stated on the same page: the QEMU environment “emulates an aarch64 architecture, but it does not emulate the hardware devices present on the BlueField Platform”, so “the tested program will not be able to access the devices needed for its successful execution, thus mandating that the testing is done on top of a physical BlueField”.^[2] Treat the emulated container as a compiler with the right sysroot and nothing more.

The other route is a cross toolchain. The applications tree exposes the meson option enable_cross_compilation_to_dpu (default false).^[8] When it is set, applications/meson.build adds -I/root/doca-cross/usr/include with the comment “Please update this folder if the base cross-compilation folder is located elsewhere”, and warns that the DPACC tool “is not support during cross compilation”.^[9] The public 3.5.0 pages fetched for this lesson do not document a doca-cross package or a meson cross file, so treat anything beyond that include path as unverified and prefer the QEMU route, which is documented end to end.^[9][2]

4The install tree and pkg-config

DOCA package assets live under /opt/mellanox/doca/ on both the BlueField and the host.^[5] Libraries are at /opt/mellanox/doca/lib/<arch>-linux-gnu/libdoca_<library>.so, with the trace flavor under a trace/ subdirectory.^[10][4] Headers resolve through pkg-config: pkg-config --cflags doca-<library> yields -I/opt/mellanox/doca/infrastructure/include.^[3] The .pc files sit under /opt/mellanox/doca/lib/<arch>-linux-gnu/pkgconfig/ on DOCA 3.3 and later, or under /opt/mellanox/doca/infrastructure/lib/pkgconfig/ on legacy and split-profile installs; whichever directory holds them must be on PKG_CONFIG_PATH. Locate the live one first with find /opt/mellanox/doca -name doca-common.pc -print -quit, verify with pkg-config --list-all | grep -i doca, and read Package 'doca-flow' was not found as either a wrong install profile or a missing path entry.^[11][4]

The paths in the table come from the quick start guide and the skills notes.^[5][10] doca_caps is off PATH by default in DOCA 3.3 and later, so call it as /opt/mellanox/doca/tools/doca_caps --version or --list-devs.^[11] Never edit anything under /opt/mellanox/doca/lib*/; switch between release and trace flavors with LD_LIBRARY_PATH or by linking the -trace pkg-config module.^[10]

5Build with meson and ninja; reserve hugepages

The toolchain baseline on Ubuntu or Debian is build-essential, meson, ninja-build, cmake, pkg-config, libjson-c-dev and liblz4-dev; RHEL uses the Development Tools group with meson, ninja-build, cmake, pkgconfig, json-c-devel and lz4-devel.^[3] The repository splits into samples/ (“simplistic code snippets that demonstrate the API usage”) and applications/ (“Advanced samples that implements a logic that might cross different SDK libs”), and its README tells you to install the doca-all profile.^[6] Building every application is cd doca-samples/applications && meson /tmp/build && ninja -C /tmp/build, producing /tmp/build/<application_name>/doca_<application_name>; the default buildtype is debug, unlike the shipped release binaries, and meson /tmp/build -Denable_trace_log=true enables DOCA_LOG_TRC messages.^[6]

Each sample owns a small meson.build. The 3.5.0 dma_local_copy one reads:

project('DOCA_SAMPLE', 'C', 'CPP',
	version: run_command(find_program('cat'),
		files('../../../VERSION'), check: true).stdout().strip(),
	license: 'BSD-3',
	default_options: ['buildtype=debug'],
	meson_version: '>= 0.61.2'
)
sample_project_arguments += ['-DDOCA_ALLOW_EXPERIMENTAL_API']
sample_project_arguments += ['-DDOCA_ALLOW_DEPRECATED_API']
# Required for all DOCA programs
sample_dependencies += dependency('doca-common')
# The DOCA library of the sample itself
sample_dependencies += dependency('doca-dma')
# Utility DOCA library for executables
sample_dependencies += dependency('doca-argp')

The sources list is dma_local_copy_sample.c, dma_local_copy_main.c, ../dma_common.c and ../../common.c, and the target is executable('doca_' + SAMPLE_NAME, ...).^[7] DPDK-based samples additionally need hugepages: echo '<count>' | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages, verified by mount | grep huge.^[3] Inside a container on a machine without a NIC “DPDK / DOCA calls that need real hardware will fail at runtime”; the build still succeeds.^[3]