Debugging and measuring with DOCA Bench

Episode 5 — Thirty percent, and a quote due tomorrow

The situation · Escalation bridge, dialed in from the lab, 21:10

Eleven people are on the bridge and only one number is being discussed: the archive’s perf lead says their BlueField-3 does thirty percent less DMA throughput than the figure in NVIDIA’s material. The procurement person, who in five weeks has asked about nothing but lead time, is on the call because the quote goes out in the morning and the PoC report is due Friday. Two people have already proposed swapping the card. Nobody has yet said what command produced the customer’s number — and the network lead, notebook three-quarters full, asks the only useful question on the bridge: show me the counter.

DOCA Bench exists so that this argument can be settled with evidence instead of seniority. It “allows users to evaluate the performance of DOCA applications, with reasonable accuracy for real-world applications”, ships under /opt/mellanox/doca/tools on both the host and the Arm image, and drives every engine through one CLI; with --csv-output-file it writes not only the stat but the configuration values used to produce it.^[5] That last property is the whole point: a run that cannot be described in flags cannot be compared.

The two most common reasons a real number comes in low are not the card. A trace-flavor library “adds runtime input-sanitation and emits additional log lines” and costs performance, and the samples build in debug by default while DOCA ships release binaries.^[3]^[2]

A number without its flags is a rumor.

So before the number, you ask about the two streams of output already scrolling on their screen.

1Two log streams, one numeric scale

Every DOCA program that uses ArgP inherits the same flags: -h/--help, -v/--version, -l/--log-level for the application, --sdk-log-level which “Sets the SDK numeric log level using the same 10-70 scale”, --log-filter which “Filters logs from specific modules (comma-separated list)”, and -j/--json to read flags from a file.^[1] The scale is 10 DISABLE, 20 CRITICAL, 30 ERROR, 40 WARNING, 50 INFO, 60 DEBUG, 70 TRACE, where TRACE “requires compilation with TRACE support”.^[1] One published NVIDIA skills file describes 30 as INFO; the Arg Parser page is the authority and says 30 is ERROR and 50 is INFO.^[3]^[1]

The two streams come from two backends. dma_local_copy_main.c registers doca_log_backend_create_standard() for the sample’s own DOCA_LOG_* calls, then doca_log_backend_create_with_file_sdk(stderr, &sdk_log) and doca_log_backend_set_sdk_level(sdk_log, DOCA_LOG_LEVEL_WARNING) so SDK internals start at WARNING; modules are named with DOCA_LOG_REGISTER(DPU_LOCAL_DMA_COPY::MAIN), which is what --log-filter matches on.^[6] ArgP itself has a fixed sequence: doca_argp_init() “must be the first function called”, then doca_argp_param_create with the short name, long name, description, callback and type setters and doca_argp_register_param, then doca_argp_start(argc, argv) and doca_argp_destroy().^[1] A running program can also move its limits at runtime; the log_limits_server sample exposes doca_log_level_set_global_lower_limit and doca_log_level_set_global_upper_limit, with its client documenting “60 = DEBUG” and “20 = CRIT”.^[9] A DOCA_LOG_LEVEL environment variable is honored only by “programs that read it”, so do not assume it.^[3]

2TRACE needs a trace build

The 70 level is a compile-time feature. The samples README explains that enabling TRACE “activates various developer log messages left out of the release compilation”, switched on with enable_trace_log in meson_options.txt or on the command line as meson /tmp/build -Denable_trace_log=true.^[2] In applications/meson.build that option becomes add_project_arguments('-DDOCA_LOGGING_ALLOW_TRACE', ...).^[11] The same README reminds you that the default meson buildtype is debug and the binaries DOCA ships are release.^[2]

The libraries have their own flavor. The setup notes describe a trace flavor under /opt/mellanox/doca/lib/<arch>-linux-gnu/trace/ that “adds runtime input-sanitation and emits additional log lines at DEBUG / TRACE level”, while the release flavor “does not emit those lines no matter how high you set --sdk-log-level”.^[4]^[3] You select it either by linking pkg-config doca-<library>-trace instead of doca-<library>, or at run time with LD_LIBRARY_PATH=/opt/mellanox/doca/lib/<arch>-linux-gnu/trace:$LD_LIBRARY_PATH; pkg-config --exists doca-flow-trace tells you whether the flavor is installed.^[3]^[10] Never edit the files under /opt/mellanox/doca/lib*/ to swap flavors.^[10] The trace flavor has measurable overhead, so switch back to release before any performance measurement.^[3]

3The debug ladder

The debugging notes order the layers install, version, build, link, runtime, program, driver and firmware, with one rule: “always start at the lowest layer the symptom is consistent with”.^[3] For a program that already builds and links, this course folds them into five rungs. Environment: pkg-config --modversion doca-<library> and /opt/mellanox/doca/tools/doca_caps --version on both sides, because headers from one DOCA version against another runtime “returns DOCA_ERROR_INVALID_VALUE from a call that should never fail”.^[3]^[8] Device: doca_caps --list-devs, ip link show, devlink dev show, dmesg | tail -200, journalctl --since "5 min ago"; the sample helper’s DOCA_ERROR_NOT_FOUND with “Matching device not found” lives here.^[3] Capability: a doca_<library>_cap_* query or DOCA_ERROR_NOT_SUPPORTED, which the Core guide attributes to an API “not implemented for that processor”.^[7]^[3] State: DOCA_ERROR_BAD_STATE is a call-order bug, DOCA_ERROR_IN_PROGRESS from a stop is normal, and --sdk-log-level 60 with a state-changed callback shows the transitions.^[3]^[7] Task: the error callback’s doca_task_get_status, the flushed-versus-real distinction, and doca_task_try_submit to validate inputs during development.^[9]

Before any change, collect the read-only triple: the program’s output with --sdk-log-level 70 on a trace build, the system view from dmesg and journalctl, and the DOCA view from doca_caps --list-devs and pkg-config --modversion.^[3] Two cautions from the same notes: attaching gdb pauses the process, which trips watchdogs on server workloads, and inside the NGC container dmesg and journalctl show only the container’s view.^[3] Never retry-loop INVALID_VALUE, BAD_STATE, NOT_SUPPORTED, INITIALIZATION or DRIVER; a task failure that moves the context to Stopping must be drained by progressing the PE before restart.^[3]^[7] When the ladder ends in the driver or firmware, escalate through the DOCA developer forum with the triple attached.^[3]

Symptom

Dev / environment: build, capabilities, performance

Start at the device rung: the tree asks what doca_caps --list-devs shows and branches down to environment or up to capability, state and task.

4DOCA Bench: one CLI for every engine

“NVIDIA DOCA Bench allows users to evaluate the performance of DOCA applications, with reasonable accuracy for real-world applications”; it is installed under /opt/mellanox/doca/tools on both DOCA-Host and the BlueField Arm image and requires DOCA 2.7.0 or later.^[5] The device is --device or -A, given as a PCIe address such as 03:00.0, an IB name such as mlx5_0, or an interface name such as ens4f0; from the Arm side target the local PCIe address.^[5] The work is --pipeline-steps, a comma-separated list drawn from doca_compress::compress, doca_compress::decompress, doca_dma, doca_ec::create, doca_ec::recover, doca_ec::update, doca_sha, doca_rdma::send, doca_rdma::receive, doca_rdma::bidir, doca_aes_gcm::encrypt, doca_aes_gcm::decrypt, doca_cc::client_producer, doca_cc::client_consumer, doca_eth::rx, doca_eth::tx and the doca_gpunetio::* steps; note that the Comm Channel name doca_cc survives in Bench.^[5]

--mode defaults to throughput, which “tries to keep each component under test as busy as possible”; bulk-latency submits batches and reports the time from the first submission to the last response in 100 buckets sized by --latency-bucket-range (for example 10us,100us; default 100ms,10ms); precision-latency “executes one job at a time” and “only supports job limited execution”.^[5] Input is --data-provider with file, file-set or random-data, sized by --uniform-job-size in bytes, with --job-output-buffer-size (default 16384) and --data-provider-job-count (default 128 per thread).^[5] Cores are --core-mask (default 0x02, at most 32 CPUs), --core-list such as 0,3,6-10, --core-count, and --threads-per-core.^[5] Limits are --run-limit-seconds/-s, --run-limit-jobs/-J, --run-limit-bytes/-b.^[5] Batching with --batch-mode and --batch-size applies only to doca_sha and doca_dma in throughput mode.^[5]

Output goes to the console and, with --csv-output-file <path>, to a CSV containing stats and the configuration values; --csv-stats "stats.*" filters columns, --csv-append-mode appends across runs, --rt-stats-interval <ms> prints transient snapshots, and --sweep runs a range of values one line per iteration.^[5] Discovery is --query device-capabilities (needs --device) and --query sweep-properties.^[5] Remote memory for DMA uses the companion agent through --companion-connection-string with --use-remote-input-buffers or --use-remote-output-buffers.^[5]

doca_bench --device 03:00.0 --pipeline-steps doca_dma \
  --data-provider random-data --uniform-job-size 4096 \
  --core-list 1-4 --threads-per-core 2 \
  --mode throughput --run-limit-seconds 10 \
  --csv-output-file /tmp/dma_throughput.csv

5Evaluating a number

Two DOCA Bench results are comparable only when the inputs that define the work match: the same --mode, the same --pipeline-steps, the same --uniform-job-size and data provider, the same --core-list or --core-mask with the same --threads-per-core, the same run limit, and the same choice of local versus remote buffers.^[5] The CSV is the evidence, because it “can contain stats and the configuration values used to produce that stat”.^[5] Then the environment: a release-flavor library and a release buildtype, since trace and debug both cost performance, and identical DOCA versions on the machines being compared.^[3]^[2]^[8] Bench itself notes that doca_dma and doca_sha “execute in constant time regardless of the input data” while compression depends on data, so data provider matters more for some steps than others.^[5]

Benchmark evaluation: worked → faded → problem

Measure DMA three ways on one device and keep every other flag constant:

B="doca_bench --device 03:00.0 --pipeline-steps doca_dma --data-provider random-data --uniform-job-size 4096 --core-list 1-4 --threads-per-core 2"
$B --mode throughput        --run-limit-seconds 10 --csv-output-file /tmp/dma_tp.csv
$B --mode bulk-latency      --run-limit-seconds 10 --latency-bucket-range 10us,100us --csv-output-file /tmp/dma_bl.csv
$B --mode precision-latency --run-limit-jobs 10000 --csv-output-file /tmp/dma_pl.csv

Reading: throughput reports bandwidth and job rate; bulk-latency adds a histogram whose first bucket is everything faster than 10us and whose last bucket is everything slower than the range; precision-latency reports the minimum and distribution per job at greatly reduced throughput. All three share device, step, job size, cores and threads, so a difference between them is the mode and nothing else. The three CSVs carry the flags, so anyone can verify that later.

Episode 5 — Case closed: the flag that explained it

How it ended

Their LD_LIBRARY_PATH pointed at a trace directory, and the binary was a default meson build — debug, not release.^[3]^[2] Rebuilt release, run against release libraries with the same job size, core list and mode, the gap closes. Both CSVs go into the PoC report.^[5]

What you say on the bridge: “Send me the command line and the CSV, not the number. Nine times out of ten it is in the configuration columns.”

The report ships Friday; the purchase order for the 32 nodes follows. The SE closes the last row of the promise spreadsheet and drinks his coffee cold. In the lab a sticker still reads START ME FIRST, and the network lead’s notebook — the counters, the two-column CPU page, the flags — is now the archive’s bring-up runbook.

Lab

Pre-flight (read-only): /opt/mellanox/doca/tools/doca_caps --version on host and Arm must match; /opt/mellanox/doca/tools/doca_caps --list-devs on the Arm for the PF address. Confirm you are using the shipped release doca_bench, not a trace build.

On the Arm side: /opt/mellanox/doca/tools/doca_bench --device <pf> --query device-capabilities — expect a per-library table including doca_dma marked capable and supported.
Throughput: doca_bench --device <pf> --pipeline-steps doca_dma --data-provider random-data --uniform-job-size 4096 --core-list 1-4 --threads-per-core 2 --mode throughput --run-limit-seconds 10 --csv-output-file /tmp/dma_tp.csv — expect a bandwidth and job-rate summary and a CSV file. If Bench reports the core list is invalid, use nproc and pick cores that exist.
Bulk latency: same flags with --mode bulk-latency --latency-bucket-range 10us,100us --csv-output-file /tmp/dma_bl.csv — expect a histogram whose buckets are 100us wide.
Precision latency: same flags with --mode precision-latency --run-limit-jobs 10000 --csv-output-file /tmp/dma_pl.csv — expect per-job minimum and percentiles; if you passed --run-limit-seconds by mistake, Bench rejects it because this mode is job-limited only.
head -3 /tmp/dma_tp.csv — expect a header row that includes configuration columns; copy the three CSVs to your notes. Log-level check: /tmp/build-dma/doca_dma_local_copy -p <pf> --sdk-log-level 60 -l 60 — expect SDK DEBUG lines interleaved with the state-change messages.
Rollback: none required; Bench generates DMA traffic against process memory and changes no device configuration. Kill a run with Ctrl-C if it exceeds its limit.

Evidence / notes (commands run, outputs, what surprised you)

Debugging and measuring with DOCA Bench

Before you read: what do you already know?

After this lesson you can