NUMA, data validation, and doca_perftest
S3·E2The first word on the line · Dell lab, Round Rock, second shift, PoC report due Thursday
Builds on: What ib_write_bw actually measures
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Analyse a perftest result header to say whether the run was NUMA-bound automatically by external affinity or not bound at all.
- Design a three-way NUMA A/B that isolates the cost of crossing the socket interconnect.
- Interpret a --data_validation line and separate a real defect from normal high-speed behaviour.
- Contrast doca_perftest with perftest on scope orchestration and start synchronisation.
Episode 2 — The first word on the line
The night-shift operator meets you at the door with a label maker in his hand and fresh tape on the bench node: BENCH-A — ASK FIRST. On the screen behind him your own runbook line returns 148 Gb/s, then 191, then 150, and the day-shift SE has typed “fabric instability” into Thursday’s draft, colouring a row in his promise spreadsheet. Both command lines match. Both perftest builds match. The fabric is the only suspect left on the list, which is exactly the moment to stop trusting the list.
The reason this lesson exists is that perftest changed underneath everybody. It now automatically detects the NUMA node of the IB device and binds the benchmark thread and memory allocations to that node, and the active NUMA node is reported in the test output header.[1] Placement used to be an accident of the scheduler; now it is a property of the run. And the binding stands down politely: perftest respects pre-existing CPU affinity set by external tools such as taskset or numactl, and if an external restriction is detected the automatic binding is skipped.[1]
You open the wrapper script the customer actually runs. The first word on the line is numactl. The header of every result names no node at all, and the operator, reading over your shoulder, offers to print a label for that too.
Read the header before the number: a run that will not say where it ran is not a measurement.
Segment 1 starts there.
1perftest now places itself, and that changes your comparisons
Current perftest “automatically detects the NUMA node of the IB device and binds the benchmark thread and memory allocations to that node”; “the active NUMA node is reported in the test output header”.[1] This is new behaviour, and it silently changes results relative to older runs on the same hardware.
Three rules govern it. If the NIC’s NUMA node cannot be determined, “perftest continues without binding” — which is exactly what happens on a veth-backed Soft-RoCE device whose numa_node reads -1.[1] If an external affinity already exists, perftest stands down: it “respects pre-existing CPU affinity set by external tools (e.g. taskset or numactl). If an external restriction is detected, automatic NUMA binding is skipped.”[1] And NUMA binding is “Not supported for raw Ethernet benchmarks”.[1]
The manual controls are --pin_cores=<cpu_list>, which takes taskset syntax such as 0, 0,2,4, 0-3 or 0-3,8,12-15; --numa_node=<id>, which needs libnuma and “is mutually exclusive with both --pin_cores and --disable_numa (rejected at parse time)”; and --disable_numa, which restores “the original behavior (OS scheduler decides placement)”.[1][2]
The host-side fact underneath all of this is one file: /sys/class/net/<if>/device/numa_node names the socket the NIC hangs off, and it must match the socket the ranks run on.[1]
ib_write_bw -d mlx5_0 -x 3 -s 65536 -q 8 --report_gbits --numa_node=0
ib_write_bw -d mlx5_0 -x 3 -s 65536 -q 8 --report_gbits --numa_node=0 10.0.0.1
perftest "is a collection of tests written over uverbs intended for use as a performance micro-benchmark". Quote this when a customer treats a result as an application SLA: "The benchmarks generate a synthetic stream of operations… The benchmarks are not designed to emulate any real application traffic."
They are "usually installed to the /usr/bin/" and "may need to be ran with sudo when running from non root". A cryptic syndrome string on stderr comes from rdma-core, not from perftest.
perftest README (master)- warnNeither --tclass nor -T: frames carry ToS 0 and land in the default, usually lossy, queue. perftest(1) man page (master)
- noteUnidirectional: "the client measures the bandwidth" — the server screen is not the result. perftest README (master)
2A NUMA A/B that survives review
A NUMA claim needs three numbers, not two. Read the NIC’s node from /sys/class/net/<if>/device/numa_node.[1] Then run the identical benchmark line three times, changing only placement: bound to the NIC’s node with --numa_node <n>, unbound with --disable_numa, and deliberately bound to the far node with --numa_node <other>.[1][2] The third arm is what turns “NUMA matters” into a number: it is the cost of crossing the socket interconnect on this specific machine, measured rather than asserted.
Keep everything else identical, because the placement flags do not exempt you from the rules of the previous lesson: the same options on both sides, the same perftest build on both hosts, an explicit -s and -q, and --report_gbits.[1][2] Record the NUMA line from each output header alongside each number; that header line is the only proof of which arm produced which figure.[1]
The honest limit: this measures the host. It says nothing about the fabric, and a NUMA-aligned run that is still slow has simply eliminated one cause. That is worth saying out loud in the report.
3Data validation: proving bytes, not just bandwidth
--data_validation checks “that every byte matches the expected pattern, detecting corruption, DMA races, and stale data during RDMA WRITE or READ operations”.[1] It is a correctness tool wearing a benchmark’s clothes, and it is worth reaching for when a customer reports intermittent bad results rather than slow ones.
The constraints are narrow: RC only; WRITE or READ bandwidth tests only, so ib_write_bw or ib_read_bw; tx_depth >= 32 and identical on both sides; host memory or --use_cuda.[1][2] It is incompatible with -a, --post_list > 1, --run_infinitely, --mr_per_qp, --gpu_touch, --use-null-mr, --payload_file_path and any non-RC connection type.[1][2] Losing -a matters: you validate one size at a time, so pick the size the customer’s workload uses.
Reading the output is where people get it wrong. The counters are errors, races, skips and retries; only errors is a defect, while “races, skips, retries are informational and reflect normal behavior at high speed”.[1] A passing line looks like VALIDATION: PASSED - 12500 chunks, 102400000 bytes [races=3, retries=7, skips=42].[1] In GPU mode two failure strings are worth recognising because neither is a data-integrity finding: “Failed to load kernel plugin library”, which means libperftest_kernels.so is not on the library path, and “Failed to create validation context”.[1] Huge pages help the validation path; the README’s own recipe is echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages.[1]
Pair validation with a counter. rx_icrc_encapsulated is the mlx5 RoCE ICRC failure counter — corruption the Ethernet FCS did not catch — and it lives in the driver’s hw_counters set.[5] The hw_counters directory itself is documented in the kernel ABI as an optional subdirectory “under either the parent device or the port subdirectories or both”, and its lifespan file “is in milliseconds and defaults to 10”, settable 0–10000, which is why a fast polling script can appear to see frozen values.[4]
4doca_perftest: the cluster-scoped successor
DOCA Perftest is “a native implementation designed for modern data centers” and is positioned as a replacement for “legacy RDMA benchmarking tools (e.g. ib_write_bw, ib_send_lat)”.[3] The differences NVIDIA claims are all about scope rather than about the per-link number: cluster-wide instead of point-to-point; built-in orchestration from a single host; native multi-process concurrency; “Hardware-aligned (Synchronized start/stop)” instead of loose serial starts; and automatic cluster-wide result aggregation.[3]
Basic syntax is doca_perftest -d mlx5_0 -n <server-hostname>, with -m bw|lat for mode, -v write|send|read|atomic_fa for the verb, -s <bytes>, -D <sec>, -b for bidirectional, --traffic_pattern, -f <json scenario file>, and -N/-C for process and core selection.[3] Traffic patterns are all_to_all (a2a), bisection (b), ring (r), one_to_one (o2o), one_to_many (o2m) and many_to_one (m2o); multi-node orchestration is via MPI.[3] Prerequisites: libibverbs from rdma-core is mandatory, while OpenMPI, DOCA Verbs, DOCA GPUNetIO and the CUDA Toolkit are optional.[3] Supported operations at DOCA 3.5.0 are Write, Read, Send, Write Immediate and the atomics (fetch-and-add, compare-and-swap).[3]
The reason synchronised start matters is not aesthetics. A perftest pair started by hand across many nodes measures partly overlapping runs, so an all-to-all pattern assembled from serial perftest launches understates congestion — every pair gets some of the fabric to itself. That is the class of measurement doca_perftest exists to fix, and it is the honest answer when a customer asks why their eight two-node perftest runs looked fine and the real job did not.
A Dell customer reports ib_write_bw at 148 Gb/s on a 200G-class link, intermittently. They have already sent both command lines and both perftest versions, and they match.
- Establish where the NIC lives.
cat /sys/class/net/ens1f0/device/numa_nodereturns1. That is the socket the card is attached to.[1] - Read their result header. If it names a NUMA node, auto-binding ran. If it names none, either the node was undeterminable or an external
taskset/numactlsuppressed binding.[1] - Arm A, aligned:
ib_write_bw -d mlx5_0 -x 3 -s 65536 -q 8 -D 30 -f 2 --report_gbits --numa_node 1on both sides.[2] - Arm B, unbound control: identical line with
--disable_numainstead.[1] - Arm C, deliberately misaligned: identical line with
--numa_node 0. Do not combine it with--pin_cores; the pair is rejected at parse time.[1] - Now test integrity, not speed, at the size that matters:
ib_write_bw -d mlx5_0 -x 3 -s 65536 -q 8 -t 32 --data_validation -D 120on both sides, with-t 32identical on both.[1][2] Read onlyerrors; noteraces,retriesandskipsand say in the report that they are normal.[1] - Bracket the validation run with
rx_icrc_encapsulatedfromhw_countersto separate wire corruption from anything else.[5]
Acceptance: three bandwidth figures each with its NUMA header line, one validation line with errors=0 or a defect, and one ICRC counter delta.
- The NIC’s socket is read from
/sys/class/net/<if>/device/________. - The header line of the result names the active ________ node when binding ran.
- Arm A binds with ________; arm B is the control using ________; arm C forces the ________ node.
--numa_nodeis mutually exclusive with ________ and ________.- Validation requires connection type ________, a bandwidth ________ or ________ test, and
-tat least ________ identical on both sides. - In the validation summary the only defect counter is ________.
- The counter that catches corruption the Ethernet FCS missed is ________.
A customer’s ib_write_bw result header shows no NUMA line at all, and their run script starts the benchmark with numactl --cpunodebind=0 --membind=0. The NIC’s device/numa_node reads 1. Write the three-step investigation and predict, before running it, which of your three arms will be fastest and why. Acceptance: you explain why auto-binding did not appear in the header, you name the flag combination that is rejected at parse time so you do not propose it, you state the prediction as a prediction, and you name the single line of evidence that would falsify it.
Three arms and a validation run
Three runs, one thing changed each time: bound to the NIC’s own node, unbound with --disable_numa, then deliberately forced to the far socket.[1] The gap between best and worst is wider than the night-to-night wobble the draft called instability, and it is a measured property of that machine rather than an assertion. Then one --data_validation run at the size the workload uses: errors=0, with races and retries recorded and labelled informational.[1] What you tell the customer: the fabric never moved, the numactl in your wrapper did. Which holds until the platform lead runs the same steady line out of GPU memory and gets a fraction of the number.
Lab
Dell-lab ConnectX pair. Read-only: no firmware, mode or QoS changes; only benchmark placement flags.
- Pre-flight inventory. On both hosts:
cat /sys/class/net/<if>/device/numa_node,lscpu | grep -E 'NUMA node\(s\)|NUMA node[0-9]',ibdev2netdev,ibv_devinfo -d mlx5_0 | grep -E 'active_mtu|link_layer', and the perftest version. Record the NIC’s node per host. - Arm A, aligned. Both sides:
ib_write_bw -d mlx5_0 -i 1 -x <v2 gid index> --tclass=104 -s 65536 -q 8 -D 30 -f 2 --report_gbits --numa_node <nic node>. Expected: a Gb/sec figure plus a NUMA line in the header naming that node. - Arm B, unbound control. Same line with
--disable_numain place of--numa_node. Expected: no binding; the OS scheduler decides. If this equals arm A, say so — it means the scheduler happened to place well, not that placement is irrelevant. - Arm C, deliberately misaligned. Same line with
--numa_node <the other node>. Expected: the lowest of the three. The A-minus-C gap in Gb/s is the socket-interconnect penalty on this machine; report it as a measured number with the exact command lines. - Integrity run. Both sides:
ib_write_bw -d mlx5_0 -x <v2 gid index> -s 65536 -q 8 -t 32 --data_validation -D 120 --report_gbits. Expected:VALIDATION: PASSEDanderrors=0. Iferrorsis non-zero, stop treating this as a performance ticket. - Bracket it with the corruption counter. Before and after step 5:
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rx_icrc_encapsulated. Expected: unchanged. A rising value points at optics, DACs or the PCIe slot before anything in QoS. - If a counter looks frozen, check
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/lifespan— the refresh window defaults to 10 ms and accepts 0–10000. Leave it as found. - Deliverable. A one-page finding with three placement numbers, their NUMA header lines, the validation summary, the ICRC delta, and one sentence naming what has been eliminated as a cause.
Soft-RoCE pair in ~/containerlab. Everything here is read-only apart from one optional hugepage setting, whose rollback is named.
- Ask the device where it lives.
cat /sys/class/net/eth1/device/numa_node. Expected on a veth-backed lab node:-1, or the file is absent. If so: record it. That is why perftest will print no NUMA line here — with no determinable node it “continues without binding”. - Confirm the negative result. Run
ib_write_bw -d rxe0 -s 65536 -q 4 -D 20 -f 2 --report_gbitson both sides and inspect the output header for a NUMA line. Expected: none. If a line appears, your lab node has real NUMA topology and steps 3–5 become meaningful rather than illustrative. - Prove the mutual exclusion. Run
ib_write_bw -d rxe0 --numa_node 0 --pin_cores 0-3. Expected: rejected at parse time. Record the exact message; it is the fastest way to remember the rule. - Prove the external-affinity rule. Run
numactl --cpunodebind=0 ib_write_bw -d rxe0 -s 65536 -D 20 --report_gbitsand compare the header with step 2. Expected: automatic binding is skipped because an external restriction was detected. - Validation drill. On both sides run
ib_write_bw -d rxe0 -s 65536 -t 32 --data_validation -n 5000. Expected: aVALIDATION: PASSEDline with chunk and byte counts and possibly non-zero races, retries or skips. If it fails to start, re-read the constraint list — RC only, WRITE or READ bandwidth only,-tat least 32 on both sides. - Deliberately break a constraint. Add
-ato the step-5 line. Expected: refused as incompatible. Then try-c UC. Expected: refused, because validation is RC only. Both transcripts go in the artifact. - Optional, mutating on the lab host only:
echo 2048 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages, re-run step 5, and roll back withecho 0 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages. Record whether the run changed at all on Soft-RoCE. - Read the doca_perftest syntax without running it. Write the
doca_perftest -d mlx5_0 -n <server> -m bw -v write -s 65536 -D 30 --traffic_pattern one_to_oneline you would use, and state in one sentence what synchronized start would change about an all-to-all measurement compared with launching perftest pairs by hand.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, why perftest now binds itself to a NUMA node and what that changes about comparing today's number with one from last year.
Sources
Facts in this lesson were checked against linux-rdma/perftest master README and man/perftest.1 (re-fetched 2026-09-07); DOCA Perftest, DOCA SDK 3.5.0; Linux kernel sysfs-class-infiniband ABI. Dates are when each page was fetched.
- OFED Performance Tests README (linux-rdma/perftest master) · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- DOCA Perftest (DOCA SDK 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
- Linux kernel ABI: sysfs-class-infiniband (stable) · fetched 2026-09-07
- Linux kernel — drivers/infiniband/hw/mlx5/counters.c · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Scenario: "the 400G NIC only does 180 Gb/s"Elsewhere in this course · Same ground: perftest, numa and hw_counters
- Proving the fabric: perftest and nccl-testsInfiniBand course · Same ground: perftest, numa and result
- NUMA alignment with Topology Manager and CPU ManagerK8sNet course · Same ground: numa and scope