What ib_write_bw actually measures
S3·E1The 180 that was never 180 · Hotel lobby in Round Rock, 6:40 a.m.
Builds on: GIDs: how a connection picks its RoCE version, ToS, DSCP, PCP, SL: where marking gets lost
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- State what a perftest latency number and a perftest bandwidth number each measure and which side of the pair produced it.
- Assemble an ib_write_bw command line whose every flag is justified and whose server and client halves match.
- Distinguish the README defaults from the man-page defaults for -s, -n and -r and say which one you verified on the host.
- Reconcile a reported bandwidth against a sysfs counter delta using -W.
Episode 1 — The 180 that was never 180
The escalation is one line long: 400G ConnectX, ib_write_bw gives us 21,500, that is half the link. No command line. No units. In ninety minutes you and the Dell SE face an imaging-analytics customer deciding whether to buy sixty-four GPU nodes on their own RoCE fabric. Their network lead has his notebook open to a clean page. The SE sets down coffee that went cold in the car: the report is due Thursday, and nobody signs for a broken fabric.
perftest exists so that argument can be settled with evidence. It is a collection of tests written over uverbs, intended for use as a performance micro-benchmark, for hardware or software tuning as well as for functional testing.[1] It is deliberately not an application: the benchmarks generate a synthetic stream of operations and are not designed to emulate any real application traffic.[1] An instrument means nothing unless you know what it was set to, and nobody here knows.
So you write down what you know. The default report is MiB/sec unless --report_gbits was passed[2] — and 21,500 MiB/s is about 180 Gb/s, the number the email calls a failure. -q defaults to 1 and is relevant only for bandwidth[2], so this is one queue pair on one path. On a unidirectional run the client is the side that measured.[1] And the same options must be passed to both server and client.[1]
A number with no command line under it is a rumour with decimals.
In the elevator you have five questions instead of an argument. Segment 1 is those questions.
1The number is a measurement, not a property
perftest is “a collection of tests written over uverbs intended for use as a performance micro-benchmark”, used “for HW or SW tuning as well as for functional testing”.[1] That sentence is the whole scope. The tests ship in bandwidth and latency pairs — ib_send_bw/ib_send_lat, ib_read_bw/ib_read_lat, ib_write_bw/ib_write_lat, ib_atomic_bw/ib_atomic_lat — plus the raw-Ethernet variants.[1]
Four semantics decide what a number means. First, “the latency benchmarks measure round-trip time but report half of that as one-way latency”, so “the results may not be accurate for asymmetrical configurations”.[1] Second, “on all unidirectional bandwidth benchmarks, the client measures the bandwidth”; on bidirectional tests each side measures the traffic it initiates and “the server reports the result to the client, who combines them together”.[1] Third, “latency tests report minimum, median and maximum latency results” and the median “is typically less sensitive to high latency variations, compared to average latency measurement”.[1] Fourth, “typically, the first value measured is the maximum value, due to warmup effects”, which is what --perform_warm_up exists to remove.[1][2]
Timing comes from the CPU cycle counter, “to get time stamps without context switch”.[1] That is why a host whose governor is parked matters and why -F only suppresses the warning about cpufreq_ondemand rather than fixing the governor.[2]
The sentence to quote when a customer treats a perftest figure as an application SLA: “The benchmarks generate a synthetic stream of operations… The benchmarks are not designed to emulate any real application traffic.”[1]
2The flags that decide what you measured
A perftest line has three jobs: name the path, shape the traffic, and choose the report. Path: -d, --ib-dev=<dev> defaults to “first device found”, -i, --ib-port=<port> defaults to 1, and -x, --gid-index=<index> picks the GID.[2] On a RoCE host the GID index is not cosmetic — every IP creates one IB/RoCE v1 and one RoCE v2 entry, and the type at the index you pick sets the RoCE version the connection uses.[4] A run with no -x cannot be reported as RoCEv2.
Marking: --tclass=<value> “Set the Traffic Class in GRH (if GRH is in use)” and works on the default out-of-band connection path, while -T, --tos=<tos value> sets ToS “to RDMA-CM QPs” and is “available only with -R flag”.[2] The ToS byte is 8 bits and DSCP is 6, so a DSCP value is multiplied by 4: NVIDIA’s own worked example is DSCP 24 giving ToS 96.[6] On a Cumulus-default fabric RoCE data rides switch priority 3 with DSCP 26, so --tclass=104 is the marking that lands there.[5]
Shape: -c, --connection defaults to RC; -m, --mtu defaults to the port MTU and is bounded 256–4096 for RDMA tests; -q, --qp defaults to 1 and is “Relevant only for bandwidth”; -t, --tx-depth is “default 128 for bw else 1”; -s sets one size and -a, --all sweeps “from 2 till 2^23”; -D runs for a fixed number of seconds with -f, --margin defaulting to 2 s excluded at each end; -N cancels the peak calculation, which is otherwise computed “up to iters=20000”; -Q, --cq-mod moderates CQE generation.[2]
Report: --report_gbits switches MiB/sec to Gbit/sec, --out_json writes a JSON report, and -W, --report-counters=<list> diffs named sysfs counters across the run, for example counters/port_xmit_data,hw_counters/out_of_buffer.[2]
ib_write_bw -d mlx5_0 -x 3 --report_gbits
ib_write_bw -d mlx5_0 -x 3 --report_gbits 10.0.0.1
perftest "is a collection of tests written over uverbs intended for use as a performance micro-benchmark". Quote this when a customer treats a result as an application SLA: "The benchmarks generate a synthetic stream of operations… The benchmarks are not designed to emulate any real application traffic."
They are "usually installed to the /usr/bin/" and "may need to be ran with sudo when running from non root". A cryptic syndrome string on stderr comes from rdma-core, not from perftest.
perftest README (master)- warn-q defaults to 1: one flow, one ECMP path, and not enough to fill 400G. Ask for -q 4..8 before blaming the fabric. perftest(1) man page (master)
- warnNeither --tclass nor -T: frames carry ToS 0 and land in the default, usually lossy, queue. perftest(1) man page (master)
- noteUnidirectional: "the client measures the bandwidth" — the server screen is not the result. perftest README (master)
3Defaults that the documentation disagrees about
Three defaults differ between the two authoritative files in the same repository. -s, --size: the man page says “Size of message to exchange (default 65536 for bw, for lat 2)” while the README’s common-options block says the default is 1.[2][1] -n, --iters: the man page says “Number of exchanges (at least 5, default for write 5000 else 1000)” and the README says at least 100, default 1000.[2][1] -r, --rx-depth: the man page says “Rx queue size (default 512)” and the README quotes 600 in its generic block.[2][1]
The practical rule is not to memorise either. State the size and the iteration count explicitly on every line you publish, and if you must quote a default, quote the one you verified with --help on the host that produced the number. The README itself says “The default value of 1000 iterations is pretty good”, and warns that the program keeps data structures proportional to the iteration count, so a very large -n can depress the measurement it is supposed to improve.[1]
4Both sides, same options, same build
The rule appears twice — once in the README and once in the man page: the same options must be passed to both server and client.[1][2] Perftest also warns that “Different versions of perftest may not be compatible with each other. Please use the same perftest version on both sides”, and specifically that “Test version 5.3 and above won’t work with previous versions of perftest. As well as 5.70 and above”.[1] The current release tag as of 2026-09-07 is v4.5-0.20.[3] --dont_xchg_versions skips the version and MTU exchange; it hides the symptom rather than fixing it.[2]
Two more operational facts. perftest applications are “usually installed to the /usr/bin/” and “may need to be ran with sudo when running from non root”.[1] And when perftest prints a cryptic syndrome to stderr, that string comes from rdma-core, not from perftest: “Perftest may print some failures with syndroms to the stderr, perftest get those errors from rdma-core.”[1] Treat it as a verbs or driver error and go read the completion status, not the benchmark.
The habit that makes a number defensible is a second, independent measurement of the same run. -W counters/port_xmit_data prints the delta of the IB port counter across the measurement; port_xmit_data counts octets divided by four (lanes), so multiplying the delta by 4 gives bytes you can compare against the bandwidth perftest reported.[2][7]
The complaint: “ConnectX-7 400G, ib_write_bw gives us 21,500. That is half the link.”
- Fix the units before anything else. The default report is MiB/sec, and
--report_gbitsis the flag that changes it.[2] 21,500 MiB/s is roughly 180 Gb/s, so the customer is comparing a MiB/s number against a Gb/s link speed. Ask for the run again with--report_gbits. - Fix the shape.
-qdefaults to 1 and is relevant only for bandwidth; one QP is one flow.[2] Ask for-q 8. Fix the message size explicitly with-s 65536rather than relying on a default the two docs disagree about.[2][1] - Fix the path. Add
-x <v2 GID index>fromshow_gidsso the run is provably RoCEv2, and--tclass=104so it rides the priority the fabric was configured for (DSCP 26 on a Cumulus default).[2][4][5] - Make the two sides identical. Same flags on server and client, and confirm the same perftest build on both hosts.[1]
- Attach independent evidence. Add
-W counters/port_xmit_dataand report both the perftest figure and the counter delta.[2]
Server: ib_write_bw -d mlx5_0 -i 1 -x 3 --tclass=104 -s 65536 -q 8 -D 30 -f 2 --report_gbits -W counters/port_xmit_data
Client: the same line plus the server address.
Acceptance: the report states the units, the number of QPs, the message size, the GID index, the traffic class, the perftest version on both hosts, and one counter delta that corroborates the bandwidth.
- The default report unit is ________; the flag that changes it is ________.
-qdefaults to ________ and is relevant only for ________ tests.- The flag that pins the GID index is ________; the flag that marks the Traffic Class in the GRH without RDMA-CM is ________.
- The rule about the two sides is ________, and the second thing that must match across hosts is the ________.
- The flag that prints a sysfs counter delta across the run is ________.
Fill the line: ib_write_bw -d mlx5_0 -i 1 ____ 3 ____=104 -s ____ -q ____ -D 30 -f 2 ____ -W counters/port_xmit_data
A customer reports ib_send_lat at “3.1” between two hosts and says it disqualifies the fabric. Write the reply. Acceptance: you state what the 3.1 is half of, you name the one configuration difference between the two directions that would make the reported figure meaningless, you name the statistic you want instead of the one they quoted, you give the full command line you want run on both sides, and you do not assert any default value without naming where you read it.
Two rows on the screen
In the meeting you put up the same run reported twice, MiB/sec and Gb/sec[2], then the same line again with -q 8 and an explicit -s 65536, and beside both a -W counters/port_xmit_data delta taken from that very run.[2] The network lead reads it before you finish the sentence and writes four flags on his clean page. What you say is short: the link is fine, the measurement was the problem, and this command line goes in your runbook — units, queue pairs, message size, GID index, both sides. Thursday survives. Then, near midnight, the lab sends the overnight sweep: that runbook line, same pair, eight times. 148. Then 191. Then 150.
Lab
Dell-lab ConnectX pair. Read-only: no firmware, mode or QoS changes.
- Pre-flight inventory.
ibv_devinfo -d mlx5_0 | grep -E 'link_layer|active_mtu|state',show_gids mlx5_0,ibdev2netdev, andib_write_bw --version(there is noperftestcommand;-V/--versionlives on each test binary) on both hosts. Expected:link_layer: Ethernet, anACTIVEport, and at least oneRoCE v2GID row with the fabric IP. If the two hosts print different perftest versions, stop and align them before measuring anything. - Baseline. Both sides:
ib_write_bw -d mlx5_0 -i 1 -x <v2 gid index> --tclass=104 -s 65536 -q 1 -D 30 -f 2 --report_gbits(client adds the server IP). Expected: one row in Gb/sec, and a NUMA line in the header naming the node perftest bound to. - The QP sweep that answers the “only half the link” complaint. Repeat step 2 with
-q 8. Expected: a materially higher figure on a 400G-class link. If it does not move, the bottleneck is not queue-pair parallelism and you have just eliminated one cause with evidence. - The unit demonstration. Repeat step 3 without
--report_gbits. Expected: the same run reported in MiB/sec. Put both numbers side by side in the report; this is the single most common false escalation. - Independent evidence. Repeat step 3 with
-W counters/port_xmit_data,hw_counters/out_of_bufferon both sides. Expected:port_xmit_datamoves in proportion to the reported bytes andout_of_bufferstays flat on a write test. Ifout_of_buffermoves, the receiver is short of buffers and that is a different lesson. - Deliverable. A three-number report:
-q 1versus-q 8at-s 65536, plus the MiB/s versus Gb/s pair, with both full command lines and both perftest versions recorded verbatim.
Soft-RoCE pair in ~/containerlab (two Linux nodes, eth1 on the same link).
- Bring up Soft-RoCE on each node. Command:
sudo modprobe rdma_rxe && sudo rdma link add rxe0 type rxe netdev eth1. Expected:rdma linklistsrxe0/1 state ACTIVE physical_state LINK_UP netdev eth1. If not: the netdev name is wrong, orrdma_rxeis missing from the container kernel — use the host kernel module and a netns instead. Rollback when finished:sudo rdma link delete rxe0 && sudo modprobe -r rdma_rxe. - Full sweep. Server:
ib_write_bw -d rxe0 -a --report_gbits. Client:ib_write_bw -d rxe0 -a --report_gbits <server-ip>. Expected: a table from 2 bytes to 8388608 bytes with BW peak and BW average columns in Gb/sec. If not: the two sides disagree on options, or one side has a different perftest build. - Fixed-size, multi-QP, duration run. Both sides:
ib_write_bw -d rxe0 -s 65536 -q 4 -D 30 -f 2 --report_gbits(client adds the server IP). Expected: a single row and no peak column noise. Write down, in one sentence each, why the number differs from step 2 at 65536 bytes: different QP count, duration instead of iterations, and 2 s trimmed at each end. - Break it on purpose. Run the server with
-s 65536and the client with-s 4096. Record the exact error or the exact wrong result. Then repeat with-qmismatched. Expected: perftest refuses or the two sides measure different things; either way the transcript is the artifact. - Reconcile against a counter. Both sides: add
-W counters/port_xmit_datato the step-3 line. Expected: a printed delta for that counter. By hand, multiply the delta by 4 (the counter is octets divided by lanes) and compare against reported bytes. If the two disagree by more than a few percent, say which one you trust and why. - Record which fields Soft-RoCE does not populate.
ls /sys/class/infiniband/rxe0/ports/1/counters/and note what is missing relative to an mlx5 device; that list is the honest limit of this lab.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in four sentences, why a single perftest number with no command line is worthless, and what three things you would ask for instead.
Sources
Facts in this lesson were checked against linux-rdma/perftest master README and man/perftest.1 (re-fetched 2026-09-07), latest tag v4.5-0.20; Cumulus Linux 5.18 RoCE defaults; MLNX_OFED 24.10-5.1.6.1 LTS RoCE page. Dates are when each page was fetched.
- OFED Performance Tests README (linux-rdma/perftest master) · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- linux-rdma/perftest release tags (GitHub API) · fetched 2026-09-07
- RDMA over Converged Ethernet (RoCE) — MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
- RDMA over Converged Ethernet — RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- RDMA over Converged Ethernet (DOCA-Host) · fetched 2026-09-07
- Linux kernel ABI: sysfs-class-infiniband (stable) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Scenario: "the 400G NIC only does 180 Gb/s"Elsewhere in this course · Same ground: semantics, perftest and flags
- Counters and errors that mean somethingInfiniBand course · Same ground: hw_counters, flags and qos
- Proving the fabric: perftest and nccl-testsInfiniBand course · Same ground: perftest, flags and versions