The counter map: sysfs, ethtool and rdma statistic
S4·E1The dashboard that swears the rail is dead · Customer network operations room, day two of the 64-node PoC
Builds on: ECN, CNP and DCQCN: closing the loop, What ib_write_bw actually measures
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Locate any RoCE counter in one of four homes and name the command that reads it.
- Explain why RoCE traffic never appears in the netdev's ordinary counters and where the bytes actually are.
- Use the hw_counters lifespan file to diagnose a counter that appears frozen.
- Enable optional congestion counters and bind a counter to one QP with rdma statistic and undo both.
Episode 1 — The dashboard that swears the rail is dead
The customer’s network lead taps the big screen twice and says the new rail is dead. Sixteen of sixty-four nodes are on the floor, the capacity they rent to train today lapses in three weeks, and the graph is flat. Two hosts behind you are finishing a benchmark that both say moved data. The Dell SE stirs a coffee he stopped drinking when it was hot.
The graph is fed from /proc/net/dev, and this is the moment RoCE’s whole reason for existing turns into a support ticket. RDMA over Converged Ethernet exists so the transport runs in the adapter instead of the host: NVIDIA states that RoCE traffic “is not shown in the associated Ethernet device’s counters since it is offloaded by the hardware and does not go through Ethernet network driver”, and that it is counted where InfiniBand traffic is counted, under /sys/class/infiniband/.[6] The MLNX_EN manual says the same thing operationally: RDMA counters bypass the kernel and “cannot be seen via standard Linux commands”.[5] The dashboard is not wrong about the netdev; it is reading a device that never sees the bytes.
The second screen is worse: a counter that has not moved in ten minutes, sampled every millisecond against a cache whose default lifespan is 10 ms.[2] Two files, two wrong stories, one room that now believes the fabric is broken. The night operator has already printed a label for the monitor: NOT THE FABRIC.
A counter you cannot name the home of is not evidence. So before anything else, the map: four homes, and one command each.
1Four homes, and why the obvious one is empty
The first thing a RoCE operator learns the hard way is that the netdev is blind. NVIDIA states it plainly: “RoCE traffic is not shown in the associated Ethernet device’s counters since it is offloaded by the hardware and does not go through Ethernet network driver. RoCE traffic is counted in the same place where InfiniBand traffic is counted; /sys/class/infiniband/<device>/ports/<port number>/counters/”.[6] The MLNX_EN manual repeats the operational consequence: RoCE counters are available only through sysfs, and RDMA counters bypass the kernel so they “cannot be seen via standard Linux commands”.[5] A monitoring stack that scrapes /proc/net/dev on a 400G RoCE host reports an idle link.[5]
So there are four homes, and each owns something the others do not.[2][3][4]
| Home | Path or command | Owns |
|---|---|---|
| IB port counters | /sys/class/infiniband/<dev>/ports/<p>/counters/ |
Byte and packet totals, port_xmit_wait, port_xmit_discards, link_downed[2] |
| RoCE hardware counters | /sys/class/infiniband/<dev>/ports/<p>/hw_counters/ |
Transport errors, RNR, retries, ICRC, the DCQCN four[1][2] |
| Ethernet counters | ethtool -S <netdev> |
Per-priority pause and discard, physical port, PCIe health[3] |
| rdma statistic | rdma statistic show link <dev>/<port> |
The same hardware counters plus optional cc_* counters and per-QP attribution[4] |
Before any of it you need the mapping between the two device names, because half the commands take mlx5_0 and the other half take eth2. ibdev2netdev prints exactly that: mlx5_0 port 1 <===> eth2.[6]
Read order: rping smoke test → ethtool -S per-priority on BOTH hosts → the switch's PFC/ECN counters → hw_counters → a packet capture, last.
No symptom selected. The board above still works as a map: pick a machine, pick a counter home, open any counter.
rp_cnp_handled / rp_cnp_ignored increment here. This is where the rate is reduced.
RoCE counters are read on both hosts in the same window, with timestamps. A single host's dump cannot separate “the receiver never reported” from “the sender never reacted”.
2hw_counters: the ones that end arguments
The hw_counters directory is optional in the sysfs ABI and may appear “under either the parent device or the port subdirectories or both”; the kernel notes that these “may match some of the counters in the counters directory, but they often include many other counters”.[2] On mlx5 that “many other” is the whole RoCE diagnostic surface, and the authoritative list of names is the in-tree driver, not a vendor page.[1]
Group them the way you will use them:[1]
- Fabric loss and reordering:
packet_seq_err,out_of_sequence,implied_nak_seq_err,duplicate_request,local_ack_timeout_err,req_transport_retries_exceeded. - Receiver starvation:
out_of_buffer,rnr_nak_retry_err,req_rnr_retries_exceeded. - Verbs-level errors mirrored in hardware:
req_cqe_error,resp_cqe_error,req_remote_access_errors,resp_local_length_error, plus the flush variants. - Congestion, the DCQCN four:
np_ecn_marked_roce_packets,np_cnp_sent,rp_cnp_handled,rp_cnp_ignored. - Physical corruption:
rx_icrc_encapsulated, RoCE ICRC failures that the Ethernet FCS did not catch. - Adaptive retransmission and slow restart:
roce_adp_retrans,roce_adp_retrans_to,roce_slow_restart,roce_slow_restart_cnps,roce_slow_restart_trans.
The np_ prefix means notification point and the rp_ prefix means reaction point, so the first two of the DCQCN four increment on the receiver and the last two on the sender.[1] NVIDIA’s own prose definitions for those counters are not on a page that can be fetched: the enterprise-support article is JavaScript-rendered and the blog that mirrors the wording returns HTTP 403 to automated clients, so treat any quoted sentence about them as unverified and read the semantics off the np_/rp_ roles instead.[9] The Windows counter page is the one NVIDIA-hosted place where equivalent names appear, as “CNPs Sent Successfully”, “RoCEv2 DCQCN Marked Packets”, “Ignored CNP Packets” and “Successfully Handled CNP Packets”.[8]
One file in that directory is not a counter. lifespan is a cache control: “The lifespan is in milliseconds and defaults to 10 unless set to something else by the driver. Users may echo a value between 0-10000 to the lifespan file.”[2] A collector polling every millisecond re-reads the same cached snapshot and reports a frozen counter.[2]
3ethtool -S: pause, discards and the physical layer
Everything about PFC lives on the Ethernet side, per priority p from 0 to 7.[3] The kernel documentation is unusually direct about what each one means, and two definitions carry the whole direction rule.[3]
rx_prio[p]_pause is “the number of pause packets received with priority p on a physical port. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from the adapter”, and it is “available only if PFC was enabled on priority p”.[3] tx_prio[p]_pause is “the number of pause packets transmitted on priority p on a physical port. If this counter is increasing, it implies that the adapter is congested and cannot absorb the traffic coming from the network.”[3] Receiving pause means you are being paused; transmitting pause means you are pausing someone else.[3]
The rest of the group is the supporting cast: rx_prio[p]_pause_duration in microseconds, rx_prio[p]_pause_transition counting Xoff-to-Xon transitions, rx_prio[p]_buf_discard for packets dropped “due to lack of per host receive buffers”, rx_prio[p]_cong_discard, and rx_prio[p]_marked for “packets ecn marked by device due to per host congestion”.[3]
Three more that decide arguments quickly:[3]
rx_pause_ctrl_phyandtx_pause_ctrl_phyare the link-layer pause counters, defined without reference to priority. The pair that settles the argument isrx_global_pauseandtx_global_pause, which the kernel documents as “only enabled when global pause mode is enabled”, against the per-priority counters, which are “available only if PFC was enabled on priority p”.[3] Global-pause counters present and norx_prio*_pausecounters at all is the documented signature of 802.3x pause rather than PFC.tx_pause_storm_warning_eventsandtx_pause_storm_error_eventsrecord a device that sent pause for so long that pause transmission was disabled by timeout, “on the period where pause frames were disabled, drop could have been” occurred. A lossless design that reaches this counter has already stopped being lossless.rx_pci_signal_integrityandtx_pci_signal_integrityare physical PCIe errors; the kernel doc’s own advice is to move the adapter to a different slot and to validate firmware and server BIOS, which is exactly the Dell escalation script.
Read order: rping smoke test → ethtool -S per-priority on BOTH hosts → the switch's PFC/ECN counters → hw_counters → a packet capture, last.
PFC “is configured” but the whole link stalls
“We turned PFC on and now everything stalls, not just the storage traffic.”
- 1Sender
ethtool -S enp1s0f0 | grep -E "pause_ctrl_phy|global_pause"
What would you expect to see? Commit your diagnosis to reveal.
- 2Sender
ethtool -S enp1s0f0 | grep rx_prio3_pause
What would you expect to see? Commit your diagnosis to reveal.
- 3Switch
nv show interface swp1 qos roce counters
What would you expect to see? Commit your diagnosis to reveal.
rp_cnp_handled / rp_cnp_ignored increment here. This is where the rate is reduced.
ethtool -S <netdev> (map it: ibdev2netdev → "mlx5_0 port 1 <===> eth2") ethtool -S enp1s0f0 | grep -E "prio[0-9]_pause|discard|pause_ctrl|storm"
Per-priority pause/discard/marked counters (priority 0–7), physical-port counters and PCIe health. Nothing RoCE-specific lives here — but PFC does, and PFC is what a lossless fabric is made of.
Gotcha. rx_prio[p]_pause exists "only if PFC was enabled on priority p". A missing counter is itself the answer.
sourceRoCE counters are read on both hosts in the same window, with timestamps. A single host's dump cannot separate “the receiver never reported” from “the sender never reacted”.
4rdma statistic: per-QP attribution and optional counters
Sysfs aggregates. NVIDIA states that “the counter index is a Queue Pair (QP) attribute given in the QP context” and that several QPs may share a counter set, in which case the value you read is cumulative.[5] When a customer asks which job produced the retries, sysfs cannot answer and the iproute2 rdma tool can.[4]
The tool’s objects are qp and mr.[4] The commands you need are few:[4]
rdma statistic show # default counter state, all devices
rdma statistic show link mlx5_0/1 # one port
rdma statistic mode supported link mlx5_0/1 # what optional counters this link offers
rdma statistic set link mlx5_0/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts
rdma statistic qp bind link mlx5_0/1 lqpn 178 # attribute one QP
rdma statistic qp set link mlx5_0/1 auto type on # or bind automatically per QP type
rdma statistic qp unbind link mlx5_0/1 cntn 4 lqpn 178
rdma statistic unset link mlx5_0/1 optional-counters # disable all optional countersOptional counters are off by default, which is why cc_rx_ce_pkts (received CE-marked packets), cc_rx_cnp_pkts (received CNPs), cc_tx_cnp_pkts and the rdma_tx_packets/rdma_rx_packets pairs read zero on a fresh host and look like a broken feature.[1][4] Enabling them is the only mutating step in this lesson, and unset is its rollback.[4]
perftest will do the sysfs sampling for you across a run: -W, --report-counters=<list of counter names>, with the man page’s own example counters/port_xmit_data,hw_counters/out_of_buffer.[7] That turns “run the test, then diff two grep . snapshots” into one command, and it is the fastest honest way to attach counter evidence to a benchmark number.[7]
Goal: prove that a 10 GiB ib_write_bw run really moved 10 GiB, using the port counter rather than the benchmark’s own report.
# 1. Map the device and confirm the port is a RoCE port.
ibdev2netdev # mlx5_0 port 1 <===> eth2
cat /sys/class/infiniband/mlx5_0/ports/1/link_layer # Ethernet
# 2. Snapshot before.
BEFORE=$(cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data)
# 3. Run a fixed amount of data: 10240 messages of 1 MiB.
ib_write_bw -d mlx5_0 -x 3 -s 1048576 -n 10240 --report_gbits <server-ip>
# 4. Snapshot after and convert.
AFTER=$(cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data)
echo $(( (AFTER - BEFORE) * 4 )) # bytes on the wireReasoning: port_xmit_data is “total number of data octets, divided by 4 (lanes)”, so the delta must be multiplied by four before it means bytes.[2] The result is larger than 10 GiB because the counter includes Ethernet, IP, UDP, BTH and ICRC overhead plus the ACK stream, not just payload.[6] The same sampling can be handed to perftest instead: add -W counters/port_xmit_data to the command line.[7]
ibdev2netdev
BEFORE=$(cat /sys/class/infiniband/mlx5_0/ports/1/________/port_xmit_data)
ib_write_bw -d mlx5_0 -x 3 -s 1048576 -n 10240 --report_gbits <server-ip>
AFTER=$(cat /sys/class/infiniband/mlx5_0/ports/1/________/port_xmit_data)
echo $(( (AFTER - BEFORE) * ____ ))Fill the blanks, then answer in one sentence each: why is the delta larger than the payload you sent, and which perftest flag would have done the sampling for you?[2][7]
A customer reports that np_ecn_marked_roce_packets “never moves” on the receiver during a congested all-reduce, and their collector polls every 2 ms.
Produce the command sequence that decides, in under two minutes, whether this is a cache artefact or a real absence of ECN marking. Acceptance criteria: you read and, if needed, temporarily change lifespan within its documented range and restore it; you re-read the counter with a poll interval at least as long as the lifespan; you state which of the four counter homes you used and why; and you name the follow-up check on the receiver if the counter is genuinely flat.[2][1]
The two-minute answer
You point the collector at /sys/class/infiniband/mlx5_0/ports/1/hw_counters/, raise lifespan above the poll interval so consecutive samples stop hitting the same cached snapshot, and enable the optional cc_* counters through rdma statistic so the congestion half of the picture is not permanently zero.[2][4] The graph fills in during the next run. The network lead copies the four paths into his notebook, one per line.
What you say to the customer: “Your fabric was never idle. Your collector was reading the one file that RoCE deliberately never touches.”[5][6]
Then the retry counters, visible for the first time all week, start climbing on eight nodes. At 01:40 the phone rings.
Lab
On the Dell-lab ConnectX host. Steps 1-3 and 7 are read-only; steps 4, 5 and 6 change host state and each names its rollback. Nothing here touches firmware or card mode.
- Pre-flight inventory, all read-only:
ibdev2netdev;ibv_devinfo -d mlx5_0;ofed_info -s;ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ | sort | tee ~/hw_counters.list. Expected:link_layer: Ethernet, and a list of counter names. This list is the artefact NVIDIA’s manuals do not publish, so keep it with the date and the driver version.[1][2] - Snapshot every value before load:
grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/* > ~/hw_before.txtandethtool -S <netdev> > ~/eth_before.txt. Ifgrepreturns permission errors, run as root. - Create real congestion between the two lab hosts:
ib_write_bw -d mlx5_0 -x <gid> -b -q 8 -D 30 --report_gbits <peer-ip>, with the identical flags on both sides. Expected: bidirectional traffic for 30 s.[7] Take the same two snapshots afterwards and diff them. - Optional counters (mutating):
rdma statistic mode supported link mlx5_0/1, thensudo rdma statistic set link mlx5_0/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts, re-run step 3, and read them withrdma statistic show link mlx5_0/1. Rollback:sudo rdma statistic unset link mlx5_0/1 optional-countersreturns the port to its default counter state.[4] - Per-QP attribution (mutating, same rollback family): start a long run, find a local QP number with
rdma resource show qp link mlx5_0/1, thensudo rdma statistic qp bind link mlx5_0/1 lqpn <n>and readrdma statistic qp show link mlx5_0/1. Rollback:sudo rdma statistic qp unbind link mlx5_0/1 cntn <c> lqpn <n>.[4] - Cache check (mutating, trivially reversible):
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/lifespan, thenecho 0 | sudo tee .../lifespan, pollnp_ecn_marked_roce_packetsat 1 ms during a congested run, and compare with the behaviour at the default. Rollback: write the original value back.[2] - Write the deliverable: a one-page table mapping each of the ten counters you saw move to its home, its command and the machine it moved on. That table is what you hand a customer.
Soft-RoCE (rdma_rxe) on two containers or two network namespaces. The point of this lab is the inventory: which counters the software device does not have.
- Create the device and confirm it exists:
sudo rdma link add rxe0 type rxe netdev eth0thenrdma link show. Expected:link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0. If not: therdma_rxemodule is missing (sudo modprobe rdma_rxe) or the netdev name is wrong. - Inventory the two sysfs trees:
ls /sys/class/infiniband/rxe0/ports/1/counters/ /sys/class/infiniband/rxe0/ports/1/hw_counters/ 2>&1. Expected: acounters/directory;hw_counters/may be absent or much shorter than on mlx5. If not present at all, record that, because the ABI markshw_countersas optional.[2] - Write down every mlx5 counter from Segment 2 that has no rxe equivalent. That list is the lesson: the DCQCN four,
rx_icrc_encapsulatedand theroce_adp_*family are hardware features, not protocol features.[1] - Ask the tool what it supports:
rdma statistic showandrdma statistic mode supported link rxe0/1. Expected: a counter list; an empty or unsupported answer for optional counters is a valid result to record.[4] - Run a fixed-size transfer with counter sampling. Server:
ib_write_bw -d rxe0 -s 65536 -n 2000. Client:ib_write_bw -d rxe0 -s 65536 -n 2000 -W counters/port_xmit_data <server-ip>— identical-sand-non both sides, since perftest requires mode options to match, and-Wonly where you want the report. Expected: perftest prints the counter delta alongside the bandwidth line.[7] If the flag is rejected, your perftest predates it; fall back to manual before/after snapshots. - Reconcile by hand: multiply the
port_xmit_datadelta by four and compare with65536 * 2000. Expected: the counter total exceeds the payload total. Explain the gap in one sentence.[2]
Rollback: sudo rdma link delete rxe0 removes the software device; nothing else in this variant changes host state.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in five sentences, why their Grafana dashboard shows zero RoCE traffic on a link that is running at 380 Gb/s, and where to point the collector instead.
Sources
Facts in this lesson were checked against Linux in-tree mlx5_ib counters.c and sysfs-class-infiniband ABI, rdma-statistic(8), mlx5 ethtool counters kernel doc, MLNX_OFED 24.10-5.1.6.1 LTS and DOCA-Host RoCE pages, fetched 2026-09-07. Dates are when each page was fetched.
- Linux kernel: drivers/infiniband/hw/mlx5/counters.c · fetched 2026-09-07
- Linux kernel ABI: sysfs-class-infiniband (stable) · fetched 2026-09-07
- Ethtool counters (mlx5): the Linux kernel documentation · fetched 2026-09-07
- rdma-statistic(8): iproute2 rdma tool man page · fetched 2026-09-07
- Ethernet Interface: MLNX_EN 5.8-4.1.5.0 LTS · fetched 2026-09-07
- RDMA over Converged Ethernet (DOCA-Host): generated PDF · fetched 2026-09-07 · DOCA 3.5.0
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- Adapter Cards Counters: NVIDIA WinOF-2 26.4.50010 · fetched 2026-09-07
- RoCEv2 Congestion Counters Explained (Oracle Linux blog, HTTP 403 to automated fetch) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Counters and errors that mean somethingInfiniBand course · Same ground: rollback, physical and hw_counters
- Reading a host's IB state in 60 secondsInfiniBand course · Same ground: sysfs, tools and hw_counters
- ECN, CNP and DCQCN: closing the loopElsewhere in this course · Same ground: dcqcn, sysfs and hw_counters