Triage in order: symptom to counter to cause
S4·E201:40, and the bridge wants to change a DCQCN parameter · Night bridge call, day four of the PoC
Builds on: The counter map: sysfs, ethtool and rdma statistic, PFC, lossless and the case for lossy
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Apply a fixed read order to a RoCE complaint and justify why each step precedes the next.
- Judge from counter direction which side of a link is the victim and which is the cause.
- Distinguish congestion faults from marking faults from physical faults using one counter each.
- Evaluate a customer incident transcript and commit to a cause before proposing any configuration change.
Episode 2 — 01:40, and the bridge wants to change a DCQCN parameter
Twelve people are on the bridge and the night operator is the only one in the building. The training rehearsal has missed its throughput target three nights running, with two weeks left on the rented capacity. Someone has found a blog with DCQCN parameter values and wants a change window before the morning stand-up. The network lead says PFC is enabled on the leaves and nothing is dropping. Nobody has read a counter yet.
You slow it down, because the counters exist for exactly this argument. A lossless RoCE design is two mechanisms that have to be checked in order: PFC, which is hop by hop and per direction, so “we enabled it on the leaf” is not “it is enabled” anywhere else in the path; and ECN, which exists so a sender is told to slow down before buffers overflow, and whose absence leaves PFC doing congestion control at pause granularity and head-of-line blocking the whole rail.[1][10] Every parameter on that blog describes how a sender reacts to notifications it receives.[9]
So you ask for np_ecn_marked_roce_packets on a receiver, and it is zero.[2] Nobody on the call has run rping either, which is where the NCCL troubleshooting page says to start.[4] Tuning a reaction to an input of zero is how three nights become four.
Read the counters in a fixed order, or you will spend the night tuning the mechanism that already works. You put the order on the screen instead.
1The order, and why it is fixed
Triage order is not taste. Each step is chosen because it is cheaper than the next one and eliminates more of the search space, and because doing them out of order produces evidence you cannot act on.
The order: rping smoke test, then ethtool -S per-priority counters on both hosts, then the switch’s PFC and ECN counters on every hop, then hw_counters on both hosts, then a capture.[4][1]
Step zero is connectivity, and NVIDIA’s own NCCL troubleshooting page gives the command: rping -s -a <server_ip> -V -C 10 on the server and rping -c -a <server_ip> -S <client_ip> -V -C 10 on the client.[4] It uses RDMA CM, so it also proves the CM path and the GID selection work before you argue about bandwidth.[4] If rping fails, every counter below is noise.
Two rules end most of the arguments that follow.[1] First, PFC is hop by hop and per direction, so “we enabled it on the leaf” is not “it is enabled”; every port in the path, both directions, including the two NICs, has to agree on the priority.[1] Second, a lossless fabric with ECN switched off is not a design: it is PFC doing congestion control at pause granularity, and it will head-of-line block the whole rail.[1][10]
Read order: rping smoke test → ethtool -S per-priority on BOTH hosts → the switch's PFC/ECN counters → hw_counters → a packet capture, last.
Fine at two nodes, collapses at scale
“Any two hosts hit line rate, but the 32-node job runs at a third of it.”
- 1Sender
ethtool -S enp1s0f0 | grep rx_prio3_pause
What would you expect to see? Commit your diagnosis to reveal.
- 2Receiver
ethtool -S enp1s0f0 | grep tx_prio3_pause
What would you expect to see? Commit your diagnosis to reveal.
- 3Switch
nv show interface swp1 qos roce counters
What would you expect to see? Commit your diagnosis to reveal.
- 4Sender
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err
What would you expect to see? Commit your diagnosis to reveal.
rp_cnp_handled / rp_cnp_ignored increment here. This is where the rate is reduced.
RoCE counters are read on both hosts in the same window, with timestamps. A single host's dump cannot separate “the receiver never reported” from “the sender never reacted”.
2Congestion: direction tells you who is the victim
The two per-priority pause counters carry the whole diagnosis. The kernel defines rx_prio[p]_pause as pause packets received, and says that if it is increasing “it implies that the network is congested and cannot absorb the traffic coming from the adapter”; tx_prio[p]_pause is pause transmitted, and rising means “the adapter is congested and cannot absorb the traffic coming from the network”.[1] So received pause identifies a victim of the fabric and transmitted pause identifies a victim of an incast.[1]
Walk the pattern rather than the value. Fine at two nodes and collapsing at 32 is the classic shape: read rx_prio3_pause on the senders and tx_prio3_pause on the receivers, then walk every hop looking for the one port whose pause counters are flat while its neighbours are moving.[1] That flat port is the hole in the lossless domain.[1]
Two counters tell you the lossless design has already broken down.[1] rx_prio[p]_buf_discard counts packets “discarded by device due to lack of per host receive buffers” and rx_prio[p]_cong_discard counts drops “due to per host congestion” — on a priority you believe is lossless, either is a contradiction.[1] tx_pause_storm_error_events counts the device “reaching time out and disabling transmission of pause frames”, after which drops could have occurred: a pause storm has converted the lossless fabric into a lossy one without telling anybody.[1]
And the cheapest possible finding: if rx_pause_ctrl_phy or tx_pause_ctrl_phy is non-zero, the link is running global 802.3x pause, not PFC — those counters exist for link-layer pause and the per-priority counters exist only where PFC is enabled on that priority.[1]
3Marking: the ECN loop, one counter per link in the chain
The congestion loop has four places to break and each has exactly one counter.[2][9]
- The switch marks. On the receiver,
np_ecn_marked_roce_packetscounts RoCE packets that arrived with CE set. Flat while the fabric is congested means the switch is not marking your queue — wrong traffic class, wrong DSCP, or the traffic is simply not where you think it is.[2][10] - The receiver reacts.
np_cnp_sentshould track it. Marked packets with no CNPs means the notification-point algorithmr_roce_ecn_npis not enabled for that priority; the enable is per priority at/sys/class/net/<interface>/ecn/roce_np/enable/<priority>.[9] - The CNP gets home. On the sender,
rp_cnp_handledshould be non-zero. CNPs sent but not handled means they are being dropped or misclassified on the return path; the usual cause is that the CNP class — DSCP 48, switch priority 6 in the NVIDIA default mapping — has no strict-priority queue of its own.[2][10] - The sender obeys.
rp_cnp_ignoredrising instead is the reaction point receiving CNPs and discarding them, which points atr_roce_ecn_rpnot being enabled for that priority.[2][9]
Reading them in that order is what stops the conversation drifting into DCQCN parameters. NVIDIA’s ECN documentation is honest about the limit of what it publishes: it gives the sysfs enable and parameter paths but the parameter tables themselves are missing from every current page, so the names and defaults under /sys/class/net/<if>/ecn/roce_rp/params/ have to be read off the host in front of you.[9] That is a good reason to fix the loop before touching any parameter.
4The faults that are not congestion at all
Roughly half of “RoCE is slow” tickets never reach the QoS layer. These are the ones with a one-line check.[4][6][7][8]
| Symptom | First check | Cause |
|---|---|---|
Call to ibv_modify_qp failed with error Invalid argument on RoCE |
show_gids, NCCL version |
GID mismatch. On NCCL 2.21 and later the GID index is chosen dynamically and NCCL_IB_GID_INDEX should not be set at all[4][12] |
Call to ibv_create_qp failed or ibv_reg_mr failed |
ulimit -l |
memlock not unlimited; in Kubernetes the pod needs IPC_LOCK instead[4][11] |
perftest Couldn't allocate MR with --use_cuda |
the GDR path | Scatter to CQE; retry with MLX5_SCATTER_TO_CQE=0[6] |
Intermittent retry-exceeded, local_ack_timeout_err climbing |
active_mtu on both ports |
MTU mismatch: perftest defaults -m to the port’s active_mtu, so force it identically on both sides to confirm[7][13] |
| Traffic ignores the DSCP the app sets | mlnx_qos -i <if> |
Trust state is still the default pcp[8] |
| Everything right on the host and still lossy | NCCL_IB_TC, perftest --tclass |
Application marking left at 0 so the traffic misses the lossless queue entirely[12][7] |
| Receiver stalls, sender reports RNR | out_of_buffer, rnr_nak_retry_err |
The application is not posting receives; this is a flow-control bug, not a fabric problem[2] |
| Rare corruption that QoS changes never fix | rx_icrc_encapsulated |
Physical corruption the Ethernet FCS missed: optics, DACs, connectors[2] |
| Throughput fine host to host, collapses GPU to GPU | nvidia-smi topo -m, lsmod |
GPUDirect not active: install order, missing peermem or DMA-BUF, or a SYS-distance NIC to GPU path[5] |
Two of these deserve a second sentence. The GID one has inverted over time: manual NCCL_IB_GID_INDEX used to be the fix and is now the fault, so the version is part of the diagnosis.[4] And the MTU one is invisible in short runs because a small message never needs the larger path MTU; the honest test is the same -m value forced on both sides, since perftest requires the same options on both ends anyway.[6][7]
Sample runs, exits cleanly, but zero throughput
5Working a case end to end
The transcript. “Any two nodes hit 380 Gb/s with ib_write_bw. The 32-node training job runs at about a third of that. We enabled PFC on the leaves. Nothing is dropping.”
Step 1, connectivity. rping -s -a 10.0.3.11 -V -C 10 and the matching client line. Passes, so addressing, GIDs and the CM path are fine and the ticket is genuinely about performance.[4]
Step 2, both hosts, per priority. On a sender: ethtool -S enp1s0f0 | grep -E "prio3_(pause|discard|buf_discard)|pause_ctrl|storm". rx_prio3_pause is climbing, so the network is pausing this adapter.[1] On a receiver the same command shows tx_prio3_pause climbing, so it cannot absorb the incast.[1] rx_pause_ctrl_phy is zero, so this really is PFC and not global pause.[1] Nothing yet contradicts the customer.
Step 3, every hop. Now the claim “we enabled PFC on the leaves” gets tested against the spines. On a Cumulus hop: nv show interface swp1 qos roce counters and nv show qos roce.[10] One spine port shows no PFC counters at all while its neighbours pause. That is the hole: PFC is hop by hop, so a single port without it makes the entire path lossy for every flow that crosses it.[1]
Step 4, confirm the consequence on the hosts. grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/{packet_seq_err,out_of_sequence,req_transport_retries_exceeded}. packet_seq_err is rising on the receivers of flows that cross that spine, which is loss, and out_of_buffer is flat, which rules out the receiving application.[2]
Step 5, the ECN half. np_ecn_marked_roce_packets is flat everywhere. So the fabric is not marking either: the design is relying on PFC alone, which is why the collapse is a cliff rather than a slope.[2][10]
Diagnosis. One spine port outside the PFC domain, plus no ECN anywhere. Recommendation: bring the missing port into the same priority-3 PFC configuration, and enable ECN so congestion is signalled before buffers fill rather than after.[10] No capture was needed.
Same transcript. Fill the blanks and justify each in one sentence.
- Smoke test:
________ -s -a 10.0.3.11 -V -C 10. Purpose: ________. - On the sender,
ethtool -S <if> | grep prio3.________climbing means the network is pausing me. - On the receiver,
________climbing means I am pausing the network. rx_pause_ctrl_phynon-zero would mean ________ instead of PFC.- The hop walk exists because PFC is ________.
- On the hosts,
________rising inhw_countersproves loss rather than receiver starvation, which would instead show________. np_ecn_marked_roce_packetsflat everywhere means ________.
New transcript. “Storage traffic to our RoCE array is fine. The training job is fine on eight nodes. When both run at once, the training job’s all-reduce time doubles and the storage clients report timeouts. We use one priority for everything because it was simpler.”
Produce a written diagnosis. Acceptance criteria:
- You name the read order you used and what each step ruled out.
- You state which counter, on which machine, would confirm head-of-line blocking rather than loss, and why the distinction changes the fix.
- You state what you expect
rx_prio[p]_pauseto look like on ports carrying no training traffic, and what that observation would prove. - You give a recommendation that changes the class design rather than a parameter, and one measurement that would show whether it worked.
- You name one thing you are not able to conclude from the host counters alone.[1][2][10]
What the order found
Working the order, the pause counters name the victim before the cause: received pause on the senders, transmitted pause on the receivers, and one spine port with no per-priority pause counters at all — the hop where the lossless domain has a hole.[1] packet_seq_err rising on the flows that cross it confirms real loss, and the flat ECN counter explains why the collapse is a cliff, not a slope.[2][10]
What you say to the customer: “No parameter change tonight. One port in the path is not in the lossless domain, and until it is, every number we tune is measuring that port.”[1]
The port joins the lossless domain by breakfast; fifteen nodes return to line rate. Node sixteen, rebuilt from the image the lab clones forty-eight times tomorrow, does not.
Lab
On the Dell-lab ConnectX pair. Host QoS is the only mutating surface; capture it first.
- Pre-flight, read-only and mandatory.
ibdev2netdev;ibv_devinfo -d mlx5_0 | grep -E "state|active_mtu|link_layer";mlnx_qos -i <netdev> | tee ~/qos-before.txt;grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/* > ~/hw-before.txt;ethtool -S <netdev> > ~/eth-before.txt.[8][3] Theqos-before.txtfile is the rollback reference for everything below. - Smoke test both directions with
rpingbefore any change.[4] - Fault 1, MTU mismatch (read-only on the fabric). With the netdev MTU at 1500, run
ib_write_bw -d mlx5_0 -x <gid> -m 4096 -s 1048576 -n 20000on both sides and watchlocal_ack_timeout_errandreq_transport_retries_exceededinhw_counters.[7][2] Expected: retries climb and the run degrades or aborts. If nothing moves, your netdev MTU already carries the 4096 path MTU; lower it and repeat. - Fault 2, trust state (mutating). Set the interface to trust DSCP and mark DSCP 26, then deliberately set it back to
pcpwhile still marking DSCP 26, and compare per-priority byte counters:ethtool -S <netdev> | grep -E "prio[0-9]_(bytes|packets)".[8][1] Expected: withpcptrust the traffic is counted on the default priority instead of priority 3. Rollback: re-apply the trust state and mappings recorded in~/qos-before.txt. - Fault 3, starved receiver (read-only).
ib_send_bw -d mlx5_0 -r 1on the server against a deep client and readout_of_bufferplusrnr_nak_retry_err.[2] Expected: both rise on the correct machines; note which one the requester sees and which one the responder sees. - Per-QP attribution for the fault you find most interesting: bind a counter to the QP as in the counter-map lesson and show the errors belong to one QP, then unbind.[14]
- Restore and verify. Re-apply
~/qos-before.txt, re-runmlnx_qos -i <netdev>and diff against the saved file; re-runrpingand the baselineib_write_bw. Expected: identical QoS output and a bandwidth number within noise of the baseline. If not, you have not finished the lab.
Fault injection over Soft-RoCE in containerlab. Break one thing at a time, record which counter or completion status moved first, then fix it before the next injection.
- Baseline. Bring up two containers with
rdma_rxeon their veth links and prove connectivity:rping -s -a <srv> -V -C 10and the client form.[4] Expected: ten ping-pong lines and exit 0. If not: therdma_rxedevice or the address is wrong; fix before continuing. - Baseline numbers:
ib_write_bw -d rxe0 -s 65536 -n 5000 --report_gbits <srv>with identical flags on both sides.[6] Record the number and the counter snapshot. - Injection A, MTU. Set one side’s netdev to 1500 and the other to 9000 (
sudo ip link set <if> mtu 1500), then repeat step 2 with-m 4096forced on both sides.[7] Expected: the run fails or collapses; record which oflocal_ack_timeout_err,packet_seq_error a completion status moved first. Rollback: restore the original MTU on both sides. - Injection B, one-way blackhole.
sudo iptables -I INPUT -p udp --dport 4791 -j DROPon one node only.[13] Expected: retry-exceeded behaviour; note that the counters on the two hosts disagree, which is the signature of a one-directional fault. Rollback:sudo iptables -D INPUT -p udp --dport 4791 -j DROP. - Injection C, starved receiver. Run
ib_send_bwwith a receive depth far smaller than the sender’s tx depth (-r 1on the server against-t 128on the client) and watchout_of_bufferandrnr_nak_retry_errinhw_counters.[2] Expected: both rise; nothing on the fabric changed. Rollback: none needed, the run ends. - Injection D, wrong marking. Run with
--tclass=0and then--tclass=104and compare where the traffic lands, given that ToS is DSCP times four.[7][8] Expected on Soft-RoCE: no queueing difference, because there is no hardware queue — record that as a fidelity limit of the no-hardware lab. - For each injection write one line: symptom, first counter that moved, the machine it moved on, and the check that would have found it in under a minute.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A Dell customer says 'RoCE is broken, we need you on site'. Explain in five sentences the order you will read counters in and why a packet capture is the last thing you do, not the first.
Sources
Facts in this lesson were checked against mlx5 ethtool counters and sysfs-class-infiniband kernel docs, mlx5_ib counters.c, NCCL 2.31.2 troubleshooting, perftest master README and man page, Cumulus Linux 5.18 RoCE page, fetched 2026-09-07. Dates are when each page was fetched.
- Ethtool counters (mlx5): the Linux kernel documentation · fetched 2026-09-07
- Linux kernel: drivers/infiniband/hw/mlx5/counters.c · fetched 2026-09-07
- Linux kernel ABI: sysfs-class-infiniband (stable) · fetched 2026-09-07
- Networking Troubleshooting: NCCL 2.31.2 documentation · fetched 2026-09-07
- GPU Troubleshooting: NCCL 2.31.2 documentation · fetched 2026-09-07
- OFED Performance Tests README (linux-rdma/perftest master) · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- Ethernet QoS (DOCA SDK 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
- Explicit Congestion Notification (ECN): MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
- RDMA over Converged Ethernet (RoCE): Cumulus Linux 5.18 · fetched 2026-09-07
- MacVLAN Network with RDMA Shared Device: NVIDIA Network Operator v25.7.0 quick start · fetched 2026-09-07
- Environment Variables: NCCL 2.31.2 documentation · fetched 2026-09-07
- RDMA over Converged Ethernet (DOCA-Host): generated PDF · fetched 2026-09-07 · DOCA 3.5.0
- rdma-statistic(8): iproute2 rdma tool man page · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- PFC, lossless and the case for lossyElsewhere in this course · Same ground: PFC, ecn and pause
- NIC firmware config in Kubernetes: RoCE, PFC and rebootsK8sNet course · Same ground: PFC, gpudirect and triage
- Telemetry-based congestion control: three loops, not oneSpectrum-X course · Same ground: PFC, ecn and qos