Playbook: 'RoCE is slow'
S3·E5The ticket that says only 'RoCE is slow' · Customer data centre, a borrowed desk on the cage floor, two days out
Builds on: Packet trimming and `lossy-multi-tc`
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Execute a nine-step RoCE verification playbook in a fixed order and justify why each step precedes the next.
- Interpret the four counters that answer 'RoCE is slow' - pause packets, ecn-marked-packets, no-buffer-discard and pg-max-usage.
- Diagnose three fault signatures from show output alone and name the single command that identified each cause.
- Assemble the host-side and switch-side evidence set an FAE should collect before escalating a congestion ticket.
Episode 5 — The ticket that says only 'RoCE is slow'
The desk is a cart in the cold aisle, and somebody has stuck a label on it that reads DIVIDE BY FOUR. There is one label maker in this building and the night-shift operator owns it. Behind you the network lead has his notebook open at the page of counters from the 02:10 call, and the architect wants the ECN thresholds lowered now, because that is what the previous vendor suggested.
You change nothing on the first pass, and it is worth saying why out loud. Every reading you have is a snapshot, and a RoCE mode change re-lays the buffer pools — so a quick fix applied during triage destroys the baseline you needed to tell whether the fix helped.[1] The first pass produces a diagnosis and a proposed one-line change; the change gets a window and a rollback.
The order is not negotiable either, and two structural facts drive it. RoCE QoS state defaults to disabled, so a switch nobody ever configured looks exactly like a switch that is misbehaving.[8] And classification is an agreement between host and switch that neither side validates — a mismarked flow forwards perfectly into traffic class 0 and raises no error anywhere.[1] Both are visible in the first two commands.
Before any counter, you ask what nobody has asked: which flows, between which hosts, compared to what number. Two days out, the fastest move is refusing to guess. Segment 1 is why the order is fixed.
1Why the order is fixed
“RoCE is slow” is not a symptom, it is a category. The playbook below is nine reads in a fixed order, and the order is the point: each step makes the next one interpretable. Reading pause counters on a switch where RoCE was never enabled tells you nothing; reading buffer occupancy before you know the traffic is in the right class tells you about the wrong queue.
Two structural facts drive the order. RoCE QoS state defaults to disabled, so the fabric may never have been configured at all.[8] And classification is an agreement between host and switch that neither side validates — a mismarked flow forwards perfectly into traffic class 0 and shows no error anywhere.[1] Those two failure modes account for most first-pass tickets and both are visible in the first two commands.
The pass is read-only by design. Every command below is a nv show, and nothing here changes configuration. That is not caution for its own sake: a RoCE mode change re-lays the buffer pools, so a “quick fix” applied during triage destroys the baseline you needed to tell whether the fix helped.[1] The first pass produces a diagnosis and a proposed one-line change; the change happens in a window with a rollback, as in lesson 3.2.
One tool warning before you start. tcpdump on a Cumulus switch “captures only control-plane traffic to/from the CPU, not the data plane” — which surprises every server engineer on their first switch ticket.[6] Data-plane questions go to WJH, SPAN or telemetry on the switch, and packet-marking questions go to a capture taken on the host.
Sample runs, exits cleanly, but zero throughput
2Steps 1-3: is it on, is it right, is it pausing
Step 1 — nv show qos roce. The system-level view: RoCE configuration and pool status.[1] You are answering one question — is it enabled, and in which mode. If the answer is disabled, you are finished with the first pass; the fix is lesson 3.2 and the conversation moves to a change window. If the mode is lossless-single-ipool, note it now, because it changes what NetQ can do for you in step 8.[4]
Step 2 — nv show interface <if> qos roce status. The per-interface view, reporting congestion-mode, enabled-tc, min-threshold, max-threshold, probability, pfc-priority, rx-enabled, tx-enabled, cable-length, trust-mode and the lldp-app-tlv fields.[1] Confirm PFC is on switch priority 3 and the pool map is what you expect.[1][2] Two fields here catch real faults on their own: a trust-mode that is not what the rest of the fabric uses, and a cable-length that does not match the physical cable. Use nv show interface qos-roce-status to sweep every interface at once when you suspect one is different.[1]
Step 3 — nv show interface <if> counters qos pfc-stats. Pause-frame counts and pause duration per switch priority 0 to 7 — the single best “is PFC firing?” read.[2] Interpretation is where people go wrong. Zero TX pause on a congested lossless fabric means classification is wrong, not that there is no congestion.[2] If the traffic were in the lossless class and the class were congested, PFC would be pausing. Zero everywhere on a busy fabric is a classification finding.
Read duration alongside count. A million pause frames of 10 µs each is a very different fabric from a thousand pause frames of 10 ms each, and only the duration figure distinguishes them.[2]
3Steps 4-6: the four counters and the queue
Step 4 — nv show interface <if> qos roce counters. The counter fields are roce-bytes, roce-packets, cnp-bytes, cnp-packets, ecn-marked-packets, no-buffer-discard, buffer-usage, buffer-max-usage, pg-usage, pg-max-usage, pause-duration and pause-packets.[1] Four of them do the diagnostic work:
| Counter | What it says | What it points at |
|---|---|---|
pause-packets / pause-duration |
PFC is engaging | Thresholds or headroom |
ecn-marked-packets |
ECN is signalling | If zero under load: thresholds too high |
no-buffer-discard |
A packet arrived with nowhere to go | Pool split or headroom too small |
pg-max-usage |
Peak priority-group occupancy | How close to the ceiling the fabric runs |
The pairs matter more than the singles. High pause with zero ECN marks means the marking thresholds sit above the pause threshold, so the emergency brake engages before the gentle signal ever fires — fix the ECN thresholds, not the pause behaviour.[1][3] Zero pause with rising no-buffer-discard means the lossless machinery is not protecting the traffic at all: either the pool is too small or the traffic is not in the lossless class.[1]
Step 5 — nv show interface <if> counters qos egress-queue-stats. Per-traffic-class transmitted frames and bytes, unicast buffer discards and WRED drops.[2] This is the classification cross-check: if the volume is on traffic class 0 rather than traffic class 3, you have proved the classification finding from step 3 rather than inferred it. WRED drops on traffic class 3 mean the ECN and RED thresholds are being exceeded.[2]
Step 6 — nv show interface <if> qos pfc-watchdog. Watchdog state and deadlock detection status per traffic class with deadlock occurrence counts.[2] Non-zero means a pause storm was detected and mitigated on this box; the cause is upstream, so this counter is a direction indicator. Remember the watchdog runs only on lossless queues on physical swp interfaces.[3]
4Steps 7-9: the whole switch, the whole fabric, the physical layer
Step 7 — nv show qos buffer. Aggregated ingress and egress data, management and descriptor pool occupancy, current and maximum.[2] This is the step for when per-interface counters are ambiguous: a leaf whose lossless pool is at its ceiling system-wide has a sizing problem, not an interface problem. Read it against the pool split you configured in lesson 3.2 — the default before any split is 100% to default-lossy.[3]
Step 8 — netq check roce. Six tests: 0 RoCE mode (“whether RoCE is configured for lossy or lossless mode”), 1 classification (“consistency of DSCP, service pool, port group, and traffic class settings”), 2 congestion control (“consistency of ECN and RED threshold settings”), 3 flow control (“consistency of PFC configuration for RoCE lossless mode”), 4 ETS mode, 5 miscellaneous (“consistency across related services”).[4] Every one of them is a cross-switch consistency check, which is what no single-switch command can give you — and inconsistent QoS on one leaf is the classic cause of intermittent loss.
The blind spot: NetQ supports RoCE lossless and lossy modes but does not support lossless-single-ipool, so a customer in that mode gets no RoCE validation at all.[4] If step 1 reported that mode, step 8 will not help and you must compare leaves by hand. The wider NetQ RoCE surface is netq show roce-config, netq show roce-counters, netq check roce and the roceconfig and tca_roce event types.[5]
Step 9 — MTU and link pause. Confirm MTU consistency end to end; the ESF guide’s default interface MTU for storage fabrics is 9216, and NetQ ships link-MTU tests 0, 1 and 2 plus a BGP MTU test precisely because mismatch is a top cause.[7][4] Then confirm link pause is not configured on a port that also runs PFC — “You can not configure link pause and PFC on the same port”, and legacy link pause on an inherited configuration is a common reason one interface behaves differently from every other one on the leaf.[3]
Traffic class 0
Every switch priority other than 3 and 6 maps to traffic class 0 — the default lossy class. The traffic class is what the scheduler, the ECN profile and the packet-trim eligibility list are written against. In this mode: TC0 is ordinary lossy traffic sharing the default pool.
TC0 — Everything unclassified: default-lossy pool, no PFC, no RoCE ECN profile.
FAE angle When a customer says "we gave RoCE its own queue", ask for `nv show interface <if> qos roce status` and read `enabled-tc`. A queue nobody classifies into is decoration.
PFC on switch priority 3 plus ECN on TC3. The buffer must never overflow for RoCE: the switch pauses upstream before it would drop.
nv set qos roce state enabled nv set qos roce mode lossless nv config apply
nv set qos egress-queue-mapping default-global switch-priority 3 traffic-class 3 nv set qos egress-queue-mapping default-global switch-priority 6 traffic-class 6 nv show interface swp16 qos roce status
- SP3 → TC3 (RoCE lossless, DWRR 50%), SP6 → TC6 (CNP, strict priority), all others → TC0. source
- Traffic class 6 (CNP) scheduler: strict. Traffic class 3 (RoCE) scheduler: WRR 50%. Identical in lossy and lossless. source
- PFC RX and TX are enabled in lossless mode and disabled in lossy mode. source
- One command sets buffer pools, classification and priority mappings, PFC and ECN thresholds, and scheduling for regular, RoCE and CNP traffic. source
RoCE (CL 5.18) · Quality of Service · Packet Trimming · NVUE QoS set
5Three signatures, and what an FAE collects
The three fault signatures worth recognising on sight:
High pause, zero ECN marks. PFC is doing all the work because ECN never fires. The marking thresholds sit above the pause threshold. Confirming command: nv show interface <if> qos roce counters showing large pause-packets and ecn-marked-packets at zero. Fix: lower the ECN min and max thresholds on the RoCE traffic class with nv set qos congestion-control <profile> traffic-class 3 min-threshold …, in a window, one step at a time.[1][8]
Zero pause, rising no-buffer-discard. Packets are arriving with nowhere to go and nothing is pausing to stop them. Either the lossless pool is too small or the headroom is wrong. Confirming command: nv show qos buffer plus nv show qos roce for the pool split. Fix: the pool split from lesson 3.2 and the cable-length arithmetic from lesson 3.3.[2][1]
Intermittent loss on flows that cross one leaf. Every counter on the leaf you are logged into looks fine. Confirming command: netq check roce — a classification or congestion-control inconsistency will name the outlier switch. Fix: bring the outlier’s configuration into line, remembering that a mode change re-lays its buffers.[4][1]
For a Dell escalation, the evidence set is what determines whether the ticket moves. Switch side: nv show qos roce, nv show interface <if> qos roce status, nv show interface <if> counters qos pfc-stats, nv show interface <if> qos roce counters, nv show interface <if> counters qos egress-queue-stats and nv show qos buffer, all captured during the event and again after clearing counters.[1][2] Host side: ethtool -S <ifname> pause and discard counters, ib_write_bw bandwidth and latency output, the NIC’s trust and PFC state from mlxreg --get, and a tcpdump of the UDP 4791 flow showing the DSCP that actually left the host.[9]
NCP-AIN objective 5.3 is verifying low-latency interconnects and objective 2.2 covers verifying QoS, ECN, PFC and telemetry; this playbook is the practical form of both.[11] One last scope note: none of the counter behaviour above exists in simulation, because a virtual Cumulus switch implements no QoS shaping, buffer management or packet marking.[10]
Ticket: “Training job step time doubled overnight. RoCE is slow between rack 4 and rack 7.”
Step 1 — bound the problem before touching a switch. Ask for: which host pairs, which leaf each host is on, what the step time was before, and an ib_write_bw result between two hosts on the same leaf as a control. Suppose the control pair runs at line rate and only cross-rack pairs are slow. You have localised the fault to paths that traverse a spine, without a single switch command.[9]
Step 2 — nv show qos roce on both leaves. Suppose both report lossless, enabled. Good: no configuration-absence finding, and NetQ validation is available in step 8 because neither is lossless-single-ipool.[1][4]
Step 3 — nv show interface <uplink> qos roce status on leaf-4 and leaf-7. Compare pfc-priority, trust-mode, cable-length and the pool map side by side. Suppose they match.[1]
Step 4 — nv show interface <uplink> counters qos pfc-stats on both. Suppose leaf-4 shows large TX pause on switch priority 3 with a substantial pause duration, and leaf-7 shows large RX pause on switch priority 3. That pairing is a story: leaf-7 is pausing leaf-4.[2]
Step 5 — nv show interface <uplink> qos roce counters on leaf-7. Suppose ecn-marked-packets is 0 and pg-max-usage is at its ceiling. Signature one: PFC is doing all the work because ECN never fired.[1]
Step 6 — nv show interface <uplink> counters qos egress-queue-stats on leaf-7 to confirm the volume really is on traffic class 3 and not traffic class 0. Suppose it is. Classification is exonerated.[2]
Step 7 — nv show interface <uplink> qos pfc-watchdog on both. Suppose zero deadlock occurrences. No pause storm; this is sustained congestion, not a loop.[2]
Step 8 — netq check roce. Suppose tests 0, 1, 3, 4 and 5 pass and test 2 (congestion control) flags leaf-7 as inconsistent with the fleet on ECN thresholds. That is the answer: leaf-7’s ECN thresholds are higher than the rest of the fabric.[4]
Step 9 — MTU consistency and link pause on the affected ports, to close out the two cheap causes before writing the report.[7][3]
Diagnosis in one line: leaf-7’s ECN thresholds sit above its PFC threshold, so cross-rack flows are being paused rather than marked, and the pause propagates back to leaf-4. Fix in one line: bring leaf-7’s qos congestion-control thresholds into line with the fleet, in a window, with nv config apply --confirm and the counters re-read afterwards.
Ticket: “Storage RoCE to the array is fine but GPU-to-GPU is slow, on one leaf only.”
- Which control measurement do you ask for before running any switch command? ________
- Step 1 command and the one question it answers:
nv show ____ ____/ ________ - Step 2 command, and the three fields you compare against a known-good leaf:
nv show interface <if> qos ____ ____/ ____ , ____ , ____ - Step 3 command:
nv show interface <if> counters qos ____. If it returns zero on every switch priority while the customer reports congestion, what have you learned? ________ - Which command proves that finding rather than inferring it, and what would you expect to see?
nv show interface <if> counters qos ____/ ________ - Which two counters from
qos roce countersseparate “thresholds wrong” from “pool too small”? ____ and ____ - Which fleet-wide command finds a single misconfigured leaf, and which RoCE mode would make it useless? ________ / ________
- Name the two closing checks. ________ and ________
- Write your diagnosis and your fix as one line each.
Three transcripts, one fabric, one shift. For each, produce the diagnosis, the single command that identified the cause, and the one-line fix. Then build the combined runbook.
- Leaf-2
pfc-stats: TX pause on switch priority 3 in the millions with a large pause duration.qos roce counters:ecn-marked-packets0,no-buffer-discard0,pg-max-usageat ceiling. - Leaf-5
pfc-stats: zero pause on every switch priority.qos roce counters:no-buffer-discardrising steadily,pg-max-usageat ceiling,ecn-marked-packets0.egress-queue-stats: nearly all volume on traffic class 3. - Flows are lossy only when they cross leaf-9. Leaf-9’s own counters look identical to leaf-2’s healthy neighbours.
nv show qos roceon leaf-9 reports modelossless-single-ipool; every other leaf reportslossless.
Acceptance criteria: each diagnosis names one of the three signatures from Segment 5; case 2 must distinguish “pool too small” from “classification wrong” and cite the egress-queue-stats evidence that rules one out; case 3 must state both the configuration inconsistency and the fact that netq check roce cannot validate that leaf, and must say what you do instead. Then write the nine-step runbook as a numbered list a Dell L2 engineer could follow without this lesson, marking which steps are read-only (all of them) and which findings require a maintenance window to act on.
Case closed — nine reads, one line of change
Nine nv show commands later the picture is unambiguous: pause near zero on switch priority 3, the volume sitting on traffic class 0 in egress-queue-stats, and netq check roce naming the leaf that was racked last as the classification outlier.[2][4] One line changes, in a window, with the counters captured before and after. Pod A passes acceptance on Friday, procurement gets its lead-time answer, and the SE finally closes row 41.
What you say to the architect: “Nothing was changed until the evidence named the change.” The network lead keeps the nine reads in his notebook, in order — and the operator has already printed a label for the runbook binder.
Lab
Goal: reproduce signature one on the Dell lab and collect the exact evidence set an FAE would ask a customer for. Read-only on the switch; step 3 writes the NIC’s PFCC register and every host step names its rollback.
Pre-flight inventory on every host involved (this is both the baseline and the rollback reference):
ibstat; ibdev2netdev
cat /sys/class/infiniband/<dev>/tc/1/traffic_class
sudo mlxreg -d <bdf> --reg_name QPTS -i "local_port=1" --get
sudo mlxreg -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --get
ethtool -S <ifname> | grep -i -E 'pause|discard|drop' > /tmp/pre-$(hostname).txt- Establish the control measurement. Between two hosts, receiver
ib_write_bw -d <dev> -F, senderib_write_bw -d <dev> -F <receiver-ip> --duration 20. Record bandwidth and average latency. Expected: close to line rate. This number is what makes every later measurement interpretable. - Confirm classification end to end as in lesson 3.1: traffic class 106 on the senders, a
tcpdump -v -n -i <ifname> udp port 4791on the receiver showingtos 0x6a. Expected: DSCP 26. If not, fix this before continuing — you would otherwise be measuring the wrong class. - Enable PFC on priority 3 on all hosts with the
PFCC --setcommand from lesson 3.2. Rollback: re-apply the values captured in the pre-flight block. - Create many-to-one incast: one receiver, two or more senders running
ib_write_bwsimultaneously for 60 seconds. Expected: aggregate received bandwidth well below the sum of individual runs, and per-sender bandwidth below the control. - During the run, collect the host evidence:
ethtool -S <ifname> | grep -i pausesampled every 2 seconds on both receiver and senders, and the fullib_write_bwoutput including latency. Expected: pause counters climbing on the receiver’s link, bandwidth falling as they climb. Save both to files named with the hostname and timestamp. - After the run, diff against the pre-flight capture:
ethtool -S <ifname> | grep -i -E 'pause|discard|drop' > /tmp/post-$(hostname).txt; diff /tmp/pre-$(hostname).txt /tmp/post-$(hostname).txt. Expected: pause counters moved, discard counters ideally did not. Discards moving means the lossless path did not hold. - Assemble the escalation package: the control measurement, the congested measurement, the pause and discard deltas, the NIC trust and PFC state from
mlxreg --get, and the tcpdump proving the DSCP. That set answers “is the host marking correctly and is it being paused” without any switch access at all — which is exactly the position you are in on most customer calls. - Restore the pre-flight NIC state and re-run step 1. Expected: the control bandwidth returns.
Optional (customer lab, read-only on the switch): during the same incast, run the full nine steps on the leaf and record every output. Then run netq check roce if NetQ is deployed. Expected: TX pause rising on switch priority 3, ecn-marked-packets non-zero if thresholds are sane and zero if they are not, deadlock occurrence count zero, and netq check roce tests 0 to 5 passing on a consistent fabric. Nothing in this optional block changes configuration.
Goal: work three fault transcripts through the playbook and rebuild the decision tree yourself. No hardware or simulator required.
- Write the nine steps from memory, in order, with the exact command for each. Then check against Segments 2 to 4. Expected:
nv show qos roce,nv show interface <if> qos roce status,counters qos pfc-stats,qos roce counters,counters qos egress-queue-stats,qos pfc-watchdog,nv show qos buffer,netq check roce, MTU and link-pause checks. If your order puts a counter before a configuration read, redo it. - For each of the three transcripts in the Worked problem above, name (a) the single command that identified the cause and (b) the one-line fix. Expected: one thresholds case, one pool or headroom case, one cross-switch consistency case.
- For transcript 2, write down the one additional read that distinguishes “the pool is too small” from “the traffic is in the wrong class”, and what each answer looks like. Expected:
egress-queue-stats— volume on traffic class 3 means pool sizing, volume on traffic class 0 means classification. - For transcript 3, write what you do instead of
netq check roce, given that NetQ does not supportlossless-single-ipool. Expected: comparenv show qos roceandnv show interface qos-roce-statusoutput between the outlier and a known-good leaf by hand, field by field. - Open the TroubleshootingTree above and walk the throughput branch. Note where its DOCA-oriented questions diverge from a RoCE fabric investigation, then draw your own RoCE tree: root “RoCE is slow”, first split “is RoCE enabled”, second split “is classification right”, third split “which counter is non-zero”. Expected: three levels, five to eight leaves, each leaf naming one
nv showcommand. - Compare your tree to the nine-step list. Every leaf should map to a step; every step should appear somewhere in the tree. Any step with no leaf is a step you do not actually understand the purpose of — go back to the segment that introduces it.
Deliverable: the nine-step runbook, the three diagnoses with commands and fixes, and your own RoCE troubleshooting tree.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A Dell customer opens a ticket that says only 'RoCE is slow'. Explain to your manager, in six sentences, the order you will work in and why you refuse to change any configuration in the first pass.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 RoCE, QoS and Network Troubleshooting pages, NVUE 5.x QoS show reference, NetQ 5.1 validation checks; re-fetched 2026-09-07. Dates are when each page was fetched.
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference - QoS show commands · fetched 2026-09-07
- Quality of Service | Cumulus Linux 5.18 · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- RoCE monitoring | Cumulus NetQ 5.1 · fetched 2026-09-07
- Network Troubleshooting | Cumulus Linux 5.18 · fetched 2026-09-07
- Cumulus Linux Configuration Guide for Ethernet Storage Fabrics · fetched 2026-09-07
- NVUE Reference - QoS set/unset commands · fetched 2026-09-07
- NVIDIA/doroce-linux - doRoCE.sh host-side RoCE configuration script · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- NetQ 5.1: validation instead of inspectionElsewhere in this course · Same ground: netq, cli and fae
- Scenario: 'the fabric is slow and adaptive routing does nothing'Elsewhere in this course · Same ground: netq, hw_counters and fae
- What Just Happened: naming the dropElsewhere in this course · Same ground: buffers, netq and cli