Tuning the lossless edge: PFC headroom, ECN thresholds, watchdog
S3·E3Zero drops, and nothing is moving · Bridge call at 02:10, seven days out
Builds on: `nv set qos roce`: the one command and its five modes
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Decompose the RoCE one-liner into its PFC profile, congestion-control profile and buffer settings and name the NVUE command behind each.
- Analyse how cable-length and the xoff/xon thresholds size PFC headroom and predict the effect of a wrong cable length in both directions.
- Distinguish absolute from dynamic ECN thresholds and explain why the published threshold numbers appear to disagree.
- Diagnose a PFC watchdog deadlock count and the link-pause conflict from show output rather than from configuration.
Episode 3 — Zero drops, and nothing is moving
The operator called first and the bridge opened second; the network lead joins from a kitchen table with the notebook already open. A training job on pod A has stalled — no drops anywhere, no interface errors, traffic on the lossless priority simply not moving. Somebody wants to reboot a leaf. The network lead says the sentence he says at least once per fabric: show me the counter.
There are two. nv show interface <if> counters qos pfc-stats gives pause packets and pause duration per switch priority, and nv show interface <if> qos pfc-watchdog gives watchdog state and deadlock occurrence counts per traffic class.[4] A non-zero occurrence count does not warn about the future — it means a pause storm already happened and was broken on that box, and the cause is upstream.[4]
Which is what PFC is for, and what it costs. It pauses one priority on one link so nothing is dropped, and the pause propagates upstream hop by hop; that is head-of-line blocking by design, and it is why a healthy RoCE fabric shows pause counters near zero.[3] The watchdog exists because pause can close a loop — A pauses B, B pauses C, C pauses A — and it waits a 100 ms polling interval times a robustness of three before it acts.[1]
A lossless fabric does not fail by losing; it fails by stopping. Nobody reboots anything. Segment 1 is the mechanism those counters are describing.
1PFC: the emergency brake and what it costs
Priority flow control pauses one priority on one link. On a Cumulus switch it is a profile: nv set qos pfc <profile-id> switch-priority <sp> selects which switch priority is lossless, … rx enable|disable and … tx enable|disable control the two directions, and … cable-length <1-100000>, … port-buffer <bytes>, … xoff-threshold <bytes> and … xon-threshold <bytes> size the buffer arithmetic.[2][1]
The documented defaults, verified on the 5.18 QoS page at write time, are xoff_size 10000 bytes, xon_delta 2000 bytes, port_buffer_bytes 25000 bytes and a recommended cable_length of 50 metres.[1] Note the tension with lesson 3.2: nv set qos roce cable-length defaults to 100 metres.[2] Two different objects, two different numbers, both current — which is why you read the running switch rather than reciting one of them.
The three thresholds work together. Occupancy on the ingress priority group climbs; when it crosses xoff-threshold the switch sends XOFF upstream; when it falls back to xoff-threshold minus xon-delta it sends XON. The gap exists so the link does not oscillate between paused and unpaused every few microseconds.
What PFC costs is scope. It stops the entire priority on that link, including flows that had nothing to do with the congestion, and the pause propagates upstream hop by hop. That is head-of-line blocking by design, and it is why a well-tuned RoCE fabric shows pause counters near zero: PFC is the mechanism that guarantees no drop, not the mechanism that manages congestion.[3]
Congestion response — which of the three actually fires
In lossless mode: PFC sends XOFF upstream, ECN marks CE at the egress queue, trimming is not active. RFC 3168: a router sets the CE codepoint only where it would otherwise have dropped the packet.
- ✓ PFC XOFF — pause upstream · ingress
PFC RX/TX is enabled on switch priority 3 in this mode: the ingress buffer crosses xoff and the switch sends an XOFF upstream.
- ✓ ECN CE mark — signal end to end · egress
The egress queue is past the ECN minimum threshold, so the switch flips the IP ECN field to CE (11) instead of dropping — RFC 3168 §5.
- ✗ Packet trim → 256 B — forward the header · Spectrum-4+
Packet trimming needs `nv set qos roce mode lossy-multi-tc` plus `nv set system forwarding packet-trim state enabled`, on Spectrum-4 or later.
FAE angle Three loops, not one. PFC is layer 2, hop by hop, switch priority 3. ECN/DCQCN is layer 3, end to end, and on Spectrum-X only fires once adaptive routing has run out of paths. Spectrum-X CC proper is RTT probes plus in-band switch telemetry driving per-destination rate contexts on the SuperNIC. Climbing pause counters on a Spectrum-X fabric usually mean adaptive routing was never eligible for that traffic.
PFC on switch priority 3 plus ECN on TC3. The buffer must never overflow for RoCE: the switch pauses upstream before it would drop.
nv set qos roce state enabled nv set qos roce mode lossless nv config apply
nv set qos congestion-control default-global traffic-class 3 ecn enabled nv set qos congestion-control default-global traffic-class 3 min-threshold 150000 nv set qos congestion-control default-global traffic-class 3 max-threshold 1500000 nv set qos congestion-control default-global traffic-class 3 probability 100 nv show interface swp16 counters qos pfc-stats
- Default ECN profile: enabled on egress queue 0, min threshold 150,000 bytes, max 1,500,000 bytes, probability 100%, RED disabled. ECN marking probability has 1% hardware granularity. source
- ECN threshold on traffic class 3 is 1,501,500 bytes. ⚠ The published numbers disagree across pages (1,501,500 B on the RoCE page; 146.48 KB / 156 KB min and 1.43 MB max in the 5.18 extraction; 150,000 / 1,500,000 B on the QoS page). Teach the shape — min ≈ 150 KB, max ≈ 1.5 MB, probability 100% — and read `nv show qos roce` on the actual switch. source
- PFC watchdog defaults: polling interval 100 ms, robustness 3 iterations; per-interface state defaults to disable. It operates only on lossless queues on physical swp interfaces. source
- Link pause and PFC cannot coexist on the same port. source
- Dynamic (percentage) ECN requires Spectrum-4 or later and reached GA in Cumulus Linux 5.18. ⚠ The NVUE reference lists the mode value as `remote` but its own example uses `relative`; verify on a running 5.18.1 switch. source
- On Spectrum-X, switches generate ECN marks only when adaptive routing capacity is exhausted — ECN is a second-line signal, not the first tool. source
- PFC RX and TX are enabled in lossless mode and disabled in lossy mode. source
- One command sets buffer pools, classification and priority mappings, PFC and ECN thresholds, and scheduling for regular, RoCE and CNP traffic. source
RoCE (CL 5.18) · Quality of Service · Packet Trimming · NVUE QoS set
2Headroom: the buffer for what is already on the wire
XOFF is not instantaneous. Between the moment the switch decides to pause and the moment the upstream port actually stops, the pause frame has to traverse the cable, the upstream port has to notice it, and everything already on the wire and in the sender’s pipeline still arrives. Headroom is the reserved buffer that absorbs exactly that, and cable length is its main input — which is why cable-length is a QoS parameter and not documentation.[2][1]
The failure is symmetric and both directions are silent. Configure 100 m on a rack of 2 m DACs and you reserve headroom for roughly fifty times the in-flight data that can exist; that memory comes out of the same pool your lossless traffic needs, so you get no-buffer-discard on a switch that looks correctly configured.[4] Configure the 100 m default on a 300 m campus run and headroom is too small, so frames arrive after XOFF with nowhere to go and the “lossless” fabric drops.
Cumulus 5.17.0 added nv set qos pfc <profile-id> small-packet-probability — “Probability percentage for small packets” — as a second input to headroom sizing, because worst-case headroom assumes full-size frames while a fabric carrying many small packets needs a different reservation. Cumulus Linux 5.18.0 promotes it: “Lossless headroom size based on small packet probability is generally available”.[2][5]
The other lever is sharing. From 5.17.0, nv set interface <id> qos shared-headroom-pool enabled lets priority groups draw from a common headroom pool instead of each reserving its own, tuned globally with nv set qos advance-buffer-config <profile> shared-headroom exclusive-headroom-per-pg and … required-headroom-per-pg (both 19456-2621440 bytes) and … oversubscription-ratio (1-256).[2] Oversubscribing headroom is a bet that not every priority group bursts at once — reasonable on a converged fabric, less so on a synchronised all-reduce.
3The watchdog, and the conflict that silently disables everything
PFC has a pathological failure: a pause loop. If A pauses B and B pauses C and C pauses A, the lossless priority deadlocks and no traffic moves at all while every counter says “no drops”. The PFC watchdog exists to detect and break that.
Its defaults are a 100 ms polling interval and a robustness of 3 polling intervals — that is, roughly 300 ms of continuous pause before it acts.[1] Its scope is narrow and worth memorising: “PFC watchdog only works for lossless traffic queues” and “You can only enable PFC watchdog on a physical interface (swp)”.[1] Per-interface state defaults to disabled.[1][2]
You read it with nv show interface <if> qos pfc-watchdog, which reports watchdog state and deadlock detection status per traffic class with deadlock occurrence counts.[4] A non-zero occurrence count is not a warning about the future; it means a pause storm already happened and was mitigated on this box. The cause is almost always upstream, so that counter is a direction indicator, not a diagnosis.
The conflict to check on any inherited configuration: “You can not configure link pause and PFC on the same port”.[1] Global link pause is a common leftover on ports that were once storage or once someone’s lab, and it is a hard conflict rather than a precedence rule. On a leaf where lossless “just doesn’t work” on one interface and works everywhere else, this is the second thing to check after classification.
4ECN thresholds, and the numbers that seem to disagree
ECN is the mechanism that should act first. Instead of pausing or dropping, the switch sets the CE codepoint on a fraction of the packets in a congested queue and lets the endpoints slow themselves down — RFC 3168’s rule being that CE should only be set “if the router would otherwise have dropped the packet”.[6]
The NVUE surface: nv set qos congestion-control <profile-id> traffic-class <tc> ecn enabled|disabled, … mode, … min-threshold <bytes>, … max-threshold <bytes>, … probability <1-100> and … red enabled|disabled, with the absolute-mode forms introduced in Cumulus Linux 5.3.0.[2] Below the minimum threshold nothing is marked; above the maximum everything is; in between the marking probability ramps toward the configured probability.
Now the numbers, which is where the published documentation looks contradictory. Three figures circulate: the generic QoS page’s default profile is “ECN on egress queue 0 with a minimum buffer threshold of 150000 bytes and a maximum buffer threshold of 1500000 bytes, probability 100%, RED disabled”; the RoCE page’s defaults table gives “Port ECN absolute threshold is 1501500 bytes for traffic class 3”; and the RoCE page’s own nv show output prints “min-threshold 146.48 KB” and “max-threshold 1.43 MB”.[1][3]
Do the arithmetic: 150000 / 1024 = 146.48, and 1501500 / 1048576 = 1.43. Two of the three figures are the same numbers rendered in kibibytes and mebibytes. That reconciliation is arithmetic performed here, not a statement NVIDIA publishes, and it does not explain every figure in circulation — the research notes also record an interface-level “156 KB” that does not reduce as cleanly. Teach the shape — minimum around 150 KB, maximum around 1.5 MB, probability 100% — and read the actual values off the box with nv show qos roce.[3][4]
One number that is not a rounding artefact: ECN marking has “a hardware granularity of one percent; effective probabilities below one percent do not produce any marking”.[1] A computed probability that lands under 1% is inert, not gentle.
5Dynamic ECN, and one keyword nobody should guess
Absolute thresholds in bytes have a scaling problem: the right number depends on the switch’s buffer size and on how many ports are congested at once, so a value tuned for one platform is wrong on the next. Dynamic ECN replaces bytes with percentages of the available buffer. It uses min-threshold-percent and max-threshold-percent (0-100), requires Spectrum-4 or later, arrived in Cumulus Linux 5.17.0 and reached general availability in 5.18.0.[2][5]
Which is where the documentation defeats you. The NVUE reference’s syntax description for nv set qos congestion-control <profile-id> traffic-class <tc> mode reads: “Specify remote to enable dynamic ECN or absolute to disable dynamic ECN.” The same page’s worked example reads nv set qos congestion-control default-global traffic-class 4,5,7 mode relative.[2] Re-checked on 2026-09-07: the contradiction is still there. remote and relative are both plausible — the feature is a relative threshold, and remote could be a typo for relative or a distinct legacy value.
This course does not publish a guess. The rule for your runbook: run the command with tab completion on the target switch, or attempt both and keep the one the parser accepts, and record the result with the switch’s exact version. Never copy either keyword into a customer change record on the strength of a doc page that argues with itself. NCP-AIN objective 2.2 covers enabling and verifying QoS, ECN, PFC, adaptive routing and telemetry — and knowing which of two documented keywords is real is precisely the operational judgement that separates a certificate from a field engineer.[10]
Two more scope notes. Cross-switch consistency of “ECN and RED threshold settings” is netq check roce test 2 — the only practical way to prove that all your leaves agree.[8] And none of this section is testable in simulation: a virtual Cumulus switch implements no QoS shaping, buffer management or packet marking.[7]
Scenario: leaf01, SN5600, 64 ports. swp1-swp32 face GPU hosts over 2 m DAC. swp33-swp40 are uplinks over 30 m of fibre to spines. RoCE is enabled, lossless, and nv show qos roce reports cable-length 100.
Step 1 — state the problem in one line. Headroom on every port is sized for 100 m of in-flight data. On the host ports the real figure is 2 m, so roughly fifty times more headroom is reserved than can ever be needed, and that memory is taken from the same buffer the lossless pool needs.[2][1]
Step 2 — decide the scope. nv set qos roce cable-length is a system-level value. Because this leaf has two cable populations, the correct instrument is a PFC profile applied per interface group, not one global number. Set the RoCE-level value to the longest run so nothing under-reserves, then tune down where you can.
Step 3 — write it:
nv set qos roce cable-length 30
nv set qos pfc host-ports switch-priority 3
nv set qos pfc host-ports rx enable
nv set qos pfc host-ports tx enable
nv set qos pfc host-ports cable-length 5
nv set interface swp1-32 qos pfc profile host-ports
nv config diff
nv config apply --confirm 5mStep 4 — predict what should change. Lossless pool occupancy under the same load should fall, because headroom is no longer reserving memory for 100 m of cable. no-buffer-discard on traffic class 3 should not rise; if it does, headroom is now too small and 5 m was too aggressive.[4]
Step 5 — verify, in this order: nv show qos roce for the system value, nv show interface swp1 qos roce status for the per-interface view, nv show interface swp1 counters qos pfc-stats for pause packets and duration, and nv show interface swp1 qos roce counters for no-buffer-discard and pg-max-usage.[4] Then confirm, then save.
Note on the 300 m case: with a 300 m campus run the same arithmetic runs the other way. The 100 m default under-reserves by a factor of three, headroom is exhausted before the upstream port stops, and the fabric drops on a lossless priority. cable-length 300 is inside the documented 1-100000 range.[2]
Scenario: leaf02, all 64 ports on 3 m DAC to hosts except swp61-swp64, which run 300 m to a second building.
- Which one number is wrong for every port on this box today, and in which direction for each population? ________
- Set the system value to the longest run so nothing under-reserves:
nv set qos roce ____ ____. - Create a profile for the short-cable ports:
nv set qos pfc rack switch-priority ____,nv set qos pfc rack rx ____,nv set qos pfc rack tx ____,nv set qos pfc rack ____ 5. - Apply it to the right ports:
nv set interface ____ qos pfc profile rack. - Read the change before committing:
nv config ____. - Commit with an automatic rollback:
nv config apply ____ ____. - Name the counter that tells you headroom is now too small: ____.
- Name the counter that tells you PFC is firing at all, and the command that prints it: ____ /
nv show interface swp1 counters qos ____. - Which one of the two changes above would you make first on a production fabric, and why?
Three transcripts arrive from a customer leaf under an all-reduce workload. For each, classify the cause as (a) thresholds too high, (b) classification wrong, or (c) lossless pool or headroom too small — then name the single command that confirms it and the one-line fix.
pfc-statsshows TX pause packets in the millions on switch priority 3 and a large pause duration;qos roce countersshowsecn-marked-packetsat 0.pfc-statsshows zero pause on every switch priority;qos roce countersshowsno-buffer-discardclimbing steadily andpg-max-usageat its ceiling.pfc-statsshows zero pause on switch priority 3 but non-zero pause on switch priority 0;egress-queue-statsshows nearly all traffic on traffic class 0 and almost none on traffic class 3.
Acceptance criteria: each answer names one class from (a)/(b)/(c), one confirming command from the NVUE show surface, and one fix expressed as an nv set command or a host-side change. At least one of your three answers must point at the host rather than the switch. State which of the three you would escalate to NVIDIA and which you would fix in the customer’s own change window.
Case update — three metres, not a hundred
The deadlock counts point one hop upstream, and on that leaf the reads explain themselves: heavy pause on switch priority 3, ecn-marked-packets at zero, and cable-length still at the documented default of 100 metres on a rack of DACs the operator’s label maker says are three.[2][4] The emergency brake was doing all the work because the gentle signal never fired, and headroom was reserving buffer for cable that does not exist.[1] The corrections go into the morning window, with a measurement between each step.
What you say to the operator: “Read the values off the box — the two documented cable-length defaults are not the same number.”[2] Five days out, the architect walks into the design review with a printed blog post about packet trimming.
Lab
Goal: make PFC visible on the Dell-lab hosts by driving congestion and watching pause counters rise as throughput falls. Mutating on the hosts.
Pre-flight inventory (save the output — it is the rollback baseline):
ibstat; ibdev2netdev
sudo mlxreg -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --get
cat /sys/class/infiniband/<dev>/tc/1/traffic_class
ethtool -S <ifname> | grep -i -E 'pause|discard'- Establish the uncongested baseline. Receiver:
ib_write_bw -d <dev> -F. Sender:ib_write_bw -d <dev> -F <receiver-ip> --duration 20. Record the bandwidth and theethtool -Spause counters before and after. Expected: pause counters essentially unchanged. - Enable PFC on priority 3 on every host involved:
sudo mlxreg -y -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --set "prio_mask_rx=0x08,prio_mask_tx=0x08,pfctx=1,pfcrx=1,pprx=0,pptx=0". Verify with--get. Rollback: re-run--setwith the field values captured in the pre-flight block. - Confirm the traffic is marked into the lossless class:
echo 106 | sudo tee /sys/class/infiniband/<dev>/tc/1/traffic_classon every sender. Rollback: restore the pre-flight value. - Create incast. Start one receiver and run
ib_write_bwfrom two or more senders simultaneously against it, each with--duration 60. If you have only two hosts, run several parallel client processes with distinct-pport numbers. Expected: aggregate received bandwidth below the sum of the individual runs. - While the test runs, sample the receiver every few seconds:
watch -n 2 "ethtool -S <ifname> | grep -i pause". Expected: pause counters climbing. Correlate the moment they start climbing with the point where per-sender bandwidth drops in theib_write_bwoutput. - Repeat step 4 with PFC disabled on the senders (
prio_mask_tx=0x00,pfctx=0) and compare. Expected: pause counters stay flat and you see loss or retransmission behaviour instead. Rollback: re-enable PFC or restore the pre-flight values. - Restore everything from the pre-flight block and re-run step 1. Expected: the original baseline bandwidth returns. If it does not, one of the register writes did not roll back — re-check with
--getagainst your saved output.
Optional (customer lab, read-only on the switch): during the same incast run nv show interface <if> counters qos pfc-stats, nv show interface <if> qos roce counters and nv show interface <if> qos pfc-watchdog on the leaf. Expected: TX pause packets and duration rising on switch priority 3, ecn-marked-packets non-zero, and a deadlock occurrence count of zero. A non-zero deadlock count during a two-host incast means the problem is upstream, not in your test.
Goal: build the threshold worksheet and settle the relative versus remote question empirically. No hardware needed; a DSX Air node covers step 5.
- Compute the two cable-length cases by hand. For a 2 m in-rack DAC and a 300 m campus run, write down: how much in-flight data the 100 m default assumes relative to reality, which direction the error runs, and which counter would reveal it. Expected: over-reservation and wasted pool memory for 2 m; under-reservation and
no-buffer-discardfor 300 m. - Write the PFC profile commands for each case, using the profile form (
nv set qos pfc <profile> …) rather than the system-level RoCE value, and say which interfaces each profile is applied to. - Take the three transcripts from the Worked problem above and classify each as (a) thresholds too high, (b) classification wrong, or (c) lossless pool or headroom too small. Expected: one of each. If two of your answers land in the same class, re-read which counter is zero in each transcript — the zero is the signal.
- For each classification write the single
nv showcommand that confirms it. Expected:counters qos pfc-statsfor pause,qos roce countersforecn-marked-packetsandno-buffer-discard,counters qos egress-queue-statsfor per-traffic-class volume and WRED drops. - On a DSX Air node, settle the keyword. Run
nv set qos congestion-control default-global traffic-class 3 modeand press Tab to see what the parser offers; then trymode relative,mode remoteandmode absolutein turn, runningnv config diffafter each. Expected: the accepted keyword or keywords appear in the pending diff and the rejected one errors. Record the exact Cumulus version fromnv show systemalongside the result. If the node refuses all three, note that too — a negative result with a version attached is still a usable answer. Clean up withnv unset qos congestion-control default-global traffic-class 3 modeandnv config apply. - Do not conclude anything about buffering, marking or pause from this simulation: a virtual Cumulus switch implements no QoS shaping, buffer management or packet marking.
Deliverable: a one-page worksheet with the two cable cases, the three transcript classifications, and a dated line recording which ECN mode keyword your Cumulus version actually accepts.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in five sentences, why a lossless fabric that is sending a lot of pause frames is a fabric that is tuned wrong rather than a fabric that is working.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 QoS and RoCE pages, NVUE 5.x QoS reference (the mode remote/absolute vs relative contradiction re-checked and still present), RFC 3168; re-fetched 2026-09-07. Dates are when each page was fetched.
- Quality of Service | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference - QoS set/unset commands · fetched 2026-09-07
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference - QoS show commands · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- RFC 3168 - The Addition of Explicit Congestion Notification (ECN) to IP · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- NVIDIA/doroce-linux - doRoCE.sh host-side RoCE configuration script · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.