PFC, lossless and the case for lossy

Episode 3 — The architect who wants more pause

The situation · Building D war room, three days out

Three days out, the pod’s two-tier Spectrum fabric on the whiteboard and one request circled: raise the PFC thresholds. The network lead has the numbers — throughput fine at eight nodes, down about forty percent at sixty-four — and the conclusion they seem to support: the fabric needs more pause, not less. The Dell SE would like you to agree, because row 41 still says “lossless: yes” and acceptance is Friday.

You ask for pause counters first, from both hosts and every hop. Direction names the victim: rx_prio[p]_pause climbing means the network cannot absorb what the adapter sends, tx_prio[p]_pause climbing means the adapter cannot absorb what the network sends.^[4] Pause on an uplink carrying no congested flow of its own is not protection; it is head-of-line blocking propagated back from somewhere else. “Show me the counter,” the network lead says, and for the first time this week you are on the same side of a sentence.

Flow control exists because RoCE needs it. NVIDIA states plainly that RoCE requires a form of flow control, that global flow control is normally undesirable for performance reasons, and that the normal and optimal way is PFC — enabled on all endpoints and switches in the flow path.^[1] IEEE 802.1Qbb was written for exactly that: pause one priority rather than the whole link, so converged traffic is not stalled to protect RDMA.^[2]

Hop-by-hop is the whole property, so one hop without PFC makes the path lossy whatever the drawing says. Before deciding how much PFC to deploy, be sure what it is already doing.

1Why RoCE asks for flow control, and why not the global kind

NVIDIA states the requirement plainly: “In order to function reliably, RoCE requires a form of flow control. While it is possible to use global flow control, this is normally undesirable, for performance reasons.”^[1] And the recommendation: “The normal and optimal way to use RoCE is to use Priority Flow Control (PFC). To use PFC, it must be enabled on all endpoints and switches in the flow path.”^[1]

That second sentence is the whole lesson in one line. PFC is hop-by-hop; a path is lossless only if every link in it, in both directions, honours pause on that priority — including the two NICs, which are endpoints, not bystanders.^[1] “We enabled it on the leaf” describes one hop of a path that has at least four.

The standard behind it is IEEE 802.1Qbb, titled “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks – Amendment: Priority-based Flow Control”, whose PAR was approved 2008-03-27 and whose standard was approved 2011-06-16.^[2] Its mechanism “implements a mechanism similar to the IEEE 802.3x PAUSE, but operating on individual priorities” identified by the VLAN-tag priority values.^[2] Whether 802.1Qbb has since been folded into a later 802.1Q revision is not stated on the IEEE project page, so treat any statement about which base standard now contains it as unverified.^[2]

The practical difference between the two pause mechanisms is what they stop. Global pause stops the link; PFC stops one priority. On a converged AI host that carries RDMA, storage and management on the same port, global pause protects RDMA by stalling everything else, which is the performance objection NVIDIA raises.^[1] On Windows the migration is explicit — global pause is turned off with Disable-NetQosFlowControl and Disable-NetAdapterQos <interface name> before PFC priorities are assigned.^[8]

lossless, no rate signal

Offered load95% of 400GECN threshold (TC3)1,501,500 B

stage 1 / 4

no CE markingno pauseno losssender did not reactgoodput ≈ 95% ⚠

⚠ Model only: a 4 MB TC3 pool at 400 Gb/s (50,000 B/µs). The shape is right; the numbers are not a simulation of your fabric.

PFC only · stage 1 of 4

1. Sender transmits

Nothing in this path can tell the sender to slow down. PFC is flow control, not congestion control.

Predict: what does the next stage do? (optional, saved)

Reaction Point (RP) — sender NIC

The sending NIC. On the host this is the r_roce_ecn_rp algorithm, enabled per priority: echo 1 > /sys/class/net/<if>/ecn/roce_rp/enable/3.

It is the only element that can lower the offered load. Marking and CNPs are worth nothing if the RP never acts: the rate stays flat and the queue keeps growing.

Tooling: Counter: rp_cnp_handled (sender). Read with rdma statistic show link mlx5_0/1, or straight from hw_counters.

FAE angle: when np_cnp_sent climbs on the receiver and rp_cnp_handled stays flat on the sender, stop tuning thresholds. The CNPs are being lost or misclassified on the way back.

Counters — which one is still moving?

np_ecn_marked_roce_packetsflatreceiver · hw_counters — CE seen in the RoCEv2 IP header. Flat while the queue is deep = the switch is not marking your queue.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_ecn_marked_roce_packets
```
np_cnp_sentflatreceiver · hw_counters — CNPs the notification point put back on the wire, DSCP 48 / priority 6.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_cnp_sent
```
rp_cnp_handledflatsender · hw_counters — CNPs the reaction point acted on. Moves on a different host from np_cnp_sent — read both.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_handled
```
rp_cnp_ignoredflatsender · hw_counters — CNPs that arrived and were dropped on the floor — r_roce_ecn_rp is not enabled for this priority.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_ignored
```
rx_prio3_pauseflatsender · ethtool -S — The switch is pausing you on priority 3. Direction identifies the victim.
```
ethtool -S <if> | grep -E "prio3_pause"
```
tx_prio3_pauseflatreceiver · ethtool -S — You pausing the network. Flat here: the receiver NIC is not the bottleneck in this scenario.
```
ethtool -S <if> | grep -E "prio3_pause"
```
packet_seq_errflatsender · hw_counters — The fabric dropped or reordered RoCE. On a lossless design this number should stay at zero.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err
```
rx_prio3_buf_discardflatreceiver · ethtool -S — Packets discarded for lack of per-host receive buffers on priority 3.
```
ethtool -S <if> | grep -E "prio3_(buf_)?discard"
```

PFC only

Hop-by-hop pause on one priority. Nothing is dropped and nothing slows down — the pause walks upstream and blocks every flow on priority 3, congesting or not.

mlnx_qos -i <if> --pfc 0,0,0,1,0,0,0,0    # host side, priority 3
nv set qos roce            # switch: lossless is the NVUE default
nv config apply

Queue is shallow; no mechanism has anything to do yet. Raise the offered load.

Cumulus RoCE (NVUE) · ECN sysfs · ZTR-RTT parameters · mlx5 counters · rdma statistic

Run the loop in pfc mode at high offered load and watch the pause propagate upstream. Then inject the pfc-gap fault to see what one hop without PFC does to the whole path.

2What lossless actually is, measured on a switch

On NVIDIA Spectrum switches with Cumulus Linux, the whole RoCE posture is one command: nv set qos roce followed by nv config apply, and “NVUE defaults to RoCE lossless. The command nv set qos roce and nv set qos roce mode lossless are equivalent.”^[3] The current release exposes five modes: lossless, lossy, lossless-single-ipool, lossy-multi-tc and dci-1.^[3]

What both main modes program is worth memorising, because it is the reference every host configuration has to match. RoCE data lands on switch priority 3 to traffic class 3, marked DSCP 26; CNP lands on switch priority 6 to traffic class 6, marked DSCP 48; everything else goes to TC0.^[3] Scheduling is TC6 strict priority, TC3 DWRR 50 percent and TC0 DWRR 50 percent.^[3] ECN is programmed on TC3 in both modes, and an LLDP App TLV advertises “UDP, Protocol:4791, Priority: 3” so hosts can learn the mapping.^[3]

What lossless adds over lossy is exactly two things: PFC on switch priority 3, and switch priority 3 mapped to the lossless buffer pool.^[9]^[3] Lossy mode is not “RoCE without congestion control”; it “Implements RFC 3168 to signal congestion experienced to the receiver” and keeps the ECN loop.^[3] That framing settles a lot of arguments: choosing lossy is choosing to drop rather than pause, not choosing to give up.

Two numbers need care in front of a customer, and they are not the same field. The default-settings table gives a port ECN absolute threshold of 1501500 bytes on traffic class 3, and both commands render it as max-threshold 1.43 MB.^[3] The minimum threshold is where the two levels disagree: nv show qos roce prints min-threshold 146.48 KB while nv show interface <interface-id> qos roce status prints min-threshold 153.00 KB, and the page explains that the system-level value intended for programming on Spectrum-4 is 150,000 bytes but “the interface-level command reflects the actual hardware-programmed value, which is 159,744 bytes (156 KB)”.^[3] Quote the minimum and the absolute threshold as two separate numbers; pairing them as one value is the order-of-magnitude error.

3Reading the argument off the counters

The judgement is not a preference, it is a measurement, and the mlx5 driver publishes both sides of it. On the host, rx_prio[p]_pause is “The number of pause packets received with priority p on a physical port. If this counter is increasing, it implies that the network is congested and cannot absorb the traffic coming from the adapter”, and it is “available only if PFC was enabled on priority p”.^[4] The mirror, tx_prio[p]_pause, is “The number of pause packets transmitted on priority p on a physical port. If this counter is increasing, it implies that the adapter is congested and cannot absorb the traffic coming from the network.”^[4] Direction names the victim.

Three more counters change the diagnosis outright. rx_pause_ctrl_phy and tx_pause_ctrl_phy count link-layer, non-PFC pause frames — non-zero there while you believe PFC is configured means global pause is doing the work and PFC is not.^[4] rx_prio[p]_buf_discard is “The number of packets discarded by device due to lack of per host receive buffers”, which says the lossless buffer for that priority is undersized for the bandwidth-delay product of the hop.^[4] And tx_pause_storm_error_events counts the case where the device sent pause frames until a timeout disabled pause transmission, “on the period where pause frames were disabled, drop could have been” occurred.^[4] That last one is a lossless fabric caught converting itself into a lossy one under load.

So the evaluation rule an FAE can defend: if the customer’s pain is loss — retransmits, sequence errors, throughput collapse with clean pause counters — the argument favours lossless, or at minimum a correctly sized lossless buffer. If the pain is stalling — rx_prio3_pause and rx_prio3_pause_duration climbing on ports that carry no congested flow, or storm events non-zero — the argument favours reducing PFC’s reach, whether by moving to lossy with ECN or by fixing the marking so that only RoCE occupies the paused priority.^[4]^[3] The two symptoms point in opposite directions and the counters distinguish them in one command.

4The switch side a Dell account will actually touch

Two switch families come up on a Dell OEM account and they are not configured the same way. On NVIDIA Spectrum with Cumulus, it is nv set qos roce plus nv config apply, verified with nv show qos roce, nv show interface <interface-id> qos roce status and nv show interface <interface-id> qos roce counters.^[3] On Dell Enterprise SONiC, RoCE is enabled with roce enable pfc-priority 3,4, modelled in Dell’s own Ansible collection as roce_enable: true with pfc_priorities: '3,4' and state: merged, the only supported state.^[6]

- name: Enable RoCE for PFC priorities
  dellemc.enterprise_sonic.sonic_roce:
    config:
      roce_enable: true
      pfc_priorities: '3,4'
    state: merged

The ordering rule in that module matters: roce_enable “Activates or deactivates RoCEv2 default buffer settings” and “Must be enabled before configuring PFC priorities”.^[6] Buffers first, then pause — the same logical order as the NVUE lossless pool.

Before promising any of it, check the model. The Dell Enterprise SONiC compatibility matrix for release 4.4.0 marks RoCEv2 supported on four of five platform families and not supported on the campus family, and in the per-model feature table RoCE V2 is yes on four of ten models while WRED, ECN and priority flow control are yes on all ten.^[7] A switch that lists PFC and ECN is not thereby a RoCE switch. Note also that this course could only fetch the 4.4.0 matrix; the lab image in question is 4.5.1, so treat model-level support as something to re-check against the customer’s release rather than quote from here.^[7]

Decide lossless or lossy for a customer, with evidence

Case: a Dell AI Factory customer runs 32 nodes on a two-tier Spectrum fabric in lossless mode. Training throughput is fine at 8 nodes and drops about 40 percent at 32. They ask to raise PFC thresholds.

Collect symmetric evidence before forming a view. From every host:^[4]

ethtool -S <if> | grep -E "prio[0-9]_pause|prio[0-9]_pause_duration|pause_ctrl_phy|pause_storm"
ethtool -S <if> | grep -E "prio[0-9]_buf_discard|prio[0-9]_discards|prio[0-9]_marked"

From the switch: nv show interface <interface-id> qos roce counters on the leaf ports and the uplinks.^[3]

Classify the pattern. Rising rx_prio3_pause on hosts whose flows are not congested, and rising pause on uplinks carrying unrelated traffic, is head-of-line blocking. Rising rx_prio3_buf_discard with low pause is an undersized lossless buffer. Non-zero tx_pause_storm_error_events anywhere is the fabric already failing over into loss.^[4]
Test the customer’s own hypothesis honestly. Raising PFC thresholds delays the pause but does not stop it propagating; it enlarges the queue the pause is protecting. If the evidence in step 2 is blocking rather than discard, say so and explain that the requested change extends the symptom.
Check whether the paused priority is carrying anything it should not. Confirm on both hosts that only RoCE marks into priority 3 — mlnx_qos -i <if> for the trust state and the DSCP map — because a management or storage flow sharing priority 3 makes every pause more expensive than it needs to be.^[5]^[3]
Confirm ECN is actually doing work before considering a mode change. NVUE programs ECN on TC3 in both modes, so a fabric where pause is the only mechanism acting means ECN is configured but not reacting - which is a host question, not a switch one, and is the subject of the next lesson.^[3]
Give a decision with a rollback. If blocking dominates: propose nv set qos roce mode lossy on a single rail or pod, nv config apply, run the same 32-node job, and compare the same counters; rollback is nv set qos roce mode lossless plus nv config apply.^[3] Note the documented asymmetry - once lossy is set, a bare nv set qos roce does not change the mode back, so the rollback must name mode lossless explicitly.^[9]
Report in the customer’s units: throughput at 8 and 32 nodes before and after, plus the pause and discard deltas that explain it. Never report a mode change without the counter pair that justified it.

Episode 3 — What the counters decided

How it ended

The counters answer for you: pause on ports with no congested flow, and storm events non-zero — the mechanism is costing throughput on healthy paths, so the argument runs against widening its reach.^[4] You propose keeping ECN on TC3, which both NVUE RoCE modes program anyway, and reducing what shares the paused priority before anyone touches a threshold.^[3] What you say to the architect: raising thresholds extends the reach of the behaviour you are already paying for, so count the hops that honour pause before calling the fabric lossless at all.^[1] The notebook gains a page. At 02:10 the operator calls: retransmits since the evening batch started, and nothing is slowing down.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: on the Dell-lab pair, prove the host half of PFC and read the direction of pause. Step 3 is mutating and names its rollback; the switch half is optional and belongs to a customer lab.

Pre-flight, read-only. Capture on both hosts:
```
ibdev2netdev
mlnx_qos -i <if> | tee /tmp/qos-before-$(hostname).txt
ethtool -S <if> | grep -E "prio[0-9]_pause|pause_ctrl_phy|pause_storm" | tee /tmp/pause-before-$(hostname).txt
```
Expected: an all-zero PFC row on a stock host, and per-priority pause counters that may be absent entirely - they are documented as available only if PFC was enabled on that priority.^[5]^[4]
Note the global-pause baseline. If rx_pause_ctrl_phy or tx_pause_ctrl_phy is already non-zero, global pause is running and must be understood before PFC is added, or you will be measuring two mechanisms at once.^[4]
Mutating - enable PFC on priority 3 on both hosts. Rollback: re-apply the PFC vector recorded in step 1, normally all zeros.^[5]
```
mlnx_qos -i <if> --pfc 0,0,0,1,0,0,0,0
mlnx_qos -i <if> | sed -n "/PFC configuration/,+3p"
```
Expected: enabled 0 0 0 1 0 0 0 0. Do not describe the path as lossless - only the two endpoints are configured, and PFC must be on at every hop.^[1]
Generate congestion and read direction, read-only. Server ib_write_bw -d mlx5_0 -b -q 8 --tclass=104, client ib_write_bw -d mlx5_0 -b -q 8 --tclass=104 <server-ip> - if the client aborts with a parameter-mismatch error, the two command lines differ, and perftest states that the options specific to modes must be the same for both server and client.^[10] Then during the run on both sides:
```
ethtool -S <if> | grep -E "prio3_pause|prio3_pause_duration|prio3_buf_discard|prio3_marked"
```
Expected on a fabric with PFC enabled somewhere upstream: one side’s rx_prio3_pause moves. If nothing moves anywhere, either no hop between the two hosts is running PFC on priority 3, or the traffic is not landing on priority 3 at all - go back and confirm the marking chain from lesson 1.^[4]
Roll back and confirm. Re-apply the captured PFC vector on both hosts, re-read mlnx_qos -i <if>, and diff against /tmp/qos-before-$(hostname).txt until it is empty.
Optional, customer lab only: repeat step 4 with a Spectrum switch in lossless and then lossy, comparing the same host counters plus nv show interface <id> qos roce counters on the switch.^[3] Rollback for the switch is nv set qos roce mode lossless and nv config apply, named explicitly because a bare nv set qos roce will not restore it.^[9]

Evidence / notes (commands run, outputs, what surprised you)

PFC, lossless and the case for lossy

Before you read: what do you already know?

After this lesson you can

Episode 3 — The architect who wants more pause

1Why RoCE asks for flow control, and why not the global kind

1. Sender transmits

2What lossless actually is, measured on a switch

3Reading the argument off the counters

4The switch side a Dell account will actually touch

Episode 3 — What the counters decided

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere