Scenario: "we do not want to configure PFC"

Episode 3 — The slide that promises no switch configuration

The situation · Hotel lobby, 07:15, ninety minutes before the customer meeting

The Dell SE turns the laptop toward you over breakfast. One slide, one line: zero touch, no switch configuration required. Row 51, and his coffee is cold again. The night-shift operator gave the real news at 02:10 in fewer words - no PFC on the hops shared with other tenants - and has labelled those two uplinks NOT OURS so nobody argues twice. In ninety minutes the customer wants you to confirm on the record that the ConnectX cards already racked will run ZTR-RTT. You have no card in front of you.

Flow control is in this conversation because RoCE was never built to absorb loss. NVIDIA is blunt: “In order to function reliably, RoCE requires a form of flow control”, and “The normal and optimal way to use RoCE is to use Priority Flow Control (PFC). To use PFC, it must be enabled on all endpoints and switches in the flow path.”^[1] Every hop is the cost the customer is refusing, and the refusal is operational: one wrong hop turns a lossless design lossy in silence, and a mis-tuned one produces pause storms the driver counts as tx_pause_storm_error_events.^[8] Zero Touch RoCE is NVIDIA’s answer to that objection - RDMA “without requiring any special switch configuration”.^[6] On the 07:40 call the SE asks the NVIDIA PM whether a later card makes it the default. “Not announced,” she says, in the tone of someone who says it weekly.

The risk is never the technology. It is the sentence you say from memory.

Start with what the customer is actually refusing.

1What the customer is actually refusing

“We do not want to configure PFC” is rarely a statement about flow control. It is a statement about operational surface: PFC has to be right on every hop.

NVIDIA is unambiguous about the requirement. “In order to function reliably, RoCE requires a form of flow control. While it is possible to use global flow control, this is normally undesirable, for performance reasons”, and “The normal and optimal way to use RoCE is to use Priority Flow Control (PFC). To use PFC, it must be enabled on all endpoints and switches in the flow path.”^[1] The standard behind it is IEEE 802.1Qbb, which “implements a mechanism similar to the IEEE 802.3x PAUSE, but operating on individual priorities” identified by the VLAN-tag priority values; the standard was approved on 2011-06-16.^[14]

The cost of getting it right is visible in the counters. rx_prio[p]_pause rising means “the network is congested and cannot absorb the traffic coming from the adapter”; tx_prio[p]_pause rising means “the adapter is congested and cannot absorb the traffic coming from the network”.^[8] One misconfigured hop turns a lossless design into a lossy one silently. And when the design is right but the tuning is not, the driver has a counter for that too: tx_pause_storm_warning_events counts “the number of times the device was sending pauses for a long period of time”, while tx_pause_storm_error_events counts the times pause transmission was disabled by timeout, during which “drop could have been” occurred.^[8] A pause storm is how a mis-tuned lossless fabric becomes a lossy fabric with extra steps.

So the first question back to the customer is not “why not”. It is: which symptom do you have today? Retransmits and rising packet_seq_err point toward needing more losslessness; rising rx_prio3_pause on ports that should be idle, and head-of-line blocking, point away from it.^[10]^[8] Those two answers lead to opposite recommendations, and you cannot write either one until you know which it is.

2Three options, and exactly what each one asks for

Option	Where it lives	What it requires	What it costs
Lossless PFC on priority 3	Switch and NIC	PFC on every hop in the flow path, both directions, hosts included	Head-of-line blocking, pause storms if mis-tuned
Lossy - ECN only	Switch and NIC	Correct ECN marking on the RoCE traffic class plus a strict-priority CNP queue	Drops under congestion; the transport retransmits
ZTR-RTT CC	NIC only	The programmable-CC path enabled on the specific card and its firmware	Nothing from the switch; a NIC-generation question instead

On the switch, both of the first two are one command. nv set qos roce - equivalently nv set qos roce mode lossless - enables RoCE and is the NVUE default; nv set qos roce mode lossy switches to ECN-only and “Implements RFC 3168 to signal congestion experienced to the receiver”; each is followed by nv config apply.^[2] Current Cumulus also offers mode lossless-single-ipool, mode lossy-multi-tc with packet trimming, and a Beta mode dci-1, “supported on Spectrum-4 through Spectrum-6”, that combines a PFC-lossless TC3 with an ECN-lossy TC4.^[2] One documented gotcha: “If you enable roce mode lossy, configuring nv set qos roce without a mode does not change the RoCE mode.”^[3]

What matters for the recommendation is how little separates the two modes. Both program port trust on, switch priority 3 to TC3 for RoCE and priority 6 to TC6 for CNP with everything else to TC0, ETS with TC6 strict and TC3 and TC0 at 50 percent WRR, an ECN absolute threshold on TC3, and an LLDP App TLV advertising UDP 4791 at priority 3. Only lossless adds PFC on switch priority 3 and maps SP3 to the lossless buffer pool.^[3] Default markings in both modes: DSCP 26 to switch priority 3, DSCP 48 to switch priority 6.^[2] Verification is nv show qos roce, nv show interface <id> qos roce status and nv show interface <id> qos roce counters.^[2]

On a Dell PowerSwitch the equivalent enablement is roce enable pfc-priority 3,4, modelled in Ansible as dellemc.enterprise_sonic.sonic_roce with roce_enable: true and pfc_priorities: '3,4', state: merged being the only supported state; the module notes that roce_enable “Activates or deactivates RoCEv2 default buffer settings” and “Must be enabled before configuring PFC priorities”.^[12] Before any of that reaches a slide, check the model: the Dell Enterprise SONiC compatibility matrix marks RoCEv2 supported on four of five platform families and not supported on the campus family, and in the per-model table RoCE V2 is yes on four of ten models while WRED, ECN and priority flow control are yes on all ten.^[13] The SmartFabric OS10 page that would give the exact class-map and policy-map sequence could not be fetched, so do not quote an OS10 configuration from memory.^[16]

lossless, no rate signal

Offered load140% of 400GECN threshold (TC3)1,501,500 B

stage 1 / 4

no CE markingPFC pausingno losssender did not reactgoodput ≈ 88% ⚠

⚠ Model only: a 4 MB TC3 pool at 400 Gb/s (50,000 B/µs). The shape is right; the numbers are not a simulation of your fabric.

PFC only · stage 1 of 4

1. Sender transmits

Nothing in this path can tell the sender to slow down. PFC is flow control, not congestion control.

Predict: what does the next stage do? (optional, saved)

Reaction Point (RP) — sender NIC

The sending NIC. On the host this is the r_roce_ecn_rp algorithm, enabled per priority: echo 1 > /sys/class/net/<if>/ecn/roce_rp/enable/3.

It is the only element that can lower the offered load. Marking and CNPs are worth nothing if the RP never acts: the rate stays flat and the queue keeps growing.

Tooling: Counter: rp_cnp_handled (sender). Read with rdma statistic show link mlx5_0/1, or straight from hw_counters.

FAE angle: when np_cnp_sent climbs on the receiver and rp_cnp_handled stays flat on the sender, stop tuning thresholds. The CNPs are being lost or misclassified on the way back.

Counters — which one is still moving?

np_ecn_marked_roce_packetsflatreceiver · hw_counters — CE seen in the RoCEv2 IP header. Flat while the queue is deep = the switch is not marking your queue.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_ecn_marked_roce_packets
```
np_cnp_sentflatreceiver · hw_counters — CNPs the notification point put back on the wire, DSCP 48 / priority 6.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_cnp_sent
```
rp_cnp_handledflatsender · hw_counters — CNPs the reaction point acted on. Moves on a different host from np_cnp_sent — read both.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_handled
```
rp_cnp_ignoredflatsender · hw_counters — CNPs that arrived and were dropped on the floor — r_roce_ecn_rp is not enabled for this priority.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_ignored
```
rx_prio3_pausemovingsender · ethtool -S — The switch is pausing you on priority 3. Direction identifies the victim.
```
ethtool -S <if> | grep -E "prio3_pause"
```
tx_prio3_pauseflatreceiver · ethtool -S — You pausing the network. Flat here: the receiver NIC is not the bottleneck in this scenario.
```
ethtool -S <if> | grep -E "prio3_pause"
```
packet_seq_errflatsender · hw_counters — The fabric dropped or reordered RoCE. On a lossless design this number should stay at zero.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err
```
rx_prio3_buf_discardflatreceiver · ethtool -S — Packets discarded for lack of per-host receive buffers on priority 3.
```
ethtool -S <if> | grep -E "prio3_(buf_)?discard"
```

PFC only

Hop-by-hop pause on one priority. Nothing is dropped and nothing slows down — the pause walks upstream and blocks every flow on priority 3, congesting or not.

mlnx_qos -i <if> --pfc 0,0,0,1,0,0,0,0    # host side, priority 3
nv set qos roce            # switch: lossless is the NVUE default
nv config apply

PFC holds the line: no drops, but the whole priority stops on that link and every other flow on priority 3 stops with it.

Cumulus RoCE (NVUE) · ECN sysfs · ZTR-RTT parameters · mlx5 counters · rdma statistic

Start in PFC and watch the pause propagate into the neighbour flow on the same priority. Then switch the mechanism to ecn and to ztr at the same offered load and compare which counter carries the signal.

3ZTR-RTT: what is verified, and what is not

ZTR-RTT is the option that asks the switch for nothing, which is exactly why it is attractive to a customer refusing fabric-wide PFC. Zero Touch RoCE “enables data centers to seamlessly deploy RDMA over Converged Ethernet (RoCE) without requiring any special switch configuration”.^[6] ZTR-RTT CC itself is “a congestion control (CC) mechanism within the DOCA Programmable Congestion Control (PCC) framework”, monitoring two indicators - RTT, “Real-time measurement of the delay between sending a packet and receiving its acknowledgment”, and CNP, “Explicit notifications sent by the network or receiver when congestion is encountered” - with additive-increase, multiplicative-decrease rate control and a “highly synchronized, hardware-assisted timestamping sequence” of four timestamps.^[4]

The published evidence is a 2021 blog: “superb RoCE performance at data center scale, without any special configuration of the switch infrastructure”, with 99 percent of conventional RoCE performance at 256 connections and 98 percent of throughput at over 16,000 connections.^[6] Quote those numbers with their date and their source, not as a current benchmark.

Now the part that decides whether the recommendation is a commitment or a slide. The ZTR-RTT pages state no supported ConnectX or BlueField generations and no minimum firmware, and the 2021 blog names none either.^[4]^[5]^[6] The only firm statements in this area belong to DOCA PCC, the framework: applications “can run on either the host machine or the NVIDIA BlueField-3 Platform (or later) target”, DOCA PCC “is supported only for the ETHERNET link type”, and the library “requires firmware version 32.38.1000 and higher”.^[7] Note what that is not: a hardware floor. Neither that page nor the ZTR-RTT pages disqualify a ConnectX part, and extending or borrowing a floor in either direction - as a requirement or as a permission - is an unsourced claim, and the single easiest way to turn this conversation into a support case. The correct move is to query the specific card.

The card-side surface is small. DOCA PCC’s prerequisites are the mlxconfig lines USER_PROGRAMMABLE_CC=1, PCC_INT_EN=0, FLEX_PARSER_PROFILE_ENABLE=10, REAL_TIME_CLOCK_ENABLE=1 and ROCE_CC_RTT_TIMESTAMP_FORMAT=2.^[7] mlxconfig -d <device> query is the discovery command, the device being either an MST path such as /dev/mst/mt4129_pciconf0 or a PCI BDF, and a change ends with “Please reboot machine to load new configurations”.^[15] The runtime evidence is the ZTR counter set - ZTR_CC_CNP_HANDLE_COUNTER, ZTR_CC_AI_INC_COUNTER, ZTR_CC_DEC_COUNTER, ZTR_CC_NUM_RTT, ZTR_CC_MIN_RTT, ZTR_CC_RTT_TIMEOUT_COUNTER and the rest - read through the PPCC register with mlxreg.^[5] If ZTR_CC_NUM_RTT is flat while ZTR_CC_RTT_TIMEOUT_COUNTER climbs, the algorithm is loaded and blind.

One corroborating signal worth knowing but labelling as inference: NVIDIA’s NIC Configuration Operator ships a Spectrum-X profile that sets User-Programmable CC: true and DCQCN: false, with CC slot 0 carrying bandwidth 400, responsiveness alpha 6553, max decrease factor 63570, base RTT 15000 and max queuing delay 250000 - the same values the ZTR-RTT page lists as the defaults for ALPHA, MAX_DEC, CONGESTION_DELAY_THRESHOLD and MAX_DELAY.^[11]^[5] That strongly suggests ZTR-RTT is the shipping default on Spectrum-X hosts, but the profile page does not say so in those words, so present it as evidence rather than as a statement. The same profile also sets trust to DSCP with ToS 96 and a PFC bitmap of 00010000; the set bit is at index 3, which agrees with ToS 96 being DSCP 24 and the default DSCP-to-priority table putting 24 on priority 3 - but one earlier reading of that page took it as priority 4, so re-read the page before a lesson or a slide states the priority.^[11]

4Writing the recommendation

A recommendation on this topic has five parts, and the last two are what make it defensible.

The symptom you measured, with counters and directions, not adjectives.^[8]

The option you recommend, with the exact command that enables it - nv set qos roce mode lossy plus nv config apply on Cumulus, or roce enable pfc-priority 3,4 on Dell Enterprise SONiC, or the mlxconfig prerequisites and a firmware check for the ZTR path.^[2]^[12]^[7]

The design artifacts: nv show qos roce captured before and after, so the customer’s change-control ticket has a diff rather than a description.^[2]

The counters that will prove it worked. For the ECN-only path, the loop has to close on two machines: np_ecn_marked_roce_packets non-zero on the receiver, np_cnp_sent tracking it, and rp_cnp_handled non-zero on the sender, with rp_cnp_ignored staying low.^[10] Host-side ECN must actually be enabled per priority through /sys/class/net/<interface>/ecn/<protocol>/enable/X for roce_rp and roce_np, or CNPs arrive and nothing reacts.^[9] For the ZTR path, ZTR_CC_NUM_RTT moving and ZTR_CC_RTT_TIMEOUT_COUNTER flat.^[5] For the PFC path, rx_prio3_pause and tx_prio3_pause on both hosts plus tx_pause_storm_error_events at zero.^[8]

The two things you are not claiming. Pick them deliberately. Good candidates: that a specific ConnectX generation supports ZTR-RTT, because no fetched page states a generation;^[4] and that any of this predicts their training-job throughput, because the switch mode changes queueing, not the application.

Positioning: three customers, one afternoon · decision 1/7A 0 · P 0 · S 0

Brief — Back-to-back calls: a VMware farm (200× R760), an AI training pod (64× XE9780 on Spectrum-X) and a storage-heavy multi-tenant inference cluster. Each has objections.

Dell account team + three end customers: VMware farm: "200 R760s on vSphere 8; NSX eats about 20% of our cores. Should we go DPU?"

Positioning drill: three customers in one afternoon, each wanting a different answer. Score yourself on positioning and safety, then re-read which options over-promised.

Episode 3, closing — What the notebook ends up saying

How it ended

In the room you answer the hardware question by not answering it from memory. On the record you name what the framework requires - the DOCA PCC mlxconfig prerequisites and firmware 32.38.1000 and higher - then ask to read the card with mlxconfig -d <device> query and the firmware from ibv_devinfo.^[7]^[15] The written recommendation offers ECN-only as the switch-side path that costs one command, and names the counters that prove whichever path they pick closed the loop.^[2]^[5] The network lead copies them onto a page she titles “must stay flat”.

What you say: “Let me read your card before I put a generation on a slide.”

Friday’s review passes at 09:40. Ten minutes later, in a briefing room, the account manager asks a question no counter answers.

Building the recommendation

The situation: a Dell customer with PowerSwitch today, evaluating Spectrum, running an inference tier plus a small training pod. They have refused fabric-wide PFC. Their complaint is intermittent stalls on unrelated flows.

Step 1 - name the symptom from evidence. On both hosts, ethtool -S <if> | grep -E "prio[0-9]_pause" shows rx_prio3_pause climbing on hosts that are not the busy ones, and packet_seq_err is flat. That is head-of-line blocking, not loss. The customer’s refusal of PFC and their symptom point the same way, which is worth saying out loud - it does not always happen.

Step 2 - pick the option that matches. ECN-only on the switch: nv set qos roce mode lossy then nv config apply, keeping DSCP 26 to priority 3 and DSCP 48 to priority 6, with TC6 strict so CNPs are never queued behind data.

Step 3 - decide whether ZTR-RTT is in scope. Run mlxconfig -d <dev> q | grep -E 'USER_PROGRAMMABLE_CC|PCC_INT_EN' and read the firmware. If the cards in this pod do not report the programmable-CC knobs, ZTR-RTT is not on the table for this deployment and the recommendation says so in one line rather than hedging.

Step 4 - the artifacts. Capture nv show qos roce before and after, and the per-interface nv show interface <id> qos roce status, as attachments.

Step 5 - the verification the customer runs. Re-run their workload; expect np_ecn_marked_roce_packets and np_cnp_sent non-zero on receivers, rp_cnp_handled non-zero on senders, rx_prio3_pause at zero everywhere because PFC is gone, and packet_seq_err bounded - if it climbs steadily, ECN-only is not enough for this load and lossless comes back on the table for the training pod alone.

Step 6 - the exclusions. Not claiming that ECN-only will hold at a larger scale than the pod measured, and not claiming any ZTR-RTT capability for these cards without the mlxconfig output.

Lab

Pre-flight inventory (read-only - every command below is a query; nothing writes firmware configuration): record sudo mst status -v output, ibv_devinfo -d mlx5_0, ofed_info -s and ibdev2netdev before you start.

sudo mst status -v to find the MST device path for the BlueField-3 or ConnectX card. Expected: a /dev/mst/mt####_pciconf0 path. If MST is not running, sudo mst start - that starts a service, it does not change device configuration.
sudo mlxconfig -d /dev/mst/mt####_pciconf0 q | grep -E 'USER_PROGRAMMABLE_CC|PCC_INT_EN'. Expected: both knobs listed with their current values. If neither name appears in the query output, this card does not expose the programmable-CC path and ZTR-RTT is not available on it - record that as the finding.
Read the firmware version two ways and record both: ibv_devinfo -d mlx5_0 | grep fw_ver and sudo mlxfwmanager --query. Expected: agreement. Compare against the DOCA PCC floor of 32.38.1000 and note whether this card meets it, without claiming that the floor applies to ZTR-RTT.
sudo mlxconfig -d <dev> q | grep -E 'ROCE_CC|CNP' to see which congestion-control knobs the firmware exposes at all. Expected: at least ROCE_CC_PRIO_MASK_P1. Record the values; do not change them.
Host QoS and ECN state, read-only: mlnx_qos -i <interface> for trust state and PFC configuration, and cat /sys/class/net/<interface>/ecn/roce_rp/enable/3 plus the roce_np equivalent. Expected: a trust state you can name and a per-priority enable bit you can quote.
Write the single sentence this lab exists to produce: “This card, at this firmware, does or does not expose the programmable congestion control path.” That sentence is what turns the recommendation from a slide into a commitment. If you later decide to enable the path, the mutating command is sudo mlxconfig -d <dev> s USER_PROGRAMMABLE_CC=1 followed by a power cycle, and the rollback is sudo mlxconfig -d <dev> s USER_PROGRAMMABLE_CC=0 plus another power cycle - schedule it as a change, not as a lab step.

Evidence / notes (commands run, outputs, what surprised you)

Scenario: "we do not want to configure PFC"

Before you read: what do you already know?

After this lesson you can

Episode 3 — The slide that promises no switch configuration

1What the customer is actually refusing

2Three options, and exactly what each one asks for

1. Sender transmits

3ZTR-RTT: what is verified, and what is not

4Writing the recommendation

Episode 3, closing — What the notebook ends up saying

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere