ECN, CNP and DCQCN: closing the loop

Episode 4 — 02:10, and nothing is slowing down

The situation · Bridge call, Building D, the night before acceptance

Two in the morning, the night before acceptance. The evening batch has been throwing RoCE retransmits for four hours, the night-shift operator has reloaded two switches and labelled the second one so nobody reloads it twice, and someone on the bridge is asking what the ECN thresholds should be set to. You are the third voice on the line and the only one who has not proposed a number.

You ask for three counters instead, and for the machine each one lives on. Marks are counted on the receiver as np_ecn_marked_roce_packets, the notifications beside them as np_cnp_sent, and the sender’s reaction as rp_cnp_handled; the prefixes are the roles, so np_ is read on the receiver and rp_ on the sender.^[5] The receiver’s marks are climbing. Its np_cnp_sent is flat.

Two bits in the IP header are the reason any of this works. RFC 3168 lets a congested router set CE instead of dropping the packet, but only on a packet whose sender already declared itself ECN-capable with an ECT codepoint.^[1] DCQCN then splits the work across three machines: the congestion point marks, the notification point sends CNPs, the reaction point reduces its transmission rate.^[3] The switch never sends a CNP, so no amount of switch tuning rescues a notification point that is switched off.

Tuning a control loop with the feedback wire cut is not tuning. Seven hours to the acceptance test. Find the break first.

1Two bits in the IP header and three roles on three machines

ECN is two bits. RFC 3168 section 5 places the ECN field in bits 6 and 7 of the IPv4 ToS octet, identically in the IPv6 traffic class, with four codepoints: Not-ECT 00, ECT(1) 01, ECT(0) 10 and CE 11.^[1] The division of labour is in the same section: “The ECT codepoints ‘10’ and ‘01’ are set by the data sender to indicate that the end-points of the transport protocol are ECN-capable”, and a router with active queue management, “instead of dropping the packet … MAY instead set the CE codepoint in the IP header”.^[1] RFC 3168 also says that protocols needing only a single ECT codepoint SHOULD use ECT(0).^[1]

NVIDIA’s framing adds the constraint that catches people: ECN “allows reliable communication by notifying all ends of communication when congestion occurs. This is done without dropping packets”, and “this feature requires all nodes in the path (nodes, routers etc) between the communicating nodes to support ECN to ensure reliable communication”.^[11]^[2] Like PFC, this is an every-hop property, not a switch feature.

On top of those two bits, DCQCN defines three roles, and NVIDIA states them explicitly. The Congestion Point “detects congestion and marks packets using the DCQCN bits”. The Notification Point, the receiving end node, “reacts to the DCQCN marked packets by sending congestion notification packets (CNPs)”. The Reaction Point, the transmitting end node, “reduces the transmission rate according to the received CNPs”.^[3] Three roles, three different machines: switch, receiver, sender.

The most common conceptual error follows directly from that split. The switch never sends a CNP. It marks a bit and moves on; the CNP is generated by the receiving NIC and travels back as its own RoCEv2 packet, which is why the fabric must give the CNP class its own treatment — Cumulus defaults CNP to DSCP 48, switch priority 6, traffic class 6, scheduled strict priority.^[6] An engineer looking for a CNP-generation knob on the switch is looking for something that does not exist.

end-to-end rate control

Offered load90% of 400GECN threshold (TC3)1,501,500 B

stage 1 / 5

no CE markingno pauseno losssender did not reactgoodput ≈ 90% ⚠

⚠ Model only: a 4 MB TC3 pool at 400 Gb/s (50,000 B/µs). The shape is right; the numbers are not a simulation of your fabric.

ECN only (lossy) · stage 1 of 5

1. Sender transmits

The RP offers traffic on priority 3 with an ECT codepoint set. If the packets left as Not-ECT (00) the switch is not allowed to mark them at all — it can only drop.

Predict: what does the next stage do? (optional, saved)

Reaction Point (RP) — sender NIC

The sending NIC. On the host this is the r_roce_ecn_rp algorithm, enabled per priority: echo 1 > /sys/class/net/<if>/ecn/roce_rp/enable/3.

It is the only element that can lower the offered load. Marking and CNPs are worth nothing if the RP never acts: the rate stays flat and the queue keeps growing.

Tooling: Counter: rp_cnp_handled (sender). Read with rdma statistic show link mlx5_0/1, or straight from hw_counters.

FAE angle: when np_cnp_sent climbs on the receiver and rp_cnp_handled stays flat on the sender, stop tuning thresholds. The CNPs are being lost or misclassified on the way back.

Counters — which one is still moving?

np_ecn_marked_roce_packetsflatreceiver · hw_counters — CE seen in the RoCEv2 IP header. Flat while the queue is deep = the switch is not marking your queue.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_ecn_marked_roce_packets
```
np_cnp_sentflatreceiver · hw_counters — CNPs the notification point put back on the wire, DSCP 48 / priority 6.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_cnp_sent
```
rp_cnp_handledflatsender · hw_counters — CNPs the reaction point acted on. Moves on a different host from np_cnp_sent — read both.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_handled
```
rp_cnp_ignoredflatsender · hw_counters — CNPs that arrived and were dropped on the floor — r_roce_ecn_rp is not enabled for this priority.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_ignored
```
rx_prio3_pauseflatsender · ethtool -S — The switch is pausing you on priority 3. Direction identifies the victim.
```
ethtool -S <if> | grep -E "prio3_pause"
```
tx_prio3_pauseflatreceiver · ethtool -S — You pausing the network. Flat here: the receiver NIC is not the bottleneck in this scenario.
```
ethtool -S <if> | grep -E "prio3_pause"
```
packet_seq_errflatsender · hw_counters — The fabric dropped or reordered RoCE. On a lossless design this number should stay at zero.
```
cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err
```
rx_prio3_buf_discardflatreceiver · ethtool -S — Packets discarded for lack of per-host receive buffers on priority 3.
```
ethtool -S <if> | grep -E "prio3_(buf_)?discard"
```

ECN only (lossy)

The switch marks CE instead of dropping; the receiver returns CNPs; the sender cuts its rate. No PFC anywhere, so a marking mistake becomes loss instead of a pause.

nv set qos roce mode lossy   # switch: drops PFC and the lossless pool, keeps ECN
echo 1 > /sys/class/net/<if>/ecn/roce_rp/enable/3
echo 1 > /sys/class/net/<if>/ecn/roce_np/enable/3

Queue is shallow; no mechanism has anything to do yet. Raise the offered load.

Cumulus RoCE (NVUE) · ECN sysfs · ZTR-RTT parameters · mlx5 counters · rdma statistic

Run the loop clean first, then inject each fault in turn - no-marking, cnp-misclassified, rp-disabled - and commit a prediction about which counter stops moving before revealing.

2The host side, and the table NVIDIA does not publish

On Linux the host half of DCQCN is two sysfs algorithms. NVIDIA exposes r_roce_ecn_rp for the reaction point and r_roce_ecn_np for the notification point, and states that “Parameters and statistics are configurable per device, port, and priority level (0-7).”^[2] Enabling is per priority:^[2]

echo 1 > /sys/class/net/<interface>/ecn/roce_rp/enable/3
echo 1 > /sys/class/net/<interface>/ecn/roce_np/enable/3
cat /sys/class/net/<interface>/ecn/roce_rp/enable/3

Parameters live beside the enable tree: read with cat /sys/class/net/<interface>/ecn/<protocol>/params/<requested attribute> and write with echo <value> > /sys/class/net/<interface>/ecn/<protocol>/params/<requested attribute>.^[2]

Now the honest part. NVIDIA’s current Linux ECN pages document the path shape and the two algorithm names but publish no parameter table and no defaults — the page was re-fetched for this lesson on 2026-09-07 and still contains no attribute list, and the parameter section trails off pointing at “Next Slide for each protocol”, an artifact of a slide deck converted into a manual.^[2]^[11] Every archived version behaves the same way, including the generated PDF.^[11] So there is no authoritative NVIDIA Linux source for min_time_between_cnps, clamp_tgt_rate or dce_tcp_g, and any lesson that quotes their defaults is quoting a forum post. The list is obtained from the host:

ls /sys/class/net/<interface>/ecn/roce_rp/params/
for f in /sys/class/net/<interface>/ecn/roce_rp/params/*; do echo "$f = $(cat $f)"; done

The nearest NVIDIA-hosted enumeration is the Windows WinOF-2 table, which lists the parameter surface with ranges and units: DcqcnEnablePrio0 through DcqcnEnablePrio7, DcqcnCnpDscp (6 bits, 0-63), DcqcnCnp802pPrio (3 bits, 0-7), DcqcnRpgTimeReset (0-131071 microseconds), DcqcnRpgByteReset, DcqcnRpgThreshold (1-31), DcqcnRpgAiRate and DcqcnRpgHaiRate in Mbit/sec, DcqcnAlphaToRateShift (0-11), DcqcnRpgMinDecFac (0-100), DcqcnRateToSetOnFirstCnp, DcqcnDceTcpG (0-1023 as a fixed-point fraction of 1024), DcqcnDceTcpRtt, DcqcnRateReduceMonitorPeriod and DcqcnInitialAlphaValue.^[4] NVIDIA states on that page that “For signaling of congestion, RCM relies on the mechanism defined in RFC3168, also known as DCQCN.”^[4] The table gives ranges and units; it does not give defaults, so even this source cannot answer “what is the default DcqcnRpgThreshold”.^[4]

3Proving the loop is closed: three counters on two machines

The four DCQCN counters come from the in-tree mlx5_ib driver and surface under /sys/class/infiniband/<dev>/ports/<p>/hw_counters/: rp_cnp_ignored, rp_cnp_handled, np_ecn_marked_roce_packets and np_cnp_sent.^[5] The prefixes are the roles, so they tell you which machine to read them on. np_* counters live on the receiver, rp_* counters on the sender. The commonly quoted prose definitions — np_cnp_sent as the CNPs sent after observing CE in the RoCEv2 IP header, rp_cnp_handled as the CNPs the reaction point acted on to throttle — come from NVIDIA support material that could not be fetched for this course, so treat the wording as unverified even though the names are confirmed in kernel source.^[5]

The diagnostic sequence follows the loop:

What you see	Where you read it	What it isolates
`np_ecn_marked_roce_packets` flat under congestion	receiver `hw_counters`	The switch is not marking your queue: wrong TC, wrong DSCP, or ECN off on that class^[6]
Marks climbing, `np_cnp_sent` flat	receiver `hw_counters`	The notification point is disabled on that priority - check `ecn/roce_np/enable/3`^[2]
`np_cnp_sent` climbing, `rp_cnp_handled` flat	receiver then sender	CNPs lost or misclassified on the return path - the CNP class is not getting its strict queue^[6]
`rp_cnp_handled` climbing, rate unchanged	sender	Reaction point enabled but not effective - parameters, or `ROCE_CC_PRIO_MASK_P1` excluding the priority^[9]

Three other surfaces corroborate. The netdev side exposes rx_prio[p]_marked, “The number of packets ecn marked by device due to per host congestion”, plus per-ring rx[i]_ecn_mark, “The number of received packets where the ECN mark was turned on”.^[10] A validated design uses the blunt version of the same check, ethtool -S <interface> | grep ecn.^[9] And iproute2 exposes optional per-link counters that must be switched on first: rdma statistic set link mlx5_2/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts, with rdma statistic mode supported link <DEV/PORT> listing what the link offers.^[12]

One caution about zeroes. On ConnectX-7 NVIDIA notes that some legacy congestion parameters are unused and that “Congestion counters related to these now unused parameters, will be also be unused and shown in perfmon always with zero values” — so a zeroed counter is not automatically a broken counter, and the counter you rely on should be one you have seen move on that hardware at least once.^[3]

4The layers below and around: firmware and tunnels

Below the driver there is a firmware layer, and it can quietly veto everything above it. The discovery command is mlxconfig -d <device> query, in practice mlxconfig -d <device> q | grep ROCE_CC.^[8]^[9] ROCE_CC_PRIO_MASK_P1 is an 8-bit bitmap of the priorities on which RoCE congestion control is active, set with mlxconfig -d <device> s ROCE_CC_PRIO_MASK_P1=<mask>; a Juniper validated design configuring NVIDIA hosts uses 255, all priorities.^[9] Device forms are the MST path /dev/mst/mt####_pciconf0 or a PCI BDF, and mlxconfig stages a next-boot value: “Please reboot machine to load new configurations”, or use mlxfwreset in band.^[8]

Two related firmware knobs, CNP_DSCP_P1 and CNP_802P_PRIO_P1, are widely cited as the per-port CNP marking controls. Neither name could be confirmed on an NVIDIA-hosted page for this course, so verify them with mlxconfig -d <dev> q on the actual card before writing them into a runbook.^[8]

Around the loop there is encapsulation. The moment RoCE rides VXLAN, the ECN bits have to survive two header operations, and RFC 6040 governs both. Normal mode requires the encapsulator to “construct the outer encapsulating IP header by copying the two-bit ECN field of the incoming IP header”; compatibility mode “clears the ECN field in the outer header to the Not-ECT codepoint”, which makes the outer header unmarkable and silently removes the underlay’s ability to signal congestion.^[7] On the way out, “the decapsulator MUST set the outgoing ECN field to the more severe marking of the outer and inner ECN fields, where the ranking of severity from highest to lowest is CE, ECT(1), ECT(0), Not-ECT”, and “If the inner ECN field is Not-ECT and the outer ECN field is CE, the decapsulator MUST drop the packet.”^[7] That last rule is a real source of mysterious loss in overlay fabrics: an unmarked inner flow that meets a congested marked underlay is dropped by specification.

Find the broken link in a live ECN loop

Case: a two-host Dell-lab pair on a fabric that is congested during a benchmark. The customer reports that ECN is enabled everywhere and rate never drops.

Confirm the traffic is on the priority you think it is. On both hosts:^[2]
```
mlnx_qos -i <if> | grep -i trust
ethtool -S <if> | grep -E "prio3_(bytes|packets|marked)"
```
Expected: Priority trust state: dscp and non-zero rx_prio3_bytes. Zero priority-3 volume means the marking chain, not ECN, is the problem - stop here and go back to lesson 1.
Read the receiver’s two counters during load:^[5]
```
watch -n1 'grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_* '
```
Expected under real congestion: np_ecn_marked_roce_packets climbing and np_cnp_sent climbing with it.
Branch on what you see. If marks are flat, the switch is not marking that class: check the switch’s ECN configuration on TC3 and confirm the DSCP the frames actually carry.^[6] If marks climb but np_cnp_sent is flat, the notification point is off on this priority:^[2]
```
cat /sys/class/net/<if>/ecn/roce_np/enable/3
```
Read the sender’s counter during the same load:^[5]
```
grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_*
```
Expected: rp_cnp_handled climbing. If it is flat while the receiver’s np_cnp_sent climbs, the CNPs are not arriving or not being recognized - check that DSCP 48 gets priority 6 and a strict queue on the return path.^[6]
If CNPs are handled and the rate still does not move, drop a layer:^[9]^[8]
```
sudo mst status -v
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep -E "ROCE_CC|CNP"
cat /sys/class/net/<if>/ecn/roce_rp/enable/3
```
Expected: the RoCE CC priority mask includes priority 3, and the reaction point is enabled on 3.
Record what the documentation does not give you, for this driver build:^[2]
```
for f in /sys/class/net/<if>/ecn/roce_rp/params/*; do echo "$(basename $f) = $(cat $f)"; done | tee /tmp/dcqcn-rp-params.txt
```
This file is the authoritative parameter list for this host, because NVIDIA publishes none.
Only now discuss thresholds, and only with the counter evidence attached.

Episode 4 — Before the batch reruns

How it ended

The notification point was never enabled on priority 3 on the receiving hosts; one write per host under ecn/roce_np/enable/3 closes the loop, and rp_cnp_handled starts moving on the senders.^[2]^[5] You confirm the CNP class still has its own strict-priority queue, since Cumulus defaults CNP to DSCP 48 on switch priority 6, TC6, strict.^[6] At 09:00 the pod passes acceptance and row 41 finally means something. Then the network lead closes the notebook and starts the next argument: phase two is another sixty-four nodes, and her team will not deploy PFC on it at all.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: on the Dell-lab pair, capture the parameter list the documentation is missing and prove the loop end to end. Step 5 is mutating and names its rollback.

Pre-flight, read-only, on both hosts:
```
ibdev2netdev
ls /sys/class/net/<if>/ecn/
cat /sys/class/net/<if>/ecn/roce_rp/enable/3
cat /sys/class/net/<if>/ecn/roce_np/enable/3
```
Expected: roce_rp and roce_np directories, each with enable/0..7 and params/.^[2] If the ecn directory is missing, the driver build does not expose it and nothing below applies.
Capture the missing table. This is the deliverable the vendor documentation does not provide:^[2]
```
for p in roce_rp roce_np; do
  for f in /sys/class/net/<if>/ecn/$p/params/*; do echo "$p/$(basename $f) = $(cat $f)"; done
done | tee /tmp/dcqcn-params-$(hostname).txt
```
Expected: a list of attribute names with current values. Record the driver version alongside it, because the set is a property of the build.
Firmware layer, read-only:^[8]^[9]
```
sudo mst status -v
sudo mlxconfig -d /dev/mst/<mtdev>_pciconf0 q | grep -E "ROCE_CC|CNP|PCC"
```
Expected: ROCE_CC_PRIO_MASK_P1 with a mask value. Record whether CNP_DSCP_P1 and CNP_802P_PRIO_P1 actually appear on this firmware - the course could not confirm those names on an NVIDIA page.

Baseline the counters on both hosts before traffic:^[5]

grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/{np_ecn_marked_roce_packets,np_cnp_sent,rp_cnp_handled,rp_cnp_ignored} | tee /tmp/cc-before-$(hostname).txt

Mutating - break the loop deliberately and restore it. Rollback is written before the change:^[2]
```
cat /sys/class/net/<if>/ecn/roce_rp/enable/3            # record: this is your rollback value
echo 0 | sudo tee /sys/class/net/<if>/ecn/roce_rp/enable/3
# run congesting traffic with identical mode flags on both sides:
#   server ib_write_bw -d mlx5_0 -b -q 8 --tclass=104 ; client ib_write_bw -d mlx5_0 -b -q 8 --tclass=104 <server-ip>
# re-read counters on both hosts and note that rp_cnp_handled no longer tracks np_cnp_sent
echo 1 | sudo tee /sys/class/net/<if>/ecn/roce_rp/enable/3     # rollback
```
Expected: with the reaction point off, the receiver still marks and still sends CNPs while the sender stops acting on them - the exact signature of a broken return-path or disabled-RP fault. Restore the recorded value and confirm with cat. If the client aborts with a parameter-mismatch error instead, the two command lines differ: perftest states that the options specific to modes must be the same for both server and client, and -b and -q are both mode options.^[13]
Close out: re-read the four counters into /tmp/cc-after-$(hostname).txt, diff against the before file, and confirm ecn/roce_rp/enable/3 and ecn/roce_np/enable/3 both match their step-1 values. Attach /tmp/dcqcn-params-*.txt to the module notes - it resolves a documented open question for this course.^[2]

Evidence / notes (commands run, outputs, what surprised you)

ECN, CNP and DCQCN: closing the loop

Before you read: what do you already know?

After this lesson you can

Episode 4 — 02:10, and nothing is slowing down

1Two bits in the IP header and three roles on three machines

1. Sender transmits

2The host side, and the table NVIDIA does not publish

3Proving the loop is closed: three counters on two machines

4The layers below and around: firmware and tunnels

Episode 4 — Before the batch reruns

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere