Telemetry-based congestion control: three loops, not one

Episode 3 — High pause, low marks, at ten past two

The situation · A bridge call from your kitchen table, 02:10, acceptance run at ten

Sixteen people are on the bridge, the acceptance run is eight hours away, and the ticket says “RoCE congestion control is not working, the fabric is slow”. The institute’s night-shift operator is awake with the counters, reading them off leaves he labelled himself: an all-reduce at sixty percent of expectation, TX pause on switch priority 3 climbing all night, ECN-marked counts almost flat, RoCE in lossless mode, and two 800G uplinks per leaf in the default VRF. Somebody proposes moving the ECN thresholds. Somebody else wants the algorithm rewritten. You ask them both to hold.

That counter combination is already a diagnosis, and it follows from why the platform is built as loops rather than one knob. On a stock DCQCN fabric, marking is the switch’s only tool. Spectrum-X puts a faster tool first: in-switch adaptive routing spreads the packet, switches generate ECN marks only when adaptive routing capacity is completely exhausted, and senders adjust their rates at RTT timescales using precise RTT probes combined with those marks^[1]. So flat marking under climbing pause is not a broken second loop. It is a first loop that never got the chance to act — and on a Spectrum-4 leaf, 800G links are not adaptive-routing eligible at all^[7].

You met that eligibility list before, in a lab, on a register that would not take yes for an answer. A counter that stays flat is evidence: a signal that never fired means a loop that never ran. Before anyone edits a threshold, separate the three loops and decide which box holds the evidence.

1Three mechanisms, not one knob

NVIDIA describes Spectrum-X as three hardware-accelerated mechanisms working together.^[1] First, in-switch adaptive routing: a quantized hardware approximation of Join-Shortest-Queue over egress queue depths, resolved at sub-microsecond intervals.^[1] Second, targeted congestion control: switches generate ECN marks only when adaptive routing capacity is completely exhausted, and senders adjust their sending rates at RTT timescales using precise RTT probes combined with ECN marks.^[1] Third, NIC-based plane load balancing: a dedicated hardware engine in the SuperNIC that distributes traffic across multiple network planes using local queue feedback and end-to-end, per-plane congestion telemetry.^[1]

The architectural claim worth teaching is the ordering. ECN is a second-line signal here. Congestion is first absorbed by rerouting; only when the fabric genuinely has no spare path does the switch start marking.^[1] That is the difference from a stock DCQCN Ethernet fabric where marking is the only tool available.^[1]

The measurement side is equally specific: sender and receiver coordinate to assess congestion paths precisely using RTT probe packets and in-band telemetry from intermediary switches, including timestamps and buffer utilization.^[1] On the endpoint, that integration is described as tightly coupled to the Spectrum-4 switch’s real-time telemetry, letting the NIC react with microsecond precision.^[3]

ASIC

Uplink speed

Interface

VRF

Traffic

Far-end NIC

AR eligible

step 0 / 9per-packet JSQdeepest queue 8

Arrival order at the receiver (modeled, revealed at the last step)

········

Adaptive routing · step 0 of 9

Burst armed: 8 packets, one RDMA QP

Adaptive routing is eligible here. The leaf picks the egress port per packet from queue occupancy and port utilization — a quantized hardware approximation of Join-Shortest-Queue, resolved within hundreds of nanoseconds. swp51 already carries a background flow.

Predict: which port takes the next packet, and why? (optional, saved)

Leaf — the adaptive routing engine

Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.

nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing

NVUE reference: adaptive routing

Eligibility gate — AR runs

ok Traffic type. RoCEv2 unicast is on the eligible list. “AR-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE.”
ok Interface type. A routed swp is a layer-3 next-hop router interface — the only eligible kind. “Adaptive routing does not support layer 3 subinterfaces, SVIs, bonds, or bond members.”
ok VRF. Default VRF — the documented scope. “Adaptive routing applies to layer-3, default-VRF next-hop router interfaces.”
ok ASIC. Spectrum-4 is the supported ASIC from Cumulus Linux 5.6 on. “Cumulus Linux 5.6 supports adaptive routing on Spectrum-4. Cumulus Linux 5.5 and earlier supports adaptive routing on Spectrum-3 and Spectrum-2 as a Beta feature for evaluation.”
ok Link speed. 400G is one of the two documented speeds. “NVIDIA Spectrum-4 at 400G/200G.”
ok Far-end NIC. A SuperNIC on a Spectrum-X switch: the NIC firmware arms AR, and it reorders the sprayed packets at the receiver. “Adaptive Routing is supported with Spectrum-X solution only.”
always Resilient hashing. Always: AR and resilient hashing are not a belt-and-braces pair. If the customer enabled resilient_hash_enable in traffic.conf as a load-sharing fix, that is a separate mechanism, not an AR helper. “Adaptive routing has no integration with resilient hashing.”

FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.

Why the hash collides (ECMP defaults)

Numbers you can quote

⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.

Loop one in isolation: uneven starting queue depths, adaptive engine. Every packet that lands on the shallow port is a packet that never triggered loop two.

2Three lanes on the whiteboard

Customers conflate three different mechanisms and then ask why “RoCE CC” is not working. Separate them before discussing anything else.^[4]^[1]

	PFC	ECN / DCQCN	Spectrum-X CC
Layer and scope	Layer 2, hop by hop between two ports^[5]	Layer 3, end to end^[11]	End to end, sender and receiver plus in-band switch telemetry^[1]
Signal	PFC pause frame on a priority^[5]	CE codepoint in the IP header, then a CNP back to the sender^[4]^[11]	RTT probes plus ECN marks plus switch timestamps and buffer utilization^[1]
Timescale	Immediate, link local^[5]	One round trip^[1]	RTT timescales for rate, sub-microsecond for the routing loop underneath^[1]
Where state lives	Per priority per port on the switch^[5]	Sender and receiver rate state^[1]	Per destination and per plane contexts on the SuperNIC^[1]
Cumulus default	On for switch priority 3 in lossless mode^[4]	ECN threshold 1,501,500 bytes on traffic class 3^[4]	Not a switch command; it is the platform behaviour^[1]
Proof it is active	`nv show interface <id> counters qos pfc-stats`^[6]	`nv show interface <id> counters qos egress-queue-stats` (WRED drops on TC3)^[6]	Host-side `mlxreg --reg_name PPCC` state and CNP counters^[8]

The Cumulus defaults behind that table are the ones from module 3: DSCP 24 to 31 map to switch priority 3 and DSCP 48 to 55 to switch priority 6; SP3 goes to traffic class 3 with a WRR 50 percent scheduler and SP6 to traffic class 6 with strict priority; in lossless mode PFC is enabled on switch priority 3.^[4] Two more constraints matter in a congestion argument: link pause and PFC cannot coexist on the same port, and ECN marking probability has 1 percent hardware granularity.^[5]

default mode

trust

stage 6 / 6

packet DSCPDSCP 26

inspect traffic class

roce-lossless memory-percent50% / 50%

stage 6 · mode lossless · trust l3 · DSCP 26

Congestion response — which of the three actually fires

In lossless mode: PFC sends XOFF upstream, ECN marks CE at the egress queue, trimming is not active. RFC 3168: a router sets the CE codepoint only where it would otherwise have dropped the packet.

✓ PFC XOFF — pause upstream · ingress
PFC RX/TX is enabled on switch priority 3 in this mode: the ingress buffer crosses xoff and the switch sends an XOFF upstream.
✓ ECN CE mark — signal end to end · egress
The egress queue is past the ECN minimum threshold, so the switch flips the IP ECN field to CE (11) instead of dropping — RFC 3168 §5.
✗ Packet trim → 256 B — forward the header · Spectrum-4+
Packet trimming needs `nv set qos roce mode lossy-multi-tc` plus `nv set system forwarding packet-trim state enabled`, on Spectrum-4 or later.

FAE angle Three loops, not one. PFC is layer 2, hop by hop, switch priority 3. ECN/DCQCN is layer 3, end to end, and on Spectrum-X only fires once adaptive routing has run out of paths. Spectrum-X CC proper is RTT probes plus in-band switch telemetry driving per-destination rate contexts on the SuperNIC. Climbing pause counters on a Spectrum-X fabric usually mean adaptive routing was never eligible for that traffic.

lossless — what the one command does

PFC on switch priority 3 plus ECN on TC3. The buffer must never overflow for RoCE: the switch pauses upstream before it would drop.

nv set qos roce state enabled
nv set qos roce mode lossless
nv config apply

Commands for this stage

nv set qos congestion-control default-global traffic-class 3 ecn enabled
nv set qos congestion-control default-global traffic-class 3 min-threshold 150000
nv set qos congestion-control default-global traffic-class 3 max-threshold 1500000
nv set qos congestion-control default-global traffic-class 3 probability 100
nv show interface swp16 counters qos pfc-stats

Default ECN profile: enabled on egress queue 0, min threshold 150,000 bytes, max 1,500,000 bytes, probability 100%, RED disabled. ECN marking probability has 1% hardware granularity. source
ECN threshold on traffic class 3 is 1,501,500 bytes. ⚠ The published numbers disagree across pages (1,501,500 B on the RoCE page; 146.48 KB / 156 KB min and 1.43 MB max in the 5.18 extraction; 150,000 / 1,500,000 B on the QoS page). Teach the shape — min ≈ 150 KB, max ≈ 1.5 MB, probability 100% — and read `nv show qos roce` on the actual switch. source
PFC watchdog defaults: polling interval 100 ms, robustness 3 iterations; per-interface state defaults to disable. It operates only on lossless queues on physical swp interfaces. source
Link pause and PFC cannot coexist on the same port. source
Dynamic (percentage) ECN requires Spectrum-4 or later and reached GA in Cumulus Linux 5.18. ⚠ The NVUE reference lists the mode value as `remote` but its own example uses `relative`; verify on a running 5.18.1 switch. source
On Spectrum-X, switches generate ECN marks only when adaptive routing capacity is exhausted — ECN is a second-line signal, not the first tool. source
PFC RX and TX are enabled in lossless mode and disabled in lossy mode. source
One command sets buffer pools, classification and priority mappings, PFC and ECN thresholds, and scheduling for regular, RoCE and CNP traffic. source

RoCE (CL 5.18) · Quality of Service · Packet Trimming · NVUE QoS set

Walk DSCP 26 to traffic class 3 and switch the congestion toggle on to see which response the current mode actually uses: pause upstream, mark CE, or trim.

3Loop three: planes, and why it is not oblivious spraying

The third loop lives entirely in the SuperNIC. The plane load balancer distributes traffic across multiple network planes using local queue feedback and end-to-end, per-plane congestion telemetry, choosing the plane with the shallowest local egress queue.^[1] Under it sits per-destination state: the SuperNIC maintains independent, stateful congestion-control contexts per destination GPU and per physical plane, each monitoring RTT probes and processing CNPs to compute a per-plane rate allowance.^[1]

The differentiator is the filtering. Any plane experiencing end-to-end congestion or link failure is temporarily filtered out of the eligible set, which is precisely what separates this from oblivious spraying — a scheme that spreads packets evenly cannot detect that one plane has degraded and will keep feeding it.^[1] That is also the mechanism behind the published resilience numbers.^[1]

Performance isolation is the composite result rather than a fourth feature: NVIDIA’s formulation is that combining high-frequency telemetry probes with flow metering ensures workloads are protected and the fabric delivers performance isolation.^[2] Module 4 lesson 5 works the multiplane configuration; here the point is only that the plane decision is a congestion-control decision made in hardware on the adapter.^[1]

4The published numbers, and the words to avoid

Use these exactly, or not at all. Failover is 2.68 ms, described as 400 times faster than software solutions at 1.08 s.^[1] Tail latency is P99 8 to 9 microseconds under 75 percent network load, against traditional Ethernet’s P99 reaching 22 microseconds.^[1] Multiplane resilience is quoted as capacity-proportional degradation: 11 percent bandwidth loss for 10 percent link failures.^[1] Multi-tenant isolation is quoted as a DeepSeek-V3 training step time of 668 ms under near-perfect isolation versus 1.18 s, a 1.6 times degradation, on traditional Ethernet.^[1] Scale is over 128,000 endpoints using shallow two-tier fat-trees.^[1]

Two things must not be said. First, “RoCE Adaptive Routing Notification” or ARN is not confirmed on any official NVIDIA page; describe the mechanism as switch-side least-congested-port selection plus SuperNIC reordering.^[1]^[2] Second, the command nv set qos roce congestion-control algorithm spectrum-x-cc appears only in third-party snippets and is absent from both the Cumulus Linux 5.18 RoCE page and the NVUE QoS reference; it is not part of the documented command surface, so do not teach it and do not run it in a customer’s change window.^[4]^[9]

5Analysing the ticket

The ticket says “RoCE CC is not working, the fabric is slow”. The analysis skill is to convert that into an ordered evidence request that separates the three loops.

From 'CC is not working' to an evidence list

Step 1. Establish the RoCE baseline on the leaf. nv show qos roce — confirm the mode is lossless rather than lossy or lossless-single-ipool, because the mode decides whether PFC exists at all and NetQ’s validation does not cover the single-pool mode.^[4] Then nv show interface <swp> qos roce status to confirm PFC on switch priority 3 and the pool map.^[4]^[6]

Step 2. Ask what the pause counters say. nv show interface <swp> counters qos pfc-stats gives pause frame counts and pause duration per switch priority.^[6] Zero TX pause on a congested lossless fabric means classification is wrong, not that there is no congestion. Sustained pause means loop one never got the chance to act.^[6]^[7]

Step 3. Ask what the marking counters say. nv show interface <swp> counters qos egress-queue-stats breaks out per-traffic-class frames, bytes, discards and WRED drops; WRED drops on TC3 mean the ECN thresholds are being exceeded.^[6]

Step 4. Check eligibility before blaming CC. Pause frames climbing on a Spectrum-X fabric usually means adaptive routing is not eligible on that traffic: check ASIC and speed, interface type and VRF, and the traffic type.^[7] This is the step customers skip.

Step 5. Move to the host. Trust mode and DSCP marking (QPTS trust state, default dscp), the acceleration fields in ROCE_ACCL, and the PPCC algorithm state including the ZTR-RTTCC slot 15 are what NVIDIA’s own script configures; ask for those three reads.^[8]

Step 6. Cross-check with NetQ if present. netq check roce for cross-switch consistency, and the adaptive routing dashboard for ECMP traffic imbalance events — remembering its prerequisites, so an empty dashboard is usually configuration.^[10]

Deliverable: one page, six evidence items, each with the command, the expected shape and what a deviation would mean. No configuration change is proposed until the evidence arrives.

Sunrise, six evidence items

How it ended

Nobody changes a threshold at three in the morning. The call ends with one page: confirm the RoCE mode and the PFC state on the leaf^[4], read pfc-stats for pause per priority and egress-queue-stats for WRED drops on traffic class 3^[6], test the uplinks against the adaptive routing eligibility rules^[7], then move to the host registers NVIDIA’s own script writes^[8]. What you say at handover is short: “Your congestion control is behaving. Your uplinks are 800G, so nothing sprayed those flows, and pause is all that was left.” Then everyone drops except the institute’s research architect, who asks whether his team could simply write their own congestion control. It is 02:52.

Lab

Goal: generate a many-to-one RoCE incast on the Dell lab and capture host-side congestion evidence. All steps read state or generate traffic; nothing is reconfigured.

Pre-flight inventory. On every host: sudo mst status -v for device paths, ibv_devinfo | grep -E 'state|link_layer' to confirm the ports are up and Ethernet, ethtool -i <ifname> for driver and firmware, and ip -br addr for the RoCE-facing IPs. Record the baseline counters before any traffic: ethtool -S <ifname> > ethtool-before.txt.

Start the receiver: ib_write_bw -d <mlx-dev> -x <gid-index> -F --report_gbits on one host. Expected: the server prints its listening line. If it exits immediately, the device or GID index is wrong — list GIDs with show_gids and pick an RoCEv2 IPv4 entry.
Start three or more senders against it, at the same time, each ib_write_bw -d <mlx-dev> -x <gid-index> -F --report_gbits <receiver-ip>. Expected: aggregate receive throughput near line rate with per-sender bandwidth well below it — that is the incast you want.
During the run, on each sender: ethtool -S <ifname> | grep -iE 'cnp|pause|ecn|congestion'. Expected: some combination of CNP sent or received counters and pause counters moving. Record which counter names this firmware actually exposes; they differ by adapter generation.
sudo mlxreg -d /dev/mst/<dev> --get --reg_name ROCE_ACCL and sudo mlxreg -d /dev/mst/<dev> --reg_name PPCC --get --indexes "local_port=1,pnat=0,lp_msb=0,algo_slot=15,algo_param_index=0" — expected: the acceleration fields and the ZTR-RTTCC algorithm slot state as NVIDIA’s doRoCE.sh addresses them.^[8] If a --get form is rejected, record the exact syntax your MFT version accepts rather than guessing.
After the run: ethtool -S <ifname> > ethtool-after.txt and diff ethtool-before.txt ethtool-after.txt. Expected: a short list of counters that moved. That diff is the artefact; keep it.
Write two sentences interpreting the diff in the language of the three loops, and one sentence naming what you cannot conclude without switch-side counters.
Optional, customer lab only: during the same run, on the leaf, nv show interface <if> qos roce counters plus nv show interface <if> counters qos pfc-stats and correlate ECN-marked and pause counts with the host diff.^[6] Read only; do not apply configuration on a customer switch.

Evidence / notes (commands run, outputs, what surprised you)

Telemetry-based congestion control: three loops, not one

Before you read: what do you already know?

After this lesson you can