Scenario: 'the fabric is slow and adaptive routing does nothing'
S6·E2Four claims and a cold coffee · Hotel lobby near Hall C, 06:50, ninety minutes before the meeting
Builds on: Adaptive routing: per-packet spraying and its eligibility rules, Telemetry-based congestion control: three loops, not one, The triage toolkit: resources, optics and what tcpdump will not tell you
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Order the evidence you request from a customer so that one round of commands settles the eligibility question.
- Judge each reported symptom against the documented adaptive-routing eligibility rules before proposing any change.
- Separate PFC, ECN and Spectrum-X congestion control when a customer uses one word for all three.
- Write a findings note that distinguishes misconfiguration from an unsupported combination from a defect that needs NVIDIA support.
Episode 2 — Four claims and a cold coffee
The Dell SE arrived at six with a coffee he stopped drinking while it was hot, and the escalation is one paragraph long: the pod is slow, collectives are thirty percent below the other pod, pause frames are climbing, adaptive routing is enabled and does nothing, they benchmarked with iperf3, and they upgraded Cumulus last month for a BGP bug. Row 14 of his promise spreadsheet says “packet spray”. Acceptance is Friday and the second pod rides on it, so he wants a root cause before you walk in. You will not have one, and that is fine; what you can walk in with is an order of operations.
Only one of those claims is a measurement. Pause frames are a counter you can read per switch priority with nv show interface <interface-id> counters qos pfc-stats.[4] “Adaptive routing does nothing” is an inference drawn from a TCP benchmark, and adaptive-routing-eligible traffic is RoCEv2 unicast or VXLAN-encapsulated RoCE, so the test never engaged the feature.[1] The upgrade is a version question: 5.18.0 is qualified for non-Spectrum-X only, and the Spectrum-X qualified build is 5.18.1.[11]
Adaptive routing exists because ordinary ECMP pins a flow to one hashed path and an AI collective is a few enormous synchronised flows that collide on it; the switch instead approximates Join-Shortest-Queue in hardware and marks ECN only once rerouting capacity is exhausted.[8] That ordering is the whole diagnosis.
Eligibility before bugs: a feature that never ran cannot be the thing that failed. Segment 1 takes the paragraph apart claim by claim.
1The ticket, and the four claims hidden inside it
The escalation arrives as one paragraph from a Dell account team: “Customer’s AI pod is slow. Collectives are 30% below the other pod. Pause frames are climbing on the leaves. Adaptive routing is enabled but does nothing. They benchmarked with iperf3 and got the same result with AR on and off. They upgraded Cumulus last month to fix a BGP bug.”
That paragraph contains four separable claims and exactly one of them is a measurement. “Collectives are slow” is an observation. “Pause frames are climbing” is a counter, which is useful if you get the actual per-priority numbers from nv show interface <interface-id> counters qos pfc-stats, the view that reports pause-frame counts and pause duration per switch priority 0 to 7.[4] “Adaptive routing does nothing” is an inference from an iperf3 TCP run, and adaptive-routing-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE, so a TCP stream never engaged the feature at all.[1] “We upgraded Cumulus” is the version question: 5.18.0 is qualified for non-Spectrum-X only and the Spectrum-X qualified build is 5.18.1.[11]
Your first reply is therefore not a fix and not a theory. It is one round of commands chosen so that the eligibility question is settled before anyone touches a threshold. Ask for output, not for opinions, and ask for it from two leaves so you can compare rather than trust.
Brief — Customer's R760 with a BlueField-3 B3220 (DPU mode) lost the DPU after a BIOS update. Host sees nothing. They are about to reflash the DPU firmware "to be safe".
Dell L2 support engineer, bridge call with the customer: We are about to reflash the DPU firmware from the host — go ahead?
2Eligibility before bugs
Adaptive routing is off by default, both globally and per interface, so the first question is whether it was ever enabled on the interfaces that carry the traffic.[2] nv show router adaptive-routing prints enable on; nv show interface swp51 router adaptive-routing prints the per-interface enable plus the link-utilization threshold.[14] Both are disabled by default, and the threshold reads 70 once the global knob is on.[2] Enabled globally with nothing enabled on the uplinks is a common and entirely silent state.
Then run the documented eligibility list against what the customer actually built. Traffic must be RoCEv2 unicast or VXLAN-encapsulated RoCE; interfaces must be layer-3 default-VRF next-hop router interfaces; the ASIC must be Spectrum-4 at 400G or 200G, and there is no 800G link support on Spectrum-4; adaptive routing is not supported on layer-3 subinterfaces, SVIs, bonds or bond members, and it does not integrate with resilient hashing.[1] This customer has bonded their spine uplinks and is testing a path inside a tenant VRF, which disqualifies the path twice over before any bug is considered.
One nuance to hold accurately, because it looks like a contradiction on the same documentation page: adaptive routing is not supported on bonds, and yet when adaptive routing is enabled the LAG hash randomizer activates for packet spraying on layer-3 bonds carrying AR-eligible packets, on Spectrum-4 and later and only on static layer-3 bonds.[1] Present both sentences together or you will be corrected by a customer holding the same page.
The far end matters too. NVIDIA’s answer on the developer forum, in a thread where mlxreg --reg_name ROCE_ACCL --set adaptive_routing_forced_en=0x1 returns success while the field stays zero, is that adaptive routing is supported with the Spectrum-X solution only.[6] The NIC will not arm the feature against a non-Spectrum-X switch no matter what is typed.
Falling back to ECMP hashing. The engine toggle says adaptive, but the scenario is not eligible, so the leaf hashes. Traffic type: TCP is not AR-eligible.
Burst armed: 8 packets, one RDMA QP
Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.
- Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
- Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
- Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
- File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.
nv set router adaptive-routing state enabled nv set interface swp51 router adaptive-routing state enabled nv config apply nv show router adaptive-routing nv show interface swp51 router adaptive-routingNVUE reference: adaptive routing
- blocks AR Traffic type. TCP is not AR-eligible. It takes the ECMP hash path, whatever the AR configuration says. “AR-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE.”
- blocks AR Interface type. AR is off on a bond or bond member. Separate feature, same page: with AR enabled the LAG hash randomizer sprays AR-eligible packets across members of a static layer-3 bond (Spectrum-4 and later, named in Cumulus Linux 5.16.0). “Adaptive routing does not support layer 3 subinterfaces, SVIs, bonds, or bond members.”
- blocks AR VRF. A tenant VRF is outside the documented scope. This is the sharpest tension between multi-tenancy and adaptive routing — check the Spectrum-X deployment guide for the RA in question. “Adaptive routing applies to layer-3, default-VRF next-hop router interfaces.”
- ok ASIC. Spectrum-4 is the supported ASIC from Cumulus Linux 5.6 on. “Cumulus Linux 5.6 supports adaptive routing on Spectrum-4. Cumulus Linux 5.5 and earlier supports adaptive routing on Spectrum-3 and Spectrum-2 as a Beta feature for evaluation.”
- ok Link speed. 400G is one of the two documented speeds. “NVIDIA Spectrum-4 at 400G/200G.”
- ok Far-end NIC. A SuperNIC on a Spectrum-X switch: the NIC firmware arms AR, and it reorders the sprayed packets at the receiver. “Adaptive Routing is supported with Spectrum-X solution only.”
- always Resilient hashing. Always: AR and resilient hashing are not a belt-and-braces pair. If the customer enabled resilient_hash_enable in traffic.conf as a load-sharing fix, that is a separate mechanism, not an AR helper. “Adaptive routing has no integration with resilient hashing.”
FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.
Why the hash collides (ECMP defaults)
- On by default: IP protocol, source IP, destination IP, source port, destination port, IPv6 flow label.
- Off by default: ingress interface, GTP TEID, inner packet fields.
- Polarization fix is a per-switch seed: nv set system forwarding hash-seed 50 (range 0–4,294,967,295).
- Resilient hashing shares 65,536 buckets by default — and adaptive routing does not use resilient hashing.
Numbers you can quote
- Failover 2.68 ms, described as 400× faster than the 1.08 s of software solutions.
- Tail latency P99 8–9 µs versus 22 µs on traditional Ethernet.
- Capacity-proportional degradation: 11% bandwidth loss for 10% link failures.
- DeepSeek-V3 step time 668 ms under isolation versus 1.18 s (1.6× degradation) on traditional Ethernet.
⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.
3Three mechanisms, one customer word
Customers say “congestion control” for three different loops, and the fix for each lives in a different place. Draw them apart before you argue about thresholds.
| Mechanism | Layer and scope | Where you read it |
|---|---|---|
| PFC | Layer 2, hop by hop, lossless on switch priority 3 by default with RoCE enabled | nv show interface <id> counters qos pfc-stats for pause frames and duration[4] |
| ECN | Layer 3, end to end, marks packets so the sender slows down | nv show qos roce for thresholds and mode; egress-queue stats for WRED drops[3][4] |
| Spectrum-X congestion control | End to end, RTT probes plus in-band switch telemetry driving per-destination rate contexts on the SuperNIC | NIC-side state and NetQ; described in NVIDIA’s platform material[8][13] |
The architectural point that makes this scenario legible: NVIDIA describes Spectrum-X as three hardware-accelerated mechanisms working together, with in-switch adaptive routing as a quantized hardware approximation of Join-Shortest-Queue that steers packets to the least-congested port within hundreds of nanoseconds, targeted congestion control in which switches generate ECN marks only when adaptive routing capacity is exhausted, and NIC-based plane load balancing.[8] Read the customer’s symptom against that ordering. Heavy pause with adaptive routing inert is exactly the fabric falling back on its second-line and third-line signals because the first one was never engaged.
The endpoint half is not optional. Per-packet spraying guarantees out-of-order arrival, and the SuperNIC re-orders packets and places them in host memory so the reordering is invisible to the application.[13] That is why a generic RoCE NIC is not sufficient and why “we will use the NICs we already have” is a design question, not a preference.[9]
One term to handle carefully in writing: if the customer uses “adaptive routing notification” or ARN, say plainly that it is not on an official NVIDIA page you can cite and describe the documented behaviour instead. It costs nothing and it is the difference between a trusted FAE and a repeater.
4Off the row, and how to say so
The last claim is the version bump. The Spectrum-X validated stack is one row per release across switch OS, NIC firmware, DOCA-Host, NetQ and the collective stack; the v2.3.1 row of September 2026 pins Cumulus Linux 5.18.1, ConnectX-8 firmware 40.50.1002, DOCA-Host 3.5.0-082 and NetQ 5.1.0.[7] Adaptive routing and congestion control are negotiated end to end between switch and NIC, so a Cumulus-only bump to chase a bug fix leaves the fabric in a combination nobody tested. NVIDIA’s guidance on that page is to deploy clusters with all components on the most up-to-date validated configuration version.[7]
The written answer therefore has three buckets, and the value you add is putting each finding in the right one.
- Misconfiguration, customer fixes today: adaptive routing not enabled on the uplink interfaces; bonded spine uplinks; the benchmark path inside a tenant VRF.[1][2]
- Unsupported combination, needs a plan not a patch: a Spectrum-X pod running 5.18.0 rather than 5.18.1, and any component off the RA row.[11][7]
- Methodology, needs a different test:
iperf3over TCP; and a request to packet-capture the leaf, which cannot work because tcpdump on a Cumulus switch monitors only control-plane traffic to and from the switch CPUs.[1][10]
Only what survives all three buckets is an escalation, and it needs an on-row, eligible configuration that still misbehaves, plus counters, netq check roce output for cross-switch consistency and a support bundle.[12]
Sample runs, exits cleanly, but zero throughput
5Write the two artifacts
Everything above is worth nothing until it is two short documents: the evidence request, and the findings note.
Artifact 1 — evidence request (send within the hour).
Subject: Data needed before we change anything on the AI pod
Please run these on leaf01 and leaf02 and paste the raw output. No changes yet.
nv show system versionnv show router adaptive-routingnv show interface swp49 router adaptive-routingand the same for every spine-facing portnv show qos rocenv show interface swp49 counters qos pfc-statsnv show interface swp49 counters qos egress-queue-statsnetq check roceOn two GPU hosts, please also send
mlxreg -d <device> --get --reg_name ROCE_ACCLand the NIC firmware version.Finally: are the spine uplinks bonded, and is the tested traffic RoCE or TCP?
Each line has a purpose. Version places them on an RA row.[7] Items 2 and 3 answer whether the feature is on where it matters, since both defaults are disabled.[2] Item 4 confirms RoCE mode and classification.[3] Items 5 and 6 quantify pause and WRED drops.[4] Item 7 checks cross-switch consistency of mode, classification, congestion control, flow control and ETS.[12] The host register read is the one that tells you whether the NIC ever armed adaptive routing.[6]
Artifact 2 — findings note (after the output arrives).
Summary. Adaptive routing was not engaged on the tested path. Three documented reasons, one version issue, one test-method issue.
Fix today. (1) Spine uplinks are members of
bond1; adaptive routing is not supported on bonds or bond members. Move the uplinks to routed ports. (2) Per-interface adaptive routing is disabled on swp49 and swp50 even though the global knob is on; both must be enabled. (3) The tested path is in VRFtenant-a; eligible interfaces are default-VRF layer-3 next hops.[1][2]Plan, not a patch. The pod runs Cumulus Linux 5.18.0, which is qualified for non-Spectrum-X only; the Spectrum-X build is 5.18.1 and is what the RA 2.3 row pins. Schedule an alignment of the whole row, not just the switch OS.[11][7]
Re-test method. A RoCE workload, not
iperf3over TCP. Adaptive routing applies to RoCEv2 unicast and RoCE in VXLAN.[1]Not yet an escalation. No finding remains once the above is corrected. If the symptom persists on an eligible, on-row configuration, we will collect
nv action generate system tech-supportfrom both leaves plus host counters and open a case.
Same customer, second pod. They report low collective throughput, zero pause frames, and adaptive routing enabled everywhere. Complete the two artifacts.
Evidence request — fill the purpose column.
| Command | Why you asked |
|---|---|
nv show system version |
____________________ |
nv show interface swp49 router adaptive-routing |
____________________ |
nv show qos roce |
____________________ |
nv show interface swp49 counters qos pfc-stats |
____________________ |
netq check roce |
____________________ |
Findings note — classify each observation as fix-today, unsupported-combination, methodology or escalation.
- Uplinks are 800G native on Spectrum-4: ____________________
- RoCE mode is
lossless-single-ipoolandnetq check rocereturns nothing: ____________________ - NetQ AR dashboard shows all zeros and
asic-monitoris stopped: ____________________ - Everything else matches the RA row and the symptom persists: ____________________
New scenario, work it alone. A different Dell customer runs a two-leaf Spectrum-4 pod at 400G with adaptive routing enabled globally and per interface, RoCE in lossless mode, all components on the RA v2.3.1 row. Collectives are fine for single-tenant jobs but degrade badly when a second tenant runs. Pause frames are low. ECN-marked packets are high on one leaf only.
Produce: (a) a five-line evidence request; (b) a findings note that puts every observation in one of the four buckets; (c) one sentence stating what you will not claim, and why.
Acceptance criteria. The evidence request includes at least one per-interface QoS counter view and one cross-switch validation. The findings note distinguishes tenant isolation behaviour from a defect, and references the documented relationship between adaptive routing and ECN marking rather than treating high ECN as an automatic fault. The final sentence names a claim you cannot source — for example the ARN terminology, or a performance number you did not measure yourself.
Episode 2 — The note you leave behind
By the time the room empties you have three buckets rather than a verdict. The spine uplinks are bonded and the benchmark ran inside a tenant VRF, so the path was disqualified twice before any defect was possible.[1] The pod is on 5.18.0 while the Spectrum-X qualified build is 5.18.1, which is a plan and not a patch.[11][7] And the test itself was TCP. What you actually say: “On the path we examined, adaptive routing was not eligible for three documented reasons — here they are with the page each one comes from.”[1]
The network lead copies all three into his notebook, one per line, then says the thing that changes your week: acceptance passes Friday, they are doubling the pod, and procurement wants a bill of materials on Monday.
Lab
Pre-flight inventory. On both Dell-lab hosts record ethtool -i <ifname>, ibstat, the NIC firmware version, the link speed, and ethtool -S <ifname> saved as a baseline file. Note which ports are ConnectX and which are BlueField-3, and confirm both hosts are on the same subnet over the same switch path. Nothing in this lab changes switch or firmware configuration.
- RDMA incast. Run a many-to-one
ib_write_bwfrom the available client processes to one server process and record aggregate bandwidth and the host counters before and after withethtool -S. Expected: pause and discard counters move on the receiving host under incast. - TCP for contrast. Run
iperf3 -c <server> -P 8over the same links and record the same counters. Expected: a very different counter signature, and no relationship to anything RoCE-specific. - Write the paragraph. In under 150 words, explain to a customer why the TCP result says nothing about adaptive routing. Ground it in the eligibility statement — AR applies to RoCEv2 unicast and VXLAN-encapsulated RoCE — rather than in an opinion about benchmarks.[1]
- Read the NIC’s own view (read-only).
mlxreg -d <device> --get --reg_name ROCE_ACCLand note theadaptive_routing_forced_enfield. Expected: the field reads zero. That is only the default - nothing on this host has tried to set it, and NVIDIA’s own host RoCE script writesadaptive_routing_forced_en=0explicitly, so the read is a baseline, not evidence of a firmware refusal. Write the distinction down: the forum thread’s finding is that a--setreturns success while the field andadaptive_routing_forced_en_field_selectboth stay0x00000000against a non-Spectrum-X switch, and you cannot reproduce that here without changing NIC state.[6] Do not run the--setform here: it changes NIC state, it would not take effect, and you would be teaching yourself a false positive. Record this row in your notes as “documentation-only”, the same verdict you will use in the module 4 coverage audit. - Optional (customer lab, read-only). With a real Spectrum-4 leaf in the path, add
nv show interface <swp> counters qos pfc-statsandnv show interface <swp> counters qos egress-queue-statsbefore and after the same two tests, and put the four counter sets side by side in your note.[4] Rollback: none required; all steps are show commands. Do not clear counters on a customer switch.
Goal. Produce the two artifacts as if for a real account, then check them against the documented rules.
- Run the escalation in
FaeScenarioSimabove and record your first three evidence requests before you look at the model answer. Expected: the model’s first round also starts with version and adaptive-routing state rather than with a threshold change. - Write the evidence request email. It must list exact commands, name which leaves and which hosts, and ask two closed questions (are the uplinks bonded; is the traffic RoCE or TCP). If not: an email that asks for “the config” gets a screenshot and costs you a day.
- In DSX Air, build a two-leaf, one-spine topology and run
nv show router adaptive-routingandnv show interface swp51 router adaptive-routing. Expected: the commands parse and reportenable offby default.[2] If not: check the release on the node withnv show system version. Note the limit honestly: adaptive routing is not supported in a virtual Cumulus Linux switch, so this step verifies the command surface and the defaults, not the behaviour.[1] - Write the findings note with the four buckets. Rank the causes and state explicitly which one is not a bug. Expected: bonded uplinks, per-interface disable and tenant VRF land in fix-today; 5.18.0 lands in unsupported-combination;
iperf3lands in methodology; nothing lands in escalation.[11] - Self-check: every claim in both artifacts must be traceable to one of this lesson’s sources. Delete any sentence you cannot cite, including the tempting ones.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to the customer's platform lead, in five sentences, why their iperf3 result does not show adaptive routing is broken, without making them feel stupid for running it.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 docs, NVUE reference, NetQ 5.1, Spectrum-X validated stack v2.3.1, 2026-09-07. Dates are when each page was fetched.
- Equal Cost Multipath Load Sharing (including Adaptive Routing) | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference: adaptive routing set and unset commands · fetched 2026-09-07
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference: QoS show commands · fetched 2026-09-07
- ECMP and Adaptive Routing monitoring | Cumulus NetQ 5.1 · fetched 2026-09-07
- NVIDIA Developer Forums: BlueField-3 (CX7) RoCE adaptive routing configuration, adaptive_routing_forced_en · fetched 2026-09-07
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- Giga-Scale AI and the Ethernet Evolution (NVIDIA Technical Blog) · fetched 2026-09-07
- Powering Next-Generation AI Networking with NVIDIA SuperNICs (NVIDIA Technical Blog) · fetched 2026-09-07
- Network Troubleshooting | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- Optimize Large-Scale AI Workloads with NVIDIA Spectrum-X (NVIDIA Technical Blog) · fetched 2026-09-07
- NVUE Reference: adaptive routing show commands · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Spectrum-X architectureDOCA course · Same ground: congestion, netq and SuperNIC
- Adaptive routing: per-packet spraying and its eligibility rulesElsewhere in this course · Same ground: eligibility, bonds and Adaptive routing
- Playbook: 'RoCE is slow'Elsewhere in this course · Same ground: netq, hw_counters and fae