Skip to content

Adaptive routing: per-packet spraying and its eligibility rules

S4·E1The register that says yes and means no · Dell lab, Round Rock, the afternoon before the readiness demo

S4·E1Analyze~30 minsources checked todaylab mutates hardwareverified against Cumulus Linux 5.18 ECMP/adaptive-routing page and NVUE reference re-fetched 2026-09-07; Spectrum-X validated stack v2.3.1

Builds on: `nv set qos roce`: the one command and its five modes

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain how per-packet adaptive routing chooses an egress port and how that differs from 5-tuple ECMP hashing.
  • Apply the documented eligibility rules to decide whether adaptive routing runs on a given interface, VRF and traffic type.
  • Configure adaptive routing with NVUE and read back the applied state, including the link utilization threshold.
  • Diagnose an 'adaptive routing does nothing' report by eliminating eligibility causes in a fixed order.

Episode 1 — The register that says yes and means no

The situation · Dell lab, Round Rock, the afternoon before the readiness demo

The demo is at nine tomorrow and slide four already promises the institute that packets get sprayed. The Dell SE wrote it; his spreadsheet of commitments to this account is at forty-one rows; his coffee went cold at eleven. Nine days later a 64-node all-reduce runs for their steering committee, and the pod-two order rides on that number. Tonight you have a BlueField-3 host, the only spare switch in the lab, and the host side of the enablement. You set the field: mlxreg --set adaptive_routing_forced_en=0x1. It reports success. You read it back: 0x00000000. You run it again, slower. Same value. Nothing ever says no.

Sit with why the feature exists before you chase the register. Classic ECMP hashes the 5-tuple and pins a RoCE queue pair to one uplink for its lifetime, so two large flows that land on the same spine port collide and stay collided. Adaptive routing improves utilization by choosing the egress port from switch state such as queue occupancy and port utilization, on a per-packet basis[1], which NVIDIA describes as a quantized hardware approximation of Join-Shortest-Queue resolved at sub-microsecond intervals[6]. That only works if both ends agree, and the answer in the forum thread you eventually find is one sentence: adaptive routing is supported with the Spectrum-X solution only[8].

Your lab switch is not part of it. A write that reports success and reads back zero is the platform saying no in the only way it has. Before you re-cable anything, learn the rules that decide whether the feature runs at all.

1What the switch is actually doing

Adaptive routing improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization, and it makes that choice on a per-packet basis.[1] That is the whole difference from ECMP. A 5-tuple hash pins a RoCE queue pair to one uplink for its lifetime, so two large flows that hash to the same spine port collide and stay collided; adaptive routing looks at the egress queues at the moment the packet arrives and puts it on the least-congested member port.[1][7]

NVIDIA describes the same mechanism from the architecture side: in-switch adaptive routing is a quantized hardware approximation of the Join-Shortest-Queue algorithm, sampling egress queue depths and steering packets to the least-congested port at sub-microsecond intervals.[6] The design goal is stated just as plainly: load balance flows packet by packet across the network, without the need for deep buffers.[7]

The documented benefits are worth quoting to a customer verbatim, because they are modest and specific: distribute eligible traffic across all ECMP member ports, remove ECMP flow collisions, distribute traffic equally or weighted to minimize latency, and overcome individual link bandwidth limits for cumulative flow rates.[1] Nothing there promises lower latency for a single small message, and nothing there applies to traffic that is not eligible.

AR eligible
GPU hostBlueField-3 SuperNICRoCEv2 unicastleaf01Spectrum-4hashdefault VRFswp51400Gqueue 3swp52400Gqueue 0swp53400Gqueue 1swp54400Gqueue 0spine01spine02spine03spine04receiverreorders0/8
step 0 / 95-tuple hashdeepest queue 3
Arrival order at the receiver (modeled, revealed at the last step)
········
ECMP hash · step 0 of 9

Burst armed: 8 packets, one RDMA QP

Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.

Leaf — the adaptive routing engine
  • Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
  • Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
  • Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
  • File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.
nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing
NVUE reference: adaptive routing
Eligibility gate — AR runs

FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.

Why the hash collides (ECMP defaults)
Numbers you can quote

⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.

Start on the eligible scenario. Step the burst with the engine on ecmp-hash, then flip to adaptive and step it again on the same queue depths.

2The eligibility list is the lesson

Everything a field engineer needs is in one short list on the Cumulus Linux 5.18 page. Eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE traffic.[1] Eligible interfaces are layer-3 interfaces and next-hop router interfaces in the default VRF.[1] Eligible hardware is switches with the Spectrum-4 ASIC at 400G and 200G speeds, on the Spectrum-X platform.[1]

The limitations are equally exact: adaptive routing does not make use of resilient hashing; Cumulus Linux does not support adaptive routing on layer 3 subinterfaces, SVIs, bonds or bond members; and the Spectrum-4 switch does not support adaptive routing on 800G links.[1] Read the last one twice. The flagship SN5600 port is 800G native, so a customer who cabled 800G end to end and enabled adaptive routing has a fabric where the feature is configured and does nothing. Breaking those ports out to 400G or 200G is what makes them eligible.[1]

Configuration Adaptive routing runs? Rule
Spectrum-4 swp51 at 400G, default VRF, RoCEv2 Yes Eligible traffic, interface, ASIC and speed[1]
Spectrum-4 swp1 native 800G No Spectrum-4 does not support adaptive routing on 800G links[1]
swp51 at 400G inside a tenant VRF No Next-hop router interfaces in the default VRF only[1]
vlan100 SVI, or swp51.100 subinterface No Not supported on SVIs or layer 3 subinterfaces[1]
bond1 member port No Not supported on bonds or bond members[1]
TCP or UDP that is not RoCE No Only RoCEv2 unicast and VXLAN-encapsulated RoCE are eligible[1]
AR ineligible — Link speed: 800G is the classic silent disqualifier: a native-800G uplink on a Spectrum-4 leaf gets no adaptive routing at all.

Falling back to ECMP hashing. The engine toggle says adaptive, but the scenario is not eligible, so the leaf hashes. Link speed: 800G is the classic silent disqualifier: a native-800G uplink on a Spectrum-4 leaf gets no adaptive routing at all.

GPU hostBlueField-3 SuperNICRoCEv2 unicastleaf01Spectrum-4hashdefault VRFswp51800Gqueue 3swp52800Gqueue 0swp53800Gqueue 1swp54800Gqueue 0spine01spine02spine03spine04receiverreorders0/8
step 0 / 95-tuple hashdeepest queue 3
Arrival order at the receiver (modeled, revealed at the last step)
········
ECMP hash · step 0 of 9

Burst armed: 8 packets, one RDMA QP

Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.

Leaf — the adaptive routing engine
  • Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
  • Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
  • Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
  • File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.
nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing
NVUE reference: adaptive routing
Eligibility gate — AR does not run

FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.

Why the hash collides (ECMP defaults)
Numbers you can quote

⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.

An ineligible scenario: 800G on Spectrum-4. Watch the engine fall back to hashing and read the rule the gating panel names, then change one field at a time until it becomes eligible.

3Configure it, then read it back

Two commands turn it on: one global, one per interface. Both default to disabled.[2]

nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv set interface swp52 router adaptive-routing state enabled
nv config apply

Link utilization is a second, separate knob, and it is gated on a profile you cannot set yourself. The 5.18 page is explicit: link utilization is off by default, you cannot enable or disable it per interface, and you can enable link utilization only when you enable the adaptive routing custom-profile.[1] With that profile in place, nv set router adaptive-routing link-utilization-threshold enabled turns the global setting on, the default link utilization threshold percentage on an interface is 70, and you can then change it per port to a value between 1 and 100.[1][2] The profile itself is not a self-service NVUE knob: when you enable adaptive routing, Cumulus Linux uses the default profile settings for your switch ASIC type, and the page tells you to contact NVIDIA Customer Support if you need adjustments — so on a stock switch expect these two commands to be refused or to do nothing, and say that before a customer runs them in a change window.[1]

nv set router adaptive-routing link-utilization-threshold enabled
nv set interface swp51 router adaptive-routing link-utilization-threshold 100
nv config apply

The Linux file form exists for the same settings and you will meet it inside cl-support bundles even on NVUE-managed switches.[1] The file is /etc/cumulus/switchd.d/adaptive_routing.conf:

adaptive_routing.enable = TRUE
adaptive_routing.link_utilization_threshold_disabled = TRUE
interface.swp51.adaptive_routing.enable = TRUE
interface.swp51.adaptive_routing.link_util_thresh = 100

followed by sudo systemctl reload switchd.service.[1] Note the inversion: the file key is a negative, link_utilization_threshold_disabled, while NVUE exposes the positive form. Teach the NVUE command and only recognise the file key when reading a support bundle; do not hand-edit files on an NVUE-managed switch.[1]

Verification is two show commands:[3]

cumulus@switch:~$ nv show router adaptive-routing
        applied
------  -------
enable  on

cumulus@switch:~$ nv show interface swp1 router adaptive-routing
                            applied
--------------------------  -------
enable                      on
link-utilization-threshold  70

4Two facts that look like contradictions

First: adaptive routing is unsupported on bonds and bond members, yet enabling adaptive routing also enables the LAG hash randomizer for packet spraying on layer-3 bonds that support adaptive-routing-eligible packets, supported on Spectrum-4 and later and only on static layer-3 bonds.[1] Both sentences are on the same page. The resolution is that the adaptive routing engine does not choose a bond member, but a separate randomizer does spread eligible packets over the members once the feature is on. The LAG hash randomizer for adaptive routing was named as a feature in Cumulus Linux 5.16.0.[5]

Second: older material and customer runbooks talk about adaptive routing profile-1 and profile-2. Cumulus Linux 5.14 states that adaptive routing now uses a unified default profile, replacing the previous profile-1 and profile-2 options.[4] Anything that still names a profile is stale from 5.14 onward, which matters because 5.14 is the release that shipped in Spectrum-X reference architecture 2.0 and every later RA row is newer still.[10]

One routing caveat travels with adaptive routing. Cumulus Linux uses IPv6 link-local addresses as BGP next hops when a route arrives with both link-local and global next hops; where there are multiple BGP peerings to the same router with adaptive routing enabled, or multiple peerings to the same router on interfaces that share a MAC address or physical interface, configure the ipv6-nexthop-prefer-global option in an inbound route map applied to the peer.[1] The syntax lives in the Set IPv6 Prefer Global section: nv set router policy route-map IPV6-PREFER-GLOBAL rule 10 action permit and nv set router policy route-map IPV6-PREFER-GLOBAL rule 10 set ipv6-nexthop-prefer-global enabled, or in vtysh route-map IPV6-PREFER-GLOBAL permit 10 with set ipv6 next-hop prefer-global.[13]

Finally, the end-to-end half. Adaptive routing is not something the switch does alone: on the host, the ROCE_ACCL register carries an adaptive_routing_forced_en field, and a field report shows the write returning success while the field stays 0x00000000 against a non-Spectrum-X switch. NVIDIA’s answer in that thread is that adaptive routing is supported with the Spectrum-X solution only.[8] That is the cheapest available proof that this is a negotiated, firmware-enforced feature rather than a switch checkbox.

5Triage: from report to rule

The analysis skill this lesson trains is turning a vague report into a named rule. Work the checks in cost order, because the first three need only the switch.

Five interfaces, one checklist

A customer sends nv show interface output for a Spectrum-4 leaf and asks why “adaptive routing is inconsistent”. Verdicts, each with the rule:

  1. swp1, native 800G to spine01. Ineligible. The Spectrum-4 switch does not support adaptive routing on 800G links.[1] Remedy: break the port out to 2x400G or 4x200G.
  2. swp51 at 400G, next hop in VRF tenant-a. Ineligible. Eligible interfaces are next-hop router interfaces in the default VRF.[1]
  3. vlan200 SVI carrying storage traffic. Ineligible. Not supported on SVIs.[1]
  4. swp9, member of static bond bond1. Ineligible for adaptive routing itself, but the LAG hash randomizer sprays eligible packets across the members once adaptive routing is enabled, on Spectrum-4 and later, static layer-3 bonds only.[1]
  5. swp53 at 200G, default VRF, RoCEv2 between SuperNICs. Eligible. Confirm with nv show interface swp53 router adaptive-routing showing enable on, and expect a link-utilization-threshold of 70 only if the switch runs the adaptive routing custom-profile with the global link utilization setting on.[1][2][3]

Report back in one sentence per port, each ending in the documented rule, and one recommendation: break out the 800G uplinks or accept hashing on them.

Nine o'clock, five facts

How it ended

You open the demo with the checklist instead of the claim. ASIC and speed first: Spectrum-4 at 400G and 200G, and explicitly not on 800G links[1]. Then interface and VRF, layer-3 next-hop ports in the default VRF and never a bond, SVI or subinterface[1]. Then the traffic actually offered, RoCEv2 unicast or VXLAN-encapsulated RoCE, which is why their planned iperf3 run would have proved nothing[1]. Then nv show router adaptive-routing for what is armed[3]. What you say to the customer: “Enabled is not the same as eligible, and my lab proved it last night in a single register.” The network lead writes it down, then asks the harder one: if every packet picks its own path, who reassembles the message? The SE already has the pod-two bill of materials open.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: prove on your own Dell-lab hardware that adaptive routing is end-to-end and firmware-enforced, not a switch-side checkbox.

Pre-flight inventory (read-only). On the BlueField-3 host: sudo mst start then sudo mst status -v to get the device path; flint -d /dev/mst/mt41692_pciconf0 q for part number and firmware; ethtool -i <ifname> for the driver and firmware string. Record all three; you will compare them with the Spectrum-X validated stack row later.[10]

  1. sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -e q | grep -i ADAPTIVE — expected: the query completes and prints any adaptive-routing-related configuration fields with default, current and next-boot values. If nothing prints, no such field is exposed on this firmware; record that and continue.[8]
  2. sudo mlxreg -d /dev/mst/mt41692_pciconf0 --get --reg_name ROCE_ACCL — expected: a register dump including adaptive_routing_forced_en. Save it: sudo mlxreg -d /dev/mst/mt41692_pciconf0 --get --reg_name ROCE_ACCL > roce-accl-before.txt. This file is your rollback reference.[8]
  3. Mutating step. sudo mlxreg -d /dev/mst/mt41692_pciconf0 --reg_name ROCE_ACCL --set "adaptive_routing_forced_en=0x1". Expected on a lab with no Spectrum-X switch: the command reports success. Rollback: re-run the same command with adaptive_routing_forced_en=0x0 and diff against roce-accl-before.txt; nothing else on the host is touched by this register write.[8]
  4. Re-read with the --get form. Expected: adaptive_routing_forced_en still reads 0x00000000, and adaptive_routing_forced_en_field_select also stays 0x00000000. That is the documented observation from the field report, and NVIDIA’s answer in the thread is that adaptive routing is supported with the Spectrum-X solution only.[8]
  5. Confirm the rollback: sudo mlxreg -d /dev/mst/mt41692_pciconf0 --get --reg_name ROCE_ACCL | diff - roce-accl-before.txt — expected: no differences.[8]
  6. Write the escalation note you would send a customer: what you set, what the firmware did, and the one-sentence rule. Keep it; module 6 reuses it.
  7. Optional, only in a customer lab with a Spectrum-4 leaf: repeat steps 2 to 4 on a host attached to a 400G port with nv set interface <swp> router adaptive-routing state enabled applied, and record whether the field now holds. Do not change any switch configuration you did not create.[1]

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why a customer's iperf3 test over a bonded 800G uplink shows no adaptive routing at all, and what would have to change for it to work.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against Cumulus Linux 5.18 ECMP/adaptive-routing page and NVUE reference re-fetched 2026-09-07; Spectrum-X validated stack v2.3.1. Dates are when each page was fetched.

  1. Equal Cost Multipath Load Sharing (includes Adaptive Routing) | Cumulus Linux 5.18 · fetched 2026-09-07
  2. NVUE Reference: Adaptive Routing set and unset commands · fetched 2026-09-07
  3. NVUE Reference: Adaptive Routing show commands · fetched 2026-09-07
  4. What's New | Cumulus Linux 5.14 · fetched 2026-09-07
  5. What's New | Cumulus Linux 5.16 · fetched 2026-09-07
  6. Giga-Scale AI and the Ethernet Evolution: How Spectrum-X Ethernet Rewrites the Rules (NVIDIA Technical Blog) · fetched 2026-09-07
  7. Optimize Large-Scale AI Workloads with NVIDIA Spectrum-X (NVIDIA Technical Blog) · fetched 2026-09-07
  8. BlueField-3 / ConnectX-7 RoCE adaptive_routing_forced_en issue (NVIDIA Developer Forums) · fetched 2026-09-07
  9. ECMP and Adaptive Routing monitoring | Cumulus NetQ 5.1 · fetched 2026-09-07
  10. NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
  11. NVIDIA-Certified Professional: AI Networking (NCP-AIN) certification page · fetched 2026-09-07
  12. NVIDIA DSX Air User Guide · fetched 2026-09-07
  13. Route Filtering and Redistribution | Cumulus Linux 5.18 · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.