What Spectrum-X is, and what it is not
S1·E1The benchmark that measured nothing · Customer AI hall, 21:40, day two of a two-week PoC
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Define Spectrum-X as a switch plus SuperNIC system and name the parts NVIDIA includes in it.
- Describe the three hardware control loops and say which end of the wire runs each one.
- Quote the published Spectrum-X performance numbers exactly and identify claims that no NVIDIA page supports.
- List the per-feature enable commands and state what is on by default on a fresh Cumulus Linux switch.
Episode 1 — The benchmark that measured nothing
The chart on the projector says the new fabric performs exactly like the old one. Day two of a two-week proof of concept, readout Thursday, and a purchase order for thirty-two leaves that is supposed to follow it. The Dell SE has stopped talking and is holding a coffee he has already reheated twice. The customer’s network lead opens a new notebook to its first page and says, “show me the counter.”
You ask three questions instead of opening a config. What traffic did they run - iperf3 over TCP, which is not adaptive-routing eligible at all, so zero improvement is the honest result.[6] What sits at the far end of the wire - a generic RoCE NIC, whose firmware will not arm adaptive routing against anything but a Spectrum-X switch.[15] And what does the switch say is enabled - adaptive routing is disabled globally and per interface out of the box, and RoCE QoS defaults to disabled as well.[7][8]
This exists at all because ordinary Ethernet pins a flow to one hashed path, and an AI collective is a handful of enormous synchronised flows, so they collide: one link saturates while its neighbours idle. NVIDIA’s answer is a loop with two ends - the switch picks the least-congested port packet by packet, and the SuperNIC puts the packets back in order in host memory so the application never sees the spraying.[2][3]
Spectrum-X is a loop with two ends; cut either end and the customer has bought ordinary Ethernet.
Segment 1 takes that two-ended sentence literally.
1A platform, with two ends
NVIDIA describes Spectrum-X as “the world’s first end-to-end, AI-optimized Ethernet platform for multi-tenant generative AI factories”.[1] Read end-to-end literally. The platform is a Spectrum switch at one end and a SuperNIC at the other, and most of what makes it different from stock Ethernet only works when both ends are present.[1][5]
NVIDIA’s Network Operator documentation gives the tightest definition: Spectrum-X is “an AI-optimized Ethernet networking platform that combines NVIDIA Spectrum switches with the BlueField-3 SuperNIC, ConnectX-7 NIC, and ConnectX-8 SuperNIC families to deliver high-bandwidth, lossless RoCE for the GPU-to-GPU compute (east-west) network”.[5] Notice the wording: BlueField-3 and ConnectX-8 are SuperNICs, ConnectX-7 is a NIC.[5] The Spectrum-X platform page pairs each with a GPU generation - BlueField-3 for Hopper-era systems, ConnectX-8 at 800 Gb/s for Blackwell, ConnectX-9 at 1,600 Gb/s per GPU for Vera Rubin NVL72.[1] The ConnectX-8 hardware manual, by contrast, never uses the words Spectrum-X or adaptive routing on its introduction page; it describes “advanced routing and telemetry-based congestion control capabilities”.[14] The platform language lives on the platform pages, not in the hardware manuals.
Diagram source (Mermaid)
flowchart LR GPU["GPU server"] --> SNIC["SuperNIC: BlueField-3 or ConnectX-8"] SNIC -- "RoCEv2 sprayed per packet" --> LEAF["Spectrum-4 / -6 leaf"] LEAF --> SPINE1["spine 1"] LEAF --> SPINE2["spine 2"] SPINE1 --> LEAF2["remote leaf"] SPINE2 --> LEAF2 LEAF2 --> SNIC2["SuperNIC reorders + places in host memory"] SNIC2 --> GPU2["remote GPU"] LEAF -. "queue-depth telemetry" .-> SNIC LEAF2 -. "RTT probes + ECN marks" .-> SNIC2
2Three loops, not one knob
NVIDIA describes Spectrum-X as three hardware-accelerated mechanisms working together.[2] First, in-switch adaptive routing: “a quantized hardware approximation of the Join-Shortest-Queue (JSQ) algorithm”, where switches sample egress queue depths and steer packets to the least-congested port “within hundreds of nanoseconds”.[2] Second, targeted congestion control: switches generate ECN marks “only when adaptive routing capacity is exhausted”, and senders adjust rates using “precise RTT probes combined with ECN marks”.[2] Third, NIC-based Plane Load Balancing (PLB), a hardware engine in the SuperNIC that distributes traffic across planes using “local queue feedback and end-to-end, per-plane congestion telemetry”, and temporarily filters out any plane suffering end-to-end congestion or a link failure.[2]
The second loop is the one that surprises people. On a stock DCQCN Ethernet fabric ECN is the only tool, so it fires constantly. On Spectrum-X congestion is first absorbed by rerouting, and marking is the second-line signal.[2] That changes what a healthy counter set looks like. NVIDIA does not publish a pause-frame heuristic, but the mechanism implies one: if pause frames are climbing on a Spectrum-X leaf, check the adaptive-routing eligibility rules - RoCEv2 unicast or VXLAN-encapsulated RoCE, layer 3 next-hop interfaces in the default VRF, Spectrum-4 at 400G or 200G - before you touch a threshold, because traffic that is not AR-eligible never got the rerouting that should have absorbed the burst.[6]
Per-packet selection guarantees out-of-order arrival, and the endpoint repairs it: “the NVIDIA BlueField-3 SuperNIC makes sure to re-order the packets and place them in the host memory, leaving the re-ordering invisible to the application”.[3] The SuperNIC also brings GPUDirect RDMA, a data path accelerator with “16 hyperthreaded cores” programmable through DOCA, and in-line crypto up to 800 Gb/s.[4]
Burst armed: 8 packets, one RDMA QP
Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.
- Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
- Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
- Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
- File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.
nv set router adaptive-routing state enabled nv set interface swp51 router adaptive-routing state enabled nv config apply nv show router adaptive-routing nv show interface swp51 router adaptive-routingNVUE reference: adaptive routing
- ok Traffic type. RoCEv2 unicast is on the eligible list. “AR-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE.”
- ok Interface type. A routed swp is a layer-3 next-hop router interface — the only eligible kind. “Adaptive routing does not support layer 3 subinterfaces, SVIs, bonds, or bond members.”
- ok VRF. Default VRF — the documented scope. “Adaptive routing applies to layer-3, default-VRF next-hop router interfaces.”
- ok ASIC. Spectrum-4 is the supported ASIC from Cumulus Linux 5.6 on. “Cumulus Linux 5.6 supports adaptive routing on Spectrum-4. Cumulus Linux 5.5 and earlier supports adaptive routing on Spectrum-3 and Spectrum-2 as a Beta feature for evaluation.”
- ok Link speed. 400G is one of the two documented speeds. “NVIDIA Spectrum-4 at 400G/200G.”
- ok Far-end NIC. A SuperNIC on a Spectrum-X switch: the NIC firmware arms AR, and it reorders the sprayed packets at the receiver. “Adaptive Routing is supported with Spectrum-X solution only.”
- always Resilient hashing. Always: AR and resilient hashing are not a belt-and-braces pair. If the customer enabled resilient_hash_enable in traffic.conf as a load-sharing fix, that is a separate mechanism, not an AR helper. “Adaptive routing has no integration with resilient hashing.”
FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.
Why the hash collides (ECMP defaults)
- On by default: IP protocol, source IP, destination IP, source port, destination port, IPv6 flow label.
- Off by default: ingress interface, GTP TEID, inner packet fields.
- Polarization fix is a per-switch seed: nv set system forwarding hash-seed 50 (range 0–4,294,967,295).
- Resilient hashing shares 65,536 buckets by default — and adaptive routing does not use resilient hashing.
Numbers you can quote
- Failover 2.68 ms, described as 400× faster than the 1.08 s of software solutions.
- Tail latency P99 8–9 µs versus 22 µs on traditional Ethernet.
- Capacity-proportional degradation: 11% bandwidth loss for 10% link failures.
- DeepSeek-V3 step time 668 ms under isolation versus 1.18 s (1.6× degradation) on traditional Ethernet.
⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.
3Numbers you may quote, and words you may not
Quote these exactly or not at all. From NVIDIA’s Spectrum-X page: Spectrum-X Ethernet “enhances network performance by 1.6x”; networks “can scale up to 128,000 GPUs in two tiers - 64x more than single-plane networks”; Spectrum-XGS Ethernet delivers “1.9x higher NCCL performance in cross-data center environments”.[1] From the giga-scale architecture blog: failover of 2.68 ms, described as 400x faster than software solutions at 1.08 s; tail latency P99 8-9 us versus 22 us for traditional Ethernet; 1.2x higher bandwidth during failures with capacity-proportional degradation of 11% bandwidth loss for 10% link failures; and a DeepSeek-V3 training step time held at 668 ms under multi-tenant load versus 1.18 s on traditional Ethernet.[2]
Two things that circulate in customer decks are not supported by any NVIDIA page fetched for this course. The term “RoCE Adaptive Routing Notification” (ARN) could not be confirmed on any official page; NVIDIA describes the equivalent behaviour as switch-side least-congested-port selection plus SuperNIC reordering, so use that description.[2][3] And the command nv set qos roce congestion-control algorithm spectrum-x-cc appears only in third-party snippets; it is absent from the Cumulus Linux 5.18 RoCE page and the NVUE QoS reference, so do not teach it or type it.[8][16]
4On the switch, nothing is on by default
A Spectrum-X switch out of the box is a Cumulus Linux switch with every Spectrum-X feature disabled. Adaptive routing defaults to disabled globally and per interface; you enable it with nv set router adaptive-routing state enabled plus nv set interface <id> router adaptive-routing state enabled, then nv config apply.[7][6] RoCE QoS is likewise off: nv set qos roce state defaults to disabled, and nv set qos roce selects lossless mode when you do turn it on.[16][8] High-frequency telemetry and What Just Happened each have their own enable and their own constraints - HFT is Spectrum-4 and later and does not work on ports using 8 lanes, and WJH must be disabled when the NetQ agent runs on the same switch.[9][10]
So the fastest sanity sweep on a customer leaf is four reads:
nv show router adaptive-routing
nv show qos roce
nv show system telemetry hft
nv show system wjh channelEligibility is the other half. Adaptive routing applies to RoCEv2 unicast and VXLAN-encapsulated RoCE on layer 3 next-hop router interfaces in the default VRF, on Spectrum-4 at 400G and 200G. It does not use resilient hashing, is not supported on layer 3 subinterfaces, SVIs, bonds or bond members, and Spectrum-4 does not support it on 800G links.[6] On the host side the firmware enforces the same platform boundary: an attempt to set adaptive_routing_forced_en against a non-Spectrum-X switch returns success while the field stays zero, and NVIDIA’s answer on its own forum was “Adaptive Routing is supported with Spectrum-X solution only”.[15]
Four commands, on the projector
You run the sweep on leaf01 while everyone watches: nv show router adaptive-routing, nv show qos roce, nv show system telemetry hft, nv show system wjh channel. Adaptive routing and RoCE QoS come back off - the shipping default, not a defect.[7][8] The bench rig has generic NICs, so half the loop was never present.[15] What you say: “You benchmarked Ethernet with Spectrum switches in it. Turn the features on, put SuperNICs on the far end, re-run with RoCE instead of TCP - then the chart means something.”[6] The network lead writes all four commands down. Then the SE turns his laptop round: the bill of materials, due at noon, thirty-two liquid-cooled chassis quoted into a hall that has no facility water.
Lab
Read-only. The Dell lab has the SuperNIC half of the platform even though it has no Spectrum switch; that half is worth knowing cold.
- Start the management tools and list devices:
Expected: a device path such assudo mst start sudo mst status -v/dev/mst/mt41692_pciconf0on a BlueField-3. Ifmstis missing, MFT is not installed - stop, do not install anything on a shared lab host without a change note. - Read the firmware level:
Expected: ansudo flint -d /dev/mst/mt41692_pciconf0 qFW Versionon the 32.x line for BlueField-3 or 40.x for ConnectX-8. Record it; lesson 4 compares it against the RA v2.3.1 row (BlueField-3 32.50.1002, ConnectX-8 40.50.1002).[11] - Look for the adaptive-routing knobs without changing them:
Expected:sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -e q | grep -i ADAPTIVE sudo mlxreg -d /dev/mst/mt41692_pciconf0 --get --reg_name ROCE_ACCLadaptive_routing_forced_enpresent and0x00000000in a lab with no Spectrum-X switch. That zero is the platform boundary, not a defect - NVIDIA’s forum answer states adaptive routing is supported with the Spectrum-X solution only.[15] Do not attempt the--setform here; there is nothing for it to arm against. - Optional, only in a customer or partner lab with a real SN5600 or SN6600 leaf: run the same four
nv showcommands from the no-hardware lab and compare the output shape with your DSX Air transcript. Read-only, nonv set, nonv config apply.[6][8]
Goal: see the Spectrum-X configuration surface and prove to yourself that every feature is off until you turn it on. Note up front: NVIDIA “no longer releases Cumulus VX as a standalone image” and points at DSX Air instead[18], and the containerlab cvx kind page returns 404 (checked 2026-09-07, not an NVIDIA source), so DSX Air is the supported path - and its virtual switch cannot exercise the Spectrum-4 data plane.[12]
- Sign in at
https://dsx-air.nvidia.com/simulations, choose Create Simulation then Blank Canvas, and drag two Cumulus leaves, two spines and four Ubuntu hosts onto the canvas.[17] Expected: nodes appear with a per-node properties panel for CPU, memory, storage and interfaces. If nothing drags, the palette needs the simulation to be in edit state - you cannot add or edit nodes after a simulation has started for the first time.[17] - Start the simulation, open a console on
leaf01and record the software version:
Expected: a 5.x version string. Write it down - lesson 4 compares it against the RA row.[11]nv show system version - Run the four-command sweep and record each answer verbatim:
Expected: adaptive routing off, RoCE not configured. If a command errors instead of returning an empty state, note the exact text - the difference between “off” and “not available in this environment” is the point of step 4.[6][8]nv show router adaptive-routing nv show qos roce nv show system telemetry hft nv show system wjh channel - Enable adaptive routing on the leaf and re-read it:
Expected:nv set router adaptive-routing state enabled nv config apply nv show router adaptive-routingenable onin the applied column.[7] Now write the honest caveat in your notes: the virtual-appliance documentation lists adaptive routing, QoS, WJH, ACLs, ISSU, PTP and SPAN as unsupported in a virtual switch, so you have verified the configuration surface and nothing about the data plane.[12] (Rollback if you want a clean node:nv unset router adaptive-routingthennv config apply.) - Deliverable: a one-page brief naming the three loops, the enable command for each feature you found, and the exact NVIDIA-sourced numbers you would quote for failover and tail latency.[2]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE in four sentences why Spectrum-X cannot be bought as a switch alone, and what you would check first on a fabric that already has the switches.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 docs, NVIDIA Spectrum-X platform page, Spectrum-X Validated Solution Stack v2.3.1, re-fetched 2026-09-07. Dates are when each page was fetched.
- NVIDIA Spectrum-X Ethernet Platform · fetched 2026-09-07
- Giga-Scale AI and the Ethernet Evolution (NVIDIA Technical Blog) · fetched 2026-09-07
- Optimize Large-Scale AI Workloads with NVIDIA Spectrum-X · fetched 2026-09-07
- Powering Next-Generation AI Networking with NVIDIA SuperNICs · fetched 2026-09-07
- NVIDIA Spectrum-X Ethernet Networking Platform (Network Operator 26.4.0) · fetched 2026-09-07
- Equal Cost Multipath Load Sharing incl. Adaptive Routing | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference: Adaptive Routing set/unset commands · fetched 2026-09-07
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- High Frequency Telemetry | Cumulus Linux 5.18 · fetched 2026-09-07
- What Just Happened (WJH) | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA DSX Air User Guide · fetched 2026-09-07
- NVIDIA ConnectX-8 SuperNIC User Manual: Introduction · fetched 2026-09-07
- NVIDIA developer forum: adaptive_routing_forced_en on BlueField-3/CX-7 · fetched 2026-09-07
- NVUE Reference: QoS set/unset commands · fetched 2026-09-07
- Custom Topology | NVIDIA DSX Air · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Spectrum-X architectureDOCA course · Same ground: SuperNIC, numbers and Spectrum-X
- Scenario: 'the fabric is slow and adaptive routing does nothing'Elsewhere in this course · Same ground: SuperNIC, Adaptive routing and defaults
- Telemetry-based congestion control: three loops, not oneElsewhere in this course · Same ground: numbers, SuperNIC and verify