Spectrum-X architecture

1Why per-flow ECMP starves AI collectives

AI training traffic is not web traffic. A collective operation is a small number of very large flows with low entropy in their headers, so a switch that load-balances by hashing the 3-tuple or 5-tuple assigns whole flows to uplinks and regularly puts two elephants on the same link while another sits idle. NVIDIA’s architecture whitepaper shows AI workloads “reduced to 60 percent of max throughput with static load balancing”.^⁠1

RoCE itself is not the problem. RDMA over Converged Ethernet runs on any suitably configured Ethernet fabric, and the customer’s existing ConnectX and generic switches will pass it. What Spectrum-X changes is that the switch and the endpoint are designed together. The whitepaper names three co-designed mechanisms:^⁠1

The platform is therefore three things sold together: Spectrum switches, SuperNICs, and software. NVIDIA lists Cumulus Linux, Pure SONiC, NetQ and DOCA as the software stack.^⁠4 The headline claim is “1.6x” AI network performance over “off-the-shelf Ethernet”; treat that as NVIDIA’s own test result, and treat the mechanisms below as the part you can reason about and verify.^⁠2

2Adaptive routing, packet spraying, and Direct Data Placement

RoCE Adaptive Routing selects, “for every packet forwarded to an ECMP group”, “the port with the minimal load over its egress queue”; only queues matching the packet’s traffic class are evaluated.^⁠1 The switches “employ packet spraying” and also “receive status notifications from neighboring switches, which can influence the forwarding decision”, so the choice is not purely local.^⁠1

Spraying has a cost: packets of one flow “may arrive out of order”. The SuperNIC “transforms any out-of-order data at the RoCE transport layer, transparently delivering in-order data to the application”.^⁠1

On the sender side the SuperNIC “can dynamically mark traffic for eligibility to packet reordering”, and the switch classifier applies spraying “only [to] these marked RoCE packets”, so traffic that needs strict ordering still gets it.^⁠1 On the receiver side DDP places sprayed packets “in the correct order in the host/GPU memory”.^⁠1

flowchart LR; A["1. Sender SuperNIC marks RoCE packets eligible for reordering"] --> B["2. Leaf: adaptive routing picks least-loaded egress queue per packet"]; B --> S1["Spine 1"]; B --> S2["Spine 2"]; B --> S3["Spine 3"]; B --> S4["Spine 4"]; S1 --> D["3. Destination leaf: packets of one flow arrive out of order"]; S2 --> D; S3 --> D; S4 --> D; D --> E["4. Receiver SuperNIC: DDP writes each packet to its offset in host or GPU memory"]; E --> F["5. Application sees in-order data"]

The whitepaper’s own result: effective bandwidth moves “from 60 percent on standard Ethernet to 95 percent with Spectrum-X”, and in its RDMA-write test all flows finish “at approximately the same time” under adaptive routing versus a wide spread under static hashing.^⁠1

3Congestion control: what adaptive routing cannot fix

Adaptive routing rebalances paths; it does not shrink a receiver’s single link. The whitepaper is explicit: “RoCE Adaptive Routing will not solve incast congestion since there is no way to route around it.” Many-to-one congestion “requires data-flow metering per endpoint”.^⁠1

The mechanism is a closed loop between switch and SuperNIC. The switch “provides real-time network telemetry information” in-band, carrying “both queuing information for accurate congestion estimation, as well as port utilization indication for fast recovery”, and the telemetry is allowed “to bypass queueing delays in congested flows”.^⁠1 The SuperNIC “manage[s] and control[s] the data sender’s data injection rate”, running the algorithm at “millions of congestion control events per second with microsecond reaction latency”.^⁠1

Why it matters for multi-job clusters: in the whitepaper’s 3-to-1 incast example without congestion control, the shared leaf back-pressures and cuts an unrelated flow “to 33 percent of available bandwidth (50 percent of expected performance)”.^⁠1 The stated goals are “multi-tenant security and multi-job performance isolation”; the product page markets the same property as “noise isolation”.^⁠1⁠2

flowchart LR; Q["Switch egress queue builds at the incast port"] --> T["In-band telemetry: queue depth + port utilization, bypasses queueing delay"]; T --> N["Sender SuperNIC congestion-control engine meters injection rate"]; N --> Q; N -. "victim flow on the shared leaf keeps its bandwidth" .-> V["Other job unaffected"]

4Switches, scale, and telemetry

The first-generation Spectrum-X switch is the SN5600: a 51.2 Tb/s Spectrum-4 ASIC with “up to 64 ports of 800GbE or 128 ports of 400GbE within a single 2U switch”.^⁠1 That is the switch NVIDIA announced with Dell, HPE and Lenovo in November 2023, using PowerEdge XE9680 servers and BlueField-3 SuperNICs in the Israel-1 reference design.^⁠4

The current generation is Spectrum-6, “102.4 Tb/s per switch chip with 200 G SerDes”; the SN6600 offers “128 ports of 800 G OSFP” and the SN6800 “512 ports of 800 G in a 5U form factor”.^⁠2 For scale, NVIDIA says “Spectrum-X Multiplane scales up to 128,000 GPUs in two tiers” and that “Spectrum-XGS accelerates multi-site NCCL collectives by 1.9x”; Spectrum-X Photonics (co-packaged optics) carries reliability and power-efficiency claims. All of these are vendor figures from NVIDIA’s own tests.^⁠2

NetQ and What Just Happened

Visibility comes from NetQ, which combines switch telemetry and DOCA telemetry; NetQ “flow telemetry maps flow paths and behavior across switch ports and RoCE queues” and is paired with What Just Happened (WJH) for drop analysis.^⁠1 The switch operating systems in the stack are Cumulus Linux and Pure SONiC.^⁠4 The validated solution stack you will read in lesson 7.3 pins Cumulus for Spectrum-4 at exact versions per release.^⁠9

5Who does the endpoint work: BlueField-3 SuperNIC, ConnectX-8, ConnectX-9

NVIDIA defines SuperNICs as “a new class of network accelerators designed to supercharge hyperscale AI workloads in Ethernet-based clouds”, “Integrated with NVIDIA Spectrum-X Ethernet”. The portfolio is BlueField-3 SuperNIC “up to 400 Gb/s”, ConnectX-8 SuperNIC “up to 800 Gb/s” and ConnectX-9 SuperNIC “up to 1.6 Tb/s throughput per GPU”.^⁠3

The BlueField-3 SuperNIC is the Hopper-era endpoint: “up to 400Gb/s of RoCE network connectivity between GPU servers” with “deterministic and isolated performance between jobs and tenants”.^⁠1

These SKUs “natively run in network interface card (NIC) mode”, keep the large number of queue pairs in on-card memory “instead of using system memory”, and include DDP.^⁠1 The modes-of-operation page agrees: “BlueField-3 SuperNIC SKUs are shipped in NIC Mode by default”, and in NIC mode the Arm cores are inactive and the device functions as a ConnectX adapter.^⁠8

ConnectX-8 and ConnectX-9

ConnectX-8 is the Blackwell-era endpoint: “up to 800 gigabits per second” through a PCIe Gen6 64 GT/s x16 edge connector, with Socket-Direct extension options.^⁠6 The HGX B300 reference architecture places “Eight NVIDIA ConnectX-8 SuperNICs on the HGX baseboard for the East/West” at “up to 800 Gb/s (2 x 400Gb/s Ethernet) per GPU”, “maintaining a 1:1 GPU-to-NIC ratio”.^⁠5 ConnectX-9 is the Vera Rubin endpoint at 1.6 Tb/s per GPU and was declared in full production at GTC in March 2026.^⁠3⁠7

The validated stack tracks exactly these pairings: H200 rows pin BlueField-3 firmware, while B300 and GB300 rows pin ConnectX-8 firmware.^⁠9 The DOCA 3.5.0 firmware table gives the three families as BlueField-3 32.50.1002, ConnectX-8 40.50.1002 and ConnectX-9 82.50.1002.^⁠10

Note what is absent: the BlueField-3 DPU SKU (Arm active) is not an east-west device in the Blackwell design. Its job is north-south, which is lesson 7.2.