Why spraying needs a SuperNIC: reordering and direct data placement

Episode 2 — The line item the server team wants to change

The situation · Institute design review, a windowless room off the data hall

The pod-two bill of materials is on the screen and the server team wants one line cheaper: same GPUs, same switches, ConnectX-7 instead of ConnectX-8 in every node. Procurement backs them on her two grounds, price and lead time, and on lead time she is right. Their argument is reasonable in isolation. It is a RoCE adapter, it does 400G, and the network team already signed off on the fabric. You are the FAE, the network lead has his notebook open at the page from last week’s demo, and the room is watching to see whether you will simply say no.

The reason that line cannot move is the reason the fabric was designed this way. Adaptive routing chooses the least-congested port for data transmission on a per-packet basis^[1], so consecutive packets of one RoCE message cross different spines and arrive out of order on purpose. Somebody has to repair that at wire rate, and NVIDIA puts the repair in the adapter: the BlueField-3 SuperNIC re-orders the packets and places them in host memory, leaving the re-ordering invisible to the application^[1], while the ConnectX-8 generation places packets directly in order into the buffer as they arrive^[2]. On the same page where NVIDIA defines the platform, ConnectX-7 is named a NIC and the other two are SuperNICs^[4].

So the answer is not no. It is: here is what the word SuperNIC is doing in that line. Ask what a prefix is doing in a product name before you delete it from a bill of materials.

1Spraying creates a problem on purpose

Adaptive routing chooses the least-congested port for data transmission on a per-packet basis.^[1] Consecutive packets of the same RoCE message therefore cross different spines, meet different queue depths, and arrive out of order. That is not a corner case to be tuned away; it is the mechanism working.

The contract is that the endpoint repairs it. The BlueField-3 SuperNIC makes sure to re-order the packets and place them in host memory, leaving the re-ordering invisible to the application.^[1] The ConnectX-8 generation states the same outcome slightly differently: SuperNICs resolve this by placing packets directly in order into the buffer as they arrive at the receiving end.^[2] Both descriptions land in the same place from the application’s point of view — the verbs layer sees an in-order message — but they are different implementations and it is worth using the wording that matches the adapter in front of you.

Rendering diagram…

Diagram source (Mermaid)

flowchart LR
  Q[Sender QP - one RoCE message] --> L[Leaf - per-packet port selection]
  L -->|pkt 1| S1[spine01]
  L -->|pkt 2| S2[spine02]
  L -->|pkt 3| S3[spine03]
  S1 --> R[Receiving SuperNIC]
  S2 --> R
  S3 --> R
  R --> M[Host or GPU memory - in order]
  M --> A((Application sees an in-order message))

One message, three paths, one in-order arrival. The only box that can be replaced by software is the last one - and that is exactly the box you must not move into software.

2What is inside a SuperNIC

The word SuperNIC is a product class, not marketing shorthand for a fast NIC. On the SuperNIC page NVIDIA names the hardware that carries the Spectrum-X endpoint job: in-hardware RoCE acceleration; GPUDirect RDMA for direct GPU-to-GPU movement that bypasses the CPU; a data path accelerator (DPA) with 16 hyperthreaded cores, programmable through DOCA for congestion control and traffic management; and in-line crypto up to 800 Gb/s with IPsec and PSP.^[2] The DPA is the piece the next lesson builds on: it is where a user-written congestion control algorithm actually runs.^[2]

The congestion control side is described as tightly integrated with the Spectrum-4 switch’s real-time telemetry capabilities, letting the NIC react with microsecond precision.^[2] Combined with the switch-side story, that is the argument for why Spectrum-X is a platform rather than a switch: the endpoint and the fabric are co-designed, and the host firmware refuses to arm adaptive routing against a switch that is not part of it.^[10]

The ConnectX-8 hardware facts are worth having exact, because they land in server BOM conversations: Ethernet at 10, 25, 100, 200 and 400 Gb/s; InfiniBand XDR and NDR up to 425 Gb/s per 4x port and HDR up to 212.5 Gb/s; PCIe Gen6 x16 at 64 GT/s through the edge connector, with Socket-Direct variants listing PCIe Gen6/5 SERDES at 64 or 32 GT/s through the x16 edge connector plus PCIe Gen5 SERDES at 32 GT/s through a PCIe auxiliary connection card, and a downstream port on Gen5 SERDES at 32 GT/s through an MCIO connector.^[3]

3Names matter: NIC, SuperNIC, DPU

NVIDIA’s Network Operator page defines the platform as one that combines NVIDIA Spectrum switches with the BlueField-3 SuperNIC, ConnectX-7 NIC and ConnectX-8 SuperNIC families to deliver high-bandwidth, lossless RoCE for the GPU-to-GPU compute (east-west) network.^[4] Read that sentence slowly: ConnectX-7 is listed as a NIC, not a SuperNIC, in the same breath as two SuperNICs.^[4] The distinction has consequences later — dual- and quad-plane multiplane deployments are ConnectX-8 only, and ConnectX-7 appears in the single-plane row.^[4]

The validated stack keeps the same split in its tracks: the H200 track pins BlueField-3 firmware, while the GB300 and B300 tracks pin ConnectX-8 firmware, and there is no separate “SuperNIC firmware” row anywhere on the page — SuperNIC firmware is the ConnectX-8 or BlueField-3 firmware column.^[6] Firmware notes carry the same membership from the other direction: ConnectX-8 firmware v40.48.1000 states that this firmware version is tested as part of Spectrum-X reference architecture release version 2.1.^[8]

Adapter	NVIDIA’s word for it	Where it shows up
BlueField-3	SuperNIC (and, in DPU SKUs, a DPU)	H200 validated-stack track; single-plane row^[6]^[4]
ConnectX-7	NIC	Single-plane row only^[4]
ConnectX-8	SuperNIC	GB300 and B300 tracks; the only dual- and quad-plane adapter^[6]^[4]

4The bond argument, and how to answer it

A customer will eventually quote the Cumulus page back at you: “adaptive routing is not supported on bonds or bond members, so your spraying story is dead on our LAG design.”^[5] Both halves of the answer are on that same page. Adaptive routing itself is unsupported on layer 3 subinterfaces, SVIs, bonds and bond members.^[5] And when you enable adaptive routing, Cumulus Linux enables the LAG hash randomizer for packet spraying on layer-3 bonds that support adaptive-routing-eligible packets, on Spectrum-4 and later, and only on static layer-3 bonds.^[5] The feature was named in the 5.16.0 release as “LAG hash randomizer for adaptive routing”.^[11]

So the honest answer is: the adaptive routing engine will not choose among bond members, and a bonded uplink is not an adaptive-routing next hop, but eligible packets are still sprayed across the members of a static layer-3 bond once adaptive routing is on. If the customer needs the queue-aware behaviour rather than randomization, the uplinks have to be individual routed ports.^[5]

ASIC

Uplink speed

Interface

VRF

Traffic

Far-end NIC

AR ineligible — Interface type: AR is off on a bond or bond member.

Falling back to ECMP hashing. The engine toggle says adaptive, but the scenario is not eligible, so the leaf hashes. Interface type: AR is off on a bond or bond member.

step 0 / 95-tuple hashdeepest queue 3

Arrival order at the receiver (modeled, revealed at the last step)

········

ECMP hash · step 0 of 9

Burst armed: 8 packets, one RDMA QP

Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.

Predict: which port takes the next packet, and why? (optional, saved)

Leaf — the adaptive routing engine

Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.

nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing

NVUE reference: adaptive routing

Eligibility gate — AR does not run

ok Traffic type. RoCEv2 unicast is on the eligible list. “AR-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE.”
blocks AR Interface type. AR is off on a bond or bond member. Separate feature, same page: with AR enabled the LAG hash randomizer sprays AR-eligible packets across members of a static layer-3 bond (Spectrum-4 and later, named in Cumulus Linux 5.16.0). “Adaptive routing does not support layer 3 subinterfaces, SVIs, bonds, or bond members.”
ok VRF. Default VRF — the documented scope. “Adaptive routing applies to layer-3, default-VRF next-hop router interfaces.”
ok ASIC. Spectrum-4 is the supported ASIC from Cumulus Linux 5.6 on. “Cumulus Linux 5.6 supports adaptive routing on Spectrum-4. Cumulus Linux 5.5 and earlier supports adaptive routing on Spectrum-3 and Spectrum-2 as a Beta feature for evaluation.”
ok Link speed. 400G is one of the two documented speeds. “NVIDIA Spectrum-4 at 400G/200G.”
ok Far-end NIC. A SuperNIC on a Spectrum-X switch: the NIC firmware arms AR, and it reorders the sprayed packets at the receiver. “Adaptive Routing is supported with Spectrum-X solution only.”
always Resilient hashing. Always: AR and resilient hashing are not a belt-and-braces pair. If the customer enabled resilient_hash_enable in traffic.conf as a load-sharing fix, that is a separate mechanism, not an AR helper. “Adaptive routing has no integration with resilient hashing.”

FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.

Why the hash collides (ECMP defaults)

Numbers you can quote

⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.

Set the interface type to bond and read the gating panel, then switch to swp and step the burst to watch arrival order at the receiver stage.

5What nobody has published

Two cautions to carry into customer conversations. First, the term “RoCE Adaptive Routing Notification”, or ARN, could not be confirmed on any official NVIDIA page during research for this course; the published material describes the behaviour as switch-side least-congested-port selection plus SuperNIC reordering, not as a named notification message.^[1]^[2] If a customer uses the term, say plainly that it is not NVIDIA’s published terminology and describe the mechanism instead.

Second, the receiving-side detail of what a non-SuperNIC adapter does with sprayed traffic is not documented by NVIDIA in the sources used here. The published claim is the positive one — SuperNICs re-order in hardware and make it invisible to the application — so the correct framing for a generic adapter is “NVIDIA does not describe that combination as supported”, not an invented failure mode.^[1]^[2] What you can demonstrate on your own hardware is the firmware gate: adaptive routing is supported with the Spectrum-X solution only, and the host register write silently fails elsewhere.^[10]

This matters for the certification too. The NCP-AIN blueprint asks candidates to describe GPU-to-GPU communications and to configure SuperNIC functionality for advanced packet processing and congestion control, so the exam expects the mechanism and the naming, not a vendor slogan.^[9]

After the room empties

How it ended

The line stays. What changes is the note beside it. Reordering is silicon on the adapter rather than code in the stack, because a per-packet software touch at 400G costs the latency and CPU budget that made RDMA worth using^[1]. You leave a three-row table behind — SuperNIC, NIC, SuperNIC — with the validated-stack track next to each, so nobody reopens it^[4]^[6]. Asked what comes after ConnectX-8, the NVIDIA PM says the two words she is paid to say. The SE’s spreadsheet reaches forty-seven rows, and you give the server lead the sentence for procurement: “The switch is only allowed to spray packets because that adapter can put them back in order.” The rehearsal is that night. The operator calls at ten past two.

Lab

Goal: inventory what your Dell-lab adapters actually expose, and map each observation to a claim in this lesson. Every step is a query; nothing is written.

Pre-flight inventory. sudo mst start then sudo mst status -v to list device paths; lspci -d 15b3: -nn to see every NVIDIA adapter in the host; uname -r and ofed_info -s (or ethtool -i <ifname>) for the driver stack. Record them before you start.

sudo flint -d /dev/mst/mt41692_pciconf0 q — expected: part number, PSID, and the firmware version. Compare that firmware with the Spectrum-X validated stack row for the H200 track (BlueField-3 firmware) and note whether your lab card is on, ahead of or behind the RA row.^[6] If flint is missing, install MFT from the DOCA-Host repository.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q — expected: the full configuration list with default, current and next-boot columns. Grep it for congestion-control and programmable-CC related fields; you will set one of them in lesson 4, so read the current values now and save the output: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q > mlxconfig-baseline.txt.
ibv_devinfo -v | head -60 — expected: device capability flags, port state, and link layer Ethernet. Note the link layer explicitly: DOCA PCC is supported for the Ethernet link type only, which you will need next lesson.
ethtool -i <ifname> and ethtool -S <ifname> | grep -iE 'out_of|reorder|cnp|pause' — expected: driver and firmware strings, plus whatever RoCE-related counters this firmware exposes. Record which counters exist; not every claim in this lesson has a matching local counter, and saying so is the point of the exercise.
Build the mapping table: one row per lesson claim (hardware reordering, GPUDirect RDMA, 16-core DPA, in-line crypto to 800 Gb/s, PCIe generation), the command you ran, and either the observed value or the words “not observable locally”.^[2]^[3]
Optional, customer lab only: repeat steps 1 to 4 on a ConnectX-8 host and contrast the PCIe generation and the firmware line against the B300 or GB300 track of the validated stack.^[3]^[6]
No rollback is required: every command in this lab reads state. If you ran anything with -y s you left the read-only path — restore from mlxconfig-baseline.txt before continuing.

Evidence / notes (commands run, outputs, what surprised you)

Why spraying needs a SuperNIC: reordering and direct data placement

Before you read: what do you already know?

After this lesson you can