Packet spraying and the reordering it hands to the NIC

1One QP, one path — where the entropy comes from

RoCEv2 is UDP encapsulation with a fixed destination port 4791 and a source port that NVIDIA describes as a field that “Can serve as an opaque flow ID to assist with ECMP (Equal-Cost Multi-Path) routing”.^[1] Read the verb: the field is available for ECMP; the page does not say the NIC varies it or how a value is derived.^[1] NVIDIA’s own mlx5 man page is just as permissive about who consumes it: the source port “is used to create entropy for network routers (ECMP), load balancers and 802.3ad link aggregation switching that are not aware of RoCE IB headers”.^[18]

The derivation lives in two kernel inlines. rdma_calc_flow_label(lqpn, rqpn) multiplies the two 24-bit QP numbers and folds the product to 20 bits; rdma_flow_label_to_udp_sport() XOR-folds that label to 14 bits and ORs in IB_ROCE_UDP_ENCAP_VALID_PORT_MIN, 0xC000.^[17] mlx5 calls them in that order: mlx5_set_path_udp_sport() takes ah->grh.flow_label and only if that is zero falls back to the QPN hash, writing the result into the address path’s udp_sport.^[2] It is gated, not universal: mlx5_set_path() calls the helper only for a RoCE address handle on an RC, UC or XRC QP with a RoCEv2 source GID, and only when the modify carries IB_QP_DEST_QPN — which is the INIT to RTR transition.^[2] The computed value is used for udp_sport only; the path’s own flow_label is set separately from the unmodified grh->flow_label.^[2]

Two consequences matter in the field. The QPN hash is symmetric because it is built on a product: the helper’s comment says it exists so that “both the requester and responder” can “calculate the same flow_label for a given connection”.^[17] RDMA-CM usually supplies a non-zero flow label first, derived from the CM port numbers as hash = (u32)sport * 31 + dport, which is not commutative.^[20] And the range convention is a should, not a rule — the same comment says “All RoCE V2 drivers should use this same convention”, yet bnxt_re writes htons(0x8CD1) into its QP1 header, below 0xC000, so a capture filter on 49152 to 65535 does not catch every RoCEv2 packet.^[17][21]

One QP therefore carries one source port at a time. mlx5dv_modify_qp_udp_sport() “enables modifying the configured UDP source port of a given RC/UC QP when QP is in RTS state”.^[18] The provider gates it on QP type, on RTS state and on the rts2rts_qp_udp_sport capability bit, and issues it as DEVX into qpc.primary_address_path.udp_sport — one source port, so it moves a QP between paths rather than spreading one QP across paths.^[3] Portable verbs have no equivalent: ibv_modify_qp() exposes only ah_attr, fixed at RTR, and alt_ah_attr for automatic path migration, gated on IBV_DEVICE_AUTO_PATH_MIG.^[19] That leaves a portable application one multipath tool — create more QPs.

2Turning the switch into a sprayer — and the phrase the docs actually use

Cumulus Linux defines the feature in one sentence, and the sentence moves with the release. On 5.14: “Adaptive routing is a load balancing feature that improves network utilization for eligible IP packets by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization.”^[4] On 5.18 the same sentence reads “for adaptive routing eligible IP packets”.^[5] Note “such as” — those are examples of switch state, not an enumerated mechanism, and the page publishes no register, counter or config key behind them.^[4] The behaviour sentence splits the same way, and it is the one to memorise: 5.14 says “With adaptive routing, the switch forwards packets to the less loaded path on a per packet basis to best utilize the fabric resources and avoid congestion”, while 5.18 says “forwards adaptive routing eligible packets” in the same position.^[4][5] Either way the basis is per packet, and that is adaptive routing’s behaviour, not the switch’s default; ordinary ECMP on the same page is per-flow hashing.^[4]

Eligibility is two lists, and which list a bullet sits in is the whole argument. Under “Cumulus Linux supports adaptive routing with”, 5.14 names Spectrum-4 at 400G and 200G speeds, RoCE2 unicast traffic, VXLAN-encapsulated RoCE traffic, layer 3 interfaces, next hop router interfaces in the default VRF, and the NVIDIA Spectrum-X networking platform.^[4] The restriction bullets that follow run unchanged into 5.18: adaptive routing “does not make use of resilient hashing”, is not supported “on layer 3 subinterfaces, SVIs, bonds or bond members”, and “The Spectrum-4 switch does not support adaptive routing on 800G links”.^[5] From 5.16 the Spectrum-X platform appears a second time, now among the restrictions — “Adaptive routing is only supported on the NVIDIA Spectrum-X networking platform” — and the support list’s traffic bullet narrows to “Adaptive routing eligible RoCE2 unicast traffic”, so quote the release the customer runs.^[5]

The syntax changed after 5.14, per command. The NVUE reference states that “In Cumulus Linux 5.14 and earlier, you specify enable on or enable off instead of state enabled or state disabled”, attached to nv set interface <interface-id> router adaptive-routing state and nv set router adaptive-routing state; both toggles were introduced in 5.1.0 and link-utilization-threshold in 5.6.0.^[6] One documented expectation carries weight in a design review: enable adaptive routing “on all the ports that are part of the same ECMP route”, because mixing adaptive routing traffic with regular ECMP routes egressing ports without it means “Unexpected route hashing might occur” — the page’s own words, in a Note rather than a Warning, with no drop, reordering or performance consequence stated.^[4]

3What spraying hands the receiver, and what strict RC does with it

Take the same sixteen-packet RDMA Write and make the switch choose per packet. Paths differ in queueing delay, so packets of one QP arrive interleaved. A strict reliable connection has one response to a gap in the packet sequence number: treat it as loss. The responder stops accepting from the gap onward and the requester retransmits, which is why out_of_sequence and packet_seq_err can climb on a fabric where nothing has actually been dropped — those two names come from the mlx5 driver, and the chain that connects them to per-packet forwarding is assembled in segment 6, not quoted from any page.^[13] The fabric spent its cleverness balancing the load, and the transport threw the result away.

The Ultra Ethernet specification is the clearest public statement of that trade because it writes both halves as conformance language. Its reliable ordered mode, ROD, is the RC-shaped one and pays the RC-shaped price: packets “MUST be transmitted in order over the network interface using a single network path (i.e., using a single entropy value)”, normatively “UET ROD mode MUST operate on a single network path”, and out-of-order new packet sequence numbers are dropped and NACKed under GoBackN recovery.^[11] Its reliable unordered mode, RUD, is the shape a multipath transport needs: it “uses selective retransmission capabilities and enables semantic processing and direct data placement out of order”, and “Packets are delivered to the semantic sublayer in the order they arrive from the network”.^[11] The specification stops there — it never says that anything above the transport restores order.^[11] The framing usually quoted for that shape is AWS’s, about SRD: reliable but out-of-order delivery that “leaves order restoration to the layers above it”.^[23]

UEC also publishes the caution that saves an FAE from over-promising: spraying alone does not balance a fabric. “Hashing is not guaranteed to spread the packets evenly. The presence of other traffic that is not sprayed may impact available bandwidth on some paths. In addition, failures may render a path unavailable.”^[11] Its conclusion is that “Improved throughput and reduced flow completion times can be achieved if the sources actively balance traffic” — which is the argument for endpoints that know something about paths, not for a switch acting alone.^[11]

cd /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ && grep . packet_seq_err out_of_sequence
rdma statistic show link mlx5_0/1
# nothing there? check for the vport_ prefix before you blame the driver:
ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ | grep -E "seq|vport_"

nv set router adaptive-routing state enabled          # 5.15+ syntax
nv set interface swp1 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routing                      # configuration, not counters

4The flag that exists, its three gates, and the chapter it is filed under

NVIDIA does document a receiver-side answer. The page is called “Out-of-order Data Placement” and its scope is stated verbatim: “This feature is only supported on: NVIDIA ConnectX-5 adapters and above; RC and XRC QPs; DC transport.”^[8] The mechanism is described plainly — packets of the same QP “may traverse different network paths from source to destination. This can lead to packets arriving out of order”, and “Instead of dropping these packets and triggering retransmissions, this feature enables them to be accepted and placed directly into host memory, even if they arrive out of order.”^[8] Do not call it reordering: the page’s own closing sentence is that data “is placed in host memory in the order it arrives”, and the words offset, segment, scatter, reorder, buffer and sort do not appear on it.^[8] Its two benefits, “Enhancing network utilization” and “Reducing latency”, carry no percentage and no benchmark.^[8]

Now the part you do not get to soften. That page sits under DOCA Programming Guide, DOCA Drivers, MLX Drivers and Technologies, InfiniBand Network, and the strings “RoCE”, “ECMP” and “Spectrum-X” appear nowhere on either version of it.^[8] The sentence to say is “NVIDIA documents out-of-order data placement only for InfiniBand.” It is not “it does not apply to RoCE” — the page neither affirms nor denies that, and NVIDIA publishes no RoCE or Ethernet counterpart page in DOCA 3.5.0.^[8]

The verbs surface, by contrast, is fully documented and it is opt-in. MLX5DV_QP_CREATE_OOO_DP is an mlx5 direct-verbs creation flag set in struct mlx5dv_qp_init_attr.create_flags and passed to mlx5dv_create_qp(); ibv_create_qp_ex() cannot set it.^[9] Three gates ride with it — query first, both ends, no fallback — and the callout below quotes them.^[9] What the flag relaxes is receive-WR consumption and arrival ordering; what it does not relax is completion ordering on either side, and RDMA Read and Atomic operations still execute in order on the responder, though RDMA Read response packets may scatter out of order on the requestor for a QP created with the flag.^[9]

flowchart LR
  A["Cumulus ECMP page: eligible packets go to the less loaded path on a per packet basis"] --> G["No NVIDIA page writes this chain as one sentence"]
  G --> B["DOCA page: packets of one QP may traverse different paths and arrive out of order - accept and place them instead of dropping"]
  A -.-> N1["5.14 and 5.15 never mention the NIC at all - 5.16 adds a NIC-side packet load balancer"]
  B -.-> N2["The words RoCE, ECMP and Spectrum-X appear nowhere on this page"]

5Why spraying and PFC argue — and what that does to last week's recommendation

The designers of MRC state the collision directly: “Spraying is hard to combine with the priority flow control (PFC) mechanism used in lossless Ethernet because a single flow reaches the last-hop switch over hundreds of paths. Further, PFC tends to create head-of-line blocking between different collectives, hurting tail latency. Thus MRC disables PFC and uses Ethernet in best-effort (lossy) mode.”^[10] Read the order of that argument: disabling PFC is a consequence of spraying, not the starting point. The same paper pairs it with “fast selective retransmission, using Selective ACK (SACK) packets” and out-of-order memory placement to absorb the resulting loss, and names its target as “best-effort (i.e. lossy) networks that support packet trimming” — where trimming is optional, since the paper says MRC “can use packet trimming”.^[10]

One scoping fact has to travel with that quote or the whole segment misleads. MRC “was designed to work with either hash-based ECMP forwarding or SRv6”, but the authors rejected static ECMP because “there still is not a simple mapping from EV to path”, deployed IPv6 segment routing instead, and “simply disabled” switch dynamic routing.^[10] The paper’s title is “…using MRC and SRv6”: the fabric it describes is not the fabric in this episode, and nothing in it endorses spraying over ECMP.^[10]

UEC says something adjacent and narrower, and the scoping matters more than the sentence: “PFC SHOULD NOT be used anywhere in a best-effort network.”^[11] Three limits travel with it: it governs UET under UET-CC only, “in a best-effort network” is the condition, and the stated cause is UET-CC’s latency assumptions, not spraying or reordering.^[11] UEC also warns that switch-autonomous spraying breaks an endpoint’s feedback loop — it is “RECOMMENDED that such behavior not be enabled for UET traffic if path-aware multipath spraying is used, as it prevents UET getting a clean signal as to which paths are congested”.^[11]

That last point is what changes the recommendation this module already shipped. NVIDIA’s own architecture narrative demotes ECN to a second-line signal when adaptive routing is the first: “the switch only generates Explicit Congestion Notification (ECN) marks when its adaptive routing capacity is completely exhausted and the queue continues to grow”, and senders “utilize precise RTT probes combined with ECN marks to adjust their sending rates at RTT timescales”.^[12] Read it as a conjunction, not a substitution: on that architecture a mark means the queue got deep and routing could not relieve it.^[12]

nv set qos roce mode lossy   # switch: drops PFC and the lossless pool, keeps ECN
echo 1 > /sys/class/net/<if>/ecn/roce_rp/enable/3
echo 1 > /sys/class/net/<if>/ecn/roce_np/enable/3

6The counters, and the two levers an operator actually has

The names come from two adjacent, separately gated arrays in the mlx5 driver. out_of_seq_q_cnts[] holds one entry, out_of_sequence; retrans_q_cnts[] holds duplicate_request, rnr_nak_retry_err, packet_seq_err, implied_nak_seq_err and local_ack_timeout_err.^[13] Each array registers only if its own capability bit is set, and the two gates are independent, so a missing packet_seq_err on a customer’s box is a capability statement rather than a broken driver.^[13] In switchdev mode on a non-host-PF port the driver uses the parallel vport_* arrays with a literal vport_ prefix, reason enough for a bare grep to come back empty.^[13] Nor are these six the whole retransmission picture: extended_err_cnts[] adds req_transport_retries_exceeded and req_rnr_retries_exceeded.^[13]

Now the honest gap, and say it out loud to the customer. The driver exports names; it defines no semantics, and NVIDIA publishes no primary page that does — including which end of a connection each counter increments on, which is why you read both ends and record which one moved.^[13] No NVIDIA page states that adaptive routing changes what out_of_sequence or packet_seq_err mean either. The chain — per-packet forwarding, out-of-order arrival, placement instead of a NAK — is assembled from separate primaries: the Cumulus ECMP page, the DOCA out-of-order page, and a blog.^[4][8][12] On the switch side there is nothing to read: nv show router adaptive-routing “Shows if adaptive routing is enabled on the switch”, and the per-interface form prints enable on and link-utilization-threshold 70 under an applied column — configuration, not counters.^[7]

Two levers remain, and both are host-side. The first is queue pairs: NCCL_IB_QPS_PER_CONNECTION is “a number between 1 and 128, default is 1”, and NVIDIA’s own rationale is routing entropy — it “can be useful on multi-level fabrics which need multiple queue pairs to have good routing entropy”, with no recommended value and no published performance number.^[14] The page does not tell you the resulting QP count; the source does, computing nQpsPerDev * ndevs on both sides and taking the larger, so on merged or multi-rail hosts the count is the environment value times the device count.^[16] The second lever is NCCL’s adaptive routing framing: NCCL_IB_ADAPTIVE_ROUTING is “Enabled (1) by default on IB networks. Disabled (0) by default on RoCE networks”, and it controls how NCCL frames sends above NCCL_IB_AR_THRESHOLD.^[14] NCCL_IB_OOO_RQ is documented as a bare force-enable — “Disabled (0) by default. Set to 1 to force enable” — with no precondition stated anywhere on the page, and that omission is the trap.^[14] The 2.31.2 source refuses the connection on an mlx5 device when the force-enable is set and ar == 0, returning ncclInternalError with “OOO RQ is force enabled but AR is not enabled”; on RoCE, where ar is 0 by default, the force-enable alone fails connection setup.^[16]

/sys/class/infiniband/<dev>/ports/<p>/hw_counters/
grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/*