Positioning multipath: what ships, what is a specification

1Four things a customer calls multipath

The paragraph names three things and means four. Sort them before you answer, because answering the wrong one costs you the room.

MRC, the transport. NVIDIA’s definition: “MRC enables a single RDMA connection to distribute traffic across multiple network paths, improving throughput, load balancing and availability for large-scale AI training fabrics.”^[1] The designers describe it as extending “the RoCEv2 Reliable Connection (RC) transport protocol to support multi-path operation borrowing several features of UET”, supporting the normal verbs interface and queue-pair abstraction “but only for write and write-with-immediate operations.”^[2] One naming note and then never again: the term is MRC, not MPRC — no vendor, specification or repository uses MPRC — and inside NVIDIA’s own DOCA corpus “MRC” already means Multi-Rack Connectivity, a service, so define the acronym before you use it.^[1][2]

Switch-side adaptive routing. A load balancing feature on the switch, enabled with switch-only NVUE commands, disabled by default, with both state toggles “Introduced in Cumulus Linux 5.1.0”.^[11] This is the one you can configure this afternoon — carrying the support list of the release the customer runs, because from Cumulus Linux 5.16 the page adds a constraint that is absent in 5.14: “Adaptive routing is only supported on the NVIDIA Spectrum-X networking platform”.^[19][20]

More queue pairs per connection. NCCL_IB_QPS_PER_CONNECTION, “a number between 1 and 128, default is 1”, whose stated rationale is routing entropy: “This can be useful on multi-level fabrics which need multiple queue pairs to have good routing entropy.”^[13] More hash draws, not per-packet spreading of one connection.

UET packet spraying. The Ultra Ethernet Specification defines three entropy modes — single path, oblivious spraying, and path-aware spraying where the entropy value changes “based on feedback provided by the destination as to which EVs saw congestion.”^[6] That is a different transport, not a mode of RoCE.

A fifth thing hides underneath and is not any of the four: out-of-order data placement, which NVIDIA documents for “ConnectX-5 adapters and above; RC and XRC QPs; DC transport” under its InfiniBand chapter.^[14] It is a receiver capability, not a path-selection mechanism. Never let it be introduced as “our MRC”.

2What NVIDIA's own pages will and will not back you on

Exactly one NVIDIA page names MRC in the multipath sense, and it is a corporate blog post published 2026-05-06 and updated 2026-06-25.^[1] Say “in the multipath sense” out loud, because a plain string search does not agree with you: searching the DOCA documentation set on 2026-09-11 returns one more occurrence of the letters MRC, a bug-fix row where MRC expands to Multi-Rack Connectivity, the DOCA service. That is a search result with a date, not a page you can send. Know precisely what the blog post carries, because in a meeting you will be quoting it from memory.

It carries the functional definition above, the collaboration list — NVIDIA “collaborated on MRC development with AMD, Broadcom, Intel, Microsoft and OpenAI” — and the provenance sentence: MRC was “proven first in production with performance optimized on NVIDIA Spectrum-X Ethernet hardware and now released as an open specification through the Open Compute Project”, where “released” hyperlinks to an OCP document.^[1][4] That document could not be read: the OCP URL returns HTTP 403 to every automated fetch, so its contents, packet formats and field widths are unverified, and you must not upgrade “released as an open specification” into “an OCP standard” or “royalty-free”.^[4]

What the page does not carry is the whole positioning argument. It names no QP type, no verbs API, no driver, no firmware setting and no configuration knob; the words “queue pair”, ibv_ and mlxconfig do not appear on it, and neither does “RoCE”.^[1] Its only hardware phrase is the generic “NVIDIA ConnectX SuperNICs” — ConnectX-8 appears zero times.^[1] ConnectX-8 enters the story only through the designers’ preprint, which names it exactly once in eighteen pages and reports no ConnectX-8 testbed, figure or measurement at all; every published experiment there runs on other vendors’ NICs.^[2]

What NVIDIA documentation does back you on is narrower and more useful. Switch-side adaptive routing has NVUE commands and a version history.^[11] Out-of-order data placement has a page and a published scope, and the DOCA page carries neither the flag nor the word “verbs” — the opt-in creation flag MLX5DV_QP_CREATE_OOO_DP and its capability query mlx5dv_ooo_recv_wrs_caps are documented in rdma-core, which is where you go for the query-then-set sequence, the both-ends requirement and the fact that there is no automatic fallback.^[14][18] And the single NVIDIA documentation page that ties NIC settings to Spectrum-X is the Kubernetes NIC Configuration Operator page — marked Tech Preview in both its title and a warning, limited to “ConnectX-8 (device ID 1023) and BlueField-3 SuperNIC (device ID a2dc)”, and dependent on a DOCA package that is not public: “To access the package, contact your NVIDIA CPM.”^[12]

3The numbers, and whose they are

Every number in this conversation belongs to a tier. Say the tier out loud with the number and you will never have to retract one.

Three worked examples of the tiering, because this is where FAEs get caught. The 1.6x headline — “Accelerate AI network performance by 1.6x over off-the-shelf (OTS) Ethernet” — carries no footnote, no workload, no cluster size and no definition of the baseline it beats, and the product page carries no chart to support it either: marketing tier, quotable only as “NVIDIA’s product page claims”, and only with the metric named.^[10] The 98% of theoretical line rate figure is better than its reputation but narrower than its usage: the blog summarises, and the paper behind it reports a 1st-percentile bandwidth of 377.23 Gbps on a 64-node subset of a Hopper cluster, under a worst-case allocation forcing every flow through a spine.^[8][9] The 2.68 ms failover number is not measured against traditional Ethernet at all: the comparison is against a software NCCL load-balancing reference on the same Spectrum-X multiplane fabric, with a different benchmark, reaching 75% of line rate on three surviving planes of a 4-plane, 1152-GPU cluster — and in the single-plane case the RDMA connection crashes outright.^[9]

The competitor’s numbers get the same treatment, not a worse one. Broadcom’s “sustain a 15:1 long-running many-to-one traffic pattern on every output port without building a trim queue” is an asserted capability from a vendor blog with no packet size, port speed, buffer configuration, metric or report behind it, and the figure appears on neither the Tomahawk 5 nor the Tomahawk 6 product page.^[5] And the Ultra Ethernet Specification makes no performance claim comparing UET to anything — the percentages that are in its 574 pages are workload characterisations (“Up to 85%”, “20-40% overall BW”) and configuration defaults, not comparisons — so any “UEC says X percent faster” number came from a deck, not from the specification.^[6]

The last one is the cross-check that ends most arguments: no primary source compares UET, MRC and SRD on the same benchmark. Each design describes itself and at best measures against TCP or against plain RoCEv2 with ECMP.^[6][2][5] Any four-column comparative chart in front of you is synthesised.

4UEC-certified is not a thing

Procurement put “UEC-certified” in the RFP because a competitor put it on a quote. It describes something the Ultra Ethernet Consortium does not issue.

What UEC actually publishes is a specification. Version 1.0.3, dated July 16, 2026, is a public 574-page PDF with no registration, licensed CC BY-ND 4.0, and written in RFC 2119 conformance keywords from end to end.^[6] Quote its conformance density only with the method attached — the customer has the same free PDF and a Ctrl-F key: text extraction on 2026-09-11 counted 969 occurrences of MUST, 98 of them MUST NOT, plus 49 SHALL.^[6] Concede that cheerfully — it is a stronger public artifact than MRC has today, and saying so costs you nothing.

The specification defines four packet delivery modes, and says the set is deliberately closed: RUD, ROD, RUDI and UUD, with the note that “Unreliable ordered delivery is not specified. The four modes provided map to all common fielded use cases.”^[6] ROD is the RC-shaped one and pays the RC-shaped price — packets “MUST be transmitted in order over the network interface using a single network path (i.e., using a single entropy value)”.^[6] RUD is the design point MRC resembles: once-and-only-once delivery, packets “delivered to the semantic sublayer in the order they arrive from the network”, selective retransmission, direct data placement out of order.^[6] It also defines three profiles, and is internally inconsistent about the middle one’s name — chapters 1 to 3 call it AI Full, chapter 7 calls it AI Extended, and the public compliance checklists say AI EXTENDED. Do not tell a customer one of the two names is stale.^[6][7]

What no document defines is a certification. The compliance page links four documents — a Readme, a test-bed recommendations document, and a Transport and a PHY-LL matrix/checklist spreadsheet — and it is the Readme, not the page, that carries the sentence introducing the other three: they are “provided to implementors for the purpose of Ultra Ethernet [UE] Compliance self-attestation”.^[7] The implementer tests “either by themselves or with the help of a third party” and then declares compliance in its own literature in a mandated format naming spec version, profile and checklist version.^[7] There is no certification, no mark, no plugfest, no interop programme and no compliant-products list — and the page’s “UEC Member Declarations” link, which looks like one, is a Necessary-Claims patent register.^[7] So the procurement-grade question is “show me your UE compliance statement — spec version, profile, checklist version”, and it is a better question than the one in the RFP.

Two more limits worth carrying. UEC does not forbid PFC: its position is that UET-CC “is designed for operation in a best-effort network and assumes that PFC is not enabled”, and “PFC SHOULD NOT be used anywhere in a best-effort network” — SHOULD-strength and conditional, not a prohibition.^[6] And NVIDIA’s own posture on UEC has no page: the Spectrum-X product page contains zero occurrences of “Ultra Ethernet” or “UEC”.^[10] Do not fill the gap with a membership tier either: the UEC members page renders its logos in JavaScript, so an automated fetch on 2026-09-11 extracted no vendor name at all, NVIDIA’s included — a fact about the page, not about the membership.^[17] Say there is nothing official to point to rather than improvising a position in the room.

MRC and UET are relatives, not enemies, and the honest bridge is already published on both sides. The MRC paper says MRC “draws upon lessons from Ultra Ethernet Transport (UET)” and that “Like UET, MRC employs packet spraying, adaptive load balancing based on ECN, out-of-order memory placement of received data, selective retransmission, and uses packet trimming to mitigate incast.”^[2] Broadcom’s article says MRC “shares many design principles with the Ultra Ethernet Transport (UET) specification, a key difference being that it appears as a minimal extension of the current Verbs specification, while UET is a clean slate redefinition with a larger scope.”^[5] Say “appears as”, not “is” — the same page calls MRC’s design decisions “effectively … a completely new transport protocol with wide-ranging network implications”.^[5]

5The hardware question underneath the architecture question

Every transport paragraph eventually becomes a slot question, and the slot question is where an FAE gets caught fastest.

Start with what NVIDIA publishes about the endpoints. Its marketing FAQ gives three SuperNIC generations: BlueField-3 for Hopper-era systems, ConnectX-8 for Blackwell at “800 Gb/s total throughput via 2×400 G, PCIe Gen6”, and ConnectX-9 for Vera Rubin NVL72 at “1,600 Gb/s per GPU via 4×200 G SerDes, PCIe Gen6”.^[10] Quote those sentences and derive nothing from them: the ConnectX-8 figure is per NIC while the ConnectX-9 figure is per GPU, and four times 200 multiplies to 800, not 1,600.^[10]

Then the Dell reality, which is what the customer can buy. The XE9680 Technical Guide carries a mixed-NIC rule that decides where the cards physically go: “the card type with the smaller quantity (≤2 cards) must be installed in Slot 31 and Slot 40 first”, after which “the remaining card type then fills the other available slots following the standard SPM order”, applying “to all high-speed NIC/DPU card types, including but not limited to BF3, CX-7, and CX-6”.^[16] The same guide carries a flat exclusion that has ended PoCs after the order landed: “PowerEdge XE9680 system with MI300X GPUs does not support SmartNIC/DPUs.”^[16] Say “SmartNIC/DPU”, not “Dell says no BlueField” — the guide never uses the word BlueField in 68 pages, and the MI300X configuration still supports several Mellanox NIC options.^[16] Ask which GPU the box shipped with before you position a DPU.

And the switch side of the quote has an absence in it that you must state as an absence. The Dell-hosted SN5600 datasheet contains zero occurrences of “adaptive”: it describes Spectrum-X, RoCE “with extensions for AI cloud servers” and “256-way equal-cost multi-path (ECMP) routing for load balancing and redundancy”, and never names adaptive routing at all.^[15] No Dell document reachable by any method publishes adaptive-routing support at model granularity for an SN-series switch. The defensible sentence is “no Dell document I can reach publishes it” — never “Dell does not support it”. Two halves, two verdicts. What Dell publishes is settled: the datasheet is in front of you and the word is not in it.^[15] What Dell supports stays open, because the Dell SONiC release notes that would settle it could not be reached by any method I tried — record that as a dated probe of your own, not as a fact about Dell’s support matrix.

6Writing the one page

The deliverable is not a meeting, it is one page the account manager can paste into the RFP response without editing. Five sections, in this order.

1. Restate the requirement. In your own words, name which of the four things each clause is asking for, and say so before you answer. “Multipath RDMA transport” is MRC; “UEC-certified switching” is a certification that does not exist; “adaptive routing across all available paths” is the switch feature, which is the one that is real and configurable.^[11]

2. One row per claim, with tier and date. Every line carries its evidence tier and the date you fetched the page. MRC is standards-summary tier, IBTA Annex 21, 2026-07-31, plus a preprint and one vendor blog; its configuration surface is not-published tier.^[3][1] Adaptive routing is documentation tier, NVUE, disabled by default.^[11] The NIC-side Spectrum-X profile is documentation tier but Tech Preview, ConnectX-8 and BlueField-3 SuperNIC only, and needs a non-public package.^[12]

3. What we can demonstrate in your lab this quarter. Two things, not three. Switch-side adaptive routing with its real scope attached — and the scope depends on the Cumulus release, so quote the one they run.^[19][20] Say in the same paragraph what the demo does not prove: the show command prints configuration, not counters, and its documented sample output is a state toggle plus a link-utilization-threshold value under an applied column, with no AR packet, rerouted-flow or event count anywhere on the page.^[21] And NCCL_IB_QPS_PER_CONNECTION for path diversity, with its ceiling stated in the same breath: each QP still rides one path at a time, so extra queue pairs buy more hash draws, not per-packet spreading of one connection, and NVIDIA publishes no recommended value.^[13] Out-of-order data placement does not go in this section, however tempting it is. NVIDIA documents it under its InfiniBand chapter and says nothing about RoCE in either direction, so on an Ethernet fabric what you can demonstrate is reading the capability and its published scope — not showing it engage.^[14][18] It belongs in the claim table as a documentation-tier row with that scope written out.

4. What we decline to assert, and why. Four sentences, each with its reason: that ConnectX-8 supports MRC as an NVIDIA statement; that MRC is an OCP standard or royalty-free; that adaptive routing requires a SuperNIC; and that any product is UEC-certified.^{[1][4][11][7]}

5. What we need from you. The Cumulus release the switches will run — the NVUE syntax changed after 5.14^[11] and the support statement changed again at 5.16, where “only supported on the NVIDIA Spectrum-X networking platform” appears for the first time^[19][20]; the GPU model in the chassis, because it decides whether a SmartNIC or DPU is supported at all^[16]; and their vendor’s UE compliance statement if the UEC clause stays in.^[7]

Then date the page and set the re-verify interval. This lesson exists because the previous episode’s answer was re-checked one quarter later; the page you write today deserves the same treatment, and the date in its footer is what makes that possible.