Multipath RC: a new transport in the annex, not in the API

1The promise multipath gives up

Module 1 built the RC contract one lesson at a time: every opcode, acknowledged delivery, retries governed by timeout and retry_cnt, and one address path installed at INIT to RTR. A competitor’s architects wrote the consequence more bluntly than NVIDIA does: RoCE “requires in-order, lossless delivery of packets in a connection”, and “the in-order requirement forces the network to use a single path per connection”.^[6] One connection, one hash, one link.

MRC keeps the connection and drops the single path. The designers describe it as “a minimal extension to RoCE” that “leverages and extends the existing Verbs API”, and, more precisely, as extending “the RoCEv2 Reliable Connection (RC) transport protocol to support multi-path operation borrowing several features of UET”.^[2] Note the verb: extends, not preserves.

The price appears immediately in the opcode column. The same paper says that “at the transport level, only the RDMA write and write-with-immediate operations are supported”, scoped twice by the qualifier that “our AI workloads only require a subset of the functionality”.^[2] That is the entire published statement. It is a statement about what is supported, never a list of what is excluded, and the words “atomic” and “RDMA read” occur nowhere in the paper.^[2] An engineer who says “MRC has no READ, no SEND, no atomics” is making an inference with no quotable sentence behind it, and the same paper reports NCCL send/recv results at 42K GPUs.^[2]

ib_read_bw -c UC   # expect failure; then -c RC

2Where MRC actually lives

First, a naming note, said once. The term is MRC, Multipath Reliable Connection, in NVIDIA’s own words, and the preprint’s Multipath RC is the same thing.^[1][2] MPRC appears in rdma-core nowhere: a case-insensitive search of libibverbs/verbs.h, providers/mlx5/mlx5dv.h and librdmacm/rdma_cma.h returns zero hits.^[15][7][16] The documentation probes run for this course on 2026-09-11 say the same: no MPRC in NVIDIA’s documentation search, nor in the DOCA, MLNX_OFED, ConnectX-8, ConnectX-9 or MFT bundles. Do not teach it or write it.

Now the trail. NVIDIA’s blog sentence is the origin: MRC was “proven first in production with performance optimized on NVIDIA Spectrum-X Ethernet hardware and now released as an open specification through the Open Compute Project”, with the word “released” hyperlinked to an OCP MRC 1.0 PDF.^[1] That PDF has never been read: opencompute.org returns HTTP 403 on every path tried, while its robots.txt is permissive - a web-application firewall challenge, not evidence the document is missing.^[5]

The IBTA leg is the one you can quote. Release 2.1 adds MRC as Annex 21, and one Link Working Group comment in the public deck reads “Include only technical sections of OCP MRC Spec” - the public evidence that the annex text was drawn from that OCP document.^[3] IBTA’s own summary says MRC “extends the Reliable Connection (RC) RoCEv2 model with explicit multipath support, congestion control, path health tracking, and failure recovery mechanisms”, and describes the software APIs “required to implement MRC in NICs, accelerators, and host stacks” while naming no vendor, product or silicon.^[3] The specification itself is a members-only download, though IBTA’s FAQ documents a non-member request path, so do not say “members only” as an absolute.^[4]

flowchart TD; P["MRC proven in production first"] --> O["OCP MRC 1.0 specification"]; O -->|HTTP 403 on every path tried| X["Never fetched - contents unverified"]; O -->|Include only technical sections| I["IBTA Vol 1 Release 2.1 Annex 21 - 31 July 2026"]; I -->|Not a public download| N["Normative text not public"]; I --> D["IBTA public overview deck"]; P --> A["arXiv 2605.04333v1 preprint"]; P --> B["NVIDIA blog post"]; D --> R["Readable and quotable - everything you may say"]; A --> R; B --> R

Two more facts belong on the slide. MRC is not a closed question even at IBTA: comments 10027 on protocol error handling and 10030 on “MRC Library and its IB Verbs relationship” are still “Under Analysis”, so the software-API question specifically is open.^[3] And the press-room check run on 2026-09-11 found no IBTA release announcing 2.1 or MRC - a probe result, not something the deck says.

3Five mechanisms, and the one that depends on the switch

The preprint lists the mechanisms in a single sentence: “Like UET, MRC employs packet spraying, adaptive load balancing based on ECN, out-of-order memory placement of received data, selective retransmission, and uses packet trimming to mitigate incast.”^[2] Five, not two, and the three usually dropped in retelling are the interesting ones.

Out-of-order placement is made possible by the wire format: “Every data packet contains the RDMA virtual address and remote key so the receiving NIC can write each arriving packet to memory immediately, no matter the arrival order.”^[2] Selective retransmission is spelled out as “fast selective retransmission, using Selective ACK (SACK) packets to indicate precisely which packets have arrived at the receiver”.^[2] And PFC goes away: spraying “is hard to combine with the priority flow control (PFC) mechanism used in lossless Ethernet because a single flow reaches the last-hop switch over hundreds of paths”, plus PFC’s head-of-line blocking between collectives, so “MRC disables PFC and uses Ethernet in best-effort (lossy) mode”.^[2] Disabling PFC is a consequence of spraying, not the starting point.

Packet trimming is the optional one, and it is not optional because it is unimportant - it is optional because it needs the switch. The paper’s own wording is “MRC can use packet trimming”, and it states that MRC’s “target deployment is best-effort (i.e. lossy) networks that support packet trimming”.^[2] A fabric whose switches do not trim does not get that mechanism.

Path selection rides an entropy value: “Each packet contains an entropy value (EV) that dictates its path through the network. The 32-bit EV is striped across the UDP source port and IPv6 flow label”, with a set size “typically 128 to 256 entries” per QP.^[2] And MRC “was designed to work with either hash-based ECMP forwarding or SRv6” - but the authors rejected static ECMP because “there still is not a simple mapping from EV to path”, deployed IPv6 segment routing instead, and “simply disabled” switch dynamic routing.^[2]

4Production first, specification after

Read NVIDIA’s provenance sentence again with the order in mind: MRC was “proven first in production with performance optimized on NVIDIA Spectrum-X Ethernet hardware and now released as an open specification through the Open Compute Project”.^[1] Specification-then-product is the shape an enterprise buyer expects. This is the reverse, and it is why the usual questions produce strange answers.

MRC is also not one vendor’s. NVIDIA attributes it to “Companies including NVIDIA, Microsoft and OpenAI” and says it “collaborated on MRC development with AMD, Broadcom, Intel, Microsoft and OpenAI”.^[1] Broadcom’s own article says the same thing from the other side, calling MRC “the result of a multi-year collaboration between AMD, Broadcom, Intel, Microsoft, NVIDIA and OpenAI”.^[6] Two competitors publishing the same collaboration list on the same day is a consortium announcement, not a competitive claim, and conceding that costs you nothing.

Now the question procurement actually asks. When can I buy it? Everything on the public record answers a different question. The preprint says “We implemented MRC in 400 and 800Gb/s RDMA NICs: NVIDIA ConnectX-8, AMD Pollara and Vulcano, and Broadcom Thor Ultra” - but the same paper’s Conclusions say “800Gb/s NICs from Nvidia, Broadcom, and AMD”, contradicting itself on the speed set, and ConnectX-8 is named exactly once in eighteen pages with no ConnectX-8 testbed, figure or measurement anywhere.^[2] Every reported experiment runs on AMD Pollara or Broadcom Thor Ultra.^[2] Implemented is not shipping; the paper says MRC “is in large-scale production use in multiple very large AI training clusters” about MRC generally, never about ConnectX-8 in orderable firmware.^[2]

And MRC is presented as one option among several, not as the next RoCE: customers “are provided with a choice of RDMA transport models”, with “Both Spectrum-X Ethernet Adaptive RDMA and MRC protocols, as well as other custom protocols” running across ConnectX SuperNICs.^[1] A choice, not a migration.

5The probes, and the real feature next door

Teach the probe, not the absence. Four probes settle the API question, and anyone can rerun them.

First, the verbs enumeration. enum ibv_qp_type in rdma-core master is unchanged - IBV_QPT_RC, UC, UD, RAW_PACKET, XRC_SEND, XRC_RECV, DRIVER - and a case-insensitive grep for “multipath”, “multi-path” and “MRC” across libibverbs/verbs.h, providers/mlx5/mlx5dv.h and librdmacm/rdma_cma.h returns zero hits.^[15][7][16] A vendor could expose a new transport behind IBV_QPT_DRIVER - that is how mlx5 ships DC - but that is not evidence anyone has.^[15] Second, documentation: the sweep run on 2026-09-11 pulled the sitemaps of fourteen networking-docs bundles - 15,425 URLs, no slug containing “mrc” or “multipath” - and the DOCA and MLNX_OFED search APIs returned only the SRP page about Linux device-mapper multipath. Third, firmware and tooling: MFT’s mlxconfig page names no multipath or spraying parameter, and the September 2026 ConnectX-8 firmware notes name no MRC, multipath or spraying.^[18][17] On the same date perftest -c still offered RC, UC, UD, XRC, DC and SRD. Fourth, standard verbs do not help either: NVIDIA documents that “RoCE does not support APM, as it relies on InfiniBand-specific mechanisms” and that “With RoCE, the alternate path is not set for RC QP”.^[11]

Now the line that must not blur. Out-of-order data placement is real, documented and different. NVIDIA’s page scopes it to “NVIDIA ConnectX-5 adapters and above; RC and XRC QPs; DC transport”, and describes packets of one QP traversing different paths being “accepted and placed directly into host memory, even if they arrive out of order” - the opposite of reordering, since data is “placed in host memory in the order it arrives”.^[10] It lives under the InfiniBand Network chapter, and the words RoCE, ECMP and Spectrum-X appear nowhere on it; that is an absence, not a denial.^[10]

Unlike MRC, it has a published API: an opt-in Direct Verbs create flag, a query-first rule and no automatic fallback, walked symbol by symbol in the Worked block below.^[8][9] What relaxes is receive-WR consumption and arrival ordering; what does not is completion delivery order, and RDMA Read and Atomic still execute in order on the responder.^[8]

6Saying it without upgrading it

Here are the sentences that survive being quoted back at you.

“MRC is real, it is in the IBTA specification as Annex 21 since 31 July 2026, and NVIDIA co-developed it with AMD, Broadcom, Intel, Microsoft and OpenAI.”^[3][1] “It extends RoCE’s Reliable Connection semantics for multipath, and at the transport level the published design supports RDMA write and write-with-immediate.”^[2] “Nobody publishes a way to enable it - no QP type, no verb, no mlxconfig parameter, no counter. If a vendor tells you it is shippable today, ask them for the configuration command.”^[15][18][17] That last one ends the argument without calling anyone a liar.

Three upgrades to refuse. “MRC is an OCP standard” and “royalty-free”: NVIDIA wrote “released as an open specification through the Open Compute Project” and nothing more - no spec version, no release date, no licence terms - and the document itself has never been fetched.^[1][5] “NVIDIA says ConnectX-8 supports MRC”: NVIDIA’s page says “ConnectX SuperNICs” and never writes ConnectX-8 at all; that name comes from a preprint with no ConnectX-8 measurement.^[1][2] And “MRC is a Spectrum-X protocol”: it is framed as an industry protocol optimized on Spectrum-X hardware, and the paper puts it on AMD and Broadcom NICs too.^[1][2]

The naming collision is worth thirty seconds, because it makes you the person who reads the documentation. Inside NVIDIA’s own DOCA corpus, “MRC” already expands to Multi-Rack Connectivity, a DOCA service; the only occurrence found is bug-fix row 4636365 in the DOCA 3.3.0 release notes, about telemetry counter and parameter names being “non-intuitive or ambiguous”.^[14] Somebody will search the docs during the meeting, find that, and conclude MRC ships. Define the acronym before you use it.

Then give them something to do this quarter. The portable multipath tool that exists today is more queue pairs per connection: NCCL_IB_QPS_PER_CONNECTION accepts “a number between 1 and 128, default is 1”, and NVIDIA’s stated rationale is routing entropy on multi-level fabrics.^[13] Scope it as the page does - can be useful, no recommended value, no published number - and on multi-rail hosts the environment value is not the resulting QP count.^[13]