Transports and opcodes: what RC, UC and UD can actually do
S1·E2The one requirement that changed the transport · Customer HQ, design review for the imaging storage tier, the morning after the lab
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Reconstruct the opcode by QP-type matrix from memory and justify why READ and the atomics are reliable-only.
- Choose a transport for a stated workload and defend the choice against the opcodes it needs.
- Explain who owns a send buffer between `ibv_post_send()` and its work completion and when that rule is suspended.
- Separate `IBV_WC_RETRY_EXC_ERR` from `IBV_WC_RNR_RETRY_EXC_ERR` and route each to the right owner.
Episode 2 — The one requirement that changed the transport
Two hours into the design review the picture on the screen is clean. Every node pushes updates straight into its peers with one-sided writes, no remote CPU is interrupted, and to keep card resources down the storage architect wants the whole tier on UC. The room likes it, including the procurement person, who likes anything that lowers a count. Then the application owner reads line nine out loud: a node that misses a value should be able to pull it from a peer.
That sentence rewrites the transport column. A read has to bring bytes back to the requester and re-request them if they are lost, so IBV_WR_RDMA_READ and the atomics are allowed on RC and XRC only, UC gets one-sided writes and plain sends and nothing more, and UD is limited to SEND.[1] The matrix is not vendor policy; it is what reliability costs, expressed as opcodes.
Which is why more than one transport exists at all. Not every workload wants an acknowledged, retried connection to every peer, so the architecture offers a ladder — reliable connected, unreliable connected, unreliable datagram, raw packet — and an application pays only for the guarantees it actually uses.[2] Somebody asks whether a future card removes the question; the NVIDIA PM on the call says “not announced” and the room moves on.
Pick the opcode the workload needs first. The transport is a consequence, not a preference.
The SE is already typing another row into the spreadsheet. There is a whiteboard behind you and five minutes before the room moves on.
1The matrix is the lesson
Everything about transport choice reduces to one table from ibv_post_send(3). IBV_WR_SEND is allowed on UD, UC, RC, XRC and RAW_PACKET; SEND_WITH_IMM on UD, UC, RC and XRC; RDMA_WRITE and RDMA_WRITE_WITH_IMM on UC, RC and XRC; RDMA_READ, ATOMIC_CMP_AND_SWP and ATOMIC_FETCH_AND_ADD on RC and XRC only; LOCAL_INV, BIND_MW and SEND_WITH_INV on UC, RC and XRC; and IBV_WR_TSO on UD and RAW_PACKET.[1] Three consequences are worth committing to memory: UD can only SEND, UC gets one-sided WRITE but no READ and no ATOMIC, and only RC and XRC get READ and the atomics.[1]
The reason is reliability, not vendor policy. A read has to bring bytes back to the requester and re-request them if they are lost; an atomic has to return the pre-operation value. Both need an acknowledged transport with a retry state machine, which is exactly what the RC-only knobs in the next lesson provide.[5] UC is connected — it has an address vector and a path_mtu — but nothing is acknowledged, so there is no retry counter and no error 12 or 13 to read.[5]
Upstream verbs enumerates the QP types as IBV_QPT_RC, IBV_QPT_UC, IBV_QPT_UD, IBV_QPT_RAW_PACKET and IBV_QPT_DRIVER, where the last is vendor-specific and not a standard transport service.[2] perftest exposes the choice as -c, --connection=<RC/XRC/UC/UD/DC/SRD> with RC as the default, and its man page states the same restrictions from the other direction: UD only for Send, UC only for Write or Send.[6][7]
| opcode | RC | UC | UD | XRC |
|---|---|---|---|---|
| ✓ | ✓ | ✓ | ✓ | |
| ✓ | ✓ | ✓ | ✓ | |
| ✓ | ✓ | — | ✓ | |
| ✓ | ✓ | — | ✓ | |
| ✓ | — | — | ✓ | |
| ✓ | — | — | ✓ | |
| ✓ | — | — | ✓ | |
| ✓ | ✓ | — | ✓ | |
| ✓ | ✓ | — | ✓ | |
| ✓ | ✓ | — | ✓ | |
| — | — | ✓ | — |
Matrix transcribed from ibv_post_send(3). RAW_PACKET (SEND, TSO) is omitted — it is not a RoCE transport.
IBV_WR_RDMA_READ
illegal on UC
One-sided read. Requires a reliable transport because the data comes back: RC and XRC only. Depth is bounded by max_qp_rd_atomic / max_dest_rd_atomic.
work-request union: wr.rdma.remote_addr + wr.rdma.rkey
ib_read_bw -c UC # expect failure; then -c RC
Connected, so it has path_mtu and an AV — but nothing is acknowledged, so there is no retry counter and no error 12/13 to read.
IBV_SEND_FENCE— "Set the fence indicator. Valid only for QPs with Transport Service Type IBV_QPT_RC" — on UC/UD it is silently meaningless.IBV_SEND_SIGNALED— Generate a completion for this WR. Needed for every WR unless sq_sig_all was set at create time.IBV_SEND_INLINE— "If the IBV_SEND_INLINE flag was set, the buffer can be reused immediately after the call returns" — otherwise the buffer belongs to the HCA until its completion is polled.IBV_SEND_SOLICITED— Raises a solicited event on the responder's completion channel.IBV_SEND_IP_CSUM— Offload the IP/L4 checksum.
“ibv_post_send() returns 0 on success, or the value of errno on failure” — it does not set errno, so perror() lies here.
IBV_WC_RETRY_EXC_ERR (12): the requester retried retry_cnt times and the responder never ACKed → loss, blackholing, or an MTU/PFC mismatch on the fabric.
IBV_WC_RNR_RETRY_EXC_ERR (13): the responder was reachable but had no receive buffers posted → application flow control (or min_rnr_timer too aggressive).
On a failed completion only wr_id, status, qp_num and vendor_err are valid. Numbers are enum positions in rdma-core verbs.h.
Sources: ibv_devinfo(1) · ibv_create_qp(3) · ibv_create_cq_ex(3) · ibv_create_srq(3) · ibv_reg_mr(3) · ibv_poll_cq(3) · ibv_query_device_ex(3) · NVIDIA Optimized Memory Access · perftest
2The work request, and who owns the buffer
struct ibv_send_wr is a linked-list node: wr_id, next, sg_list, num_sge, opcode, send_flags, a union for imm_data/invalidate_rkey, and a union whose rdma arm carries remote_addr and rkey, whose atomic arm adds compare_add and swap, and whose ud arm carries ah, remote_qpn and remote_qkey.[1] That union is the transport matrix in structural form: the fields a UD send needs do not exist on the path a one-sided write uses.
Five send flags exist: IBV_SEND_FENCE (“Set the fence indicator. Valid only for QPs with Transport Service Type IBV_QPT_RC”), IBV_SEND_SIGNALED, IBV_SEND_SOLICITED, IBV_SEND_INLINE and IBV_SEND_IP_CSUM.[1] Two behaviours around them cause real bugs. First, the ownership rule: a posted buffer cannot be reused “until the request is fully executed and a work completion has been retrieved from the corresponding completion queue (CQ)” — except that “if the IBV_SEND_INLINE flag was set, the buffer can be reused immediately after the call returns”.[1] Second, on UD the address handle “must remain valid until completion retrieval” or behaviour is undefined.[1]
And the error convention: “ibv_post_send() returns 0 on success, or the value of errno on failure” — it does not set errno.[1] Code that posts, gets a non-zero return and then prints strerror(errno) prints something unrelated. That single line has cost more debugging hours than the matrix.
3Reading a completion, especially a bad one
struct ibv_wc carries wr_id, status, opcode, vendor_err, byte_len, an immediate/invalidated-rkey union, qp_num, src_qp, wc_flags, pkey_index, slid, sl and dlid_path_bits.[3] On a non-success completion only wr_id, status, qp_num and vendor_err are guaranteed valid — reading byte_len or opcode from a failed WC is reading noise.[3] Two more rules from the same page: “Each polled completion is removed from the CQ and cannot be returned to it”, and “The user should consume work completions at a rate that prevents CQ overrun”.[3]
The status enum in verbs.h runs, in order: IBV_WC_SUCCESS, LOC_LEN_ERR, LOC_QP_OP_ERR, LOC_EEC_OP_ERR, LOC_PROT_ERR, WR_FLUSH_ERR, MW_BIND_ERR, BAD_RESP_ERR, LOC_ACCESS_ERR, REM_INV_REQ_ERR, REM_ACCESS_ERR, REM_OP_ERR, RETRY_EXC_ERR, RNR_RETRY_EXC_ERR, and on through FATAL_ERR, RESP_TIMEOUT_ERR and GENERAL_ERR.[4] Because the codes are numbered by position, IBV_WC_RETRY_EXC_ERR is 12 and IBV_WC_RNR_RETRY_EXC_ERR is 13.[4] ibv_wc_status_str() is what turns them into the strings “transport retry counter exceeded” and “RNR retry counter exceeded” that appear in customer logs.[4]
One more piece of enum trivia that pays off when reading someone else’s poll loop: IBV_WC_RECV is defined as 1 << 7 (128) precisely “so consumers can test if a completion is a receive by testing (opcode & IBV_WC_RECV)”.[4]
4Choosing a transport in front of a customer
The decision is short. If the workload needs one-sided reads or atomics — a key-value store pulling remote buffers, a lock manager, most storage targets — it is RC or XRC and there is nothing to discuss.[1] If it only pushes and can tolerate loss at the application layer, UC removes the acknowledgement machinery. If it is many-to-many with small messages and the application already has its own reliability, UD plus a shared receive queue keeps QP count from exploding, at the cost of SEND-only semantics.[1][10]
Two constraints bound that choice. First, an SRQ can only be attached to RC or UD QPs — creation fails for any other type with an SRQ attached.[2] Second, UD has no path_mtu at all: IBV_QP_PATH_MTU is “valid for RC/UC QPs only”, so a UD message is bounded by the port MTU minus headers, which is why UD behaves differently from RC under an MTU mismatch.[5]
On the DOCA side the same choices appear with different names: DOCA RDMA offers send, send-with-immediate, receive, read, write, write-with-immediate, atomic compare-and-swap and atomic fetch-and-add, and its DC (“dynamically connected”) transport support is documented as “in alpha level only”.[9] DOCA also states plainly that its RDMA operations “are not atomic” in the sense that the application must synchronise, and that task buffers must remain valid until completion — the same ownership rule as raw verbs.[9]
The goal is to demonstrate, with output rather than assertion, that UC cannot read and UD can only send. Run everything on the two-node pair; the same options must be passed to server and client.
- Baseline the reliable case. Server:
ib_read_bw -d mlx5_0 -c RC -s 65536 -n 1000. Client:ib_read_bw -d mlx5_0 -c RC -s 65536 -n 1000 <server_ip>. Expected: a normal result table. Reasoning: RC is the perftest default and allowsRDMA_READ.[1][7] - Break it on purpose. Client:
ib_read_bw -d mlx5_0 -c UC -s 65536 <server_ip>. Expected: the tool refuses the combination; the man page states UC is valid for Write or Send only. Record the exact message — that string is what a customer will paste into a ticket.[7] - Show what UC can do. Server and client:
ib_write_bw -d mlx5_0 -c UC -s 65536 -n 1000. Expected: a result table. Reasoning:RDMA_WRITEis allowed on UC.[1] - Show the UD boundary. Server and client:
ib_send_bw -d mlx5_0 -c UD -s 1024 -n 1000. Expected: a result table. Thenib_write_bw -d mlx5_0 -c UD— expected: rejected, because UD is Send only.[7][1] - Show the MTU boundary that follows from having no
path_mtu. Runib_send_bw -c UD -s 4096; the message size is bounded by the port MTU minus headers rather than by anIBV_MTU_*value, becauseIBV_QP_PATH_MTUis valid for RC and UC only.[5] - Record for each run: the transport, the opcode implied by the binary name, and whether it ran or was refused. That three-column table is the matrix, re-derived from your own hardware.
- Reliable baseline:
ib_read_bw -d ____ -c ____ -s 65536 -n 1000on both sides. RC allowsIBV_WR_____. - The refusal:
ib_read_bw -c ____fails because that transport is documented as valid for ____ or ____ only. - UC’s one-sided capability is proven with the binary
ib_______bw -c UC. - UD is limited to the
ib_______bw -c UDbinary because its only data opcode isIBV_WR_____. - UD has no
IBV_QP_____, so its message size is bounded by the ____ MTU minus headers instead. - If a run fails for a reason unrelated to the matrix, the perftest README warns that a cryptic ____ string comes from ____, not from perftest itself.
A Dell customer runs a distributed cache. Each node must (a) push updates into peers without interrupting their CPUs, (b) occasionally pull a remote value it does not have cached, and (c) hold 4000 peer connections without exhausting card resources.
Produce: one transport recommendation, the specific opcode from the matrix that forces it, one alternative you rejected and the exact capability it lacks, and one resource mitigation from this module for the connection count. Acceptance criteria: your answer names an opcode, names a transport that the matrix forbids for that opcode, and names the receive-side object that reduces per-connection buffer cost together with the one restriction on which QP types may attach to it.
After the review
You redraw the tier with two paths: the pushes stay one-sided, the pull forces RC, and a shared receive queue brings the per-connection buffer cost back down — remembering that an SRQ attaches to RC or UD queue pairs only.[1][2] The architect keeps the read. Procurement asks what RC costs in lead time, which is, for once, the right question. You leave the operations team one triage rule for later: status 12 is transport retry exceeded, status 13 is RNR retry exceeded, and they have different owners.[4] What you say: “Pick the opcode first. The transport is not a preference, it follows.” The cutover window is Thursday at midnight — the first time this traffic ever leaves a single rack.
Lab
Dell-lab ConnectX or BlueField-3 pair. Read-only: only benchmarks, no configuration change, so nothing to roll back.
- Pre-flight inventory:
ibv_devinfo -l,ibdev2netdev, andibv_devinfo -d mlx5_0 | grep -E "active_mtu|link_layer". Record theactive_mtu— it is the default perftest will use. - Repeat the four matrix runs from the no-hardware lab with
-d mlx5_0and the real netdev IP. Same flags on both sides, always. - Add the MTU demonstration:
ib_send_bw -d mlx5_0 -c UD -s 4096 -n 1000. Compare againstib_write_bw -d mlx5_0 -c RC -s 4096 -m 4096. Expected: the RC run accepts an explicit-m, the UD run has nopath_mtuto set. If the UD run fails outright at 4096, the port MTU minus headers is below 4096 — record the port MTU and say so. - Diff the refusal messages against the ones Soft-RoCE produced. Any difference is a driver-level difference worth knowing before a customer reports it.
- Optional, only on a customer or partner lab that has one: repeat with
-c DC. This transport is not reachable from the containerlab path at all, and DOCA documents its DC support as alpha-level, so treat any result as informational rather than as a supported configuration.
Soft-RoCE on two containerlab Linux nodes. Read-only: nothing is reconfigured, only benchmarks are run.
- Bring up Soft-RoCE on both nodes:
modprobe rdma_rxethenrdma link add rxe0 type rxe netdev eth1. Expected: no output from either command;rdma linkthen listslink rxe0/1 state ACTIVE physical_state LINK_UP netdev eth1on both nodes. If not:modinfo rdma_rxeto confirm the module exists on this kernel, andip -br linkto confirm the netdev name. - Run the transport matrix by hand, recording the exact output of each:
ib_send_bw -d rxe0 -c UD,ib_write_bw -d rxe0 -c UC,ib_read_bw -d rxe0 -c UC,ib_read_bw -d rxe0 -c RC. Server first, then the client with the server IP appended, and the same flags on both sides. - Expected: runs 1, 2 and 4 produce result tables; run 3 is refused. If run 3 instead produces a table, you have a perftest build that accepts the flag and fails later — capture whatever it prints and treat that string as the answer.
- For every refusal write down the literal error text next to the matrix row it proves. The perftest README warns that a cryptic syndrome string comes from rdma-core, not from perftest, so a syndrome is a verbs error and belongs in the matrix column, not in a bug report against the tool.
- Produce the contrast yourself instead of reading it: start
ib_write_bw -d rxe0 -D 30on both sides, thenip link set eth1 downon the server. Expected: the client ends with “transport retry counter exceeded” (status 12). Bring the link back up withip link set eth1 up, then re-run with the server started but no client receives posted and classify the second failure. Write one sentence for each saying which counter you would ask the customer for next. If not: if the client hangs instead of failing, shorten the budget with-u 8and repeat. - Limitation to record: Soft-RoCE is a software implementation, so the set of transports it accepts may be narrower than a ConnectX. Note any combination the hardware lab later accepts that rxe refused.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in four sentences, why their storage team can use UC for a write-only path but the same code fails the moment someone adds a remote read, and what error the fabric team should look for instead of blaming the switch.
Sources
Facts in this lesson were checked against rdma-core man pages and libibverbs verbs.h (master), 2026-09-07. Dates are when each page was fetched.
- ibv_post_send(3) - rdma-core man page · fetched 2026-09-07
- ibv_create_qp(3) - rdma-core man page · fetched 2026-09-07
- ibv_poll_cq(3) - rdma-core man page · fetched 2026-09-07
- rdma-core libibverbs/verbs.h (master) · fetched 2026-09-07
- ibv_modify_qp(3) - rdma-core man page · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- linux-rdma/perftest README · fetched 2026-09-07
- RDMA over Converged Ethernet (DOCA-Host) - generated PDF · fetched 2026-09-07
- DOCA RDMA - DOCA-Host · fetched 2026-09-07
- ibv_create_srq(3) - rdma-core man page · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.