Skip to content
nodeza.learn
RoCE course/1. Foundations/Lesson 1.3

RoCEv1 vs RoCEv2: the encapsulation

S1·E3Works in the rack, dies between racks · Customer data center, cutover night, bridge call at 02:10

S1·E3Analyze~25 minsources checked todaylab mutates hardwareverified against NVIDIA DOCA-Host RoCE documentation and the IANA port registry, 2026-09-07

Builds on: The verbs object model: device, PD, QP, CQ, MR

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Build the RoCEv1 and RoCEv2 header stacks from the outside in and state which layer makes each one routable or not.
  • Explain why a firewall or ACL for RoCEv2 must match the UDP destination port and never the source port.
  • Locate the F/R and B/R bits in the BTH and separate them from the IP ECN field that actually carries RoCEv2 congestion state.
  • Predict what disappears from a host when RoCE is disabled on a device or a virtual function.

Episode 3 — Works in the rack, dies between racks

The situation · Customer data center, cutover night, bridge call at 02:10

The maintenance window closes at four. The imaging tier came up inside every rack an hour ago and dies the moment traffic crosses one. Fourteen people are on the bridge. The security team pastes their rule into chat: UDP 4791 permitted, source and destination. The fabric team reports the sending host’s Ethernet interface counters are flat, so in their reading nothing leaves the NIC. Two pieces of evidence, both misleading. The night-shift operator has walked every cross-rack cable twice and labels the one everybody keeps blaming: SPINE 2, NOT THE PROBLEM.

RoCEv2 is UDP with a dedicated destination port, 4791, while the source port is not identity at all: it carries an opaque flow identifier so network devices can do things like ECMP while staying agnostic to the protocol header format.[1] An ACL that pins the source port permits whatever happens to match and blackholes the rest. The flat counters prove nothing either — RoCE traffic is offloaded by the hardware, never goes through the Ethernet network driver, and is counted where InfiniBand traffic is counted.[1]

The encapsulation exists for exactly this problem. RoCEv1 is a dedicated ethertype, 0x8915, with no IP header, so it lives inside one Ethernet broadcast domain; RoCEv2 replaces the GRH with a real IP header that traverses L3 routers, plus a UDP header acting as a stateless encapsulation layer for the RDMA transport.[1] That is what makes rack-to-rack possible.

Match the destination port and leave the source port alone: it is a hash, not an identity.

You ask the operator to start a capture.

1Two encapsulations, one transport

NVIDIA states the whole design in one sentence: “RoCE encapsulates IB transport in one of the following Ethernet packets: RoCEv1 - dedicated ether type (0x8915); RoCEv2 - UDP and dedicated UDP port (4791)”.[1] Everything above the encapsulation — queue pairs, opcodes, PSNs, the retry state machine — is unchanged between them. Only the wrapper differs.

RoCEv1 is a layer-2 protocol: it “is defined as RDMA over Ethernet header… It uses ethertype 0x8915 and can be used with or without the VLAN tag. The regular Ethernet MTU applies on the RoCE frame”, and it is limited to a single Ethernet broadcast domain.[1][2] RoCEv2 replaces the InfiniBand GRH with real network headers: “Instead of the GRH used in RoCE, IP routable RoCE packets carry an IP header which allows traversal of IP L3 Routers and a UDP header (RoCEv2 only) that serves as a stateless encapsulation layer for the RDMA Transport Protocol Packets over IP.”[1]

The port is not folklore. The IANA registry row reads roce,4791,udp,IP Routable RocE,[InfiniBand_Trade_Association],[Diego_Crupnicoff],2014-10-17, and TCP/4791 is marked Reserved rather than assigned.[3] The IETF draft on exporting RoCEv2 telemetry restates it: “RoCEv2 runs on top of UDP (port 4791) and inherits the transport layer protocol of the InfiniBand Architecture (IBA)”.[4] The specification itself is an IBTA annex; the annex number and year that circulate in blog posts could not be confirmed on an IBTA-hosted page, because the public specification page exposes no annex list to non-members, so treat “Annex A17, 2014” as widely repeated but unverified — the verifiable date is the 2014-10-17 IANA registration.[3]

DSCP
ToS 104 · SL 3 · PCP 3 · prio 3NVUE default for RoCE data → switch priority 3
wire order — first byte on the wire at the topEthernet MAC headerethertype = IPv4 / IPv614 BIPv4 headerDSCP 26 · ECN bits 6–720 BUDP headersport = flow hash · dport 47918 BBTH ⚠opcode · dest QP · PSN · FECN/BECN12 BRDMA payloadpath MTU 256…4096 (RC/UC)≤ MTUICRC ⚠rx_icrc_encapsulated4 BEthernet FCSrecomputed at every hop4 B
RoCEv2 frame · untagged · IPv4 GID

UDP header 8 B

Destination port 4791, fixed. IANA registry row, verbatim: roce,4791,udp,IP Routable RocE,[InfiniBand_Trade_Association],[Diego_Crupnicoff],2014-10-17. The source port is not a port at all: "the UDP source port field is used to carry an opaque flow-identifier that allows network devices to implement packet forwarding optimizations (e.g. ECMP)".

UDP is what makes RoCEv2 routable — "a stateless encapsulation layer for the RDMA Transport Protocol Packets over IP".

tcpdump -i <netdev> -nn udp port 4791 -c 20   # fixed dport, varying sport
FAE angle

The single most useful sentence for a customer with a firewall: match dport 4791 only, never sport — the source port is a per-flow hash and changes packet to packet. Second most useful: echo 0 > /sys/devices/<bdf>/roce_enable and 4791 becomes ordinary UDP again, which is how you prove a "RoCE is broken" ticket is really a NIC-config ticket. On Windows the port is a driver parameter (roce_udp_dport; default 4791 as of WinOF v5.35).

IANA service-name / port registry
Which GID picked this version

Every IP on the port creates two GID entries — same value, types IB/RoCE v1 and RoCE v2 — and each occupies two hardware entries, so the table fills twice as fast as people expect. An IPv4 GID is an IPv4-mapped IPv6 address.

A connected QP has no “set RoCEv2” knob: the address vector names a GID index, and the GID type at that index sets the version. RDMA-CM instead reads a module-global. That is why ib_write_bw -x <index> and an app using the CM can disagree on the same host.

show_gids            # DEV PORT INDEX GID IPv4 VER DEV, then n_gids_found=<count>
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/3
cma_roce_mode -d mlx5_0 -p 1        # read the RDMA-CM global
cma_roce_mode -d mlx5_0 -p 1 -m 2   # 1 = RoCEv1, 2 = RoCEv2
echo 0 > /sys/devices/<bdf>/roce_enable   # no GID tables, Raw Ethernet QPs only

show_gids ships in mlnx-tools at sbin/show_gids — not the old ofed_scripts/ path.

mlnx-tools show_gids · MLNX_OFED — RoCE

⚠ marks a number no fetched NVIDIA or IETF page states: the GRH, BTH and ICRC byte sizes (the IBTA spec is members-only) and whether a switch classifies RoCEv1 on the GRH traffic-class byte.

Toggle version between v1 and v2 and watch which headers appear and disappear. Click each row for what the field does and which tool reads it.

2Reading the frame

Build a RoCEv2 frame from the outside in: Ethernet MAC header, optionally an 802.1Q VLAN tag, an IP header and a UDP header - NVIDIA’s wording is that the IP header “allows traversal of IP L3 Routers” and the UDP header “serves as a stateless encapsulation layer for the RDMA Transport Protocol Packets over IP”[1] - and inside that the InfiniBand Base Transport Header, the RDMA transport header carrying the QP information, which RoCEv2 inherits from the InfiniBand Architecture, then the RDMA payload.[4] The frame closes with the InfiniBand invariant CRC and the Ethernet FCS. The RoCEv1 version replaces the IP and UDP headers with the GRH and sets ethertype 0x8915 on the MAC header.[1]

The UDP header is where the interesting operational behaviour lives. The destination port “unequivocally distinguishes the datagram”, while “the UDP source port field is used to carry an opaque flow-identifier that allows network devices to implement packet forwarding optimizations (e.g. ECMP) while staying agnostic to the specifics of the protocol header format”.[1] Read that twice before writing an ACL: the source port is entropy, not identity.

The BTH is the InfiniBand transport header that RoCE never changed. The IPFIX draft, which defines how to export it as telemetry, enumerates its fields: OpCode as an 8-bit value, Partition Key as 16 bits, Destination QP and Source QP as 32-bit elements with 24 effective bits, PSN with 24 effective bits, and flag groups carrying SE, M, PadCnt and TVer, an AckReq bit, and one octet the draft calls F/R and B/R - the InfiniBand FECN and BECN positions, which the draft labels FECN/Res1 and BECN/Res1 because they are reserved in RoCEv2 usage.[4] Know they exist in the header, but do not expect them to carry congestion state on a RoCE fabric: NVIDIA states that for signalling congestion RoCEv2 Congestion Management “relies on the mechanism defined in RFC3168”, the two-bit ECN field in the IP TOS octet with codepoints Not-ECT, ECT(1), ECT(0) and CE, plus the notification packet the receiver sends back - which is module 2.[11][7]

One field with no analogue in a TCP stack: the ICRC, an invariant CRC computed across the RoCE headers and payload. The in-tree mlx5 driver exposes its failures as the hardware counter rx_icrc_encapsulated.[8] NVIDIA publishes no fetchable page defining that counter, so treat the reading - corruption the Ethernet FCS did not catch, a physical-layer signal rather than a congestion one - as the field interpretation to confirm on hardware, not as a quotable definition.

3Turning RoCE off, and what that proves

RoCE is on by default: “By default, RoCE is enabled on all mlx5 devices. When RoCE is enabled, all traffic to UDP port 4791 is treated as RoCE traffic by the device.”[1] It can be turned off per PCI device with echo <0|1> > /sys/devices/{pci-bus-address}/roce_enable, and read back with cat on the same path.[1][2] The consequence is stated plainly: “Once RoCE is disabled, only Ethernet traffic will be supported. Therefore, there will be no GID tables and only Raw Ethernet QPs will be supported.”[1]

There is a legitimate production use for that switch. Disabling RoCE on a virtual function from the hypervisor saves VM memory, and it also means “a VM can have the capability of utilizing the RoCE UDP port (4791) for standard UDP traffic” — because the device no longer intercepts the port.[1] In the lab it is also the cleanest way to prove that a “RoCE is broken” ticket is really a NIC-configuration ticket: with RoCE off, GIDs vanish and every RDMA tool fails in a specific, recognisable way.

Both versions are available at once unless you intervene: “Both RoCEv1 and RoCEv2 are supported by default; the driver associates all GID indexes to RoCEv1 and RoCEv2, thus, a single entry for each RoCE version.”[1][6] Which of the two a given connection uses is decided by the GID index, which is the whole of the next lesson.

Finally, the counter trap that wastes an afternoon in every first RoCE deployment: “RoCE traffic is not shown in the associated Ethernet device’s counters since it is offloaded by the hardware and does not go through Ethernet network driver. RoCE traffic is counted in the same place where InfiniBand traffic is counted; /sys/class/infiniband/<device>/ports/<port number>/counters/.”[1]

Worked → faded → problem: prove the encapsulation with a capture

Goal: show, from your own capture, that RoCEv2 is UDP/4791 with a fixed destination port and a varying source port, and that the BTH follows the UDP header.

  1. Confirm the device-to-netdev mapping so you capture on the right interface: ibdev2netdev. Expected output form: mlx5_0 port 1 <===> eth2.[1]
  2. Start the capture on the receiver, filtered to the RoCE port: tcpdump -i eth2 -nn -X udp port 4791 -c 20. Reasoning: the destination port is well known and fixed, so a port filter is safe; a source-port filter would not be.[1]
  3. Generate traffic. Server: ib_write_bw -d mlx5_0 -s 4096 -n 2000. Client: ib_write_bw -d mlx5_0 -s 4096 -n 2000 <server_ip>. The same flags go on both sides.[9]
  4. Read the capture. Expected: every packet has destination port 4791; the source port differs between flows and is not 4791. The bytes immediately after the 8-byte UDP header are the BTH, whose first byte is the opcode.[1][4]
  5. Repeat with a datagram transport to show the encapsulation is unchanged by the transport: ib_send_bw -d mlx5_0 -c UD -s 1024. Expected: still UDP/4791, source port still varying.[9]
  6. Now prove the offload claim. Run ethtool -S eth2 | grep -E "^ *tx_packets|^ *rx_packets" before and after a run, and compare with cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data before and after. Expected: the InfiniBand counter climbs by the traffic volume while the Ethernet netdev counters barely move, because RoCE does not traverse the Ethernet driver.[1]
  7. Record the numbers. A customer who has seen this once stops filing “no traffic on the interface” tickets.

Before the window closes

How it ended

The capture settles it on one screen: the destination port is constant at 4791 and the source port is different on every flow, exactly as the opaque flow identifier is meant to be.[1] Security wildcards the source port, the cross-rack path comes up, and the InfiniBand port counters climb while the Ethernet ones stay flat, as documented.[1] The network lead copies both counter names into the notebook. The change lands with an hour to spare. What you say: “Stop reading the interface counters — RoCE is not counted there.” At 03:40 the SE sends one line from the lab: ib_write_bw now crosses the subnet boundary, and the customer’s storage client still will not.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Dell-lab ConnectX or BlueField-3 pair. Steps 1 to 5 are read-only. Step 6 is mutating and names its rollback.

  1. Pre-flight inventory, recorded before anything else: ibv_devinfo -l, ibdev2netdev, ip -br addr show <netdev>, show_gids, and lspci -D | grep -i mellanox to get the PCI bus address you will need in step 6. Save all of it to a file.
  2. Capture on the mapped netdev: tcpdump -i <netdev> -nn -X udp port 4791 -c 20.
  3. Generate with ib_write_bw -d mlx5_0 -s 4096 -n 2000 on both sides, server first. Expected: fixed destination port 4791, varying source port.
  4. Prove the offload: record ethtool -S <netdev> and cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data before and after a run. Expected: the InfiniBand counter moves by the traffic volume, the Ethernet counters barely move.
  5. Check the corruption counter while you are here: cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rx_icrc_encapsulated. Expected: zero. Non-zero means physical-layer corruption; note the value and look at optics before anything else.
  6. Mutating step — lab hosts only, never a shared or production card. Disable RoCE on the device: echo 0 > /sys/devices/<pci-bus-address>/roce_enable. Verify with cat /sys/devices/<pci-bus-address>/roce_enable (expect 0), then show_gids (expect the GID table to be empty) and re-run ib_write_bw (expect failure). Rollback, mandatory before you leave the host: echo 1 > /sys/devices/<pci-bus-address>/roce_enable, then re-run show_gids and confirm the GID entries recorded in step 1 are back, and re-run ib_write_bw to confirm traffic flows again. If the GID table does not return, reload the driver stack with /etc/init.d/openibd restart and re-check.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell network architect, in four sentences, why their existing leaf-spine ACLs break RoCEv2 traffic and what one line of the ACL has to change.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against NVIDIA DOCA-Host RoCE documentation and the IANA port registry, 2026-09-07. Dates are when each page was fetched.

  1. RDMA over Converged Ethernet (DOCA-Host) - generated PDF · fetched 2026-09-07
  2. RDMA over Converged Ethernet (DOCA-Host, HTML) · fetched 2026-09-07
  3. IANA Service Name and Transport Protocol Port Number Registry (CSV) · fetched 2026-09-07
  4. Export of RoCEv2 Base Transport Header (BTH) Information Using IPFIX (draft-lin-opsawg-ipfix-rocev2-01) · fetched 2026-09-07
  5. RoCE v2 UDP Port - NVIDIA WinOF v5.50.53000 · fetched 2026-09-07
  6. RDMA over Converged Ethernet (RoCE) - MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
  7. RFC 3168 - The Addition of Explicit Congestion Notification (ECN) to IP · fetched 2026-09-07
  8. Linux kernel - drivers/infiniband/hw/mlx5/counters.c · fetched 2026-09-07
  9. perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
  10. ibv_modify_qp(3) - rdma-core man page · fetched 2026-09-07
  11. Ethernet Network - NVIDIA WinOF-2 26.4.50010 (RoCEv2 Congestion Management) · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.