Skip to content
nodeza.learn
RoCE course/1. Foundations/Lesson 1.4

GIDs: how a connection picks its RoCE version

S1·E4The benchmark crosses the subnet and the application does not · Hotel lobby, 07:20, ninety minutes before the go/no-go

S1·E4Apply~30 minsources checked todaylab mutates hardwareverified against NVIDIA DOCA-Host RoCE page, mlnx-tools show_gids (master) and the Linux sysfs-class-infiniband ABI, all re-fetched 2026-09-07

Builds on: RoCEv1 vs RoCEv2: the encapsulation

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Read a GID table from `show_gids` or from sysfs and say which entry is RoCEv2 on which netdev.
  • Determine which RoCE version a given connection is using and name the mechanism that chose it.
  • Apply `cma_roce_mode` and perftest `-x` correctly to force a version on each of the two connection paths.
  • Explain why an application using RDMA-CM and a perftest run on the same host can disagree about the RoCE version.

Episode 4 — The benchmark crosses the subnet and the application does not

The situation · Hotel lobby, 07:20, ninety minutes before the go/no-go

Lobby coffee, laptop on your knees, and the SE’s 03:40 message still open. On the same pair of hosts, ib_write_bw runs clean across the subnet boundary and the customer’s storage client will not. At nine the room decides whether the archive cutover holds, and the sentence they are already rehearsing is “turn on RoCEv2 for storage”.

There is nothing to turn on. For a connected queue pair the version is not a flag: moving an RC or UC queue pair from INIT to RTR requires an address vector, the address vector names an index into the port’s GID table, and the GID type at that index sets the RoCE type of the queue pair.[1] perftest without -R does its own out-of-band exchange and lets an engineer pin that index with -x.[7] The storage client does not: it uses RDMA-CM, which decides the source GID itself and breaks the tie between the two entries with a global value of the RDMA-CM module, read and written with cma_roce_mode.[1] One benchmark and one application on one host, legitimately speaking different versions.

The table looks the way it does because both versions ship at once: every IP address configured on the port creates a pair of GID entries with the same value and different types.[1] Something has to choose between them, and which mechanism chooses depends entirely on how the connection was made.

Nobody enables RoCEv2. Somebody picks a GID index — find out who.

Ninety minutes, two hosts, one GID table. Start with where the entries come from.

1Where GIDs come from

A GID table entry is not something an administrator creates directly. “GID table entries are created whenever an IP address is configured on one of the Ethernet devices of the NIC’s ports”, and each entry carries three things: a GID value, a GID type and a network device.[1] Because both RoCE versions are supported at once, “the GID table is occupied with two GIDs, both with the same GID value but with different types”.[1] Re-verified against the current NVIDIA page at write time: each RoCE-capable port receives two GID entries per address, differing only in type.[2]

Address family shows up in the value: “The GID format can be of 2 types; IPv4 and IPv6. IPv4 GID is an IPv4-mapped IPv6 address, while IPv6 GID is the IPv6 address itself.”[1] And the family plus the version together decide the layer-3 header: “Layer 3 header for packets associated with IPv4 GIDs will be IPv4 (for RoCEv2) and IPv6/GRH for packets associated with IPv6 GIDs and IPv4 GIDs for RoCEv1.”[1]

Two capacity facts belong together. Each IP produces a pair, because both versions are supported at once - so the entry count is twice the address count, not equal to it.[1][3] Older MLNX_OFED builds added that in RoCEv1/RoCEv2 mode each GID-table entry consumes two entries in hardware as well; the current NVIDIA page no longer carries that sentence, so treat hardware GID-table capacity as version-dependent and confirm it against the customer’s firmware rather than quoting a number.[3] A host with a base interface, three VLAN sub-interfaces and both address families still burns through the table faster than a naive count suggests.

DSCP
ToS 104 · SL 3 · PCP 3 · prio 3NVUE default for RoCE data → switch priority 3
wire order — first byte on the wire at the topEthernet MAC headerethertype = IPv4 / IPv614 B802.1Q VLAN tagPCP 3 · DEI · VID4 BIPv4 headerDSCP 26 · ECN bits 6–720 BUDP headersport = flow hash · dport 47918 BBTH ⚠opcode · dest QP · PSN · FECN/BECN12 BRDMA payloadpath MTU 256…4096 (RC/UC)≤ MTUICRC ⚠rx_icrc_encapsulated4 BEthernet FCSrecomputed at every hop4 B
RoCEv2 frame · VLAN tagged · IPv4 GID

IP header 20 B / 40 B

IPv4 (20 B) for IPv4 GIDs, IPv6 (40 B) for IPv6 GIDs: "Layer 3 header for packets associated with IPv4 GIDs will be IPv4 (for RoCEv2)." The ToS octet (IPv6: Traffic Class) is split 6 + 2: DSCP in the top six bits, ECN in "bits 6 and 7 of the IPv4 TOS octet" — Not-ECT 00, ECT(1) 01, ECT(0) 10, CE 11 (RFC 3168 §5).

This is the byte the fabric reads under trust dscp, and the byte the congested switch rewrites to CE. ToS = DSCP × 4 — "to set DSCP value of 24, set the ToS bit to 96 (24x4=96)".

ib_write_bw --tclass=104          # DSCP 26 on the AV path
tcpdump -i <netdev> -nn -v udp port 4791   # read the DSCP back
FAE angle

The ECN bits here are the IP-layer signal. They are NOT the BTH FECN/BECN bits one header down — a distinction that decides which counter you go read.

RFC 3168 (ECN)
Which GID picked this version

Every IP on the port creates two GID entries — same value, types IB/RoCE v1 and RoCE v2 — and each occupies two hardware entries, so the table fills twice as fast as people expect. An IPv4 GID is an IPv4-mapped IPv6 address.

A connected QP has no “set RoCEv2” knob: the address vector names a GID index, and the GID type at that index sets the version. RDMA-CM instead reads a module-global. That is why ib_write_bw -x <index> and an app using the CM can disagree on the same host.

show_gids            # DEV PORT INDEX GID IPv4 VER DEV, then n_gids_found=<count>
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/3
cma_roce_mode -d mlx5_0 -p 1        # read the RDMA-CM global
cma_roce_mode -d mlx5_0 -p 1 -m 2   # 1 = RoCEv1, 2 = RoCEv2
echo 0 > /sys/devices/<bdf>/roce_enable   # no GID tables, Raw Ethernet QPs only

show_gids ships in mlnx-tools at sbin/show_gids — not the old ofed_scripts/ path.

mlnx-tools show_gids · MLNX_OFED — RoCE

⚠ marks a number no fetched NVIDIA or IETF page states: the GRH, BTH and ICRC byte sizes (the IBTA spec is members-only) and whether a switch classifies RoCEv1 on the GRH traffic-class byte.

Switch gidFamily between ipv4 and ipv6, and version between v1 and v2, and watch which layer-3 header the frame gets. That mapping is the content of this segment.

2Reading the table by hand

show_gids is the convenience script, and it is worth knowing what it does because on a stripped host you will have to do it yourself. Re-verified against the current source: it walks /sys/class/infiniband/*/ports/*/gids/*, skips the all-zero GID and the bare fe80:0000:0000:0000:0000:0000:0000:0000 entry, reads the type from gid_attrs/types/<i> and the netdev from gid_attrs/ndevs/<i>, derives the dotted-quad from the IPv4-mapped bytes, prints the header DEV PORT INDEX GID IPv4 VER DEV, and finishes with n_gids_found=<count>.[4] It takes an optional device-name argument and enumerates everything under /sys/class/infiniband/ without one.[4] It ships in mlnx-tools at sbin/show_gids, not the ofed_scripts/ path older documentation still points at.[6][4]

The kernel ABI is the authority for what those files contain, and it is short enough to memorise. gid_attrs/types/<gid-index> is “either ‘IB/RoCE v1’ for IB and RoCE v1 based GIDs or ‘RoCE v2’ for RoCE v2 based GIDs”; gid_attrs/ndevs/<gid-index> is “The net-device’s name associated with the GID”.[5] That is the whole contract: value, type, netdev.

NVIDIA’s own published table for mlx5_0 port 1 shows the pattern: index 0 is a link-local value with type v1 on p4p1, index 1 is the same value with type v2, index 2 is ::ffff:0a0a:0a01 (that is 10.10.10.1) with type v1, index 3 the same value with type v2, and mlx5_1 then starts again at index 0.[1][3]

3Which version does this connection use?

For a connected queue pair there is no version switch. NVIDIA states the mechanism: “To modify an RC/UC QP (connected QP) from INIT to RTR, an Address Vector (AV) must be given. The AV, among other attributes, should specify the index of the port’s GID table for the source GID of the QP. The GID type in that index will be used to set the RoCE type of the QP.”[1] Re-verified at write time on the current HTML page: the GID type at the specified index determines the RoCE mode for the QP.[2] IBV_QP_AV is one of the required attributes for the INIT to RTR transition on RC and UC, and it does not exist for UD.[8] So the rule to teach is: you do not set RoCEv2 — you pick a GID index, and the index picks the version.

RDMA-CM works differently, and the difference is the source of an entire class of tickets. “RDMA_CM interface requires only the active side of the peer to pass the IP address of the passive side. The RDMA_CM decides upon the source GID to be used and obtains it from the GID table. Since more than one instance of the GID value is possible, the lookup should be also according to the GID type. The type to use for the lookup is defined as a global value of the RDMA_CM module.”[1] That global is read with cma_roce_mode -d <dev> -p <port> and written with cma_roce_mode -d <dev> -p <port> -m <1|2>, where 1 is RoCEv1 and 2 is RoCEv2 — re-verified on the current page at write time.[1][2]

perftest sits on both sides of that fence, which is what makes it a good teaching instrument. Without -R it does an out-of-band exchange and lets you pin the index with -x, --gid-index=<index>; with -R, --rdma_cm it “connect[s] QPs with rdma_cm and run[s] test on those QPs” and therefore inherits whatever the CM global says.[7] Two runs on one host, minutes apart, can legitimately use different RoCE versions.

The same split appears one layer up in NCCL. NCCL_IB_GID_INDEX “Defines Global ID index used in RoCE mode” with a default of -1, but NVIDIA’s current troubleshooting guidance is explicit: on NCCL 2.21 and later the GID index is selected dynamically and you should not set the variable; only earlier versions needed it set by hand after running show_gids.[9][10] A runbook that still hard-codes an index is a runbook written for an older NCCL.

Worked → faded → problem: prove which version a connection is using

Goal: determine, with evidence rather than configuration files, which RoCE version each of two connection paths is using on one pair of hosts.

  1. Read the table on both hosts: show_gids. Expected: pairs of rows sharing a GID value, with the VER column reading v1 and v2, and the final n_gids_found=<count> line. Record the index of the v2 entry whose IPv4 column matches the address you will connect over, and the index of its v1 twin.[4]
  2. Cross-check one index by hand so you can do it on a host with no mlnx-tools: cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/3 should print RoCE v2, and cat .../gid_attrs/ndevs/3 should print the netdev.[5]
  3. Read what RDMA-CM would choose: cma_roce_mode -d mlx5_0 -p 1. Record the value.[1]
  4. Start a capture on the mapped netdev in a second shell: tcpdump -i <netdev> -nn -c 20 "udp port 4791 or ether proto 0x8915". This filter catches both versions, which is the point.[1]
  5. Out-of-band path, pinned to the v2 index. Server: ib_write_bw -d mlx5_0 -x <v2 index> -s 4096 -n 2000. Client: same flags plus the server IP. Expected in the capture: UDP packets to port 4791.[7]
  6. Out-of-band path, pinned to the v1 index. Re-run with -x <v1 index> on both sides. Expected in the capture: frames with ethertype 0x8915 and no UDP header.[1]
  7. CM path. Run ib_write_bw -d mlx5_0 -R -s 4096 -n 2000 on both sides. Expected: whichever encapsulation matches the cma_roce_mode value you recorded in step 3, and note that -x is not what decided it here.[7]
  8. Write the three results as a table: path, index or CM mode, and observed encapsulation. That table is the artifact you would show a customer.

Nine o'clock, with evidence

How it ended

You walk in with three artefacts instead of an opinion: show_gids from both hosts, the output of cma_roce_mode -d mlx5_0 -p 1, and a capture filtered to catch both encapsulations at once.[4][1] The CM global was set to 1, so every CM-based application on that device was speaking RoCEv1 and stopping at the first router, while the benchmark kept working because it names its own GID index.[1] One command changes it, and the change stops exactly there. What you say: “You do not turn RoCEv2 on. You choose a GID, and for this client the chooser is cma_roce_mode.” The archive gets its go. The fabric is handed over on Friday, and on Monday the training job starts dying after forty minutes.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Dell-lab ConnectX or BlueField-3 pair. Steps 1 to 5 are read-only; step 6 is mutating and names its rollback.

  1. Pre-flight inventory: ibv_devinfo -l, ibdev2netdev, ip -br addr, show_gids, and cma_roce_mode -d mlx5_0 -p 1. Save the output; the last value is what you must restore later.
  2. Verify one row by hand against sysfs: cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/<i> and .../gid_attrs/ndevs/<i> for the index you plan to use. Expected: RoCE v2 and the netdev name from step 1.
  3. Start the dual-version capture: tcpdump -i <netdev> -nn -c 20 "udp port 4791 or ether proto 0x8915".
  4. Out-of-band, pinned: ib_write_bw -d mlx5_0 -x <v2 index> -s 4096 -n 2000 on both sides. Expected: UDP/4791 in the capture. Repeat with the v1 index and expect ethertype 0x8915.
  5. CM path: ib_write_bw -d mlx5_0 -R -s 4096 -n 2000 on both sides. Expected: the encapsulation implied by the cma_roce_mode value recorded in step 1, regardless of any index you used earlier.
  6. Mutating step — lab hosts only. Change the CM default: cma_roce_mode -d mlx5_0 -p 1 -m 1. Verify with cma_roce_mode -d mlx5_0 -p 1, re-run the -R test from step 5 and confirm the capture now shows ethertype 0x8915. Rollback, mandatory: cma_roce_mode -d mlx5_0 -p 1 -m 2 (or whatever value step 1 recorded), verify by reading it back, and re-run step 5 to confirm the original behaviour returns. Leaving a host on RoCEv1 breaks every CM-based application that crosses a subnet.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why their benchmark and their application can be running different RoCE versions on the same card at the same time, and the two commands that show which is which.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against NVIDIA DOCA-Host RoCE page, mlnx-tools show_gids (master) and the Linux sysfs-class-infiniband ABI, all re-fetched 2026-09-07. Dates are when each page was fetched.

  1. RDMA over Converged Ethernet (DOCA-Host) - generated PDF · fetched 2026-09-07
  2. RDMA over Converged Ethernet (DOCA-Host, HTML) - re-verified at write time · fetched 2026-09-07
  3. RDMA over Converged Ethernet (RoCE) - MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
  4. mlnx-tools sbin/show_gids (source) - re-verified at write time · fetched 2026-09-07
  5. Linux kernel ABI: sysfs-class-infiniband (stable) - re-verified at write time · fetched 2026-09-07
  6. GitHub API - Mellanox/mlnx-tools master tree · fetched 2026-09-07
  7. perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
  8. ibv_modify_qp(3) - rdma-core man page · fetched 2026-09-07
  9. Environment Variables - NCCL 2.31.2 documentation · fetched 2026-09-07
  10. Networking Troubleshooting - NCCL 2.31.2 documentation · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.