The verbs object model: device, PD, QP, CQ, MR

1The device and its port, read two ways

Everything starts with a device context. ibv_devinfo is the tool that prints what the driver knows about it, with the synopsis ibv_devinfo [-d device] [-i port] [-l] [-v]; -d/--ib-dev=DEVICE selects one device, -i/--ib-port=PORT one port, -l/--list prints only the device names, and -v/--verbose prints “all available information about RDMA devices”.^[1] On a RoCE port the block looks like this.^[2]

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         16.28.0578
        node_guid:                      ...
        vendor_id:                      0x02c9
        vendor_part_id:                 4121
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

Two lines in that block cause most of the confusion in a first RoCE ticket. transport: InfiniBand (0) on an Ethernet port is normal — RoCE keeps the InfiniBand transport and only swaps the link; link_layer is the field that reports the medium.^[2] And sm_lid/port_lid are zero because “LID is a layer 2 attribute of the InfiniBand protocol stack, it is not set for a port and is displayed as zero when querying the port”.^[2] That follows from a bigger fact: for RoCE “the presence of a Subnet Manager (SM) is not required in the fabric”, which also means path queries are impossible and the path record must be filled in by hand or by RDMA-CM.^[2]

The same facts are readable without the tool: cat /sys/class/infiniband/mlx5_0/ports/1/state returns ACTIVE and cat /sys/class/infiniband/mlx5_0/ports/1/link_layer returns Ethernet, while the firmware version sits one level up, at cat /sys/class/infiniband/mlx5_0/fw_ver.^[2] Keep both routes in your hands — on a customer box where mlnx-tools is missing, sysfs still answers.

2Protection domain: the scope that makes a key mean something

A protection domain is not an object that does work — it is the boundary that makes the other objects safe. MRs, QPs, SRQs and address handles are all allocated inside a PD, and a key registered in one PD is meaningless to a QP in another.^[6][3] The API is blunt about this in one place people trip over: ibv_advise_mr() takes the PD as its first argument, not the MR, because the advice applies to a scatter-gather list of keys that all live in that domain.^[7]

Above the PD sits the extended device query, ibv_query_device_ex(), whose struct ibv_device_attr_ex carries orig_attr, comp_mask, odp_caps, completion_timestamp_mask, hca_core_clock, device_cap_flags_ex, tso_caps, rss_caps, max_wq_type_rq, packet_pacing_caps, raw_packet_caps, tm_caps, cq_mod_caps, max_dm_size, atomic_caps, xrc_odp_caps and phys_port_cnt_ex.^[8] That structure is where you check whether a feature a customer is asking for exists on their firmware at all, rather than arguing about it.

ibv_alloc_pd(context)

3QP and CQ: two queues, one completion ring

A queue pair is a send queue plus a receive queue. It is created from struct ibv_qp_init_attr, which is qp_context, send_cq, recv_cq, srq, a cap block, qp_type and sq_sig_all.^[3] The cap block is max_send_wr, max_recv_wr, max_send_sge, max_recv_sge and max_inline_data.^[3] Upstream verbs offers IBV_QPT_RC, IBV_QPT_UC, IBV_QPT_UD, IBV_QPT_RAW_PACKET and the vendor-specific IBV_QPT_DRIVER; the next lesson is about what each of them can actually do.^[3] perftest exposes the same choice as -c, --connection=<RC/UC/UD/XRC/DC/SRD> with a default of RC, which is the fastest way to demonstrate the matrix without writing code.^[12]

sq_sig_all decides how noisy the CQ is: “If set, each Work Request (WR) submitted to the SQ generates a completion entry”; unset it and only WRs flagged IBV_SEND_SIGNALED complete.^[3] Attach a shared receive queue and two rules bite: creation “fails if attempting a QP type other than RC or UD with SRQ attachment”, and max_recv_wr/max_recv_sge are ignored because the SRQ owns them.^[3]

The completion queue has its own surprises. ibv_create_cq_ex() “may create a CQ with size greater than or equal to the requested size” — read cqe back from the returned CQ instead of assuming.^[4] The channel may be NULL if you will not use completion events, and comp_vector “must be >= 0 and < context->num_comp_vectors”.^[4] An SRQ is created from struct ibv_srq_init_attr holding max_wr, max_sge and srq_limit, where srq_limit “is irrelevant for ibv_create_srq” and is armed later with ibv_modify_srq; both SRQ and CQ may return larger values than requested.^[5]

Teardown has dependencies you should be able to recite: ibv_destroy_srq() fails while any QP is still associated with it, and ibv_destroy_qp() fails if the QP is still in a multicast group.^[5][3]

4Memory registration: lkey, rkey, and what registration costs

ibv_reg_mr(pd, addr, length, access) pins a range of memory and gives the HCA a translation for it; ibv_reg_mr_iova() is the same with an explicit HCA virtual address.^[6] The access flags are IBV_ACCESS_LOCAL_WRITE, REMOTE_WRITE, REMOTE_READ, REMOTE_ATOMIC, FLUSH_GLOBAL, FLUSH_PERSISTENT, MW_BIND, ZERO_BASED, ON_DEMAND, HUGETLB and RELAXED_ORDERING.^[6] Two rules matter more than the list: “If IBV_ACCESS_REMOTE_WRITE or IBV_ACCESS_REMOTE_ATOMIC is set, then IBV_ACCESS_LOCAL_WRITE must be set too”, and “Local read access is always enabled for the MR” — there is no IBV_ACCESS_LOCAL_READ.^[6]

Registration returns two keys. The lkey goes in the local QP’s scatter-gather entries; the rkey is what you hand the peer so it can do one-sided reads, writes and atomics against that range.^[6] ibv_dereg_mr() fails while a memory window is still bound to the MR.^[6]

Registration cost is page-pinning cost, which is why on-demand paging exists. NVIDIA describes ODP as “a technique to alleviate much of the shortcomings of memory registration”: an explicit ODP MR “does not need to have valid mappings at registration time”, and an implicit one gives the application “a special memory key that represents their complete address space”.^[10] The implicit recipe from the man page is IBV_ACCESS_ON_DEMAND with addr 0 and length SIZE_MAX.^[6] ODP moves the cost to first touch, so ibv_advise_mr(pd, IBV_ADVISE_MR_ADVICE_PREFETCH, IBV_ADVISE_MR_FLAG_FLUSH, sg_list, num_sge) exists to pay it early; with IBV_ADVISE_MR_FLAG_FLUSH the call is synchronous and the pages are guaranteed present in the HCA on success, without it the call is best-effort.^[7] NVIDIA notes one flat limitation: “ODP does not support contiguous pages.”^[10]

Finally, keep the vocabulary aligned with the DOCA side of the house, because Dell customers will mix them in one sentence: DOCA RDMA supports InfiniBand and “Ethernet using RoCE”, and on the BlueField platform the doca_dev handed to DOCA RDMA must be a scalable function - with one exception, the DPA datapath, which currently supports PFs only.^[11]

ibv_reg_mr(pd, addr, length, access)   # or ibv_reg_mr_iova(pd, addr, length, hca_va, access)