IPoIB: the management plane on an IB fabric

1A native driver, not a tunnel

IPoIB is easy to under-estimate as “IP in a tunnel over RDMA”. It is not. The Linux driver is “an implementation of the IP over InfiniBand protocol as specified by RFC 4391 and 4392… It is a ‘native’ implementation in the sense of setting the interface type to ARPHRD_INFINIBAND and the hardware address length to 20”.^[2] The link layer really is InfiniBand; only the payload is IP.

That 20-octet hardware address is [8-bit reserved][24-bit QPN][128-bit GID], and RFC 4391 recommends “that the GID formed by the combination of the IB subnet prefix and the port’s ‘Port GUID’ be included in the link-layer/hardware address”.^[1] Two consequences follow immediately. First, the address contains a queue-pair number, because delivery on InfiniBand is to a QP and not to a port. Second, ARP cannot reuse Ethernet’s parameters: RFC 4391 section 9.2 states that for IPv4 the ARP hardware type is 32 and “The length of the hardware address used in ARP packet header therefore is 20”.^[1]

Every datagram carries a small header of its own: “All IP and ARP datagrams transported over InfiniBand are prefixed by a 4-octet encapsulation header”, whose Reserved field “MUST be set to zero on send and ignored on receive”.^[1] Remember that 4 - it is the entire explanation for the MTU arithmetic two segments from now.

Why any of this matters on a fabric bought for RDMA: everything that is not an RDMA application still needs IP. On an InfiniBand-only pod that is ssh, the scheduler, monitoring, and - specifically - SHARP, which requires “Enabled IPoIB interface in compute servers in order to enable using UD multicast for result distribution”.^[7] MLNX-OS makes the same point from the switch side: “IP interface ‘ib0’ is running under the default PKEY (0x7fff) and can be used for in-band management connectivity to the system.”^[9]

2One interface per port, and children per partition

“When the IPoIB driver is loaded, it creates one interface for each port using the P_Key at index 0.”^[2] That single sentence decides your naming and your tenancy model. The base ib0 belongs to whatever partition occupies block 0, index 0 of the port’s PKey table - by default the default partition 0x7fff.^[9][5]

Additional partitions get child interfaces, created and destroyed through sysfs:

echo 0x8001 > /sys/class/net/ib0/create_child   # creates ib0.8001 with P_Key 0x8001
echo 0x8001 > /sys/class/net/ib0/delete_child   # removes it

Both lines are quoted from the kernel documentation.^[2] The pkey and parent sysfs files expose the mapping afterwards, and “Child interface create/delete can also be done using IPoIB’s rtnl_link_ops, where children created using either way behave the same”.^[2]

Which partition lands at index 0 is controlled from the subnet manager, not the host. In partitions.conf, indx0 “indicates that this pkey should be inserted in block 0 index 0”.^[5] So if a customer wants their tenant partition to be the base ib0 rather than a child, that is a one-token change in the SM’s partition file - and it changes what the default partition’s interface is called on every host at once.^[5]

# /etc/opensm/partitions.conf   —   opensm -P /etc/opensm/partitions.conf -Z both
Default=0x7fff,ipoib,defmember=full : ALL=full, SELF=full ;
tenant-a=0x7ff1,indx0,ipoib,mtu=5,defmember=full :  ;

3Datagram or connected: the MTU arithmetic

“The mode is set and read through an interface’s /sys/class/net/<intf name>/mode file.”^[2] There are two.

Datagram mode uses the IB unreliable-datagram transport, and “the interface MTU has is equal to the IB L2 MTU minus the IPoIB encapsulation header (4 bytes).”^[2] On a typical 2K fabric that is 2048 - 4 = 2044 bytes, which RFC 4391 makes the floor for every implementation: the IB components “may support maximum payloads of 256, 512, 1024, 2048, or 4096 octets… this document requires that all IPoIB implementations support an MTU of 2044 octets”.^[1]

Connected mode “takes advantage of the connected nature of the IB transport and allows an MTU up to the maximal IP packet size of 64K.”^[2] Two caveats travel with it and both matter operationally: “In connected mode, the interface’s UD QP is still used for multicast and communication with peers that don’t support connected mode”, and “Stateless offloads are supported only in datagram mode.”^[2] So connected mode is a large-message optimisation that keeps a datagram path underneath it, and it costs you the offloads.

NVIDIA’s DOCA page adds the numbers a Dell customer will actually hit: IPoIB supports “Varies MTU values: up to 4k in Datagram mode”, the physical port MTU defaults to 4k while “the IPoIB logical MTU defaults to 2k”, and in OpenSM partition files you use “mtu=5 for 4k and mtu=4 for 2k”.^[3] The same page describes Enhanced IPoIB, which “enables offloading ULP basic capabilities to a lower vendor specific driver, in order to optimize IPoIB data path” with stateless offloads, multiple queues and interrupt moderation - and is UD mode only.^[3]

The trap in one line: the partition file’s mtu= default is 4, meaning 2048, so a host configured for 4092 or 65520 will look up and behave broken until someone compares the two.^[5][3]

4The partition side: where a working IPoIB interface actually comes from

An IPoIB interface needs a multicast broadcast group to exist before ARP can work, and that group is created by the subnet manager, not by the host. In partitions.conf the ipoib flag “indicates that this partition may be used for IPoIB, as a result the IPoIB broadcast group will be created with the mgroup_flag flags given, if any”.^[5] The flags default to rate=3 (10 Gbps), mtu=4 (2048), sl=0, scope=2 (link local), Q_Key=0x0b1b for IP groups, TClass=0 and FlowLabel=0.^[5] One of them carries an explicit warning: “WARNING: changing this [Q_Key] for the broadcast group may break IPoIB on client nodes!!!”^[5]

Membership is the second half. P_Key bit 15 is the membership bit - “‘1’ means full member, ‘0’ means limited member” - and “A P_Key match occurs only if at least one of the nodes (sending / receiving) is a full member”.^[6] RFC 4391 turns that into a requirement for IPoIB specifically: “a ‘Full Membership’ P_Key (high-order bit is set to 1) MUST be used so that all members may communicate with one another”, and every IPoIB interface “MUST ‘FullMember’ join the IB multicast group defined by the broadcast-GID”.^[1]

Now put it beside the partition-file default: defmember “specifies default membership for port guid list. Default is limited.”^[5] So a stanza written without =full produces limited members, the broadcast group cannot be full-member joined by them, and the interface comes up and resolves nothing. Two limited members of the same partition also cannot reach each other - that follows from the match rule quoted above, though note that the OpenSM man page never spells out the limited-to-limited case verbatim; the closest primary statement is RFC 4391’s full-membership requirement.^[6][1]

The canonical known-good one-liner is worth keeping in a snippet file: Default=0x7fff,ipoib:ALL=full; - “equivalent to how OpenSM used to run prior to the partition manager”.^[5] And on the QoS side, OpenSM matches IPoIB traffic by PKey: ipoib :, ipoib, pkey 0x7fff : and any, pkey 0x7fff : are equivalent match rules.^[8]

Default=0x7fff, ipoib : ALL=full ;

tenant-a=0x8001, ipoib, mtu=5, defmember=full : 0x0002c90300a1b2c1, 0x0002c90300a1b2c2, 0x0002c90300a1b2c3 ;

echo 0x8001 > /sys/class/net/ib0/create_child
cat /sys/class/net/ib0.8001/pkey
ip -d link show ib0.8001

Default=0x7fff, ipoib : ALL=____ ;
tenant-b=0x8002, ______, ______, defmember=______ : 0x…d1, 0x…d2 ;

storage=0x7ff2 : 0x0002c90300b00001, 0x0002c90300b00002 ;
compute=0x7ff3, ipoib : ALL_CAS ;