Partitions and PKeys: the most common mistake in IB

1What a P_Key is, and the bit that decides everything

NVIDIA frames partitions as the IB analogue of VLANs: “InfiniBand (IB) provides network isolation through network partitions, like Ethernet VLANs (802.1Q).”^[2] The mechanism is a 16-bit Partition Key carried in the packet’s BTH — “The P_Key is embedded in the packet’s BTH (L4 header) and is enforced by both the HCA and the switch” — and it applies to “all IB packets, except for SMP MADs.”^[2] That exception is why a fabric with a broken partition design still answers smpquery and ibdiagnet while applications hang.

The layout is where partitions stop resembling VLANs. “15 LSB (P_Key[14:0]): This field defines the P_KEY value… 1 MSB (P_Key[15]): This field defines the membership type. ‘1’ means full member, ‘0’ means limited member.”^[2] So 0xFFFF and 0x7FFF are the same partition at two membership levels — “The default P_Key value is 0xFFFF/0x7FFF, depending on the membership type.”^[2] OpenSM’s config file takes only the value half: “PKey — P_Key value for this partition. Only low 15 bits will be used. When omitted will be autogenerated.”^[1]

And here is the rule that produces most of the tickets: “A P_Key match occurs only if at least one of the nodes (sending / receiving) is a full member. Full members can send packets to both full and limited members; limited members can only send packets to full members.”^[2] Two limited members of the same partition therefore cannot exchange traffic at all. Worth knowing where that fact lives: the opensm man page never states it, and only NVIDIA’s own security page spells it out — so cite that page, not the man page, when a customer asks you to prove it.^[2][1]

2The asymmetry that breaks working clusters

OpenSM always has a default partition: “The default partition will be created by OpenSM unconditionally even when partition configuration file does not exist or cannot be accessed. The default partition has P_Key value 0x7fff. OpenSM’s port will always have full membership in default partition.”^[1]

The trap is what happens to everyone else, and the man page states it in one sentence. All other end ports “will have full membership if the partition configuration file is not found or cannot be accessed, or limited membership if the file exists and can be accessed but there is no rule for the Default partition.”^[1] Written as the two implicit rules it means:^[1]

# no /etc/opensm/partitions.conf at all  ->  behaves as:
Default=0x7fff : ALL=full ;

# a partitions.conf exists with no Default rule  ->  behaves as:
Default=0x7fff : ALL=limited, SELF=full ;

Combine that with the match rule from segment 1 and you have the failure in full. Every compute node is now a limited member of 0x7fff; limited members can only talk to full members; the only full member is the SM’s own port. The fabric is Active everywhere, ibstat is clean, ibdiagnet runs — and MPI hangs. Nothing looks broken because nothing is broken at the link layer.

The grammar you need to avoid it is short. A definition is [PartitionName][=PKey][,indx0][,ipoib_bc_flags][,defmember=full|limited] followed by a colon, a port-GUID list, and a semicolon.^[1] defmember=full|limited|both “specifies default membership for port guid list. Default is limited”, and a port written as <PortGUID>[=[full|limited|both]] with the suffix “omitted (or unrecognized)” is assumed limited — so a typo demotes a node just as effectively as an omission.^[1] The keywords are ALL, ALL_CAS, ALL_SWITCHES, ALL_ROUTERS and SELF, the last meaning the subnet manager’s port, and “Empty list means no ports in this partition.”^[1]

Two more rules that bite on copy-paste. “PartitionName does not need to be unique, PKey does need to be unique. If PKey is repeated then those partition configurations will be merged and first PartitionName will be used.”^[1] And the man page gives the safe first line outright: “The following rule is equivalent to how OpenSM used to run prior to the partition manager: Default=0x7fff,ipoib:ALL=full;”.^[1]

# /etc/opensm/partitions.conf   —   opensm -P /etc/opensm/partitions.conf -Z both
tenant-a=0x7ff1,indx0,ipoib : 0x0002c90300a1b201, 0x0002c90300a1b301 ;

3Designing two tenants, and what NVIDIA recommends

The multi-tenant answer is three decisions, and they follow directly from the match rule.

First, keep 0x7fff as the management partition but demote the tenants in it. NVIDIA’s own hardening guidance is to “Define all end nodes as limited members on the default PKEY” with default_membership = limited.^[4] Management stays reachable because the management node is a full member; tenants cannot reach each other over the default partition because two limited members never match.^[2]

Second, give each tenant its own PKey with full membership inside it, so that tenant nodes can talk to their own peers:^[1]

Default=0x7fff, ipoib : ALL=limited, SELF=full ;
tenant-a=0x7ff1, ipoib, indx0, defmember=full : 0x0002c90300a1b201, 0x0002c90300a1b301 ;
tenant-b=0x7ff2, ipoib, defmember=full : 0x0002c90300c7d101 ;
storage=0x7ff8, defmember=limited : 0x0002c90300a1b201, 0x0002c90300c7d101, 0x0002c90300b4c101=full ;

Read the last line carefully: the storage target is the only full member, every client is limited, so clients reach storage and not each other. That is the hub-and-spoke pattern the membership bit was designed for.

Third, make the switch do the enforcement rather than trusting the hosts. -Z/--part_enforce [both | in | out | off] sets “the partition enforcement type (for switches)… Default is both”, and UFM ships part_enforce = both (default- outbound and inbound) marked read-only.^[1][11] Leave it at both. Related and off by default: -W/--allow_both_pkeys decides “whether both full and limited membership on the same partition can be configured in the PKeyTable. Default is not to allow both pkeys.”^[1]

Two side effects to plan for. If a partition will carry IPoIB it needs the ipoib flag, which “indicates that this partition may be used for IPoIB, as a result the IPoIB broadcast group will be created with the mgroup_flag flags given, if any” — and the group’s defaults are rate=3, mtu=4 (2048), sl=0, scope=2, Q_Key=0x0b1b for IP groups, with an explicit “WARNING: changing this [Q_Key] for the broadcast group may break IPoIB on client nodes!!”^[1] And RFC 4391 requires that “a ‘Full Membership’ P_Key (high-order bit is set to 1) MUST be used so that all members may communicate with one another”, with every IPoIB interface FullMember-joining the broadcast group.^[8] A tenant partition of limited members with the ipoib flag therefore produces an interface that comes up and never resolves ARP.^[8][9]

On the switch side the same objects exist in MLNX-OS — ib partition my-partition pkey 0x7ff2, then partition my-partition and member all, inspected with show ib partition — with the caveat that “The partitions configuration is applicable and to be used only when the SM is enabled and running on the system.”^[5] The UFM appliance CLI exposes the management partition’s default membership as ib partition management defmember <full|limited>, and “It is not possible to modify the defmember in case OpenSM or UFM are running.”^[10]

4Verify against the fabric, never against the file

A partition design is only real once the SM has programmed it, and there are three read-back paths.

Per port, from the SMA and with no SA involvement, smpquery takes an operation name and a LID and port: smpquery PKeys <lid> <port> returns that port’s PKey table, in the same family as smpquery SL2VL and smpquery VLArb.^[6] This is the narrowest possible check and the one to run on the one node that cannot reach its peer.

Fabric-wide, ibdiagnet writes ibdiagnet2.pkey — “pkey tables” — alongside ibdiagnet2.slvl and ibdiagnet2.sm.^[7] Ask a customer for that file rather than for their partitions.conf, because the file states intent and the dump states outcome, and the whole class of bug in this lesson lives in the gap between the two.

On the host, IPoIB exposes what it is actually using: the driver “creates one interface for each port using the P_Key at index 0”, child interfaces are created with echo 0x8001 > /sys/class/net/ib0/create_child producing ib0.8001, and /sys/class/net/ib0/pkey reports the key in force.^[12] NVIDIA’s DOCA page documents the same mechanism in shorthand — echo 1 also yields ib0.8001, because “the actual PKey used is a 16-bit number with the most significant bit set”.^[9] That is where indx0 connects to something visible: the partition marked indx0 is the one the base interface picks up.^[1][12]