LIDs, GIDs, GUIDs and LMC

1Three identifiers, three lifetimes

RFC 4392 gives all three definitions in a few lines, and they are worth quoting exactly because everything else follows from them. “A port, in a data packet, is identified by a Local Identifier (LID) and optionally a Global Identifier (GID).”^[1] “The LID is a 16-bit value that is assigned when the port becomes active.”^[1] “The GID is 128 bits long and is formed by the concatenation of a 64-bit IB subnet prefix and a 64-bit EUI-64-compliant portion. The EUI-64 portion of a GID is referred to as the Global Unique Identifier (GUID)”.^[1]

Then the sentence that decides how you write scripts for the next ten years: “The GUID is the only persistent identifier of a port. However, it cannot be used as an address in a packet.”^[1] And its companion: “The subnet manager may attempt to keep the LID values constant across reboots, but that is not a requirement.”^[1]

When each one is on the wire is also specified: “The GID in the packet is needed only when communicating across an IB subnet, though it may always be included… A LID is always needed in a packet since the destination within a subnet is determined by it.”^[1] So inside one subnet - which is every fabric in this module - the LID is doing the steering, and the GID is optional context.

NVIDIA restates the GUID as a security property rather than an addressing one: “Each node and port are identified through a 64-bit Global Unique Identifier (GUID) maintained by hardware, which helps restrict impersonation or spoofing.”^[3] That is the reason every fabric-configuration file keys on GUIDs - root-GUID files, compute-node GUID files and partitions.conf port lists all take GUIDs, never LIDs.^[8][2]

One honest gap: you will see LID range tables in the wild - unicast up to 0xBFFF, multicast above 0xC000, 0xFFFF permissive. No primary page fetched for this course states those boundaries; RFC 4392 says only that the LID is a 16-bit value.^[1] Treat the ranges as folklore until you can cite the IBA specification itself, and never put them in a customer answer as if NVIDIA published them.

2The GID in detail: prefix, scope and multicast

The subnet prefix is a knob, not a constant. OpenSM’s default is 0xfe80000000000000, settable with --subnet_prefix; UFM ships the same value as subnet_prefix = 0xfe80000000000000, marked read-write.^[2][5] Because that is the link-local prefix, a bare GID read off two different fabrics looks identical up to the GUID - which is a mild trap when you are comparing captures from two customer sites.^[2]

Unicast GID scopes mirror IPv6: link-local FE80/10, which is never forwarded past the IB subnet; site-local FEC0/10; and global.^[1] Multicast GIDs have their own shape: FFxy:<112 bits>, where the x nibble is the flags field 000T - T=0 is a permanently assigned or well-known group and T=1 a transient one - and the y nibble is the scope, with 2 = link-local, 5 = site-local, 8 = organization-local and 0xE = global.^[1] You will meet those numbers again in lesson 4, because IPoIB’s broadcast group is a multicast group with a scope and a rate.^[8]

Reading identifiers on a live fabric splits into two routes. From the SMA, smpquery fetches attributes directly: Nodeinfo (NI), Nodedesc (ND), Portinfo (PI), GUIDInfo (GI) and more, addressed by LID (smpquery portinfo 3 1), by GUID with -G, or by direct route with -D.^[6] From the local driver, the stable kernel sysfs ABI exposes ports/<port-num>/lid (“Port LID”), ports/<port-num>/lid_mask_count (“Port LID mask count”), ports/<port-num>/sm_lid (“Subnet manager LID for port’s subnet”), plus device-level node_guid and sys_image_guid.^[7] The whole infiniband-diags suite shares the same addressing flags: -D for directed route, -G by GUID, -L by LID, -s for the SM LID.^[9]

ibnetdiscover -p          # every port: LID, portnum, GUID, width, speed, NodeDescription
iblinkinfo                # link state and rate per switch port
ibroute <lid>             # this switch's forwarding table

ibstat                       # port state, base LID, port GUID, rate
ibaddr                       # GID of the local port
smpquery -G nodeinfo <guid>  # query by GUID, not by LID

3LMC: more LIDs per port, and what it costs

LMC is the LID Mask Control value, and OpenSM’s man page defines it in one sentence: “This option specifies the subnet’s LMC value. The number of LIDs assigned to each port is 2^LMC. The LMC value must be in the range 0-7.”^[2] Without -l the default is 0 - one LID per port, therefore one path between any two ports.^[2] UFM ships the same default, lmc = 0, read-write.^[5]

The point of LMC is to give the SM a way to express several distinct paths between the same pair of ports, because the path is chosen by destination LID. The man page adds the constraint that stops people using it as a magic performance knob: “LMC values > 0 should only be used if the subnet topology actually provides multiple paths between ports, i.e. multiple interconnects between switches.”^[2] When LMC is greater than zero, MinHop adds tie-breaks - prefer a port going to a different systemImageGuid, then a different NodeGuid, then fall back to path count.^[4]

Now the trap. “Note: LMC > 0 is not supported by fat-tree routing. If this is specified, the default routing algorithm is invoked instead.”^[4] Nothing errors out. The customer set LMC to get multipathing, and quietly lost the routing engine they actually wanted. LASH does not support LMC > 0 either.^[2] One more silent correction lives nearby: OpenSM’s own source clamps an out-of-range value back to the built-in default and logs it rather than refusing to start, so lmc 9 in a config file becomes lmc 0 with a log line most people never read - osm_subn_verify_config() reports “Invalid Cached Option Value:lmc = 9: Using Default:0” and carries on.^[11]

Changing LID assignment at all is a maintenance-window action: -r / --reassign_lids forces reassignment, and “Specifying -r on a running subnet may disrupt subnet traffic. Without -r, OpenSM attempts to preserve existing LID assignments resolving multiple use of same LID.”^[2] UFM ships reassign_lids = FALSE (disabled).^[5]

4Field consequence: key on GUIDs, verify with both

Put the lifetimes side by side. The GUID is burned in and is what every configuration file references.^[3][8] The LID is assigned at activation, only preserved on a best-effort basis, and can be forced to change by one flag.^[1][2] Therefore: inventory, alerting and automation key on GUIDs; commands and forwarding key on LIDs.

A concrete workflow for a maintenance window: capture ibnetdiscover and iblinkinfo before and after, then diff by GUID and let the LID column be the thing that moves. iblinkinfo supports this directly with --diffcheck <keys> taking a comma-separated list from port, state, lid and nodedesc, and notes that “if port is specified alongside lid or nodedesc, remote port lids and node descriptions will also be compared”.^[10] Its exit status is usable in a cron: “0 on success, -1 on failure to scan the fabric, 1 if check mode is used and inconsistencies are found”.^[10]

Duplicate GUIDs deserve one paragraph of their own because they are the failure mode people misdiagnose as a bad card. “The SM has built-in logic that knows to detect duplicate GUIDs, and to isolate (not to configure) devices with duplicate GUIDs.”^[3] By default OpenSM treats that as fatal and exits; -y / --stay_on_fatal keeps the SM alive when it “discovers duplicated guids or a 12x link with lane reversal badly configured”.^[2] So the symptom of a cloned GUID is not a broken link - it is a node that never gets configured, or an SM that will not stay up.^[3][2]