Who's who in an InfiniBand subnet

Episode 1 — Nobody on the call knows who the master is

The situation · Customer bridge call, 07:10, day one of a sixty-four-node proof of concept

The bridge opens with eleven people on it and three theories. In thirty days an acceptance script decides whether the imaging company buys these sixty-four nodes and the fabric under them. Today their jobs hang at random: always on the same collective, never on the same nodes. The Dell SE — coffee already cold, spreadsheet where every promise gets a row — is being asked to swap a third cable. Everyone is describing a physical problem.

You ask something that sounds beside the point: who is your Subnet Manager, and is there exactly one? Silence, then two answers. Someone enabled the switch-embedded manager during bring-up, on a platform whose own manual recommends no more than two subnet managers for any single fabric.^[6] Someone else installed UFM last month, and UFM’s documentation is explicit that in Management or High Availability mode every other Subnet Manager must be disabled first, because any that remain active will prevent UFM from running.^[7]

This is the architecture. An InfiniBand switch does not learn where addresses live; it forwards on tables a central manager wrote, which is how the fabric gets deterministic routing and centralised access control instead of any host arbitrarily adjusting network capabilities.^[1] One authority is the design. Two authorities is the outage.

On InfiniBand the first question is never “which cable” — it is “who is master”.

You ask for sminfo from two hosts and the ibdiagnet2.sm file.^[5] The network lead opens a green notebook to a clean page. First, the vocabulary of who does what.

1The entity list, in NVIDIA's own words

Start with vocabulary, because every later lesson leans on it. NVIDIA’s architecture list for an InfiniBand subnet is: Node (a host with an HCA), HCA (Host Channel Adapter, attached to the host over PCIe), Switch (moves IB packets port-to-port “based on routing tables configured by the Subnet Manager”), Router (“functions as a switch but operates between different subnet domains”), Subnet Manager (SM), and the UFM Appliance.^[1]

The sentence that matters most is the one about the switch. An InfiniBand switch does not learn where addresses live. It forwards on a table that the Subnet Manager wrote into it. NVIDIA describes the whole thing as “a software-defined network” in which the SM “enables security by providing centralized access control to network resources and preventing any host from arbitrarily adjusting network capabilities”.^[1] That is why a fabric with no SM comes up physically and moves nothing: the links train, but a switch forwards only on the tables the Subnet Manager wrote, so there is nothing wrong with the cable.^[1] Lesson 5 reads the resulting pair off a real host - physical state LinkUp with logical state Initializing - and names the sysfs files that show it.

A second consequence follows immediately: a host cannot pick its own privileges. “The application cannot specify the partition but uses the existing index in the local partition table, ensuring it cannot connect to a partition of which the node is not a member.”^[1] Compare that with Ethernet, where a host with root can tag any VLAN it likes. On InfiniBand the tenant boundary is programmed from the centre, and the card enforces it.^[1]

Rendering diagram…

Diagram source (Mermaid)

flowchart LR; SM["Subnet Manager - discovers, assigns LIDs, computes paths"] -- "SMPs on QP0 / VL15" --> SMA1["SMA on HCA"]; SM -- "SMPs on QP0 / VL15" --> SMA2["SMA on switch"]; SMA1 --> HCA["HCA in a node - PCIe attached"]; SMA2 --> SW["Switch - forwards on tables the SM wrote"]; SM --- SA["Subnet Administrator - answers path and service queries"]; SW --> RT["Router - joins different subnet domains"]; UFM["UFM Appliance"] -.-> SM

The control plane and the data plane of one subnet. Everything the switch does on the right was programmed from the left.

2How the SM actually talks to the fabric

“Each HCA and switch run a Subnet Manager Agent (SMA), which is responsible for communicating with the SM. The SM configures the network by sending Subnet Manager Packets (SMPs) via management datagrams (MADs).”^[1] Those SMPs are not ordinary traffic: “SMPs can be either lid-routed (LIDR) or direct-routed (DR). SMPs are sent via Queue Pair (QP) 0 on Virtual Lane 15 (lossy, management VL).”^[1] RFC 4392 states the rule from the other side - “All ports must support VL15 which is reserved exclusively for subnet management datagrams.”^[2]

Direct-routed is the detail worth holding onto. A DR SMP carries the path as a list of output ports rather than a destination LID, so it works on a subnet where no LID has been assigned yet. That is exactly why the base diagnostic tools function on an unconfigured fabric: infiniband-diags states that “the base utilities use directed route MAD’s to perform their operations. They may therefore work even in unconfigured subnets”, while the higher-level tools need LID routing and SA access.^[11]

There are three documented conversation patterns: SM to node (an SMP request and its response), node to SM (a TRAP, which the SM answers with TRAP_REPRESS), and SM to SM (SMINFO GET/SET during mastership election).^[1] Traps are how a switch tells the manager that a link changed state, and they are the reason a flapping cable can turn into a storm of full topology re-scans two modules from now.^[1]

The glossary the same document set publishes is worth memorising because the abbreviations appear raw in logs: AM = Aggregation Management, BTH = Base Transport Header, CC = Congestion Control, DR = Direct Routed, GMP = General Management Packets, LIDR = LID Routed, PMA = Performance Manager Agent.^[10]

3What the SM does, what the SA answers, and where LIDs live

NVIDIA lists four SM responsibilities: discover the physical topology; assign LIDs to end nodes, switches and routers; establish paths among end nodes; and sweep the subnet to manage topology changes such as node reboots and link flaps.^[1] Alongside it, the Subnet Administrator “runs on the same node running the Subnet Manager (SM) and compliments it by responding to information or registration requests” - events, service attributes, topology and switch forwarding tables.^[1] A short way to keep them apart: the SM writes, the SA answers.

Addressing has one shape that surprises people coming from Ethernet. RFC 4392 is blunt: “The ports of a switch are not addressable by LIDs/GIDs or, in other words, are transparent to other end nodes”.^[2] The corollary every tool relies on - that you reach a switch through the LID on its management port, port 0, plus a port number - is IBA behaviour that RFC 4392 does not spell out; you see it directly in an ibnetdiscover capture, which prints port LIDs per node and can also show the switch external-port correspondence.^[8] The next lesson takes the addressing apart properly; here, just fix the shape: end ports get LIDs, switch external ports do not.^[2]

ibnetdiscover is where you see this on a real fabric. It “performs IB subnet discovery and outputs a human readable topology file. GUIDs, node types, and port numbers are displayed as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed (full topology).”^[8] Add -p for “list of connected ports with relevant information (like LID, portnum, GUID, width, speed, and NodeDescription)” and -g for vendor grouping that “may also show the switch external ports correspondence”.^[8]

baseline shape — every inter-node flow crosses the spine

Scalable units1 SU

Path

Click one HCA for the source, another for the destination. Hops are counted as switch ASICs traversed — a link count is hops + 1.

Fat tree (node-aligned)

Leaf switches under spine (root) switches. Here every rail of a node drops on that node's own leaf, so nothing is rail-aligned: all inter-node traffic is leaf → spine → leaf, 3 switch hops.
Quantum-2 switches "enable a variety of topologies, including Fat Tree, SlimFly, DragonFly+, multi-dimensional Torus, and more"; the QM9700 family is positioned for "small to medium-sized deployments" on a two-level fat tree.
Ceiling: a two-level fat tree on Quantum-X800 Q3400 supports "up to 10,368 network interface cards (NICs)".

FAE angle. The empty leaf is deliberate. For a partly populated SU the RA says to buy the full leaf count and leaf-spine cables anyway and leave the positions empty — it "will ensure optimal traffic routing". Shrink the fabric instead and the tree turns asymmetric, ftree's rank rules stop holding, and job-to-job timing goes non-deterministic.

QM97XX User Manual — Introduction

LIDs, GIDs, GUIDs on this picture

Every HCA port gets a LID: "a 16-bit value that is assigned when the port becomes active", and "the subnet manager may attempt to keep the LID values constant across reboots, but that is not a requirement."
The GID is 128 bits — a 64-bit IB subnet prefix plus a 64-bit EUI-64 GUID. OpenSM and UFM both default the prefix to 0xfe80000000000000 (--subnet_prefix).
"The GUID is the only persistent identifier of a port. However, it cannot be used as an address in a packet."
Switch external ports — every leaf-spine and node-leaf port drawn here — carry no LID: they "are transparent to other end nodes". Only switch port 0, the management port, has one.
LMC is 0 by default: one LID, one path per port. -l/--lmc gives 2^LMC LIDs, range 0–7, and "should only be used if the subnet topology actually provides multiple paths between ports". Fat-tree routing does not support LMC > 0 — set it and OpenSM invokes the default engine instead.
FAE angle: never key a script, a partitions file or a root-GUID file on a LID. LIDs move at the next activation; GUIDs are burned in.

ibstat                       # port state, base LID, port GUID, rate
ibaddr                       # GID of the local port
smpquery -G nodeinfo <guid>  # query by GUID, not by LID

RFC 4392 §1.2

Scalable-unit arithmetic — 1 SU

DGX H200 SuperPOD compute fabric — QM9700 NDR

Predict before you look: how many leaf and spine switches does 1 SU need here? (optional, saved)

Sources: opensm(8) · UFM SM defaults · MLNX-OS Subnet Manager · SHARP environment · Quantum-X800 switches · MCA4J80 ACC · ibdiagnet · ibdiagnet dump files · Dell XE9680 technical guide

Set the overlay to Address and click a leaf switch, then an HCA. Note which elements are given an identifier you can put in a packet and which are not.

4The first question on every bring-up call

On a Dell plus NVIDIA InfiniBand build there are three legal places for the SM to run - the switch-embedded MLNX-OS SM, a host running opensm, or UFM - and only one of them should be master.^[9]^[7] Module 2 chooses between them properly. For now, learn the constraint that turns this into a first question rather than a design question.

NVIDIA’s own switch manual says “No more than two subnet managers are recommended for any single fabric”.^[6] UFM goes further: “In Management or High Availability mode, ensure that all Subnet Managers in the fabric are disabled before running UFM. Any remaining active Subnet Managers will prevent UFM from running.”^[7] And the election rule is fixed: “At system bring up, all SMs are in discovery mode. Once an SM discovers another SM in the network, the one with the highest priority setting takes precedence. If two SMs have the same priority, the SM with the lower GUID takes over.”^[1]

The evidence is two commands. sminfo “issues and dumps the output of an sminfo query in human readable format”, targeting the SM listed in the local port info.^[4] And ibdiagnet writes ibdiagnet2.sm, the file that lists the Subnet Managers seen in the fabric - which is what you ask a customer to send when you suspect two masters and cannot log in yourself.^[5] A healthy host SM also writes SUBNET UP to both /var/log/messages and /var/log/opensm.log; seeing it once is health, seeing it every few minutes is a symptom.^[3]

One capability trap belongs here even though its lesson is later: the switch-embedded SM “does not support adaptive routing, fault routing (i.e., SHIELD or FRN), congestion control, SHIELD, and SHARP”, and also does not support DragonFly+ topologies.^[9] A customer who runs the on-board SM to avoid a UFM licence and then reports that adaptive routing does nothing is not seeing a defect.^[9]

Episode 1 — Case note: two masters, one fabric

How it ended

The ibdiagnet2.sm file lists two Subnet Managers, and the two sminfo captures, ten minutes apart, name different masters.^[5] That is the ticket: two authorities writing forwarding tables, so the fabric re-routes and jobs stall in ways that look like cabling. You add: if they keep the switch-embedded manager, adaptive routing, fault routing, congestion control and SHARP are off the table on this fabric.^[9] What you say to the network lead: “Your cables are fine. Two managers are programming your switches, and until exactly one of them is master, nothing else we test will be conclusive.” She writes it down. Then the night-shift operator asks why node 41 came back from a reboot wearing a different address — and whether it is allowed to do that.

Lab

Read-only on the Dell-lab host. No configuration is changed, so no rollback is required.

Pre-flight inventory, before you touch anything: lspci | grep -i mellanox, ibv_devices, and ibdev2netdev. Record the device names and their netdev mapping.
Confirm the port is InfiniBand and not Ethernet: ibstat | grep -A1 "Link layer" or cat /sys/class/infiniband/mlx5_0/ports/1/link_layer. Expected: InfiniBand. If it says Ethernet, this host’s port is in Ethernet mode and belongs to the RoCE course - the IB labs from lesson 3 onward will not apply.
Identify the local HCA and its port GUID: ibstat and ibstat -p. Expected: a Port GUID in 0x… form plus a Base lid. Write both down - lesson 2 is about the difference between them.
Ask the port which SM it knows about: sminfo. Expected: one line naming an SM LID, GUID, priority and state. If the command returns nothing, no SM has programmed this port yet - which is the correct reading, not a tool failure.
Read the same facts from the driver rather than the fabric: ibv_devinfo -v | head -40. Expected: sm_lid and port_lid match what ibstat reported.
Optional, only where a real IB switch exists (customer or NVIDIA lab): ibnetdiscover -p and locate a Switch entry. Confirm that the switch appears once with a port-0 LID and that its external ports are listed as ports of that entry with no LIDs of their own. This is the single fastest way to make the RFC 4392 rule concrete.

Evidence / notes (commands run, outputs, what surprised you)

Who's who in an InfiniBand subnet

Before you read: what do you already know?

After this lesson you can

Episode 1 — Nobody on the call knows who the master is

1The entity list, in NVIDIA's own words

2How the SM actually talks to the fabric

3What the SM does, what the SA answers, and where LIDs live

Fat tree (node-aligned)

4The first question on every bring-up call

Episode 1 — Case note: two masters, one fabric

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere