Bringing up OpenSM

Episode 1 — Both ports are Active and nobody started a subnet manager

The situation · Dell lab, Round Rock, day one of a nine-day build

Day one of nine. The two-node staging rig in the Dell lab will become a 384-node fabric for an imaging co-op - four hospital research groups buying one cluster between them - and the Dell SE has already written “fabric up” in row 4 of the spreadsheet where he keeps every promise, next to a coffee that went cold an hour ago. ibstat shows both ports Active. The co-op’s network lead opens a grid-ruled notebook to a clean page and asks who started the subnet manager. Nobody in the lab did.

That is the reason a subnet manager exists at all. InfiniBand pushes addressing and forwarding out of the switch: a port has no LID until something assigns one, and a switch has no forwarding table until something computes and writes it, so a perfectly good cable leaves a port in Initializing until an SM sweeps.^[6] OpenSM is that something - “an InfiniBand compliant Subnet Manager and Administration”, required at least once per subnet “in order to initialize the InfiniBand hardware”.^[1]

So you go looking. sminfo answers, and the master is a host nobody has logged into all week: a package install started opensm at the default priority of 0, the lowest there is.^[1]^[6] The night-shift operator prints a label reading DO NOT POWER OFF and waits to hear which box earns it.

Active ports are not evidence of health - they are evidence that some subnet manager, somewhere, already did the work.

Segment 1 starts where you do: one SM, bound to one port, on purpose.

1What OpenSM is responsible for, and where it says so

OpenSM is “an InfiniBand compliant Subnet Manager and Administration, and runs on top of OpenIB”, and at least one instance is required per InfiniBand subnet “in order to initialize the InfiniBand hardware”.^[1] Nothing on an IB fabric addresses itself: until an SM sweeps, assigns LIDs and programs forwarding tables, a perfectly good cable gives you a port stuck in Initializing.^[6]

Scope is narrower than people expect. OpenSM “attaches to a specific IB port on the local machine and configures only the fabric connected to it. (If the local machine has other IB ports, opensm will ignore the fabrics connected to those other ports).”^[1] A dual-port ConnectX in a dual-rail cluster therefore needs two instances, and the man page’s own example of separating them in syslog uses --log_prefix with the names “mpi” and “storage”.^[1]

Success has one marker. Logging goes by default to two files — /var/log/messages for general major events and /var/log/opensm.log for details of reported errors — and “Both log files should include the message SUBNET UP if opensm was able to setup the subnet correctly.”^[1]^[4] The man page adds a line worth quoting to a customer: “All errors reported in this second file should be treated as indicators of IB fabric health issues.”^[1] When a fatal, non-recoverable error occurs opensm exits, unless you passed -y/--stay_on_fatal.^[1]

One more scoping fact for sizing conversations: “opensm defaults were designed to meet the common case usage on clusters with up to a few hundred nodes.”^[1] That is not a hard limit, but it is the reason a 2,000-node customer ends up tuning timeouts and caches rather than running stock.

2The flags you will actually type

Bind explicitly. -g/--guid selects the local port GUID, and “OpenSM may be bound to 1 port at a time. If GUID given is 0, OpenSM displays a list of possible port GUIDs and waits for user input.”^[1] On a two-port card, opensm -g 0 is the safe way to discover the choices interactively before you script it.

The defaults you must know cold, because every customer question is really “is this default or did someone change it”:^[1]

Flag	Default when omitted	What it controls
`-p, --priority`	0 (lowest); range 0–15	Which SM wins handover, priority first then GUID
`-s, --sweep`	10 seconds; `-s 0` disables	Light sweep interval
`-t, --timeout`	200 milliseconds	Transaction timeout
`--retries`	3	Transaction retries
`-l, --lmc`	0; valid range 0–7	LIDs per port is 2^LMC
`-Z, --part_enforce`	`both`	Switch-side partition enforcement
`-Q, --qos`	disabled	QoS setup

Read that priority row twice. Zero is the lowest priority and it is the default, so an SM started with no -p loses every election to any SM that has one.^[1] And these are upstream numbers: UFM ships sm_priority = 15 marked read-only and qos = TRUE, so “the default” depends entirely on who runs the SM.^[2]^[1]

For a real bring-up you add daemon plumbing and a capture. -B/--daemon runs in the background and -J/--pidfile writes the PID, which is what gives you a clean stop.^[1] -c/--create-config “will dump its configuration to the specified file and exit. This is a way to generate OpenSM configuration file template.”^[1] -F/--config then selects a config file; MLNX_OFED documents the packaged path as /etc/opensm/opensm.conf and the options cache as /var/cache/opensm/opensm.opts.^[4] On a large cluster add -A/--ucast_cache, which “prevents routing recalculation (which is a heavy task in a large cluster) when there was no topology change detected during the heavy sweep… A very common case that is handled by the unicast routing cache is host reboot.”^[1]

There is a trap in the config file that produces silent, wrong behaviour. OpenSM’s own osm_subn_verify_config() clamps out-of-range values back to the built-in default and logs them, with source lines of the form if (p_opts->lmc > 7) and if (15 < p_opts->sm_priority) each calling log_report(" Invalid Cached Option Value:…: Using Default:%u\n", …).^[3] So sm_priority 20 does not become 15 and does not fail to parse — it becomes 0, the lowest priority in the fabric, while the customer believes they pinned the master.

score 0 / 12

Symptom → command · 1 / 12

One host, 60 seconds: you need LID, SMLID, port state, link width active and physical state straight from the local IB driver — from a binary, not a shell script.

Predict first: type it from memory before you pick (optional, saved)

Symptom → command

A customer sentence. Pick the one tool that answers it.

Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.

NCP-AIN · Troubleshooting Tools (20%) names:

ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo

NCP-AIN exam page

Run the symptom round. For each prompt pick the command first and only then the flag - that is the order that survives a bridge call.

3Reading state after the bring-up

Once SUBNET UP appears, four kinds of evidence exist and they answer different questions.

The cache lives in OSM_CACHE_DIR, default /var/cache/opensm: guid2lid holds the LID range assigned per GUID, guid2mkey the per-port M_Keys, and neighbors “a map of the GUIDs at either end of each link in the fabric”.^[1] That is what makes a restart non-disruptive — without -r/--reassign_lids, “OpenSM attempts to preserve existing LID assignments”, and the man page warns that “Specifying -r on a running subnet may disrupt subnet traffic.”^[1]

The dumps land in OSM_TMP_DIR, default /var/log: opensm-subnet.lst, opensm.fdbs (unicast forwarding tables) and opensm.mcfdbs (multicast).^[1] --dump_files_dir, or dump_file_dir in the config file, takes precedence over the environment variable.^[1]

The signals are your two live controls: SIGHUP “starts a new heavy sweep as if a trap was received or a topology change was found”, and SIGUSR1 reopens /var/log/opensm.log for logrotate.^[1] A kill -HUP is how you force a re-discovery after re-cabling without restarting the daemon and re-running the election.

The fabric view comes from the diagnostics, not from the SM’s own logs. sminfo “Issues and dumps the output of an sminfo query in human readable format”, and ibdiagnet writes ibdiagnet2.sm, described as “Subnet Managers” — the file that proves how many SMs the fabric can see.^[6]^[5] Note that opensm and ibutils2 are separate packages: the DOCA-Host installation page lists opensm, ibutils2, hcoll and sharp as proprietary packages requiring separate installation, so doca-all alone gives you a driver with no subnet manager and no ibdiagnet.^[7]

4Proving there is exactly one master

Mastership is decided by priority first and GUID second: “Once an SM discovers another SM in the network, the one with the highest priority setting takes precedence. If two SMs have the same priority, the SM with the lower GUID takes over.”^[10] That is two facts, and the second is the one people forget — a tie is broken by the lower GUID, which is not something you get to choose, so if you care which host is master you must set priority explicitly on both.

The election matters even in a lab, because a managed switch may already be running its own SM, and a customer’s monitoring host may have had opensm enabled by a package post-install months ago. Use the simulator below to run the cases before you meet them: two hosts at equal priority, one host against an enabled switch SM, and a master that disappears mid-run.

Inject:

step 1 / 5 · Discovery

Predict: who is master after the next step, and why? (optional, saved)

Timeline

1. Discovery

All 2 enabled SMs start in discovery mode. Each sweeps the subnet with directed-route SMPs and finds the others through SM→SM SMINFO GET/SET — that exchange is the election traffic.

Candidates — toggle, then set priority (0–15, higher wins) and port GUID

leaf01 my-smswitchpriorityGUID
opensm @ mgmt01hostpriorityGUID
opensm @ mgmt02hostpriorityGUID

Mastership rule opensm(8)MLNX-OS SM UFM run modes UFM SM defaults UFM HA sminfo ibdiagnet2.sm Fabric Summary lines iblinkinfo(8)osm_subnet.c QM97xx SM ceiling

Two host SMs at equal priority. Step through discovery and election and read which GUID wins - then set one to priority 14 and re-run.

Bring up a host SM on a fresh lab fabric and prove it

Case: a two-node Dell lab, ConnectX ports already in IB mode, cabled through one switch. No SM is running anywhere and both ports show Initializing.

Confirm the physical layer before blaming the SM. On both hosts:
```
ibstat
```
Expected: Physical state: LinkUp, State: Initializing, Link layer: InfiniBand. LinkUp plus Initializing means the link trained and no SM has programmed the port — an SM problem, not a cable problem.^[9]^[6]
Confirm nothing is already managing the subnet:
```
sminfo
```
Expected: a failure or no master. If this returns a master, stop — you are about to create a second SM and the rest of this runbook is the wrong procedure.^[6]
Read the port GUID you intend to bind, letting opensm enumerate rather than transcribing hex by hand:
```
opensm -g 0
```
Expected: a list of possible port GUIDs and a prompt for input; Ctrl-C out once you have the GUID.^[1]
Capture the configuration you are about to run, before you run it:
```
opensm -c /tmp/opensm.conf
grep -E '^(sweep_interval|transaction_timeout|transaction_retries|sm_priority|lmc|part_enforce|qos) ' /tmp/opensm.conf
```
Expected: sweep_interval 10, transaction_timeout 200, transaction_retries 3, sm_priority 0, lmc 0, part_enforce both, qos FALSE.^[1] This file is also the artifact you ask a customer for instead of a screenshot.
Start it bound to one port, in daemon mode, with a pidfile:
```
sudo opensm -g 0x248a070300a28c4d -p 13 -B -J /var/run/opensm.pid
```
-p 13 is deliberate: leaving the default 0 means any other SM that appears later takes over silently.^[1]
Prove it worked, in this order:
```
grep -c "SUBNET UP" /var/log/opensm.log
ibstat | grep -E "State:|Base lid"
sminfo
```
Expected: exactly one SUBNET UP; ports now Active with a non-zero base LID; one master reported by sminfo.^[1]^[6]
Exercise the live control once, so you have seen it before you need it:
```
sudo kill -HUP $(cat /var/run/opensm.pid)
tail -20 /var/log/opensm.log
```
Expected: a new heavy sweep in the log and still exactly one master.^[1]
Rollback for the whole exercise: sudo kill $(cat /var/run/opensm.pid). Ports return to Initializing, which is the proof that the SM was doing the work.^[1]

One SM, on purpose

How it ended

You bind deliberately: -g on a port GUID you read rather than transcribed, -p 13 so no stray default-priority SM inherits the fabric later, -B and -J for a clean stop, and opensm -c archived before anything changes.^[1] Then the three-command proof: one SUBNET UP in /var/log/opensm.log, ports Active with a non-zero base LID, one master from sminfo.^[1]^[6]

What you say to the network lead: “Your ports were Active because a package install started a subnet manager at priority zero. We run one now, on purpose, and we can prove which one in three commands.”

She writes it down, then turns the notebook around. The design review is in two days, and the SM on the architecture slide is not this host. It is inside a switch.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: bring up a real subnet manager on the Dell lab and prove it. Step 3 starts a daemon that changes the state of every port in the subnet; its rollback is named.

Pre-flight inventory, read-only. On every host with an IB-mode port:
```
ibstat | tee /tmp/ibstat-before-$(hostname).txt
ibstat -p
sminfo || echo "no SM answered"
systemctl is-active opensm || true
rpm -q opensm ibutils2 2>/dev/null || dpkg -l | grep -E 'opensm|ibutils2'
```
Expected: port state Initializing with Physical state: LinkUp on a fabric with no SM; a port GUID list from ibstat -p; sminfo failing; the opensm service inactive. If sminfo answers, an SM is already running — do not start a second one; go to the next lesson and decide placement first.^[6]^[1]
Capture the configuration before changing anything: opensm -c /tmp/opensm-lab.conf and archive it next to the ibstat capture.^[1]
Mutating — start the SM bound to one port. Rollback: sudo kill $(cat /var/run/opensm.pid), after which ports return to Initializing.
```
sudo opensm -g <port_guid_from_step_1> -p 13 -B -J /var/run/opensm.pid
```
Expected: the process daemonizes and the pidfile exists.^[1]
Verify in the documented order, read-only:
```
grep -n "SUBNET UP" /var/log/opensm.log
ibstat | grep -E "State:|Base lid|SM lid"
sminfo
```
Expected: exactly one SUBNET UP; every port Active with a non-zero base LID; sminfo naming one master with the priority you set.^[1]^[6] If SUBNET UP repeats every few minutes, stop and look for a flapping link or a second SM before tuning anything.^[11]
Exercise the signals, read-only in effect:
```
sudo kill -HUP $(cat /var/run/opensm.pid); tail -30 /var/log/opensm.log
sudo kill -USR1 $(cat /var/run/opensm.pid)
```
Expected: a heavy sweep after HUP; the log file reopened after USR1 with no other change.^[1]
Inspect the artifacts the SM just wrote, read-only: ls -l /var/cache/opensm/ and ls -l /var/log/opensm-subnet.lst /var/log/opensm.fdbs /var/log/opensm.mcfdbs. Expected: guid2lid, guid2mkey and neighbors in the cache directory and the three dumps in /var/log.^[1] Open guid2lid and match one line to a GUID from step 1.
Restart-stability check, mutating but self-reverting. Stop the SM, restart it with the same command, and diff the LIDs:
```
sudo kill $(cat /var/run/opensm.pid); sleep 5
sudo opensm -g <port_guid> -p 13 -B -J /var/run/opensm.pid; sleep 20
ibstat | grep "Base lid" | tee /tmp/lids-after.txt
```
Expected: identical LIDs, because without -r OpenSM preserves existing assignments using guid2lid.^[1] Do not add -r on anything but an idle lab: the man page warns it may disrupt subnet traffic.^[1]
Rollback and confirm: sudo kill $(cat /var/run/opensm.pid), then ibstat and diff against /tmp/ibstat-before-$(hostname).txt until only the expected fields differ. Optional, customer lab only: start a second opensm on another host with --log_prefix and a different -p, then re-read sminfo from a third node — that is lesson 2’s territory and should be done deliberately, not by accident.^[1]

Evidence / notes (commands run, outputs, what surprised you)

Before you read: what do you already know?

After this lesson you can