Where the SM runs: switch, host, or UFM
S2·E2The slide says embedded SM, and underneath it says SHARP · NVIDIA briefing room, day three, six days to the acceptance test
Builds on: Bringing up OpenSM
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish the three legal subnet-manager placements by the features each one can and cannot deliver.
- Analyse a customer requirement and choose a placement citing the documented unsupported-feature list rather than preference.
- Predict the outcome of an SM election and a handover from priority and GUID and explain what UFM changes about it.
- Assemble the evidence set that proves how many subnet managers a customer fabric is actually running.
Episode 2 — The slide says embedded SM, and underneath it says SHARP
The slide on the screen is the Dell SE’s, from March: switch-embedded subnet manager, no extra hardware, and a bullet underneath that reads adaptive routing and SHARP. Row 9 of the promise spreadsheet. The network lead’s notebook is open at day one’s three commands; procurement wants to know what any change costs in lead time, since the expansion PO is written against an acceptance test six days out. The NVIDIA PM says “roadmap”, then “not announced”, closing that branch.
Placement is a real question because the same protocol role can live in three places, and they are not the same product. The embedded SM on a managed switch is the no-extra-hardware option; a host running opensm is the flexible one; UFM is the managed one that also brings a dashboard, health reporting and alerting - and those are not subnet-manager functions at all, which is why no amount of tuning gets them out of the other two.[9] It is settled by a documented list of what the embedded SM does not implement, not by preference.[1]
You do not argue with the slide. You ask what the cluster will run and what the co-op wants to see at 02:00 when it misbehaves: training jobs, SHARP, and a dashboard four hospital groups can read. Scale is not the argument - 384 nodes sits far inside the embedded SM’s documented 2048-node ceiling.[1]
Choose where the SM lives by the feature list it does not implement, not by the node count it can reach.
Segment 1 takes the three placements apart in that order.
1Three placements, and the cliff between them
Every InfiniBand subnet needs one master subnet manager, and there are exactly three legal places to put it: the embedded SM inside a managed Quantum switch, a Linux host running opensm, or UFM.[5][2] Choosing between them looks like a scale question and is really a feature question.
The switch-embedded SM is disabled by default and is enabled per SM node in MLNX-OS: switch (config) # ib smnode my-sm enable, with switch (config) # ib smnode my-sm sm-priority <priority> placing it in the election.[1] It manages fabrics “up to 2048 nodes on x86 based systems”, a figure the switch hardware manual restates and the QM9700 product text rounds to “up to 2,000 nodes”.[1][5][13] That ceiling is generous — and almost never the reason a customer outgrows it.
The reason is this sentence, which is the single most commonly missed fact in an InfiniBand pre-sales conversation: “Subnet manager running via MLNX-OS does not support adaptive routing, fault routing (i.e., SHIELD or FRN), congestion control, SHIELD, and SHARP.”[1] It also “does not support Dragonfly+ topology or combination of Fat-Tree and Dragonfly+ topologies.”[1] A 512-node AI training pod is far inside the node ceiling and still cannot use the embedded SM, because the features that make an AI fabric worth buying are not implemented in it.
A note on which switch you are talking about. The MLNX-OS pages this course could fetch are v3.12.6200 LTS, which predates Quantum-X800; the CLI syntax above is confirmed for MLNX-OS and should be attributed that way rather than to a Quantum-3 class switch running NVOS.[1]
| Product | Speed | PCIe | Role | GPU generation |
|---|---|---|---|---|
NIC | ||||
SuperNIC (no Arm) | ||||
SuperNIC (no Arm) | ||||
DPU | ||||
DPU | ||||
SuperNIC (Arm inactive) | ||||
DPU / storage processor | ||||
Ethernet switch | ||||
Ethernet switch | ||||
InfiniBand switch | ||||
InfiniBand switch |
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2Election and handover, in practice
Mastership is decided the same way regardless of placement: “At system bring up, all SMs are in discovery mode. Once an SM discovers another SM in the network, the one with the highest priority setting takes precedence. If two SMs have the same priority, the SM with the lower GUID takes over.”[4] Upstream opensm gives priority a range of 0 to 15 where 0 is the default and the lowest.[10]
The dangerous part is that there is no fabric-wide default. UFM ships sm_priority = 15 marked read-only, alongside sweep_on_trap = TRUE so every trap 128 and 144 causes a heavy sweep, and ignore_other_sm = FALSE.[3] The switch SM has whatever number someone typed into sm-priority. So “the SM is at the default priority” is not a statement anyone can make about a mixed fabric; the only correct move is to read each candidate’s running configuration.[3][10]
Run the cases in the simulator before you meet them on a call. The three that matter are: two host SMs at equal priority, where the GUID decides; a host SM against an enabled switch SM, where the number you typed decides; and a master that disappears mid-run, where you watch the standby take over and then ask what happens when the old master returns.
1. Discovery
All 3 enabled SMs start in discovery mode. Each sweeps the subnet with directed-route SMPs and finds the others through SM→SM SMINFO GET/SET — that exchange is the election traffic.
- switch
- host
- host
- UFM
3UFM: run modes, install shape, and the precondition
UFM has four run modes: Management, Monitoring, High Availability, and High Availability with failover to an external SM.[2] The precondition is the thing customers hit first: “In Management or High Availability mode, ensure that all Subnet Managers in the fabric are disabled before running UFM. Any remaining active Subnet Managers will prevent UFM from running.”[2] That includes the switch SM someone enabled during bring-up and forgot.
Monitoring mode is the read-only posture. It disables the Subnet Manager and plugins, SM configuration, automatic fabric partitioning, central device management, QoS, HA failover, traffic-aware routing, device management, scheduler integration and unhealthy ports, while keeping fabric discovery, the topology map, the dashboard, monitoring, alerts and SNMP traps, and the fabric logical model.[2] In that mode UFM connects “to the fabric using port ib0 only”, monitoring_mode is set to yes in /opt/ufm/conf/gv.cfg, and /opt/ufm/files/conf/opensm/opensm_mon.conf is overwritten on startup — “Any other configuration is not valid for Monitoring mode.”[2]
UFM Enterprise installs as a container. The documented standalone command mounts the Docker socket, the systemd directory, /opt/ufm/files/ for persistent state and a temp directory holding the licence, and names the fabric interface:[6]
docker run -it --name=ufm_installer --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/systemd/system/:/etc/systemd_files/ \
-v /opt/ufm/files/:/installation/ufm_files/ \
-v /tmp/license_file/:/installation/ufm_licenses/ \
mellanox/ufm-enterprise:latest \
--install \
--fabric-interface ib1Then systemctl daemon-reload and systemctl start ufm-enterprise.[6][2] Underneath, UFM “utilizes the Open Fabric community OpenSM Subnet Manager” — which is why every opensm parameter you learned in lesson 1 still applies, just read from /opt/ufm/files/conf/opensm/opensm.conf instead of /etc/opensm/.[3]
4High availability, and what the notes can and cannot tell you
The NCP-AIN blueprint’s InfiniBand domain lists “Perform initial configuration and provisioning with HA” as an objective, so this is exam material as well as field material.[11]
The documented HA design is a two-server pair: DRBD “synchronizes a replicated partition between the two servers”, with /opt/ufm/files mounted on the DRBD device; a heartbeat layer is “responsible for starting UFM on master node and stopping it on the slave” and sends keepalives; and a virtual IP moves to the surviving node on failover.[7] Before a second failover the pair must re-establish readiness — management-network connectivity, DRBD state, disk space and IB cluster membership — and the manual is explicit that “UFM can sustain a second failover only a few minutes after the new UFM standby server is up and running”, depending on partition size and link speed.[7] That sentence is the one to quote when a customer asks why a rapid double failover left them down.
Operationally the pair is driven by ufm_ha_cluster, whose documented subcommands include start, stop, failover, takeover, status, cleanup and help.[2] Be careful how much further you go: the fuller subcommand list some search results show (config, detach, attach, enable-maintain, reset, is-master) and the frequently repeated “configure the standby first, then the master” ordering could not be confirmed on a page this course fetched — the HA pages that resolved returned the older DRBD and heartbeat text.[7][2] Treat those as unverified and check the customer’s own UFM version’s HA guide before writing them into a runbook.
For monitoring, the artifact to ask for is the System Health report: UFM can “create reports that run a series of checks on the UFM server”, run from a “Run New Report” button with results displayed inline, covering UFM Configuration (release, licence type and expiration, functionality level, device limits, running mode), UFM Processing (the status of OpenSM, ibpm, ModelMain, httpd and MySql), Memory, CPU, Disk and Fabric Interface.[9] Two caveats for honesty with a customer: the other System Health tabs (Fabric Health, Daily Reports, IBDiagnet) exist but the fetched page gives no detail, and this course contains no UFM telemetry, REST or GUI walkthrough — so “Use UFM for monitoring” is only partially covered here.[9][11]
Case: a Dell customer runs 380 nodes of CFD on a two-tier fat tree behind QM9700s. No collectives beyond MPI, no AI training planned. They also want “a dashboard the ops team can look at”, and they have already enabled the switch SM during bring-up.
- Size first, and get it out of the way. 380 nodes is far inside the documented 2048-node ceiling for the embedded SM, so scale is not an argument for anything.[1]
- Test the requirement against the unsupported list, item by item. Adaptive routing: not needed. SHIELD or FRN: not asked for. Congestion control: not asked for. SHARP: not asked for. DragonFly+: not the topology. Nothing in the workload requires a feature the embedded SM lacks.[1]
- Test the second requirement, which is the one that actually decides. A dashboard, health reporting and alerting are not subnet-manager functions at all — they are UFM functions, delivered by the System Health report and the monitoring surface.[9] The embedded SM has no equivalent.
- Offer the mode that satisfies both without a fabric change: UFM in Monitoring mode alongside the existing switch SM. Monitoring mode disables UFM’s own SM and its configuration, partitioning and QoS functions while keeping discovery, topology, dashboard, monitoring and alerts — and it connects on
ib0only.[2] - State the consequence honestly rather than selling around it: in Monitoring mode the customer gets no automatic partitioning, no QoS from UFM, no HA failover and no device management. If they later want any of those — or ever want adaptive routing or SHARP — they must move to Management mode, which requires disabling the switch SM first.[2][1]
- Write down the evidence you will ask for before and after. Before:
show ib smnodefrom each switch,sminfofrom two hosts, and oneibdiagnetrun foribdiagnet2.sm.[1][12][8] After: the same three plus a UFM System Health report showing OpenSM’s state in the UFM Processing group.[9] - Record the decision in one sentence a Dell account manager can repeat: “Switch SM keeps the fabric, UFM Monitoring gives you visibility, and the day you want adaptive routing or SHARP the switch SM comes off.”
- Size check: 380 nodes against the documented ceiling of ____ nodes. Conclusion: scale is ____ the deciding factor.
- Feature check against the embedded SM’s unsupported list: ____ routing, fault routing (____ or FRN), ____ control, and ____ . None required here.
- The real requirement is a dashboard and alerting, which is a ____ function, not a subnet-manager function.
- Recommend UFM in ____ mode, which keeps discovery, topology, dashboard, monitoring and alerts, and connects on port ____ only.
- Name the cost: no automatic ____ , no ____ from UFM, no ____ failover, no device management.
- Evidence before:
show ib ____per switch,____from two hosts, and theibdiagnet2.____file. Evidence after: add a UFM ____ ____ report. - One-sentence decision for the account manager.
A different customer on the same account: 96 nodes of GPU training on QM9700s, currently managed by the switch SM. They have opened a case saying adaptive routing is enabled in their design document but ibdiagnet shows no adaptive-routing tables, and SHARP jobs fail to allocate a tree. They also have a second site with a UFM pair that they want to standardise on, and they ask whether the UFM standby at the second site can manage this fabric too.
Write the response. Acceptance criteria: (a) state whether the reported behaviour is a defect and cite the exact documented sentence that settles it; (b) name the placement change required and the precondition that must be satisfied before UFM will start; (c) answer the cross-site question in terms of what a subnet manager’s scope actually is; (d) give the ordered cutover steps with the rollback for each mutating one; (e) name the two pieces of evidence you would attach to close the case.
What the room signs off
It lands on UFM in Management mode, justified by the unsupported-feature sentence rather than a node count, with the switch SM disabled first - any remaining active subnet manager prevents UFM from running.[1][2] Priorities get written down for both candidates in the same session, because there is no fabric-wide default: UFM ships sm_priority = 15 marked read-only, and a switch SM carries whatever number someone typed.[3]
What you say to the room: “Nothing on your slide is wrong except where the SM lives. Move it, and adaptive routing and SHARP stop being a support case.”
The network lead is already on the next page. Four hospital groups, one subnet, and a question for Friday’s change window: what keeps their traffic apart?
Lab
Goal: prove the election rule on the Dell lab with two subnet managers you control. Steps 3 and 5 are mutating and name their rollbacks. The switch and UFM half is customer-lab only.
- Pre-flight inventory, read-only, on both hosts:
Expected: one master reported byibstat | tee /tmp/ibstat-before-$(hostname).txt ibstat -p sminfo ps -ef | grep -c "[o]pensm"sminfo(the SM you started in lesson 1), the port GUIDs of both hosts, and exactly oneopensmprocess in the subnet.[12] - Record the current master’s priority and GUID from
sminfooutput and from the command line you used. You are about to make a second SM and you need the “before” to interpret the “after”.[12] - Mutating — start a second
opensmon the other host at a lower priority, with its own log prefix so syslog stays readable. Rollback:sudo kill $(cat /var/run/opensm-standby.pid).
Expected: the master is unchanged — the priority-13 SM still wins.[10]sudo opensm -g <host_b_port_guid> -p 5 -B -J /var/run/opensm-standby.pid --log_prefix standby sleep 20 sminfo - Prove it from the fabric rather than from the process list, read-only:
Expected:ibdiagnet cat /var/tmp/ibdiagnet2/ibdiagnet2.smibdiagnet2.smlists both subnet managers with one marked master.[8] Note the real output directoryibdiagnetprinted — the docs disagree between/var/tmp/ibdiagnet2/and/var/tmp/ibdiagpath/, so read the path off the run rather than trusting either page.[8] - Mutating — force a handover by killing the master. Rollback: restart it with the original command from lesson 1.
Expected: the priority-5 SM is now master and its log shows asudo kill $(cat /var/run/opensm.pid) sleep 30 sminfo grep -n "SUBNET UP" /var/log/opensm.log | tail -3SUBNET UP. Restart the original and confirm mastership returns to it.[10] - Equal-priority tiebreak, mutating and self-reverting. Restart the standby with
-p 13so both SMs match, then re-readsminfoand check the winner against the two port GUIDs — the lower GUID should hold mastership.[4] Rollback: stop the standby. - Clean up: stop the standby SM, confirm one master with
sminfo, and diffibstatagainst/tmp/ibstat-before-$(hostname).txt. - Optional, customer lab only with a managed switch and UFM. Enable the switch SM with
ib smnode my-sm enableandib smnode my-sm sm-priority 5, verify withshow ib smnode, then attempt to start UFM in Management mode and observe it refuse while any SM remains active.[1][2] Rollback:ib smnode my-sm disableon the switch and stop any host SM before starting UFM. Do not perform this on a production fabric — disabling the running SM re-elects mastership.
Goal: build the placement decision table and the evidence request, with no hardware. All steps are documentation and simulator work.
- Build the three-row decision table yourself, one row per placement (switch-embedded, host
opensm, UFM), with columns: node ceiling, adaptive routing, SHIELD or FRN, congestion control, SHARP, DragonFly+, monitoring and health reporting, documented HA. Fill it only from the sources in this lesson. Expected: the embedded-SM row carries five “no” cells sourced from one sentence, and the host-opensmrow’s monitoring cell is empty rather than “no” —opensmsimply is not a monitoring product.[1][2] - Decide three customer asks against that table and write one sentence of justification each, quoting the deciding line:
- a 2,000-node HPC cluster with no AI collectives;
- a 512-node training pod that wants adaptive routing and SHARP;
- a monitoring-only takeover of a fabric someone else manages. Expected: switch SM, UFM Management, UFM Monitoring respectively — and for the middle one your justification must cite the unsupported-feature sentence rather than a node count.[1][2]
- Run the election simulator through all three of these cases: two hosts at equal priority (predict the winner from the GUIDs before stepping), a switch SM at priority 5 against a host at 13, and a master-down event. Write your prediction before each run. Expected: your predictions match; where they do not, the panel names the rule you missed.
- Write the evidence request you would send a customer, as an actual message. It must ask for
sminfofrom two named nodes, oneibdiagnetrun with theibdiagnet2.smfile attached,show ib smnodefrom every managed switch, andopensm -coutput from every host that runs a subnet manager.[12][8][1] Expected: under 120 words, no diagnosis in it, and every item something the customer can produce without a maintenance window. - Record the honest gaps in your own notes, because you will be asked: this course has no UFM GUI, telemetry or REST walkthrough, so “Use UFM for monitoring” is only partly covered; and the
ufm_ha_clustersubcommand list beyond start, stop, failover, takeover, status, cleanup and help - plus the “configure the standby first” ordering - is unverified here.[2][7]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A Dell account team is quoting a 400-node InfiniBand cluster and asks whether the customer needs UFM or whether the managed switch will do. Answer in five sentences without saying it depends.
Sources
Facts in this lesson were checked against MLNX-OS v3.12.6200 LTS Subnet Manager page and UFM Running UFM Server Software page re-fetched 2026-09-07 (run modes, disable-all-other-SMs rule, ufm_ha_cluster subcommands); UFM Subnet Manager Default Properties 6.24.1 re-fetched 2026-09-07 (sm_priority 15 RO). Dates are when each page was fetched.
- Subnet Manager - NVIDIA MLNX-OS User Manual v3.12.6200 LTS · fetched 2026-09-07
- Running UFM Server Software - UFM Enterprise 6.15.0 · fetched 2026-09-07
- UFM Enterprise 6.24.1: UFM Subnet Manager Default Properties · fetched 2026-09-07
- Security in InfiniBand - NVIDIA InfiniBand Security Overview and Guidelines · fetched 2026-09-07
- QM97XX User Manual: Software Management · fetched 2026-09-07
- Installing UFM on Docker Container - Standalone Mode (UFM Enterprise 6.24.1) · fetched 2026-09-07
- High Availability - NVIDIA UFM Enterprise User Manual 6.11.2 · fetched 2026-09-07
- ibdiagnet Dump Files - IBUtils2 Utility Documentation · fetched 2026-09-07
- UFM Health Tab - UFM Enterprise User Manual 6.25.1 · fetched 2026-09-07
- opensm(8) man page source (linux-rdma/opensm master) · fetched 2026-09-07
- AI Networking Certification (NCP-AIN) - NVIDIA · fetched 2026-09-07
- InfiniBand Fabric Utilities - MLNX_OFED 23.07-0.5.1.2 · fetched 2026-09-07
- QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Adaptive routing, SHIELD and hash-based forwardingElsewhere in this course · Same ground: mlnxos, ufm and ibdiagnet
- Who's who in an InfiniBand subnetElsewhere in this course · Same ground: election, evidence and E-switch
- DPF: DPUs at Kubernetes scaleDOCA course · Same ground: install, modes and Scale-out