Routing engines and how OpenSM picks a path

1The -R list is a fallback chain, not a preference

OpenSM offers ten routing engines: minhop, updn, dnup, ftree, lash, dor, torus-2QoS, dfsssp, sssp and nue, and the man page’s supported list also includes file.^[1][2] You select them with -R, and the important sentence is about ordering: multiple routing engines can be specified separated by commas so that a specific ordering of routing algorithms will be tried if earlier routing engines fail, and if all configured routing engines fail, OpenSM will always attempt to route with Min Hop unless no_fallback is included in the list.^[1] MinHop is also what you get when no engine is given at all.^[1]

That single sentence explains most “why is this fabric slow” tickets on a tree. A customer configures ftree, the tree does not qualify, OpenSM falls through to MinHop, the log still says SUBNET UP, every ibstat reads Active, and the fabric behaves like an unbalanced mesh under a shift pattern. Nothing is broken; the engine you asked for simply did not run.^[1]

NVIDIA layers its own engine names on top through UFM. The appliance CLI lists them verbatim as ar-dor, ar-ftree, ar-torus, ar-updn, chain, dfp, dfp2, dor, file, ftree, minhop, pqft, torus-2QoS, updn, set with a command such as ib sm routing-engines ar-updn.^[4] UFM’s shipped SM default is routing_engine = ar_updn, which is the opposite of the upstream feel of the tool.^[3] So “what is the default engine” has two correct answers and you must ask who runs the SM before giving either.^[2][3]

flowchart TD; S(["Customer fabric"]) --> Q1{"Is the shape a tree?"}; Q1 -->|"pure fat tree, all CAs at one rank"| F["ftree or NVIDIA ar_ftree - feed -a root_guid_file and -u cn_guid_file"]; Q1 -->|"tree-like but irregular or partly cabled"| U["updn or ar_updn - give -a root_guid_file, do not trust the statistical root autodetect"]; Q1 -->|"mesh or hypercube, cabled as such"| D["dor - deadlock free on meshes and hypercubes"]; Q1 -->|"2D or 3D torus"| T["torus-2QoS with --torus_config - routes around failures, keeps SL values"]; Q1 -->|"arbitrary or faulty, no regular shape"| N["nue - deadlock free on any topology, --nue_max_num_vls default 1"]; Q1 -->|"DragonFly plus"| DF["dfp or dfp2 - UFM engine set only, not upstream OpenSM"]; F --> C{"Add no_fallback?"}; U --> C; D --> C; T --> C; N --> C; DF --> C; C -->|"yes"| C1["Engine failure is loud: no MinHop safety net"]; C -->|"no"| C2["Engine failure is silent: MinHop routes the fabric and SUBNET UP still appears"]

2What each tree engine demands of the cabling

MinHop is the baseline: a two-stage algorithm that builds a min-hop matrix and then assigns LFT output ports with per-port LID counters for balancing.^[1] It always works and it optimises nothing about your traffic pattern.

Up/Down is the engine for trees that are not pure. The reference says it should be used if the subnet is not a pure fat tree and one of its loops may experience a deadlock.^[1] Two properties matter in the field. First, root detection is statistical: since the algorithm is statistical, it may not find any root nodes, and if that stage cannot find any root nodes and the user did not specify a GUID list file, OpenSM defaults back to the Min Hop routing algorithm.^[1] Give it -a/--root_guid_file rather than hoping.^[2] Second, Up/Down routing does not allow LID-routed communication between switches located inside spine switch systems, so you cannot run the SM on switches other than the leaf switches of the fabric.^[1] On a customer design with a chassis spine, that sentence disqualifies an SM placement before you have drawn a single cable.

The fat-tree engine optimises for the shift communication pattern, handles non-constant K, cases where not all leaves are present, and any Constant Bisectional Ratio, and like Up/Down it prevents credit-loop deadlocks.^[1] Its rules with no root GUID file are exact: tree rank between two and eight inclusively; same-rank switches must have identical numbers of up-going and down-going port groups and identical port counts per group, where a port group is ports that are connected to the same remote switch; and all the CAs have to be at the same tree level.^[1] Supply a root GUID file and the topology need not be a pure fat tree - only rank 2 to 8 and all compute nodes at the same rank still hold.^[1] One more rule catches designers: LMC greater than 0 is not supported by fat-tree routing, and if it is specified the default routing algorithm is invoked instead.^[1]

Three GUID files shape the result: -a/--root_guid_file for roots, -u/--cn_guid_file for compute nodes, and -G/--io_guid_file with -H/--max_reverse_hops for I/O nodes that may traverse a switch the wrong way around a bounded number of times.^[2] The last one carries its own warning: it should never be used to connect nodes with high bandwidth traffic between them, because routes the other way around can in theory cause credit loops.^[1] ftree also writes opensm-ftree-ca-order.dump next to the OpenSM log, giving the compute-node order that matches the routing tables.^[1]

opensm -R ftree,updn,no_fallback -a /etc/opensm/root_guids -u /etc/opensm/cn_guids
ibroute <lid>        # this switch's linear forwarding table
dump_lfts            # every switch's LFT

3The deadlock-freedom family

Four engines exist because deadlock, not bandwidth, is the hard constraint on an irregular fabric.

LASH is Layered Shortest Path Routing: it groups shortest paths into SLs, or layers, so that no layer contains a cycle, opening a new layer when a path would deadlock and then rebalancing.^[1] It has a behaviour that surprises people debugging one direction of a flow: LASH ensures the same SL is used for all source and destination pairs in both directions, and there is no guarantee that the return path for a given destination-source pair will be the reverse of the route source-destination.^[1] Activate it with -R lash -Q; --do_mesh_analysis preconditions switch-port assignments in regular cartesian meshes, which may reduce the number of SLs required, and --lash_start_vl defaults to 0.^[2] LMC greater than 0 is not supported by LASH routing.^[2]

DOR avoids port equalisation except for redundant links between the same two switches, which is what makes it deadlock-free on hypercubes and meshes when they are actually cabled as such.^[1] Torus-2QoS is DOR-based for 2D and 3D torus, supports two QoS levels, and can route around multiple failed fabric links or a single failed fabric switch without introducing deadlocks and without changing path SL values granted before the failure; its config file comes from --torus_config.^[1][2] Nue is the general answer: 100 percent applicable and deadlock-free for any arbitrary or faulty network topology and any number of virtual lanes, including the absence of VLs.^[1] Its --nue_max_num_vls defaults to 1 to enforce deadlock-freedom even if QoS is not enabled, with 0 meaning auto-detect the fabric maximum; a higher value results in better path balancing.^[2]

The file engine is the odd one out. It just loads the LFTs from the file specified, with no reaction to real topology, and multicast is not affected by the file routing engine because that still uses min-hop tables.^[1] Treat it as a forensic and reproduction tool, not a production engine.

One consequence to carry into the rest of the module: SHARP does not accept any engine you like. Its documented requirement is tree-based topologies with updn, ar_updn, ftree or ar_ftree; DragonFly+ with dfp; and hypercube topologies with the dor routing engine with dor_hyper_cube_mode enabled.^[9] Choosing lash or nue for deadlock reasons therefore takes SHARP off the table, and that trade belongs in the design conversation, not in a later escalation.^[9]

4After the engine: stickiness, cache and balance

Routing is not recomputed as often as customers assume. OpenSM will preserve existing routing in any case where there is no change in the fabric switches unless -r/--reassign_lids is specified, and if a link is added or removed, OpenSM does not recalculate the routes that do not have to change.^[1] -A/--ucast_cache goes one step further: it prevents routing recalculation, which is a heavy task in a large cluster, when there was no topology change detected during the heavy sweep, and the man page names host reboot as the common case it covers, which would otherwise force two full recalculations.^[2] UFM ships use_ucast_cache = TRUE.^[3]

Two knobs exist purely to break unlucky symmetry. --port-shifting counteracts the case where, in some fabrics and particularly cluster environments, routes commonly align and congest with other routes due to algorithmically unchanging traffic patterns.^[2] --scatter-ports randomises port selection instead of round-robin, and if the value is 0, which is the default, the scatter ports option is disabled; MLNX-OS adds that it is recommended to use a prime number for the seed.^[2][5] UFM does not ship the upstream default here either: its table shows scatter_ports = 8.^[3] -X/--guid_routing_order_file fixes the order in which port GUIDs are routed for MinHop and Up/Down, which is how storage or service nodes get first pick of link balancing.^[2][5]

Then prove it. ibroute dumps a switch’s unicast forwarding table, with a default range of all valid entries from 1 to FDBTop, -a to show invalid entries and -M to switch to multicast.^[8] dump_lfts does the same for every switch.^[2] Fabric-wide, ibdiagnet -r provides a report of the fabric qualities and validates unicast routing, static and adaptive, plus multicast, and its credit-loop success string is -I- no credit loops found.^[6] The evidence files are ibdiagnet2.fdbs for unicast FDBs and ibdiagnet2.mcfdbs for multicast, and note that ibdiagnet2.fdbs is one of the files disabled by default, so it needs --enable_output.^[7]

opensm -R ftree,updn,no_fallback \
  -a /etc/opensm/root_guids.conf \
  -u /etc/opensm/cn_guids.conf \
  -A -F /etc/opensm/opensm.conf
ibroute 3                 # one switch LFT, by LID
dump_lfts                 # every switch LFT
ibdiagnet -r              # routing validation, credit loops

5Choosing on a customer call

The analysis skill here is turning a cabling drawing into an engine plus its inputs plus its failure mode, in that order.