Escalation bridge call: five InfiniBand tickets

1How the first ten minutes work

An InfiniBand escalation is a layer-selection problem. The customer’s symptom - the job is slow, the job hangs, the port is down - is compatible with a cable, a subnet manager, a partition, a routing decision and an application bug, and your only advantage is that some checks eliminate an entire layer for free.

Three rules make the call go well. First, run the cheapest decisive check before the likeliest one. Second, ask for artefacts rather than descriptions: ibdiagnet writes the same files on their fabric as on yours - ibdiagnet2.log, ibdiagnet2.db_csv, ibdiagnet2.pkey, ibdiagnet2.sm, ibdiagnet2.net_dump_ext - and -f/--load_from_file reloads a customer’s db_csv so you can re-run the analysis without touching their fabric.^[4][7] Third, change nothing on a live fabric to test a theory. The ibportstate man page states the reason: “InfiniBand HCA port state may be changed locally without the knowledge of the Subnet Manager.”^[15]

Ask three questions before any command runs. Since when, because counters are cumulative from the last clear. Which node or nodes, because a fault on one rail and a fault on one node are different problems. And what changed, because a maintenance window is the most common answer and nobody volunteers it.

2Ticket 1 and ticket 2: the port and the hang

Ticket 1 - a port will not go Active. Read ibstat first: it “displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state.”^[2] If the physical state is not LinkUp, this is the physical layer - seating, module support, PHY data. If physical is LinkUp and the logical state is Initializing, no SM has programmed the port; check sminfo, which “Issues and dumps the output of an sminfo query in human readable format”, then start opensm on a node or enable the switch SM.^[12][1][9] If the state is Active but the rate is low, the link negotiated down: ibportstate ... query “also validates port characteristics (link width, speed, espeed, and fdr10) based on the peer port” when the queried port is a switch port, iblinkinfo shows widths and speeds fabric-wide, and ibdiagnet --ls and --lw assert an expected speed and width across the whole fabric.^[15][3][7]

And one Dell-specific stop: if iDRAC shows a ConnectX-7 InfiniBand card at “Single Data Rate (SDR)” while the OS shows the real rate, the answer is Dell KB 000221452 - “This is expected behavior and is cosmetic in the iDRAC. This can be safely ignored as the card functions at and reflects Next Data Rate (NDR) speeds within the operating system.”^[8] Send the KB and the ibstat output together and the ticket closes in one reply.

Ticket 2 - the fabric is up and the application hangs. Everything is Active, nothing communicates. Read ibdiagnet2.pkey before you theorise: partition membership is the most common cause, and the pkey tables are the ground truth rather than the partitions file someone believes is in effect.^[4] Then ibdiagnet -r --r_opt check_sl, which validates that “all SL2VL mappings don’t route to VL15” - a service level mapped to VL15 is silently dropped.^[5] Then ibdiagnet -r for credit loops, whose success string is -I- no credit loops found.^[5] Then ibqueryerrors --data -r, remembering that “The default threshold is zero (0)” and that -r/--report-port adds LID, port, link speed setting and the remote node description to every line.^[6] PortXmitDiscards concentrated on one switch port is congestion or routing, not a cable.^[6] Finally ib_write_bw between exactly the two hosts that hang: if it runs at rate, the fabric path works and the ticket moves up the stack.^[13]

3Ticket 3 and ticket 4: the subnet manager misbehaving

Ticket 3 - two SMs, or a failover that went wrong. Start with sminfo from a node, but know its limit: it reports “The target SM is the one listed in the local port info”, so it tells you which SM your port is pointing at, not how many exist.^[12] The proof is ibdiagnet2.sm, the file that lists every subnet manager seen in the fabric.^[4] Mastership is decided by priority then GUID: opensm’s -p/--priority runs “from 0 (default and lowest priority) to 15 (highest)” and “This will effect the handover cases, where master is chosen by priority and GUID.”^[1] A managed switch joins the same election with ib smnode <name> sm-priority <priority>, and its embedded SM is disabled by default until ib smnode <name> enable.^[9] After any handover, confirm SUBNET UP in /var/log/opensm.log on the new master - the log line that means the fabric was actually configured, not merely that a process started.^[1]

Two facts turn this ticket from a debate into a decision. The MLNX-OS embedded SM does not support adaptive routing, fault routing, congestion control or SHARP, and is documented for fabrics “up to 2048 nodes on x86 based systems”.^[9] And the upstream default priority is 0 while UFM ships its SM at a different value - always read the running configuration rather than assuming a default.^[1]

Ticket 4 - repeated heavy sweeps. A heavy sweep runs when a trap arrives or a topology change is detected, and SIGHUP triggers one manually “as if a trap was received or a topology change was found”.^[1] So recurring heavy sweeps in opensm.log mean the topology keeps changing. The cause is a flapping link or a rebooting node, and the sweeps are the symptom. Confirm with iblinkinfo -d, which prints “only nodes which have a port in the ‘Down’ state”, and with ibdiagnet --get_phy_info, which queries “all ports for phy information” and feeds ibdiagnet2.net_dump_ext with FEC, BER and PHY data per link.^[3][14][4]

You can mitigate churn with -A/--ucast_cache, which “prevents routing recalculation (which is a heavy task in a large cluster) when there was no topology change detected during the heavy sweep” and explicitly handles the host-reboot case.^[1] Say the word mitigate out loud on the call: the cache reduces the cost of the churn and does not stop the link flapping.

4Ticket 5: SHARP is not working

This ticket is a preconditions ticket, and the order is fixed because each step is cheaper than the next.

First, the routing engine. NVIDIA’s requirement list is explicit: “Tree based topologies: updn, ar_updn, ftree, ar_ftree; DragonFly+ topology: dfp; Hypercube topologies: dor routing engine with dor_hyper_cube_mode enabled.”^[10] Any other engine and no aggregation trees are built at all.

Second, who runs the SM. The MLNX-OS embedded SM lists SHARP among its unsupported features, so a SHARP customer needs UFM or a host SM stack that supports it.^[9] The SHARP prerequisites say the same thing from the other side: “Make sure onboard Subnet Manager is disabled in the managed switches”, and the Aggregation Manager “is a central entity running on a dedicated server with a master Subnet Manager. This dedicated server cannot serve as a compute node.”^[10]

Third, IPoIB. NVIDIA requires an “Enabled IPoIB interface in compute servers in order to enable using UD multicast for result distribution in SHARP.”^[10] A fabric with no IPoIB broadcast group has no result-distribution path, and the symptom is a job that creates and then stalls.

Only then run the smoke test: sharp_hello -d mlx5_0:1 -v 3, which “Verifies that a SHARP client can communicate with sharp_am and successfully create a SHARP job” and prints a tree_info line naming an LLT tree followed by Test Passed.^[11] Setting SHARP_COLL_ENABLE_SAT=1 adds a streaming-aggregation tree line.^[11]

And the sentence that ends the argument, quoted verbatim from NVIDIA: “While ibdiagnet can be used to check for general fabric errors, it does not verify communication between libsharp and sharp_am, nor whether SHARP jobs can be successfully created.”^[11] A clean ibdiagnet is not evidence that SHARP works. The fabric-side view you can still use is ibdiagnet --sharp, which collects SHARP configuration into ibdiagnet2.sharp.^[7][4]

5The reply, and the evidence bundle

Your written reply has three parts and they belong in this order: what the evidence shows, what you conclude, and what you need next. Not the other way round - a conclusion first invites a debate about your conclusion instead of about the evidence.

The bundle you request is the same for four of the five tickets, which is why it is worth having as a saved snippet: run ibdiagnet and send the whole output directory; send ibstat and ibstatus from the affected hosts; send sminfo from any node; and state when counters were last cleared.^[7][2][12] From that one directory you get the partition tables, the SM list, the per-link PHY and BER data, the counters and the discovered topology - and -f/--load_from_file lets you re-analyse it offline as many times as you like.^[4][7]

For ticket 5 add two things the bundle does not contain: the SM’s routing engine setting, and the output of sharp_hello -d <dev>:1 -v 3 from one compute node.^[10][11]

Two habits separate a good bridge call from a long one. Name the artefact, never the theory - “send me ibdiagnet2.pkey” beats “I think it is a partition problem”, because the first ends in a fact and the second ends in an argument. And write down the check you did not run and why; on the second call, the list of eliminated layers is the most valuable thing you own.