Reading a host's IB state in 60 seconds
S1·E5The card the technician is already holding · Customer data center, acceptance day, handover at four
Builds on: Speeds, widths, and the link that says SDR
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Run the four-command host triage in order and state what each command proves that the previous one does not.
- Distinguish port state from port physical state and read Initializing as an SM fault rather than a cable fault.
- Apply NCCL's four acceptance criteria to a port and decide whether the host is ready before any fabric work starts.
- Locate a port's raw and hardware counters in sysfs without needing a diagnostic package installed.
Episode 5 — The card the technician is already holding
Twelve of the sixty-four compute nodes are failing the acceptance script, and a technician is standing in the aisle with a replacement adapter out of its bag. Procurement wants a number for how many spares to order. The Dell SE wants to know whether this is a platform fault, because that line of the promise spreadsheet has his name on it. The night-shift operator stayed past his shift and has the label maker loaded. You have not touched a keyboard yet, and you have about a minute before somebody unseats a card that is perfectly good.
You ask for one command, on one failing host and one healthy one: ibstat. It needs nothing but the local driver, and it displays LID, SMLID, port state, link width active and port physical state in a single screen.[1]
One screen is enough because of how this fabric is built. On InfiniBand a trained link is not yet a usable link — a port has no LID and no place in any forwarding table until the subnet manager programs it — so the two facts are reported separately. The kernel’s stable sysfs ABI keeps state as DOWN, INIT, ARMED or ACTIVE and phys_state as Sleep, Polling, LinkUp in different files for exactly that reason.[6] NCCL’s own troubleshooting page opens its InfiniBand section the same way, checking the subnet manager with sminfo before anything else.[5]
A link that is up is a cable fact; a port that is Active is a subnet manager fact.
Four commands, in order, cheapest first.
1Four commands, in this order
Almost every InfiniBand host ticket is settled or correctly escalated by four commands, and the order matters because each one costs more access than the last.
ibstat first, because it needs nothing but the local driver. It “is a binary which displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state.”[1] It “is similar to the ibstatus utility but implemented as a binary rather than a script. It has options to list CAs and/or ports and displays more information than ibstatus.”[1] Its options are -l/--list_of_cas, -s/--short and -p/--port_list, with positional ca_name and portnum to narrow it down - so ibstat -l, ibstat -p and ibstat mthca0 2 are all valid.[1]
ibstatus second, for the rate. It “is a script… Output includes LID, SMLID, port state, link width active, and port physical state”, invoked as ibstatus [-h] [devname[:port]]....[2] NVIDIA’s MLNX_OFED page words the same tool slightly differently and calls out what you actually want from it: “LID, SMLID, port state, port physical state, port width and port rate”.[4] That is the practical division of labour between the two - the binary for detail, the script for the rate string.[1][4]
sminfo third. It “issues and dumps the output of an sminfo query in human readable format. The target SM is the one listed in the local port info or the SM specified by the optional SM LID or by the SM direct routed path.”[3] NCCL’s own troubleshooting page opens its InfiniBand section with it: “InfiniBand fabrics require a Subnet Manager (SM) to be running. Check SM status with: sudo sminfo”, and “If sminfo fails or shows no SM, ensure the SM is running on at least one node (for example, the opensm service).”[5]
ibdiagnet last, because it is the first step that touches the whole fabric. Its ibdiagnet2.sm output file lists every Subnet Manager discovered - the evidence you ask a customer for when you suspect two masters.[9]
One host, 60 seconds: you need LID, SMLID, port state, link width active and physical state straight from the local IB driver — from a binary, not a shell script.
A customer sentence. Pick the one tool that answers it.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo2Two states, and the one that is misread most
An InfiniBand port has a physical state and a logical state, and they answer different questions. The kernel’s stable sysfs ABI documents them as separate attributes: ports/<port-num>/state is “Port state (DOWN, INIT, ARMED, ACTIVE or ACTIVE_DEFER)” and ports/<port-num>/phys_state is “Port physical state (Sleep, Polling, LinkUp, etc)”.[6] Alongside them the ABI gives ports/<port-num>/rate as “Port data rate (active width * active speed)”, plus lid, lid_mask_count and sm_lid.[6] Those files are your fallback on a customer box where infiniband-diags was never installed.[6]
Read them as a pair:
| Physical state | Logical state | What it means | Who owns it |
|---|---|---|---|
Not LinkUp (Polling, Sleep) |
DOWN |
The two ends never trained | Physical layer - cable, module, port disabled |
LinkUp |
INIT |
Trained, but nothing has configured the port | Subnet Manager |
LinkUp |
ACTIVE |
Trained and programmed | Nothing yet - check rate and width next |
The middle row is the one that costs people afternoons. A port at physical LinkUp and logical Initializing has a perfectly good cable and no manager. The correct next command is sminfo, and the correct next question is whether opensm is running anywhere - NCCL’s guidance is exactly that.[5] A healthy host-side manager writes SUBNET UP to both /var/log/messages and /var/log/opensm.log; if it is running and the port is still Initializing, read those logs before doing anything else.[8]
3NCCL's acceptance criteria, and where the counters live
NCCL publishes the shortest useful definition of a healthy IB port, and it is worth adopting verbatim because it is what a customer’s own AI stack will check. Read ibstat or ibstatus and require: port state Active (not Down, Init or Armed); physical state LinkUp; link layer matching what you expect - InfiniBand versus Ethernet; and “Rate matches your fabric specification without unexpected downshift”.[5]
The link-layer criterion is not filler. A ConnectX or BlueField port can be in Ethernet mode, in which case every InfiniBand diagnostic below this line is meaningless on it and the conversation belongs to RoCE instead.[5] Check it before anything else on an unfamiliar host.
The rate criterion is where lesson 3 plugs in. If the OS reports the expected rate and only iDRAC says SDR, that is Dell KB 000221452 and it is cosmetic - “This is expected behavior and is cosmetic in the iDRAC” and the card “functions at and reflects Next Data Rate (NDR) speeds within the operating system”.[10] If the OS also reports a downshift, you have a real finding and the tools from lesson 3 apply.[10]
For counters, two locations matter. In sysfs, RDMA counters are exposed at /sys/class/infiniband/<device>/ports/*/hw_counters/ and /sys/class/infiniband/<device>/ports/*/counters - readable with no package installed.[6] Via the fabric, perfquery with no arguments reads the local port’s performance counters, which is what makes it usable on a single card with no switch attached.[11] NCCL points at three specific counters as cabling and switch indicators when read with sudo perfquery -x <lid>: SymbolErrorCounter, LinkErrorRecoveryCounter and LinkDownedCounter.[5] Module 4 does counters properly; for the 60-second read, non-zero and growing is the signal, and the absolute number means nothing without knowing when they were last cleared.[11]
Host does not see the BlueField
For each reading, the verdict and the single next command. The readings below are field-value combinations built from the documented attributes, not captured transcripts.
1. No device. ibstat -l lists nothing; /sys/class/infiniband/ is empty.
Verdict: this is not an InfiniBand question yet - either the driver is not loaded or the card is not enumerated. Next: lspci | grep -i mellanox. If the card is on the bus, the fault is host software; if it is not, the fault is hardware or the slot.[1]
2. Physical state Polling, state Down. Next: nothing on the fabric. The link never trained, so this is the physical layer - cable seated, module supported, remote port enabled.[6]
3. Physical state LinkUp, state Initializing. Verdict: trained, unmanaged. Next: sminfo; if it returns nothing, start opensm on one node or enable the switch SM, then re-read.[5][8]
4. State Active, physical LinkUp, link layer InfiniBand, width 4X, rate matching the fabric spec. Verdict: all four NCCL criteria met - the host is not the problem.[5] Next: move up a layer, to the fabric or the application.
5. State Active, physical LinkUp, but the rate is one generation below the purchase order and iDRAC also said SDR. Verdict: the OS agrees with iDRAC, so this is not the cosmetic KB case - it is a real downshift.[10] Next: smpquery -x portinfo <lid> 1 for the negotiated versus supported extended speed, then the switch-side peer validation from lesson 3.
6. Link layer Ethernet. Verdict: the port is not in InfiniBand mode. Next: stop applying IB triage. Confirm with cat /sys/class/infiniband/<dev>/ports/1/link_layer and hand the question to the RoCE playbook.[5][6]
Three more readings. Fill in the blanks.
7. State Active, physical LinkUp, width 1X on a card cabled for 4X.
Verdict: ____________ is intact and ____________ is degraded, which points at ____________ rather than ____________. Next command: ____________.
8. ibstat looks perfect on this host, but sminfo returns a different SM GUID than it did an hour ago.
Verdict: ____________. Next: ask for ____________ from two hosts and the ____________ file, because the thing you must establish is ____________.
9. State Active everywhere, but /sys/class/infiniband/mlx5_0/ports/1/counters/link_downed is large.
Verdict: the port is up now, and the counter is ____________, so the number alone tells you ____________. The question you must ask the customer first is ____________.
Problem - solve alone, 10 minutes. A Dell customer opens a case: “Two of our eight rails are slow after last night’s maintenance. iDRAC shows SDR on all of them.”
Write the reply. It must contain:
- The one-line verdict on the iDRAC observation and the KB number that supports it.
- The exact four commands you want run on one affected host, in order, with one sentence each on what that command will prove.
- A decision rule stated in advance: which output would make this a fabric escalation, which would make it a host issue, and which would close the case as no-fault.
- One sysfs read that works even if the customer’s image has no
infiniband-diagsinstalled.
Acceptance criteria: every command you name appears in this lesson with a citation; your decision rule is written before you see the outputs; and your reply asks for no hardware action of any kind.
Episode 5 — Case closed, four o'clock
All twelve failing hosts read physical state LinkUp with port state Initializing; the healthy host reads Active.[6] Trained cables, unconfigured ports — so the next command is sminfo, and the next question is whether opensm is running anywhere, which is precisely what NCCL prescribes.[5] It is not, since the morning’s reboot; started again, the twelve go Active with an hour to spare. Acceptance passes, procurement orders no spares, and the operator prints one last label, for the master node, reading THIS ONE IS IN CHARGE. The network lead closes a notebook whose first page now reads, in order: one master; GUIDs name, LIDs address; ask which layer; read the partition; state before hardware.
Lab
Mutating. Step 6 stops and restarts a subnet manager. Do this on a Dell-lab host only, and only when no one else is using that fabric.
- Pre-flight inventory, saved to a file you can diff against later:
ibstat > /tmp/ib-before.txt; ibstatus >> /tmp/ib-before.txt; sminfo >> /tmp/ib-before.txt 2>&1; ibv_devinfo -v | head -40 >> /tmp/ib-before.txt. Expected: a baseline containing state, physical state, width, rate and the SM the port sees. - Read the same values from the driver’s sysfs and confirm they agree:
cat /sys/class/infiniband/mlx5_0/ports/1/{state,phys_state,rate,lid,sm_lid,link_layer}. Expected:ACTIVE,LinkUp, a rate string, andInfiniBandas the link layer. Iflink_layersaysEthernet, stop - this port is not in IB mode. - Apply the four NCCL criteria out loud against your own output and record a pass or fail for each. Expected: four passes on a healthy lab port.
- Dump the counters:
ls /sys/class/infiniband/mlx5_0/ports/1/counters/andcat /sys/class/infiniband/mlx5_0/ports/1/counters/{link_downed,symbol_error}where present, then compare withperfqueryrun with no arguments for the local port. Expected: the two agree. Record the values with a timestamp - a counter with no timestamp is not evidence. - Confirm which SM this port is being managed by:
sminfo. Expected: one SM LID, GUID, priority and state. Write them down. - Mutating - the whole lesson in one toggle. Stop the subnet manager:
systemctl stop opensm(orkill $(cat /var/run/opensm.pid)if you started it by hand). Wait for the next sweep, then re-runibstat. Expected: the port falls toInitializingwhile physical state staysLinkUp- a fabric fault with a perfectly healthy cable, produced on demand. Rollback:systemctl start opensm, wait forSUBNET UPin/var/log/opensm.log, then re-runibstatand diff against/tmp/ib-before.txtuntil the port isActiveagain with the same LID and rate. - Optional: repeat step 6 while a
perfqueryloop is running and watch which counters move. Expected: state changes are not error counters - a clean transition should not create symbol errors. Rollback: as step 6.
Transcript and man-page drill. No InfiniBand hardware.
- In a container:
apt-get update && apt-get install -y infiniband-diags man-db. Expected:man ibstatandman ibstatusboth open. - Read both pages and list three fields or capabilities the binary offers that the script does not. Expected: your list is grounded in the wording “displays more information than ibstatus” plus the
-l,-sand-poptions. Write down which page you would cite to a customer for each claim. - Work the six readings in the Worked block above without looking at the verdicts, then check yourself. Expected: six different next commands. If two of your answers are the same command, re-read which state changed between the two readings.
- Build the state table from the kernel ABI rather than from this lesson: find the documented values for
stateandphys_stateand write the pair table yourself. Expected: DOWN, INIT, ARMED, ACTIVE, ACTIVE_DEFER for state; Sleep, Polling, LinkUp among others for phys_state. - Write the sysfs-only version of the triage - the four
catcommands you would send a customer whose image has no diagnostics installed. Expected:state,phys_state,rateandsm_lidunder/sys/class/infiniband/<dev>/ports/1/. - Finish with the NCCL criteria: write them out from memory, then check against the page. Expected: four criteria, one of which is the link layer - the one people forget.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell lab technician, in four sentences, the order you read a host's InfiniBand state in and why stopping the subnet manager is the fastest way to teach someone the difference between a cable problem and a fabric problem.
Sources
Facts in this lesson were checked against rdma-core man pages ibstat(8) ibstatus(8) perfquery(8) (master), MLNX_OFED 23.07-0.5.1.2 and 5.6-2.0.9.0 fabric utilities pages, NCCL 2.30.7 networking troubleshooting, Linux kernel sysfs-class-infiniband ABI, opensm(8) - fetched 2026-09-07. Dates are when each page was fetched.
- ibstat(8) man page source - rdma-core master · fetched 2026-09-07
- ibstatus(8) man page source - rdma-core master · fetched 2026-09-07
- InfiniBand Fabric Utilities - MLNX_OFED 23.07-0.5.1.2 · fetched 2026-09-07
- InfiniBand Fabric Utilities - MLNX_OFED 5.6-2.0.9.0 · fetched 2026-09-07
- Networking Troubleshooting - NCCL 2.30.7 documentation · fetched 2026-09-07
- Linux kernel ABI - sysfs-class-infiniband (stable) · fetched 2026-09-07
- IP Over InfiniBand - DOCA 3.5.0 · fetched 2026-09-07
- opensm(8) man page source - linux-rdma/opensm master · fetched 2026-09-07
- ibdiagnet Dump Files - IBUtils2 Utility Documentation · fetched 2026-09-07
- Dell KB 000221452 - Link Speed of NVIDIA ConnectX-7 Single Port NDR200 InfiniBand Card Shows as SDR · fetched 2026-09-07
- perfquery(8) man page source - rdma-core master · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- The counter map: sysfs, ethtool and rdma statisticRoCE course · Same ground: sysfs, tools and hw_counters
- GIDs: how a connection picks its RoCE versionRoCE course · Same ground: addressing, sysfs and tools
- Triage in order: symptom to counter to causeRoCE course · Same ground: tools, NCCL_IB_TC and triage