Link surgery and PHY triage

1ibportstate: the sharpest tool in the box

ibportstate “allows the port state and port physical state of an IB port to be queried (in addition to link width and speed being validated relative to the peer port when the port queried is a switch port), or a switch port to be disabled, enabled, or reset.”^[1] Its man page also contains the single most important safety sentence in InfiniBand operations: “InfiniBand HCA port state may be changed locally without the knowledge of the Subnet Manager.”^[1]

Read that as an operations rule. Nothing asks the SM for permission, nothing warns you, and the SM finds out the way it finds out about a cable pull — through a trap and a sweep.^[1][9]

The supported operations are enable, disable, reset, speed, espeed, fdr10, width, query, on, off, down, arm, active, vls, mtu, lid, smlid, lmc, mkey, mkeylease, mkeyprot, and “(Default is query)”.^[1] off “change the port state to disable” and on “change the port state to enable (only when the current state is disable)”.^[1]

Three details separate a competent operator from a dangerous one.^[1]

• “speed and width are allowed on any port”, with values checked against LinkSpeedSupported, and “NOTE: Speed and width changes are not effected until the port goes through link renegotiation”. A change that appears to do nothing has in fact been armed.^[1]
• query “also validates port characteristics (link width, speed, espeed, and fdr10) based on the peer port. This checking is done when the port queried is a switch port as it relies on combined routing… This peer port validation feature of query op requires LID routing to be functioning in the subnet.” So the peer validation you want is available on switch ports and needs a working SM.^[1]
• “mkey, mkeylease, and mkeyprot are only allowed on CAs, routers, or switch port 0 (An error is generated if attempted on external switch ports).”^[1]

Working invocations from the man page, worth copying exactly: ibportstate -C qib0 -P 1 3 1 disable, ibportstate -C qib0 -P 1 3 1 enable, ibportstate -D 0 1, ibportstate 3 1 reset, ibportstate 3 1 speed 1, ibportstate 3 1 width 1, ibportstate -D 0 1 lid 0x1234 arm.^[1]

2What a BER number means before you argue about it

“The Bit Error Rate (BER) is the number of bit errors per unit time divided by the total number of transferred bits during a studied time interval.”^[2] Everything else about BER reporting is about which errors are counted and against what threshold.^[2]

The generation split decides the vocabulary: EDR devices report “effective errors” while HDR and NDR devices report “symbol errors”, and in both cases these are “the actual errors seen at the application level after error correction”.^[2] A customer comparing an EDR spine to an NDR leaf is comparing two different measurements and will conclude one of them is broken.^[2]

The default threshold is 10^-12, expressed in the tooling as decimal 1000000000000 or hex 0xe8d4a51000, and “When threshold is set to 0, all BER values are reported regardless of magnitude.”^[2] The FEC modes that appear alongside are STD-RS, STD-LL-RS, MLNX_RS_544_514_PLR, MLNX_RS_271_257_PLR and RS_FEC_544_514.^[2] When a link crosses the line, the report is a fixed string worth quoting rather than paraphrasing:^[2]

BER exceeds threshold - BER type: Symbol BER, FEC mode: [MODE], BER value = [VALUE] / threshold = [THRESHOLD]

Fabric-wide, the same data arrives through ibdiagnet --get_phy_info, which queries all ports for PHY information, with --get_ppamp for opAMP data and --show_cap_reg to dump capability registers that are hidden by default.^[7] The per-link result lands in ibdiagnet2.net_dump_ext, the “Extended fabric link dump with FEC, BER, and phy data” — that file, not a screenshot, is what you ask for.^[8]

3mlxlink: the module, the counters and the eye

mlxlink is “used to check and debug link status and related issues” and “can be used on different links and cables (passive, active, transceiver and backplane)”.^[3] The documentation calls it a tool for advanced users, which in practice means it will happily tell you things you have to interpret rather than read.^[3]

The flags that carry a triage:^[3]

Three operational facts from the same page. “mlxlink errors, warnings and notes are printed on stderr console” — so a script that captures only stdout throws away the reason the command failed.^[3] “In ConnectX-7 and later cards, configuring the loopback can be applied when the link is fully down (not in polling state)” — a loopback test on a modern card needs the link taken down first, which is itself a change.^[3] And the warning that mirrors the ibportstate one: do not disable ports connecting hosts to unmanaged switches using --port_state dn.^[3]

Sequence the three read-only calls deliberately. -m answers “what is plugged in” — vendor, part number, temperature. -c answers “how is it performing” — physical counters and BER. --rx_fec_histogram answers “how much margin is left” — where the corrected errors sit across the bins.^[3] Only --amber_collect produces the CSV you would attach for someone else to analyse.^[3]

4The switch side, and the order that ends in a cable swap

NVOS exposes per-port link diagnostics in three code families: PHY Firmware Indication (codes 0-1023) covering auto-negotiation failures, link training problems, signal integrity issues and cable compliance mismatches; Firmware Management Issues (codes 1024-2047) covering unplugged cables, unsupported modules, temperature shutdowns and power budget exceeded; and Link Down Reason Indication (codes 0-43).^[4] Named reasons from that last table include Hi_SER_or_Hi_BER, Block_Lock_loss and FEC_sync_loss, alongside management and thermal events.^[4] The entry points are nv show interface <id> link phy-diag and nv show interface <id> link phy-detail, next to nv show interface <id> link state and link counters.^[5]

That code family split is the diagnosis in miniature. Hi_SER_or_Hi_BER and FEC_sync_loss point at the medium; “temperature shutdown” and “power budget exceeded” point at the cage and the environment; an unplugged-cable indication points at somebody’s hands.^[4]

So the order for a flapping link is:^[6][7][3][4]

1. iblinkinfo -d — which ports have a port in the Down state right now.^[6]
2. ibdiagnet --get_phy_info, then read ibdiagnet2.net_dump_ext for FEC and BER per link.^[7][8]
3. mlxlink -d <mst> -p <port> -m -c for module identity, temperature and BER on the suspect port.^[3]
4. Map the NVOS link-down reason code on the switch side.^[4]
5. Only then swap the cable.^[4]

Do not skip step 1, and do not misread the SM’s logs on the way: repeated heavy sweeps are triggered by a trap or a detected topology change, so their recurrence is the SM reacting to a link that keeps flapping rather than the cause of it.^[9]

One Dell-specific stop sign belongs in the same checklist. If the only evidence is an iDRAC inventory screen showing a ConnectX-7 NDR200 card as SDR, there is nothing to triage: the UEFI and PXE drivers configure the link speed to SDR when the cable supports it so the link comes up quickly at boot, and Dell’s resolution is verbatim “This is expected behavior and is cosmetic in the iDRAC. This can be safely ignored as the card functions at and reflects Next Data Rate (NDR) speeds within the operating system.”^[10] The proof is ibstat in the OS, and the KB number is 000221452.^[10][11]

# 1. What is actually down now, fabric-wide.
iblinkinfo -C mlx5_0 -P 1 -d

# 2. Per-link FEC and BER, from one sweep.
ibdiagnet -o /tmp/phy --get_phy_info
grep -i "rack4-leaf" /tmp/phy/ibdiagnet2.net_dump_ext

# 3. The module on the suspect port (read-only).
mst start
mlxlink -d /dev/mst/mt4129_pciconf0 -p 1 -m -c 2>&1 | tee /tmp/mlxlink-p1.txt

ibportstate -C mlx5_0 -P 1 <lid> <port> ________       # validates against the peer
ibdiagnet -o /tmp/assert --ls ________ --lw ________    # assert the design fabric-wide
mlxlink -d <mst> -p <port> --________                   # where do the corrected errors sit