Counters and errors that mean something
S4·E2Seven digits and a purchase order · Bridge call, 22:10 Monday, joined from your kitchen table
Builds on: Discovering a fabric: ibnetdiscover and iblinkinfo
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Read PortCounters with perfquery and convert the Data components into real bytes.
- Separate physical-layer counters from MAD-pressure and congestion counters before proposing a fix.
- Sweep a whole fabric with ibqueryerrors and tune what it reports with a threshold file.
- Reset the right counter block on the right port and measure a rate over a known window.
Episode 2 — Seven digits and a purchase order
The operator shares his screen: rack 4, leaf port 17, SymbolErrorCounter with seven digits on it — the link the network lead read out this morning. Procurement has drafted an order for forty cables and wants a lead time tonight. The Dell SE, coffee cold, offers to get agreement on the call so it ships; his spreadsheet has grown a row. The acceptance benchmark takes the fabric at midnight.
You ask when those counters were last cleared. Nobody knows. That question is the whole meeting, because an InfiniBand port counter is cumulative from the last clear: seven digits can be years of a healthy link or forty minutes of a dying one, and the screenshot cannot tell you which.
Counters exist because a port cannot describe its own health; it keeps tallies and lets someone else do the arithmetic. Every device carries a Performance Management Agent, and perfquery “uses PerfMgt GMPs to obtain the PortCounters (basic performance and error counters)” and its sibling attributes from the PMA at the node and port you name.[1] The kernel exposes the same values per port in sysfs, where each file “contains the corresponding value from the port’s Performance Management PortCounters attribute, as described in the InfiniBand Architecture Specification”.[10] NCCL’s troubleshooting guidance is built on exactly these names: run sudo perfquery -x <lid> and look for non-zero SymbolErrorCounter, LinkErrorRecoveryCounter and LinkDownedCounter.[3]
A counter with no start time is a number; with a start time it is a measurement.
Before you agree to forty of anything, you go back to what perfquery is actually reading.
1What perfquery actually reads
perfquery talks to the Performance Management Agent on a node: it “uses PerfMgt GMPs to obtain the PortCounters (basic performance and error counters), PortExtendedCounters, PortXmitDataSL, PortRcvDataSL, PortRcvErrorDetails, PortXmitDiscardDetails, PortExtendedSpeedsCounters, or PortSamplesControl from the PMA at the node/port specified.”[1] The synopsis is perfquery [options] [<lid|guid> [[port(s)] [reset_mask]]], and with no arguments at all it reads the local port — which is the one form that works on a host with a single card and no switch.[1]
Two facts in that man page cost more support hours than anything else in this module.[1]
The first is units: “In PortCounters, PortCountersExtended, PortXmitDataSL, and PortRcvDataSL, components that represent Data (e.g. PortXmitData and PortRcvData) indicate octets divided by 4 rather than just octets.”[1] Anyone reconciling a benchmark against a counter without that factor concludes the fabric moved a quarter of the traffic it moved.[1]
The second is port selection: “Inputting a port of 255 indicates an operation be performed on all ports”, and “For PortCounters, ExtendedCounters, and resets, multiple ports can be specified by either a comma separated list or a port range.”[1] So perfquery -a 32 1-10 and perfquery -l 32 1,4,8 are both legal, and -a/--all_ports “show aggregated counters for all ports of the destination lid”, falling back to iterating ports when the device does not support AllPortSelect.[1]
Flags worth memorising, because each one selects a different attribute rather than a different format: -x/--extended (and note “extended port counters attribute is optional”), -X/--xmtsl, -S/--rcvsl, -D/--xmtdisc, -E/--rcverr, -T/--extended_speeds, -c/--smplctl.[1] UFM’s own one-line description of the tool is the honest summary: perfquery “Dumps (and optionally clears) the performance counters”.[7]
iblinkinfo — print only the nodes that have a port in the Down state.
You know the tool. Pick the flag that makes it answer the question.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo2Reading, clearing, and measuring a rate
Counters are cumulative from the last clear, which means an absolute value is not evidence. The only defensible statement about a fabric is a rate over a known window, and building one takes three commands: record, clear, re-read.[1]
perfquery -x 32 1 # extended counters, LID 32 port 1: record these
perfquery -R 32 1 0x0fff # reset ONLY the error counters of port 1
# ... run the workload for a known interval ...
perfquery -x 32 1 # re-read; the deltas are now errors per intervalThe mask is the part people get wrong. The man page’s own examples are perfquery -R 32 2 **0x0fff** = “reset only error counters of port 2” and perfquery -R 32 2 **0xf000** = “reset only non-error counters of port 2”.[1] -r/--reset_after_read reads and then clears in one step, -R/--Reset_only clears without printing.[1]
Clearing is the one irreversible act in this lesson. There is no undo: whatever history was in those counters is gone, so the rollback is procedural — write the values down before you clear them, and say in the ticket when you cleared and what the pre-clear values were.[1] The fabric-wide equivalents live in ibdiagnet: --pc “Resets all fabric IB spec compliant port counters” and --scr “Resets all the Mellanox counters”, with --sc reporting the Mellanox set.[5]
For the QoS and congestion attributes there is a long tail of optional flags — --oprcvcounters, --flowctlcounters, --vloppackets, --vlopdata, --vlxmitflowctlerrors, --vlxmitcounters, --swportvlcong, --rcvcc, --slrcvfecn, --slrcvbecn, --xmitcc, --vlxmittimecc — which exist when the device implements them and are the right tool only after you know which class of problem you have.[1] Host-side, the same numbers surface in sysfs: every file under /sys/class/infiniband/<device>/ports/<port-num>/counters/ “contains the corresponding value from the port’s Performance Management PortCounters attribute, as described in the InfiniBand Architecture Specification”, and the optional hw_counters/ subdirectory carries the vendor’s own set — “they may match some of the counters in the counters directory, but they often include many other counters”. That pair is what a collector should scrape.[10]
3ibqueryerrors: the whole fabric at once
perfquery asks one node. ibqueryerrors asks all of them: “The default behavior is to report the port error counters which exceed a threshold for each port in the fabric. The default threshold is zero (0).”[2] On a real fabric that first run is unreadable, and the fix is not to stop running it but to tell it what you consider normal.[2]
Three mechanisms do that.[2]
-s/--suppress <err1,err2,...>drops named counters, and-c/--suppress-commondrops “some of the common ‘side effect’ counters. These counters usually do not indicate an error condition and can be usually be safely ignored.”[2]--threshold-file <filename>, default<IBDIAG_CONFIG_PATH>/error_thresholds, inname=vallines with#comments. The documented example is the shape you copy:[2]
# Define thresholds for error counters
SymbolErrorCounter=10
LinkErrorRecoveryCounter=10
VL15Dropped=100- Scope filters
--switch,--caand--routerrestrict which node types are swept.[2]
For output detail, -r/--report-port adds “LID, port, external port (if applicable), link speed setting, remote GUID, remote port, remote external port (if applicable), and remote node description information” — that is what turns a counter into a cable someone can find.[2] --data adds transmit and receive data counters, --counters prints data counters only, --details adds “receive error and transmit discard details”.[2]
Two cautions, both in the man page. --skip-sl “Use the default sl for queries. This is not recommended when using a QoS aware routing engine as it can cause a credit deadlock.”[2] And the clears, --clear-errors/-k and --clear-counts/-K, carry the warning that “clearing data or error counters will occur regardless of if they are printed or not” — so a run scoped to one switch can still wipe the fabric’s counters if you are careless with the flags.[2] Exit status follows the same three-state convention as iblinkinfo: -1 scan failed, 0 clean, 1 beyond threshold.[2]
4Which counter class are you looking at?
Counter names are documented on NVIDIA’s pages; their prose definitions are not, at least not on an NVIDIA page that can be fetched. NVIDIA’s own counter-definition articles did not return body text when the research for this course was collected, so the definitions below are quoted from the Linux kernel’s stable sysfs ABI, which states that each of these files carries the value of the port’s PortCounters attribute as described in the InfiniBand Architecture Specification.[10] The names themselves are safe: NCCL’s troubleshooting page tells you to run sudo perfquery -x <lid> and look for non-zero SymbolErrorCounter, LinkErrorRecoveryCounter and LinkDownedCounter as cabling or switch indicators.[3]
| Class | Counters | What it is asking you to check |
|---|---|---|
| Physical | SymbolErrorCounter, LinkErrorRecoveryCounter, LinkDownedCounter | Cable, module, cage, connector — the medium.[3] |
| Management pressure | VL15Dropped | SM and MAD load. VL15_dropped is the “Number of incoming VL15 packets dropped due to resource limitations (e.g., lack of buffers) of the port”, and VL15 carries the management traffic.[10] |
| Congestion or routing | PortXmitDiscards, PortXmitWait | Where traffic is going, not what it is going through. port_xmit_discards is the “Total number of outbound packets discarded by the port because the port is down or congested”; port_xmit_wait is “The number of ticks during which the port had data to transmit but no data was sent during the entire tick (either because of insufficient credits or because of lack of arbitration)”.[10] |
Three questions, asked in this order, get you to the right class before you touch anything: since when (counters are cumulative from the last clear), which counter (the table above), and what is the rate (errors per hour after a clean reset, not the absolute number).[1][2]
ibdiagnet collects the same families in one sweep — “PortCounters”, “PortCountersExtended”, “PortRcvErrorDetails”, “PortXmitDiscardDetails” and LLRCounters from supporting devices — and its -P/--counter <cntr=threshold,...> flag filters by threshold with a default of 0, mirroring ibqueryerrors.[4][5] The results land in ibdiagnet2.pm (spec-compliant) and ibdiagnet2.mlnx_cntrs (Mellanox), which are the two files to ask a customer for rather than a screenshot.[8]
Sample runs, exits cleanly, but zero throughput
Three ports on one leaf switch (LID 32) are suspected. Decide which one is a physical problem.
# 1. Snapshot all three ports without changing anything.
perfquery -x 32 1 | tee /var/tmp/p1.before
perfquery -x 32 2 | tee /var/tmp/p2.before
perfquery -x 32 3 | tee /var/tmp/p3.before
# 2. Record the pre-clear values in the ticket, then clear ONLY the error block.
perfquery -R 32 1,2,3 0x0fff
# 3. Run the workload for exactly one hour, then re-read.
perfquery -x 32 1; perfquery -x 32 2; perfquery -x 32 3Reading the result: port 1 shows SymbolErrorCounter and LinkErrorRecoveryCounter climbing with LinkDownedCounter still zero — a marginal medium, degraded but training successfully, which is the profile that goes to PHY triage next.[3] Port 2 shows only PortXmitDiscards climbing while its data counters are the largest of the three — that is congestion or routing on a busy port, and swapping its cable changes nothing.[2] Port 3 shows only VL15Dropped, which is management-traffic pressure and points at the SM rather than at the link.[2] The comma list in step 2 is legal because resets accept a comma separated list or a port range.[1]
To convert port 2’s load into bytes: take PortXmitData from the delta and multiply by four, because the Data components are octets divided by 4.[1]
Same three ports, but you want the whole fabric in one command and you already know that ten symbol errors a week is normal on this customer’s DAC plant.
cat > /etc/infiniband-diags/error_thresholds <<'EOF'
SymbolErrorCounter=____
LinkErrorRecoveryCounter=____
VL15Dropped=____
EOF
ibqueryerrors --________ /etc/infiniband-diags/error_thresholds -r --switch
echo "exit status: $?" # 0 means ____ , 1 means ____ , -1 means ____Fill the blanks and justify each threshold number in one sentence. Then say which flag you would add to include the transmit and receive data counters, and which flag you must not add if you want to keep the customer’s history.[2]
A customer reports “errors across the whole fabric since the upgrade”. They have already run ibqueryerrors -k twice.
Produce the plan that still yields a defensible answer. Acceptance criteria: you state explicitly what was lost when they cleared and how that limits your conclusions; you define one measurement window with a start command and an end command; you classify whatever you find into the three counter classes before recommending an action; you name the two ibdiagnet files you will ask for so the evidence is fabric-wide rather than per-port; and every command you propose is either read-only or has its pre-values recorded first.[2][1][8]
Errors per hour, on a named port
You do not approve the cables. The pre-clear values go into the ticket, you clear only the error block on the suspect port, and the midnight benchmark becomes the measurement window.[1] By 01:00 rack 4 port 17 has a rate, while two other ports the customer had flagged show only PortXmitDiscards — packets the port received correctly and could not send, which is congestion and not medium.[10] What you say on the call: give me one hour with a known start time and you get errors per hour on a named port. On Wednesday, day two of the acceptance test, half the ranks stop dead at the first collective — and the rate on port 17 explains none of it.
Lab
On the Dell lab with an InfiniBand-mode port. Steps 1-4 are read-only. Step 5 clears counters, which is irreversible and has a recording step instead of a rollback.
- Pre-flight, read-only:
ibstat, thenibnetdiscover -p | headto get the LID of the port you will use. Expected:State: Active,Physical state: LinkUp, and a LID you can name. If the state isInitializing, fix the SM before reading counters.[3] - Local port with no arguments:
perfquery. Expected: the local port’s performance counters print, which works even without a switch. If it fails, the PMA is unreachable and nothing else in this lab will work.[1] - Named port:
perfquery -x <lid> 1 | tee /var/tmp/pq.before. Expected: extended counters print. If the device rejects-x, record that: the extended attribute is optional.[1] - Drive real traffic between two lab hosts with
ib_write_bwfor a fixed duration, then re-readperfquery -x <lid> 1 | tee /var/tmp/pq.afterand compute the PortXmitData delta multiplied by four. Expected: the byte figure is the same order as the bandwidth reported by the benchmark; a factor-of-four gap means you forgot the unit rule.[1] - Irreversible, record first: copy
/var/tmp/pq.beforeinto your notes with the date, thenperfquery -R <lid> 1 0x0fffand immediatelyperfquery -x <lid> 1. Expected: the error block reads zero while the data counters keep their values, which is the proof that the mask did what you meant.[1] There is no rollback for a counter clear — the recorded pre-values in your notes are the only history that survives, which is exactly the discipline you want a customer to have. - Fabric-wide baseline, read-only:
ibqueryerrors --data -rand save the output with the date. Expected: per-port lines with LID, port, link speed setting, remote GUID and remote node description.[2] Do not add-kor-Kon a customer fabric; those clear regardless of what is printed.[2] - Optional on a multi-switch fabric: run
ibdiagnetonce and compareibdiagnet2.pmagainst youribqueryerrorsoutput. Expected: the same ports appear. Keep the file — it is the artefact an escalation asks for.[8]
Paper-and-man-page work plus a container. No fabric required.
- Install the tools and read the two man pages: in a container,
apt-get install -y infiniband-diags man-db, thenman perfqueryandman ibqueryerrors. Expected: you can find the units sentence and the reset-mask examples yourself. If the man pages are missing, read the sources listed in this lesson.[1][2] - Take three supplied
perfquery -xoutputs (from a captured ticket, or the ones in the Worked example) and, for each one, write down: the three physical counters and their values, PortXmitDiscards, VL15Dropped, and PortXmitData converted to real bytes. Expected: your byte figure is four times the raw counter delta.[1] - Decide which of the three is a cable problem and write one sentence of justification per port. Expected: your reasoning names the counter class, not the port number. If two ports look identical, say what extra reading would separate them.[3]
- Write the exact command that resets only the error counters of the suspect port on LID 32 port 2, and the command that would have reset only the non-error counters. Expected:
perfquery -R 32 2 0x0fffandperfquery -R 32 2 0xf000.[1] - Author an
error_thresholdsfile for a production fabric with three entries and a comment line, and justify each number against the customer’s cable plant and its measurement window. Expected: the file parses asname=vallines. Then write theibqueryerrorscommand line that uses it, restricted to switches, with port reporting on.[2] - Finish with the drill above in
mixedmode. Anything under 80 percent on the counter questions means re-reading segment 4 rather than moving to the next lesson.[7]
Nothing here touches a fabric; the container is the only thing to remove.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A customer sends a screenshot of a port with SymbolErrorCounter in the millions and asks whether to replace the cable. Explain in five sentences what you need before you can answer.
Sources
Facts in this lesson were checked against rdma-core master man pages (perfquery, ibqueryerrors), IBUtils2 2.24.0 basic commands and 2.26.0 port counters, NCCL networking troubleshooting, UFM Enterprise 6.24.1 diagnostic utilities, fetched 2026-09-07. Dates are when each page was fetched.
- perfquery(8) man page source (rdma-core master) · fetched 2026-09-07
- ibqueryerrors(8) man page source (rdma-core master) · fetched 2026-09-07
- Networking Troubleshooting (NCCL user guide) · fetched 2026-09-07
- Port Counters (IBUtils2 2.26.0) · fetched 2026-09-07
- ibdiagnet Basic Commands (IBUtils2 2.24.0) · fetched 2026-09-07
- InfiniBand Fabric Utilities (MLNX_OFED 23.07-0.5.1.2) · fetched 2026-09-07
- Diagnostic Utilities (UFM Enterprise 6.24.1) · fetched 2026-09-07
- ibdiagnet Dump Files (IBUtils2 2.24.0) · fetched 2026-09-07
- IP Over InfiniBand (DOCA 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
- sysfs-class-infiniband (Linux kernel stable ABI) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.