ibdiagnet: the command you run before escalating
S4·E3Three theories and eleven hours · Hall 2, Wednesday, day two of the three-day acceptance test
Builds on: Discovering a fabric: ibnetdiscover and iblinkinfo, Counters and errors that mean something
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Run one ibdiagnet sweep and state which fabric faults it has already ruled out.
- Select the flags that answer a specific question: routing, counters, expected speed and width, PHY, SHARP.
- Name the output file that holds each piece of evidence and ask a customer for it by name.
- Re-analyse a customer capture offline with -f and produce a three-line escalation summary.
Episode 3 — Three theories and eleven hours
The customer’s HPC lead has until Friday to sign. The fabric is up, ibstat is green on all forty hosts, and half the ranks of the acceptance benchmark hang at the first collective. Three people at the table, three theories — partitions, the subnet manager, a downshifted link — and the network lead’s notebook is filling faster than the evidence is. The NVIDIA PM, dialled in, confirms that the roadmap item which would help here is “not announced”. Eleven hours until the run restarts.
Chasing three theories with three tools in parallel is how an acceptance test dies. So you ask for one command, and for its whole output directory rather than a screenshot.
ibdiagnet exists for this exact moment. It “Sweeps the InfiniBand fabric and collects information from the following InfiniBand devices: Switches, HCAs, Routers, Aggregation Nodes, Gateways”,[1] and one bare run checks duplicated node and port GUIDs and duplicated node descriptions, “Performs correct LID assignment and duplicated LID check”, finds links in INIT logical state and unresponsive devices, performs a “correctness check of the switch routing tables as well as checking of credit-loop free routing”, validates width and speed, and “Reports links with high Bit Error Rate (BER)”.[1] Even the console stage names it prints — Fabric Discovery, Duplicated GUIDs check, LID Check, Links Check — are a diagnosis, because where a run stops tells you which layer to argue about.[2]
One sweep, one directory, then argue.
Their engineer types it and reports there is no such command on the host. That, too, is a finding.[12]
1One sweep, and what it has already ruled out
ibdiagnet “Sweeps the InfiniBand fabric and collects information from the following InfiniBand devices: Switches, HCAs, Routers, Aggregation Nodes, Gateways.”[1] The reason it is the command you run before escalating is the checklist it performs without being asked.[1]
In one bare run it checks duplicated node and port GUIDs and duplicate node descriptions; “Performs correct LID assignment and duplicated LID check”; finds “links in INIT logical state” and unresponsive devices; performs a “correctness check of the switch routing tables as well as checking of credit-loop free routing”; validates width and speed; “Reports links with high Bit Error Rate (BER)”; and compares counters against thresholds between snapshots.[1] It also handles cable information, PHY data, PCI dumps, virtualization data, partition validation and the security keys.[1]
The stage names you will see on the console are listed on the “without parameters” page: Fabric Discovery, Duplicated GUIDs check, Duplicated Node Description Check, LID Check, Links Check and others.[2] Reading them in order is itself a diagnosis: a run that stops in Fabric Discovery is not the same problem as one that completes and complains in Links Check.[2]
Two packaging facts before the flags. ibdiagnet ships in the ibutils2 package, and on a DOCA host that package is not installed by the profiles: “The installation procedure does not install proprietary packages. Those packages are installed upon request”, with ibutils2 and opensm on that list.[12][8] And the manual moved: the old ibdiagnetusermanual URL now redirects to the IBUtils2 Utility Documentation, so a customer quoting an old page may be reading a different flag set than the one on their box.[8]
Half the nodes joined, ibstat says Active, MPI hangs. Which ibdiagnet file tells you what partition membership each port actually got?
Where the proof lives: ibdiagnet2.* files, OpenSM logs and caches, sysfs.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo2The flags that answer a question
There are dozens of flags. The ones worth memorising map one-to-one onto questions a customer asks.[3]
| Question | Flag | Documented behaviour |
|---|---|---|
| Where do the files go? | -o, --output_path |
“Specifies the directory where the output files will be placed.”[3] |
| Is the routing sane? | -r, --routing plus --r_opt |
Validates unicast and multicast routing; check_sl validates that no SL2VL mapping routes to VL15; mcast adds the multicast credit-loop check; static_ca2ca forces static CA-to-CA checks with adaptive routing on; dump_only writes files without checks.[4] |
| Are we at the speed we bought? | --ls, --lw |
“Specifies the expected link speed” and “Specifies the expected link width”, 0 disables each.[3] |
| What do the counters say now? | --sc, --pc, --scr |
Report Mellanox counters; reset the IB spec compliant port counters; reset the Mellanox counters.[3] |
| Is the topology rail-optimised? | --rail_validation |
“Checks topology being rail optimized (default - disabled).”[3] |
| What does the PHY say? | --get_phy_info |
“Query all ports for phy information”, with --get_ppamp for PPAMP data and --show_cap_reg to dump capability registers.[6] |
| Can I work offline? | -f, --load_from_file |
“Loads ibdiagnet.db_csv from external file. Use this option to skip discovery stage.”[3] |
The timing knobs matter on a large or a sick fabric, because the defaults are tuned for a healthy one: --mads_timeout default 500 ms, --mads_retries default 2, --max_hops default 64, --smp_window 8, --gmp_window 128, --sl 0, --screen_num_errs 5, and --pm_pause_time “(default=1)” second between the first and second counter samples.[3] If a sweep is timing out on a congested fabric, raise the timeout before you conclude a device is dead.[3]
Scope control keeps a sweep from becoming an outage of attention: --scope <file.guid> includes, --exclude_scope excludes, --host_file names hosts, --skip <stage> drops a stage such as dup_guids, lids, pm or speed_width_check, and --skip_plugin unloads a plugin entirely.[3] Topology comparison uses -t/--topo_file to read an expected topology and -w/--write_topo_file to record the current one.[3]
3The files: what you ask a customer to send
The escalation skill is naming files. One sweep writes a directory; each file is one topic, and asking for the directory rather than a screenshot is what makes a first reply useful.[5]
ibdiagnet2.log— the log;ibdiagnet2.db_csv— “Internal database”, the file-freloads.[5]ibdiagnet2.lst— fabric links in LST format;ibdiagnet2.net_dump— link dump with split cable mapping and FEC info;ibdiagnet2.net_dump_ext— “Extended fabric link dump with FEC, BER, and phy data”.[5]ibdiagnet2.pm— “IB spec compliant Ports Counters”;ibdiagnet2.mlnx_cntrs— Mellanox diagnostic counters.[5]ibdiagnet2.pkey— pkey tables;ibdiagnet2.slvl— SL2VL tables of fabric switches;ibdiagnet2.sm— the subnet managers seen in the fabric.[5]ibdiagnet2.nodes_info— “Nodes Information (FW version, etc)”, the fastest fleet-wide firmware inventory there is.[5]ibdiagnet2.fdbsandibdiagnet2.mcfdbs— unicast and multicast forwarding tables;ibdiagnet2.ar,ibdiagnet2.far,ibdiagnet2.far_flid— adaptive-routing tables;ibdiagnet2.rn,ibdiagnet2.rnc,ibdiagnet2.rnc2— SHIELD configuration and counters.[5]ibdiagnet2.rails,ibdiagnet2.sharp,ibdiagnet2.cables,ibdiagnet2.guid,ibdiagnet2.ibnetdiscover,ibdiagnet2.iblinkinfo— rail validation, SHARP configuration, cable info, the scope file, and the two discovery views from the previous lesson.[5]
Three of them are disabled by default and need --enable_output: ibdiagnet2.fdbs, ibdiagnet2.ar and ibdiagnet2.rnc.[5] Ask for a forwarding-table dump without saying that and you will get a directory without one.[5] Routing validation adds two more files of its own, ibdiagnet2.vl2vl and ibdiagnet2.plft.[4]
Where the directory lands is genuinely unsettled in the documentation. The routing-validation page and the SHARP guide both read files from /var/tmp/ibdiagnet2/, while the UFM 6.24.1 diagnostic-utilities page states the default output directory is /var/tmp/ibdiagpath/; the IBUtils2 basic-commands page documents -o without stating any default at all.[4][11][10][3] Do not quote a path to a customer from memory: run ibdiagnet | tail on the box, read the path it prints, or remove the ambiguity by always passing -o yourself.[3]
Firmware: Dell DUP vs NVIDIA mlxfwmanager / bf-fwbundle — which one wins?
4Offline, and from the switch
Two run modes matter for an FAE who is rarely on the customer’s management network.[3][9]
Offline. -f/--load_from_file loads a saved ibdiagnet.db_csv and skips discovery, so the entire analysis runs on your laptop against a file the customer emailed.[3] That is the highest-value drill in this whole module: the fabric is untouched, the customer does not need to grant access, and you can iterate flags for as long as you like.[3] --discovery_only is the complement, dumping only the db_csv output with discovery.[3]
From an NVOS switch. When no host has ibutils2 installed, the switch can run the sweep itself: nv action run ib cmd "ibdiagnet [options]", with the documented restriction “Only ibdiagnet command is currently supported in command string” and the example nv action run ib cmd "ibdiagnet --get_phy_info".[9] Then nv show ib ibdiagnet shows the log content, nv show ib ibdiagnet files lists the archives, and the whole output set is packed as ibdiagnet2_output.tgz.[9] Getting it off the switch is nv action upload ib ibdiagnet files <archive> <remote-url> over ftp, tftp, scp or sftp, and nv action delete ib ibdiagnet files <archive-name> cleans up afterwards.[9]
A customer reports “MPI jobs hang on half the nodes, but the fabric is up”. They can run one command. Get everything in one sweep, then work it offline.
# On their management host (ibutils2 installed):
ibdiagnet -o /var/tmp/case12345 -r --r_opt check_sl,mcast --sc
tar czf case12345.tgz /var/tmp/case12345
# ... they send you case12345.tgz ...
# On your laptop:
tar xzf case12345.tgz && cd var/tmp/case12345
ibdiagnet -f ./ibdiagnet2.db_csv -o /tmp/reanalysis
grep -c . ibdiagnet2.pkey ibdiagnet2.sm ibdiagnet2.nodes_infoReasoning: -o removes the default-directory ambiguity, so you know exactly what to ask for.[3] -r with check_sl validates that no SL2VL mapping lands on VL15, which silently drops a service level, and mcast adds the multicast credit-loop check — both are prime suspects for “up but hanging”.[4] --sc adds the Mellanox counter report without resetting anything.[3] Offline, -f re-runs the analysis with no fabric access.[3] Then three files carry the three usual causes: ibdiagnet2.pkey for partition membership, ibdiagnet2.sm to prove how many subnet managers are present, and ibdiagnet2.nodes_info for a firmware spread.[5]
The escalation summary is three lines: what the fabric is (node and switch counts from the sweep), what the sweep found (the verbatim finding, e.g. whether -I- no credit loops found appeared), and what you have attached (the directory, named).[4][5]
Same customer, but now they also claim the fabric downshifted to NDR200 after a switch replacement, and they need the forwarding tables in the capture.
ibdiagnet -o /var/tmp/case12346 \
--ls ____ --lw ____ \
--enable_output ________ \
-r --r_opt ________Fill the blanks for a fabric that should be 400 Gb/s at 4x, then answer: why does --enable_output appear at all here, and which single file will you open first when the archive arrives?[3][5]
You are handed only ibdiagnet2.db_csv from a fabric you have never seen and a one-line complaint: “collectives are slow since Tuesday”.
Produce the offline analysis plan. Acceptance criteria: every command runs with no fabric access; you name at least four ibdiagnet2.* files and what each one would have to say to confirm or kill a hypothesis; you state which questions the database cannot answer and what you would have to ask the customer to run instead; you avoid quoting a default output path; and you end with a three-line summary in the form fabric / finding / attached.[3][5][4]
The directory, not the screenshot
Once ibutils2 is installed the sweep runs, and one directory settles all three theories before lunch: ibdiagnet2.pkey for partition membership, ibdiagnet2.sm for how many subnet managers the fabric actually has, ibdiagnet2.nodes_info for the firmware spread across every node the sweep saw.[5][12] You reload their ibdiagnet2.db_csv on your own laptop with -f and keep iterating flags while they go on benchmarking, so the fabric is never yours to touch.[3] What you tell the HPC lead: send me the directory, not the screenshot. The same sweep reports one link with a high BER — rack 4, port 17 — and by 06:00 the operator has opened a ticket that reads: send us a cable.
Lab
On the Dell lab, with a ConnectX or BlueField-3 port in InfiniBand mode and a subnet manager running somewhere. Every step here is read-only; nothing resets counters and nothing changes state.
- Pre-flight inventory:
ibstat,sminfo,ibnetdiscover -p | head. Expected:ActiveandLinkUp, exactly one master SM, and a topology you recognise. Ifibdiagnetis not installed, installibutils2first.[12] - First sweep with an explicit output path:
ibdiagnet -o /tmp/ibd1. Expected: the stage list from segment 1 prints in order and/tmp/ibd1fills with files.[2][3] - Settle the documentation conflict on your own hardware:
ibdiagnet | tailwith no-o, and record the path it actually prints. Expected: one path. Write it in your notes with the ibutils2 version from step 1, because the public pages disagree between/var/tmp/ibdiagnet2/and/var/tmp/ibdiagpath/.[4][11] - Read your own partition work from module 2 back out of the fabric:
cat /tmp/ibd1/ibdiagnet2.pkey. Expected: the P_Key tables you configured. If a host is missing, that is a membership problem and not a link problem.[5] - Firmware inventory in one file:
cat /tmp/ibd1/ibdiagnet2.nodes_info. Expected: FW version per node. Keep it — the next lesson on firmware starts from this file.[5] - Assert the design:
ibdiagnet -o /tmp/ibd2 --ls <expected-speed> --lw <expected-width>using the values your lab is built for. Expected: a pass, or a named port that is not at the expected setting.[3] - Routing and SL validation:
ibdiagnet -o /tmp/ibd3 -r --r_opt check_sl. Expected:-I- no credit loops foundon a healthy fabric and no SL mapped to VL15.[4] - PHY data for the next lesson:
ibdiagnet -o /tmp/ibd4 --get_phy_infoand openibdiagnet2.net_dump_ext. Expected: FEC mode and BER columns per link.[6][5] - Optional, on a customer lab with an NVOS switch:
nv action run ib cmd "ibdiagnet --get_phy_info", thennv show ib ibdiagnet filesandnv action upload ib ibdiagnet files ibdiagnet2_output.tgz scp://user@host/path/. Expected: the archive lands on your collection host. Clean up withnv action delete ib ibdiagnet files <archive-name>when the case closes.[9]
A plain x86 Linux box, no InfiniBand hardware. Everything here is read-only.
- Install DOCA-Host and then, explicitly, the tool package: follow the DOCA-Host installation page for your distribution, then install
ibutils2on request — the profiles do not install it.[12] Expected:ibdiagnet --versionruns. If the binary is missing afterdoca-all, that is the lesson, not a failure.[12] - Confirm what you have:
rpm -q ibutils2 opensmordpkg -l | grep -E 'ibutils2|opensm'. Expected: both named, or a clear statement of which is absent.[12] - Run the analysis offline against a supplied capture:
ibdiagnet -f /path/to/ibdiagnet2.db_csv -o /tmp/offline. Expected: the stages run without any fabric access and/tmp/offlinefills withibdiagnet2.*files.[3] If it refuses, check that you passed the database file and not the directory.[3] - Walk the evidence in this order and write one sentence per file:
ibdiagnet2.log(what ran),ibdiagnet2.pkey(partition membership),ibdiagnet2.sm(how many subnet managers),ibdiagnet2.nodes_info(firmware spread),ibdiagnet2.net_dump_ext(FEC, BER and phy per link).[5] - Search the log for the routing verdict:
grep -n "credit loops" /tmp/offline/ibdiagnet2.log. Expected: the verbatim line-I- no credit loops foundwhen routing is clean.[4] If the string is absent, record that the check did not run rather than assuming a failure —-rhas to be requested.[4] - Produce the three-line escalation summary: fabric (counts), finding (verbatim strings, not paraphrase), attached (the directory name). Expected: it fits in a chat message and needs no follow-up question.[5]
- Finish with the drill above in
filemode until you can name the file for each kind of evidence without looking.[5]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell services engineer, in five sentences, why 'send me the ibdiagnet output directory' is a better first request than 'send me a screenshot of the error', and what you will do with the directory.
Sources
Facts in this lesson were checked against IBUtils2 2.24.0 basic commands / dump files / plugins and 2.26.0 functionality, routing validation and without-parameters pages; NVOS ibdiagnet commands v25.02.3000; DOCA-Host 3.5.0 installation page. Re-fetched 2026-09-07. Dates are when each page was fetched.
- ibdiagnet Functionality (IBUtils2 2.26.0) · fetched 2026-09-07
- Running ibdiagnet without Parameters (IBUtils2 2.26.0) · fetched 2026-09-07
- ibdiagnet Basic Commands (IBUtils2 2.24.0) · fetched 2026-09-07
- Routing Validation (IBUtils2 2.26.0) · fetched 2026-09-07
- ibdiagnet Dump Files (IBUtils2 2.24.0) · fetched 2026-09-07
- Phy Diagnostic Plugin (IBUtils2 2.24.0) · fetched 2026-09-07
- Cable Diagnostic Plugin (IBUtils2 2.24.0) · fetched 2026-09-07
- IBUtils2 Utility Documentation (ibdiagnet user manual index) · fetched 2026-09-07
- ibdiagnet Commands (NVOS UM for InfiniBand Switches v25.02.3000) · fetched 2026-09-07
- Diagnostic Utilities (UFM Enterprise 6.24.1) · fetched 2026-09-07
- Testing NVIDIA SHARP Setup (SHARP 3.0.0) · fetched 2026-09-07
- DOCA-Host Installation and Upgrade (DOCA 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Escalation bridge call: five InfiniBand ticketsElsewhere in this course · Same ground: phy, qos and routing
- Adaptive routing, SHIELD and hash-based forwardingElsewhere in this course · Same ground: nvos, ibdiagnet and qos
- MTU, RoCE LAG, and what has to be installedRoCE course · Same ground: tuning, packaging and select