Discovering a fabric: ibnetdiscover and iblinkinfo
S4·E1Everything came back · Hall 2, 06:40 Monday, after a 02:10 leaf reboot
Builds on: Reading a host's IB state in 60 seconds
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Produce a full fabric topology with ibnetdiscover and read GUIDs, node types, LIDs and NodeDescriptions out of it.
- Choose between ibnetdiscover and iblinkinfo for a given question and name the flag that answers it.
- Capture a cache before a maintenance window and diff the fabric against it afterwards.
- Use iblinkinfo exit status to build a health check that fails only on a real inconsistency.
Episode 1 — Everything came back
The night-shift operator meets you at the cage door with a sentence you have heard before: everything came back. He rebooted a leaf at 02:10 for a firmware task, the acceptance benchmark restarted at 04:00, and two of the forty hosts are missing from it. He has label-printed every port in rack 4, including the empty ones. The Dell SE is on the bridge call from the parking lot, cold coffee in hand, spreadsheet row 12: fabric accepted Friday. Friday is the signature.
You ask the only question that matters: is there a picture of this fabric from before the window? There is not, so the argument is about memory rather than evidence, and memory loses.
This is the problem the infiniband-diags suite exists to end. A fabric is a graph that nobody drew, and the tools that map it were built to work before anything else does: the base utilities use directed-route MADs and “may therefore work even in unconfigured subnets”, while higher level utilities require LID routed MADs and to some extent SA/SM access.[3] So ibnetdiscover can produce “a human readable topology file” with GUIDs, node types, port numbers, port LIDs and NodeDescriptions on a subnet where no subnet manager has assigned a single LID,[1] and iblinkinfo “reports link info for each port in an IB fabric, node by node”.[2]
Take the picture while the fabric is healthy. Afterwards it is not evidence, it is a memory.
You start with the two views: what is out there, and what state it is in.
1Two views of the same fabric
Discovery is the first thing you do on a fabric you did not build. The infiniband-diags package frames its own job plainly: it “is a set of utilities designed to help configure, debug, and maintain infiniband fabrics”, and “The base utilities use directed route MAD’s to perform their operations. They may therefore work even in unconfigured subnets.”[3] That sentence is worth more than it looks: ibnetdiscover and iblinkinfo can answer questions on a fabric where no subnet manager has assigned a single LID yet, while “Other, higher level utilities, require LID routed MAD’s and to some extent SA/SM access.”[3]
The two tools answer different questions.[1][2]
| Question | Tool | Why |
|---|---|---|
| What is out there and how is it wired? | ibnetdiscover |
“performs IB subnet discovery and outputs a human readable topology file. GUIDs, node types, and port numbers are displayed as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed (full topology).”[1] |
| What state is every link in? | iblinkinfo |
“reports link info for each port in an IB fabric, node by node. Optionally, iblinkinfo can do partial scans and limit its output to parts of a fabric.”[2] |
NVIDIA’s own utility list keeps them together as the basic-connectivity pair, and the NCP-AIN Troubleshooting Tools domain names iblinkinfo explicitly in the command list it expects you to know.[4][8] The habit to build now is: ibstat answers a question about this host, these two answer questions about the fabric.[6]
One host, 60 seconds: you need LID, SMLID, port state, link width active and physical state straight from the local IB driver — from a binary, not a shell script.
A customer sentence. Pick the one tool that answers it.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo2ibnetdiscover: shaping the output
Run bare, ibnetdiscover prints the whole topology to standard output “unless a topology file is specified”, so the first discipline is to redirect it into a dated file you can attach to a ticket.[1] The flags then shape how much of it you have to read.[1]
-l/--listprints a list of connected nodes,-H/--Hca_list,-S/--Switch_listand-R/--Router_listnarrow it to one node type.[1]-g/--grouping“correlates IB nodes by different vendor specific schemes. It may also show the switch external ports correspondence” — this is what turns 32 internal chips back into the switch a customer can point at in the rack.[1]-p/--portsprints a “list of connected ports with relevant information (like LID, portnum, GUID, width, speed, and NodeDescription)”.[1]-f/--fulladds “ports’ speed and width, vlcap”.[1]-s/--showprints “progress information during discovery”;-m/--max_hopsreports the maximum number of hops discovered and does not bound the walk.[1] Radius control on a partial scan belongs toiblinkinfo -n/--hops, which specifies “the number of hops away from a specified node to scan”.[2]
The addressing and transport flags are shared with the rest of the suite: -C picks the CA, -P the port, --node-name-map substitutes readable names for GUIDs, -t sets the MAD timeout, -y supplies an M_Key and -z/--config a config file.[1][3] On a host with two cards, forgetting -C is how you end up mapping the wrong subnet: a discovery tool “attaches” to one port and sees only the fabric behind it.[3]
The node-name map deserves one minute of your time on any customer engagement. Its format is documented in this man page and reused by OpenSM, so the same file makes ibnetdiscover, iblinkinfo and the SM print names instead of 64-bit GUIDs.[1] A topology dump full of 0x0002c9... is unreadable in an escalation mail; the same dump with rack-and-slot names is a diagram.[1]
3The cache family: proving what changed
The four cache flags are the reason ibnetdiscover belongs in a change window and not only in a first triage: --cache writes the discovery result to a file, --load-cache reads one back, and --diff / --diffcheck compare the live fabric against it.[1] Without a cache taken while the fabric was healthy, “did the reboot change anything?” is unanswerable and the conversation degenerates into opinion.[1]
iblinkinfo has its own comparison mode with keys: --diffcheck takes a comma-separated list from port, state, lid and nodedesc, and “If port is specified alongside lid or nodedesc, remote port lids and node descriptions will also be compared.”[2] Pick the keys deliberately: after an SM restart with -r the LIDs may legitimately move, so diffing on lid produces noise while diffing on port and state still finds the cable someone did not plug back in.[2]
--filterdownports <filename> closes the last gap. It filters “downports indicated in a ibnetdiscover cache. If a port was previously indicated as down in the specified cache, and is still down, do not output it”, and it exists “for environments where switches are not fully populated”.[2] A 64-port leaf with 30 ports in use is not 34 faults.[2]
4iblinkinfo in automation
For a cron or a CI job, the useful part of iblinkinfo is not its printing but its exit status: “0 on success, -1 on failure to scan the fabric, 1 if check mode is used and inconsistencies are found.”[2] Three states, not two — a script that treats every non-zero as a failure will page someone at 03:00 for a fabric it could not reach and for a fabric that is merely different, and will never tell the two apart.[2]
The scan-shaping flags matter here too, because a health check should be cheap: --switches-only and --cas-only filter node types, -G <guid> or -D <direct route> pick a start node for a partial scan, -a/--all prints all nodes found in that partial scan, and -n/--hops sets the radius.[2] One warning the man page states outright about partial scans: “For switches results are printed for all ports not just switch port 0.”[2]
You do not always have to run either tool. One ibdiagnet sweep writes both views to disk as ibdiagnet2.ibnetdiscover (“Network in ibnetdiscover format”) and ibdiagnet2.iblinkinfo (“Network in iblinkinfo format”), which is how you get a discovery view out of a capture a customer already sent.[5]
iblinkinfo — print only the nodes that have a port in the Down state.
You know the tool. Pick the flag that makes it answer the question.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfoA leaf switch was rebooted overnight. The customer says “everything came back”. Prove it in four commands.
# Before the window, on a management host with a working HCA:
ibnetdiscover -C mlx5_0 -P 1 --cache /var/tmp/fabric-pre.cache
ibnetdiscover -C mlx5_0 -P 1 -p > /var/tmp/fabric-pre.ports
# After the window:
ibnetdiscover -C mlx5_0 -P 1 --load-cache /var/tmp/fabric-pre.cache --diff
iblinkinfo -C mlx5_0 -P 1 -dReasoning: the first command stores the known-good topology, because --cache is the only way to have something to compare against later.[1] The -p dump is the human-readable companion with LID, port number, GUID, width, speed and NodeDescription per connected port, which is what you paste into the ticket.[1] After the window, --load-cache plus --diff names what moved.[1] iblinkinfo -d then reduces the answer to the nodes that have a port in the Down state right now.[2] Nothing here changes fabric state, and all of it works even if the SM has not finished its sweep, because these are directed-route tools.[3]
Same window, but you also need to know whether any remote node description changed, and the fabric has 12 leaf switches that are only half populated.
ibnetdiscover -C mlx5_0 -P 1 --________ /var/tmp/fabric-pre.cache
# ... maintenance ...
iblinkinfo -C mlx5_0 -P 1 --diffcheck ________,________ --load-cache /var/tmp/fabric-pre.cache
iblinkinfo -C mlx5_0 -P 1 -d --________ /var/tmp/fabric-pre.cacheFill the blanks, then answer in one sentence each: which two --diffcheck keys did you choose and why not lid, and what does the third command stop reporting?[2]
Write a health check that a cron can run every 15 minutes on a 3-switch lab fabric.
Acceptance criteria: it runs only read-only commands; it distinguishes the three documented exit statuses of iblinkinfo and reacts differently to each; it does not alert on ports that were already down in the reference cache; it names the file it compares against and how that file is refreshed after an intentional change; and it prints, on failure, the single command a human should run next.[2][1]
A port, not an opinion
By 07:30 the diff has a name on it. One link on the rebooted leaf never came back, and iblinkinfo -d prints the node that owns it, so the 08:00 call gets a port instead of a theory.[1][2] Before you leave you make the next window cheaper: a cache written while the fabric is healthy, the -p dump beside it, both files dated.[1] Then the customer’s network lead opens a green notebook, points at the link that did come back, and reads out a SymbolErrorCounter with seven digits on it. “Show me the counter,” he says. It is a counter. That is exactly the problem.
Lab
On the Dell lab, with a ConnectX or BlueField-3 port in InfiniBand mode. Steps 1-5 are read-only. Step 6 bounces a link and names its rollback.
- Pre-flight inventory, read-only:
ibstat,ibstatus,ibdev2netdev,sminfo. Expected: the port reportsState: ActiveandPhysical state: LinkUp, andsminforeturns exactly one master.[6][7] If the state isInitializing, stop here: the link trained but no SM has programmed the port, which is an SM problem and not a discovery problem.[7] - Full topology:
ibnetdiscover -C mlx5_0 -P 1 -p | tee /var/tmp/fabric-$(date +%F).ports. Expected: one line per connected port with LID, port number, GUID, width, speed and NodeDescription.[1] If the output is empty, re-check-C/-Pagainstibstat -l.[6] - Grouped view:
ibnetdiscover -C mlx5_0 -P 1 -g | head -40. Expected: nodes correlated by the vendor scheme, with switch external-port correspondence where the vendor supplies it.[1] - Reference cache:
ibnetdiscover -C mlx5_0 -P 1 --cache /var/tmp/fabric.cacheandiblinkinfo -C mlx5_0 -P 1 -l | tee /var/tmp/links-before.txt. Expected: the cache file exists and the link list shows the widths and speeds you expect for the generation in the lab.[1][2] - Read-only layering check:
sudo ip link set ib0 down, then re-runibstatandiblinkinfo -C mlx5_0 -P 1 -d. Expected:ip linkshowsib0DOWN whileibstatstill reportsState: ActiveandPhysical state: LinkUp, and-dprints nothing new.ib0is the IPoIB network interface created over the port — “The first port on the first HCA in the host is called interface ib0” — whileibstatreports the CA port’s own state, so the two are different layers.[9][6] Ifibstatdoes change, you took down something other than the IPoIB interface. Restore withsudo ip link set ib0 up. - Mutating, reversible — lab port only: unseat the cable on the lab port, wait 10 seconds, then run
iblinkinfo -C mlx5_0 -P 1 -dandibnetdiscover -C mlx5_0 -P 1 --load-cache /var/tmp/fabric.cache --diff. Expected: the port appears in the-doutput and the diff names it.[2][1] If-dstill prints nothing, the port never leftActive— confirm withibstatthat you pulled the right cable.[6] Rollback: reseat the cable, wait for the link to train, thenibstatshowsState: ActiveandPhysical state: LinkUpagain and the diff is clean.[6] Do this only on a lab port; never on a link a customer workload is using. - Optional, on a customer or multi-switch fabric: repeat steps 2-4 and save both the cache and the
-pdump with the date in the filename. Those two files plusiblinkinfo -dare the discovery evidence pack for an escalation, and oneibdiagnetrun would have produced the same two views asibdiagnet2.ibnetdiscoverandibdiagnet2.iblinkinfo.[5]
No InfiniBand hardware needed. The infiniband-diags package exists in every distribution’s repositories independently of NVIDIA, so the man pages and the flag surface are all available in a plain container.[3]
- Start a container and install the package:
docker run -it --rm ubuntu:24.04 bash, thenapt-get update && apt-get install -y infiniband-diags man-db. Expected:ibnetdiscover --helpprints the usage block. If not, the package name on RHEL-family images is the same but you needdnf install infiniband-diags. - Read the two man pages and build a one-page table:
man ibnetdiscoverandman iblinkinfo. For each of-l,-g,-H,-S,-p,-f,-mand for each of-d,-l,-p,-a,-n,--switches-only,--cas-onlywrite the question the flag answers in your own words. Expected: your table matches segments 2 and 4. If a flag is missing from your build, note the version — flags accrete over releases.[1][2] - Take two saved
ibnetdiscover -pdumps from either side of a maintenance window —pre.portsandpost.ports— and produce the comparison by hand:diff -u pre.ports post.ports. Expected: a short diff in which every changed line carries a LID, a port number or a NodeDescription; classify each difference as expected (LIDs moved after an SM restart) or unexpected (a link that is gone). If the diff runs to thousands of lines you are reading LID churn rather than breakage — diff on the GUID column instead, or useiblinkinfo --diffcheck port,state. Note that the--cachefile is not one of these dumps: it isibnetdiscover’s own cache, written with--cacheand read back only with--load-cache, not a text report you can hand todiff.[1][2] - Write the health-check skeleton and test its branches without a fabric:
#!/usr/bin/env bash
out=$(iblinkinfo -C "$CA" -P 1 --diffcheck port,state --load-cache "$REF" 2>&1); rc=$?
case $rc in
0) exit 0 ;; # fabric matches the reference
1) echo "FABRIC CHANGED"; echo "$out"; exit 1 ;; # inconsistencies found
*) echo "SCAN FAILED - check ibstat on $(hostname)"; exit 2 ;;
esacExpected: forcing rc to each of 0, 1 and 255 exercises three different messages. If your script collapses 1 and -1 into one branch, fix it — that is the bug this step exists to prevent.[2]
5. Rehearse the vocabulary with the drill above in mixed mode until the symptom-to-command step is automatic; the NCP-AIN Troubleshooting Tools domain names iblinkinfo in its published command list.[8]
Nothing in this variant changes any host or fabric state; removing the container is the only cleanup.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, why you ask for an ibnetdiscover cache before a scheduled switch reboot and what you do with it the next morning.
Sources
Facts in this lesson were checked against rdma-core master man pages (ibnetdiscover, iblinkinfo, infiniband-diags), MLNX_OFED 23.07 fabric utilities, IBUtils2 2.24.0 dump files, fetched 2026-09-07. Dates are when each page was fetched.
- ibnetdiscover(8) man page source (rdma-core master) · fetched 2026-09-07
- iblinkinfo(8) man page source (rdma-core master) · fetched 2026-09-07
- infiniband-diags(8) overview man page (rdma-core master) · fetched 2026-09-07
- InfiniBand Fabric Utilities (MLNX_OFED 23.07-0.5.1.2) · fetched 2026-09-07
- ibdiagnet Dump Files (IBUtils2 2.24.0) · fetched 2026-09-07
- ibstat(8) man page source (rdma-core master) · fetched 2026-09-07
- Networking Troubleshooting (NCCL user guide) · fetched 2026-09-07
- AI Networking Certification (NCP-AIN) exam page · fetched 2026-09-07
- IP Over InfiniBand (DOCA 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.