Speeds, widths, and the link that says SDR
S1·E3A photograph of an iDRAC screen · Dell lab, Round Rock, Friday afternoon, nine days to acceptance
Builds on: LIDs, GIDs, GUIDs and LMC
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Map a per-lane signalling rate to its InfiniBand generation using NVIDIA's own force_link_speed value tables.
- Separate link width from link speed and read a negotiated link as width times speed.
- Choose the right command to read or assert a rate - ibstat, iblinkinfo, smpquery -x, ibportstate query or ibdiagnet --ls/--lw - and say what each one can and cannot prove.
- Analyse an SDR-in-iDRAC escalation and decide from the reporting layer alone whether it is cosmetic.
Episode 3 — A photograph of an iDRAC screen
The escalation is an image file. The customer’s platform team has photographed an iDRAC inventory screen showing an NVIDIA ConnectX-7 InfiniBand card reporting Single Data Rate, with the purchase order for a 400 Gb/s fabric pasted underneath. Procurement has asked for the lead time on sixty-four replacement adapters — its only question, and roughly once a year the right one.
You are in the lab with the Dell SE, his coffee cold again, the promise spreadsheet open at the 400G row, and an identical card in a PowerEdge two racks away. Down the hall an NVIDIA PM is answering another customer’s roadmap questions with “not announced”.
Nobody is wrong about what they saw: the iDRAC did say SDR. The open question is which layer said it, and when in the card’s life.
Rates are negotiated per link and then reported by whatever software happens to be looking, which is why NVIDIA publishes a numbered value table rather than a slogan. force_link_speed_ext maps 1 to 14.0625 Gbps for FDR, 2 to 25.78125 for EDR, 4 to 53.125 for HDR and 8 to 106.25 Gbps for NDR, and all three of the force_link_speed parameters are annotated “Do NOT change” because each default already tells the subnet manager to use whatever the port reports as supported.[1] The number on the purchase order is four of those lanes: NDR peaks at 400 Gbps per direction.[7]
A rate is a claim by whichever layer is reporting it — ask which layer, and when.
Before you answer a photograph, get the ladder straight.
1The ladder, read off the subnet manager's own value table
There is no need to trust a marketing slide for InfiniBand rates: the Subnet Manager exposes them as a numbered value table you can read in the UFM SM properties. force_link_speed_ext maps 1 = 14.0625 Gbps (FDR), 2 = 25.78125 Gbps (EDR), 4 = 53.125 Gbps (HDR), 8 = 106.25 Gbps (NDR), with combination values for multi-rate support, “30: Disable extended link speeds” and a default of “31: set to PortInfo:LinkSpeedExtSupported”.[1] The older force_link_speed covers the legacy ladder - “1: 2.5 Gbps, 3: 2.5 or 5.0 Gbps, 5: 2.5 or 10.0 Gbps, 7: 2.5 or 5.0 or 10.0 Gbps”, reserved values, and default “15: set to PortInfo: LinkSpeedSupported”.[1] XDR needed a third field: force_link_speed_ext2, where “1: 212.50 Gbps” and the default is “7: set to PortInfo:LinkSpeedExtSupported2”.[1]
All three are annotated “Do NOT change” in NVIDIA’s table, and the reason is in the defaults themselves: each one’s default tells the SM to use whatever the port reports as supported.[1] Forcing a speed is how you lose a rate, not how you gain one.
Those are per-lane signalling rates. The port rate a customer quotes is four lanes wide: NVIDIA states “NVIDIA InfiniBand XDR has a peak speed of 800 Gbps per direction… NVIDIA InfiniBand NDR [has] a peak speed of 400 Gbps per direction”.[7] Do the arithmetic and note the mismatch on purpose: 4 x 106.25 is 425, not 400. The per-lane number is the signalling rate on the wire; the marketed number is the usable data rate after line coding. Quote the right one for the question - a customer asking about cable and transceiver capability wants the per-lane figure, a customer sizing a job wants the port figure.[1][7]
The InfiniBand Trade Association does publish a roadmap covering 1x, 2x, 4x and 12x port widths - but on the fetched page the rate table is delivered as an image, so no per-generation numbers are readable from the page text.[13] That is why this lesson sources the ladder from NVIDIA’s SM parameter tables instead.
2Width times speed: two numbers, one link
A negotiated InfiniBand link is described by two independent attributes: how many lanes are active, and how fast each lane is running. Both can degrade on their own, and they degrade for different reasons - a lane failure gives you 4x to 1x at the same speed, while a marginal medium or a mismatched cable gives you the full width at a lower speed.
The fastest place to see both at once on a real fabric is an ibdiagnet fabric summary, which prints a line of the form Links at 4x50 alongside the node and switch counts.[11] Four lanes at the 50 Gb/s class is an HDR fabric, whose per-lane signalling rate is 53.125 Gb/s.[11][1] One line, whole fabric, no interpretation needed.
Per link, iblinkinfo “reports link info for each port in an IB fabric, node by node”, with -l/--line to “print all information for each link on one line” for grep-friendly output and -d/--down to “print only nodes which have a port in the ‘Down’ state”.[4] It can also compare against a previous capture with --diffcheck <keys> over port, state, lid and nodedesc, and it returns exit status 1 in check mode when inconsistencies are found - which makes it usable directly in a health cron.[4]
Locally, ibstat “displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state”.[12] Note what that list contains and what it does not - the rate is absent from both the ibstat and ibstatus man pages.[12] NVIDIA’s own utilities page is the one that words ibstatus as printing “port width and port rate”.[14] Lesson 5 drills that pair properly.
| PowerEdge | BF-3 offering | DPU mode | NIC mode | Aux power | vSphere DSE | Notable KB |
|---|---|---|---|---|---|---|
R660 16G | ||||||
R760 16G | ||||||
R760XA 16G (GPU) | ||||||
XE9680 16G HGX H100/H200 | ||||||
XE9680L 16G liquid | ||||||
R7725 17G (AMD) | ||||||
R770 17G (Intel) | ||||||
XE9780 / XE9785 17G HGX B300 — Dell AI Factory |
✓ yes · ✗ no · ◐ conditional · ? unknown · — n/a. Click a cell for the evidence.
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
3Reading a rate, asserting a rate, and what each command can prove
Four tools, four different claims.
smpquery -x reads PortInfo from the SMA. The flag matters: -x/--extended “Set SMSupportsExtendedSpeeds bit 31 in AttributeModifier (only impacts PortInfo queries)”.[6] Without it you are reading the legacy speed field on a port whose real rate lives in the extended one. Its examples are worth memorising in the shape smpquery portinfo 3 1, smpquery -G switchinfo 0x2C9000100D051 1 and smpquery -D nodeinfo 0.[6]
ibportstate both reads and writes. Its supported operations are “enable, disable, reset, speed, espeed, fdr10, width, query, on, off, down, arm, active, vls, mtu, lid, smlid, lmc, mkey, mkeylease, mkeyprot (Default is query)”.[3] Two sentences from that page govern how you use it. First: “Speed and width changes are not effected until the port goes through link renegotiation.”[3] Second, and this is what makes query valuable: it “also validates port characteristics (link width, speed, espeed, and fdr10) based on the peer port… This checking is done when the port queried is a switch port as it relies on combined routing”, and that validation “requires LID routing to be functioning in the subnet”.[3] So ibportstate query on a switch port is the only one of these that asks the fabric whether the two ends agree.
perfquery -T/--extended_speeds reads PortExtendedSpeedsCounters - the FEC and error counters for a link that is up but marginal.[5] Module 4 uses them properly; here, know that they exist and that they are the difference between “the link is at the right rate” and “the link is at the right rate and is not working for it”.
ibdiagnet turns a reading into an assertion: --ls <0|2.5|5|10|14|25|50|100|200|FDR10> sets the expected link speed and --lw <0|1x|2x|4x|8x|12x> the expected width, with 0 disabling the check.[10] That is what you hand a customer as an acceptance test, because it produces a pass or a fail rather than a table someone has to read.
4Analysis: the link that says SDR
Now the escalation this lesson is named after, re-fetched and re-quoted today. Dell KB 000221452 (article version 4, last modified 19 May 2026): the symptom is that “The NVIDIA ConnectX-7 InfiniBand card may show that its speed is reflected as Single Data Rate (SDR)” when inspected in the iDRAC UI.[2] The cause is deliberate: “The UEFI and PXE driver configure the Link speed to SDR if the cable supports SDR” so that the link comes up quickly during boot.[2] And the resolution is that “This is expected behavior and is cosmetic in the iDRAC” - the card operates at NDR speeds within the operating system.[2]
The affected list is long and covers most of the Dell GPU and general-purpose fleet: R650, R6515, R6525, R6615, R6625, R750, R750XA, R7515, R7525, R760, R760XA, R7615, R7625, R860, R960, XE8640, XE9640 and XE9680.[2]
Analyse it as a reporting-layer problem rather than a link problem. iDRAC inventories what the pre-boot driver negotiated; the OS reports what the OS driver negotiated after it loaded and renegotiated. Two layers, two moments, two legitimate answers.[2] The decision rule follows directly: if only the pre-boot layer says SDR and any OS-side read says otherwise, it is cosmetic. If the OS also says SDR, you have a real downshift and the tools in segment 3 apply - ibstat for width and state, ibstatus for the rate string, smpquery -x portinfo for the negotiated extended speed, then ibportstate query on the switch port to see whether the peer agrees.[12][6][3]
Dell platform: POST, iDRAC, vSphere, firmware
Case A. A ConnectX-7 port on an XE9680. iDRAC inventory shows SDR. In the OS, ibstat reports port state Active, physical state LinkUp, link width active 4X; ibstatus prints the rate string for a 400 Gb/s link.
Reading: two layers, two moments. The pre-boot driver brought the link up at SDR on purpose to shorten boot, which is exactly the documented behaviour of KB 000221452, and the OS driver renegotiated.[2] Verdict: cosmetic, no action. Next command: none on the hardware - reply with the KB number and the ibstat output as proof.[12]
Case B. Same card, but in the OS ibstat shows width 4X and the negotiated extended speed reads 53.125 Gb/s per lane rather than 106.25.
Reading: full width, one generation down - that is HDR class on a port that should be NDR.[1] Width is intact so this is not a broken lane; the two ends agreed on a lower rate. Verdict: real downshift, medium or capability. Next command: smpquery -x portinfo <lid> 1 to confirm LinkSpeedExtActive against LinkSpeedExtSupported on this port - the -x flag is mandatory or you read the wrong field - then ibportstate <lid> <port> query on the switch side so the switch validates the peer.[6][3]
Case C. An ibdiagnet fabric summary from the customer shows Links at 4x50 on every link, and the customer’s purchase order says NDR.
Reading: this is not one bad link, it is a fabric-wide generation mismatch - the whole fabric negotiated the HDR class.[11][1] A single-link investigation will waste the call. Verdict: fabric-wide, almost certainly cable or transceiver generation, or an adapter generation the customer did not expect. Next command: ibdiagnet --ls 100 --lw 4x to turn “should be NDR” into a pass/fail list of offending ports, then iblinkinfo -l for the per-link inventory to hand back.[10][4]
Case D. A port on an R760xa. iDRAC says SDR. In the OS, ibstat shows state Active, physical state LinkUp, link width active 1X.
Fill in the blanks:
- The iDRAC reading is explained by ____________ and is ____________.
- But the OS shows width ____, which means ____________ lanes trained, so this is a ____________ problem, not the KB.
- The next command is
____________to see whether other links on the same switch show the same width, and____________ queryon the switch port to have the switch compare both ends. - The one thing you must NOT do first is ____________, because ____________.
Case E - solve alone. A customer with an eight-rail XE9680 pod reports that “two nodes are slow”. They send you: an iDRAC screenshot showing SDR on all eight cards of both nodes, an ibstat from one node showing 4X and Active, and nothing else.
Produce:
- A one-sentence verdict on the iDRAC screenshot, with the KB number.
- The exact list of artefacts you ask for next, with the command that produces each one and what each will prove.
- The single assertion command you would have them run to convert “slow” into a pass/fail result, with the values you would put in its two flags for an NDR fabric.
Acceptance criteria: your verdict cites KB 000221452 and does not request any hardware action; every artefact you ask for is produced by a command quoted in this lesson; and your assertion command names both an expected speed and an expected width.
Episode 3 — The reply you send instead of a cage visit
You bring the lab card up and read it from the OS: 4X width, port Active, and the rate string the fabric expects.[12] That is the documented behaviour: the UEFI and PXE driver configure the link speed to SDR if the cable supports SDR, and it “is expected behavior and is cosmetic in the iDRAC” while the card runs at NDR in the OS.[2] No cards move; procurement’s thread closes. What you write back: “Send me ibstat and ibstatus from the host. If the OS reports NDR, this is KB 000221452 and there is nothing to replace. If the OS says SDR too, we have a real downshift.” Monday, a new line in the network lead’s notebook: the tenant network is up on every host and answers nothing.
Lab
Mutating. Steps 1-5 and 7 are read-only reads of the Dell-lab ConnectX or BlueField-3 port. Step 6 forces a link speed on a lab-only port and names its rollback - do not run it against a production or unmanaged switch.
- Pre-flight inventory:
ibstat,ibstatus,ibv_devinfo | head -20, and note the device name, port number, state, physical state, link width active and rate. Save it to a file - this is your baseline. - Read the negotiated extended speed from the SMA rather than the driver:
smpquery -x portinfo <lid> 1. Expected: the extended-speed active and supported fields are populated. If they read as unsupported or zero, you forgot-x- run it again without the flag and compare, so you see the difference for yourself. - Read the extended-speeds counters:
perfquery -T <lid> 1. Expected: a PortExtendedSpeedsCounters block. If the attribute is not supported on this device, record that - optional attributes really are optional, and knowing which of your lab cards supports them is useful. - Compare the layers on this exact machine: pull the adapter’s reported link speed from iDRAC (web UI or
racadm) and put it beside your step 1ibstatusrate. Expected: if iDRAC says SDR and the OS does not, you have reproduced KB 000221452 on your own hardware - keep that screenshot pair, it is the best teaching artefact you will ever have for this ticket. - If a second IB port and a cable are available, bring the pair up back to back and compare the negotiated rate against the cable’s rated generation. Expected: the link negotiates the lower of the two capabilities. Record both.
- Optional and mutating - lab-only port, never a host link into a production or unmanaged switch:
ibportstate <lid> <port> speed <value>to force a lower enabled speed, then bounce the link so renegotiation happens, and re-read withibstat. Rollback:ibportstate <lid> <port> speed 15to restore “all supported speeds enabled”, bounce the link again, and confirm the original rate from your step 1 baseline file. Remember the man page warning that an HCA port’s state may be changed locally without the Subnet Manager’s knowledge. - Optional, customer or NVIDIA lab with a real switch:
ibportstate <lid> <port> queryon a switch port and read the peer-validation output. Expected: width, speed, espeed and fdr10 compared against the neighbour. This is the step you cannot rehearse on a back-to-back link.
No InfiniBand hardware. Everything here is documentation and artefact analysis.
- In a container with
infiniband-diags, read the three man pages that carry this lesson’s facts:man ibportstate,man smpquery,man perfquery. Expected: you can quote the renegotiation sentence, the-xextended-speeds sentence, and the-Tcounter flag from memory afterwards. If a page is missing,apt-get install -y man-db infiniband-diags. - Build the rate ladder as a table from the source, not from memory: open the UFM SM default properties page and copy the
force_link_speed,force_link_speed_extandforce_link_speed_ext2value mappings into a file. Expected: five per-lane rates from 14.0625 to 212.50 and three defaults that all mean “use what the port supports”. - Compute the port rates: multiply each per-lane figure by 4 and write both your arithmetic result and NVIDIA’s published port rate next to each other. Expected: they do not match exactly for NDR (425 vs 400) - write one sentence explaining why, and which number you would use with a customer asking about a cable.
- Work Cases A, B and C from the Worked block above using only the quoted field values. Expected: three different verdicts and three different next commands. If two of your verdicts are the same, re-read which layer reported the number.
- Reproduce the SDR escalation end to end: open Dell KB 000221452, extract the symptom, cause and resolution sentences verbatim, and write the two-sentence customer reply plus the one proof command. Expected: your reply names the KB, quotes the cosmetic sentence, and asks for OS-side output.
- Write the acceptance test you would hand a customer for an NDR fabric: the full
ibdiagnetcommand line with--lsand--lwset, and one sentence on what a failure of that command does and does not prove.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, why their iDRAC says SDR while the OS says NDR on the same ConnectX-7 card, and what evidence you want before anyone opens the rack.
Sources
Facts in this lesson were checked against UFM Enterprise 6.24.1 SM defaults and Dell KB 000221452 (article version 4, 19 May 2026) both re-fetched and re-quoted 2026-09-07; rdma-core man pages (master), QM97XX and Q32xx/Q34xx specifications, IBUtils2 2.24.0. Dates are when each page was fetched.
- UFM Enterprise 6.24.1 - UFM Subnet Manager Default Properties · fetched 2026-09-07
- Dell KB 000221452 - Link Speed of NVIDIA ConnectX-7 Single Port NDR200 InfiniBand Card Shows as SDR · fetched 2026-09-07
- ibportstate(8) man page source - rdma-core master · fetched 2026-09-07
- iblinkinfo(8) man page source - rdma-core master · fetched 2026-09-07
- perfquery(8) man page source - rdma-core master · fetched 2026-09-07
- smpquery(8) man page source - rdma-core master · fetched 2026-09-07
- DGX SuperPOD with DGX B300 - Key Components of the DGX SuperPOD · fetched 2026-09-07
- QM97XX InfiniBand Switch Systems User Manual - Specifications · fetched 2026-09-07
- NVIDIA Q32xx/Q34xx XDR Switch Systems User Manual - Specifications · fetched 2026-09-07
- ibdiagnet Basic Commands - IBUtils2 Utility Documentation · fetched 2026-09-07
- NVIDIA SHARP 3.8.0 - Testing NVIDIA SHARP Setup · fetched 2026-09-07
- ibstat(8) man page source - rdma-core master · fetched 2026-09-07
- InfiniBand Trade Association - InfiniBand Roadmap · fetched 2026-09-07
- InfiniBand Fabric Utilities - MLNX_OFED 5.6-2.0.9.0 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Link surgery and PHY triageElsewhere in this course · Same ground: ibportstate, opensm and link
- The generation map: which SuperNIC is on the baseboardRA course · Same ground: speeds, sources and generation
- QoS: service levels, virtual lanes, SL2VL and VL arbitrationElsewhere in this course · Same ground: ibdiagnet, opensm and lane