Skip to content
nodeza.learn
InfiniBand course/3. Routing & design/Lesson 3.5

Quantum switches and the LinkX BOM

S3·E5The cable that fits and then cooks · Customer data center staging area, day one of the 2-SU bring-up

S3·E5Evaluate~35 minsources checked todayverified against QM97XX user manual (introduction, specifications, software management) and its SKU list; Q32xx/Q34xx XDR user manual (introduction, specifications); LinkX MMA4Z00-NS, MCP4Y10, MCP7Y00, MCA4J80, MCA4K00 and MMS4B10-XM pages; MFT 4.30.0 mlxlink; IBUtils2 2.24.0. Research notes fetched 2026-09-07

Builds on: Topologies: fat-tree, rail-optimized, twin-plane

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Choose between QM9700, QM9701 and QM9790 for a customer and defend the choice from the management and power differences.
  • Justify a Quantum-X800 model choice among Q3400-RA, Q3401-RD, Q3200-RA and Q3450-LD from rack, power and cooling constraints.
  • Convert a cabling map into a LinkX line-item list, counting OSFP cages rather than ports.
  • Judge a proposed cable BOM against the three classic errors: twin-port arithmetic, top type and copper reach.
  • State which parts of an XDR InfiniBand optics BOM cannot be sourced from the material this course has, and say so to a customer.

Episode 5 — The cable that fits and then cooks

The situation · Customer data center staging area, day one of the 2-SU bring-up

Day one of the bring-up, a staging area full of open boxes, and an installer holding a transceiver against a chassis it was never meant to sit in. Two rows over, 256 leaf-to-spine runs are laid out in 2 m passive DAC for a gap that measures about four metres. The partner who built the BOM is on speakerphone and certain the list is right, because the port counts add up. The night-shift operator has seen where this goes and is already at the label maker.

They add up against the wrong unit. The switches use twin-port OSFP cages supporting two transceiver engines in a single OSFP form-factor plug, creating 800 Gb/s electrical to the switch and 2x400G optics using two MPO-12/APC optical connectors.[7] One cage, two links, so the node-side quantity was never a single number taken from ports.[11] Reach is a catalogue fact rather than a judgement call: passive MCP4Y10 ships at 0.5, 1, 1.5 and 2 m, and active copper MCA4J80 covers 3, 4 and 5 m.[8][10]

The top type is why this part-number family exists at all. The 400G switches require finned-top 2x400G transceivers for additional cooling due to the reduced air flow inlets in the switches, while flat-top parts are what the DGX H100 and liquid-cooled ends take.[7][9] Both fit the same cage. Only one of them stays cool.

Count cages, measure the gap, then read the top type. A cable that fits can still cook.

Start with the switch on the rail in front of you.

1Quantum-2: three models, one decision that is really about management

The QM97XX family delivers 64 ports of NDR 400 Gb/s InfiniBand per port in a 1U standard chassis, with an aggregated bidirectional throughput of 51.2 Tb/s and more than 66.5 billion packets per second of capacity.[1] Physically that is 32 OSFP cages carrying the 64 ports, and with NVIDIA port-split technology the same switch provides a double-density radix for 200 Gb/s, supporting up to 128 ports of NDR200.[1] The specification table states the ASIC as the NVIDIA Quantum-2 IC and gives switching as 25.6 Tbps, the unidirectional figure - both numbers are official and you should quote the pair rather than pick one.[2][1]

The model split is a management decision. QM9700 and QM9701 are internally managed with an on-board subnet manager, running MLNX-OS with CLI, WebUI, SNMP and JSON management; QM9790 is externally managed and can use the advanced UFM feature sets.[1] The interface matrix is where the consequence lives: QM9700 has a front USB 3.0 type A port, one front MGT port, a front console, two replaceable PSUs and seven fans; QM9701 has the same interfaces but is DC powered using a busbar; QM9790 has no USB, no MGT and no console, only a front I2C connector in a USB 3.0 type A form.[1]

The rest of the platform data is what a Dell rack conversation needs. Environmentals: forward airflow 0 to 35 C, reverse airflow 0 to 40 C operational, non-operational minus 40 to 70 C, humidity 10 to 85 percent, altitude 3,050 m, noise 78.4 dBA.[2] Power: 1x or 2x 200-240 Vac at 10 A, with QM9700 typical 747 W on passive cables and 1,720 W maximum with active cables, and QM9790 at 640 W typical and 1,610 W maximum.[2] QM9701 measures 438 x 43.6 x 854 mm at 16.88 kg, is DGX mountable, and takes 40 to 59.5 Vdc at 720 W typical and 1,660 W maximum.[2] The ordering names and airflow codes: MQM9700-NS2F forward and MQM9700-NS2R reverse, MQM9790-NS2F and MQM9790-NS2R, MQM9701-NS2R, where P2C is power-to-connector (forward) and C2P is connector-to-power (reverse).[4]

Firmware follows the same split: the firmware of managed switch systems is handled automatically by MLNX-OS, while externally managed systems require a firmware burning tool such as flint or mlxburn from the MFT package.[3]

ProductSpeedPCIeRoleGPU generation
NIC
SuperNIC (no Arm)
SuperNIC (no Arm)
DPU
DPU
SuperNIC (Arm inactive)
DPU / storage processor
Ethernet switch
Ethernet switch
InfiniBand switch
InfiniBand switch

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

The shipped matrix is the DOCA product and stack view, not a Quantum switch table. Use it here for the adapter side of the pairing - ConnectX and BlueField generations - and keep the switch-side numbers in this lesson, which come from the QM97XX and XDR switch manuals.

2Quantum-X800: four models and a facilities conversation

The Q3400-RA and Q3401-RD 4U switches are the first to leverage 200 Gb/s per lane SerDes and feature 144 ports at 800 Gb/s distributed across 72 OSFP cages.[5] The throughput and radix table reads: Q3200-RA at 72 XDR ports over 36 OSFP cages and 57.6 Tb/s; Q3400-RA and Q3401-RD at 144 XDR ports over 72 cages and 115.2 Tb/s; Q3450-LD at 144 XDR ports over MPO connectors and 115.2 Tb/s.[5] The Q3200 is two independent switches within a single enclosure, each providing 36 ports at 800 Gb/s, and the spec table states two independent 28.8 Tb/s switches; note that some distributor listings describe it as 36 ports over 18 cages, and the manual is what you quote.[5]

Two platform features change a design. All Quantum-X800 switches include a dedicated OSFP InfiniBand in-band management port specifically for UFM, separated on the front panel from the other ports, so the full set of standard ports stays available for data - the FNM port.[5] The B300 reference architecture uses exactly that: UFM 3.5 nodes are connected to four FNM ports on the Q3400 switches.[14] And the switches feature optional router capabilities, facilitating the expansion of InfiniBand clusters across multiple sites.[5]

The Q3450-LD is the co-packaged-optics model: integrating silicon photonics directly with the switch ASIC eliminates the need for pluggable optical transceivers, reduces the high-speed electrical path to a few millimetres within the substrate, and cuts insertion loss to roughly 4 dB compared with 22 dB in traditional pluggable designs.[5] Physically it is 4U, 175 x 438 x 840 mm, 57.20 kg, with 144 MPO connectors, a 4xUDQ4 liquid-cooling connection type, two DGX-compliant busbar connectors at 40-60 V input, a maximum 4.3 kW with active cables, two replaceable fans and operational reverse airflow at 5 to 40 C.[6] For comparison, the Q3400-RA is 4U at 177.8 x 438 x 850 mm and 60 kg with 8 PSUs, 10 fans, 200-240 Vac at 10 A per PSU, 2,900 W typical passive and 7,000 W maximum active; the Q3401-RD is busbar powered at up to 6,327 W with reverse airflow 0 to 40 C; the Q3200-RA is 2U, 88 x 438 x 680 mm, 49 kg, 4 PSUs, 862 W typical and 3,486 W maximum.[5][6] All four carry an Intel CFL 4-core i3-8100H management CPU and the Quantum-3 switch ASIC.[6]

One honest limit for this whole segment: the switch CLI syntax this course teaches is MLNX-OS, and the only MLNX-OS subnet-manager page available is v3.12.6200 LTS, which predates Quantum-3. Attribute ib smnode and ib partition syntax to MLNX-OS, never to Quantum-X800, and confirm the switch OS on the platform in front of you.[3]

3LinkX arithmetic: cages, tops and reach

Everything about the NDR cable plant follows from one sentence: the switches use twin-port OSFP cages supporting two transceiver engines in a single OSFP form-factor plug, creating 800 Gb/s electrical to the switch and 2x400G optics using two MPO-12/APC optical connectors.[7] One cage, two links. Count cages.

Reach. Passive DAC MCP4Y10-Nxxx covers switch-to-switch runs up to 2 m, shipping at 0.5, 1, 1.5 and 2 m with 30AWG for the 0.5 to 2 m range, and with flat-top variants such as MCP4Y10-N001-FLT.[8][9] Active copper MCA4J80-Nxxx covers 3, 4 and 5 m, and adds an -FLT flat-top and an -FTF flat-to-finned part.[10] So the boundary in NVIDIA’s own catalogue is passive up to 2 m, active copper 3 to 5 m, then optics.[8][10]

Tops. This is thermal, not preference. The 400G IB and Ethernet switches require finned-top 2x400G transceivers for additional cooling due to the reduced air flow inlets in the switches, while flat-top transceivers, ACCs and DACs are required in the DGX H100; flat top is designated -FLT in the part number and is used for liquid-cooled switches and DGX H100 systems.[7][9] The B300 RA shows the same rule in a shipping design: compute nodes use an OSFP twin-port flat-top transceiver with MMF passive fibre, while the Ethernet storage switch side uses an OSFP twin-port finned transceiver.[14]

Splitters. MCP7Y00 splits 1:2 into two 400G, and MCP7Y50 splits 1:4 into four 200G; active copper versions exist at 4 and 5 m, and QSFP112 ends live on MCP7Y10 and MCP7Y40.[11] The four documented twin-port use cases are switch-to-switch at 800G or two 400G links to two switches, switch to two 400G ConnectX-7 or BlueField-3, switch to four 200G adapters via 1:2 splitters, and switch to the DGX H100 Cedar-7 complex.[7]

Fibre. Two fibres per transceiver, each may be a different length, but both need to be the same type - both straight or both splitters, not mixed - and both should be approximately the same length to avoid inducing different latency delays in the fibres, at 4.5 ns per metre.[7] NVIDIA supplies multimode crossover and straight fibre cables up to 100 m straight and 50 m for splitters.[7] The named assemblies are MFP7E10-Nxxx straight multimode, MFP7E20-Nxxx 1:2 multimode splitter, MFP7E30-Nxxx MPO-to-MPO single-mode and MFP7E40-Nxxx single-mode 1:2 splitter.[7][13]

Power. Twin-port multimode OSFP transceivers remain at 15 W for all configurations; a 400G single-port transceiver is 8 W; using 1:2 fibre splitters automatically creates 200 Gb/s transceivers by activating only two channels and reduces power from 8 to 5.5 W.[7] That is the number that turns a cable choice into a switch power number, since the QM9700’s own maximum figure is quoted with active cables.[2]

a cabling contract, not a switch feature
Rail-optimized fat tree — one slice: 3 nodes × 4 rails, 4 leaf, 2 spineleaf→spine: optics across the rownode→leaf: DAC ≤ 2 m / ACC 3–5 mS0spine (root)OSFP cagesS1spine (root)OSFP cagesL0leaf — rail 0OSFP cagesL1leaf — rail 1OSFP cagesL2leaf — rail 2OSFP cagesL3leaf — rail 3OSFP cagesnode n0 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112node n1 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112node n2 — 4 of 8 rails shownr0OSFP / QSFP112r1OSFP / QSFP112r2OSFP / QSFP112r3OSFP / QSFP112
2 SU
Path

Click one HCA for the source, another for the destination. Hops are counted as switch ASICs traversed — a link count is hops + 1.

Rail-optimized fat tree

  • Every GPU's rail-N port lands on leaf N: "Traffic per rail of the DGX H200 systems is always one hop away from the other 31 nodes in a SU. Traffic between nodes, or between rails, traverses the spine layer."
  • Same rail, different node = 1 switch hop. Different rail = 3 hops through a spine.
  • H200 SuperPOD compute fabric: QM9700 NDR, "Rail-optimized, non-blocking, full fat-tree network with eight NDR400 connections per system".
  • Nothing is enabled on the switch to get this. It is decided by where the technician plugs the cable.

FAE angle. On an eight-rail PowerEdge XE9680 the rail map is the deliverable. Dell's slot priority for NDR400 1P cards is 33, 37, 35, 39, 32, 36, 34, 38, leaving 31 and 40 for the storage NIC or a BlueField-3 — DPUs over 75 W are legal only in 31/40. Check it against ibdev2netdev and each card's numa_node, then validate with ibdiagnet --rail_validation (→ ibdiagnet2.rails).

DGX H200 SuperPOD RA — Network Fabrics

What each link class costs in LinkX parts
  • Node-to-leaf runs stay inside or beside the rack; leaf-to-spine runs cross the row. That distinction is the whole cable BOM.
  • NDR copper reach in NVIDIA's own catalogue: passive DAC MCP4Y10-Nxxx up to 2 m (0.5 / 1 / 1.5 / 2 m), active copper MCA4J80-Nxxx at 3, 4 and 5 m. Anything longer is optics.
  • Twin-port OSFP is the arithmetic trap: one cage carries "two transceiver engines… creating 800Gb/s electrical to the switch and 2x400G optics". One MCP4Y10 is one 800G switch-to-switch link; one MCP7Y00 1:2 splitter is two 400G node links. Count OSFP cages, not ports.
  • Top type is thermal, not preference: finned-top at the air-cooled switch, flat-top (-FLT) at DGX and liquid-cooled ends, -FTF ("flat to finned") for an asymmetric run.
  • Fiber rules: multimode "up to 100-meters straight and 50-meters for splitters"; both fibers of a twin-port link same type and roughly the same length — 4.5 ns/m of delay.
  • A QM9700 gives 64 NDR 400 Gb/s ports over 32 OSFP cages (or 128 × NDR200 split); a Q3400-RA gives 144 × 800 Gb/s over 72 OSFP cages.
mlxlink -d <mst_dev> -p 1 -m       # vendor, PN, length, temperature
ibdiagnet --get_cable_info         # fabric-wide → ibdiagnet2.cables

LinkX twin-port OSFP overview

Scalable-unit arithmetic — 2 SU

DGX H200 SuperPOD compute fabric — QM9700 NDR

Sources: opensm(8) · UFM SM defaults · MLNX-OS Subnet Manager · SHARP environment · Quantum-X800 switches · MCA4J80 ACC · ibdiagnet · ibdiagnet dump files · Dell XE9680 technical guide

Cable overlay. Node-to-leaf runs stay in or beside the rack; leaf-to-spine crosses the row. Click a link of each class and price it in your head against the reach table before reading the panel.

4The XDR cable gap, said out loud

For NDR the ordering tables are complete enough to build a BOM from. For XDR-generation InfiniBand optics they are not, and pretending otherwise is how a quote goes wrong.

What exists in the fetched material: MCA4K00, a 1600 Gb/s OSFP active copper cable with eight high-speed electrical copper pairs each operating at up to 200 Gb/s, whose firmware supports InfiniBand, Ethernet and NVL5 and detects the protocol of the switch it attaches to; its only ordering length is 1.1 m, in three OPNs, and its stated applications are 1600 Gb/s NVL5 switch-to-switch and 800 Gb/s HCA-to-HCA.[18][19] Also catalogued are MCA4K50 and MCA7K10, the latter a 1600 Gb/s to 2x800 Gb/s OSFP to 2xOSFP active copper splitter.[13]

What does not exist here: an XDR InfiniBand-protocol transceiver ordering table. The one 1600G optic captured in detail, MMS4B10-XM, is a Gen2 TRO 1600 Gb/s 2xDR4 single-mode transceiver supporting Ethernet protocol only, with 500 m maximum reach, 19 W maximum power and CMIS 5.0 - and its 500 m figure assumes two optical patch panels or four optical connectors in the link.[12] Other 1600G optics are named in the catalogue index - MMS4C1X FRO and MMS4A00, both 2xDR4 twin-port OSFP at 1310 nm up to 500 m - but without fetched ordering tables.[13]

So the honest position with a customer is: an XDR copper BOM can be built from the named active-copper parts and their published lengths, and an XDR optical BOM has to be confirmed against NVIDIA’s current ordering guide or your NVIDIA contact before it goes on a quote. Say that rather than transferring an NDR part number to an XDR row.[12][13]

5Judging a BOM, and proving what was installed

Review a 2-SU NDR cable BOM

A partner sends the cable BOM for a 2-SU NDR pod: 63 nodes at 8 rails, 16 leaf and 8 spine QM9700s, leaves and spines in adjacent racks about 4 m apart, one liquid-cooled GPU chassis row.

Their list: 504 MCP4Y10-N002 for node-to-leaf; 256 MCP4Y10-N002 for leaf-to-spine; no fibre; no patch panels; all parts finned-top.

The review, defect by defect:

  1. Copper reach on the leaf-to-spine runs. At about 4 m, passive DAC is out: MCP4Y10 tops out at 2 m and MCA4J80 active copper covers 3, 4 and 5 m.[8][10] Either move those 256 runs to MCA4J80-N004 or, if the row grows, to optics with fibre and MPO counts.[7]
  2. Twin-port arithmetic on the node runs. 504 node ports is not 504 switch cages. A leaf’s cage feeding two 400G node ports is one MCP7Y00 1:2 splitter, so the node-side count must be derived from cages at the leaf and ports at the node, not from a single number.[11][7]
  3. Top type at the liquid-cooled end. Flat-top -FLT is required for liquid-cooled and DGX ends while finned-top belongs at the air-cooled switch, and for a run with one of each there is the -FTF flat-to-finned active copper part.[9][10]
  4. Missing passive plant. If any run becomes optics, the fibre assemblies and the MPO patch-panel count belong in the same quote, with both fibres of a twin-port link the same type and approximately the same length at 4.5 ns per metre.[7]
  5. Power knock-on. Active cables move the switch from its typical figure toward its maximum: QM9700 is 747 W typical with passive cables and 1,720 W maximum with active ones.[2] A BOM that changes cable class changes the rack power line too.

The deliverable back to the partner: a corrected table with one row per run class - node-to-leaf, leaf-to-spine, management - each with quantity, part number, length, top type at each end, and a note on any run that crosses into optics.

Case closed

How it ended

The corrected BOM: one row per run class - node-to-leaf, leaf-to-spine, management - with quantity, part number, length and top type. Cross-row runs move to active copper at 4 m, and the rack power line is re-read with them - QM9700 is 747 W typical on passive cables and 1,720 W maximum with active ones.[10][2] At handover you read what arrived, per port: mlxlink -d <mst_dev> -p 1 -m.[15]

What you tell the partner: none of these is a bad cable. The BOM counted ports where the switch counts cages, and picked the wrong top type.

Acceptance passes: rail validation clean, routing clean, right engine in the log. The operator’s second label of the week, on the spine rack: CAGES, NOT PORTS. The network lead closes a full notebook. Case closed.

Lab

Goal: read the installed cable and module inventory from the host side of the Dell lab and compare it with what the paperwork claims.

Pre-flight inventory (read-only). sudo mst start then sudo mst status -v for the MST device path; ibstat for device, port state and rate; ibdev2netdev for the mapping. Save all three with today’s date.

  1. sudo mlxlink -d <mst_dev> -p 1 - expected: the link status block for port 1, including state and speed. Note that mlxlink errors, warnings and notes appear on stderr, which matters if you pipe this into a file.[15]
  2. sudo mlxlink -d <mst_dev> -p 1 -m - expected: module information with vendor, part number, length and temperature. Record all four. This is the single most useful command on this page and it is entirely read-only.[15]
  3. sudo mlxlink -d <mst_dev> -p 1 -c - expected: physical counters and BER data for the port. Save the output; it is the baseline you would compare against after any cable change.[15]
  4. Compare what the module reports against what the lab paperwork or the label says: vendor, part number and length. Expected: agreement. Any mismatch is a finding worth writing up even in a lab, because it is exactly the class of error this lesson is about.
  5. Write the one-paragraph handover note you would give a customer: which fields you read, which command produced them, and what a discrepancy in each field would imply.
  6. Re-run the pre-flight commands and diff. Expected: identical; nothing here mutates the card. Do not use mlxlink to change port state in this lab - the tool’s own guidance warns against disabling ports that connect hosts to unmanaged switches.[15]
  7. Optional, only in a customer lab with a cabled fabric and written permission: ibdiagnet --get_cable_info and compare ibdiagnet2.cables against the shipped BOM, flagging any mixed finned and flat run.[17] Expected: one row per cable with vendor and part data. Note in your write-up that the Cable Diagnostic plugin is deprecated and will be entirely removed in the next version, so this evidence path has a shelf life.[16]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell configuration specialist, in five sentences, why counting InfiniBand ports is the wrong way to price a cable BOM, and what to count instead.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against QM97XX user manual (introduction, specifications, software management) and its SKU list; Q32xx/Q34xx XDR user manual (introduction, specifications); LinkX MMA4Z00-NS, MCP4Y10, MCP7Y00, MCA4J80, MCA4K00 and MMS4B10-XM pages; MFT 4.30.0 mlxlink; IBUtils2 2.24.0. Research notes fetched 2026-09-07. Dates are when each page was fetched.

  1. QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
  2. QM97XX User Manual: Specifications · fetched 2026-09-07
  3. QM97XX User Manual: Software Management · fetched 2026-09-07
  4. QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual (SKU and OPN list) · fetched 2026-09-07
  5. NVIDIA Q32xx and Q34xx XDR 800Gb/s InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
  6. NVIDIA Q32xx/Q34xx XDR Switch Systems User Manual: Specifications · fetched 2026-09-07
  7. MMA4Z00-NS 800Gb/s Twin-port OSFP, 2x400Gb/s Multimode 2xSR4, 50m: Overview · fetched 2026-09-07
  8. MCP4Y10-Nxxx 800Gbps Twin-port OSFP passive DAC: Ordering Information · fetched 2026-09-07
  9. MCP4Y10-Nxxx 800Gbps Twin-port OSFP passive DAC: Overview · fetched 2026-09-07
  10. MCA4J80-Nxxx 800Gbps Twin-port OSFP InfiniBand LACC: Ordering Information · fetched 2026-09-07
  11. MCP7Y00-Nxxx 800Gbps Twin-port OSFP passive DAC splitter: Overview · fetched 2026-09-07
  12. MMS4B10-XM TRO 1600Gbps OSFP Twin-Port 2xDR4 transceiver · fetched 2026-09-07
  13. NVIDIA Networking Interconnect (LinkX docs index) · fetched 2026-09-07
  14. SuperPOD with DGX B300 RA: Key Components · fetched 2026-09-07
  15. mlxlink Utility | NVIDIA Firmware Tools (MFT) Documentation v4.30.0 · fetched 2026-09-07
  16. Cable Diagnostic Plugin | IBUtils2 · fetched 2026-09-07
  17. ibdiagnet Basic Commands | IBUtils2 · fetched 2026-09-07
  18. MCA4K00 1600Gbps to 1600Gbps OSFP Active Copper Cable: Product Specifications · fetched 2026-09-07
  19. MCA4K00: Ordering Information · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.