Skip to content
nodeza.learn
RA course/2. Sizing/Lesson 2.2

Dual plane: two fabrics, not a bond

S2·E2The bond with no partner · Phase-two design review, video call, ten weeks to the filing

S2·E2Analyze~30 minsources checked todayverified against HGX AI Factory RA Networking Physical Topologies and Components re-fetched 2026-09-07 (dual/single plane wording; 8x400 minimum and 16x400 recommended); NVL72 AI Factory Networking Physical Topologies re-fetched 2026-09-07; NIC Configuration Operator Spectrum-X page re-fetched 2026-09-07 (device IDs 1023 and a2dc; hwplb ConnectX-8 only; CNP DSCP 0 vs 48); Network Operator 26.4.0 Spectrum-X multiplane matrix re-fetched 2026-09-07 (swplb GA / hwplb tech preview); Cumulus Linux 5.13 ECMP page re-fetched 2026-09-07 (no adaptive routing on 800G)

Builds on: Rail-optimized design and what breaks it

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain dual plane as two independent fabrics rather than a link aggregation and cite the sentence that says so.
  • Derive the leaf-count and transceiver consequences of choosing dual plane over single plane at the same per-GPU bandwidth.
  • Determine from a NIC device ID whether hardware plane load balancing is even eligible on a given card.
  • Distinguish the plane decision from the rail decision when reading a customer's quote.

Episode 2 — The bond with no partner

The situation · Phase-two design review, video call, ten weeks to the filing

The network lead opens with his condition, notebook flat on the desk: bond the two 400 Gb/s halves of every GPU link into one LACP aggregate, the way every other server in the hall is attached. The Dell SE has written “check with networking” in the promise spreadsheet. You are networking, and the answer is no — useful only if you can say why in his terms.

There is nothing on the far end to negotiate with. The Enterprise RA is explicit: “Dual Plane Topology involves breaking the interface to 2x400 Gb/s interfaces. Every such interface is then connected to a different leaf switch, and every such leaf switch is part of an independent fabric.”[1] Two independent fabrics share no control plane, so no peer exists to agree on an aggregation — and none is needed: “Tracking of each plane, load balancing, and failure handling is handled by the ConnectX-8 SuperNIC on the hardware level.”[1]

That is the problem the topology solves. A redundant leaf pair shares an address plan, a routing domain and an upgrade window; two independent fabrics share none of it, which is why “a single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion”.[7] With a filing date fixed, the insurer needs a fabric that degrades rather than dies.

An NVIDIA PM joins for ten minutes; asked when hardware plane load balancing leaves tech preview, she says “not announced”.[6]

Two planes are two fabrics, not two cables.

Segment 1 starts with the sentence that settles the argument.

1The sentence that settles the argument

Dual plane is the most misread line item in a Blackwell quote, and one sentence in the Enterprise RA settles it: “Dual Plane Topology involves breaking the interface to 2x400 Gb/s interfaces. Every such interface is then connected to a different leaf switch, and every such leaf switch is part of an independent fabric that scales to 1024 interfaces of 400 Gb/s.”[1] The GB300 NVL72 RA says the same thing about the same silicon: “With each GPU generating 800 Gb/s bandwidth through the ConnectX-8 SuperNICs, dual plane topology involves breaking the interface to 2x400 Gb/s interfaces.”[2]

Three words carry the whole design. Different leaf switch — not a redundant pair of the same leaf. Independent fabric — not two halves of one fabric with a link between them. And 1024 interfaces of 400 Gb/s is the size of one plane, not the size of the cluster.[1]

Nothing about this is a bond. “Tracking of each plane, load balancing, and failure handling is handled by the ConnectX-8 SuperNIC on the hardware level.”[1] There is no LACP peer, no MLAG, no bond device to configure — the adapter owns the decision. That is why a customer network team that plans to aggregate the two halves is planning against a topology in which no common peer exists.

rail →rail 0rail 1rail 2rail 3rail 4rail 5rail 6rail 7node 0n0·g0n0·g1n0·g2n0·g3n0·g4n0·g5n0·g6n0·g7node 1n1·g0n1·g1n1·g2n1·g3n1·g4n1·g5n1·g6n1·g7node 2n2·g0n2·g1n2·g2n2·g3n2·g4n2·g5n2·g6n2·g7node 3n3·g0n3·g1n3·g2n3·g3n3·g4n3·g5n3·g6n3·g7800Geach 800 Gb/s ConnectX-8 port broken out to 2 interfaces, one per fabricfabric 1leaf 0·P1leaf 1·P1leaf 2·P1leaf 3·P1leaf 4·P1leaf 5·P1leaf 6·P1leaf 7·P1fabric 2leaf 0·P2leaf 1·P2leaf 2·P2leaf 3·P2leaf 4·P2leaf 5·P2leaf 6·P2leaf 7·P2spine 1spine — fabric 1: independent, scales to 1,024 interfaces of 400 Gb/sspine 2spine — fabric 2: independent, scales to 1,024 interfaces of 400 Gb/s
showing 4 of 8 nodes · 8 rails · solid links are fabric 1; a selected GPU draws its link in every fabric

Rail-optimized

GPUs are connected "using a rail-optimized network topology through their respective NVIDIA ConnectX-8 SuperNICs" — the same-index NIC on every node lands on the same leaf. Rail count equals GPUs per node.

Click any GPU cell to follow its SuperNIC to a leaf and a fabric.

Dual planerecommended at 64+ nodes

2 × 400 Gb/s per GPU (800 Gb/s ConnectX-8 port, broken out)

"Each GPU generating 800 Gb/s bandwidth through the ConnectX-8 SuperNICs, dual plane topology involves breaking the interface to 2x400 Gb/s interfaces" — each interface goes "to a different leaf switch", and "Each leaf switch is part of an independent fabric that scales to 1024 interfaces of 400 Gb/s". Load balancing is done "by the ConnectX-8 SuperNIC at hardware level".

Transceiver: twin-port transceiver — MMA4Z00-NS, 800 Gb/s twin-port OSFP, 2 × 400 Gb/s, multimode, 50 m max reach. The RA states the fork in the optics: "dual-plane using twin-port transceivers or single-plane using single-port OSFP transceivers".

Host side: NUM_OF_PLANES_P1=2 (hwplb; 0 for swplb/uniplane) · LAG_RESOURCE_ALLOCATION=1 for hwplb · CNP DSCP 48 for hwplb vs 0 · multiplaneMode: "swplb" | "hwplb"

Network Operator 26.4.0 matrix: Dual-Plane on ConnectX-8 with B300 / GB300 — "swplb" GA, "hwplb" tech preview. "hwplb multiplane mode is only supported on ConnectX-8 (device ID 1023)".

  • It is not a bond and not redundancy pairing: two independent fabrics with no links between them.
  • BOM consequence: leaf count doubles versus single plane at the same per-GPU bandwidth, and the customer must operate two IP fabrics.
  • Twin-planar resilience claim on the DGX B300 SuperPOD RA: "A single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion".
RA topology pageMMA4Z00-NSNIC Configuration OperatorNetwork Operator 26.4.0
Sizing this picture

8 nodes × 8 rails = 64 server-facing 400 Gb/s interfaces per fabric × 2 fabrics = 128 total.

Rail-aligned needs at least one leaf per rail per fabric: 8 rails × 1 leaf per rail = 8 leaves per fabric, 16 leaves total. Pure port arithmetic would only need 1 per fabric — the difference is what rail alignment costs. derived

Leaf radix: SN5610 / SN5600 presents 64 OSFP cages = 128 × 400 Gb/s logical; a non-blocking leaf faces 64 of them at the servers and 64 at the spines. derived — the RAs do not print a per-leaf split

Transceiver at the NIC: twin-port OSFP (MMA4Z00-NS class) — this is a BOM fork, not a field option.

8 GPUs / node: HGX B300 node: "Eight NVIDIA B300 GPUs on an HGX B300 baseboard" with "Eight NVIDIA ConnectX-8 SuperNICs per NVIDIA HGX B300 baseboard" — a 1:1 GPU-to-SuperNIC ratio at "800 Gb/s (2 x 400Gb/s Ethernet) per GPU". 8 GPUs per node means 8 rails.

8 nodes: Below the 64-node line where the RA starts recommending dual plane.

Keeping traffic on its rail from the host

  • NCCL_IB_HCA selects the HCAs, format <hca>[:<port>[:<rail>[:<plane>]]]; ^ excludes, = forces an exact match. Default is "All available IB interfaces".
  • NCCL_IB_TC "Defines the InfiniBand traffic class field", default 0, range 0–15 — set it to the traffic class the switch marks as RoCE.
  • NCCL_IB_SL "Defines the InfiniBand Service Level", default 0, range 0–15. NCCL_IB_GID_INDEX "Defines the Global ID index used in RoCE mode", default -1.
  • NCCL_NET_GDR_LEVEL controls GPUDirect RDMA between a NIC and a GPU: LOC (disabled), PIX, PXB, PHB, SYS. NCCL_SOCKET_IFNAME auto-selects, "favoring interfaces starting with ib".
  • NicInterfaceNameTemplate gives "rail/plane-based network device naming via udev rules" — that is how a rail survives a reboot with a stable interface name.
  • SpectrumXRailPoolConfig carries "rail topology, PF selection, IPAM binding, resource exposure"; CIDRPool allocates IPs per rail or per rail/plane.

FAE angle. "We bought the RA and we get 60% of line rate on all-reduce" is the escalation. Check in order: (1) is the cabling actually rail-aligned, or did the installer cable node-sequentially? (2) is NCCL_IB_HCA pinned so rank N uses rail N? (3) is NCCL_IB_TC set to the traffic class the switch marks as RoCE? Two of those three are free to fix.

Where the RA stops. The RA names six logical networks over one physical fabric — GPU Compute, CPU Converged, Storage, Customer, Support Server Networking and OOB — with "VLAN isolation ... to provide logical separation". It publishes no BGP/EVPN configuration, no IP addressing, no MTU, no DSCP/PFC/ECN values and no adaptive-routing or congestion-control parameters. Those live in the Cumulus and Spectrum-X software docs.

HGX AI Factory topologiesNVL72 AI Factory topologiesDGX B300 SuperPOD fabricsDGX B300 XDR fabricsGB200 SuperPOD fabricsNCCL env varsSN5600 series datasheetHGX logical architectureNVL72 logical architectureQuantum-X800
Fabric split is on. Follow one GPU's 800G port through the breakout and note that the two halves never meet again inside the diagram.

2Single plane is a supported option, priced in bandwidth

The single-plane alternative is documented, not improvised. Each interface “is connected to a leaf switch within a single compute fabric that scales to 1024 interfaces of 400 Gb/s as part of the reference architecture. While this approach reduces the total GPU bandwidth by 50%…”[1] Fifty percent is NVIDIA’s number, not an estimate, and the RA positions the option for workloads that do not warrant maximum throughput.[1]

The RA’s own reference configurations show where it expects each. The logical architecture page describes the GPU Compute east-west network as a “separate, isolated dual-plane” for both the 64-node and the 128-node configurations, so dual plane is the shape the document builds at scale.[3] Meanwhile the Components page prints an east-west floor and a recommendation per node: minimum “400 GB/s (8x 400 Gb/s Ethernet NICs)” and recommended “800 GB/s (16x 400 Gb/s Ethernet NICs using breakout)”.[8] Read the units carefully — the aggregate is in bytes and the per-NIC figure is in bits, and eight 400 Gb/s NICs is 3,200 Gb/s which is 400 GB/s, so the page is internally consistent even though it looks like a typo.

Note also what the plane choice is not. It is orthogonal to rails: rails decide which leaf a given NIC index lands on, planes decide which fabric. A dual-plane pod is still rail-optimized, and the same-index-NIC rule applies inside each plane.[7]

3Why the RAs break 800G into 2x400G at all

There is a mechanism reason, not just a topology reason, and it comes from the switch software. Cumulus Linux documents adaptive routing on “Switches with the Spectrum-4 ASIC at 400G and 200G speeds” and then states the constraint plainly: “The Spectrum-4 switch does not support adaptive routing on 800G links.”[9] A customer who insists on running the SuperNIC’s 800 Gb/s straight into one switch port gives up adaptive routing on that link — the very mechanism that keeps elephant flows from colliding on a single hashed path.

The rest of the adaptive-routing page is worth knowing before a design review, because two of its facts change plans. The default link-utilization threshold on an interface is 70 percent, and it is configurable.[9] And enabling or disabling adaptive routing “globally or on an interface reloads the switchd service”, which resets ports — so it is a design-review decision, never a live-cluster change.[9] Adaptive routing also does not run on layer-3 subinterfaces, SVIs, bonds or bond members, and “Adaptive routing does not make use of resilient hashing.”[9]

That last exclusion is another reason the bond idea dies: even if you could aggregate the two plane interfaces, bonds are outside adaptive routing’s supported list.[9]

4The host side: modes, device IDs and what is actually GA

On the host, the plane decision is firmware and driver configuration, and it is gated by silicon. The NIC Configuration Operator applies LINK_TYPE_P1=2 for Ethernet, SRIOV_EN=1, NUM_OF_VFS=1, plus the two parameters that carry the plane decision: NUM_OF_PLANES_P1 and LAG_RESOURCE_ALLOCATION, the latter set to 0 for the software modes and 1 for hwplb.[5] Congestion-notification marking moves with the mode too: CNP DSCP is 0 for swplb, uniplane and none, and 48 for hwplb.[5]

The hard constraint is a device ID. “The hwplb multiplane mode is only supported on ConnectX-8 (device ID 1023)”; the other supported adapter, the BlueField-3 SuperNIC, is device ID a2dc and gets the software modes only.[5] Reading the device ID off a card is therefore a five-second answer to “can we use hardware plane load balancing here”.

The maturity matrix matters just as much in a Dell deal. Network Operator 26.4.0 lists Single-Plane (none) as GA on BlueField-3 SuperNIC, ConnectX-7 and ConnectX-8 with H100/H200/B200 and GB200; Dual-Plane on ConnectX-8 with B300 and GB300 with swplb GA and hwplb tech preview; and Quad-Plane on ConnectX-8 with B300 on the same GA/tech-preview split.[6] NVIDIA’s stated purpose for the feature is scale: multiplane “splits each SuperNIC across two or more independent network planes — enabling Ethernet to scale from thousands to hundreds of thousands of GPUs in a flat, two-tier topology, with improved performance and resiliency over single-plane networks.”[6]

ProductSpeedPCIeRoleGPU generation
NIC
SuperNIC (no Arm)
SuperNIC (no Arm)
DPU
DPU
SuperNIC (Arm inactive)
DPU / storage processor
Ethernet switch
Ethernet switch
InfiniBand switch
InfiniBand switch

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

ConnectX-8 row first. Then open the BlueField-3 SuperNIC row and note which one can carry hwplb.

5What dual plane does to the BOM

Say the consequence out loud, because it is the part that shows up in price. Dual plane doubles the server-facing 400 Gb/s interface count relative to single plane, which doubles the leaf count for the same GPU population — and buys twice the per-GPU bandwidth in exchange, since single plane is documented as exactly 50 percent.[1] It also fixes the transceiver: the RA expresses the choice as “dual-plane using twin-port transceivers or single-plane using single-port OSFP transceivers”.[4] That is a part-number fork on the quote, not a field option.

The leaf side is the SN5610 or SN5600 class: 64 OSFP cages that become 128 logical 400 GbE interfaces in 2U.[10] How many of those cages face servers versus spines is not published in the RAs — the non-blocking half-and-half split you will see everywhere, including in the next lesson, is arithmetic, and you should label it as yours every time you say it.

The operational line item is the one customers forget: two independent fabrics means two address plans, two routing domains, two monitoring scopes and two upgrade windows.[1] Before quoting dual plane, ask whether their automation assumes one fabric per NIC. On the DGX SuperPOD side the equivalent design is called twin-planar, with “Each GPU has 2x 400GbE connectivity through two different planes”, and the RA’s payoff sentence is that “A single switch failure, transceiver failure or cable failure will not lead to a catastrophic job abortion.”[7]

Read a GB300 quote and say what topology it actually buys

A Dell account sends a GB300 NVL72 quote for one rack and asks you to sanity-check the network lines.

  1. Fix the per-GPU bandwidth from the platform, not the quote. Each GPU generates 800 Gb/s through its ConnectX-8 SuperNIC.[2] A GB300 NVL72 rack is 72 GPUs, so the east-west requirement is 72 x 800 Gb/s.
  2. Apply the dual-plane rule. Each 800 Gb/s interface is broken to 2x400 Gb/s, each landing on a different leaf in an independent fabric.[2] Server-facing 400 Gb/s interfaces = 72 x 2 = 144, i.e. 72 per plane. (This multiplication is mine; the RA states the rule, not the total.)
  3. Check the count against the published fabric cap. Each plane “scales to 1024 interfaces of 400 Gb/s”.[1] 72 is far inside it, so one rack does not stress the design — which is the honest answer to “will this scale”.
  4. Read the optics line. Twin-port transceivers means the quote is built for dual plane; single-port OSFP means single plane and half the GPU bandwidth.[4][1] If the line says single-port and the drawing says dual plane, the drawing is wrong.
  5. Read the NIC line for the mode question. If the customer intends hardware plane load balancing, the adapter must be ConnectX-8, device ID 1023, and hwplb is still listed as tech preview in Network Operator 26.4.0.[5][6]
  6. Check the converged network separately. GB300 compute trays take “dual 400 Gb/s ports” to two separate switches and management nodes take “four 200 Gb/s ports”; that is a different fabric from the two compute planes and is sized in the next lesson.[2]
  7. Write the one-line verdict. “Quote is dual plane if and only if the optics line is twin-port; compute planes are two independent fabrics of 72 interfaces each; hwplb is tech preview so the design should assume swplb.”

Episode 2 — Case note: ask for the optics line

How it ended

You stop arguing about the drawing and ask for the transceiver line instead: twin-port OSFP is the RA’s dual plane, single-port OSFP is single plane.[4] Phase two is quoted twin-port, so the two fabrics were bought before anyone said the word bond. What you say: “There is no LACP partner — the halves land in separate fabrics on purpose and the SuperNIC owns the load balancing and the failover in hardware.[1] What you inherit is two fabrics to address, monitor and upgrade.” The network lead writes that down. Procurement, who has been silent for forty minutes, asks the only question they ever ask: how many switches, and what is the lead time.

Lab

Read-only inspection on the Dell lab’s ConnectX or BlueField-3 host. Nothing here writes firmware.

  1. Pre-flight inventory. Identify the card and its PCI address before touching anything: 
    lspci -nn | grep -i mellanox
ibv_devinfo | grep -E 'hca_id|fw_ver|board_id'
    Expected: a device ID in the vendor:device form. Record it — the next step depends on it.
  2. Answer the hwplb eligibility question from the device ID. hwplb is supported only on ConnectX-8, device ID 1023; BlueField-3 SuperNIC is a2dc.[5] Write down which of the two (or neither) your card is, and therefore whether hardware plane load balancing is even on the table for this host.
  3. Read the plane-related firmware configuration without changing it. 
    sudo mlxconfig -d /dev/mst/mt<device> q | grep -Ei 'NUM_OF_PLANES_P1|LAG_RESOURCE_ALLOCATION|LINK_TYPE_P1|SRIOV_EN|NUM_OF_VFS'
    Expected: current values printed. If mst devices are missing, start the tools with sudo mst start (this loads a kernel module and starts a service; roll it back with sudo mst stop when finished). If a parameter does not appear at all, that is itself the finding: the firmware on this card does not expose the multiplane parameters.
  4. Record, do not change. Copy the query output into your notes verbatim. This lab is read-only: mlxconfig set writes non-volatile configuration and requires a power cycle to take effect, so it is out of scope here.
  5. Confirm the link type and speed the card is actually running. 
    ibdev2netdev
ethtool <iface> | grep -E 'Speed|Duplex|Link detected'
    Expected: a speed consistent with the installed optics. Note whether this host could ever present two 400 Gb/s netdevs from one cage.
  6. Optional, customer lab only. Observe a real dual-plane leaf pair and its twin-port transceivers, and confirm from the switch side that the two leaves carry no links to each other. Read-only commands only.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

A Dell SE reads dual plane on a GB300 quote and asks whether it is just redundancy so the cluster survives a switch failure. Explain in five sentences what it really is what it buys and what it costs the operations team.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against HGX AI Factory RA Networking Physical Topologies and Components re-fetched 2026-09-07 (dual/single plane wording; 8x400 minimum and 16x400 recommended); NVL72 AI Factory Networking Physical Topologies re-fetched 2026-09-07; NIC Configuration Operator Spectrum-X page re-fetched 2026-09-07 (device IDs 1023 and a2dc; hwplb ConnectX-8 only; CNP DSCP 0 vs 48); Network Operator 26.4.0 Spectrum-X multiplane matrix re-fetched 2026-09-07 (swplb GA / hwplb tech preview); Cumulus Linux 5.13 ECMP page re-fetched 2026-09-07 (no adaptive routing on 800G). Dates are when each page was fetched.

  1. Networking Physical Topologies — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
  2. Networking Physical Topologies — NVIDIA NVL72 AI Factory (GB300) Enterprise RA · fetched 2026-09-07
  3. Networking Logical Architecture — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
  4. Networking Hardware — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
  5. NVIDIA Spectrum-X NIC Configuration — NIC Configuration Operator (Tech Preview) · fetched 2026-09-07
  6. NVIDIA Spectrum-X Ethernet Networking Platform — Network Operator 26.4.0 · fetched 2026-09-07
  7. Network Fabrics — DGX SuperPOD with DGX B300, Spectrum-4 Ethernet and DC Busbar Power RA · fetched 2026-09-07
  8. Components — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
  9. Equal Cost Multipath Load Sharing including Adaptive Routing — Cumulus Linux 5.13 · fetched 2026-09-07
  10. NVIDIA Spectrum SN5600 Series Switches Datasheet (Dell-branded) · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.