Multiplane fabrics and multi-tenant performance isolation
S4·E5Quad-plane on the hardware they already own · NVIDIA briefing room, second day of an executive briefing
Builds on: Telemetry-based congestion control: three loops, not one, BGP-EVPN symmetric IRB for tenant isolation
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Explain what a plane is, how a SuperNIC is split across planes, and how a degraded plane is removed from service.
- Read the Network Operator multiplane support matrix and decide which architecture a given adapter can run.
- Identify the Kubernetes objects that define a Spectrum-X deployment and the RA version field inside them.
- Analyse the tension between multi-tenant VRF isolation and the default-VRF-only rule for adaptive routing.
Episode 5 — Quad-plane on the hardware they already own
The acceptance number is a week old and the pod-two order is on the infrastructure director’s desk, unsigned. He puts two things on the table. First, a quote of yours listing Network Operator 26.7.0, next to a printed NVIDIA page that stops at 26.4.0, and he wants to know which of the two is wrong. Second, the real ask behind the meeting: their existing BlueField-3 fleet, running quad-plane, next quarter. The SE’s spreadsheet is at sixty-one rows and has grown a column headed “which document says so”.
Take the second one first, because it decides the design. A plane is not a switch setting. Each SuperNIC, corresponding to a GPU, is split into two or more ports, each connecting to a separate network plane, and the adapter load balances packet by packet in hardware[4] — which is how NVIDIA claims up to 128,000 GPUs in two tiers, sixty-four times more than single-plane networks[5]. The support matrix then decides who can do it: dual-plane and quad-plane list the ConnectX-8 SuperNIC only, while BlueField-3 and ConnectX-7 sit in the single-plane row[1]. Their fleet cannot reach planes through software.
The version question is not a contradiction either. The Network Operator page maps RA 2.2 to 26.4.0 and stops; the validated solution stack lists 26.7.0 in its v2.3.1 row[1][2]. Two indexes, both current on the day they were fetched. Read the support matrix before the release notes, and the release notes long before the roadmap.
1What a plane is
Multiplane is a topology decision that lives half in the adapter. In Dell’s wording, each SuperNIC — which corresponds to a GPU — is split into two or more ports, each of which connects to a separate network plane, and the SuperNIC intelligently routes network traffic to different planes, load-balancing packet by packet in hardware.[4] The claimed payoff is scale on a shallow fabric: Spectrum-X multiplane enables up to 128,000 GPUs in two tiers, which NVIDIA calls 64 times more than single-plane networks.[5] The technical blog states the same scale as over 128,000 endpoints using shallow two-tier fat-trees.[3]
Resilience is the second argument. The SuperNIC leverages high-frequency telemetry from the switch and integrates with the NVIDIA Collective Communications Library to route traffic away from planes with problematic link outages or congestion issues.[4] That is the same stateful filtering described in the previous lesson: any plane experiencing end-to-end congestion or link failure is temporarily filtered out of the eligible set, which is what separates this from oblivious spraying.[3] The published consequence is capacity-proportional degradation — 11 percent bandwidth loss for 10 percent link failures, and 1.2 times higher bandwidth during failures.[3]
Diagram source (Mermaid)
flowchart TB G[GPU] --> N[ConnectX-8 SuperNIC] N -->|port 1| P1[Plane 1 leaf] N -->|port 2| P2[Plane 2 leaf] P1 --> S1[Plane 1 spine] P2 --> S2[Plane 2 spine] S1 --> R1[Remote plane 1 leaf] S2 --> R2[Remote plane 2 leaf] R1 --> N2[Remote SuperNIC] R2 --> N2 N2 --> G2[Remote GPU] P2 -. congestion or link failure .-> F[Plane filtered out of the eligible set]
2Read the matrix before you promise anything
The Network Operator page carries the support matrix, and it is short enough to memorise.[1]
| Architecture | NICs | GPU platforms | Multiplane mode | Status |
|---|---|---|---|---|
| Single-Plane | BlueField-3 SuperNIC, ConnectX-7 NIC, ConnectX-8 SuperNIC | H100/H200/B200, GB200 | none (1 plane) |
GA |
| Dual-Plane | ConnectX-8 SuperNIC | B300, GB300 | swplb (2 planes), hwplb (2 planes) |
swplb GA, hwplb tech preview |
| Quad-Plane | ConnectX-8 SuperNIC | B300 | swplb (4 planes), hwplb (4 planes) |
swplb GA, hwplb tech preview |
Two readings matter commercially. First, multiplane is a ConnectX-8-only capability in this matrix; BlueField-3 SuperNIC and ConnectX-7 appear only in the single-plane row, so a customer’s existing BlueField-3 fleet cannot be configured into planes.[1] Second, swplb is the software plane load balancer and is GA, while hwplb is the hardware plane load balancer and is tech preview; both are set through the multiplane mode field on NicConfigurationTemplate.[1]
Versions are the other half of the answer, and here two current NVIDIA pages disagree. As fetched on 2026-09-07, the Network Operator page maps Spectrum-X RA 2.1 to Network Operator 26.1.0 and RA 2.2 to 26.4.0, and stops there.[1] The Spectrum-X validated solution stack, fetched the same day, lists Network Operator 26.7.0 in its latest row, v2.3.1 of September 2026, alongside Cumulus Linux 5.18.1, NetQ 5.1.0, DOCA-Host 3.5.0-082, ConnectX-8 firmware 40.50.1002 and BlueField-3 firmware 32.50.1002.[2] Neither page is wrong; they are indexed differently. Quote both, with their dates, and let the customer’s actual RA row decide.[1][2]
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
3The Kubernetes objects a customer will hand you
Five custom resources define a Spectrum-X deployment on the Kubernetes side.[1] NicClusterPolicy is cluster-wide and enables the Spectrum-X Operator, the SR-IOV Network Operator, the NIC Configuration Operator, NV-IPAM and Multus. NicConfigurationTemplate carries NIC firmware and PF configuration: link type, VF count, multiplane mode and the RA version. NicInterfaceNameTemplate drives rail- and plane-based netdev naming through udev rules. SpectrumXRailPoolConfig describes the rail topology, PF selection, IPAM binding and resource exposure. CIDRPool is the NV-IPAM object that allocates addresses per rail, or per rail and plane.[1]
The detail worth pausing on is that NicConfigurationTemplate carries NIC firmware and PF configuration — link type, numVfs, multiplane mode — and the Spectrum-X RA version in the spectrumXOptimized.version field; Network Operator 26.4.0 accepts the values RA1.3, RA2.0, RA2.1 and RA2.2.[1] The reference architecture number is therefore machine-readable configuration inside the cluster, not just a line in a document — which means a version mismatch between the cluster manifest and the deployed firmware is discoverable with kubectl, not only by reading release notes.
The page ships three separate quick-start walkthroughs — single-plane, swplb, and hwplb as tech preview — so a customer asking “how do I deploy this” should be pointed at the one that matches their adapter row rather than a generic guide.[1] The NCP-AIN blueprint expects deployment and verification of the Network Operator to manage RDMA interfaces in Kubernetes clusters, so this is exam surface as well as field surface.[11]
4The switch side of a plane
Cumulus Linux 5.18 added the BGP hook for planes: nv set vrf <vrf-id> router bgp plane-id.[7] The same release lists EVPN unreachability in disjoined planes as generally available, and EVPN multihoming without a unique SVI IP address — both multiplane- and multi-tenant-relevant.[8] Two more 5.18 items belong in any tenancy conversation because they change how tenants are carved up and both are Beta: layer 3 VXLAN device mode and profile-based switch configuration.[8]
Monitoring caught up in the same window. NetQ 5.1 adds multiplane monitoring with filtering capabilities, plane-specific validations, and topology integration using system labels configured through NVUE commands.[9] That labelling is the bridge between the plane concept and the operational view: without the labels, a plane is just a set of unrelated switches on a dashboard.
Tenant isolation itself, on the switch, is BGP-EVPN — and it is an explicit NCP-AIN objective to configure multi-tenancy BGP-EVPN to isolate tenant workloads.[11] The reference-design framing NVIDIA uses with enterprise and government customers pairs Spectrum-4 switches and BlueField-3 SuperNICs with telemetry-driven congestion control, adaptive routing, hardware-enforced network segmentation, and the Network Operator plus NetQ for management and visibility.[12]
5Isolation, and the rule that complicates it
NVIDIA’s plain statement is that performance isolation comes from adaptive routing and congestion control together: AR finely balances data paths so one tenant’s flows do not collide with another’s between leaf and spine, and CC’s metering plus telemetry prevents victim flows from forming.[10][3] The formulation on the platform blog is that combining high-frequency telemetry probes with flow metering ensures that workloads are protected and the fabric delivers performance isolation.[10] The quantified claim is the DeepSeek-V3 step time: 668 ms under near-perfect isolation versus 1.18 s, a 1.6 times degradation, on traditional Ethernet.[3]
Now the tension. Adaptive routing is documented as applying to layer-3 interfaces and next-hop router interfaces in the default VRF.[6] Tenant isolation on a Cumulus fabric is normally a VRF per tenant. Under the 5.18 statement, routed traffic inside a tenant VRF is therefore not adaptive-routing eligible, which puts the two halves of the isolation story in different scopes. This is not a contradiction you can resolve from the public pages; it is a question to take to the Spectrum-X deployment guide for the customer’s specific RA release, and to say so out loud rather than assume.[6]
A Dell customer with a B300 pod and ConnectX-8 SuperNICs asks for a dual-plane design with four tenants, each isolated, “with adaptive routing everywhere”.
- Adapter gate. ConnectX-8 supports dual-plane; the matrix lists
swplbGA andhwplbtech preview, so the design isswplbwithhwplbas a lab item.[1] - Version row. Pick the track (B300) and read the whole RA row rather than a single number; v2.3.1 is Cumulus 5.18.1 with ConnectX-8 firmware 40.50.1002 and DOCA-Host 3.5.0-082, and the Operator value to quote is 26.7.0 from the validated stack, noting the Network Operator page still tops out at RA 2.2 to 26.4.0.[2][1]
- Kubernetes objects.
NicConfigurationTemplatewithmultiplane: swplb, the VF count and the RA version field;SpectrumXRailPoolConfigbinding rails to aCIDRPool;NicInterfaceNameTemplatefor plane-based naming; all under aNicClusterPolicy.[1] - Switch side. Plane identity through
nv set vrf <vrf-id> router bgp plane-id, tenant separation through BGP-EVPN, and NetQ 5.1 multiplane monitoring with NVUE system labels so the planes are visible as planes.[7][11][9] - The caveat, stated before they find it. Adaptive routing is documented for next-hop router interfaces in the default VRF, so “adaptive routing everywhere, including inside each tenant VRF” is not something the 5.18 page supports; confirm the intended behaviour against the Spectrum-X deployment guide for the RA row before committing.[6]
- What isolation actually buys them, in NVIDIA’s numbers: the DeepSeek-V3 668 ms versus 1.18 s comparison, and capacity-proportional degradation of 11 percent for 10 percent link failures.[3]
The same ask, on a customer with BlueField-3 SuperNICs on an H200 pod:
- Adapter gate: BlueField-3 appears only in the ____ row, so the answer to dual-plane is ____.[1]
- Version row: the ____ track is the BlueField-3 track; v2.3.1 pins Cumulus ____ and BlueField-3 firmware ____.[2]
- Kubernetes:
NicConfigurationTemplatewith multiplane mode ____ and the ____ version field.[1] - Switch side: tenant separation through ____; plane identity is not required because ____.[11]
- Caveat to state anyway: adaptive routing applies to next-hop router interfaces in the ____ VRF.[6]
- What you would need to change to make multiplane possible for this customer: ____.[1]
Write the one-paragraph steer for a B300 or GB300 customer who has read about hwplb and wants it in production this quarter, plus a second paragraph for their architect who insists that per-tenant VRFs and adaptive routing can both be fully enabled.
Acceptance: the first paragraph names the GA status of each mode and offers a concrete alternative with a trial path; the second states the default-VRF rule verbatim in substance, names the document that would settle it for their RA, and does not invent a workaround. Both paragraphs cite the Network Operator page, the validated stack and the Cumulus 5.18 adaptive routing page, and neither asserts a version number without saying which page and date it came from.[1][2][6]
Case closed — three lines and two links
Plane count is decided by the adapter, so quad-plane is a refresh conversation for their BlueField-3 fleet and not a configuration change[1]. If pod two ships with ConnectX-8, swplb is generally available and hwplb is a tech preview to trial in a lab rather than run in production[1]. The version answer is both pages, with their dates, and their own RA row decides[2]. The director signs. The sentence he keeps: “Your software is not blocking planes. Your adapters are.” That night, in the data hall, the operator peels PLANE A and PLANE B off two uplinks he labelled the week of the first demo, and puts the tape back in the drawer for the pod that will earn it.
Lab
Goal: use the Dell lab to prove the hardware gate — that the adapter, not the software, decides the plane count. Every step reads state.
Pre-flight inventory. sudo mst start; sudo mst status -v for device paths; lspci -d 15b3: -nn for every NVIDIA adapter present; ip -br link for the netdevs.
sudo flint -d /dev/mst/mt41692_pciconf0 q— expected: part number and PSID identifying a BlueField-3 adapter, plus the firmware version. Record the exact part number string; it is what you will match against the support matrix.[1]sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep -iE 'plane|pf_num|num_of_vfs|link_type'— expected: link type and VF-related fields. Note whether any plane-related field exists at all on this adapter and firmware; record the answer either way.- Open the support matrix and locate your adapter. Expected: BlueField-3 SuperNIC appears in the single-plane row only, so no multiplane mode is applicable to this card regardless of the switches or the Network Operator version.[1]
- Write the two-sentence customer answer: the adapter you have is single-plane in the support matrix, and dual- or quad-plane would require ConnectX-8 SuperNICs on a B300 or GB300 platform.[1]
- Compare your card’s firmware with the H200 track of the validated stack row you intend to run, and note whether the lab card is on, ahead of or behind that row.[2]
- Optional, customer lab with ConnectX-8 hosts: read the multiplane mode actually configured on the NIC through
mlxconfig ... q, and on the attached leaves read the plane configuration withnv show vrf <vrf-id> router bgpfor the plane identifier added in 5.18.[7] Read only; make no changes on customer equipment. - No rollback is required. If you ran
mlxconfigwith-y sat any point you left the read-only path — restore the field and power cycle before leaving the lab.
Goal: author the two Kubernetes objects a customer would hand you, then write the steer for a tech-preview request.
- Write a
NicConfigurationTemplatecontaining, at minimum: the multiplane modeswplb, a VF count, the link type, and the RA version field. Expected: those fields are the ones the Network Operator page names for that CRD.[1] If you cannot name the field that carries the RA version, re-read the CRD list before continuing. - Write a
SpectrumXRailPoolConfigthat describes the rail topology, PF selection and IPAM binding, plus theCIDRPoolit binds to for per-rail or per-rail-and-plane allocation.[1] - Validate the syntax if a cluster is available:
kubectl apply --dry-run=client -f nicconfigurationtemplate.yamlagainst a local kind cluster. Expected: the objects parse. If no cluster is available, review field by field against the Network Operator page and mark any field you invented — inventing fields is the failure mode this step is designed to catch.[1] - Build the adapter-to-architecture table from memory, then check it: which adapters appear in single-plane, which in dual- and quad-plane, and the GA status of each mode.[1]
- Write the one-paragraph steer for a B300 customer asking for
hwplbtoday. Expected content:swplbis GA for both dual- and quad-plane,hwplbis tech preview for both, and the trial belongs in a lab with the customer’s own RA row.[1] - Write two sentences on the Network Operator version disagreement, quoting both pages with the date you checked them.[1][2]
- Finish with the isolation paragraph: where isolation comes from, the published DeepSeek-V3 numbers, and the default-VRF caveat for adaptive routing.[10][3][6]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell account SE, in five sentences, why a customer's choice of ConnectX-8 versus BlueField-3 decides whether multiplane is on the table at all.
Sources
Facts in this lesson were checked against Network Operator 26.4.0 Spectrum-X page and Spectrum-X validated stack v2.3.1 both re-fetched 2026-09-07; Cumulus Linux 5.18 new-commands page re-fetched 2026-09-07. Dates are when each page was fetched.
- NVIDIA Spectrum-X Ethernet Networking Platform | Network Operator 26.4.0 · fetched 2026-09-07
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- Giga-Scale AI and the Ethernet Evolution: How Spectrum-X Ethernet Rewrites the Rules (NVIDIA Technical Blog) · fetched 2026-09-07
- Dell PowerSwitch SN6000 Ethernet Switch Series spec sheet · fetched 2026-09-07
- NVIDIA Spectrum-X Ethernet Platform · fetched 2026-09-07
- Equal Cost Multipath Load Sharing (includes Adaptive Routing) | Cumulus Linux 5.18 · fetched 2026-09-07
- New and Removed NVUE Commands | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus NetQ 5.1 · fetched 2026-09-07
- Optimize Large-Scale AI Workloads with NVIDIA Spectrum-X (NVIDIA Technical Blog) · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) certification page · fetched 2026-09-07
- Networking Infrastructure | AI Factory Reference Design (NVIDIA AI Enterprise) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.