Spectrum-X rails inside Kubernetes

1What the Spectrum-X Operator adds on top of everything so far

By this point in the course a pod can already get a second interface on an SR-IOV VF, with RDMA, scheduled by an extended resource. Spectrum-X is the layer above that: it “combines Spectrum switches with BlueField-3 SuperNIC, ConnectX-7, and ConnectX-8 NICs to deliver high-bandwidth, lossless RoCE for the GPU-to-GPU compute (east-west) network”, and the operator’s job inside Kubernetes is “discovering the NICs, configuring rails, and exposing them to pods as schedulable resources”.^[1] Nothing new appears in the Kubernetes API: a rail is still an extended resource, and the test pod still requests nvidia.com/rail0: "1" and nvidia.com/rail1: "1" with the annotation k8s.v1.cni.cncf.io/networks: rail0,rail1.^[2]

Three moving parts carry the whole feature. A flow-controller DaemonSet programs per-rail OVS flows and owns the SpectrumXRailPoolConfig CRD; it is “essential for any Spectrum-X mode (single-plane, swplb, hwplb)”.^[3] A NIC tuning profile ships as a versioned ConfigMap and is applied through spectrumXOptimized in a NicConfigurationTemplate.^[1] And a VRF meta-plugin optionally isolates each rail in its own routing table.^[3]

The hardware pairing is worth stating out loud in a Dell conversation, because it is the same platform matrix as the rest of the operator: ConnectX-6 through ConnectX-9, BlueField-3 and BlueField-3 SuperNIC in NIC mode only, up to 800 Gb/s.^[6] A BlueField in DPU mode is not a Spectrum-X rail NIC as far as this operator is concerned.

2multiplaneMode, and the xPlane rule you can read off the rail topology

multiplaneMode takes three values. none is single-plane and covers BlueField-3 and ConnectX-7/ConnectX-8 on H100, H200 and B200, plus GB200. hwplb is Hardware Multiplane on ConnectX-8 with B300/GB300 and is documented as “recommended mode”. swplb is Software Multiplane on the same hardware.^[1] Hardware Multiplane pulls in a component: “DOCA xPlane is a required component.”^[1] Hardware Multiplane with DOCA xPlane went GA on ConnectX-8 in 26.7.0.^[5]

You do not have to remember which mode deploys xPlane, because the rail topology tells you. xPlane is a DaemonSet deployed “per rail topology entry that selects more than one PF”.^[3] That gives a table you can reconstruct from first principles:

For hardware multiplane the operator waits for the selected PFs to enter switchdev mode before it creates the br-xplane bridge.^[3] That wait is why a hwplb cluster can sit for minutes with nothing apparently happening and still be healthy.

The version story is a straight mapping and an FAE should have it memorised: RA 2.3 with operator 26.7.x (GA), RA 2.1 with 26.4.x (Tech Preview), RA 2.1 with 26.1.x (GA).^[1] When a customer names an RA version, you can name the operator version, and vice versa.

3The 26.7.0 profile ConfigMap: a name that must match character for character

Up to 26.4, spectrumXOptimized.version selected a built-in Reference Architecture. From 26.7.0 the NIC tuning ships as a versioned Spectrum-X profile ConfigMap, and spectrumXOptimized.version names that ConfigMap.^[1][5] In the quick start the block reads {enabled: true, version: "spectrum-x-ra2.3-profile", overlay: "none", multiplaneMode: "none", numberOfPlanes: 1} inside a NicConfigurationTemplate.^[2]

The verification page is blunt about the matching rule. List the candidates with kubectl get configmap -A -l network.nvidia.com/operator.nic-configuration.spectrum-x-profile; “the label value is ignored — only the key must be present”, and the ConfigMap name is what spectrumXOptimized.version must match, character for character.^[4] Confirm the daemon actually loaded it with kubectl -n nvidia-network-operator logs -l app.kubernetes.io/name=nic-configuration-daemon | grep "Loaded Spectrum-X profile".^[4]

Three of the documented failure rows come straight out of this one field:^[4]

The rest of the NIC-side tuning is the ordinary NicConfigurationTemplate you already know, where pciPerformanceOptimized, roceOptimized and gpuDirectOptimized live — and where pciPerformanceOptimized.maxAccOutRead is now documented as “Deprecated, ignored”.^[7][5]

4SpectrumXRailPoolConfig, the VRF rule and the MTU that must agree three times

The rail pool is one CR. In spectrumx.nvidia.com/v1alpha2, the quick start sets draEnabled: false, networkNamespace: default, numVfs: 1, and a railTopology list whose entries look like {name: rail0, nicSelector: {pfNames: ["net_rail0"]}, cidrPoolRef: rail-0, mtu: 9216}.^[2] The pfNames values are not accidents: the Spectrum-X flow adds a NicInterfaceNameTemplate with netDevicePrefix: "net_rail%rail_id%" and rdmaDevicePrefix: "rdma_rail%rail_id%", so the interface names are manufactured to be selectable per rail.^[2]

The IPAM side is where the VRF rule bites. The VRF meta-plugin “isolates each rail in its own VRF, named after the rail topology entry”, and VRFs apply only to rails whose cidrPoolRef resolves to an IPv6 CIDRPool; rails using inline railTopology[].ipam do not get VRF isolation at all.^[3] IPv6 rail isolation via the VRF CNI meta-plugin is itself new in 26.7.0.^[5] Each rail also gets the RDMA CNI with rdmaQoS: {tos: 96, tc: 96}, and the pod’s RDMA device is named rdma_<rail topology name>.^[3]

Rail MTU in NVIDIA’s own quick start is 9216.^[2] That number only helps if the whole path agrees with it. Calico’s rule for the primary network is “set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU)”, with per-encapsulation subtractions of 20 for IP-in-IP, 50 for VXLAN IPv4 and 70 for VXLAN IPv6.^[8] On a secondary interface the constraint is different in kind: a macvlan child’s mtu is capped at the master’s MTU, so you can never raise a pod above its host interface.^[10] Rail traffic bypasses the primary CNI entirely, but the primary CNI still carries the NCCL bootstrap, so both budgets have to be right or you get a job that starts and then crawls.

kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "9166"}}'
kubectl rollout restart daemonset/calico-node -n kube-system   # the doc says: restart the DaemonSet
ip link show   # the tunnel device MTU must read 9166
kubectl exec -it <pod> -- ip link   # eth0 must read 9166, net1 must read 9216
# SpectrumXRailPoolConfig:  railTopology: [{name: rail0, nicSelector: {pfNames: ["net_rail0"]}, cidrPoolRef: rail-0, mtu: 9216}]
kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> -o jsonpath='{.status.syncStatus}'

5Verification, and the two failures you manufacture yourself

The operator gives you an ordered set of checks, and the order matters because each one depends on the previous.^[4]

# 1. did the profile load on this node?
kubectl -n nvidia-network-operator logs -l app.kubernetes.io/name=nic-configuration-daemon \
  | grep "Loaded Spectrum-X profile"

# 2. what does the NIC think happened?
kubectl -n nvidia-network-operator get nicdevices
kubectl -n nvidia-network-operator get nicdevice <name> -o jsonpath='{.status.conditions}' | jq

# 3. did the rail pool sync?
kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.syncStatus}'
kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.nodeStates}' | jq

# 4. does a pod see the rail as an RDMA device?
kubectl exec -it spectrum-x-test -- rdma link
kubectl exec -it spectrum-x-test -- ip -6 route show vrf rail0

# 5. traffic, inside the VRF
ip vrf exec rail0 ib_write_bw -R ...

syncStatus takes Unknown, InProgress, Failed or Succeeded, with per-node detail in .status.nodeStates.^[4] The NIC device condition reasons are the vocabulary for step 2: UpdateSuccessful is the target; UpdateStarted, PendingReboot, PendingNodeMaintenance and PendingFirmwareUpdate are in-progress states; PartiallyApplied means some parameters were unsupported and the message lists which; IncorrectSpec and SpecValidationFailed mean the template is invalid for that device; NonVolatileConfigUpdateFailed, RuntimeConfigUpdateFailed and FirmwareError mean the apply itself failed.^[4]

Two failures on this page are ones the operator did not cause. The first is ip vrf exec: “RDMA-CM is not VRF-aware. Traffic tests that use RDMA-CM (for example ib_write_bw -R) must be run inside the VRF context.”^[4] Forget it and you get a benchmark that connects to nothing on a fabric that is working. The second is capabilities: the quick start’s spectrum-x-test pod carries IPC_LOCK and NET_RAW, while the verify page says workloads that need ip vrf exec require IPC_LOCK, NET_ADMIN, DAC_OVERRIDE and BPF.^[2][4] These are not contradictory pages, they are two different pods: start from the narrower set for a plain RDMA workload and widen only when the workload enters a VRF itself.

Two more rows close the table. Configuration that flaps means the SR-IOV Operator’s mellanox plugin is still enabled and fighting the NIC Configuration Operator — set disablePlugins: [mellanox].^[4][2] A missing OVS bridge means the manageSoftwareBridges feature gate was not set on the SR-IOV Operator.^[4][2] And OVS uplinks missing after a reboot is documented as “expected on Hardware Multiplane” with the evidence in /var/log/xplane-ovs-pre.log.^[4]

Once the plumbing verifies, NCCL is the measurement layer. The rail CNI’s rdmaQoS sets tos: 96 and tc: 96;^[3] NCCL exposes the same lever as NCCL_IB_TC, default 0, and NCCL_IB_GID_INDEX, default -1.^[9] Teach these as one setting seen from two sides: a runbook that hardcodes a traffic class in the job while the CNI sets another one is a contradiction that no single page will report.