Skip to content
nodeza.learn

Kubernetes networking for AI clusters

How an AI cluster actually wires GPUs together: the primary CNI (Calico/Cilium) that owns eth0 and Services, and the separate fast path — Multus, SR-IOV VFs, host-device, IPoIB, RDMA CNI — that carries NCCL traffic and never touches the primary CNI. Builds from the Kubernetes network model up through the NVIDIA Network Operator (v26.7.0), NIC Configuration Operator firmware templates, GPUDirect RDMA and NUMA alignment, to Spectrum-X rails, DPF, schedulers, triage and Dell AI Factory scenarios. Interactive diagrams, two-variant labs (containerlab/kind with no NVIDIA hardware, or the Dell-lab BlueField-3 and ConnectX hosts), checkpoints.

4 modules. Dots: grey not started, blue in progress, green complete with quiz ≥ 80%, amber complete but below 80%. Each module ends in a checkpoint that unlocks the next.

Kubernetes networking for AI clusters

0%

0 of 20 cases closed

Foundations

Fast path

Operators

AI factory & FAE

Two-layer networking: primary CNI and the fast path

Explain to a customer why an AI cluster has a primary CNI plus a separate second interface, and build the second path's software pieces (Multus, macvlan/ipvlan/host-device, cluster-wide IPAM, MTU) on hardware you already have.

  1. 1.1 The Kubernetes network model and where a second CNI fitsEight nodes, thirty days, one interface
  2. 1.2 Primary CNI choices: Calico and CiliumThe kernel that said no
  3. 1.3 MTU budgeting across switch, host PF and podThree numbers and one screenshot
  4. 1.4 Multus and the NetworkAttachmentDefinitionThe perfect manifest
  5. 1.5 macvlan, ipvlan, host-device and cluster-wide IPAMWhose ticket it actually was
  6. Checkpoint

The fast path: device plugins, SR-IOV and RDMA in pods

Make an RDMA-capable interface appear inside a pod, choose correctly between the shared and exclusive designs, and name every place the path can fail.

  1. 2.1 How kubelet learns a NIC existsThe pod that never left Pending
  2. 2.2 SR-IOV plumbing: device plugin, CNI and operatorThe NAD that set a priority and no VLAN
  3. 2.3 RDMA in pods: shared plugin vs exclusive netnsThe node that was hardened at 02:10
  4. 2.4 RoCE, GID tables and the NCCL knobs that read themThe slow fabric that was never on the fabric
  5. 2.5 vfio-pci for VMs, and the DRA SR-IOV driverThe appliance that owned the card
  6. Checkpoint

Operators: Network Operator, NIC firmware, GPUDirect and NUMA

Deploy and version the NVIDIA operator stack correctly, push RoCE and GPUDirect settings into NIC firmware safely, and align NIC to GPU so a training job runs at full speed.

  1. 3.1 Network Operator and the NicClusterPolicyThe policy that was applied and never became anything
  2. 3.2 Version matrices, upgrades and heterogeneous clustersThe bookmark that says 1.31 is fine
  3. 3.3 NIC firmware config in Kubernetes: RoCE, PFC and rebootsThe change request that says no outage expected
  4. 3.4 GPUDirect RDMA: dma-buf, nvidia-peermem and GDSlsmod says nothing, so GPUDirect must be broken
  5. 3.5 NUMA alignment with Topology Manager and CPU ManagerSix nodes are slower and nobody can say which
  6. Checkpoint

AI factory scale: Spectrum-X, DPF, schedulers, triage and Dell scenarios

Reason about a full AI-factory network you cannot build at home, run the triage ladder in order, and answer the Dell OEM and NCA-AIIO questions an FAE actually gets asked.

  1. 4.1 Spectrum-X rails inside KubernetesThe rail that verified clean and moved nothing
  2. 4.2 BlueField in Kubernetes: DPF versus Network OperatorTwo cards on the loading dock
  3. 4.3 Schedulers on top: MPI Operator, Run:ai and Slurm on KubernetesThe scheduler argument nobody needed to have
  4. 4.4 The triage ladder: sos-report, error strings and the checklists02:10, and the restart that deletes the answer
  5. 4.5 Dell AI Factory scenarios and the NCA-AIIO positioningThe numbers you are allowed to say out loud
  6. Checkpoint