Kubernetes networking for AI clusters
0 of 20 cases closed
Foundations
Fast path
Operators
AI factory & FAE
Kubernetes networking for AI clusters
Multus, SR-IOV, RDMA in pods, the NVIDIA Network Operator, and the fabric under a Dell AI Factory.
How an AI cluster actually wires GPUs together: the primary CNI (Calico/Cilium) that owns eth0 and Services, and the separate fast path — Multus, SR-IOV VFs, host-device, IPoIB, RDMA CNI — that carries NCCL traffic and never touches the primary CNI. Builds from the Kubernetes network model up through the NVIDIA Network Operator (v26.7.0), NIC Configuration Operator firmware templates, GPUDirect RDMA and NUMA alignment, to Spectrum-X rails, DPF, schedulers, triage and Dell AI Factory scenarios. Interactive diagrams, two-variant labs (containerlab/kind with no NVIDIA hardware, or the Dell-lab BlueField-3 and ConnectX hosts), checkpoints.
Who it is for: Platform engineers wiring RDMA and GPUDirect into Kubernetes AI clusters.
- Not started
- In progress
- Complete, case open
- Case closed
- Locked
- Checkpoint
- A case closes when the lesson’s retrieval check reaches 80%.
Explain to a customer why an AI cluster has a primary CNI plus a separate second interface, and build the second path's software pieces (Multus, macvlan/ipvlan/host-device, cluster-wide IPAM, MTU) on hardware you already have.
- 1.1The Kubernetes network model and where a second CNI fitsEight nodes, thirty days, one interfaceUnderstand20 min
- 1.2Primary CNI choices: Calico and CiliumThe kernel that said noAnalyze30 min
- 1.3MTU budgeting across switch, host PF and podThree numbers and one screenshotApply25 min
- 1.4Multus and the NetworkAttachmentDefinitionThe perfect manifestApply25 min
- 1.5macvlan, ipvlan, host-device and cluster-wide IPAMWhose ticket it actually wasAnalyze30 min
- Module 1 checkpoint12 questions · pass mark 80% · 12 min
Make an RDMA-capable interface appear inside a pod, choose correctly between the shared and exclusive designs, and name every place the path can fail.
- 2.1How kubelet learns a NIC existsThe pod that never left PendingUnderstand20 min
- 2.2SR-IOV plumbing: device plugin, CNI and operatorThe NAD that set a priority and no VLANApply30 min
- 2.3RDMA in pods: shared plugin vs exclusive netnsThe node that was hardened at 02:10Analyze30 min
- 2.4RoCE, GID tables and the NCCL knobs that read themThe slow fabric that was never on the fabricAnalyze30 min
- 2.5vfio-pci for VMs, and the DRA SR-IOV driverThe appliance that owned the cardAnalyze25 min
- Module 2 checkpoint14 questions · pass mark 80% · 12 min
Deploy and version the NVIDIA operator stack correctly, push RoCE and GPUDirect settings into NIC firmware safely, and align NIC to GPU so a training job runs at full speed.
- 3.1Network Operator and the NicClusterPolicyThe policy that was applied and never became anythingApply30 min
- 3.2Version matrices, upgrades and heterogeneous clustersThe bookmark that says 1.31 is fineEvaluate30 min
- 3.3NIC firmware config in Kubernetes: RoCE, PFC and rebootsThe change request that says no outage expectedAnalyze30 min
- 3.4GPUDirect RDMA: dma-buf, nvidia-peermem and GDSlsmod says nothing, so GPUDirect must be brokenAnalyze25 min
- 3.5NUMA alignment with Topology Manager and CPU ManagerSix nodes are slower and nobody can say whichEvaluate30 min
- Module 3 checkpoint13 questions · pass mark 80% · 12 min
Reason about a full AI-factory network you cannot build at home, run the triage ladder in order, and answer the Dell OEM and NCA-AIIO questions an FAE actually gets asked.
- 4.1Spectrum-X rails inside KubernetesThe rail that verified clean and moved nothingAnalyze30 min
- 4.2BlueField in Kubernetes: DPF versus Network OperatorTwo cards on the loading dockAnalyze25 min
- 4.3Schedulers on top: MPI Operator, Run:ai and Slurm on KubernetesThe scheduler argument nobody needed to haveEvaluate25 min
- 4.4The triage ladder: sos-report, error strings and the checklists02:10, and the restart that deletes the answerEvaluate30 min
- 4.5Dell AI Factory scenarios and the NCA-AIIO positioningThe numbers you are allowed to say out loudCreate35 min
- Module 4 checkpoint14 questions · pass mark 80% · 12 min