Skip to content
nodeza.learn
K8sNet course/4. AI factory & FAE/Lesson 4.3

Schedulers on top: MPI Operator, Run:ai and Slurm on Kubernetes

S4·E3The scheduler argument nobody needed to have · The customer's design review, a whiteboard already full, eight days before acceptance

S4·E3Evaluate~25 minsources checked todaylab mutates hardwareverified against kubeflow/mpi-operator README (v0.8.2, v2beta1) re-fetched 2026-09-07; NVIDIA Run:ai self-hosted 2.24 cluster system requirements re-fetched 2026-09-07; GPU Operator platform support re-fetched 2026-09-07

Builds on: How kubelet learns a NIC exists, Version matrices, upgrades and heterogeneous clusters

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Reframe a 'which scheduler' question into 'does the scheduler know about the second NIC' and justify the reframing.
  • Read an MPIJob and predict whether its workers will land on nodes that actually have the fabric.
  • Judge a proposed Run:ai deployment against the published version matrices and name the contradiction rather than resolving it.
  • Decide what you can and cannot state about Slinky and Kubeflow Trainer from the pages that were actually fetched.

Episode 3 — The scheduler argument nobody needed to have

The situation · The customer's design review, a whiteboard already full, eight days before acceptance

Three products are on the board and everybody has a favourite: Run:ai because it is NVIDIA’s, the MPI Operator because it is small, Slurm because half the team came from HPC. The Dell SE is refilling a coffee he will not drink and taking notes for row 22. The network lead says the sentence he says once per project: show me the counter.

He is right, and the counter is on the node rather than on the switch. Device plugins exist so a vendor can advertise hardware to kubelet without modifying Kubernetes core code, using names like nvidia.com/gpu, and what they advertise becomes an extended resource — integer-only, non-overcommittable, not shareable between containers.[5] That is the whole scheduling contract, and all three products place work by counting those names.[5][8] A rail that nobody advertises is invisible to every one of them: the job is admitted, the container starts, and the collective library quietly falls back to whatever the primary CNI gave it. No error is raised anywhere, which is why the complaint arrived three weeks late and worded as “training is slow”.

Do not choose a scheduler until you know the node has something to schedule on.

kubectl describe node | grep -A20 Allocatable settles it in one command. Then, and only then, does the product conversation start — and it starts with two version matrices that contradict each other.[2][3]

1The reframe: extended resources are the only thing a scheduler can see

Customers ask “which scheduler should we run” and expect a product recommendation. The useful answer starts one level lower. Kubernetes device plugins exist so vendors can advertise hardware to kubelet “without modifying Kubernetes core code”, using the naming convention vendor-domain/resourcetype such as nvidia.com/gpu.[5] Those advertised names become extended resources, which are “integer-only and not overcommittable”, and the devices “cannot be shared between containers”.[5]

That is the entire scheduling contract. The MPI Operator, Run:ai and Slurm-on-Kubernetes all place work by counting extended resources, so a rail NIC that is not advertised as nvidia.com/rail0 or rdma/rdma_shared_device_a is invisible to every one of them.[5][8] The job lands somewhere, starts, and runs over whatever the primary CNI gives it.

Two consequences worth saying out loud to a customer. First, a fabric problem can present as a scheduler problem and vice versa, and the discriminator is one command: kubectl describe node | grep -A20 Allocatable. Second, because extended resources are integer and non-shareable, “two pods sharing one VF” is not a thing — either they used the shared RDMA device plugin, which deliberately over-advertises one HCA as a scheduling counter, or they are looking at two different resource names.[5]

◐ Level 2 — limited (same annual cycle)
  • Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
  • This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
  • ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).

Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗

⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.

Use the stack checker as the shape of the artifact you will build in the lab: one row per component, one column per published constraint, contradictions marked rather than resolved.

2MPI Operator: the smallest thing that shows the problem

The Kubeflow MPI Operator “makes it easy to run allreduce-style distributed training on Kubernetes”, currently at release v0.8.2 with API kubeflow.org/v2beta1.[1] An MPIJob carries runPolicy.cleanPodPolicy, mpiReplicaSpecs holding Launcher and Worker pod templates, and slotsPerWorker; status carries conditions, replicaStatuses and timing metadata.[1] The example launcher runs mpirun --allow-run-as-root -np 2 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib.[1]

Two things about that command line are teaching material. -bind-to none -map-by slot is the standard shape for GPU jobs where the process should not be pinned by MPI because the GPU and NIC affinity is decided elsewhere. And -x NCCL_DEBUG=INFO is already in NVIDIA’s own example, which is convenient: NCCL_DEBUG prints warnings and debug information, and NCCL_IB_DISABLE=1 is the fastest A/B test for whether RDMA was ever engaged.[9]

The gap matters as much as the content. The README documents no SSH configuration, no hostNetwork, no IPC_LOCK and no RDMA specifics.[1] So when you build an RDMA-capable MPIJob you are borrowing from the Network Operator’s pod examples, where every RDMA pod adds securityContext.capabilities.add: ["IPC_LOCK"] and requests the RDMA resource in the same container as the GPU.[7] Say that provenance out loud rather than implying the MPI Operator documents it.

3Run:ai: the version matrix you quote rather than resolve

Run:ai’s cluster requirements are published and specific. The latest cluster release supports Kubernetes 1.33 to 1.35 and OpenShift 4.17 to 4.20; Multi-Node NVLink on GB200 needs Kubernetes 1.32 or later.[2] Required operators: GPU Operator versions 25.3 to 25.10, Network Operator v24.4 and above for clusters with RDMA or Multi-Node NVLink-capable nodes such as B200 and GB200, and DRA driver 25.3 to 25.8 for MNNVL.[2] Run:ai’s RDMA requirement is stated as Network Operator with SR-IOV InfiniBand support.[2]

Now put that next to the GPU Operator’s own matrix: 26.7.x supported, 26.3.x deprecated, 25.10.x and lower end of support, on Kubernetes 1.33 to 1.37.[3] A customer who follows Run:ai’s matrix faithfully lands on a GPU Operator release that its own vendor lists as end of support. Do not pick a side in that. Quote both numbers, name the collision, and let the customer’s support posture decide. This is also a live example rather than a permanent fact — re-check it before teaching it, because the whole point is that it is true today.

Two more Run:ai constraints bite in real deployments. Pod Security Admission: “NVIDIA Run:ai supports restricted policy for Pod Security Admission (PSA) on OpenShift only. Other Kubernetes distributions are only supported with privileged policy.”[2] That agrees with the Network Operator, whose prerequisites include Pod Security Admission enforcement set to privileged for its namespace.[7] A hardened customer running restricted everywhere on vanilla Kubernetes will see the driver DaemonSet fail to schedule with no networking symptom at all. Ingress and DNS: ingress controllers and FQDNs are conditionally required — unnecessary when control plane and cluster deploy together, mandatory when they deploy separately — and external inference access needs a wildcard DNS record such as *.runai-inference.mycorp.local resolving to the cluster’s public IP.[2]

4Slurm on Kubernetes, and the discipline of not over-claiming

“Slinky™ is SchedMD’s set of projects to enable interoperability between Slurm and Kubernetes”, and the slurm-operator lets you “Run Slurm on Kubernetes. Manage and scale Slurm clusters on Kubernetes as pods.”[4] The page references Slurm 26.05.[4]

That is the whole of what the fetched page supports. Slinky’s CRDs, its Kubernetes networking model and its version numbers live on a separate site that was not fetched, so they are unverified here and must not be described to a customer as if they were.[4] The same discipline applies to Kubeflow Trainer: TrainJob, TrainingRuntime and ClusterTrainingRuntime are commonly cited, but the kubeflow.org page is a stub and the project’s own documentation site was not fetched for this course, so this lesson does not teach them.

What you can say confidently, because it follows from the mechanism in segment 1, is that Slurm-on-Kubernetes changes none of the fabric arithmetic: the Slurm daemons run as pods, pods are placed on extended resources, and a rail that is not advertised is not schedulable regardless of which scheduler is asking.[5]

Positioning: three customers, one afternoon · decision 1/7A 0 · P 0 · S 0

Brief — Back-to-back calls: a VMware farm (200× R760), an AI training pod (64× XE9780 on Spectrum-X) and a storage-heavy multi-tenant inference cluster. Each has objections.

Dell account team + three end customers: VMware farm: "200 R760s on vSphere 8; NSX eats about 20% of our cores. Should we go DPU?"

Run the positioning scenario and watch how often the right answer is a question about what the node advertises rather than a product recommendation.
Evaluating a proposed scheduler stack

A customer sends a one-page plan: “Kubernetes 1.32, GPU Operator 25.10, Network Operator 26.7.0, Run:ai latest, MPIJobs for training, restricted PSA cluster-wide, rail NICs configured by the Spectrum-X Operator.” Evaluate it.

  1. Kubernetes version. Network Operator 26.7.0 supports 1.32 to 1.36; GPU Operator supports 1.33 to 1.37; Run:ai supports 1.33 to 1.35.[6][3][2] The three-way intersection is 1.33 to 1.35. 1.32 is outside two of the three. Verdict: change the Kubernetes version, and say which three windows produced the number.
  2. GPU Operator version. Run:ai lists 25.3 to 25.10 as supported; GPU Operator lists 25.10.x and lower as end of support.[2][3] Verdict: quote both, do not choose. Flag that 25.10 satisfies Run:ai and simultaneously sits on GPU Operator’s end-of-support list.
  3. Pod Security Admission. restricted cluster-wide on vanilla Kubernetes contradicts Run:ai, which supports restricted on OpenShift only, and contradicts the Network Operator’s privileged-namespace prerequisite.[2][7] Verdict: this is the item most likely to cause a confusing outage, because the symptom is a DaemonSet that never schedules.
  4. The fabric question. The plan says rails are configured, not that they are advertised. Ask for kubectl describe node | grep -A20 Allocatable from one worker.[5] Until nvidia.com/rail0 appears there, no scheduler in the plan can place a job on the fabric.[8]
  5. MPIJob specifics. The worker template must request the rail resource in the same container as the GPU, and must add IPC_LOCK; the MPI Operator’s own README documents neither, so cite the Network Operator pod examples as the source.[1][7]

Written verdict: three version corrections, one security-policy conflict, one piece of missing evidence, and one template requirement — none of which is a criticism of the scheduler choice.

Case notes, the counter

How it ended

Allocatable lists GPUs and nothing else: the rails were never advertised, so no scheduler could have placed on them.[5] What you say is this: advertise the NICs first, because until a node lists that resource name Run:ai, the MPI Operator and Slinky are equally blind — then build inside Kubernetes 1.33 to 1.36, since Run:ai supports 1.33 to 1.35 and Network Operator 26.7.0 supports 1.32 to 1.36.[2][6]

The change goes in that evening. At 02:10 the night-shift operator calls: a pod named rdma-shared-dp-ds is crash-looping, and the operations lead already has his hand on the keyboard.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

On the Dell-lab host with the BlueField-3 or ConnectX in NIC mode and a single-node Kubernetes (kind, k3s or RKE2) plus the Network Operator already deployed from module 3.

  1. Pre-flight inventory. Record kubectl version, helm list -A, kubectl get nicclusterpolicy nic-cluster-policy -o yaml and kubectl describe node | grep -A20 Allocatable. Save it — this is your rollback reference.
  2. Confirm what is advertised. From the Allocatable output, note the exact RDMA resource name including its prefix, for example rdma/rdma_shared_device_a. If nothing is advertised, stop here and go back to the module 3 lab; there is nothing for a scheduler to place on.
  3. Add the resource to the MPI worker template. Edit your MPIJob so the worker container requests that exact name and adds securityContext.capabilities.add: ["IPC_LOCK"], following the Network Operator pod examples rather than the MPI Operator README. Expected: workers become Running on the node that advertises the resource. Rollback: kubectl delete mpijob <name>; no node state is changed.
  4. Record the transport question. With the job running, capture which interface the launcher-to-worker traffic uses — the primary CNI interface or the secondary one — using ip -br addr inside the launcher pod and the pod’s annotations. Write the answer down; the MPI Operator does not document it, so your measurement is the evidence.
  5. Re-run with NCCL_DEBUG=INFO if the workload is a collective, and then once more with NCCL_IB_DISABLE=1. Expected: a visible difference if RDMA was engaged, none if it never was. Rollback: none, these are environment variables in a job you delete afterwards.
  6. Optional, customer site only: run the same job across two nodes over a real fabric and compare the transport line against the single-node result.
  7. Restore. Delete every MPIJob you created and re-run the Allocatable command from step 1 to confirm the node is exactly as you found it.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why 'which scheduler should we use' is the wrong first question for an AI cluster, and what the right first question is.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against kubeflow/mpi-operator README (v0.8.2, v2beta1) re-fetched 2026-09-07; NVIDIA Run:ai self-hosted 2.24 cluster system requirements re-fetched 2026-09-07; GPU Operator platform support re-fetched 2026-09-07. Dates are when each page was fetched.

  1. GitHub — kubeflow/mpi-operator · fetched 2026-09-07
  2. NVIDIA Run:ai Self-hosted v2.24 — Cluster System Requirements · fetched 2026-09-07
  3. NVIDIA GPU Operator — Platform Support · fetched 2026-09-07
  4. Slurm Workload Manager — Slinky · fetched 2026-09-07
  5. Kubernetes — Device Plugins · fetched 2026-09-07
  6. NVIDIA Network Operator v26.7.0 — Platform Support · fetched 2026-09-07
  7. NVIDIA Network Operator v26.7.0 — Deployment Guide with Kubernetes · fetched 2026-09-07
  8. NVIDIA Network Operator v26.7.0 — Spectrum-X Quick Start · fetched 2026-09-07
  9. NCCL User Guide — Environment Variables · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.