NUMA alignment with Topology Manager and CPU Manager
S3·E5Six nodes are slower and nobody can say which · Dell customer lab, three days from the acceptance run
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Evaluate a Topology Manager policy and scope choice against a given node topology and workload shape.
- Justify why a TopologyAffinityError rejection is the system working rather than a setting to disable.
- Critique a CPU Manager migration plan and identify the step whose omission fails silently.
- Recommend a remedy for an unalignable node - policy option, pod shape, or physical NIC placement.
Episode 5 — Six nodes are slower and nobody can say which
Twelve dual-socket nodes, one image, one container, and the same training job finishes forty minutes later on six of them than on the other six. Nothing is red. The pods are Running, the fabric counters are boring, and a ticket has already been opened against the switch. The network lead asks for the counter and for once you have to tell him there is not one. He writes that down and underlines it.
The reason this component exists is the reason the problem is invisible. A two-socket server is two machines sharing a name: memory and PCIe slots are split, and a transfer between the halves costs time that no log records. Kubernetes does not read that off the hardware. Topology Manager only considers resources whose hint providers actually supply topology hints, and a device plugin supplies one by putting TopologyInfo, carrying the NUMA node of the device it advertises, in its response.[3] With the default policy none there is no alignment at all and every hint provider is ignored, so the pod is admitted and the job is simply slow.[1]
The platform lead asks you to make the six slow nodes fast. What you are about to do instead is make the slowness loud, and the first thing this customer will see is pods that refuse to start.[1]
Alignment is a vote, and only the resources that vote are counted.
1Half speed is a configuration argument
On a dual-socket PowerEdge with eight GPUs and several NICs, “the job runs at half speed” is usually a NIC on one socket feeding a GPU on the other. Nothing errors. The pod is Running, the fabric is clean, the counters are boring, and the transfers cross the inter-socket link on every message.
Kubernetes has one component whose whole job is to prevent that. Topology Manager offers four policies: none, the default, performs no alignment and ignores every hint provider; best-effort tries to align to a single NUMA node and admits the pod either way; restricted admits only when all requested resources align to a common set of NUMA nodes and rejects otherwise; single-numa-node requires everything to come from one NUMA node or the pod is rejected.[1] It requires Kubernetes v1.18 or later and aligns pods of every QoS class, but only for resources whose hint providers actually supply topology hints.[1]
That last clause is the whole design. Alignment is not something the scheduler infers from hardware - it is a vote, and only resources that vote are considered. A device plugin votes by putting TopologyInfo in its response, which carries the NUMA node for the device it is advertising.[3] Dynamic Resource Allocation does the newer equivalent by advertising devices as ResourceSlice objects with attributes including PCIe bus id and NUMA node.[5]
The Network Operator GPUDirect example requests nvidia.com/hostdev: 1 and nvidia.com/gpu: 1 in the same container, with securityContext.capabilities.add: ["IPC_LOCK"]. That single co-request is what makes NUMA alignment possible at all.
Selecting a device explains it and lets you move it to the other socket — the physical change an FAE actually recommends. Devices are keyboard reachable: Tab, then Enter.
Set the policy, the placement and the request, then submit the pod to see the admission decision.
Default. No alignment: every Hint Provider is ignored.
The pod is always admitted, so the failure never shows up as an error — it shows up as throughput. This is the state most customer clusters are in when they call about "the job runs at half speed".
Default. Containers are aligned one at a time, with no grouping: a GPU-only container and a NIC-only container in the same pod can legally land on different NUMA nodes. Co-requesting in one container is what makes both Hint Providers vote on the same decision.
Default. CFS quota only; the workload can migrate across cores and sockets, and CPU casts no topology hint. Alignment then rests on the device plugins alone.
Both options sit behind the TopologyManagerPolicyOptions feature gate. prefer-closest-numa-nodes defaults to false and applies to best-effort and restricted only. ⚠ max-allowable-numa-nodes is recorded in the course notes as “default unlimited, caps how many NUMA nodes a pod's resources may span”; the upstream page could not be re-read to confirm whether it caps the pod span or the machine's NUMA count — verify before quoting it to a customer.
# /var/lib/kubelet/config.yaml topologyManagerPolicy: "none" topologyManagerScope: "container" cpuManagerPolicy: "none"
spec:
containers:
- name: trainer
securityContext:
capabilities:
add: ["IPC_LOCK"]
resources:
limits: # requests must match limits → Guaranteed QoS
nvidia.com/gpu: "1"
nvidia.com/hostdev: "1"
cpu: "8"On a Spectrum-X rail deployment the NIC side is requested as nvidia.com/rail0 / nvidia.com/rail1 instead of nvidia.com/hostdev; on the RDMA shared plugin it is rdma/rdma_shared_device_a. Same alignment question, different resource name.
“The job runs at half speed” on a dual-socket PowerEdge is usually a NIC on socket 0 feeding a GPU on socket 1. The Kubernetes-side fix is single-numa-node + scope: pod + CPU Manager static, with nvidia.com/gpu and nvidia.com/hostdev in the same container so both hint providers vote. Then the failure inverts into TopologyAffinityError — and that rejection is the system working.
Map the box before you argue about policy: lstopo, numactl --hardware, nvidia-smi topo -m, cat /sys/class/net/<pf>/device/numa_node.
Documented limitation: a hard limit of 8 NUMA nodes per system; systems with more are unsupported, with no workaround. Check it before recommending an NPS4 BIOS setting on a high-core-count AMD node — dual socket × NPS4 is already 8.
Dell publishes the 2-8-9-400 configuration on PowerEdge XE9680: up to 12 worker nodes, "each equipped with two high-performance processors and eight NVIDIA H200 SXM GPUs". Two sockets, eight GPUs — that is the board this planner draws.
Topology Manager aligns pods of all QoS classes, but only for resources whose Hint Providers actually supply topology hints. Topology Manager requires Kubernetes v1.18 or later.
GPUDirect RDMA moves data NIC↔GPU without touching host memory, which is exactly why the NIC and the GPU want to hang off the same NUMA node. Verify the module path with lsmod | grep nvidia and the transport with NCCL_DEBUG=INFO.
⚠ The per-node NIC count and the 24 allocatable CPUs per NUMA node in this diagram are illustrative: the Dell brief confirms two sockets and eight H200 SXM GPUs, but the “9 NICs” reading of 2-8-9-400 is unverified.
2Scope: the difference between one decision and two
The policy says how strict alignment is. The scope says what is being aligned. container is the default and aligns containers one at a time with no grouping; pod aligns all containers in a pod onto a common set of NUMA nodes, and is set with topologyManagerScope.[1] Under pod scope the resource total uses the effective-request formula - the maximum of the sum of the app containers, or the maximum of the init containers.[1]
The practical consequence is the one that catches people. A pod with a GPU container and a separate NIC container is two independent alignment decisions even under single-numa-node: each is individually satisfiable, both are admitted, and they can land on opposite sockets. That is why NVIDIA’s GPUDirect example requests nvidia.com/hostdev: 1 and nvidia.com/gpu: 1 in the same container with IPC_LOCK rather than splitting them.[4] A co-request is one decision with two voters.
There are therefore two ways to buy the same guarantee, and they belong to different people. Co-requesting is a pod-spec change that an application team can make without touching the platform. topologyManagerScope: pod is a kubelet configuration change that a platform team makes for everyone.[1] On a Dell AI Factory node where the same operator set is deployed across the fleet, deciding which of the two you are recommending is half the conversation.[7]
3CPU Manager static, and the step that fails silently
CPU alignment is a separate subsystem with its own policy. none is the default and uses CFS quota, letting workloads migrate; static assigns exclusive CPUs to Guaranteed pods with integer CPU requests and keeps a shared pool for BestEffort and Burstable pods - and for Guaranteed pods with fractional requests.[2] Only under static does CPU become a hint provider that votes alongside the GPU and the NIC.[1]
The static policy needs a reservation greater than zero. Precedence is --reserved-cpus as an explicit list first; otherwise the reservation comes from --kube-reserved or --system-reserved, taken in ascending order by physical core id.[2] The policy options and their status as of the page fetched on 2026-09-09: full-pcpus-only GA 1.33+, distribute-cpus-across-numa Beta 1.33+, align-by-socket Alpha 1.25+, distribute-cpus-across-cores Alpha 1.31+, strict-cpu-reservation GA 1.35+, prefer-align-cpus-by-uncorecache GA 1.36+, gated by CPUManagerPolicyOptions with CPUManagerPolicyBetaOptions on by default and CPUManagerPolicyAlphaOptions off by default.[2]
Changing the policy is a ritual, and it has a step people skip. Drain the node, stop the kubelet, remove /var/lib/kubelet/cpu_manager_state, update the kubelet configuration, start the kubelet.[2] Then uncordon - that step is yours, not the documentation’s. Skip the delete and the kubelet refuses to start, reporting that the configured policy differs from the state checkpoint policy.[2] The failure is loud in the kubelet log and invisible everywhere else - a node that is cordoned and not coming back looks, from a dashboard, like a node that is being worked on.
# kubelet configuration - the four lines this lesson is about
cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node
topologyManagerScope: pod
topologyManagerPolicyOptions:
prefer-closest-numa-nodes: "true"[1][2] max-allowable-numa-nodes is deliberately absent: it defaults to 8, only accepts values greater than 8, and belongs to a node with more than 8 NUMA nodes - not to this recipe.[1] The policy options require the TopologyManagerPolicyOptions feature gate; a re-read of the page on 2026-09-09 also names a per-option gate PreferClosestNUMANodes, so verify both against the cluster’s Kubernetes minor before promising a customer the option is available.[1]
4TopologyAffinityError is the system working
Turn alignment on and the failure mode inverts. Instead of jobs that run slowly, you get pods rejected with TopologyAffinityError - the exact string to grep for, produced under restricted and single-numa-node when alignment cannot be satisfied.[1] The pod is rejected rather than automatically retried onto a better node, which is what makes it look like a bug to a customer whose node has free GPUs and free NICs sitting right there.
It is not a bug. It is the node telling you that no NUMA node holds both a free GPU and a free NIC, which was equally true yesterday when the pod was admitted and ran at half speed. The remedies, in the order worth proposing: co-request the devices in one container so alignment is even attempted; move from single-numa-node to restricted with prefer-closest-numa-nodes, which still refuses a genuinely crossed placement while allowing a multi-node set when the request cannot fit one node; and, when the topology is simply wrong, move a card.[1][4] Turning Topology Manager back to none is not on the list - it does not change where the devices are, it only removes the report.
Two constraints bound the argument. Above 8 NUMA nodes the kubelet does not enable Topology Manager by default. The documented escape is the max-allowable-numa-nodes policy option (GA since Kubernetes 1.35), which the docs flag as low-data and at your own risk - so treat it as a caution before an aggressive NPS setting on a high-core-count AMD platform, not as a hard wall.[1] And that option defaults to 8: the value you set must itself be greater than 8, so raising it is how you allow alignment on a very large machine - it is not a per-pod span limit.[1]
The Network Operator GPUDirect example requests nvidia.com/hostdev: 1 and nvidia.com/gpu: 1 in the same container, with securityContext.capabilities.add: ["IPC_LOCK"]. That single co-request is what makes NUMA alignment possible at all.
Selecting a device explains it and lets you move it to the other socket — the physical change an FAE actually recommends. Devices are keyboard reachable: Tab, then Enter.
Set the policy, the placement and the request, then submit the pod to see the admission decision.
Admit only if all requested resources align to a common set of NUMA nodes; otherwise reject.
A multi-node set is allowed when the request genuinely cannot fit one node (8 GPUs on a 4-per-socket box), but a GPU on one socket and a NIC on the other is rejected. This is the setting to recommend when single-numa-node is too strict.
All containers in the pod land on a common set of NUMA nodes. Pod-scope resource math uses the effective-request formula: the max of the sum of the app containers, or the max of the init containers.
Exclusive CPUs for Guaranteed pods with integer CPU requests; BestEffort and Burstable pods stay in the shared pool. Only then does CPU become a Hint Provider and vote with the GPU and the NIC.
Both options sit behind the TopologyManagerPolicyOptions feature gate. prefer-closest-numa-nodes defaults to false and applies to best-effort and restricted only. ⚠ max-allowable-numa-nodes is recorded in the course notes as “default unlimited, caps how many NUMA nodes a pod's resources may span”; the upstream page could not be re-read to confirm whether it caps the pod span or the machine's NUMA count — verify before quoting it to a customer.
full-pcpus-only=true— GA 1.33+distribute-cpus-across-numa=true— Beta 1.33+align-by-socket=true— Alpha 1.25+distribute-cpus-across-cores=true— Alpha 1.31+strict-cpu-reservation=true— GA 1.35+prefer-align-cpus-by-uncorecache=true— GA 1.36+
Static-policy reservation precedence, first match wins: --reserved-cpus (explicit list, highest), then --kube-reserved, then --system-reserved. The reservation must be greater than zero.
Switching CPU Manager from none to static: drain the node, stop kubelet, rm /var/lib/kubelet/cpu_manager_state, update the config, start kubelet. Skipping the state-file delete is a classic silent failure — the new policy simply does not take effect.
# /var/lib/kubelet/config.yaml topologyManagerPolicy: "restricted" topologyManagerScope: "pod" topologyManagerPolicyOptions: # needs the TopologyManagerPolicyOptions feature gate prefer-closest-numa-nodes: "true" max-allowable-numa-nodes: "2" cpuManagerPolicy: "static"
spec:
containers:
- name: trainer
securityContext:
capabilities:
add: ["IPC_LOCK"]
resources:
limits: # requests must match limits → Guaranteed QoS
nvidia.com/gpu: "1"
nvidia.com/hostdev: "1"
cpu: "8"On a Spectrum-X rail deployment the NIC side is requested as nvidia.com/rail0 / nvidia.com/rail1 instead of nvidia.com/hostdev; on the RDMA shared plugin it is rdma/rdma_shared_device_a. Same alignment question, different resource name.
“The job runs at half speed” on a dual-socket PowerEdge is usually a NIC on socket 0 feeding a GPU on socket 1. The Kubernetes-side fix is single-numa-node + scope: pod + CPU Manager static, with nvidia.com/gpu and nvidia.com/hostdev in the same container so both hint providers vote. Then the failure inverts into TopologyAffinityError — and that rejection is the system working.
Map the box before you argue about policy: lstopo, numactl --hardware, nvidia-smi topo -m, cat /sys/class/net/<pf>/device/numa_node.
Documented limitation: a hard limit of 8 NUMA nodes per system; systems with more are unsupported, with no workaround. Check it before recommending an NPS4 BIOS setting on a high-core-count AMD node — dual socket × NPS4 is already 8.
Dell publishes the 2-8-9-400 configuration on PowerEdge XE9680: up to 12 worker nodes, "each equipped with two high-performance processors and eight NVIDIA H200 SXM GPUs". Two sockets, eight GPUs — that is the board this planner draws.
Topology Manager aligns pods of all QoS classes, but only for resources whose Hint Providers actually supply topology hints. Topology Manager requires Kubernetes v1.18 or later.
GPUDirect RDMA moves data NIC↔GPU without touching host memory, which is exactly why the NIC and the GPU want to hang off the same NUMA node. Verify the module path with lsmod | grep nvidia and the transport with NCCL_DEBUG=INFO.
⚠ The per-node NIC count and the 24 allocatable CPUs per NUMA node in this diagram are illustrative: the Dell brief confirms two sockets and eight H200 SXM GPUs, but the “9 NICs” reading of 2-8-9-400 is unverified.
5Judging a configuration
Node: dual-socket, 8 GPUs (4 per socket), 4 RDMA NICs all physically on socket 0. Workload: training pods requesting 1 GPU, 1 NIC and 8 CPUs.
Proposal A - single-numa-node with scope pod and CPU Manager static.
Verdict: correct in intent, unusable on this node as built. Socket 1 has GPUs and no NIC, so every pod that lands on a socket-1 GPU is rejected with TopologyAffinityError.[1] Half the GPUs become unschedulable for this workload. Recommend it only alongside a hardware change.
Proposal B - none, because the pods must run.
Verdict: wrong, and the reason is worth stating to the customer in one sentence. none performs no alignment and ignores every hint provider, so the crossed placements return - and they return silently, as the throughput problem that started this.[1] This is not a compromise, it is a decision to stop measuring.
Proposal C - restricted with prefer-closest-numa-nodes, scope pod, CPU Manager static, plus a hardware ticket to move two NICs.
Verdict: recommend. restricted admits only aligned allocations and still permits a multi-node set where the request genuinely cannot fit one node, and the option biases what it does allow toward the closest nodes.[1] CPU Manager static makes CPU a hint provider so the 8 CPUs are drawn from the same node as the devices.[2] The hardware ticket is the actual fix; the configuration is what makes the problem visible until the ticket lands.
The migration, done correctly, for the CPU Manager half:
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
sudo systemctl stop kubelet
sudo rm /var/lib/kubelet/cpu_manager_state
# edit the kubelet config: cpuManagerPolicy: static
sudo systemctl start kubelet
kubectl uncordon <node>[2] Skip the rm and the kubelet will not start, reporting that the configured policy differs from the state checkpoint policy.[2]
Proving it afterwards: a pod that co-requests GPU and NIC in one container with IPC_LOCK should be admitted on a socket-0 GPU and rejected on a socket-1 GPU.[4] Then one training run with NCCL_DEBUG=INFO to confirm the transport, and record the throughput as the new baseline.[8]
Same node, but the customer has moved two NICs so each socket now has two. Evaluate a fourth proposal: single-numa-node, scope container, CPU Manager static, pods with a GPU container and a separate NIC sidecar container.
- The scope is ____ which means alignment is decided ____ rather than for the pod as a whole.[1]
- Therefore the GPU container and the NIC container are ____ decisions and can land on ____[1]
- Will either container be rejected? ____ Why? ____[1]
- What will the customer observe? ____ rather than an error.[1]
- The two independent fixes are ____ (a platform change) and ____ (a pod-spec change).[1][4]
- Which of the two would you recommend to an application team that cannot get kubelet config changed, and why? ____
- If the customer also wants the 8 CPUs on the same node as the devices, which policy must be set and what does it require of the pod? ____[2]
A Dell customer runs twelve dual-socket nodes. Six are balanced (2 NICs per socket) and six have all four NICs on socket 0. They run one kubelet configuration for the whole fleet. Today the fleet is on topologyManagerPolicy: none and CPU Manager none, and their complaint is that the same training job takes 40% longer on some nodes than others, apparently at random.
Produce a written recommendation containing: (a) the measurement you would ask for first, and what result would confirm the diagnosis; (b) a fleet-wide kubelet configuration you would propose, with the policy, the scope, the CPU Manager policy and any policy options, and one sentence justifying each; (c) what will happen on the six unbalanced nodes when that configuration lands, named with the exact string the customer will report to you; (d) your answer to their inevitable request to set the policy back to none; (e) the migration procedure for the CPU Manager change with the step you will call out in bold and why; (f) the hardware recommendation and how you would sequence it against the configuration change.
Acceptance criteria: your measurement uses nvidia-smi topo -m or the NIC’s numa_node sysfs value rather than a fabric test; your configuration names a policy that can reject and says why; you name TopologyAffinityError verbatim; your answer to (d) argues that none removes the report rather than the problem; your migration includes removing /var/lib/kubelet/cpu_manager_state while the kubelet is stopped; and your sequencing does not leave half the GPUs unschedulable while the hardware ticket is open.[1][2][3][4]
Case closed
The fleet moves to restricted with prefer-closest-numa-nodes, scope pod, and CPU Manager static - which is also what makes CPU a hint provider alongside the GPU and the NIC.[1][2] Six nodes begin rejecting GPU-plus-NIC pods with TopologyAffinityError and the platform lead calls it a regression.[1] It is an inventory: those six carry every NIC on one socket, which was equally true last week when the job merely ran slowly. Procurement discovers the lead time on two cards is shorter than the argument about them.
The acceptance run finishes on twelve nodes. The night-shift operator relabels the six that changed. The spreadsheet closes.
Lab
Goal: map the real topology of a dual-socket PowerEdge and decide which GPU:NIC pairs can satisfy single-numa-node. Read-only: every command below observes. No kubelet configuration is changed on this node and no firmware is touched.
Pre-flight inventory:
lstopo --output-format txt 2>/dev/null || lstopo-no-graphics
numactl --hardware
nvidia-smi topo -m
for pf in /sys/class/net/*/device/numa_node; do echo "$pf: $(cat $pf)"; done
kubectl get node <node> -o jsonpath='{.status.allocatable}' | tr ',' '\n'Expected: a NUMA node count, a GPU-to-GPU and GPU-to-NIC affinity matrix, and one NUMA node number per network interface. If nvidia-smi topo -m is unavailable, lstopo plus the sysfs numa_node values are enough to complete the lab - they report the same NUMA number a device plugin publishes as TopologyInfo.[3]
- Draw the node. Two columns, one per socket. Place every GPU and every RDMA-capable PF in the column its
numa_nodesays.[3] Expected: an honest picture. This drawing is the deliverable customers never have. - Read
nvidia-smi topo -mand mark, for each GPU, the NIC with the closest path it reports. Expected: some pairs are on the same socket and some are not - and the socket it puts them on is the NUMA node the plugin would advertise inTopologyInfo.[3] - List the pairs that could satisfy
single-numa-node- a GPU and a NIC on the same NUMA node - and the pairs that could not.[1] Expected: a count. If every pair aligns, this node would take the strict policy with no schedulability cost. - Count the NUMA nodes and compare against 8, the default ceiling above which the kubelet does not enable Topology Manager at all.[1] Expected: 2 on a standard dual-socket configuration; more if an NPS setting has split them, which is worth recording before anyone proposes changing it.
- Check what the cluster advertises: the GPU resource and the NIC resource in the node’s allocatable list, and confirm both come from plugins that supply topology hints.[3] Expected: both present. A resource with no hint does not vote and cannot be aligned.
- Predict, then verify. Write down what would happen to a pod requesting 1 GPU, 1 NIC and 8 CPUs under each of the four policies on this node, before checking the current policy with
kubectl get --raw "/api/v1/nodes/<node>/proxy/configz". Expected: your prediction for the node’s actual policy matches what pods on it are doing today. - Deliverable: the drawing, the alignable-pair count, the NUMA node count, and a one-paragraph recommendation of which policy this node could carry today without any hardware change. Nothing in this lab needs a rollback because nothing in it mutated the node.
Goal: produce a TopologyAffinityError on purpose on a kind or single-node k3s cluster, then do the CPU Manager migration both correctly and incorrectly and compare. No NVIDIA hardware needed. Read-only in the sense that nothing here touches firmware or a NIC - but you are editing kubelet configuration on a disposable cluster, so use a cluster you can delete.
- Inventory the node’s topology first:
numactl --hardwareandlscpu | grep -i numa. Expected: at least two NUMA nodes, or the rest of this lab has nothing to reject. If you have a single-NUMA machine, use a VM configured with two NUMA nodes; alignment cannot fail on a node with one. - Record the current kubelet configuration before you change it - this is your rollback:
cp /var/lib/kubelet/config.yaml ~/kubelet-config-before.yaml. - Set
topologyManagerPolicy: single-numa-nodeandtopologyManagerScope: podin the kubelet config and restart the kubelet.[1] Expected: the kubelet comes up and existing pods are unaffected. Rollback: restore the saved file and restart. - Deploy a topology-aware fake device plugin so a hint provider exists whose hints you control.[3] Use
k8stopologyawareschedwg/sample-device-plugin, which emulates devices per NUMA node for exactly the case where the machine has no multi-NUMA hardware to offer:[10]
Expected:git clone https://github.com/k8stopologyawareschedwg/sample-device-plugin cd sample-device-plugin kubectl create -f config/device-A-configmap.yaml -f config/device-B-configmap.yaml REPOOWNER=<your quay org> make push deploykubectl describe node | grep -A20 Allocatablelists the emulated resources. If not: check that the plugin’s socket appeared under/var/lib/kubelet/device-plugins/and that the plugin pod is Running.[3] - Request the fake resource plus integer CPUs in a Guaranteed pod and force a placement that cannot align - the repo ships
manifests/test-both-taerror.yamlfor exactly that, withmanifests/test-both-success.yamlas the control.[10] Expected: the pod is rejected and the event containsTopologyAffinityError.[1] If it is admitted instead, check that the pod is genuinely Guaranteed with integer CPU requests and that CPU Manager isstatic- a fractional request keeps the pod in the shared pool and removes a voter.[2] - Now do the CPU Manager migration incorrectly: change
cpuManagerPolicytostaticand restart the kubelet without deleting the state file. Expected: the kubelet does not start and the log reports that the configured policy differs from the state checkpoint policy.[2] Record the exact message. - Do it correctly: drain, stop the kubelet,
rm /var/lib/kubelet/cpu_manager_state, update the config, start the kubelet, uncordon.[2] Expected: the kubelet starts and a Guaranteed integer-CPU pod now receives exclusive CPUs. - Compare and write it up. Two paragraphs: what the incorrect path looked like from a dashboard versus from the kubelet log, and why that difference matters when a customer is doing this at 2 a.m. Rollback for the whole lab: restore
~/kubelet-config-before.yaml, restart the kubelet, delete the fake device plugin, or delete the cluster.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell customer in five sentences why their training job runs at half speed on some nodes, and why the fix might be a pod that refuses to start.
Sources
Facts in this lesson were checked against Re-fetched 2026-09-09: Kubernetes Topology Manager (policies, scopes, policy options) and CPU Management Policies (static options with GA/Beta status, reservation precedence, cpu_manager_state migration). Dates are when each page was fetched.
- Kubernetes - Control Topology Management Policies on a node · fetched 2026-09-09
- Kubernetes - Control CPU Management Policies on the Node · fetched 2026-09-09
- Kubernetes - Device Plugins · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Deployment Guide with Kubernetes · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - DRA SR-IOV Driver · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Spectrum-X Quick Start · fetched 2026-09-07
- Dell AI Factory with NVIDIA - NVIDIA 2-8-9-400 configuration (Solution Brief) · fetched 2026-09-07
- NCCL User Guide - Environment Variables · fetched 2026-09-07
- NVIDIA GPU Operator - GPUDirect RDMA and GPUDirect Storage · fetched 2026-09-09
- k8stopologyawareschedwg/sample-device-plugin - a NUMA topology based device plugin · fetched 2026-09-09
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.