How kubelet learns a NIC exists

1A vendor agent and two sockets

Kubernetes ships no knowledge of your NIC. The device-plugin framework exists so vendors can advertise hardware to kubelet “without modifying Kubernetes core code”, which is why every NIC resource in this course - nvidia.com/hostdev, rdma/rdma_shared_device_a, nvidia.com/rail0 - is produced by a DaemonSet and not by Kubernetes itself.^[1] The framework has been stable since v1.26.^[1]

The handshake has one fixed address and one variable one. Kubelet listens on the hardcoded registration socket /var/lib/kubelet/device-plugins/kubelet.sock, exporting a single Registration service.^[1] The plugin serves its own gRPC socket in that same directory and, in the Register call, hands kubelet three things: the name of that socket, the Device Plugin API version, and the ResourceName it intends to advertise.^[1]

Order matters and the documentation says so: start the gRPC service first, register second.^[1] A plugin that registers before it can serve can be asked for devices it does not yet have.

Once registered, the plugin implements five RPCs: GetDevicePluginOptions, ListAndWatch (a stream), Allocate, GetPreferredAllocation and PreStartContainer.^[1] ListAndWatch is the one that carries the inventory, and it is also how a plugin reports that a device has gone unhealthy, so kubelet stops scheduling onto it.^[1] It is also the RPC that carries the topology hint: a plugin that wants Topology Manager alignment sends TopologyInfo back as part of the Device struct during ListAndWatch, and each Device in that stream can name the NUMA node it sits on.^[1]

The last piece of the contract is restart handling. Plugins are told to watch for deletion of their own Unix socket, because that is how a kubelet restart appears from the outside, and to re-register afterwards.^[1] A NIC resource that vanishes from a node after a reboot and never returns is almost always a plugin that failed to re-register - not a card that failed.

flowchart LR; A["1. Plugin DaemonSet starts and initializes the devices"] --> B["2. Plugin opens its own gRPC socket in /var/lib/kubelet/device-plugins/"]; B --> C["3. Register on kubelet.sock: socket name + API version + ResourceName"]; C --> D["4. ListAndWatch streams the device list and health"]; D --> E["5. Kubelet publishes the name and count in node Allocatable"]; E --> F["6. Scheduler places a pod that requests that exact name"]; F --> G["7. Allocate returns device nodes, env vars, mounts, CDI names"]; G --> H["8. Container starts with the device visible"]; E -. "kubelet restart deletes the socket" .-> B

2What a resource name is, and the arithmetic it obeys

The naming convention is vendor-domain/resourcetype - nvidia.com/gpu is the canonical example.^[1] That string is not decoration. It is the entire scheduling contract: the scheduler compares the name a pod requests against the names nodes publish, character for character.

Extended resources obey three rules that shape every design in this module. They are integer only, they are not overcommittable, and devices “cannot be shared between containers”.^[1] There is no 0.5 of a VF and no burst above the advertised count.

That is why the RDMA shared device plugin looks like it breaks the rules. Its required rdmaHcaMax field is the “maximum number of RDMA resources that can be provided by the device plugin resource”, it has no default, and it is what lets one physical HCA be advertised many times over.^[3] Nothing in the kernel has been partitioned. The number is a scheduling budget, and the isolation you might assume from an integer resource simply is not there.^[3][1] Lesson 3 turns that into a design choice.

The prefix half of the name has four different origins, and this is the most expensive default in the stack. In the upstream sriov-network-device-plugin, resourceName is required while resourcePrefix is optional and defaults to intel.com - on a 15b3 NVIDIA card, on any card.^[2] The RDMA shared device plugin defaults its prefix to rdma, which is where rdma/rdma_shared_device_a comes from.^[3][4] The NVIDIA Network Operator writes nvidia.com explicitly in its sriovDevicePlugin block, producing nvidia.com/hostdev.^[4] OpenShift uses the openshift.io prefix for SR-IOV resources instead.^[5]

Four sources, four strings, one scheduler that does exact matching.

3Allocate: what actually reaches the container

Allocate is where the abstraction becomes a file. The AllocateResponse tells the runtime how to modify the container: annotations, device nodes under /dev, environment variables, mounts, and fully-qualified CDI device names.^[1] CDI support went alpha in v1.28, beta in v1.29 and GA in v1.31, behind the DevicePluginCDIDevices feature gate.^[1]

For the SR-IOV plugin the environment-variable half is the part you will read in the field. It injects PCIDEVICE_<RESOURCE_NAME> holding the allocated device IDs and PCIDEVICE_<RESOURCE_NAME>_INFO holding JSON with mount points and metadata; the resource name is upper-cased with . and / replaced by underscores.^[2] So nvidia.com/hostdev becomes PCIDEVICE_NVIDIA_COM_HOSTDEV. When you want proof from inside a running pod that a specific VF was handed to it, that variable is the proof.

One field does not travel in Allocate at all, and it is the one that matters most later. TopologyInfo is sent back “as part of the Device struct during ListAndWatch”, not in the AllocateResponse: it names the NUMA node of the device, and it is the only reason Topology Manager can align a NIC with a GPU at all.^[1] Topology Manager aligns pods of every QoS class, but only for resources whose hint providers actually supply hints.^[7] A plugin that publishes no TopologyInfo makes its devices invisible to alignment, and a single-numa-node policy will then align everything except the NIC.^[7] Module 3 spends a whole lesson on the consequences.

Which devices show up at all depends on what the plugin was told to select and on what the operator supports. Network Operator 26.7.0 lists ConnectX-6, ConnectX-6 Dx, ConnectX-7, ConnectX-8 SuperNIC and ConnectX-9 SuperNIC, plus BlueField-3 DPU and BlueField-3 SuperNIC with the qualifier NIC mode only.^[6] A BlueField left in DPU mode is not a Network Operator story.^[6]

4Three places the name must agree

Every Pending-forever NIC pod in this course is the same bug in a different costume: the name is spelled three times and one spelling differs.

1. Advertised by the device plugin, from resourcePrefix plus resourceName in its ConfigMap.^[2][3]
2. Referenced in the NetworkAttachmentDefinition annotation, so Multus knows which device plugin allocated the PCI address it must pass down.^[4]
3. Requested in the pod’s resources.requests and resources.limits.^[4]

The evidence chain is short. On the node, ls -l /var/lib/kubelet/device-plugins/ shows whether the plugin is talking to kubelet at all.^[1] From the API, kubectl describe node | grep -A20 Allocatable shows the name and count kubelet publishes.^[8] If the name is there and the pod is still Pending, the pod is asking for a different string. If the name is not there, work down: the plugin DaemonSet, then its ConfigMap selectors, then the host itself, where NVIDIA’s own collector runs lsmod, ibstat, ibv_devinfo, mst status and dmesg.^[8]

# ConfigMap sriovdp-config — no resourcePrefix key
{ "resourceList": [{
    "resourceName": "hostdev",
    "selectors": { "vendors": ["15b3"], "isRdma": true }
}]}