Multus and the NetworkAttachmentDefinition

Episode 4 — The perfect manifest

The situation · The customer's staging cluster, day sixteen, screen-shared late in the afternoon

The platform engineer has the manifest open and it is correct. The NetworkAttachmentDefinition exists, kubectl get network-attachment-definitions lists it, the delegate config is the macvlan block from NVIDIA’s guide. The pod starts, runs, serves traffic - and ip -br addr inside it shows lo and eth0 and nothing else. No error. Everyone on the call is now looking at the NIC, procurement is asking twice a day about lead times for cards nobody has ordered, and the SE has given up pretending the coffee is warm.

Nothing is wrong with the NIC, because Multus never builds an interface itself. “Multus CNI enables attaching multiple network interfaces to pods in Kubernetes” - a meta-plugin that delegates to other CNI plugins for every attachment beyond the cluster network, following the Kubernetes Network Custom Resource Definition De-facto Standard rather than its own object model.^[1] That indirection is why it exists: the CNI API gives a pod one network, and rather than fork the API, the Network Plumbing Working Group added a name - an annotation on the pod, a CRD holding the delegate config, and a shim that reads one and calls the other.

Which means almost every failure here is a lookup failure rather than a networking failure: a name that does not resolve, a namespace that does not match, JSON that does not parse.^[2] The network lead wants a counter; what settles this one is a string.

Ask for kubectl describe pod verbatim before anybody touches hardware.^[10]

Four checks, two minutes. Start with how the annotation is actually written.

1Two ways to install it, and why the heavier one is recommended

“Multus CNI enables attaching multiple network interfaces to pods in Kubernetes.”^[1] It is a meta-plugin: it delegates to other CNI plugins for every attachment beyond the cluster network, and it follows the Kubernetes Network Custom Resource Definition De-facto Standard rather than inventing its own object model.^[1]

There are two deployment shapes. The thick plugin arrived in version 4.0 and consists of two binaries - multus-daemon, a per-node agent, plus the multus-shim CNI plugin. It supports capabilities the thin model did not have, such as metrics, and the project is explicit about the trade-off: the thick plugin “comes with the trade-off of consuming more resources than the ‘thin plugin’”, and is nonetheless recommended for most deployments.^[1] The thin plugin is the older single-binary model, kept for resource-constrained environments.^[1]

# thick (recommended)
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml

# thin
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml

^[1] In a Network Operator deployment you usually install neither by hand: NicClusterPolicy.spec.secondaryNetwork.multus deploys the multus-cni image from nvcr.io/nvidia/mellanox at the operator’s own version tag.^[3] Knowing the upstream manifests still matters, because that is what a customer’s platform team probably applied before you arrived, and the two paths produce different owners for the same DaemonSet.

2Every form of the annotation, and what each one is for

One annotation drives all of this: k8s.v1.cni.cncf.io/networks. It has four documented forms and each exists for a different reason.

Comma-separated list - the common case, attaching two networks in order:

k8s.v1.cni.cncf.io/networks: macvlan-conf-1, macvlan-conf-2

^[2]

Interface pinning with an @ suffix, when the workload needs a predictable device name rather than whatever net1, net2 ordering it happens to get:

k8s.v1.cni.cncf.io/networks: macvlan-conf-1@macvlan1

^[2]

Namespace qualification with a slash, when the NAD lives in a shared networking namespace rather than the workload’s own:

k8s.v1.cni.cncf.io/networks: testns1/macvlan-conf-3

^[2]

JSON list form, which is the only one that can carry per-attachment options. It supports name, namespace, interface and default-route:

k8s.v1.cni.cncf.io/networks: '[{"name":"macvlan-conf","default-route":["192.168.2.1"]}]'

^[2] That last example is worth pausing on: it moves the pod’s default route onto the secondary interface. That is occasionally what an appliance-style workload wants and almost never what a training job wants, because it takes the pod’s egress off the primary CNI and with it anything that depended on cluster-managed routing.

primary

delegate

ipam

options

Checks

no forbidden combination

warnThe NAD is in net-attach and the pod in default: the annotation must be namespace-qualified (net-attach/macvlan-net, or the JSON form with "namespace"). It works upstream; it will not work unchanged on OpenShift, where namespace isolation is on by default. [NAD namespace + OpenShift Multus namespace isolation]
infomacvlan mtu has range 0 to the master’s MTU — raise the PF first, then the pod. One master cannot be enslaved by macvlan and ipvlan at once. [macvlan mtu is capped at the master]
infoCalico owns eth0 only. Confirm it has not autodetected ens1f0 as its uplink: sharing one interface with the fast path works until Felix reconfigures it, and then RDMA disappears. [primary CNI must not claim the fast-path interface]

Predict before you read the checks: what breaks in this combination? (optional, saved)

selected block

Delegate — macvlan

Functions like a switch already connected to the host interface; each virtual interface gets a distinct MAC, so existing DHCP servers work.

Required: name, type, ipam (may be empty for an address-less interface). Optional: master (defaults to the default-route interface), mode, mtu, linkInContainer.
Modes: bridge (default), private, vepa, passthru. mtu range is 0 to the master’s MTU — you cannot raise a pod above the host interface.
One master cannot be enslaved by macvlan and ipvlan at once; most wireless cards cannot be enslaved at all.
Operator route: MacvlanNetwork (mellanox.com/v1alpha1: networkNamespace, master, mode, mtu, ipam) generates the NAD for you.

FAE angle: macvlan gives an IP-level second interface only. RDMA over it exists solely through the shared RDMA device plugin (rdma/rdma_shared_device_a) — a scheduling counter, not isolation.

source

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net
  namespace: net-attach
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "macvlan-net",
      "type": "macvlan",
      "master": "ens1f0",
      "mode": "bridge",
      "mtu": 1500,
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.2.225/28",
        "exclude": [
          "192.168.2.229/30",
          "192.168.2.236/32"
        ]
      }
    }

Operator route: MacvlanNetwork (mellanox.com/v1alpha1: networkNamespace, master, mode, mtu, ipam) generates the NAD for you. Network Operator deployment guide · Multus how-to-use · quick start

Start with sameNamespace off and read the validation message - that is failure mode one from the checklist below, before you have written a single YAML file. Then turn it on and watch which of the three emitted artifacts changes.

3Two places a NAD's configuration can live

A NetworkAttachmentDefinition normally carries its CNI JSON inline in spec.config:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{"cniVersion":"0.3.1","type":"macvlan","master":"eth1","mode":"bridge","ipam":{"type":"whereabouts","range":"192.168.2.225/28"}}'

^[2]^[6]^[8]

There is a second, less-used shape. If a NAD has no spec, Multus looks for a file in the default configuration directory - /etc/cni/multus/net.d/ - whose CNI config name field matches the resource.^[2] That exists for cases where the configuration is managed on the node rather than in the API, and it is perfectly legitimate. It is also how you end up with a cluster where some attachments are described in Kubernetes and others only on disk, and where the answer to “what is this network?” depends on which node you SSH into.

In the Network Operator flow you often write neither by hand. The operator’s own CRs - MacvlanNetwork, HostDeviceNetwork, IPoIBNetwork - generate the NAD for you from fields like networkNamespace, master, mode, mtu and an inline ipam string.^[3] The exception is the InfiniBand SR-IOV case, where you still hand-write the chained plugin list.^[3] Knowing both surfaces matters when a customer says “we did not create any NADs” and kubectl get net-attach-def -A disagrees with them - the operator did.^[9]

4The four-item failure checklist

Nearly every “the pod has no second interface” ticket is one of four things, and all four are checkable in about two minutes.

Wrong namespace. The NAD is not in the pod’s namespace and the annotation did not qualify it. On OpenShift this is even more common, because Multus there runs with namespaceIsolation enabled by default: the pod must live in the NAD’s namespace unless the NAD sits in default, openshift-multus, openshift-sriov-network-operator or openshift-cnv. Under Network Operator the lever is the networkNamespace field on MacvlanNetwork, HostDeviceNetwork, IPoIBNetwork, OVSNetwork and SriovNetwork, which places the generated NAD where the pods actually run.^[4]
Misspelled annotation. The key is k8s.v1.cni.cncf.io/networks. Anything else is silently ignored - Kubernetes stores unknown annotations happily and nothing complains.^[2]
Invalid spec.config JSON. The NAD exists and looks fine in kubectl get, but the delegate configuration inside it never parsed.^[2]
A resourceName matching no advertised device-plugin resource. This one is different in kind: the pod stays Pending at scheduling and never reaches CNI ADD at all, because extended resources are scheduling input and are integer-only and non-shareable.^[7] The default resourcePrefix for the SR-IOV network device plugin is intel.com, which is why NVIDIA’s own configurations always set "resourcePrefix": "nvidia.com" explicitly - and why a pod copied from one cluster to another can ask for a resource that exists everywhere except by that name.^[5]^[3]

The artifact that separates them is always the same: kubectl describe pod, read verbatim - the command Multus’s own quickstart uses to confirm an attachment.^[10] Cases 1 to 3 surface a delegate error in the pod’s events; case 4 produces a scheduling message and nothing from Multus at all, and that absence is the diagnosis.^[7]

Build the three artifacts for a macvlan second interface

Ask: a training pod in namespace ml-jobs needs a second interface on the host’s ens1f0, named net1, addressed from a cluster-wide pool, with the pod’s default route left on the primary CNI.

Choose the delegate. No hardware isolation is required and no RDMA yet, so macvlan in the default bridge mode.^[6]
Choose the IPAM. Cluster-wide, so whereabouts with a range - host-local would hand duplicate addresses to pods on different nodes.^[8]

Write the NAD, in the pod’s own namespace so no qualification is needed:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: fast-net
  namespace: ml-jobs
spec:
  config: '{"cniVersion":"0.3.1","type":"macvlan","master":"ens1f0","mode":"bridge","ipam":{"type":"whereabouts","range":"192.168.2.225/28"}}'

^[2] ^[6] ^[8]

Write the annotation. Default interface naming is acceptable here, so the plain form is enough: k8s.v1.cni.cncf.io/networks: fast-net.^[2]
Decide the default route. Leave it alone - do not add a default-route key, because that would move pod egress off the primary CNI.^[2]
Resources block: none. macvlan does not consume a device-plugin resource, which is exactly why this configuration schedules anywhere and gives no hardware isolation.^[7]
Verify: kubectl exec -- ip -d link should show net1 of type macvlan with ens1f0 as its master, plus the unchanged eth0.

Two minutes, four checks

How it ended

The events name the delegate that failed.^[10] The NAD lives in a shared net-infra namespace, the pods live in the training namespace, and the annotation names the NAD without qualifying it - a lookup for something that does not exist there.^[2] Write it net-infra/rail-a, or move the NAD next to the pods, and net1 appears on all eight nodes at once. Nobody touched the NIC, the driver or the switch. The operator relabels the ports and goes home; at 02:10 he calls anyway. Two pods, two different nodes, one address on the rail subnet - and the network team has already opened a ticket against the leaves.

Lab

Goal: the same attachment against a real ConnectX PF on the Dell BF-3 host, and confirmation that the interface is genuinely a child of that PF. Read-only apart from creating and deleting NADs and pods - no firmware, no mode change, no host interface reconfiguration.

Pre-flight inventory:
```
ip -br link
ethtool -i <pf> | head -3
ibstat | head -20
kubectl get net-attach-def -A
kubectl -n kube-system get ds | grep -i multus
```
Expected: the PF is present and mlx5_core, ibstat sees the HCA, and you have a written record of any NADs that already existed - so you can prove at the end that you left none behind.
Confirm which interface the primary CNI is using, exactly as in lesson 1, and confirm it is not the PF you are about to use as a macvlan master. If it is, stop and choose a different PF; enslaving the CNI’s own uplink is not a read-only act in effect even though every command here is.

Create a macvlan NAD whose master is the real PF:

kubectl apply -f - <<'EOF'
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata: { name: cx-macvlan }
spec:
  config: '{"cniVersion":"0.3.1","type":"macvlan","master":"ens1f0","mode":"bridge","ipam":{"type":"whereabouts","range":"192.168.2.225/28"}}'
EOF

^[2]^[6]^[8] Rollback: kubectl delete net-attach-def cx-macvlan.

Launch a pod against it with interface pinning so you can see the @ form work on real hardware:
```
kubectl run cx-probe --image=nicolaka/netshoot --restart=Never \
  --annotations="k8s.v1.cni.cncf.io/networks=cx-macvlan@fast0" -- sleep 3600
kubectl exec cx-probe -- ip -d link
```
^[2] Expected: an interface literally named fast0, of type macvlan, whose master is the ConnectX PF, alongside the untouched eth0. Rollback: kubectl delete pod cx-probe.
Confirm what this attachment did not give you. Inside the pod, rdma link and ibv_devinfo - expected: nothing useful. A macvlan child link is an IP-level attachment; RDMA over it needs the shared device plugin, which is module 2’s subject.^[9]
Confirm the host is unchanged: re-run every command from step 1 and diff against your inventory. The PF’s MTU, addresses and HCA state must be identical, and kubectl get net-attach-def -A must return to exactly the set you recorded.
Deliverable: the ip -d link output from step 4 pasted beside the equivalent output from the kind lab. The point to note in writing is how little of it differs - the whole difference between a laptop demo and a GPU node is the master field.

Rollback for the whole lab: delete the pod, delete the NAD, re-run step 1 and confirm it matches the starting inventory.

Evidence / notes (commands run, outputs, what surprised you)

Multus and the NetworkAttachmentDefinition

Before you read: what do you already know?

After this lesson you can

Episode 4 — The perfect manifest

1Two ways to install it, and why the heavier one is recommended

2Every form of the annotation, and what each one is for

Checks

Delegate — macvlan

3Two places a NAD's configuration can live

4The four-item failure checklist

Two minutes, four checks

Lab

Retrieval check

Explain it to a Dell SE

Sources

The same idea elsewhere