macvlan, ipvlan, host-device and cluster-wide IPAM

1macvlan: a switch already connected to the host interface

The macvlan plugin describes itself in one sentence that is worth memorising, because it predicts most of its behaviour: it “functions like a switch that is already connected to the host interface”.^[1] The host interface is enslaved, and each virtual interface shares the physical device while keeping a distinct MAC address - which is why existing DHCP servers keep working against macvlan children.^[1]

Required fields are name, type set to "macvlan", and ipam, which may be an empty dictionary if you want an interface with no address at all.^[1] The optional fields are the ones that matter in a lab: master (defaults to the default-route interface, which is almost never what you want on a GPU node), mode, mtu, and linkInContainer.^[1]

Two documented facts carry weight later. First, mtu has an explicit range - zero to the master’s MTU - so a pod interface can never be raised above the PF it hangs off.^[1] That is the hard cap from the MTU lesson, restated as a plugin rule rather than as arithmetic. Second, one master interface cannot be enslaved by macvlan and ipvlan at the same time, so the two plugins are mutually exclusive per PF, not per cluster.^[1] Wireless cards mostly cannot be enslaved at all, which matters only as a reminder that “enslavement” is a driver capability and not a guarantee.^[1]

kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "8950"}}'
kubectl rollout restart daemonset/calico-node -n kube-system   # the doc says: restart the DaemonSet
ip link show   # the tunnel device MTU must read 8950
kubectl exec -it <pod> -- ip link   # eth0 must read 8950, net1 must read 9000
# NAD delegate:  {"type": "macvlan", "master": "<pf>", "mtu": 9000, "ipam": {...}}

2ipvlan: same wire, one MAC, two limits people trip over

ipvlan is the same idea with one variable changed. The documentation states it directly: “Like its cousin macvlan, it virtualizes the host interface. However unlike macvlan which generates a new MAC address for each interface, ipvlan devices all share the same MAC.”^[2] Fields are master, mode (one of "l2", "l3", "l3s", defaulting to "l2"), mtu (defaulting to the kernel-chosen value rather than to the master’s), ipam, and linkInContainer.^[2]

The shared MAC buys you a switch that does not have to learn a new address per pod. It costs you two things, both documented, both the source of tickets.

The first is DHCP: “Because all ipvlan interfaces share the MAC address with the host interface, DHCP can only be used in conjunction with ClientID (currently not supported by DHCP plugin).”^[2] In practice that means static or cluster-wide IPAM, which is the next segment anyway.

The second is the one that surprises senior people. A container cannot reach its own host through the ipvlan interface - the plugin page says so plainly and tells you to attach another network, such as ptp, if host communication is needed.^[2] Anything that assumes a pod can curl a node-local agent, a node exporter or a health endpoint over the fast interface will fail on ipvlan and work on macvlan, for a reason that has nothing to do with either being broken.

3host-device: moving the whole thing, once

host-device is the bluntest of the three: “This simple plugin will move the requested device from the host’s network namespace to the container’s.”^[3] That single sentence contains the entire capacity model. The device is moved, not shared, so the node has exactly as many host-device attachments as it has devices, and the second pod that asks for the same one does not get a degraded interface - it gets nothing.

Four fields select the device: device (a name such as eth0), hwaddr (a MAC), kernelpath (for example /sys/devices/pci0000:00/0000:00:1f.6) and pciBusID (for example 0000:00:1f.6).^[3] On top of those it supports the deviceID runtime capability, which takes a PCI address - that is the hook a device plugin uses to hand host-device a specific NIC at allocation time rather than having it hard-coded in the NAD.^[3]

Three behaviours to know before you write the lab. CNI_IFNAME is disregarded, so the interface arrives in the pod under its own name and any manifest that expects net1 is wrong.^[3] On CNI DEL the device is returned to the host’s network namespace, which is what makes this reversible and therefore lab-safe.^[3] And when the device is bound to a DPDK driver - selected through pciBusID or the deviceID runtime argument - IPAM configuration is bypassed entirely, because there is no kernel interface left to address.^[3]

In a Network Operator deployment you rarely write this JSON by hand. The HostDeviceNetwork CR takes networkNamespace, resourceName and ipam and generates the NAD, and the matching pod requests nvidia.com/hostdev: 1 in both requests and limits alongside the IPC_LOCK capability.^[6] That resource request is what enforces the one-pod-per-device rule at scheduling time, because extended resources are integer-only, non-overcommittable, and cannot be shared between containers.^[8]

4Cluster-wide IPAM: why host-local is a latent outage

The default IPAM plugin most people copy from an example is host-local, and it keeps its allocation state on the node it runs on - Whereabouts exists precisely because “host-local only knows how to assign IPs to pods on the same node”.^[4] On the primary CNI that is fine, because each node owns a disjoint pod CIDR. On a secondary network where every node draws from the same flat subnet, it is a defect: Whereabouts’ own README notes that people “assume it’ll work across nodes – and then wind up with IP address collisions”, and the resulting duplicate-address symptoms on a rail fabric look exactly like a switch problem, so the ticket arrives at the wrong team.^[4]

Whereabouts is “An IP Address Management (IPAM) CNI plugin that assigns IP addresses cluster-wide.”^[4] Required fields are type set to whereabouts and range.^[4] Optional fields cover the shapes a real subnet needs: range_start and range_end, exclude (a list of CIDRs to skip), network_name (independent allocation for two networks that use the same CIDR), gateway, routes, enable_overlapping_ranges which defaults to true, and node_slice_size, which turns on the experimental Fast IPAM mode by pre-allocating a slice per node.^[4]

{
  "cniVersion": "0.3.0",
  "name": "whereaboutsexample",
  "type": "macvlan",
  "master": "eth0",
  "ipam": {
    "type": "whereabouts",
    "range": "192.168.2.225/28",
    "exclude": ["192.168.2.229/30"]
  }
}

^[4] State lives in two CRs - IPPool stores the allocated addresses and OverlappingRangeIPReservation tracks assignments across ranges - so you can read the allocation table with kubectl instead of guessing.^[4] Two known issues are worth quoting to a customer before they find them: system crashes may leave stranded allocations, which on an unstable cluster can exhaust the range over weeks; and a wide IPv6 CIDR of /64 or larger is only addressable within its first /65 because of a uint64 offset constraint.^[4]

nv-ipam is NVIDIA’s answer and has three parts: ipam-controller, a controller that watches IPPool and CIDRPool CRs in a predefined namespace and assigns each node a cluster-unique range through CR status; ipam-node, a node daemon that installs the CNI binary, performs allocations and persists them to disk behind a gRPC service; and nv-ipam, the CNI binary that calls that daemon.^[5] The CNI config takes poolName (comma-separated for dual-stack), poolType (ippool or cidrpool, case-insensitive), daemonSocket, daemonCallTimeoutSeconds and logLevel.^[5]

The two pool kinds allocate differently and confusing them is a real mistake:

^[5] IPPool also takes subnet, gateway, exclusions, perNodeExclusions, defaultGateway and routes; CIDRPool takes cidr, gatewayIndex, exclusions, perNodeExclusions, staticAllocations, defaultGateway and routes.^[5] NVIDIA’s Spectrum-X rail configuration uses CIDRPool with perNodeNetworkPrefix: 31 and gatewayIndex: 0 - one /31 per node per rail, which is the smallest thing that still has a gateway.^[10] Network Operator deploys nv-ipam itself through NicClusterPolicy.spec.nvIpam, so on an operator-managed cluster it is already versioned and patched with everything else.^[6]

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-net
  namespace: default
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "ipvlan-net",
      "plugins": [
        {
          "type": "ipvlan",
          "master": "ens1f0",
          "mode": "l2",
          "ipam": {
            "type": "host-local",
            "subnet": "10.56.217.0/24"
          }
        },
        {
          "type": "rdma"
        }
      ]
    }

5The decision tree you actually give a customer

The customer question is never “compare these three CNI plugins”. It is “which one for GPUDirect”, and the answer has to survive being repeated by someone else in a meeting you are not in. Three lines:

• host-device moves the whole PF into the pod - full line rate, full RDMA, and exactly one pod per NIC per node.^[3][8]
• SR-IOV slices the PF into virtual functions, so several pods share one NIC with isolation enforced in hardware.^[7]
• macvlan and ipvlan give an IP-level secondary interface, and RDMA over them is available only through the shared RDMA device plugin, which over-advertises one HCA as a counted scheduling resource rather than isolating it per pod.^[9]

NVIDIA’s own quick-start use-case list is organised along exactly this seam, which is useful because you can point a customer at their vendor’s page rather than at your opinion: “SR-IOV Network with RDMA” and “Host Device Network with RDMA” on one side, “IP over InfiniBand with RDMA Shared Device” and “MacVLAN Network with RDMA Shared Device” on the other, with “SR-IOV InfiniBand Network with RDMA” for the IB case.^[7] The words “RDMA Shared Device” in two of those five titles are doing all the work.