The Kubernetes network model and where a second CNI fits

1Four requirements, and Kubernetes implements none of them

Start where a customer starts. The Kubernetes network model is four sentences, and Calico writes them out: “Every pod gets its own IP address”; “Containers within a pod share the pod IP address and can communicate freely with each other”; “Pods can communicate with all other pods in the cluster using pod IP addresses (without NAT)”; and “Isolation (restricting what each pod can communicate with) is defined using network policies”.^[1]

Now the part that surprises people in a design review: Kubernetes itself implements none of it. Implementations plug in through the CNI API, and an implementation consists of two things - a network plugin that attaches pods to the network, and an IPAM plugin that assigns their addresses.^[1] Service load balancing is a third, separate job, handled by kube-proxy, which spreads traffic from virtual IPs across the backing pods.^[1]

Read the four requirements again with an AI cluster in mind and notice what is missing. Nothing in the model mentions bandwidth, latency, MTU, hardware queues, or direct memory access. The model is about reachability and isolation. A cluster can satisfy every requirement perfectly and still be useless for distributed training, because a 1500-byte software-switched veth pair is a correct implementation of “pods can communicate without NAT”.

That gap is the entire reason this course exists. The rest of the module is about the second network you add because the first one is doing a different job well.

2What the primary CNI owns

Whichever plugin you install - Calico, Cilium, OpenShift’s OVN-Kubernetes - it becomes the primary CNI. It owns eth0 inside every pod, and with it the Service ClusterIPs, cluster DNS, and NetworkPolicy enforcement.^[1] Calico’s component list shows how much machinery that is: Felix “Programs routes and ACLs, and anything else required on the host to provide desired connectivity for the endpoints on that host”, BIRD distributes routes to BGP peers, confd regenerates BIRD’s configuration from the datastore, and Typha shields the datastore from a few hundred Felix instances.^[9]

None of that machinery is in the path of an ibv_post_send. In an AI cluster the RDMA and GPU east-west traffic rides a secondary interface, attached by Multus onto an SR-IOV virtual function, a whole host device, or a macvlan or IPoIB child link - and it never goes through the primary CNI.^[2][4]

flowchart LR
  subgraph POD[Training pod]
    E0[eth0 from primary CNI]
    N1[net1 from Multus delegate]
  end
  E0 --> CNI[Calico or Cilium datapath]
  CNI --> SVC[Services / DNS / NetworkPolicy]
  SVC --> MGMT[(Management fabric)]
  N1 --> DEL[macvlan / host-device / SR-IOV VF]
  DEL --> PF[ConnectX or BlueField-3 PF]
  PF --> RAIL[(RDMA fabric)]
  DP[Device plugin] -. advertises resource .-> N1

The practical consequence for triage is a sorting rule, and it is worth internalising before any of the plumbing. Symptoms on the top path - pods cannot resolve a Service, a NetworkPolicy is not taking effect, a node’s pods are unreachable - belong to the primary CNI. Symptoms on the bottom path - a job’s collective stalls, ib_write_bw will not run between two pods, a second interface never appears - belong to Multus, the delegate CNI, the device plugin or the driver.^[2][3]

The failure that breaks this rule is the one to remember: when the primary CNI has autodetected an RDMA-capable PF as its own uplink, both layers share one interface, and a routine Calico or Cilium reconfiguration takes the fast path down with it.^[9]

3The four pieces that build the second interface

The second path is built by four cooperating pieces, and knowing the order they run in is most of the debugging skill.

Multus is first. “Multus CNI enables attaching multiple network interfaces to pods in Kubernetes” - it is a meta-plugin that delegates to other CNI plugins for every attachment beyond the cluster network.^[3] It reads a pod annotation, resolves the named NetworkAttachmentDefinition, and calls the plugin named inside it.

The delegate CNI is second, and it is the piece that actually creates or moves an interface: macvlan, ipvlan, host-device, sriov, ib-sriov or ipoib. NVIDIA’s own quick start pairs them explicitly, offering “IPoIB with Shared Device” and “MacVLAN Network with Shared Device” against “SR-IOV with RDMA” and “Host Device with RDMA”.^[7]

The IPAM plugin is third, named inside the delegate’s config, and it is a genuinely separate program.^[1] Lesson 5 is about why host-local is the wrong answer for a cluster-wide fabric.

The device plugin runs beside all of this rather than in the chain. Device plugins let vendors advertise hardware to kubelet “without modifying Kubernetes core code”, using the resource naming convention vendor-domain/resourcetype, and the resources they advertise are integer-only, not overcommittable, and cannot be shared between containers.^[6] That is what makes a pod’s resources block able to ask for a NIC at all, and why a pod that asks for a resource nobody advertises stays Pending forever with no CNI error to read.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-net
  namespace: default
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "macvlan-net",
      "type": "macvlan",
      "master": "ens1f0",
      "mode": "bridge",
      "mtu": 1500,
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.2.225/28",
        "exclude": [
          "192.168.2.229/30",
          "192.168.2.236/32"
        ]
      }
    }

Above all of it sits the NVIDIA Network Operator, whose scope statement is a precise description of this second path and nothing else: it “leverages Kubernetes CRDs and Operator SDK to manage Networking related Components in order to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster”.^[2] The components it can deploy are the pieces above - the DOCA-OFED driver container, the RDMA shared device plugin, the SR-IOV network device plugin, InfiniBand Kubernetes, Multus, the container networking plugins, the IPoIB CNI and NVIDIA’s own IPAM.^[2] There is no primary CNI on that list, and that absence is the answer to the customer’s first question.

4Scope boundaries: what the hardware and the certification actually say

Two boundaries are worth fixing now because both come up in the first customer meeting.

The hardware boundary. Network Operator 26.7.0 supports Kubernetes >=1.32 and <=1.36, NICs from ConnectX-6 through ConnectX-9, and BlueField-3 DPU and SuperNIC in NIC mode only, up to 800 Gb/s.^[5] The mode restriction matters directly in a Dell lab: a BlueField-3 sitting in DPU mode is outside the supported configuration for everything this course builds, and the fix is a mode change, not a CNI change.^[5]

The certification boundary. This course maps against the NVIDIA-Certified Associate: AI Infrastructure and Operations blueprint, whose Domain 2 objectives include 2.5 “Identify key components and considerations of a cluster of an accelerated infrastructure” and 2.7 “Determine networking requirements for AI workloads”.^[8] Be honest about the fit: the blueprint never names CNI, Multus, SR-IOV or the Network Operator anywhere, and the two domains this course touches least carry 60 percent of the exam.^[8] This module goes far deeper than 2.5 and 2.7 ask. Treat it as over-serving a subset of the blueprint, not as exam preparation.