Primary CNI choices: Calico and Cilium
S1·E2The kernel that said no · A design review on a video call, day nine, Dell and the customer's platform team
Builds on: The Kubernetes network model and where a second CNI fits
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Decompose Calico into Felix, BIRD, confd and Typha and say which one a given symptom belongs to.
- Distinguish Cilium's encapsulation and native-routing modes by what each one requires of the underlay.
- Predict whether a PowerEdge node's kernel and NIC can carry netkit, IPv4 BIG TCP or IPv6 BIG TCP.
- Analyse a customer's primary-CNI choice against what the vendor documentation does and does not claim about Multus.
Episode 2 — The kernel that said no
The slide says netkit. The platform lead has read the Cilium tuning page twice and wants the fastest datapath on the eight PowerEdge nodes before procurement will release racks two and three. Somebody types uname -r into the shared terminal: 6.5. The room goes quiet, because netkit needs kernel 6.8 or newer.[5] The NVIDIA PM is asked twice whether that floor moves, and says “not announced” both times, pleasantly.
This is the moment a primary CNI stops being a logo and becomes five specific facts. Cilium’s fast paths are kernel-gated and it does not soften the landing: with kubeProxyReplacement=true it “exits if kernel support missing” rather than falling back.[4] Calico decomposes just as bluntly - Felix programs routes and ACLs on the host, BIRD distributes them to BGP peers, confd regenerates BIRD’s configuration, Typha shields the datastore - and a symptom belongs to exactly one of them.[1]
Both products exist because Kubernetes hands the whole implementation to a plugin, and each answered the same question differently: wrap pod traffic and ask nothing of the underlay, or route it natively and require that “the network connecting the cluster nodes must be capable of routing PodCIDRs”.[3] Two racks with a routed spine turn that fork into a question for the network lead, who is already writing it on page four.
Do not compare products. Decompose them, then check what the kernel and the fabric already allow.
Start with Calico’s four programs, because one of them is on the rail port.
1Calico is four programs, and the symptom tells you which one
Calico is easier to reason about once you stop treating it as one thing. Felix is the per-node agent that “Programs routes and ACLs, and anything else required on the host to provide desired connectivity for the endpoints on that host”, covering interface management, route programming, ACL programming and state reporting.[1] BIRD is the BGP client: it “gets routes from Felix and distributes to BGP peers on the network for inter-host routing”, and can act as a route reflector at scale.[1] confd “Monitors Calico datastore for changes to BGP configuration and global defaults such as AS number, logging levels, and IPAM information” and regenerates BIRD’s configuration from it.[1] Typha sits between the datastore and Felix, “reducing each node’s impact on the datastore” by holding a single connection, caching state and deduplicating events across many Felix instances; Calico quantifies only the fan-out, not a cluster size: “one Typha instance can support hundreds of Felix instances”, which is why the trigger to reach for it is datastore load, not a node count.[1] The datastore behind all of it is either the Kubernetes API datastore (kdd) or etcd.[1]
That decomposition is a triage table. Routes or ACLs wrong on one node: Felix. Routes missing at the leaf: BIRD. BGP configuration not reflecting a datastore change: confd. API server groaning under pod churn at scale: Typha.
The mode that matters most for an AI cluster is the one that turns half of this off. calico/node can run “in policy only mode where Felix runs without BIRD and confd”, selected with CALICO_NETWORKING_BACKEND=none.[1] Reach for it when another fabric owns routing - which is precisely the situation on a rail-optimised GPU cluster, where the east-west path is a separate network that Calico has no business advertising into.
2Cilium: two datapaths, and the flags that pick one
Cilium’s first design decision is whether pod traffic is wrapped or routed. Encapsulation is the default and comes in two flavours: “VXLAN (Default)” on 8472/UDP, and Geneve on 6081/UDP.[3] VXLAN costs a documented “50 bytes per network packet”, which the documentation itself suggests mitigating with jumbo frames.[3]
Native routing removes the wrapper and hands the job to the underlay. Its requirement is stated plainly: “the network connecting the cluster nodes must be capable of routing PodCIDRs”, either through a default route to an upstream router or by every node knowing every other node’s pod IPs.[3] You enable it with routing-mode: native plus ipv4-native-routing-cidr: x.x.x.x/y, and auto-direct-node-routes: true inserts the node routes automatically when all nodes share a single L2 domain.[3] That last condition is where designs break: a two-rack cluster with a routed spine cannot use the shortcut, so the leaves have to carry pod routes and somebody has to own that configuration.
The second decision is kube-proxy. --set kubeProxyReplacement=true makes Cilium handle Services itself, and Cilium “exits if kernel support missing” rather than falling back.[4] Two companions are mandatory - --set k8sServiceHost=${API_SERVER_IP} and --set k8sServicePort=${API_SERVER_PORT} (typically 6443) - because kube-proxy is what normally provisions the API-server Service, and without it Cilium cannot reach the API server to learn about Services in the first place.[4] Load balancing then offers SNAT (default), DSR, and Hybrid, described as “DSR for TCP, SNAT for UDP to avoid MTU adjustments”, selectable per Service with the service.cilium.io/forwarding-mode annotation.[4] Socket-level load balancing does its work at connect, sendmsg and recvmsg time by rewriting the destination before any packet exists, adding no lower-layer NAT at all.[4]
Diagram source (Mermaid)
flowchart TB
P[Pod eth0] --> Q{routing-mode}
Q -- tunnel default --> V[VXLAN 8472 or Geneve 6081]
V --> C1[50 bytes overhead per packet]
C1 --> U1[(Underlay needs only node reachability)]
Q -- native --> N[routing-mode native plus ipv4-native-routing-cidr]
N --> D{All nodes one L2 domain}
D -- yes --> A[auto-direct-node-routes true]
D -- no --> L[Leaves must route PodCIDRs]
A --> U2[(Underlay carries pod prefixes)]
L --> U23The kernel cliffs on a PowerEdge
Cilium’s best performance features are kernel-gated, and the numbers are worth memorising because they decide whether a design is deliverable on the OS a customer has already standardised on.
The netkit datapath needs kernel 6.8 or newer, installed as --set bpf.datapathMode=netkit --set routingMode=native --set bpf.masquerade=true --set kubeProxyReplacement=true.[5] eBPF host-routing is automatic when the kernel supports it and is enabled together with --set bpf.masquerade=true --set kubeProxyReplacement=true; it “Bypasses iptables and the upper host stack”.[5] IPv6 BIG TCP needs kernel 5.19 or newer and IPv4 BIG TCP needs 6.3 or newer, both with --set enableIPv6BIGTCP=true or --set enableIPv4BIGTCP=true alongside native routing, masquerade and kube-proxy replacement.[5]
The detail that makes this an NVIDIA conversation rather than a Linux one: the NIC list documented for BIG TCP is mlx4, mlx5 and ice.[5] mlx5 is the ConnectX and BlueField driver, so a Dell node with ConnectX-7 or a BlueField-3 SuperNIC in NIC mode is on the supported list.[5][7] And BIG TCP raises the GSO/GRO limits rather than the interface MTU - Cilium documents that no network interface MTU changes are required for it - which makes it the one throughput lever in this lesson that does not drag the whole MTU budget of lesson 3 behind it.[5]
| PowerEdge | BF-3 offering | DPU mode | NIC mode | Aux power | vSphere DSE | Notable KB |
|---|---|---|---|---|---|---|
R660 16G | ||||||
R760 16G | ||||||
R760XA 16G (GPU) | ||||||
XE9680 16G HGX H100/H200 | ||||||
XE9680L 16G liquid | ||||||
R7725 17G (AMD) | ||||||
R770 17G (Intel) | ||||||
XE9780 / XE9785 17G HGX B300 — Dell AI Factory |
✓ yes · ✗ no · ◐ conditional · ? unknown · — n/a. Click a cell for the evidence.
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
Where does Calico sit on the same question? It does not offer these knobs; its equivalent lever is the MTU arithmetic, and its own guidance is a single rule - “Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU)”.[2] Cilium’s answer to the same problem is to measure instead: the tuning page states that “Cilium will automatically detect the MTU of the underlying network devices”.[5] Note the honest gap here - Cilium’s dedicated MTU page was not reachable when these notes were verified, so the exact configuration key for overriding that detection is not something this course will name until it has been read on a live page.[5]
4Chaining, Multus, and the sentence the docs do not contain
Cilium can run as the second plugin in someone else’s chain. CNI chaining works “on top of any CNI plugin that is using a veth device model. The majority of CNI plugins use such a model.”[6] The generic-veth path is a ConfigMap holding a CNI conflist whose second plugin is cilium-cni with "chaining-mode": "generic-veth", referenced at install time with cni.chainingMode=generic-veth, cni.customConf=true and cni.configMap=cni-configuration.[6]
Chaining is not free. The documentation states that Layer 7 Policy and IPsec Transparent Encryption are constrained in this mode.[6] If a customer chained Cilium and never noticed losing L7 policy, that tells you the chaining was inherited from a platform installer rather than chosen.
Now the sentence that is not in the documentation: the chaining page says nothing about Multus.[6] Multus itself is explicit about what it is - “Multus CNI enables attaching multiple network interfaces to pods in Kubernetes”, following the Kubernetes Network Custom Resource Definition De-facto Standard - and it is agnostic about the primary CNI.[8] But agnostic is not the same as validated. Treat “Cilium plus Multus” as an integration the customer must test in their own environment, and say so in the design review rather than in the post-mortem.
Customer: eight GPU nodes across two racks, each rack its own L2 domain with a routed spine, kernel 6.5, ConnectX-7 east-west NICs, a separate rail fabric already carrying GPU traffic, and a requirement for NetworkPolicy on the management network. They are running Cilium because the platform installer chose it.
- Does the underlay route PodCIDRs? Two racks, routed spine, so nodes do not share one L2 domain.
auto-direct-node-routesis therefore unavailable - it only inserts routes when all nodes are in a single L2 domain.[3] Native routing is still possible, but only if the leaves are configured to route the pod prefixes.[3] - Is that configuration in scope? Assume the network team says no for this phase. Then keep the default: VXLAN encapsulation on 8472/UDP, and budget the documented 50 bytes per packet.[3]
- Which kernel features are reachable on 6.5? IPv6 BIG TCP needs 5.19 and IPv4 BIG TCP needs 6.3 - both available. netkit needs 6.8 - not available without an OS change.[5]
- Is the NIC on the BIG TCP list? ConnectX-7 uses mlx5, which is documented.[5] So enable
--set enableIPv4BIGTCP=truewith its required companions, and note that this needs no interface MTU change.[5] - kube-proxy? If they want kubeProxyReplacement, it must be set with
k8sServiceHostandk8sServicePortpointing at the API server on 6443, and it will exit rather than degrade if the kernel is short.[4] - What about the rail fabric? Nothing in the Cilium configuration should touch it. The check is that the Cilium-managed uplink is the management NIC, not the ConnectX-7 rail port - the same check as lesson 1.
- Deliverable to the customer: stay on VXLAN this phase; enable IPv4 BIG TCP on the GPU nodes; open 8472/UDP between racks; schedule native routing as a phase-two item that depends on the network team owning pod prefixes at the leaf.
Same customer, one year later: they have upgraded to kernel 6.9, the network team now advertises pod prefixes from the leaves, and a new team wants Multus for a second interface on the same nodes.
- Nodes share one L2 domain? ____ . So
auto-direct-node-routesis ____ . - Native routing now requires which two settings?
____and____.[3] - On kernel 6.9, is netkit available? ____ (floor is ____).[5]
- If they enable netkit, which four flags does the documented install line use? ____ , ____ , ____ , ____ .[5]
- What does Cilium’s documentation say about running underneath Multus? ____ .[6]
- If they instead chain Cilium with generic-veth, which two capabilities are documented as constrained? ____ and ____ .[6]
- One-sentence recommendation on the Multus question, phrased so it survives being quoted back to you in six months: ____ .
A Dell customer escalates: “since the Cilium upgrade our Multus secondary interfaces intermittently disappear and our L7 policies stopped matching”. They run RKE2, kernel 6.8, BlueField-3 SuperNICs in NIC mode, VXLAN encapsulation, kubeProxyReplacement enabled.
Produce: (a) the single hypothesis that explains both symptoms at once, and the exact configuration you would ask for to confirm it; (b) two other facts you would collect before accepting that hypothesis; (c) what you can and cannot promise about Cilium plus Multus, quoted from the documentation rather than from experience; (d) whether the BlueField-3 SuperNIC in NIC mode is inside NVIDIA’s supported configuration for this cluster, with the source.
Acceptance criteria: your hypothesis names generic-veth chaining and cites the constrained-capability list; your promise about Multus is an absence-of-documentation statement, not a support claim; and your hardware answer cites the platform-support page rather than a product page.[6][7]
Two facts instead of a product name
Netkit comes off the slide and the OS upgrade goes on the roadmap. Calico stays, and CALICO_NETWORKING_BACKEND=none goes into the plan: Felix without BIRD and confd, because the rail fabric already owns east-west routing and does not want Calico’s opinion in it.[1] On Multus you say the honest thing - the chaining page never names it, so it is a combination they test, not one you promise.[6] Then the network lead turns the notebook around. The leaf ports are 9216, he says, and page six is the screenshot that proves it. So why does ip link inside a pod say 1500?
Lab
Goal: confirm on the real ConnectX or BlueField-3 host that the NIC qualifies for BIG TCP, and measure the GSO/GRO change. This lab mutates a running Cilium install; every mutating step names its rollback.
- Pre-flight inventory. Record before changing anything:
Expected:uname -r ethtool -i <pf> | head -3 ip link show <pf> ethtool -k <pf> | grep -E 'gso|gro|tso'ethtool -ireportsdriver: mlx5_core. That is the qualifying fact - Cilium documents BIG TCP on mlx4, mlx5 and ice.[5] If the driver is anything else, stop: the rest of this lab does not apply. - Confirm the card is one Network Operator 26.7.0 supports, and in a supported mode - BlueField-3 is supported in NIC mode only.[7] Record the model and mode in your log.
- Record the current Cilium values so you can restore them exactly:
This file is your rollback. Do not proceed without it.helm get values cilium -n kube-system -o yaml > /tmp/cilium-values-before.yaml - Mutating step: enable IPv4 BIG TCP with its documented companions. Pre-condition, read first: these flags switch the cluster’s primary datapath to native routing, and native routing requires that “the network connecting the hosts on which Cilium is running on must be capable of forwarding IP traffic using addresses given to pods” - either the underlay already routes your PodCIDRs, or all nodes share one L2 domain and you add
--set autoDirectNodeRoutes=true.[3] If neither is true on this host, stop here and read step 5’s measurement on a node that already runs native routing; do not convert this cluster.
Expected: the DaemonSet rolls out and cross-node pod-to-pod still passes. If the rollout succeeds but the ping fails, the underlay is not routing PodCIDRs - roll back immediately rather than measuring on a broken datapath. No interface MTU change is required for BIG TCP.[5] Rollback:helm upgrade cilium cilium/cilium -n kube-system --reuse-values \ --set enableIPv4BIGTCP=true \ --set ipv4.enabled=true \ --set routingMode=native \ --set ipv4NativeRoutingCIDR=<cluster pod CIDR> \ --set bpf.masquerade=true \ --set kubeProxyReplacement=true kubectl -n kube-system rollout status ds/cilium --timeout=300s kubectl exec <pod-a> -- ping -c3 <pod-b IP> # pods on two different nodeshelm upgrade cilium cilium/cilium -n kube-system -f /tmp/cilium-values-before.yamlfollowed bykubectl -n kube-system rollout restart ds/cilium. - Measure the effect on the host interface:
Expected: the GSO/GRO maximum sizes are larger than the values you recorded in step 1. If they are unchanged, the kernel is below 6.3 for IPv4 - checkip -d link show <pf> | grep -E 'gso_max_size|gro_max_size' ethtool -k <pf> | grep -E 'gso|gro'uname -ragainst the floor before assuming a NIC problem.[5] - Confirm you did not disturb the fast path.
ip -br addr showandibstatshould be unchanged from step 1, and the Cilium-managed uplink should still be the management interface, not the RDMA PF. - Roll back with the command in step 4 whether or not the test succeeded, then re-run step 5 and confirm the original values returned. Leaving a lab host on a configuration you cannot explain later is how a lab stops being a reference.
Safety note: steps 4 and 7 restart the cluster’s CNI DaemonSet, which briefly interrupts pod networking on this host. Do not run this lab while anything else is using the cluster.
Goal: build both primary CNIs side by side and measure the differences you have just read about. No NIC required; everything runs in kind.
- Cluster A, Calico:
Expected:kind create cluster --name cni-calico --config - <<'EOF' kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true podSubnet: "192.168.0.0/16" EOF kubectl --context kind-cni-calico apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.0/manifests/calico.yamlcalico-nodereaches Ready. If not, read the first error inkubectl -n kube-system logs ds/calico-node -c calico-node. - Cluster B, Cilium with kube-proxy replaced:
Expected: install succeeds. The twokind create cluster --name cni-cilium --config - <<'EOF' kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true kubeProxyMode: none nodes: - role: control-plane - role: worker - role: worker EOF API=$(docker inspect -f '{{ .NetworkSettings.Networks.kind.IPAddress }}' cni-cilium-control-plane) helm repo add cilium https://helm.cilium.io/ && helm repo update helm install cilium cilium/cilium -n kube-system \ --set kubeProxyReplacement=true \ --set k8sServiceHost=$API \ --set k8sServicePort=6443k8sService*values are required precisely because kube-proxy normally provisions the API-server Service.[4] If Cilium exits at start, readkubectl -n kube-system logs ds/cilium- it hard-fails on missing kernel support rather than degrading.[4] - Diff the two dataplanes. On each cluster:
pluskubectl get nodes -o wide uname -r docker exec <node> iptables-save | wc -lkubectl -n kube-system exec ds/cilium -- cilium statuson B andkubectl -n kube-system exec ds/calico-node -- calico-node -show-statuson A. Expected: a large difference in the iptables rule count. Record both numbers; the ratio is the argument, not the absolute value. - Check your kernel against the documented floors: 6.8 for netkit, 6.3 for IPv4 BIG TCP, 5.19 for IPv6 BIG TCP.[5] Write down which of the three your
uname -rallows. - Now break it deliberately, on the multi-node cluster B. Native routing is configured with
routing-mode: nativeplusipv4-native-routing-cidr, andauto-direct-node-routesis what installs the PodCIDR routes between nodes - leave that last one off and nothing routes PodCIDRs.[3] Note the Helm values are camelCase even though the ConfigMap keys are hyphenated:
Then schedule a pod on each worker and ping across:helm upgrade cilium cilium/cilium -n kube-system --reuse-values \ --set routingMode=native \ --set ipv4NativeRoutingCIDR=10.244.0.0/16 \ --set autoDirectNodeRoutes=false kubectl -n kube-system rollout status ds/cilium --timeout=300s
Expected: same-node traffic still works, cross-node traffic fails - kind’s Docker network does not route PodCIDRs, and “the network connecting the hosts on which Cilium is running on must be capable of forwarding IP traffic using addresses given to pods”.[3] Capture the failure. Rollback:kubectl run a --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"cni-cilium-worker"}}' -- sleep 3600 kubectl run b --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"cni-cilium-worker2"}}' -- sleep 3600 kubectl exec a -- ping -c3 $(kubectl get pod b -o jsonpath='{.status.podIP}')helm rollback ciliumand confirm connectivity returns. - Deliverable: a table with one row per fact you measured - iptables rule count, kernel, which BIG TCP variants are reachable, and what native routing broke.
Rollback for the whole lab: kind delete cluster --name cni-calico && kind delete cluster --name cni-cilium.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in five sentences, how you would decide between Calico and Cilium for a customer's GPU cluster without arguing about which is better.
Sources
Facts in this lesson were checked against Cilium stable tuning, routing, kube-proxy-free and generic-veth-chaining pages, Calico latest architecture and MTU pages, and NVIDIA Network Operator v26.7.0 platform-support, all re-fetched 2026-09-07. Dates are when each page was fetched.
- Calico - Component architecture · fetched 2026-09-07
- Calico - Configure MTU to maximize network performance · fetched 2026-09-07
- Cilium - Routing (encapsulation and native routing) · fetched 2026-09-07
- Cilium - Kubernetes Without kube-proxy · fetched 2026-09-07
- Cilium - Tuning Guide · fetched 2026-09-07
- Cilium - Generic Veth Chaining · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Platform Support · fetched 2026-09-07
- GitHub - k8snetworkplumbingwg/multus-cni · fetched 2026-09-07
- Calico - About Kubernetes networking · fetched 2026-09-07
- Dell AI Factory with NVIDIA - 2-8-9-400 configuration solution brief · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.