MTU budgeting across switch, host PF and pod
S1·E3Three numbers and one screenshot · The customer's data hall, day twelve, second day of a throughput escalation
Builds on: Primary CNI choices: Calico and Cilium
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Compute the correct Calico pod MTU for a given fabric MTU and encapsulation using the documented subtraction table.
- Apply the veth_mtu patch and verify the result on both the tunnel device and inside a pod.
- Determine the maximum MTU a macvlan or ipvlan secondary interface can carry on a given host PF.
- Diagnose an RDMA path that connects and then stalls on large messages as an MTU disagreement rather than a fabric fault.
Episode 3 — Three numbers and one screenshot
The switch team got there first with their evidence: every rail port at 9216, screenshot attached, ticket returned to the compute side. The night-shift operator has already labelled the port in his usual tape, and the label is right. The run connects, bootstraps, then crawls. No error anywhere, which is why this is day two of the escalation and day twelve of thirty.
Three numbers decide the path and only the smallest one matters: switch port, host PF, pod interface. The fabric answer covers exactly one of them. Calico expects you to compute the pod value yourself and publishes the arithmetic - 1450 for VXLAN over IPv4 on a 1500 network, 8950 on a 9000 network, a flat 50 bytes either way.[1] Cilium, the option this account turned down, measures instead: it “will automatically detect the MTU of the underlying network devices”, which means it faithfully detects whatever the host was left at.[2] Neither of them raises a host interface for you.
MTU budgeting exists because the failure is silent by construction. Small packets fit under any ceiling, so ping, curl, the TCP handshake and the collective’s own bootstrap all pass; only the bulk phase builds frames large enough to meet the smallest MTU on the path. On the secondary interface the same mismatch reads as RDMA that connects and then stalls on large messages - which is how it gets filed against a switch whose rails NVIDIA’s own configuration puts at 9216 anyway.[6]
Ask for three numbers, not one.
Start with the rule; the arithmetic serves it.
1Three numbers, and only the smallest one matters
The rule to carry into every AI cluster conversation is short: three MTUs must agree - switch port, host PF, pod interface. Everything in this lesson is arithmetic in service of that sentence.
The reason it is worth a whole lesson is the failure shape. A mismatch does not produce an error. It produces a path that comes up, passes a ping, completes a TCP handshake, runs NCCL’s bootstrap successfully, and then crawls on the data phase - because small messages fit under any MTU and only the large ones hit the ceiling. On the secondary interface the same mismatch produces RDMA that connects and then stalls on large messages, which reads like a fabric problem and gets escalated to the switch team.[6]
Two vendors handle this in opposite ways, and knowing which one you are looking at determines your first question. Calico expects you to compute the number and set it, using a documented per-encapsulation subtraction.[1] Cilium expects to measure it: “Cilium will automatically detect the MTU of the underlying network devices”, so you configure the host for jumbo frames and it follows.[2] On Calico the bug is usually a number nobody updated; on Cilium it is usually a host interface nobody raised.
2Calico: the subtraction table, and the two commands that prove it
Calico publishes the arithmetic directly. On a network MTU of 1500 the recommended values are 1500 with no encapsulation, 1480 for IP-in-IP, 1450 for VXLAN over IPv4, 1430 for VXLAN over IPv6, 1440 for WireGuard over IPv4 and 1420 for WireGuard over IPv6; on a 9000-byte network the same row reads 9000, 8980, 8950, 8930, 8940 and 8920.[1] Read the deltas out of it and you have the whole table in six numbers: 0, 20, 50, 70, 60, 80.[1]
The rule that goes with the table is one sentence and it is where people go wrong: “Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU)”.[1] Setting the pod MTU correctly and leaving the tunnel at its old value satisfies neither path.
On a manifest install the change is a ConfigMap patch:
kubectl patch configmap/calico-config -n kube-system --type merge \
-p '{"data":{"veth_mtu": "1440"}}'[1] Then restart the DaemonSet, because the ConfigMap alone changes nothing that is already running.[1] Verification is two commands and both are required: ip link show on the host, where the IP-in-IP tunnel appears as tunlx with its MTU, and ip link inside a newly created pod.[1] Existing pods keep the MTU they were created with, which is why re-testing with the pod you already had open is the most common way to conclude wrongly that the patch did not work.
One caution for anyone carrying an old runbook: FelixConfiguration.mtuIfacePattern is not on the current Calico MTU page.[1] It may exist elsewhere in the product, but this course will not teach it as the path, and neither should you quote it to a customer without reading it on a live page first. The documented path is the veth_mtu key.[1]
Pod eth0 — 8950 B
9000 − 50 = 8950. network MTU − 50. Cilium documents the same 50 bytes per packet for VXLAN, which is why jumbo frames pay for themselves on either CNI.
Calico: You set it. Calico does not measure the fabric for you.
- Rule, verbatim: “Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU).”
- Manifest installs change it through the calico-config ConfigMap key veth_mtu, then the DaemonSet is restarted.
- Verify the tunnel with ip link show; the IP-in-IP tunnel appears as tunlx.
- Policy-only mode (CALICO_NETWORKING_BACKEND=none) is the clean answer when a different fabric owns routing — then there is no tunnel to budget for.
Workload MTU and tunnel MTU are both 8950 — Calico wants them identical so every path has the same MTU.
- ⚠ macvlan mtu has range 0 to the master's MTU. A NAD asking mtu: 9216 on a 9000-MTU master is out of range — the pod does not get a bigger MTU than the PF, ever.
- ⚠ Classic bug shape: a 9000 fabric with the pod network left at 1500. NCCL's TCP bootstrap still works and the data path crawls, so the ticket reads "training is slow", never "MTU is wrong".
- Pod eth0 (8950) and pod net1 (9000) differ by 50 bytes. That is correct — the secondary path skips the tunnel — but net1 must still equal the PF and the switch port.
kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "8950"}}'
kubectl rollout restart daemonset/calico-node -n kube-system # the doc says: restart the DaemonSet
ip link show # the tunnel device MTU must read 8950
kubectl exec -it <pod> -- ip link # eth0 must read 8950, net1 must read 9000
# NAD delegate: {"type": "macvlan", "master": "<pf>", "mtu": 9000, "ipam": {...}}3Cilium: measure instead of compute
Cilium’s position is that you should not be doing this arithmetic at all. The tuning page states that “Cilium will automatically detect the MTU of the underlying network devices”, so the operational task moves to the host: raise the PF, and the pod interfaces follow.[2]
That does not make the overhead disappear. VXLAN is the default encapsulation and costs a documented “50 bytes per network packet”, and the documentation’s own suggested mitigation is jumbo frames.[3] The arithmetic is the same; who performs it is different.
The Cilium-specific thing worth knowing is what BIG TCP does not require. Cilium documents that no network interface MTU changes are needed for it - it raises the GSO and GRO limits instead of the frame size.[2] That makes BIG TCP the only throughput lever in this lesson that is a per-node decision rather than a fabric-wide project, and it is available on mlx4, mlx5 and ice NICs, with kernel floors of 6.3 for IPv4 and 5.19 for IPv6.[2]
Be honest about the boundary of what has been verified here. Cilium’s dedicated MTU page was not reachable when this lesson was written, so the exact configuration key for overriding the auto-detected value is not stated in this course - what is cited is the auto-detection behaviour and the 50-byte VXLAN figure, both read on live pages.[2][3] If a customer needs to pin the value rather than let it be detected, read the current page with them rather than quoting a key from memory.
4The caps on the secondary path
The fast path has its own ceilings, and they are lower than people expect.
macvlan takes an mtu field whose documented range is 0 to the master’s MTU.[4] You cannot lift a pod above the host interface it is enslaved to - so a 9216 pod on a 9000 PF is not a configuration you can request, it is a host change you make first. NVIDIA’s own MacvlanNetwork CR carries the same mtu field alongside master, mode and ipam, and the deployment guide’s example sets it to 1500, which is a default rather than a recommendation.[7]
ipvlan has no mtu promise to give you: the field “Defaults to the value chosen by the kernel”, which in practice means it inherits the master.[5] Two other ipvlan facts belong in the same breath because they surprise people mid-design: all ipvlan interfaces share the host interface’s MAC, and “ipvlan does not allow virtual interfaces to communicate with the master interface. Therefore the container will not be able to reach the host via ipvlan interface.”[5]
The rail case sets the top of the budget. NVIDIA’s own Spectrum-X quick start configures rails at 9216.[6] That is the number to hold a customer’s leaf configuration against when they say the fabric is jumbo-enabled - and the number your host PF and pod interface both have to reach before any of it means anything.
Customer: Calico with VXLAN over IPv4, fabric configured at 9000, host PFs currently at 1500, a macvlan secondary network planned on the same PF.
- Fabric MTU: 9000, confirmed on the switch port, not from a design document.
- Host PF: must be raised to 9000.
ip link set <pf> mtu 9000. Until this is done nothing downstream can exceed 1500. - Calico pod MTU: 9000 minus 50 for VXLAN over IPv4 = 8950, which matches the published table’s 9000-row entry.[1]
- Apply it:
kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "8950"}}', then restart thecalico-nodeDaemonSet.[1] - Tunnel MTU: must equal the workload MTU, so also 8950 - Calico’s rule is that they are the same value.[1]
- Verify, twice:
ip link showon the host and confirmtunl0reads 8950; then create a new pod and runip linkinside it.[1] - Secondary path: the macvlan NAD may request at most the master’s MTU, so with the PF at 9000 the ceiling is
mtu: 9000.[4] Note that the secondary interface is not encapsulated, so it does not pay the 50 bytes - the 8950 figure applies to the primary path only. - Result: switch 9000, PF 9000, pod eth0 8950 (Calico, VXLAN), pod net1 9000 (macvlan). Every number is derived, none is copied.
Same customer moves the primary CNI to WireGuard over IPv6 for a compliance requirement, and raises the fabric to 9216 to match a new Spectrum-X rail.
- Fabric MTU = ____ , which matches NVIDIA’s documented rail value of ____ .[6]
- Host PF must be set to ____ .
- Calico subtraction for WireGuard over IPv6 = ____ bytes.[1]
- Calico pod MTU = ____ minus ____ = ____ .
- Command:
kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "____"}}'followed by ____ .[1] - Tunnel MTU must be ____ , because Calico’s rule is ____ .[1]
- Maximum
mtuthe macvlan NAD may now request = ____ .[4] - How much did the compliance requirement cost the primary path, in bytes per packet, compared with VXLAN over IPv4? ____ .[1]
A customer reports: “our Spectrum-X fabric is at 9216 and NCCL still gets nowhere near line rate. The switch team confirmed all ports are 9216.” They run Cilium, VXLAN encapsulation, and a macvlan secondary network for the fast path on PFs that were installed six months before the fabric was upgraded.
Produce: (a) the three artifacts you request, in order, with the exact command for each; (b) for each artifact, the value that would clear it and the value that would convict it; (c) which of the two paths - primary or secondary - you expect to be the actual problem here, and why the switch team’s evidence does not settle it; (d) the one host-level change that has to happen before any CNI-level change can help, and what it breaks while it is being applied.
Acceptance criteria: your answer uses Cilium’s auto-detection behaviour rather than a Calico subtraction, names the macvlan master cap as the binding constraint on the secondary path, and treats “all ports are 9216” as evidence about one of three numbers rather than all three.[2][4][6]
What you write in the ticket
Three commands settle it: ip link inside a pod, ip link on the host PF, and the veth_mtu value in calico-config.[1] The fabric is 9216, the PF is 1500 because those cards were installed before the fabric upgrade, and the pod inherited it. The macvlan secondary could never have been raised past the master anyway - the plugin documents mtu as a range of 0 to the master’s MTU.[4] So you raise the PF first, patch the ConfigMap, restart the DaemonSet and create a new pod - which is when the network lead notices what the new pod does not have. Only lo and eth0, against a NetworkAttachmentDefinition everyone swears is correct.
Lab
Goal: prove the macvlan master cap on a real ConnectX or BlueField-3 PF. Mutating: it changes a host interface MTU and creates NADs; every step names its rollback.
- Pre-flight inventory. Record every one of these values before changing anything - this list is your rollback target:
Expected: the PF’s current MTU is written down,ip link show <pf> ethtool -i <pf> | head -3 ibstat | head -20 kubectl get net-attach-def -Aethtool -ireportsmlx5_core, andibstatshows the HCA. Ifibstatis empty, stop - the driver is not loaded and nothing below will be meaningful. - Mutating step: raise the PF to 9000.
Expected: the interface reportsip link set <pf> mtu 9000 ip link show <pf>mtu 9000. Rollback:ip link set <pf> mtu 1500, or whatever value you recorded in step 1. Warning: this briefly disturbs traffic on that interface. Do not run it on a PF carrying anything you care about. - Create a macvlan NAD whose
masteris that PF and whosemtuis 9000 - at the master’s ceiling, which is the top of the documented range.[4]
Rollback:kubectl apply -f - <<'EOF' apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: { name: macvlan-9000 } spec: config: '{"cniVersion":"0.3.1","type":"macvlan","master":"ens1f0","mode":"bridge","mtu":9000,"ipam":{"type":"host-local","subnet":"10.56.217.0/24"}}' EOFkubectl delete net-attach-def macvlan-9000. - Create a second NAD identical except
"mtu":9216- above the master. Rollback:kubectl delete net-attach-def macvlan-9216. - Launch one pod against each NAD with the
k8s.v1.cni.cncf.io/networksannotation and compare:
Expected: the 9000 pod’skubectl exec <pod-9000> -- ip link show net1 kubectl exec <pod-9216> -- ip link show net1net1carries 9000. The 9216 case is the one to document carefully - record verbatim whether the pod fails to start, whether the interface appears clamped to the master’s value, and the exactkubectl describe podevent text. The documented rule is that the value must be within 0 to the master’s MTU; capture what your kernel and plugin version actually do with a request outside it rather than assuming.[4] - Confirm the fast path is still healthy after the MTU change:
ibstatandibv_devinfoshould match step 1, andip -br addr showshould show no address changes. - Roll everything back in reverse order - delete both pods, delete both NADs, return the PF to its recorded MTU - and re-run step 1 to confirm the host matches the inventory you started from.
Deliverable: the verbatim event text from step 5, added to your symptom-to-cause deck. “macvlan mtu above master” is a card you will use.
Goal: find your own MTU cliff experimentally in containerlab, then apply and verify the Calico patch in kind. Mutating: it changes interface MTUs and a ConfigMap; both roll back.
- Bring up two Dell Enterprise SONiC 4.5.1 leaves with two Linux hosts in containerlab. The kind is
dell_sonic, default credentialsadmin:admin, andeth1maps toEthernet0with subsequent ports incrementing by four.[9] Expected:containerlab inspectshows all four nodes running; the SONiC nodes take about two minutes to boot.[9] If a SONiC node stays unhealthy, give it the full two minutes before restarting it. - Set a 9216 port MTU on the two leaf ports that face the hosts, using the SONiC CLI. Record the pre-change value first. Rollback: set the same ports back to the recorded value.
- Raise both Linux hosts’ data interfaces to 9216 with
ip link set eth1 mtu 9216. Rollback:ip link set eth1 mtu 1500. - Walk the MTU down until fragmentation is refused:
Expected: with everything at 9216 all three succeed. Now lower one leaf port to 9000 and repeat: the 9188 case fails with “Message too long” or “Frag needed”, the other two still pass. That single asymmetric result is the entire lesson - one number out of three, and only large packets notice.ping -M do -s 9188 -c 2 <peer> ping -M do -s 8972 -c 2 <peer> ping -M do -s 1472 -c 2 <peer> - Switch to a kind cluster with Calico. Read the current value, patch it, restart, verify:
[1] Rollback: patch the key back to the value you read in the first line, then restart the DaemonSet again.kubectl -n kube-system get cm calico-config -o jsonpath='{.data.veth_mtu}'; echo kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "1440"}}' kubectl -n kube-system rollout restart ds/calico-node kubectl -n kube-system rollout status ds/calico-node --timeout=180s - Verify on both sides, and note the trap:
Expected:docker exec <kind-node> ip link show tunl0 kubectl run mtu-probe --image=nicolaka/netshoot --restart=Never -- sleep 600 kubectl exec mtu-probe -- ip link show eth0tunl0and the new pod both read 1440.[1] Now check a pod that existed before the patch: it still has the old MTU. Write that down - it is why “the patch did not work” tickets exist. - Deliverable: a three-column table - switch port, host interface, pod interface - filled in for each stage of the lab, with the largest successful
ping -M do -sbeside each row.
Rollback for the whole lab: restore the SONiC port MTUs, ip link set eth1 mtu 1500 on both hosts, restore veth_mtu, delete the probe pods, destroy the containerlab topology and the kind cluster.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, why a jumbo-frame fabric can still deliver 1500-byte pod networking, and what you would ask for to prove it in one round trip.
Sources
Facts in this lesson were checked against Calico latest MTU page and Cilium stable tuning page re-fetched 2026-09-07; cni.dev macvlan and ipvlan pages re-fetched 2026-09-07; NVIDIA Network Operator v26.7.0 Spectrum-X quick start. Dates are when each page was fetched.
- Calico - Configure MTU to maximize network performance · fetched 2026-09-07
- Cilium - Tuning Guide · fetched 2026-09-07
- Cilium - Routing (encapsulation and native routing) · fetched 2026-09-07
- CNI plugins - macvlan · fetched 2026-09-07
- CNI plugins - ipvlan · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Spectrum-X Quick Start · fetched 2026-09-07
- NVIDIA Network Operator v26.7.0 - Deployment Guide with Kubernetes · fetched 2026-09-07
- Calico - Component architecture · fetched 2026-09-07
- Containerlab - Dell Enterprise SONiC (dell_sonic kind) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- BGP-EVPN symmetric IRB for tenant isolationSpectrum-X course · Same ground: vxlan, mtu and diagnose
- Network Operator and the NicClusterPolicyElsewhere in this course · Same ground: macvlan, misconception and triage
- Spectrum-X rails inside KubernetesElsewhere in this course · Same ground: mtu, triage and verify