NIC firmware config in Kubernetes: RoCE, PFC and reboots

Episode 3 — The change request that says no outage expected

The situation · Dell customer change-advisory call, Wednesday, for a Monday acceptance run

The change request is on the screen and the box marked service impact says none. The platform lead wrote it in good faith: everything else this quarter - the CNI, the NADs, the device plugin config - was a kubectl apply that took effect the moment a process re-read it. The Dell SE has already put “RoCE enabled, Thursday” in the spreadsheet, in the optimistic column, next to a coffee he has not touched since it was hot.

You have to say no, and the reason is where the settings live. numVfs, linkType, PFC and trust are NIC firmware configuration - nvconfig, the same store mlxconfig writes - so applying them is a reboot, not a reconcile.^[1] That is architecture, not caution: the number of virtual functions changes which PCI functions the card exposes, and the bus enumerates those at boot.

The NIC Configuration Operator does not pretend otherwise. It deploys a configuration daemon on each desired node and uses the Maintenance Operator to cordon and drain a node before making changes.^[1]^[3] So the answer is not whether there is downtime but how much - arithmetic you can do on this call from maxParallelOperations and maxUnavailable rather than discover during the window.^[3]

The night-shift operator, who will be the one in the cold aisle at 02:10 with the label maker, asks how many nodes go dark at once. It is the best question anyone has asked all week, because it is the only knob you have.

The question is never whether to reboot. It is how many at a time.

1Firmware is not CNI

Everything in module 1 and module 2 was software: a NAD, an annotation, a device plugin config. This lesson is about the settings that are not. numVfs, linkType, PFC and trust live in NIC firmware configuration - nvconfig, the same store mlxconfig writes - which means applying them is a reboot, not a reconcile.^[1] That single sentence changes the shape of the customer conversation from a change request to a maintenance window.

The NIC Configuration Operator “deploys a configuration daemon on each of the desired nodes to configure NVIDIA NICs there” and uses the Maintenance Operator to prepare nodes before making changes.^[1] You enable it from the nicConfigurationOperator section of NicClusterPolicy, alongside everything you built in lesson 1.^[10]

The CR you write is NicConfigurationTemplate in API group configuration.net.nvidia.com/v1alpha1. Its spec has four top-level fields: nodeSelector (empty means all matching nodes), nicSelector, resetToDefault (a bool that “Performs nvconfig reset and node reboot”), and template.^[2] nicSelector matches on nicType - a device ID in hex such as 101b, 101d or a2dc - and on pciAddresses.^[1]

Inside template, the ConfigurationTemplateSpec fields are numVfs, linkType (Ethernet or Infiniband, required unless networkBay is set), pciPerformanceOptimized, roceOptimized, gpuDirectOptimized, runtimePerformanceOptimized (ethtool-level ring buffers, channels and LRO), spectrumXOptimized (module 4), networkBay (ConnectX-9 only, nicType 1025), rawNvConfig and force, where force passes --force to mlxconfig.^[2] Note linkType in that list: switching a port between Ethernet and InfiniBand is a firmware change like any other, which is why an IB-to-RoCE migration is a reboot per node and not a config push.

2The RoCE block, and its switch-side twin

Here is the documented worked example, verbatim in shape:

spec:
  nicSelector:
    nicType: "101b"
    pciAddresses:
      - "0000:03:00.0"
  template:
    numVfs: 2
    linkType: Ethernet
    pciPerformanceOptimized:
      enabled: true
      maxReadRequest: 4096
    roceOptimized:
      enabled: true
      qos:
        trust: dscp
        pfc: "0,0,0,1,0,0,0,0"
    gpuDirectOptimized:
      enabled: true
      env: Baremetal
    rawNvConfig:
      - name: THIS_IS_A_SPECIAL_NVCONFIG_PARAM
        value: "55"

^[1] pciPerformanceOptimized carries enabled, maxReadRequest (PCI read request size in bytes) and maxAccOutRead, which the CRD reference marks “Deprecated: this field is ignored and no longer maps to MAX_ACC_OUT_READ”.^[2]^[6] A runbook that still sets maxAccOutRead is not broken, it is inert - which is worse, because it reads like it did something. roceOptimized carries enabled, a qos block with trust, pfc, tos, cableLen, ecn and pauseFrames, and roceMode where 1 is RoCE v1 and 2 is RoCE v2.^[2] gpuDirectOptimized carries enabled and env such as Baremetal.^[2]

Now read trust: dscp and pfc: "0,0,0,1,0,0,0,0" as one half of a policy. The bitmap enables PFC on exactly one priority - index 3 - and trust: dscp says the NIC derives that priority from the DSCP bits of the IP header rather than from an 802.1p tag. The switch must be told the same two things or the policy is a mismatch that only shows up under congestion. Dell names the same three mechanisms on the SONiC side of an AI fabric: RoCEv2 fully integrated, adaptive routing, and congestion management using ECN marking and PFC.^[7] One policy, two halves, and only the switch half runs on your bench at home.

Fabric / switch port MTU9216 B

1 warning

Predict: what pod MTU does this configuration need? (bytes, optional — saved)

Pod eth0

Pod eth0 — 9216 B

9216 − 0 = 9216. Pod MTU = network MTU. Nothing is subtracted because nothing is wrapped.

Calico: You set it. Calico does not measure the fabric for you.

Rule, verbatim: “Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU).”
Manifest installs change it through the calico-config ConfigMap key veth_mtu, then the DaemonSet is restarted.
Verify the tunnel with ip link show; the IP-in-IP tunnel appears as tunlx.
Policy-only mode (CALICO_NETWORKING_BACKEND=none) is the clean answer when a different fabric owns routing — then there is no tunnel to budget for.

Workload MTU and tunnel MTU are both 9216 — Calico wants them identical so every path has the same MTU.

⚠ Classic bug shape: a 9216 fabric with the pod network left at 1500. NCCL's TCP bootstrap still works and the data path crawls, so the ticket reads "training is slow", never "MTU is wrong".

Set it and verify it

kubectl patch configmap/calico-config -n kube-system --type merge -p '{"data":{"veth_mtu": "9216"}}'
kubectl rollout restart daemonset/calico-node -n kube-system   # the doc says: restart the DaemonSet
ip link show   # the tunnel device MTU must read 9216
kubectl exec -it <pod> -- ip link   # eth0 must read 9216, net1 must read 9216
ip link set <pf> mtu 9216 && kubectl exec -it <pod> -- ip link show net1

Calico MTU · SR-IOV VF · ipvlan · Spectrum-X verify · Network Operator · Calico architecture · Cilium routing

Set the fabric to 9216 and follow the number down to the pod interface on the SR-IOV path. Then switch the encapsulation and watch the primary-CNI number diverge from the secondary path - they are budgeted separately.

The MTU half of the same conversation is easy to get wrong because the two paths are budgeted independently. Calico subtracts per-encapsulation overhead from the workload MTU - 20 bytes for IP-in-IP, 50 for VXLAN over IPv4, 70 for VXLAN over IPv6, 60 and 80 for WireGuard over IPv4 and IPv6 - and expects the workload and tunnel MTUs to match.^[8] None of that arithmetic applies to the SR-IOV or host-device interface, which is not tunnelled at all; there the number that matters is the fabric MTU, which NVIDIA’s own Spectrum-X quick start sets to 9216.^[5] A pod with a correct eth0 MTU and a wrong net1 MTU connects, exchanges small messages, and stalls on large ones.

3Firmware images are a different workflow

Do not confuse firmware configuration with firmware images. Configuration is NicConfigurationTemplate. Images are two other objects. NicFirmwareSource takes binUrlSources, a list of URLs to signed firmware zips - the documented example points at a Mellanox download URL for a ConnectX-6 Dx image.^[1] NicFirmwareTemplate then takes nodeSelector, nicSelector.nicType, and a template carrying nicFirmwareSourceRef and updatePolicy: Update.^[1]

Two operational facts to carry into the customer call. First: “A configuration reset triggers a node reboot. Ensure that workloads are drained or that the Maintenance Operator is configured to handle the node maintenance automatically.”^[1] That is NVIDIA telling you that resetToDefault on a busy node without maintenance configured is an outage you caused. Second, and specific to the hardware Dell sells most of into AI: on DGX or HGX systems with an external BMC, configure FW_RESET_AFTER_CONFIG_UPDATE: "true" so the firmware is explicitly reset before the reboot, because OS-level reboots do not reload firmware on those platforms.^[1] Without it the change does not persist and the node can loop: apply, reboot, discover the config did not take, apply again.

One thing this lesson deliberately does not claim. A search snippet suggests the NIC Configuration Operator will move to being deployed through NicClusterPolicy because the Helm chart is to be deprecated in future releases. That sentence was not present in the release-notes body when this lesson was written, so treat it as unverified and do not put it in a customer deck.^[6]

4The Maintenance Operator handshake

The Maintenance Operator provides a “Kubernetes API(Custom Resource Definition) to allow node maintenance operators in K8s cluster in a coordinated manner”: it cordons, drains, and then marks the node ready.^[3] Two objects matter.

MaintenanceOperatorConfig carries logLevel, maxParallelOperations and maxUnavailable.^[3] Those two numbers are the entire wall-clock story of a rolling firmware change. Twelve nodes with maxParallelOperations: 4 and a ten-minute reboot cycle is three rounds; with maxParallelOperations: 1 it is twelve. Agree the number before the window, not during it.

NodeMaintenance carries nodeName, requestorID, cordon, waitForPodCompletion and drainSpec with force mode, pod selectors, timeouts and eviction filters.^[3] The flow is a handshake, and each step is a place it can stop: a requestor creates the NodeMaintenance; the operator cordons the node, waits for the named pods, drains according to drainSpec; it sets the condition Ready: True; the requestor then deletes the CR.^[3] A node stuck before Ready: True is usually a pod that will not evict - a PodDisruptionBudget, a bare pod with no controller, or a training job that waitForPodCompletion is patiently waiting on. That is not a firmware problem and no amount of looking at the NIC will reveal it.

In Spectrum-X installs the Maintenance Operator is turned on explicitly with --set maintenanceOperator.enabled=true.^[5]

5Reading the conditions

When a firmware change does not land, the device conditions tell you which of three questions to ask.^[4]

PartiallyApplied means “Some parameters unsupported” for that device ID or mode - a capability answer.^[4] You asked a ConnectX-6 for something a ConnectX-7 has, or asked for an Ethernet-only setting on a port in InfiniBand mode. The fix is the template content, matched to nicType.

IncorrectSpec and SpecValidationFailed mean the NicConfigurationTemplate itself is invalid - a validation answer, and nothing reached the device.^[4]

NonVolatileConfigUpdateFailed, RuntimeConfigUpdateFailed and FirmwareError mean the apply itself failed - an execution answer. Go to the nic-configuration-daemon logs on that node.^[4]

Underneath all of it is the same host-level checklist NVIDIA’s own collector runs: lsmod, ibstat, ibv_devinfo, mst status and dmesg.^[9] For firmware work mst status and dmesg are the two that earn their place - mst status says whether the management interface can see the card at all, and dmesg carries the PCIe and firmware errors that explain an execution failure.

Symptom

Firmware: Dell DUP vs NVIDIA mlxfwmanager / bf-fwbundle — which one wins?

Walk the firmware branch. At each fork, name the single command that would decide it before you click - then check whether the tree agrees.

FAE angle

The Dell shape of this call is a maintenance-window argument dressed as a technical question. The customer asks whether RoCE can be turned on without downtime; the answer is no, because trust and PFC are nvconfig and nvconfig needs a reboot.^[1] What you can control is how much downtime: maxParallelOperations and maxUnavailable on MaintenanceOperatorConfig turn a vague window into an arithmetic one.^[3] On XE9680-class nodes with an external BMC, ask whether FW_RESET_AFTER_CONFIG_UPDATE is set before the window opens - it is the difference between one reboot per node and a loop.^[1] And do the switch half in the same change: Dell’s own SONiC AI story is RoCEv2 with ECN marking and PFC, so a NIC configured for DSCP trust into a switch that was never told is a fabric that works until it is busy.^[7]

One safe firmware change, end to end

Goal: change pciPerformanceOptimized.maxReadRequest to 4096 on one ConnectX device in the Dell lab, with a rollback that does not depend on memory.

Record the current state first. This is the rollback.
```
mst start
mst status
mlxconfig -d /dev/mst/mt4125_pciconf0 q > ~/nvconfig-before-$(date +%F).txt
```
mst status is the check that the management interface can see the card at all.^[9] mlxconfig -d <dev> q queries the device’s supported configurations, and that capture is the rollback artifact - everything you might change must appear in the file before you proceed.^[12]
Identify the device precisely. Take nicType from the device ID and the PCI address from mst status, and use both in nicSelector so the template cannot match a NIC you did not intend.^[1]
Confirm maintenance is configured. MaintenanceOperatorConfig with maxParallelOperations: 1 for a single-node lab, and the Maintenance Operator enabled.^[3]^[5]

Write the minimal template - one field changed, nothing else:

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: lab-maxreadreq
spec:
  nodeSelector:
    kubernetes.io/hostname: bf3-lab-01
  nicSelector:
    nicType: "101d"
    pciAddresses:
      - "0000:03:00.0"
  template:
    linkType: Ethernet
    pciPerformanceOptimized:
      enabled: true
      maxReadRequest: 4096

^[2]^[1] Note what is absent: no resetToDefault, no numVfs, no roceOptimized. A minimal template is a minimal blast radius.

Apply and watch the device conditions move through UpdateStarted, PendingReboot and UpdateSuccessful.^[4] PendingReboot is the expected resting state before the node cycles - it is not a stall.
Let maintenance do the reboot. The NodeMaintenance object cordons and drains, the node reboots, and the condition settles after it returns.^[3]

Verify against the before-file:

mlxconfig -d /dev/mst/mt4125_pciconf0 q > ~/nvconfig-after.txt
diff ~/nvconfig-before-$(date +%F).txt ~/nvconfig-after.txt

Expected: exactly one line differs.

Rollback if needed: re-apply the template with the recorded original value, or use resetToDefault: true accepting that it resets everything and reboots.^[2]

After the window

How it ended

The change goes out with maxParallelOperations: 4, which turns twelve nodes and a twelve-minute reboot into three rounds rather than twelve.^[3] FW_RESET_AFTER_CONFIG_UPDATE is set first, because these are external-BMC platforms where an OS reboot does not reload firmware and the node can otherwise loop.^[1] Every node comes back with one line different from its before-capture, and the switch half - DSCP trust and PFC on the same priority - lands in the same window.^[7]

At 02:10 the operator calls anyway, and not about the fabric. The customer’s ML engineer has run lsmod on a GPU node, found no nvidia_peermem line, and filed GPUDirect as broken.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: apply exactly one firmware configuration change on the Dell BF-3 / ConnectX host and watch the conditions move. Mutating - this reboots the node. Do it in a window you own.

Pre-flight inventory. Do not skip any line; this block is your rollback:

mst start
mst status
mlxconfig -d <dev> q > ~/nvconfig-before-$(date +%F-%H%M).txt
uname -r
ip -br link show
kubectl get nicclusterpolicy nic-cluster-policy -o jsonpath='{.status.conditions}'
kubectl get nicdevice -A

Expected: mst status names the device and PCI address^[9], and mlxconfig -d <dev> q queries the device and prints the full current nvconfig into a file you keep.^[12] If mst status finds nothing, stop - nothing below will work and the problem is the host.

Confirm the NIC Configuration Operator is running, deployed through the nicConfigurationOperator section of your NicClusterPolicy, and that NICConfigurationOperatorReady is True.^[10] Rollback: remove the section from the policy.
Configure maintenance conservatively. MaintenanceOperatorConfig with maxParallelOperations: 1 and a maxUnavailable you are comfortable with on a single-node lab.^[3] Rollback: delete the config object.
Apply the minimal template from the worked example - pciPerformanceOptimized.maxReadRequest: 4096 and nothing else. Rollback: re-apply the template with the maxReadRequest value recorded in your before-file, and let the node reboot again in the same window. Do not use resetToDefault as your first rollback.^[2]
Watch the device conditions:
```
kubectl get nicdevice -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions}{"\n"}{end}'
```
Expected: UpdateStarted then PendingReboot. PendingReboot is a resting state, not a hang.^[4]
Watch the NodeMaintenance handshake. kubectl get nodemaintenance -A -o yaml - expect cordon, drain, then condition Ready: True.^[3] If it parks before Ready, list the pods that will not evict; that is your blocker, not the NIC.
After the reboot, confirm UpdateSuccessful and diff the nvconfig:
```
mlxconfig -d <dev> q > ~/nvconfig-after.txt
diff ~/nvconfig-before-*.txt ~/nvconfig-after.txt
```
Expected: exactly one line differs.^[12] More than one line means your template matched more than you intended - check nicSelector.^[1]
Do not set resetToDefault: true on this host as an experiment. It resets all nvconfig and reboots, and on an external-BMC platform without FW_RESET_AFTER_CONFIG_UPDATE set it can leave you cycling.^[1]^[2]
Collect the artifact: kubectl netop-sosreport --verbose --log-lines 10000 after the change, and keep it beside the before and after nvconfig captures.^[9]
Full rollback in order: re-apply the original maxReadRequest, allow the reboot, diff nvconfig against your before-file until it matches, then delete the NicConfigurationTemplate and the MaintenanceOperatorConfig.

Evidence / notes (commands run, outputs, what surprised you)

NIC firmware config in Kubernetes: RoCE, PFC and reboots

Before you read: what do you already know?

After this lesson you can