The triage ladder: sos-report, error strings and the checklists

1Collect before you change

The Network Operator ships its own collector. The script is kubectl-netop_sosreport, invoked as kubectl netop-sosreport [OPTIONS], or standalone as ./network-operator-sosreport.sh [OPTIONS].^[1] It requires kubectl in PATH, a valid kubeconfig, cluster-admin recommended, Bash 4.0+, Python 3.6+ for the HTML report, plus tar, gzip and sha256sum.^[1]

What it gathers is the argument for running it first. It collects every Network Operator custom resource and CRD; operator deployments, pods, ConfigMaps, secret metadata, RBAC, events and webhook configurations; DaemonSet and Deployment specs, pod details, current and previous container logs, related ConfigMaps and Services; and node details with labels, annotations, conditions, status and allocatable resources.^[1] The phrase that decides your first move is previous container logs. Restart the driver DaemonSet to “try something” and you have deleted the evidence of what it did last time.

Two documented invocations cover the field cases: --skip-diagnostics --log-lines 1000 when you need it fast, and --verbose --log-lines 10000 when you are preparing a support case.^[1]

A documentation note worth carrying, because a stale bookmark wastes time: the 26.7.0 troubleshooting index resolves to the SOS-Report page, and the per-symptom material — troubleshooting pod failures, quick health checks, preparing for a support case — appears as example workflows inside that page, not as separate URLs.^[2][1] Guessed per-symptom slugs return 404.

2The host ladder: five commands, stop at the first failure

The collector runs its diagnostics inside the OFED driver pod on each node: lsmod | grep mlx, ibstat, ibv_devinfo, mst status, uname -r and dmesg (last 200 lines), plus ip link / ip addr.^[1] That list is also the manual checklist, with one caveat — run by hand on the host, the middle three need OFED and MFT userspace installed there; inside the driver pod they always work. Run them in that order and stop at the first one that fails, because every later rung assumes the earlier ones passed.

Only after all five pass does the problem become a Kubernetes problem — and then it is one of a short list: the device plugin never registered, the NAD or annotation or namespace is wrong, the resource-name prefix does not match, the RDMA netns mode is wrong for the deployed design, or the MTU disagrees somewhere.^[1][7][10]

On the Kubernetes side there is a rung above all of them that costs one command: the Network Operator acts only on a NicClusterPolicy named exactly nic-cluster-policy, and an instance with any other name is silently ignored.^[13] “We installed the operator and nothing happened” is that sentence more often than it is anything else.

3Error strings, memorised

Some of this job is genuinely recall. These strings appear verbatim in logs and each maps to exactly one cause.

Exiting.. can not change : incorrect RDMA subsystem network namespace — the RDMA shared device plugin started while the node’s RDMA subsystem is in exclusive mode; the pod rdma-shared-dp-ds sits in CrashLoopBackOff.^[3] Prove it with rdma system show, which displays the netns mode.^[4] This is not a bug to work around: exclusive netns is required for per-pod RDMA isolation with SR-IOV plus rdma-cni, and it breaks the shared plugin, so the two models cannot run on one node.^[12][3]

EBUSY from rdma system set netns exclusive — “if there are active network namespaces and if one or more RDMA devices exist, changing mode from shared to exclusive returns error code EBUSY.”^[4] The mode must be set before namespaces exist. The man page also advises against changing the subsystem mode while RDMA traffic is running, even though it is supported.^[4]

spectrum-x config version <name> not found — the profile ConfigMap is missing, lacks the watch label, or the name does not match spectrumXOptimized.version character for character.^[5]

TopologyAffinityError — Topology Manager under restricted or single-numa-node could not satisfy alignment, so the pod is rejected and is not automatically retried onto a better node.^[6]

nicdevice condition reasons — PartiallyApplied means some parameters were unsupported for that device and the message lists them; IncorrectSpec and SpecValidationFailed mean the NicConfigurationTemplate is invalid for that device; NonVolatileConfigUpdateFailed, RuntimeConfigUpdateFailed and FirmwareError mean the apply itself failed and the next stop is the nic-configuration-daemon log for that node.^[5]

rdma system show
rdma system set netns exclusive
rdma system set netns shared
echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf

4The two checklists that catch the rest

The resource-name checklist. One name has to agree in three places: what the device plugin advertises, what the NAD references, and what the pod requests. Four different defaults collide there — intel.com is the sriov-network-device-plugin’s default resourcePrefix, which is why NVIDIA’s own configs always set "resourcePrefix": "nvidia.com" explicitly; the RDMA shared device plugin defaults to rdma, producing names like rdma/rdma_shared_device_a; and OpenShift writes openshift.io.^[7][8][9] A pod requesting the wrong prefix stays Pending forever with no CNI error at all, because scheduling never gets far enough to call CNI.^[7]

# ConfigMap sriovdp-config — no resourcePrefix key
{ "resourceList": [{
    "resourceName": "hostdev",
    "selectors": { "vendors": ["15b3"], "isRdma": true }
}]}

The NAD checklist. Five items, all checkable in two minutes.^[10][9][12]

1. Is the NAD in the pod’s namespace? On OpenShift, Multus is pre-configured with namespace isolation enabled by default, so a NAD in namespace A cannot be referenced from namespace B.^[9] Namespace-qualify as testns1/macvlan-conf-3 or with the JSON "namespace" key.^[10]
2. Is the annotation key exactly k8s.v1.cni.cncf.io/networks?^[10]
3. Is spec.config valid JSON? A NAD may instead have no spec and be backed by a file in /etc/cni/multus/net.d/ on the node — a half-migrated cluster has both and neither works.^[10]
4. Does the NAD’s resourceName match an advertised resource, prefix included?^[7]
5. For chained RDMA, does {"type":"rdma"} come after the sriov or ib-sriov plugin in the plugins array?^[12]

The MTU checklist. Three MTUs must agree: switch port, host PF, pod interface. Calico’s own rule is to “set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU)”, subtracting 20 for IP-in-IP, 50 for VXLAN IPv4, 70 for VXLAN IPv6, 60 for WireGuard IPv4 and 80 for WireGuard IPv6.^[11] A mismatch on the secondary interface produces RDMA that connects and then stalls on large messages — which reads like a fabric problem and is reported as “training is slow”.^[11]

5Running the ladder under pressure