NetQ 5.1: validation instead of inspection
S5·E2Two engineers on two leaves, both right · Bridge call, 07:00, the rest of Pod B still unexamined
Builds on: What Just Happened: naming the drop, Playbook: 'RoCE is slow'
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Run the right `netq check` object for a reported fault and map each failed test number to the switch-side command that confirms it.
- Explain what NetQ validates across a fabric that no single-switch `nv show` can answer.
- Determine whether a customer's NetQ deployment can monitor their fabric from version, agent and service state.
- Identify the two configurations NetQ cannot help with and say what to use instead.
Episode 2 — Two engineers on two leaves, both right
The bridge has been open since 05:00. The Dell SE is on it with a coffee that went cold before sunrise and the spreadsheet where every promise about Pod B goes, two columns, promised and demonstrated. “Fabric-wide RoCE health” is still in the first column. Two of the customer’s engineers are logged into two different leaves, reading nv show output at each other. Both switches look correct. Both engineers are right, and that is the problem: leaf07 was named at 04:00 and RoCE still drops for seconds during checkpoint writes.
RoCE correctness is not a property of a switch, it is a property of the path: classification, congestion control and flow control have to agree on every hop, and a per-switch read confirms one hop. That is the gap the validation tooling exists to close: netq check roce runs numbered tests across every device that reports in — mode, classification of DSCP, service pool, port group and traffic class, congestion control, flow control, ETS and miscellaneous — so the output is where the fabric disagrees with itself rather than what one leaf believes.[3]
The tool is built for that job: four flat families, check, show, config and trace, runnable from the standard prompt with no modes.[2] The network lead says the sentence she will repeat weekly all project — show me the counter — and one question decides whether any answer means anything: is every agent reporting.[2]
A per-switch read can only ever prove one hop; the fault lives between them.
Start with the shape of the tool.
1A flat CLI built around four verbs
NetQ’s command line is deliberately unlike a switch CLI: “you can run all commands from the standard command prompt instead of only in a specific mode, at the same level”.[2] There are four families — netq check <protocol-or-service>, netq show <protocol-or-service>, netq config <action> <object>, and netq trace <destination> from <source>.[2]
check is the one that changes how you work. Its objects are addresses, agents, bgp, clag, cl-version, evpn, interfaces, mlag, mtu, ntp, roce, sensors, topology, vlan, vxlan.[2] Each object runs a numbered set of tests across every device that reports in, so the output is not “here is the state of this switch” but “here is where the fabric disagrees with itself”.[3]
Agent lifecycle is the other half of config: netq config (add|del|show) agent and netq config (start|stop|status|restart) agent.[2] Almost every “NetQ shows nothing” call ends at one of those two commands.
Click a tool, or tab to it and press Enter. Dashed edge: WJH streamed through the NetQ agent instead of read locally.
One uplink runs hot while its siblings idle. The hash is not spreading the load.
Commit first: pick the surface you would reach for. The map stays quiet until you do.
Pick a symptom, then a tool. Every panel gives the exact command, the sampling or export constraint, the ASIC floor, and what the tool will not answer.
2What `netq check` actually tests
The RoCE tests are the ones a Spectrum-X FAE uses most:[3]
| # | Name | Checks |
|---|---|---|
| 0 | RoCE mode | whether RoCE is configured for lossy or lossless mode |
| 1 | RoCE classification | consistency of DSCP, service pool, port group and traffic class settings |
| 2 | RoCE congestion control | consistency of ECN and RED threshold settings |
| 3 | RoCE flow control | consistency of PFC configuration for RoCE lossless mode |
| 4 | RoCE ETS mode | consistency of Enhanced Transmission Selection settings |
| 5 | RoCE miscellaneous | consistency across related services |
There is a hard limit to record next to that table: NetQ supports RoCE lossless and lossy modes but does not support the single shared buffer mode lossless-single-ipool.[3] A customer running that mode gets no NetQ RoCE validation at all and has to fall back to per-switch nv show qos roce reads.[3][8]
EVPN has seven tests: 0 EVPN BGP session, 1 VNI type consistency (L2 and L3 uniform), 2 type 2 (IP-MAC binding and location consistency across VTEPs), 3 type 3 (replication group consistency), 4 session settings and FDB configuration, 5 VLAN consistency (VLAN-to-VNI mapping), 6 VRF consistency (VRF-to-L3 VNI mapping).[3] BGP tests cover session establishment, address families, router ID, hold time, keep-alive interval, IPv4 and IPv6 stale-path timers and interface MTU as test 7; interface tests include speed (2) and auto-negotiation (3); link-MTU tests are physical (0), SVI (1) and bridge (2).[3] That MTU appears three times over is not redundancy — MTU mismatch is a top cause and NetQ checks it from three angles.[3]
For RoCE specifically NetQ 5.1 also gives netq show roce-config, netq show roce-counters, netq show events message_type roceconfig, netq show events message_type tca_roce and netq show events-config message_type roceconfig.[4]
3The adaptive routing and ECMP views
NetQ 5.1 exposes ECMP with netq show ecmp and netq show ecmp-hash-config, and an Adaptive Routing dashboard showing devices with AR configured (enabled or disabled), the RoCE mode in use (lossy or lossless), interface-level AR configuration, and events including ECMP traffic imbalances.[5] The events tab displays up to 10 switches, sortable by highest P95, standard deviation or widest port deviation; queue lengths render as histograms exposing P95, standard deviation, mean and maximum.[5] The ECMP resource view on the full-screen switch card shows forwarding-resource utilization: maximum next-hop capacity, current usage and percentage consumed.[5]
The dashboard has four prerequisites: Spectrum-4 switches; Cumulus Linux 5.5.0 or later; RoCE lossless mode enabled; ASIC monitoring enabled via the asic-monitor service.[5] Two failure modes follow from that list and both look like data rather than errors. First, “switches without adaptive routing configuration won’t appear in UI/CLI results” — an empty dashboard is usually a configuration problem, not a NetQ problem.[5] Second, disabling asic-monitor makes the histogram metrics report zero, not “unavailable”.[5] A wall of zeroes reads as a quiet fabric and is really a dead measurement.
4NetQ 5.1 release facts and the version gap
NetQ 5.1 is available exclusively for on-premises deployments; you can upgrade to it from 5.0 or 4.15, and the 5.1 server works with agents 5.1 and 5.0.[1] Supported switch and host operating systems are Cumulus Linux 5.16, 5.15, 5.11.3 and 5.9.4 plus Ubuntu 24.04 and 22.04.[1] Grafana integration is generally available, OTLP metrics retention moved to 15 days instead of the previous three-day retention, installation is roughly 50% faster, threshold-crossing rules gained a wider range of measurement units, and NetQ no longer sends multiple successive events for breaches of the same threshold-crossing rule.[1] For WJH, L1 frame error events (symbol and CRC) were removed from the WJH dashboard; those counters live in the Link Health view.[1]
Now the gap you will be asked about. The Spectrum-X Validated Solution Stack row for RA v2.3.1 (September 2026) pairs NetQ 5.1.0 with Cumulus Linux 5.18.1.[7] The NetQ 5.1 supported-OS list stops at Cumulus Linux 5.16 — re-checked on 2026-09-07, and it still does.[1] Both statements are current NVIDIA pages. State both to the customer with their dates, and check the NetQ release notes for their exact build before you promise support for a 5.18.1 fabric.[1][7]
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
The last piece of deployment shape is gNMI. The NetQ gNMI agent is disabled by default: netq config add agent gnmi-enable true, default port 9339, then netq config restart agent.[9] You can run gNMI-only with netq config add agent opta-enable false, but both agents cannot be disabled simultaneously.[9] If NetQ telemetry is unreachable while both agents are active, the OpenConfig models openconfig-interfaces, openconfig-if-ethernet, openconfig-system and nvidia-if-ethernet-ext stop streaming to gNMI while WJH and platform data continue.[9]
A customer reports intermittent RoCE loss on an eight-leaf fabric. NetQ 5.1, agents 5.1, Cumulus 5.16.6.
- Establish that NetQ can see everyone before trusting any result:
netq check agents.[2] Expected: every switch reporting. A missing agent makes every later check a partial answer. netq check roce. Suppose tests 0, 1, 4 and 5 pass and test 3 fails on leaf05.[3]- Translate the test number: 3 is flow control — consistency of PFC configuration for lossless mode.[3]
- Go to leaf05 and read the value directly:
nv show interface <swp> qos roce statusfor the PFC priority and pool map, thennv show interface <swp> counters qos pfc-statsfor pause counts per switch priority.[8] Compare against a passing leaf. - Confirm the cross-fabric shape with
netq show roce-config, which shows the configuration NetQ collected from each device.[4] - Fix on leaf05 only, then re-run
netq check roceand expect a clean pass. Record both runs in the ticket — the before and after is the evidence. - If the customer’s mode is
lossless-single-ipool, stop at step 2: NetQ does not validate that mode and the whole workflow becomes per-switch reads.[3]
netq check ____— prove every agent reports before trusting a result.netq check rocefails test ____ on one leaf; that test is ____.- Switch-side confirmation:
nv show interface <swp> ____ ____ statusandnv show interface <swp> counters qos ____. - Cross-fabric view of collected config:
netq show ____. - The one RoCE mode that makes this whole workflow unavailable: ____.
You are given three outputs from a customer fabric: (a) netq check evpn failing test 6 on two leaves; (b) netq check roce failing test 1 on one leaf; (c) an Adaptive Routing dashboard where every histogram reads zero across all switches.
For each, write the failed-test meaning, the single switch-side command that confirms the cause, and the one-line fix. Then answer the harder question: which of the three is most likely not a fabric fault at all, and what would you check to prove it. Acceptance: your answer for (c) names asic-monitor and explains why zero is not the same as unavailable, and your answer for (a) names VRF-to-L3 VNI mapping rather than “EVPN is broken”.
What closed the bridge
netq check agents first, so the result is a whole fabric and not seven eighths of one.[2] Then netq check roce: test 3 fails on one leaf, which is flow control — PFC consistency for lossless mode — and the switch-side read on that leaf shows it.[3] One leaf drifted, not a fabric. And before you promise monitoring coverage, you check the NetQ release against the switch releases they actually run.[1][7]
Then the network lead turns to page eleven and reads out the complaint nobody has touched: collectives that stall for a few milliseconds, several times a run, on a fabric every dashboard calls comfortable. That one is not drift. That one is a sampling interval.
Lab
Pre-flight inventory: on each Dell-lab host record ethtool -i <ifname>, sudo mst start && flint -d /dev/mst/<dev> q, and the installed DOCA-Host version. Nothing in this lab changes state.
- Read-only. Produce the host half of the RoCE consistency picture that NetQ would compare across switches: the ToS or DSCP the host actually marks, from
cat /sys/class/infiniband/<dev>/tc/1/traffic_class, and the trust and PFC state frommlxreg -d /dev/mst/<dev> --reg_name QPTS --getand--reg_name PFCC --get.[8] - Run
ib_write_bwbetween two hosts and capture the receiver side withtcpdump -v -i <ifname> udp port 4791 -c 20to read the TOS octet on the wire. Record the DSCP that lands.[8] This is exactly what NetQ RoCE test 1 checks for consistency on the switch side.[3] - Write the mapping note: for each of RoCE tests 0-5, state whether the host can produce any evidence at all for it. Most cannot — the tests are about switch configuration — and being able to say that quickly on a call is the point.[3]
- Optional, customer lab with a live NetQ: run
netq check agents,netq check roce,netq check evpnandnetq show ecmp-hash-config, and reconcile each failure with the switch-side reads from the RoCE playbook lesson. Read-only; none of these commands changes configuration.[2][3][5] - Optional, same fabric:
netq show wjh-drop detailsfor the same window and check whether the drop reasons corroborate the failed validation tests.[6]
There is no NetQ server in this lab, so the work is reading and decision-making on real output shapes.
- Build the test-to-command table. For every RoCE test 0-5 and every EVPN test 0-6, write the switch-side command that confirms the same fact on one device.[3][8] Expected shape: 13 rows, each with test number, what it checks, and one
nv showorshowcommand. If you cannot name a command for a test, mark it and say why — test 5 (miscellaneous) is legitimately hard to pin to one command. - Build the decision table this module exists for: three columns — question, can a single-switch
nv showanswer it, can NetQ answer it. Rows: is PFC on SP3 on this leaf; is classification identical fabric-wide; is MTU consistent across a BGP session; is VLAN-to-VNI mapping consistent; what is the buffer occupancy on this port right now; which switches show ECMP imbalance.[3][5] - Write the two-line “NetQ will not help here” section:
lossless-single-ipoolgets no RoCE validation, and switches with no AR configuration never appear in the AR views.[3][5] - Write the pre-engagement questionnaire as a numbered list a Dell SE could send without you: NetQ version and deployment type; agent versions; each switch’s Cumulus release against the supported list;
asic-monitorstate on Spectrum-4 leaves; whether the local WJH service is disabled where the agent does WJH; whether RoCE mode is lossless.[1][5][6] - Verification of your own work: for each questionnaire line, cite which page the requirement comes from. A line you cannot cite is one you invented — delete it or verify it.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in four sentences, the difference between what NetQ tells them and what logging into a leaf and running `nv show` tells them.
Sources
Facts in this lesson were checked against NetQ 5.1 What's New re-fetched 2026-09-07 (supported-OS list still ends at Cumulus Linux 5.16); NetQ 5.1 validation, RoCE, ECMP/AR, WJH and gNMI pages and the Spectrum-X Validated Solution Stack — research fetched 2026-09-07. Dates are when each page was fetched.
- What's New | Cumulus NetQ 5.1 · fetched 2026-09-07
- NetQ Command Line Overview | Cumulus NetQ 5.1 · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- RoCE monitoring | Cumulus NetQ 5.1 · fetched 2026-09-07
- ECMP and Adaptive Routing monitoring | Cumulus NetQ 5.1 · fetched 2026-09-07
- Configure and Monitor What Just Happened | Cumulus NetQ 5.1 · fetched 2026-09-07
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- gNMI Streaming | Cumulus NetQ 5.1 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Playbook: 'RoCE is slow'Elsewhere in this course · Same ground: netq, cli and fae
- Spectrum-X architectureDOCA course · Same ground: netq, Adaptive routing and versions
- Scenario: 'the fabric is slow and adaptive routing does nothing'Elsewhere in this course · Same ground: netq, Adaptive routing and fae