What Just Happened: naming the drop
S5·E1Three nights of drops nobody can name · Customer data hall, 02:10, night three of a stalled acceptance run
Builds on: Playbook: 'RoCE is slow'
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish the WJH service from the WJH agent and name which one a given deployment must disable.
- Configure a WJH channel with a trigger, a drop filter and an aggregation window using the 5.18 NVUE surface.
- Read `nv-wjh-cli` and `netq show wjh-drop` output and classify each event by category, drop-reason id range and severity.
- Diagnose the four documented reasons a WJH deployment reports nothing.
Episode 1 — Three nights of drops nobody can name
The night-shift operator calls at 02:10 because he has run out of things to label. A training job on Pod B has stalled three nights running, always in the same phase, and Friday is the acceptance re-test that releases the rest of the customer’s order. He has a terminal open on leaf07 with tcpdump running on the port the rank-0 host is cabled to, and it shows nothing at all. Interface discards on that port climb through every run. The customer’s network lead is on the call too, notebook open at page three, and what she wants is not a theory.
You say the quiet thing first, because it is why three nights were wasted: on a Cumulus switch tcpdump captures only control-plane traffic to and from the CPU, not the data plane, so a forwarding drop leaves no trace in a capture.[7] WJH, SPAN and telemetry are the data-plane answers.[7]
That gap is the reason the feature exists. A discard counter counts; it cannot explain. What Just Happened is the ASIC reporting why it discarded a frame — a service that diagnoses drops locally across layer 1, layer 2, layer 3, tunnels, buffers and ACLs, plus an agent that streams the same contextual telemetry off-box to NetQ.[1]
The operator’s runbook says what-just-happened poll. On this 5.18 box the unit is nv-wjh.service and the CLI is nv-wjh-cli.[1]
A counter tells you that something died; only WJH tells you what killed it.
Start with which of those two components this switch is actually running.
1Two components with one name
What Just Happened is not one thing. Cumulus Linux ships a WJH agent that streams detailed contextual telemetry off the switch to NetQ, and a WJH service that diagnoses dropped packets locally across layer 1, layer 2, layer 3, tunnels, buffers and ACLs.[1] Which one a site runs decides every troubleshooting step afterwards, because the two cannot both own the drop-capture path.[1]
The reason this matters more on a switch than on a server: tcpdump on a Cumulus switch captures only control-plane traffic to and from the CPU, not the data plane.[7] A server engineer’s reflex — capture the interface and look — returns nothing about a forwarding drop. WJH, SPAN and telemetry are the data-plane answers.[7]
Names changed under people. On Cumulus Linux 5.18 and later the systemd unit is nv-wjh.service and the local CLI is nv-wjh-cli; older runbooks and knowledge-base articles say what-just-happened poll, which does not exist on a 5.18 box.[1] Cumulus Linux 5.18.0 also moved WJH filtering and threshold controls into NVUE and made WJH and gNMI subscriptions able to run at the same time, which they previously could not.[4][6]
Click a tool, or tab to it and press Enter. Dashed edge: WJH streamed through the NetQ agent instead of read locally.
An application reports loss. Interface discard counters move, but nothing says why.
Commit first: pick the surface you would reach for. The map stays quiet until you do.
Pick a symptom, then a tool. Every panel gives the exact command, the sampling or export constraint, the ASIC floor, and what the tool will not answer.
2Configuring a channel: trigger, filter, aggregation
A channel is the unit of configuration. You create one and attach a trigger:
nv set system wjh channel <name> trigger <type>
nv config applyTrigger types are l1, l2, l3, tunnel, buffer and acl.[1] Two constraints bite immediately: channel names must be 4 to 16 characters, and each trigger can be enabled on only one channel — so you cannot have a buffer channel per tenant.[1]
Filtering narrows what a channel records:
nv set system wjh channel <id> drop-filter <filter-id> drop-type <type> drop-reason <reason>
nv set system wjh channel <id> drop-filter <filter-id> drop-type <type> severity <level>
nv set system wjh channel <id> drop-filter <filter-id> ip <address>
nv unset system wjh channel <id> drop-filter <filter-id>
nv show system wjh channel <id> drop-filter[1] Aggregation controls how much the switch summarizes before you read it: nv set system wjh channel <id> polling-interval <5-300> in seconds and nv set system wjh channel <id> aggregate-cache-size <500-5000>.[1] On a busy AI fabric the cache size is the knob that decides whether a burst of identical drops evicts everything else.
Buffer channels can carry thresholds so the channel reports congestion and latency events rather than only discards:
nv set system wjh channel <id> buffer-threshold latency tc <class> interface <ports> high <ms>
nv set system wjh channel <id> buffer-threshold congestion tc <class> interface <ports> high <percent>
nv show system wjh channel <id> buffer-threshold[1] Spectrum-6 does not support latency thresholds — on an SN6000-class leaf the congestion form is the one that works.[1]
3Reading it: on-box, and through NetQ
On the switch the three display buffers map to the three questions you ask:
nv show system wjh packet-buffer # individual drops
nv show system wjh aggregate-buffer # aggregated events
nv show system wjh l1-buffer # layer 1 events
nv show system wjh channel # all channels
nv action export system wjh packet-buffer [no-metadata] # to PCAP[1] The Linux form is nv-wjh-cli poll --data <drops|aggregates|l1> --output-type <table|json|pcap> [--channels <list>] [--no-metadata], plus nv-wjh-cli poll dump.[1] PCAP files land in /var/run/nv-wjh/ and diagnostics dumps in /var/log/nv-wjh/diags-dump/<timestamp>.[1]
The categories are what make the output worth reading. L1: link training failure, port state changes, symbol and CRC errors. L2: MLAG isolation, VLAN mismatch, spanning-tree filtering, loopback detection, multicast source MAC. L3/Router: non-routable packets, blackhole routes, unresolved neighbors, loopback and multicast IPs, TTL exhaustion, MTU violations. Tunnel: overlay MAC issues, decapsulation errors, encapsulation isolation. Buffer: tail drop, WRED, congestion thresholds, latency thresholds. ACL: ingress and egress port or router ACL denials.[1]
Through NetQ the same data arrives centrally. Enable it with sudo netq config add agent wjh then sudo netq config restart agent; WJH itself “is enabled by default on NVIDIA Spectrum switches running Cumulus Linux 4.4.0 or later”.[2] Read it with netq [<hostname>] show wjh-drop [severity <sev>] [details] [between <t1> and <t2>] [around <t>] [json], and filter the detailed per-type form by ingress-port, severity, reason, src-ip, dst-ip, proto, src-port, dst-port, src-mac, dst-mac, egress-port, traffic-class, rule-id-acl or vlan.[2] The aggregate output is a six-row table you should be able to read at a glance:
Drop type Aggregate Count
L1 560
Buffer 224
Router 144
L2 0
ACL 0
Tunnel 0[2] With details the reasons appear: WRED, Blackhole route, Packet Latency Threshold Crossed, Port TC Congestion Threshold, Oper down.[2]
When WJH is streamed over gNMI the reasons arrive as numeric ids, and the ranges are the fastest way to classify a stream you did not generate: L1 10021-10031, L2 201-210, Router 301-327, Tunnel 402-404, ACL 601-604, Buffer 503-506, each carrying a severity of Notice, Warning or Error.[3] The models are nvidia-if-wjh-drop-aggregate for aggregated drops and nvidia-if-ethernet-ext for PFC, WRED, ACL and buffer drop counters; the reason mapping file on the switch is /usr/etc/wjh_lib_conf.xml.[3]
4Why WJH reports nothing
Four documented causes, in the order they are worth checking.
-
Wrong name. On 5.18 the unit is
nv-wjh.serviceand the CLI isnv-wjh-cli. A runbook that callswhat-just-happened pollfails, and a monitoring check that watches the old unit reports the service down forever.[1] -
The NetQ agent owns WJH. When the NVIDIA NetQ agent is used for WJH, the local WJH service must be disabled:
nv set system wjh state disabled sudo systemctl disable nv-wjh && sudo systemctl stop nv-wjh -
ASIC floor. Buffer packet-drop monitoring requires Spectrum-2 or later, and WJH uses a SPAN destination, so it cannot exceed the switch ASIC’s SPAN limits.[1] On a box already using its SPAN resources for a mirror session, a buffer channel is competing for them.
-
The panel moved. NetQ 5.1 removed L1 frame error events (symbol and CRC) from the WJH dashboard; the Link Health view retains those counters.[5] An operator watching the old panel after an upgrade sees a regression that is a relocation.[5]
A leaf running Cumulus Linux 5.18.1 with two channels — l3drops (trigger l3) and bufdrops (trigger buffer) — during a training-job complaint.
- Confirm the channels exist and the triggers are where you think:
nv show system wjh channel. Expected: two rows, each with one trigger, no trigger listed twice — a trigger can be on only one channel.[1] - Pull aggregates as JSON so you can filter them:
nv-wjh-cli poll --data aggregates --output-type json --channels bufdrops.[1] - Classify by category first, not by header. Buffer reasons are ids 503-506; Router reasons are 301-327.[3]
- Suppose the top three rows are: Buffer / WRED / 18,400 events on swp17; Buffer / Port TC Congestion Threshold / 900 on swp17; Router / Blackhole route / 12 on swp3.[2]
- Root cause lines: WRED on one uplink at that volume is ECN doing its job or a threshold set too low — cross-check
nv show interface swp17 counters qos egress-queue-statsfor WRED drops on the RoCE traffic class before you touch anything. The congestion-threshold events are the same event seen through the threshold you configured. The dozen blackhole-route events are unrelated and belong in a different ticket.[1][2] - If you need packets rather than counts, narrow first with a drop filter, then
nv action export system wjh packet-bufferand open the PCAP from/var/run/nv-wjh/.[1]
nv show system wjh ____— confirm each trigger appears on exactly ____ channel.nv-wjh-cli poll --data ____ --output-type json --channels bufdrops.- Reason ids: Buffer ____ to ____, Router ____ to ____, L1 ____ to ____.
- A high WRED count on one uplink means check ____ next, on traffic class ____.
- To get packets: set a ____ on the channel, then
nv action ____ system wjh packet-buffer, and read the file from ____.
You are handed a netq show wjh-drop details capture from a customer fabric. It contains 560 L1 events on leaf03 swp9, 224 Buffer events spread over four leaves, and 144 Router events on one leaf with reason Blackhole route. The customer has NetQ 5.1 and reports that “L1 errors vanished from the WJH dashboard after the upgrade”.
Produce: (a) three separate one-line root-cause statements, one per category, each naming the next command you would run; (b) the sentence you send the customer about the missing L1 panel; and (c) which of the three findings, if any, is a switch configuration problem rather than a fabric fault. Acceptance: your L1 answer names a physical suspect and the Link Health view, your Buffer answer distinguishes threshold events from tail drops, and your Router answer does not blame the fabric.
04:00 — the drop has a name
By 04:00 leaf07 carries two channels — one on l3, one on buffer — staged together and applied once.[1] The aggregate view names the fault: WRED on a single uplink, the congestion-threshold events you configured beside it, and a dozen blackhole-route entries that belong to a different ticket entirely.[2] The L1 errors that vanished from the customer’s NetQ dashboard at 5.1 were relocated into the Link Health view, not lost.[5]
What you say to the operator, who is already printing a label for that uplink: your discards were never mysterious, they were unnamed. Then the network lead writes leaf07 on page three and asks the question that ends the night — if this leaf was wrong, how do you know the other thirty-one are right?
Lab
Pre-flight inventory on the Dell lab hosts before anything else: ip -br link, ethtool -i <ifname> for each ConnectX/BlueField-3 port, and sudo mst start && flint -d /dev/mst/<dev> q for firmware. Save the output; it is the baseline for every later step.
-
Read-only, host side. Collect the drop evidence a host can produce:
ethtool -S <ifname> | grep -Ei 'discard|error|drop|pause'andnetstat -s. Expected: counters that go up during a load test.[7] -
Run a load that produces discards (many-to-one
ib_write_bwfrom two senders into one receiver), then re-read the same counters. Record the deltas. -
Write the comparison that is the point of this lab: for each host counter, state the question it answers and the question it cannot. A host discard counter gives a port and a count; WJH gives the reason and the ingress port on the switch. Neither replaces the other, and on a switch
tcpdumpgives you neither because it sees only control-plane traffic to and from the CPU.[7] -
Optional, customer or partner lab with a Spectrum-2 or later leaf — mutating. Before the change:
nv config saveandnv config show -o commands > wjh-pre.confoff-box.[8] Then:nv set system wjh channel bufdrops trigger buffer nv set system wjh channel bufdrops polling-interval 30 nv config apply --confirm 5Confirm within the window if the box is healthy; if it is not, do nothing and the switch rolls back automatically.[8] Rollback:
nv unset system wjh channel bufdropsthennv config apply, or restore the saved command file withnv config replace wjh-pre.conffollowed bynv config apply.[1][8] -
Optional, same leaf: generate congestion, then
nv show system wjh aggregate-bufferandnv action export system wjh packet-buffer, and copy the PCAP from/var/run/nv-wjh/to your laptop to open locally.[1] Rollback: as step 4; the export itself changes no configuration. -
If the leaf runs a NetQ agent, do not enable the local service at all — check first with
netq config show agentand, if the agent does WJH, leavenv set system wjh state disabledin place and read the data withnetq show wjh-drop detailsinstead.[1][2]
Goal: build the channel configuration in simulation, then do the analysis work on real output shapes.
-
In DSX Air (https://dsx-air.nvidia.com) start a simulation with at least one Cumulus leaf. Confirm the release first:
nv show system version. Expected: a 5.x string; record it, because the WJH command surface in this lesson is the 5.18 one.[1] -
Create two channels:
nv set system wjh channel l3drops trigger l3 nv set system wjh channel bufdrops trigger buffer nv set system wjh channel bufdrops polling-interval 30 nv set system wjh channel bufdrops aggregate-cache-size 2000 nv config applyExpected: apply succeeds and
nv show system wjh channellists both.[1] If the apply is rejected: check the channel name length (4-16 characters) and that no other channel already holds the same trigger.[1] -
Try to break the rule on purpose:
nv set system wjh channel l3copy trigger l3thennv config apply. Expected: a rejection or a configuration that leaves only one channel holdingl3. Record the exact message — it is the one a customer will paste into a ticket.[1] -
Add a drop filter and read it back:
nv set system wjh channel bufdrops drop-filter f1 drop-type buffer severity error nv config apply nv show system wjh channel bufdrops drop-filter[1] Expected:
nv show system wjh channel bufdrops drop-filterlists filterf1with drop-typebufferand severityerror.[1] If the filter is not listed: you read before the apply — runnv config applyand read again. If the apply is rejected: check the drop-type value against the documented set —acl,buffer,l2,l3ortunnel— and that the channel already exists with the trigger you expect.[1] Record what the filter does to the output before you move on: the documented purpose of a drop filter is to prevent matching drops from being monitored, so aseverity errorfilter removes error-severity events from the channel rather than isolating them.[1] -
Now read the buffers:
nv show system wjh aggregate-bufferandnv-wjh-cli poll --data aggregates --output-type json. Expected in simulation: the command surface works and the result is empty or synthetic. Write one sentence in your notes stating plainly that the ASIC drop reasons are not produced in simulation, so this step validates syntax only.[1] -
Analysis exercise, no simulator needed: take the aggregate shape from this lesson (L1 560, Buffer 224, Router 144, L2 0, ACL 0, Tunnel 0) plus the reason list (WRED, Blackhole route, Packet Latency Threshold Crossed, Port TC Congestion Threshold, Oper down) and classify each reason by category and by the gNMI id range it would occupy.[2][3] Deliverable: a table of reason, category, id range, severity you would expect, and the next command.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, what WJH tells them that an interface discard counter does not — and what has to be true on the switch for it to say anything at all.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 WJH, What's New and New/Removed NVUE Commands pages; NetQ 5.1 WJH and gNMI pages — research fetched 2026-09-07, NetQ 5.1 What's New re-fetched 2026-09-07. Dates are when each page was fetched.
- What Just Happened (WJH) | Cumulus Linux 5.18 · fetched 2026-09-07
- Configure and Monitor What Just Happened | Cumulus NetQ 5.1 · fetched 2026-09-07
- gNMI Streaming | Cumulus NetQ 5.1 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus NetQ 5.1 · fetched 2026-09-07
- New and Removed NVUE Commands | Cumulus Linux 5.18 · fetched 2026-09-07
- Network Troubleshooting | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE CLI | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.