Streaming telemetry: OTLP, high-frequency counters, latency measurement
S5·E3The stall that never appears on the dashboard · Dell lab, Round Rock, Tuesday, three days before the acceptance re-test
Builds on: What Just Happened: naming the drop
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Configure OTLP export from a Cumulus Linux 5.18 leaf with the correct destination, transport security and VRF.
- Select stat groups and sample intervals for a stated question and name the prerequisite each group depends on.
- Schedule a high-frequency telemetry job within the documented ASIC, port and concurrency limits.
- Decide when latency measurement applies and identify the link types it cannot measure.
Episode 3 — The stall that never appears on the dashboard
The collector is up, the lab leaf is streaming, and every panel says the fabric is comfortable. The customer’s engineers say a collective stalls for milliseconds several times a run, with the job timings to show it. Someone says the fabric is fine and the problem is the hosts. On the call the network lead turns her notebook to page eleven: show me the counter.
Look at the sample interval before you accept that. Interface statistics default to one second, and so does the AI-Ethernet group.[1] A buffer that fills and drains inside two milliseconds is invisible to a once-a-second sample; the counter is not lying, it is never looking at the right moment.
That blind spot is the whole reason high-frequency telemetry exists: it collects counters at single-digit millisecond to microsecond intervals, with an interval knob valid from 100 to 12,750 microseconds in multiples of 50 and a default of 5000.[2]
It comes with a constraint that decides your Friday. HFT is Spectrum-4 and later, and it is not supported on ports using 8 lanes — on Spectrum-4, swp1 through swp64 use all 8 lanes, so those ports must be broken out first.[2] The uplinks in Pod B are native 800G, so a telemetry question just became a cabling question — and the procurement person, who only asks price and lead time, decides whether Friday happens.
You cannot sample your way to a burst shorter than your interval; here, visibility is a cabling decision.
Start with the always-on stream; it decides what the scalpel is compared against.
1OTLP export: the always-on stream
Cumulus Linux 5.18 exports telemetry over OpenTelemetry to a gRPC collector:
nv set system telemetry export otlp state enabled
nv set system telemetry export otlp grpc destination <IP/hostname>
nv set system telemetry export otlp grpc port <port>
nv set system telemetry export otlp grpc certificate <ca-certificate>
nv set system telemetry export otlp grpc insecure enabled
nv set system telemetry export vrf <vrf-name>[1] The default port is 8443. Secure mode requires a CA certificate, and insecure mode must be explicitly enabled if you provide none — the most common reason a first attempt sends nothing. Export uses the default VRF unless you override it, which matters when the collector is only reachable through the management VRF.[1] The feature requires a Spectrum-2 ASIC or later.[1]
What you get is organised into stat groups: acl-stats, ai-ethernet-stats, buffer-stats, control-plane-stats, dot1x-stats, histogram, interface-stats, lldp, platform-stats, router and software-stats.[1] Each has its own sample interval:[1]
| Statistic | Default | Range (seconds) |
|---|---|---|
| acl-stats | 5 s | 1-86400 |
| ai-ethernet-stats | 1 s | 1-86400 |
| control-plane-stats | 1 s | 1-86400 |
| dot1x-stats | 60 s | 1-86400 |
| interface-stats | 1 s | 1-86400 |
| interface debounce | 10 s | 10-86400 |
| lldp | 5 s | 1-86400 |
| platform-stats | 60 s | per-class adjustable |
| router | 30 s | multiples of 10, up to 60 |
| software-stats | 60 s | 60-86400 |
Click a tool, or tab to it and press Enter. Dashed edge: WJH streamed through the NetQ agent instead of read locally.
p99 latency spikes while average link utilization sits under 40%.
Commit first: pick the surface you would reach for. The map stays quiet until you do.
Pick a symptom, then a tool. Every panel gives the exact command, the sampling or export constraint, the ASIC floor, and what the tool will not answer.
2Choosing groups, scopes and labels
The Spectrum-X group is ai-ethernet-stats:
nv set system telemetry ai-ethernet-stats export state enabled
nv set system telemetry ai-ethernet-stats sample-interval <1-86400>It carries adaptive routing, SRv6 and packet-trimming data — and it “require[s] enabling adaptive routing, packet trimming, or SRv6 separately”. Turning on the stat group alone produces nothing.[1] That is the single most common empty-dashboard cause on a Spectrum-X pod, because adaptive routing is disabled by default and packet trimming is a separate feature with its own enable.[5][6]
Interface statistics can be scoped so you are not shipping every counter on every port:
nv set system telemetry interface-stats ingress-buffer priority-group <0-15>
nv set system telemetry interface-stats egress-buffer traffic-class <0-15>
nv set system telemetry interface-stats switch-priority <0-7>
nv set system telemetry interface-stats class phy state enabled
nv set system telemetry interface-stats class debounce sample-interval <10-86400>[1] Histogram export has its own state and a temporality of delta or cumulative.[1] Different collectors can receive different data through per-destination stat groups:
nv set system telemetry stats-group <group> <statistic> export state enabled
nv set system telemetry stats-group <group> <statistic> sample-interval <value>
nv set system telemetry export otlp grpc destination <IP> stats-group <group>[1] Granular metric selection went GA in 5.18.0 — metric-list, include-list and exclude-list, applied globally or per stats-group — which is how you cut volume without giving up a whole group.[1][4]
Labels are how the data becomes searchable on the collector: nv set system telemetry label "<name>" description "<text>" plus the per-interface form, with a maximum of 10 device labels and 10 interface labels.[1] Ten is enough for rack, pod, role, plane and fabric; it is not enough for a label per tenant, so decide the taxonomy before you deploy.
Not everything has to be sampled. Congestion notifications are event-driven:
nv set interface <iface> telemetry congestion-event egress-buffer traffic-class <tc-list> buffer-threshold <bytes>
nv set system telemetry congestion-event throttle-duration <milliseconds>
nv set system telemetry congestion-event export state enabled[1] On the receiving side, NetQ 5.1 keeps OTLP metrics for 15 days instead of the previous three-day retention, with Grafana integration generally available — worth quoting when a customer asks how long the data lives.[7]
3High-frequency telemetry: the microburst scalpel
HFT exists to “collect counters at very short sampling intervals (single digit milliseconds to microseconds)” — the bursts a one-second sample never sees.[2] The interval knob is nv set system telemetry hft sample-interval-usec, valid 100 to 12,750 microseconds in multiples of 50, default 5000.[2]
nv set system telemetry export otlp grpc destination 10.1.1.100
nv set system telemetry hft counter rx-byte
nv set interface swp1s0-3 telemetry hft state enabled
nv set system telemetry hft export state enabled
nv config apply[2] Counters include rx-byte, tx-byte, tc-occupancy, pg-occupancy, pg-watermark, if-in-discards, pause frames, tc-ecn-marked, sp-octets-rx and adaptive routing grades; the standard profile collects tx-byte, rx-byte and traffic-class buffer occupancy.[2][8] Data can stream out through OpenTelemetry export or IPFIX, or be collected locally as JSON under /var/run/cumulus/hft for later upload — the local form is what you use when the collector is not reachable during a maintenance window.[2] Scheduled collection looks like:
nv action schedule system telemetry hft job 2024-07-17 10:00:00 duration 30 profile profile1 ports swp1s0-swp9s0The limits are what make HFT an operational decision rather than a default:
- Spectrum-4 and later only.[2]
- No 8-lane ports. On Spectrum-4, swp1 through swp64 use all 8 lanes; to run HFT you must break those ports out.[2] An all-native-800G SN5600 therefore cannot run HFT on those ports as configured.
- A maximum of 25 scheduled sessions, and only one HFT job at a time on the switch ASIC.[2]
- A single gRPC destination for streamed HFT and a single IPFIX destination; a maximum of 4 custom profiles plus the default.[2]
- Time sync across switches is required — NTP works, PTP is recommended — or the timestamps from two leaves cannot be compared.[2]
4Latency measurement, and what it will not measure
Latency monitoring is a separate feature from both of the above, available on Spectrum-4 and later switches and on layer 3 and untagged physical links only.[3]
nv set system telemetry latency-measurement state enabled
nv set system telemetry latency-measurement sample-interval <seconds>
nv set interface <if> latency-measurement traffic-class <TC> protocol ipv4|ipv6 [dscp <v>]
nv set system telemetry latency-measurement export state enabled[3] The probe interval has a floor that scales with how many traffic classes a port measures: 1-4 traffic classes at least 1 s, 5-8 at least 2 s, 9-12 at least 3 s, 13-16 at least 4 s.[3]
The unsupported list is the part that decides whether the feature answers the customer’s question at all: no leaf-to-leaf or spine-to-spine measurement, no GPU or server-facing interfaces, no link-local IPv6, no one-time or on-demand measurement, and no one-way latency.[3] A customer asking “what is my GPU-to-GPU latency” cannot get it from this feature; what they get is per-link latency on measurable links, which they then have to compose. Say that plainly rather than letting a proof of concept discover it.
Requirement: a Spectrum-4 leaf in a Spectrum-X pod, adaptive routing enabled, one collector at 10.1.1.100 reachable in the default VRF, no PKI yet, and a standing question about buffer occupancy on the four uplinks.
-
Enable the transport and be explicit about security:
nv set system telemetry export otlp state enabled nv set system telemetry export otlp grpc destination 10.1.1.100 nv set system telemetry export otlp grpc insecure enabledPort defaults to 8443 and the default VRF is used unless overridden.[1] The insecure line is required because no CA certificate is provided; write it in the change record as a temporary state with a date.[1]
-
Turn on the feature before the group. Adaptive routing is disabled by default, and
ai-ethernet-statsemits nothing without it:nv set router adaptive-routing state enabledplus the per-interface enables.[5][1] -
Enable the groups you can justify:
nv set system telemetry ai-ethernet-stats export state enabled nv set system telemetry ai-ethernet-stats sample-interval 1 nv set system telemetry interface-stats egress-buffer traffic-class 3Traffic class 3 is where RoCE lives on a default lossless configuration — switch priority 3 maps to traffic class 3 — so scoping the buffer view there is the honest answer to the standing question.[9]
-
Label the device so the collector can group it:
nv set system telemetry label "rack" description "r12"and the per-interface form for the uplinks; you have 10 of each.[1] -
Add event-driven congestion notification on the uplinks rather than lowering every interval:
[1]nv set interface swp49-52 telemetry congestion-event egress-buffer traffic-class 3 buffer-threshold 1000000 nv set system telemetry congestion-event export state enabled -
nv config apply, then verify withnv show system telemetry export,nv show system telemetry export otlp grpc destinationandnv show system telemetry interface-stats.[1] -
Do not enable HFT as part of this baseline: it is one job per ASIC and the uplinks are 8-lane native ports on this box, so it needs breakout first.[2]
- Transport:
nv set system telemetry export otlp ____ enabled,... grpc destination ____, and because there is no CA certificate,... grpc ____ enabled. Default port is ____. - Before the AI-Ethernet group can emit anything, enable ____ (or packet trimming or SRv6).
- Scope the buffer view:
nv set system telemetry interface-stats egress-buffer traffic-class ____— the class RoCE uses by default. - Labels: maximum ____ device labels and ____ interface labels.
- Event-driven alternative to a faster sample:
nv set interface <if> telemetry ____ egress-buffer traffic-class 3 buffer-threshold <bytes>. - Why HFT is not in this baseline: ____ per ASIC, and 8-lane ports need ____.
New requirement on the same leaf: the customer reports a collective that stalls for a few milliseconds at a time and wants proof it is a microburst rather than a host problem. You have a two-hour window, one collector, and the uplinks are native 800G.
Write the plan: what has to change physically before HFT can run, which counters you would collect and at what sample-interval-usec, how you would schedule it, how the data leaves the switch given a single gRPC destination is already in use by the standing OTLP stream, and what you would compare it against to say “burst” rather than “host”. Acceptance: your plan names the 8-lane restriction and the one-job-per-ASIC limit, chooses an interval inside 100-12,750 in multiples of 50, and includes at least one measurement taken on the hosts.
What went in Tuesday's write-up
One uplink broken out, one scheduled job, an interval well inside the legal range, and buffer occupancy sampled at a timescale the stall can actually appear in.[2] The one-second stream stays where it was, as the background record the burst is measured against.[1] The night operator labels the broken-out uplink HFT — DO NOT PATCH, then labels the patch panel too, because the tape is cheap and he is thorough.
What you say to the customer: the stall is a microburst, and only one HFT job runs on the ASIC at a time, so agree now who owns that slot.[2] Then the network lead points out what you have really proved — one leaf, correct. Friday needs thirty-two of them identical, and nobody is logging into thirty-two boxes by Thursday.
Lab
Pre-flight inventory on the Dell-lab BlueField-3/ConnectX hosts: ethtool -i <ifname>, sudo mst start && flint -d /dev/mst/<dev> q, and the installed DOCA-Host version (dpkg -l 'doca*' | grep '^ii' or rpm -qa 'doca*'). Record all three before touching anything; the DOCA version must match the RA row you are working to.
-
Read-only. Determine whether the installed DOCA-Host provides DOCA Telemetry Service on this host at all: list the installed DOCA packages and check for a telemetry service component. If it is not present, stop and record that — do not install packages to make a lab step work on a shared lab host.
-
If DTS is available, point it at the same collector from the no-hardware lab and confirm metrics arrive in the debug exporter. Compare the metric set with the switch stat groups: which of
interface-stats,buffer-statsandai-ethernet-statshave any host-side analogue, and which are switch-only.[1] -
Read-only host burst evidence for the comparison exercise: run
ib_write_bwmany-to-one and sampleethtool -S <ifname>every 100 ms into a file, then plot or diff consecutive samples. Write one sentence on what a 100 ms host sample can and cannot resolve compared with a 5000 microsecond switch sample.[2] -
Optional, customer or partner lab with a Spectrum-4 leaf — mutating. Before any change:
nv config saveandnv config show -o commands > telemetry-pre.confoff-box. Enable HFT on a breakout port only:nv set system telemetry hft sample-interval-usec 1000 nv set interface swp1s0-3 telemetry hft state enabled nv set system telemetry hft export state enabled nv config apply --confirm 5Confirm inside the window only if the leaf is healthy.[2] Rollback:
nv unset system telemetry hftandnv unset interface swp1s0-3 telemetry hftthennv config apply, or restore withnv config replace telemetry-pre.conffollowed bynv config apply. -
Optional, same leaf: schedule a local JSON collection instead of streaming if the collector is not reachable —
nv action schedule system telemetry hft job <date> <time> duration 30 profile <profile> ports <ports>and collect the files from/var/run/cumulus/hft.[2] Rollback: none needed for a scheduled job that has completed; delete the collected files after you copy them off. -
Write the closing note an FAE keeps: why the same job cannot run on a native 800G 8-lane port, and who owns the single ASIC HFT slot on that switch.[2]
Goal: make the collector side real, then write and syntax-check the switch side.
-
In
~/containerlab, run an OpenTelemetry collector so the receiving end is genuinely working. Minimalotel-config.yaml:receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:8443 exporters: debug: verbosity: detailed service: pipelines: metrics: receivers: [otlp] exporters: [debug]Run it as a container in your lab topology and confirm the port is listening:
ss -ltnp | grep 8443. Expected: the collector process bound to 8443, which is the Cumulus default OTLP port.[1] If not: check the container published the port and that nothing else holds 8443. -
Prove the pipeline with a synthetic exporter (any OTLP metrics generator, or a second collector configured to forward) so you know the receiver works before you blame a switch. Expected: metric batches in the debug exporter output.
-
Write the switch configuration for a Spectrum-4 leaf that streams
ai-ethernet-statsand traffic-class 3 egress-buffer stats to that collector, with an include-list, two device labels and one interface label. State in a comment line which prerequisite feature must be enabled for the AI-Ethernet group to emit anything.[1][5] -
In DSX Air, paste the same commands onto a Cumulus node and
nv config apply. Expected: the commands parse andnv show system telemetry exportreflects them. Record explicitly that this validates syntax only — the ASIC counters behind these groups are not produced in simulation.[1] -
HFT arithmetic, no hardware needed: for sample intervals of 100, 125, 5000 and 12,800 microseconds, mark each legal or illegal and say why (multiples of 50, range 100-12,750).[2] Then, for an SN5600 with all native 800G ports, write the sentence explaining what has to change before any HFT job can run.[2]
-
Latency measurement worksheet: for ports carrying 3, 6 and 14 traffic classes, give the minimum probe interval, and list the five link, path or mode types the feature cannot measure.[3]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell platform engineer, in four sentences, why you would run both a one-second OTLP stream and a five-millisecond HFT job on the same leaf instead of picking one.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 High Frequency Telemetry page re-fetched 2026-09-07 (adds IPFIX export, /var/run/cumulus/hft, 4 custom profiles); OTLP Export, Latency Monitoring, RDMA over Converged Ethernet (RoCE) and What's New pages — research fetched 2026-09-07. Dates are when each page was fetched.
- Open Telemetry Export | Cumulus Linux 5.18 · fetched 2026-09-07
- High Frequency Telemetry | Cumulus Linux 5.18 · fetched 2026-09-07
- Latency Monitoring | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- Equal Cost Multipath Load Sharing (adaptive routing) | Cumulus Linux 5.18 · fetched 2026-09-07
- Packet Trimming | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus NetQ 5.1 · fetched 2026-09-07
- High Frequency Telemetry | Cumulus Linux 5.16 · fetched 2026-09-07
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Observability: DTS, BlueMan, FireflyDOCA course · Same ground: hft, config and telemetry
- The triage toolkit: resources, optics and what tcpdump will not tell youElsewhere in this course · Same ground: otlp, hft and telemetry
- Spectrum-X architectureDOCA course · Same ground: congestion, Adaptive routing and versions