Skip to content
nodeza.learn
DOCA course/5. Services/Lesson 5.2

Observability: DTS, BlueMan, Firefly

S5·E2Twenty-four cards that never answered · A bridge call at 01:20, wave one live in Hall 2, twenty-two days to move-in

S5·E2Apply~35 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs (DTS, BlueMan, Firefly service guides), 2026-09-06

Builds on: The MFT toolbox: read-only vs mutating, DOCA Ethernet and OVS-DOCA, HBN: BGP/EVPN on the server

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Describe the telemetry path from DTS providers through the container to each exporter, naming ports and file paths.
  • Deploy DTS on a host or DPU, enable providers in dts_config.ini, and scrape the Prometheus endpoint on port 9100.
  • Bring up BlueMan by satisfying its DTS and DPE dependencies and reading its configuration file.
  • Compose a Firefly static-pod environment block for a given PTP profile and check the firmware and clock prerequisites.

Episode 2 — Twenty-four cards that never answered

The situation · A bridge call at 01:20, wave one live in Hall 2, twenty-two days to move-in

The night-shift operator has been on the bridge an hour before you join. Wave one is up: twenty-four nodes, HBN peering with the leaves, the tenant’s test workload perfectly happy. The monitoring wall has one dark column where the DPUs should be, and he has already labelled that monitor with his label maker — DPU, PENDING — because that is what he does with things that do not work yet. The network lead unmutes for four words. Show me the counter. Nobody can say whether a port is dropping packets or whether the cards are simply quiet.

Telemetry on a DPU is awkward because the interesting counters live in a dozen places: sysfs, ethtool, tc, the vNIC, nvidia-smi. DTS exists to collect from all of those providers in one container and hand them out through one set of exporters, so an existing monitoring stack gains a scrape target instead of a project.[1] Providers are switched on a line at a time in dts_config.ini, the service restarts itself within 60 seconds of a config change, and Prometheus scrapes port 9100.[1]

There are two more asks on this call. The operator wants a screen a night-shift engineer can read without building Grafana; that is BlueMan, which needs DTS and the DPE daemon already running.[2] And the tenant moving into Hall 2 timestamps everything it does, so somebody will eventually ask whether the clock on these cards is disciplined at all.[3] You cannot argue with a counter nobody is collecting. Follow the data from where it is produced.

1The telemetry path: providers, DTS, exporters

The DOCA Telemetry Service (DTS, GA in 3.5.0) is a container that collects counters from providers and hands them to exporters.[6] Providers named by the guide: sysfs, ethtool, tc, vNIC, ifconfig, nvidia-smi, DCGM, bfperf, storage, amber, diagnostic_data_low_freq, diagnostic_data_high_freq, plus aggregation inputs from Fluent Bit, Prometheus and OpenTelemetry.[1] Exporters: a Prometheus scrape endpoint (default port 9100), Fluent Bit push, OTLP push, Prometheus Remote Write, and binary files under /opt/mellanox/doca/services/telemetry/data.[1]

Two more inputs matter. DOCA applications publish their own counters through the DOCA Telemetry Exporter library over IPC sockets in /tmp/ipc_sockets/ (root).[1] And for data the container cannot read on its own, the DOCA Privileged Executer (DPE) runs the provider outside the container: start it with systemctl start dpe and name the provider with a grpc. prefix, for example enable-provider=grpc.sysfs.fw_pages.[1] BlueMan requires DPE as well.[2]

Rendering diagram…
Diagram source (Mermaid)
flowchart LR
 PR[Providers sysfs ethtool tc nvidia-smi DCGM bfperf storage amber] --> DTS[DTS container]
 APP[DOCA apps via Telemetry Exporter lib] -->|ipc sockets| DTS
 DPE[DPE daemon for grpc providers] --> DTS
 FF[Firefly ptp4l phc2sys synced] -->|telemetry export| DTS
 DTS -->|scrape port 9100| PROM[Prometheus]
 DTS -->|push| FB[Fluent Bit]
 DTS -->|OTLP push| OT[OpenTelemetry collector]
 DTS -->|remote write| RW[Prometheus remote write]
 DTS -->|binary files| BIN[telemetry data directory]
 DTS --> BM[BlueMan dashboard on DPU OOB IP]
 DPE --> BM
The DTS telemetry path in DOCA 3.5.0: inputs on the left, exporters on the right. BlueMan reads through DTS and DPE; Firefly can export its clock statistics into DTS.

The library side is separate from the service: the DOCA Telemetry library offers per-domain reader contexts (doca_telemetry_pcc, _dpa, _diag, _adp_retx, _phy) created with doca_telemetry_<domain>_create(dev), configured with setters, then started; in 3.5.0 it “now exposes RTT counters (hardware congestion control counters)”.[8][5]

2Deploying DTS on the DPU and on the host

On a DPU, DTS is already there: a static pod at /etc/kubelet.d/doca_telemetry_standalone.yaml starts at boot; remove the file to disable it. The service directory is /opt/mellanox/doca/services/telemetry/.[1] Static pods follow the common DOCA model: kubelet watches /etc/kubelet.d/, sudo crictl pods and sudo crictl logs <id> show state, and a pod takes up to 20 seconds to start.[4]

On an x86 host (any Dell PowerEdge with Docker), the guide gives a two-command recipe. First initialize a host-flavoured config directory, then run the service:[1]

docker run -v "/opt/mellanox/doca/services/telemetry/config:/config" --rm \
  nvcr.io/nvidia/doca/doca_telemetry:<version> \
  /bin/bash -c "DTS_CONFIG_DIR=host /usr/bin/telemetry-init.sh"

docker run -d --net=host --privileged --device=/dev/mst/ --gpus all \
  -v "/opt/mellanox/doca/services/telemetry/config:/config" \
  -v "/opt/mellanox/doca/services/telemetry/ipc_sockets:/tmp/ipc_sockets" \
  nvcr.io/nvidia/doca/doca_telemetry:<version> /usr/bin/telemetry-run.sh

Configuration lives in /opt/mellanox/doca/services/telemetry/config/dts_config.ini: enable-provider=<name> (one line per provider), update=1000 (ms), prometheus=http://0.0.0.0:9100, output=/data, sync-time-limit=10000, level-labels-file=/config/level_labels.ini, add-pci-address-label=true, add-numa-node-label=true, and sysfs-allowed-ports=60,62,64-72. Counter and field filters are .cset and .fset files. Config changes are picked up automatically: the service restarts within 60 seconds.[1]

The high-frequency telemetry (HFT) path adds an HTTP API on port 9117 for the diagnostic-data provider: POST /ad-hoc-collection/start, GET /ad-hoc-collection?collection_id=X (returns a tar.gz), and DELETE. It needs the fwctl driver loaded (fwctl and mlx5_fwctl, with a /dev/fwctl/fwctl* node visible inside the container) and firmware 28.43.1000 or later on ConnectX-7, or 32.43.1000 or later on BlueField-3; retention defaults to 1 GB, one week maximum age and ten minutes minimum age.[1] In a DPF fleet the same service is installed from the Helm chart doca-telemetry version 1.25.5.[7]

3BlueMan: the dashboard on the DPU

BlueMan (GA) “runs in the DPU as a standalone web dashboard and consolidates all the basic information, health, and telemetry counters into a single interface”: system information, health, and DTS counters with graphs.[2][6] Its dependencies are explicit: “DTS and the DOCA Privileged Executer (DPE) daemon must be up and running”, and the BlueField image must be version 3.9.3.1 or higher.[2]

Deployment is the static-pod pattern again: copy its YAML into /etc/kubelet.d/. Two images are involved, doca_blueman_fe_service_<version>-doca<version>_arm64.tar (front end) and doca_blueman_conv_service_<version>-doca<version>_arm64.tar (converter). Then browse to http://<DPU_OOB_IP> and log in with the DPU’s SSH credentials.[2] Thresholds and other back-end settings sit in /opt/mellanox/doca/services/telemetry/config/blueman_config.ini and are reflected within 60 seconds.[2]

4Firefly: PTP and SyncE as a service

Firefly (GA) packages time synchronization: ptp4l (linuxptp v4.2), phc2sys, testptp for PPS, synced for SyncE, plus Firefly Monitor and Firefly Servo.[3][6] Profiles: default (PTP default profile, client plus server, phc2sys on), media (SMPTE 2059-2), l2-telco (G.8275.1, client-only, SyncE on), and custom.[3][9] Config paths are layered: built-in /etc/linuxptp inside the container, user overrides in /etc/firefly/ on the host (mounted), and the finalized files in /tmp/; the NGC resource ships scripts/doca_firefly/<version>/ptp4l-default.conf, ptp4l-media.conf and a configs/ directory.[3]

The static-pod YAML is driven by environment variables: PTP_INTERFACE: "p0" (space-separated for several ports), PTP_PROFILE: default|media|l2-telco|custom, PHC2SYS_ARGS: "-a -r", MONITOR_STATE: disable|<ip>, PPS_STATE: defined_by_profile, SERVO_STATE: disable, SYNCE_INTERFACE: "p0", and per-parameter overrides of the form CONF_<TYPE>_<SECTION>_<PARAMETER>, for example CONF_PTP_global_priority1: "64".[3] The NVIDIA skill adds PTP_STATE, PTP_CONFIG_FILE, MONITOR_CLIENT_TYPE, PHC2SYS_STATE and PHC2SYS_CONFIG_FILE.[9]

Prerequisites: firmware 24.34.1002 or later; BlueField image 3.9.0 or later; the real-time clock enabled with sudo mlxconfig -d 03:00.0 set REAL_TIME_CLOCK_ENABLE=1; INTERNAL_CPU_MODEL=1 for embedded or =0 for separated mode; NTP disabled; OVS hardware offload on; and PPS-capable hardware for PPS.[3] The host variant runs with Docker, for example sudo docker run --privileged --net=host -v /var/log/doca/firefly:/var/log/firefly -v /etc/firefly:/etc/firefly -e PTP_INTERFACE='eth2' -it nvcr.io/nvidia/doca/doca_firefly:1.7.0-doca3.0.0-host /entrypoint.sh; the 3.5.0-matched tag was not captured on the fetched page, so read it from NGC before you copy this line.[3] Logs: /var/log/doca/firefly/firefly.log, ptp4l.log, phc2sys.log, synced.log, or sudo crictl logs <id>.[3]

What changed in 3.5.0: the servo detects “dynamic packet rate”; the SyncE backend auto-detects dpll on kernel 6.8+ and falls back to mft; Firefly exports telemetry to DTS; and a multi-interface boundary-clock JBOD mode exists.[3]

Worked → faded → problem: a Firefly env block for a broadcast customer

Ask. A media customer wants SMPTE 2059-2 on port p0, wants the DPU to be a preferred grandmaster candidate (lower priority1), and wants Firefly stats in their existing DTS.

  1. Check prerequisites on the host: sudo flint -d /dev/mst/mt41692_pciconf0 q (firmware must be 24.34.1002 or newer) and sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q REAL_TIME_CLOCK_ENABLE INTERNAL_CPU_MODEL. If REAL_TIME_CLOCK_ENABLE is 0, it must be set to 1 and the card power-cycled before Firefly is useful.[3]
  2. Disable NTP on the DPU OS and confirm OVS hardware offload is on (ovs-vsctl get Open_vSwitch . other_config:hw-offload should be "true").[3]
  3. Write the env block in the Firefly static-pod YAML (2-space indentation, as kubelet requires):[4]
env:
  - name: PTP_INTERFACE
    value: "p0"
  - name: PTP_PROFILE
    value: "media"
  - name: PHC2SYS_ARGS
    value: "-a -r"
  - name: CONF_PTP_global_priority1
    value: "64"
  - name: MONITOR_STATE
    value: "disable"
  - name: PPS_STATE
    value: "defined_by_profile"

Reasoning: media selects the SMPTE 2059-2 built-in; CONF_PTP_global_priority1 overrides one ptp4l parameter without a custom file; PHC2SYS_ARGS: "-a -r" keeps the system clock following the PHC.[3] 4. sudo cp doca_firefly.yaml /etc/kubelet.d/ and verify with sudo crictl pods and tail -f /var/log/doca/firefly/ptp4l.log.[3][4]

5Which tool for which customer question

Customer asks Answer with Key fact to cite
“Can we get DPU counters into our Prometheus?” DTS scrape endpoint port 9100, one enable-provider line per source[1]
“Something is wrong on this DPU, show me now” BlueMan http://<DPU_OOB_IP>, needs DTS + DPE[2]
“We need PTP for 5G / broadcast” Firefly profiles l2-telco / media, firmware 24.34.1002+[3]
“Deep firmware diagnostics on demand” DTS HFT API port 9117, fwctl, firmware 28.43.1000+ (ConnectX-7) or 32.43.1000+ (BlueField-3)[1]
“Our own DOCA app must report metrics” Telemetry Exporter library IPC sockets in /tmp/ipc_sockets[1]

End of Episode 2 — The column fills in

How it ended

By 02:05 the dark column has numbers in it. You enabled the providers the operator actually needed, waited out the sixty-second restart, curled port 9100 to prove the metrics existed before blaming anyone’s Prometheus, and left BlueMan open on one DPU so the night shift had a page to look at.[1][2] Firefly goes on the follow-up list with its firmware and real-time-clock prerequisites, not into tonight’s change.[3] What you say to the network lead at breakfast: iDRAC sees the card, DTS sees what the card sees. At 06:40 her runbook lands in your inbox. It scripts gnoic against port 9339, and she wants to know which of you is out of date before her team sees your slide.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Pre-flight on the Dell-lab BlueField-3 (read-only):

  1. On the DPU: sudo crictl pods | grep -i telemetry (DTS should already be running from boot), ls /opt/mellanox/doca/services/telemetry/config, and curl -s http://127.0.0.1:9100/metrics | head. Record cat /etc/mlnx-release (BlueField image version; BlueMan needs 3.9.3.1 or higher) and, on the host, sudo flint -d /dev/mst/mt41692_pciconf0 q for the firmware version. Note that the Firefly guide’s floor, 24.34.1002, sits on the BlueField-2 firmware line while a BlueField-3 reads 32.x (lesson 1.2), so the two numbers are not comparable: record what the card reports and treat the gate as unresolved rather than reading 32 > 24 as a pass.[1][2][3][10]

Mutating steps, each with rollback:

  1. Start DPE. sudo systemctl status dpe; if inactive, sudo systemctl start dpe. Expected: active. Rollback: sudo systemctl stop dpe.[1]

  2. Deploy BlueMan. Copy the BlueMan YAML from the NGC resource into /etc/kubelet.d/. Expected within about 20 seconds: sudo crictl pods shows the fe and conv pods; http://<DPU_OOB_IP> prompts for the DPU SSH credentials and shows system info, health and DTS graphs. If not: sudo crictl logs <id> and confirm DTS and DPE are up. Rollback: sudo rm /etc/kubelet.d/<blueman>.yaml.[2][4]

  3. Tune a threshold. Edit /opt/mellanox/doca/services/telemetry/config/blueman_config.ini, change one threshold, wait 60 seconds and refresh the page. Rollback: restore the original file.[2]

  4. Firefly, only if prerequisites hold. If the firmware gate recorded in step 1 is resolved (not simply assumed from the 32-versus-24 comparison) and sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q REAL_TIME_CLOCK_ENABLE already reads 1, deploy the Firefly YAML with PTP_INTERFACE: "p0" and PTP_PROFILE: "default" after disabling NTP on the DPU. Expected: tail /var/log/doca/firefly/ptp4l.log shows port state changes. Rollback: remove the YAML and re-enable NTP. If REAL_TIME_CLOCK_ENABLE is 0, stop here: setting it is a firmware change that needs a power cycle and is out of scope for this lab; note it for the M2 mode-switching procedure.[3]

  5. Record. Save curl -s http://127.0.0.1:9100/metrics | wc -l before and after enabling grpc.sysfs.fw_pages in dts_config.ini (needs DPE from step 2). Rollback: remove the line.[1]

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, how a customer gets BlueField counters into the Prometheus they already run, and what BlueMan adds on top.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs (DTS, BlueMan, Firefly service guides), 2026-09-06. Dates are when each page was fetched.

  1. DOCA Telemetry Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  2. DOCA BlueMan Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  3. DOCA Firefly Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  4. DOCA Container Deployment Guide · fetched 2026-09-06 · DOCA 3.5.0
  5. Changes and New Features (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  7. doca-platform v26.4.1: dpuservicetemplate_dts.yaml · fetched 2026-09-06
  8. NVIDIA/skills doca-telemetry SKILL.md · fetched 2026-09-06
  9. NVIDIA/skills doca-firefly SKILL.md · fetched 2026-09-06
  10. DOCA General Support (per-family firmware table) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.