DMS: one interface for every device
S5·E3The port number in last year's runbook · A hotel lobby at 07:10, ninety minutes before the automation workshop, nineteen days to move-in
Builds on: Modes of operation: DPU, NIC, Zero-Trust, rshim and the BFB path, Out-of-band: BMC, Redfish, and the BlueField-4 path
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Describe what the DOCA Management Service manages in 3.5.0 and how dms-cli deduces GET, SET and ACTION from a YANG path.
- Use dms-cli introspection (--tree, --describe, list-targets) to find the path for a device setting before touching it.
- Apply the operating-mode and BFB-provisioning flows through dms-cli and state what a reboot or power cycle still requires.
- Distinguish the 3.5.0 dms-cli plus /nvidia/* surface from the 3.4.0 dmsd gNMI/gNOI server and explain what is and is not documented for each.
Episode 3 — The port number in last year's runbook
Coffee, a laptop on your knees, and the network lead’s runbook open next to your deck. Your deck says dms-cli. Her runbook, written last year against a release her team read carefully, scripts gnoic against port 9339. This morning’s workshop is where the operator decides whether provisioning the remaining 168 cards goes into their existing automation or stays a manual runbook, and the Dell SE has already written “fully automated” on row 47. The NVIDIA PM dials in from two time zones away and will answer anything about the roadmap with “not announced”, which is his job.
DMS exists because a fleet does not tolerate hand work. Every device setting reached through mlxconfig and every image pushed through bfb-install is one more thing an engineer does per card; DMS is the centralized way to configure and operate BlueField and ConnectX devices from a script instead.[1] In 3.5.0 the documented client is dms-cli, alias dmsc, and it is path-centric: you give it a target and a path such as /nvidia/mode/operating-mode, and it deduces GET, SET or ACTION from the path and its arguments.[1]
Her notes are not wrong, they are older. The 3.4.0 guide documented dmsd, a systemd gRPC service on localhost:9339 with TLS on by default and gNMI Get and Set but no Subscribe.[2] The 3.5.0 provisioning flows moved to native DMS APIs.[3] Ask which release they read before you tell them they are wrong. Before you answer her, know exactly what DMS is today.
1What DMS is in DOCA 3.5.0
The 3.5.0 guide defines the DOCA Management Service as “a centralized solution for configuring and operating BlueField and ConnectX devices”. Its client is dms-cli (alias dmsc) at /opt/mellanox/doca/services/dms/dms-cli, and the tool “is path-centric: GET, SET, and ACTION are deduced from the path and arguments”.[1] The services index lists DMS at Alpha quality in 3.5.0, lower than any other service in this module.[4]
The guide’s table of contents shows the scope: Introduction, System Components, YANG Models, DMS-CLI Flags, Operations, Introspection, Provisioning and Reset, Prerequisites and Dependencies. It also describes “Profile-based NIC provisioning plans (e.g. SPC-X / MRC): plan, install, verify, cleanup”.[1] Two 3.5.0 change-log items explain the shape: “Updates DMS provisioning flows to use native DMS APIs instead of gNOI” and “Adds doca_mgmt API support to resolve vhca_id through libdms and CLI”; a third “Introduces DMS-facing Core API to query native_port_num”.[3]
Diagram source (Mermaid)
flowchart LR OP[Operator script or DPF provisioning controller] --> CLI[dms-cli alias dmsc] CLI --> LIB[libdms native DMS API] LIB --> T1[target pci 0000:03:00.0] LIB --> T2[target pci 0000:03:00.0 pf0vf5] T1 --> M[nvidia-mode operating-mode nic or dpu] T1 --> P[nvidia-provisioning install BFB async] T1 --> R[nvidia-reset factory-reset] T1 --> N[nvidia-nvconfig nvidia-link nvidia-icm and other modules] LEG[Legacy 3.4.0 client gNMI gNOI] -.-> D[dmsd systemd service localhost 9339] D -.-> T1
2Paths, targets and introspection
DMS “uses NVIDIA-native YANG models - paths follow the /nvidia/… structure rather than the OpenConfig conventions used in the legacy interface”. Top-level modules include nvidia-root, -link, -lag, -eswitch, -multiplane, -qos, -roce, -cc, -data-direct, -icm, -platform, -mode, -pci, -nvconfig, -emulation, -misc, -raw, -artifacts, -provisioning, -blueprints, -reset, -operation and -diagnostics.[1]
You never guess a path. Introspection first:[1]
/opt/mellanox/doca/services/dms/dms-cli --tree /nvidia
/opt/mellanox/doca/services/dms/dms-cli --describe /nvidia/mode
/opt/mellanox/doca/services/dms/dms-cli list-targetsThen address a target with -t. A PCI function is pci/0000:03:00.0; a VF is appended, as in -t pci/0000:03:00.0,pf0vf5 for /nvidia/icm/quota.[1] GET is a bare path; SET is the same path followed by a value:[1]
/opt/mellanox/doca/services/dms/dms-cli -t pci/0000:03:00.0 /nvidia/mode/operating-mode # GET
/opt/mellanox/doca/services/dms/dms-cli -t pci/0000:03:00.0 /nvidia/mode/operating-mode dpu # SET
/opt/mellanox/doca/services/dms/dms-cli -t pci/0000:03:00.0,pf0vf5 /nvidia/icm/quotaThe operating-mode enum has two values: nic (“Standard NIC, ARM cores disabled”) and dpu (“DPU mode, ARM cores enabled”).[1] In M1 terms these are the two states of INTERNAL_CPU_OFFLOAD_ENGINE on BlueField-3 (ENABLED(0) is DPU mode, DISABLED(1) is NIC mode), and the modes page still requires a power cycle to apply a mode change.[9] Zero-trust is not an operating-mode value; it remains a host-privilege restriction set with mlxprivhost from the Arm side.[9]
3Provisioning and reset through DMS
Provisioning is asynchronous: dms-cli -t pci/0000:03:00.0 /nvidia/provisioning/install source=staged:bf3.bfb installs a staged BFB. Reset is an action node: dms-cli /nvidia/reset/factory-reset.[1] The guide’s “Provisioning and Reset” section also covers the profile-based NIC plans (SPC-X / MRC) with a plan, install, verify, cleanup sequence.[1]
What the 3.5.0 page does not mention: mlxfwreset. Reset is exposed as /nvidia/reset/* YANG actions; the MFT mlxfwreset command remains a host-side tool from M2 and is not part of the documented DMS surface.[1] Likewise the page does not describe rshim: the BFB path in M2 (bfb-install over /dev/rshim0) and the DMS install action are two routes to the same result, and the guide does not state how the staged image reaches the device.[1]
In DPF, DMS is the provisioning back end for host-trusted clusters: the reference DPFOperatorConfig sets provisioningController: {dmsTimeout: 900}, and the v26.4.1 release notes say the “host agent install phase now completes once DMS returns”, which made host-trusted installs faster.[8][7] When a DPU sits in the OS Installing phase for a long time, that 900-second budget is the first number to check.[8]
Ask. A B3140H SuperNIC arrived in NIC mode; the customer wants HBN, which needs DPU mode, and a fresh 3.5.0 image.
- Find the target:
dms-cli list-targets→ note the PCI address, herepci/0000:03:00.0.[1] - Read before writing:
dms-cli -t pci/0000:03:00.0 /nvidia/mode/operating-mode→nic.[1] - Check what the node accepts:
dms-cli --describe /nvidia/mode→ enumnic|dpu.[1] - Write:
dms-cli -t pci/0000:03:00.0 /nvidia/mode/operating-mode dpu.[1] - Power-cycle the server (a mode change is a firmware configuration change and takes effect only after a power cycle).[9]
- Confirm: repeat step 2 →
dpu. Cross-check on the host withmlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE→ENABLED(0).[9] - Stage the image where DMS expects it and install:
dms-cli -t pci/0000:03:00.0 /nvidia/provisioning/install source=staged:bf3.bfb. The call returns immediately; poll the provisioning status path (find it with--tree /nvidia/provisioning).[1]
Reasoning: the Arm image is useless while the Arm cores are disabled, so the mode change and its power cycle come first; introspection replaces guessing at every step.
dms-cli ____to find the target.dms-cli -t ____ /nvidia/mode/operating-mode(GET).dms-cli --describe ____.dms-cli -t pci/0000:03:00.0 /nvidia/mode/operating-mode ____(SET).- Apply with a ____.
- Cross-check:
mlxconfig ... q ____should read____. dms-cli -t pci/0000:03:00.0 /nvidia/provisioning/install source=____.
A host has two BlueField-3 cards. Write the dms-cli sequence that (a) reads the ICM quota of VF 5 on the first card, (b) factory-resets the second card, and (c) proves both results. Acceptance: correct -t syntax for the VF target, an explicit note of which step is an ACTION rather than a SET, and the introspection command you would run first if the quota path were unknown.
4The pivot: 3.4.0 dmsd gNMI/gNOI versus 3.5.0 dms-cli
The 3.4.0 guide describes a server: “On the DPU, dmsd comes pre-installed with the BlueField BFB-Bundle and runs automatically as a systemd service bound to localhost:9339”, with flags --bind_address (default :9339), --auth shadow|credentials|cert, --allowed_users, --tls_enabled (on by default), --tls_cert_file, --tls_key_file, --ca and --enable_mlxconfig.[2] It spoke gNMI Get, Set and Capabilities, and stated that “DMS does not currently support the gNMI Subscribe command (streaming telemetry data)”. Its gNOI services were OS Install/Activate/Verify (BFB), System Reboot and reboot-status, Factory-reset, File stat/get/put, mlxconfig set/query/reset, and Containerz; clients were dmsc, dmscli or any gNOI client such as gnoic. Status: “currently in beta stage, with General Availability (GA) support for SPC-X use cases”.[2]
The 3.5.0 guide contains none of the strings dmsd, gNMI, gNOI, 9339, gRPC, server, libdms, gnmic, gnoic or Subscribe. It refers “Existing DMS users … looking for the legacy, OpenConfig-based tool” to a separate legacy guide.[1]
| Aspect | 3.4.0 guide | 3.5.0 guide |
|---|---|---|
| Client | gNMI/gNOI clients, dmsc, dmscli |
dms-cli / dmsc (the 3.5.0 change log also names libdms behind the doca_mgmt API)[3] |
| Server | dmsd systemd service on localhost:9339 |
not documented |
| Data model | OpenConfig-style | NVIDIA-native /nvidia/* |
| Streaming | gNMI Subscribe unsupported | not documented |
| Provisioning | gNOI OS install | /nvidia/provisioning/install native API |
| Quality | beta (GA for SPC-X) | Alpha in the services index[4] |
Two things remain unverified and you should say so to a customer: whether the dmsd binary still ships in the 3.5.0 BFB and whether the legacy gRPC server is functional or supported in 3.5. The NVIDIA doca-dms skill (main branch) still describes dmsd plus dmspe, four auth modes and even claims gNMI Subscribe STREAM and ONCE are implemented in gnxi/gnmi/server.go, which contradicts the 3.4.0 public guide; the skill itself warns to verify against the version-matched guide.[5][6][2]
End of Episode 3 — What goes on the slide instead
You rewrite one slide before nine. It carries both surfaces — the 3.4.0 dmsd gNMI and gNOI server on 9339, and the 3.5.0 dms-cli with its /nvidia/ paths and introspection commands — with the arrow pointing one way.[1][2] You add the quality level, because leaving it out would be the real mistake: DMS is Alpha in the 3.5.0 services index.[4] What you say to the network lead: build toward dms-cli, keep your gnoic path until we test it together on the lab card, and do not make DMS the only way you can reach a DPU this year. At 09:50 the tenant’s own requirement arrives. Their nodes ship diskless.
Lab
All steps are read-only: introspection and GET only. Do not issue a SET or an ACTION in this lab.
-
Pre-flight. On the Dell-lab BlueField-3 Arm OS:
cat /etc/mlnx-release(image version) andls -l /opt/mellanox/doca/services/dms/. Expected:dms-clipresent on a 3.5.0 BFB. If not: DMS may not be installed on this image; stop and record.[1] -
Targets.
sudo /opt/mellanox/doca/services/dms/dms-cli list-targets. Expected: at least onepci/0000:xx:00.0entry for the local device.[1] -
Browse.
sudo /opt/mellanox/doca/services/dms/dms-cli --tree /nvidia | head -60andsudo /opt/mellanox/doca/services/dms/dms-cli --describe /nvidia/mode. Expected: the module list from Segment 2 and theoperating-modeenumnic|dpu.[1] -
Read the mode.
sudo /opt/mellanox/doca/services/dms/dms-cli -t pci/<addr> /nvidia/mode/operating-mode. Expected:dpuon the lab card. Cross-check from the host:sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE→ENABLED(0).[1][9] -
Read a VF quota. If VFs exist (
ls /sys/class/net | grep -E "pf0vf"), runsudo /opt/mellanox/doca/services/dms/dms-cli -t pci/<addr>,pf0vf0 /nvidia/icm/quota. Expected: a quota value; if no VFs, record the error text.[1] -
Answer the unverified question.
systemctl status dmsd 2>&1 | head -5andss -ltnp | grep 9339. Expected: either an active legacydmsdonlocalhost:9339(the 3.4.0 behaviour surviving in 3.5.0) or nothing. This is the evidence the lesson could not obtain from documentation; record it with the image version.[2] -
Log. Save the outputs of steps 2 through 6 with
cat /etc/mlnx-releaseat the top; you now hold the lab’s DMS baseline for later mutating exercises.
Goal: prove the 3.4.0-to-3.5.0 pivot from primary sources and read the fleet-side contract, without a BlueField.
-
Count the gRPC vocabulary on each guide. Run
for v in 3-4-0 3-5-0; do echo "== $v"; curl -sL "https://networking-docs.nvidia.com/doca/archive/$v/doca-management-service-guide" | grep -o -i -E "gnmi|gnoi|dmsd|9339|dms-cli|/nvidia/" | tr 'A-Z' 'a-z' | sort | uniq -c; done. Expected: the 3.4.0 page returns counts forgnmi,gnoi,dmsdand9339; the 3.5.0 page returns none of those and does returndms-cliand/nvidia/. If not: the site may have served the index shell; add-A Mozilla/5.0and retry.[1][2] -
Read the 3.5.0 change log for DMS.
curl -sL https://networking-docs.nvidia.com/doca/archive/3-5-0/changes-and-new-features | grep -o -i -E "[^.]*(gnoi|libdms|native_port_num)[^.]*\.". Expected: the three DMS-related sentences (native DMS APIs instead of gNOI;vhca_idthroughlibdms;native_port_num).[3] -
Find the fleet timeout.
curl -s https://raw.githubusercontent.com/NVIDIA/doca-platform/v26.4.1/docs/public/user-guides/host-trusted/use-cases/hbn-ovnk-snap/manifests/03-dpf-system-installation/operatorconfig.yaml | grep -n -A1 provisioningController. Expected:dmsTimeout: 900.[8] -
Check the DOCA host container for the CLI.
docker run --rm nvcr.io/nvidia/doca/doca:devel-3.5.0-host ls -l /opt/mellanox/doca/services/dms/ 2>&1 | head. Expected: either thedms-clibinary (then run./dms-cli --tree /nvidia | head -40to browse the model offline) or “No such file”, which tells you DMS ships in the BFB rather than the host devel image. Record which.[1] -
Write a path plan. From the module list in Segment 2, write a five-row table: setting the customer wants (mode, link type, ICM quota for a VF, factory reset, BFB install), the
/nvidia/...module you would browse with--describe, and whether it is GET, SET or ACTION.[1] -
Read the skill conflict.
curl -s https://raw.githubusercontent.com/NVIDIA/skills/main/skills/doca-dms/SKILL.md | grep -n -i -E "subscribe|dmsd|9339|verify". Expected: lines claiming Subscribe support and a warning to verify against the version-matched guide. Note in your log why you would trust the 3.4.0 guide’s “no Subscribe” statement over the skill.[5][2]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE, in four sentences, what DMS is for and why the 3.5.0 version looks different from what a customer may have read about last year.
Sources
Facts in this lesson were checked against DOCA 3.5.0 DMS guide (dms-cli) and 3.4.0 DMS guide (dmsd), 2026-09-06. Dates are when each page was fetched.
- DOCA Management Service Guide (3.5.0, dms-cli) · fetched 2026-09-06 · DOCA 3.5.0
- DOCA Management Service Guide (3.4.0, dmsd gNMI/gNOI) · fetched 2026-09-06 · DOCA 3.4.0
- Changes and New Features (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
- DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
- NVIDIA/skills doca-dms SKILL.md · fetched 2026-09-06
- NVIDIA/skills doca-dms CAPABILITIES.md · fetched 2026-09-06
- doca-platform v26.4.1 release notes · fetched 2026-09-06
- doca-platform v26.4.1: operatorconfig.yaml (dmsTimeout) · fetched 2026-09-06
- BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.