Skip to content
nodeza.learn
DOCA course/5. Services/Lesson 5.5

Security: Argus, App Shield, and zero trust

S5·E5The agent that watches the watcher · A conference room at the operator's office, day one of the tenant's security review, eight days to move-in

S5·E5Evaluate~35 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs (Argus service guide, App Shield library, modes page), 2026-09-06

Builds on: Modes of operation: DPU, NIC, Zero-Trust, Switching modes safely, Storage offload: SNAP, Virtio, DevEmu, STA

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain how Argus performs agentless host introspection from the DPU and why that is stronger than a host-resident agent.
  • Evaluate whether a given server meets every Argus prerequisite (mode, firmware, BFB, PF_BAR2, IOMMU, OS) before proposing it.
  • Map Argus event families and output formats onto a customer's SIEM pipeline.
  • Judge what a host administrator loses in zero-trust mode and when that trade is right for a Dell OEM customer.
  • Choose between the Argus service and the App Shield library for Linux, Windows and custom-integration cases.

Episode 5 — The agent that watches the watcher

The situation · A conference room at the operator's office, day one of the tenant's security review, eight days to move-in

The tenant sent its own security architect, and her first question is the fair one you were expecting: they already run an agent on every host, so why pay for a second pair of eyes on the same machine? The operator cannot answer it for her, and that is the shape of the whole deal — in Hall 2 the tenant owns the host operating system and the operator owns the card. The network lead has stopped writing. The Dell SE has stopped reheating his coffee, which is how you know the room matters.

Because the first pair of eyes lives inside the thing it is watching. An agent in the host OS can be stopped, blinded or lied to by anything that gains enough privilege on that host. Argus does the observation from the other side of the PCIe bus: it is agentless, using hardware-level live machine introspection and DOCA DMA to read selected structures in volatile host memory.[1] The service runs on the Arm cores, so a compromised host root user has no process to kill, and in zero-trust mode cannot reconfigure the card to stop it either.[1]

Her SOC counterpart is already on the practical question: what arrives in the pipeline, in what format, and through which collector.[1] The night-shift operator is quiet, because he suspects what you know — this only works on a machine that meets every gate. The only witness a compromised host cannot lie to is the one it cannot reach. Start with how the watching actually happens.

1Agentless introspection: watching the host from outside

Argus is a security service that performs “hardware-level live machine introspection” and uses “NVIDIA DOCA DMA to read selected structures in volatile host memory”. No software runs on the host; the DPU reads the host’s memory and interprets it.[1] The services index lists Argus at Beta in 3.5.0, and NVIDIA’s own skill describes it as the “currently-promoted successor” to the App Shield library.[6][3]

The security argument is structural. An agent inside the host shares the attacker’s privilege domain: root can stop it, patch it or blind it. Argus runs on Arm cores the host cannot reach, over a DMA path the host did not set up. In zero-trust mode (Segment 4) the host cannot even reconfigure the card to remove that path.[1][7] What Argus sees, per the guide’s event families: processes created, terminated, zombie or hidden; shell history cleared; reverse shells; foreign binaries executed; kernel modules loaded or changed; containers started; and GPU identity, metrics and topology.[1] 3.5.0 added “key GPU metrics without a host tool/agent”, per-container “runtime manifests”, and Vector as an output.[5]

2The gates: what must be true before Argus runs

The guide is precise, and each item is a go/no-go check:[1]

Gate Requirement
Device and mode “Operates only on BlueField-3 or later configured in DPU mode”
Firmware “Requires firmware version 24.35.0388 or later”
BlueField image “Supported BlueField image versions: 4.11.0 or later”
Host firmware keys mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1
Host kernel GRUB iommu=pt plus intel_iommu=on or amd_iommu=on
Host OS Linux only; KVM tested; Kata only with NVIDIA-DPU support
Container Runs privileged; image on NGC as nvidia/doca/doca_argus
[1]

The DPU-mode requirement inherits everything from M1: INTERNAL_CPU_OFFLOAD_ENGINE must read ENABLED(0), and a SuperNIC shipped in NIC mode must be switched and power-cycled first.[7] The PF_BAR2 keys are the ones to watch in mixed designs: SNAP-4’s base configuration sets PF_BAR2_ENABLE=0, and the DPF reference flavor does the same, so a “storage plus security on one card” proposal is unproven until you test it.[8]

OS profiles tell Argus where kernel structures live. They are “auto-generated at runtime”, or produced manually with python3 doca_apsh_config.py --files memregions symbols --os <linux/windows> from /opt/mellanox/doca/tools/doca_apsh_config.py, which needs dwarf2json and the distribution’s debug symbols (Ubuntu ddebs, RHEL debuginfo); the result is a set of JSON files on the DPU.[1]

The service reads a SERVICE_CONFIG_FILE with: service_log_level (10 DISABLE to 70 TRACE, default 50), system_scanner_sleep_time, auto_scan, profile_generation, and a systems[] list where each host has representor_id (a VUID such as MT2333XZ06YAMLNXS0D0F0VF1), memory_regions_path and os_symbol_path (or auto), os_type, dma_device_name (from ibv_devinfo | grep hca_id), limits, and events.container_filter/enrichment/sbom.[1] Debug logs land in /var/log/doca_argus/.[1]

3Events, severity and the SIEM hand-off

Every Argus message carries a message_header (vendor NVIDIA, product DOCA_ARGUS, type EVENT, ALERT or SYSTEM_ACTIVITY, a severity, schema 1.0, and an ISO-8601 timestamp with nanoseconds), then bluefield_system_information, workload_information (with a container_context when applicable) and activity_data.[1] The guide marks nine event types ALERT/HIGH: Shell History Cleared, Shell History Disabled, Reverse Shell Detected, Process Hidden, Foreign Binary Executed, Binary Executed Not as Intended, Foreign Binary Executed File Size Mismatch, Foreign Library Loaded and Foreign Library Loaded File Size Mismatch. The remaining process, kernel-module, container and GPU records are activity data a SIEM correlates.[1]

Output is configured in the same file: log_events_to_stdout, log_folder_path, telemetry_address, telemetry_tag, and telemetry_format: JSON|syslog. The documented transports are Fluent Bit over TCP 24224 or Vector, which is how the stream reaches Splunk, Elastic or a cloud SIEM the customer already runs.[1]

4Zero trust: what the host gives up

Zero Trust (Restricted) mode is “A specialized variation of DPU Mode that enhances security by preventing the host system administrator from accessing BlueField from the host side”; the card “must be fully controlled by the data center administrator via the Arm cores or the BMC connection, rather than through the host”.[7] The host loses five things, each individually restrictable: port ownership (“The host cannot assign itself as the port owner”), hardware counters, tracer functionality, the RShim interface, and firmware flashing from the host.[7]

It is set from the Arm side, never the host: sudo mlxprivhost -d /dev/mst/<device> r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner, and restored with sudo mlxprivhost -d /dev/mst/<device> p. On the host, mlxconfig q INTERNAL_CPU_OFFLOAD_ENGINE shows an RO flag when zero trust is active. Rules: “Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode”; perform a system-level reset when leaving zero trust; and “Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior”.[7]

DPU mode + host locked out of the card
Host (x86 / Arm server) — mlx5 netdevsArm subsystem — representors + bridgeUplinks (Arm netdevs p0 / p1)PF netdevhost pf0 (mlx5)VF netdevsSR-IOV pf0vf0…SF netdevsen3f0pf0sf4 …PCIe Gen5 x16e-switch (embedded switch, switchdev)inside the NIC ASIC — flow tables programmed by the e-switch managerp0uplink 0p1uplink 1Wire — QSFP112 400G / 200Gpf0hpfpf0vf0pf0sf0OVS-DOCA bridge ovsbr1 (p0 + pf0hpf)or HBN br-hbnArm appsovs-vswitchd · DOCA Flow · tcpdump -i pf0hpfhost restricted: no rshim · no port owner · no counters · no tracer · no fw flash
hop 1 / 4
Zero Trust (Restricted) mode · Host → Arm

1. Host sends on the PF netdev

Same start as any transmit. Nothing on the host says "this goes to the Arm".

What the host lost (mlxprivhost … r)
  • --disable_rshim — RShim interface — no /dev/rshim0 console, no bfb-install, no tmfifo_net0 from the host
  • --disable_port_owner — Port ownership — the host cannot assign itself as port owner
  • --disable_counter_rd — Hardware counters — the host is denied access to hardware counters
  • --disable_tracer — Tracer functionality is blocked
  • (always) — Firmware flash from the host is restricted
Mode configuration
sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE   # ENABLED(0) with the RO flag = Zero Trust active
sudo mlxprivhost -d /dev/mst/<device> r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner

Run on the Arm OS, not the host. Back to plain DPU mode: sudo mlxprivhost -d /dev/mst/<device> p (then a system-level reset). Via BMC: PATCH …/HostPrivilegeConfig/Settings {"PrivilegeMode":"Restricted"} + power cycle.

  • Power cycle required.
  • Zero Trust → NIC is not direct: restore DPU mode first, reset, then set INTERNAL_CPU_OFFLOAD_ENGINE=1.
  • The card must be managed from the Arm or the BMC — the host cannot flash firmware or push a BFB.
BlueField Modes of Operation · Virtual Switch on BlueField · OVS-DOCA
Mode zerotrust, direction host-to-arm: step through what a host-originated management request meets. Compare with mode dpu to see which paths (rshim, counters, firmware flash) disappear.

At fleet scale the same idea becomes a deployment mode. DPF zero-trust treats the host as untrusted: “DPUs are managed through their Baseboard Management Controller (BMC) via Redfish”, all management runs over the DPU OOB network, and “The host sees the DPU as a standard NIC”.[9] Argus and zero trust compose: Argus supplies the observation, zero trust guarantees the observed party cannot switch it off or re-flash the observer.[1][7]

5Service or library: Argus versus App Shield

The App Shield library (doca_apsh) exposes the same introspection as callable APIs: doca_apsh_create, doca_apsh_dma_dev_set, doca_apsh_start, then per host doca_apsh_system_create, doca_apsh_sys_os_type_set, doca_apsh_sys_dev_set, doca_apsh_sys_os_symbol_map_set, doca_apsh_sys_mem_region_set, doca_apsh_sys_kpgd_file_set, doca_apsh_system_start; and getters doca_apsh_processes_get, modules_get, libs_get, threads_get, vads_get, netscan_get, process_netscan_get, yara_get, attestation_get, containers_get, interfaces_get, doca_apsh_process_info_get, plus Windows-only privileges_get, envars_get, handles_get, ldrmodules_get, process_parameters_get and sids_get. It requires firmware 24.32.1010 or later; its prerequisites state supported system types as Windows or Linux, and it lists specific Windows 10 builds (x86 10240 through 19041, x64 15063 through 19041) only for the netscan_get and process_netscan_get getters.[2] The 3.5.0 samples (apsh_pslist, apsh_modules_get, apsh_libs_get, apsh_threads_get, apsh_vads_get, apsh_netscan_get, apsh_containers_get, apsh_injection_detect_get and others) follow one flow: open a DMA-capable device, create the apsh context, open the remote PCI device by VUID, create the system handler, call the getters.[4][10]

Worked → faded → problem: evaluate an Argus request

Request. “Deploy Argus on our 40 R760 hosts with B3220 cards; hosts run Ubuntu 22.04 on KVM; cards came with firmware 32.41.x and BFB 4.9.x; the same cards will also run SNAP-4 for boot-from-NVMe-oF.”

Evaluation, gate by gate:[1]

  1. Device and mode: B3220 is a BlueField-3 DPU SKU, ships in DPU mode. Pass (verify INTERNAL_CPU_OFFLOAD_ENGINE reads ENABLED(0) on a sample card).[7]
  2. Firmware: the Argus guide requires “24.35.0388 or later” on a page that also says Argus “operates only on BlueField-3 or later”. Those are not comparable numbers: 24.x is the BlueField-2 firmware line and BlueField-3 is 32.x (lesson 1.2), and the DOCA 3.5.0 table pins BlueField-3 at 32.50.1002.[1][11] So record the actual flint -d /dev/mst/<dev> q value (32.41.x here) against 32.50.1002, treat the gate as unresolved, and raise the guide’s cross-family minimum with NVIDIA instead of declaring a pass by comparing 32 to 24.
  3. BlueField image 4.11.0 or later: 4.9.x fails. A BFB upgrade to 4.11.0 or newer is a prerequisite task with its own change window (M2).
  4. Host keys PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1: conflict. SNAP-4’s documented base line sets PF_BAR2_ENABLE=0. Neither guide documents coexistence; this is a lab test before any commitment.[8]
  5. GRUB iommu=pt intel_iommu=on: Ubuntu on Intel, pass after a host reboot per node.
  6. Linux hosts, KVM tested: pass.
  7. SIEM: ask which collector; plan Fluent Bit or Vector with telemetry_format set to the SOC’s preference.

Verdict: conditional go. Two blocking items (BFB upgrade, PF_BAR2 coexistence test) and one per-node reboot. Write that as the proposal’s dependencies, not as a footnote.

End of Episode 5 — What you commit to, and what you do not

How it ended

You leave them one demo and one honest list. The demo is three events, not thirty, arriving in their collector with no agent anywhere in the host.[1] The list is the gates: BlueField-3 in DPU mode, firmware 24.35.0388 or later, BlueField image 4.11.0 or later, the two PF_BAR2 keys with a power cycle, IOMMU passthrough in GRUB, and Linux hosts only — the Argus service is Linux-only and App Shield is a library, not a product.[1][2] Then zero trust runs into cable: it assumes the BlueField’s own BMC is reachable on its OOB port, and iDRAC is the server’s BMC, not the DPU’s.[9][7] The operator is already down the aisle with his label maker, tagging 168 uncabled OOB ports.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Pre-flight gates on the Dell-lab BlueField-3 (read-only). If any gate fails, stop after step 1 and record; do not proceed to the mutating steps.

  1. Host: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE PF_BAR2_ENABLE PF_BAR2_SIZE (need ENABLED(0); record the two BAR2 values as your rollback baseline; an RO flag would mean the card is already zero-trust), sudo flint -d /dev/mst/mt41692_pciconf0 q (firmware at least 24.35.0388), cat /proc/cmdline | grep -o -E "iommu=pt|intel_iommu=on|amd_iommu=on". DPU: cat /etc/mlnx-release (image at least 4.11.0), ibv_devinfo | grep hca_id (value for dma_device_name), and sudo crictl pods | grep -i snap (if SNAP is running, the PF_BAR2 change below will conflict; do not proceed on a shared card).[1][7][8]

Mutating steps, each with rollback:

  1. Host firmware keys. sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1, then power-cycle the server. Expected: the query from step 1 shows the new values. Rollback: set both keys back to the step-1 baseline and power-cycle.[1]

  2. Host IOMMU. Add iommu=pt intel_iommu=on (or amd_iommu=on) to GRUB_CMDLINE_LINUX in /etc/default/grub, sudo update-grub, reboot. Expected: /proc/cmdline contains both. Rollback: remove the tokens, update-grub, reboot.[1]

  3. Service config. On the DPU write the SERVICE_CONFIG_FILE from noHw step 3 with the real representor_id (VUID) and dma_device_name from step 1, profile_generation enabled for auto profiles, log_events_to_stdout: true. Rollback: back up any pre-existing file (sudo cp <path> <path>.bak) before writing, and after removing the pod in step 5 delete the file you added (sudo rm <path>, where <path> is whatever the pod YAML’s SERVICE_CONFIG_FILE points at) or restore the .bak.[1]

  4. Deploy Argus. Copy the doca_argus static-pod YAML into /etc/kubelet.d/. Expected: sudo crictl pods shows the pod; sudo crictl logs <id> shows events with product: DOCA_ARGUS; on the host, start and stop a harmless process (sleep 300 &) and see Process Created and Process Terminated records. If not: /var/log/doca_argus/ and raise service_log_level to 60. Rollback: sudo rm /etc/kubelet.d/<argus>.yaml.[1]

  5. Do not enter zero-trust mode in this lab. Note the M2 procedure instead: from the Arm side mlxprivhost -d /dev/mst/<device> r ... restricts, mlxprivhost -d /dev/mst/<device> p plus a system-level reset restores, and a restricted card must return to DPU mode before any NIC-mode change.[7]

  6. Restore. Roll back steps 5, 4, 3 and 2 in that order and repeat the step-1 query to prove the baseline.

Retrieval check

11 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, what Argus gives a security team that an EDR agent does not, and what it costs to turn on.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs (Argus service guide, App Shield library, modes page), 2026-09-06. Dates are when each page was fetched.

  1. DOCA Argus Service Guide · fetched 2026-09-06 · DOCA 3.5.0
  2. DOCA App Shield (library) · fetched 2026-09-06 · DOCA 3.5.0
  3. NVIDIA/skills doca-argus SKILL.md · fetched 2026-09-06
  4. doca-samples 3.5.0: samples/doca_apsh/README.md · fetched 2026-09-06 · DOCA 3.5.0
  5. Changes and New Features (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  7. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  8. SNAP-4 Service Deployment (PF_BAR2_ENABLE=0) · fetched 2026-09-06 · DOCA 3.5.0
  9. doca-platform v26.4.1: getting started, zero-trust mode · fetched 2026-09-06
  10. doca-samples 3.5.0 full git tree (API) · fetched 2026-09-06 · DOCA 3.5.0
  11. DOCA General Support (per-family firmware table) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.