Programmable congestion control with DOCA PCC
S4·E4Coffee, a laptop, and a customer who wants their own algorithm · Hotel lobby, 06:50, four hours after the bridge call
Builds on: Telemetry-based congestion control: three loops, not one
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Describe the DOCA PCC architecture: host library, device libraries on the DPA, and the Reaction Point and Notification Point roles.
- Write the enablement runbook for PCC, including the mlxconfig fields, the power-cycle requirement and the firmware floor.
- Choose a probe mechanism among CCMAD, IFA1 and IFA2 and say what each provides.
- Place PCC correctly among the three Spectrum-X loops, and name what it does not replace.
Episode 4 — Coffee, a laptop, and a customer who wants their own algorithm
The research architect left the bridge call with a sentence you have been turning over since: we want to run our own congestion control. He gets twenty minutes now, the re-cabled acceptance run is at ten, and the SE is across the table with a coffee he will not drink. You also have a lab card in your bag that did not behave last night — you set the field, warm-rebooted the host, and the value read back as if you had never touched it.
Both problems have the same answer, and it starts with why this component exists. Rate decisions have to close inside a round trip and touch every flow context on the adapter, so NVIDIA made the algorithm itself programmable in place: DOCA PCC allows users to design and implement their own congestion control algorithm, with a host library managing the PCC context configuration and device libraries running on the BlueField data-path accelerator, split into a Reaction Point that adjusts transmission rates and a Notification Point that processes congestion notifications[1]. That accelerator is the same 16 hyperthreaded cores you inventoried on the SuperNIC[4]. The gates are narrow — BlueField-3 or later, ETHERNET link type only, firmware 32.38.1000 and higher[1] — and last night’s failure is on the same page: the change requires a graceful shutdown and a power cycle, not a warm reboot[1].
A field that reads back unchanged after a reboot is usually a field that needed the power cycle you skipped. So walk in with a runbook and a boundary, not with a yes.
1What PCC is, and where each piece runs
DOCA PCC allows users to design and implement their own congestion control algorithm, delivered on BlueField-3 as a DOCA component.[1] The architecture has two halves. A host library offers a unified interface for managing the DOCA PCC context configuration; device libraries run on the BlueField data-path accelerator (DPA) and are split into two roles.[1] The Reaction Point (RP) monitors network conditions actively and dynamically adjusts data transmission rates to alleviate congestion promptly. The Notification Point (NP) passively receives congestion notifications from external sources and processes them.[1]
That DPA is the same hardware you inventoried in lesson 2: 16 hyperthreaded cores on the SuperNIC, programmable through DOCA for congestion control and traffic management.[4] So “programmable congestion control” is literal — the algorithm is code on the adapter, not a profile selection.
Support is narrow and worth stating up front: applications run on the host or on the NVIDIA BlueField-3 platform or later, and DOCA PCC is supported only for the ETHERNET link type.[1] The library requires firmware version 32.38.1000 and higher.[1]
Diagram source (Mermaid)
flowchart TB
subgraph HOST[Host]
A[Application] --> L[DOCA PCC host library]
L -->|doca_pcc_create / set_app / start| C[PCC context]
end
subgraph NIC[BlueField-3 SuperNIC]
C --> DPA[Data-path accelerator - 16 hyperthreaded cores]
DPA --> RP[Reaction Point - adjusts tx rate]
DPA --> NP[Notification Point - handles notifications]
end
RP -->|CCMAD / IFA1 / IFA2 probes| FAB[(Spectrum-X fabric)]
FAB -->|RTT + in-band telemetry| NP
NP --> RP2The enablement runbook
Two persistent fields turn the two roles on. Enable RP with mlxconfig -d <mlx_device> -y s USER_PROGRAMMABLE_CC=1; enable NP with mlxconfig -d <mlx_device> -y s PCC_INT_EN=0.[1] Both changes require a graceful shutdown and then a power cycle of the host.[1] A warm reboot leaves the adapter powered and the previous value live, which is the single most common reason an engineer reports that “mlxconfig did nothing”.
The pre-flight list before you type either command:
| Gate | Check | Rule |
|---|---|---|
| Device | BlueField-3 platform or later[1] | flint -d /dev/mst/<dev> q |
| Link type | ETHERNET only[1] | ibv_devinfo | grep link_layer |
| Firmware | 32.38.1000 or higher[1] | flint -d /dev/mst/<dev> q |
| Stack alignment | The RA row you run: v2.3.1 pins DOCA-Host 3.5.0-082 and BlueField-3 firmware 32.50.1002 on the H200 track[5] | Compare host and adapter against the validated stack |
| Rollback reference | Baseline captured before any write | mlxconfig -d /dev/mst/<dev> q > pcc-before.txt |
The host API sequence around the context is short: doca_pcc_create(), doca_pcc_set_app(), doca_pcc_set_thread_affinity(), then doca_pcc_start().[1] The device side implements callbacks — on the RP doca_pcc_dev_user_init(), doca_pcc_dev_user_set_algo_params() and doca_pcc_dev_user_algo(), and on the NP doca_pcc_dev_np_user_packet_handler().[1]
3Probes: what the algorithm gets to see
An algorithm is only as good as its signal, and PCC supports three probe mechanisms. CCMAD provides information about the network’s round-trip time, used for proactive congestion detection. IFA1, In-band Flow Analyzer 1, provides in-band congestion feedback. IFA2 offers an optimized alternative method for in-band congestion feedback.[1]
Those map onto the platform story from the previous lesson: sender and receiver coordinate to assess congestion paths precisely using RTT probe packets and in-band telemetry from intermediary switches, including timestamps and buffer utilization.[6] CCMAD is the RTT half; IFA1 and IFA2 are the in-band-telemetry half.
The feature history is worth knowing because customers ask “since when”. DOCA 2.7 was the release that delivered lossless RoCE using adaptive routing and DOCA programmable congestion control for Spectrum-X RA 1.0.1, along with NP programmability to trigger alerts on congestion events, support for multiple probe packets, and Tx/Rx byte tracking at NIC endpoint ports.[2] DOCA 3.3.0 later introduced support for injecting response timestamps in NP operations.[1]
4Where PCC sits among the three loops
Map PCC onto the diagram from lesson 3 before you touch a customer’s fabric. PCC replaces the sender-side rate control algorithm: the reaction point is the thing that decides how fast to send after reading probes and notifications.[1] It does not replace in-switch adaptive routing, which is a switch behaviour on Spectrum-4 hardware, and it does not replace plane load balancing, which is a dedicated hardware engine on the SuperNIC.[6]
That boundary is the answer to “can we write our own Spectrum-X?” No: you can write your own rate control on the endpoint and leave the other two loops in place. The corollary matters commercially — a customer who wants custom CC still needs Spectrum-X switches for loop one and SuperNICs for loops two and three.[6]
On the same host, NVIDIA’s own doRoCE.sh shows the neighbouring knobs an FAE will meet on the way: trust mode through QPTS with trust_state, default dscp; PFC masks through PFCC; lossy-mode acceleration and selective repeat through ROCE_ACCL; and ZTR-RTTCC through PPCC with cmd_type=2 and algo_slot=15.[3] Script defaults are ToS 106, GID index 3 and trust dscp; note that ToS 106 encodes DSCP 26, which falls inside the switch’s default RoCE range of DSCP 24 to 31 mapping to switch priority 3.[3]
5Runbook practice
Objective: enable the PCC reaction point on one BlueField-3 lab host, verify it, and be able to undo it.
- Pre-flight.
sudo mst start; sudo mst status -vfor the device path.sudo flint -d /dev/mst/mt41692_pciconf0 q— confirm the part is BlueField-3 and firmware is at or above 32.38.1000.[1]ibv_devinfo | grep link_layer— must read Ethernet, because PCC supports the ETHERNET link type only.[1] Compare the firmware and the installed DOCA-Host against the RA row you intend to run: v2.3.1 pins DOCA-Host 3.5.0-082 with BlueField-3 firmware 32.50.1002 on the H200 track.[5] - Baseline.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q > pcc-before.txt. Keep the file; it is the rollback reference. - Change.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -y s USER_PROGRAMMABLE_CC=1.[1] - Power cycle. Graceful shutdown of the host, then a full power cycle — not a warm reboot, because the field is read at adapter initialisation.[1]
- Verify.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep -i USER_PROGRAMMABLE_CC— the current value must read 1. If it still reads 0, the machine was warm-rebooted or the write targeted a different device. - Exercise. Build and run a DOCA PCC sample from the installed DOCA-Host and observe the device callbacks firing:
doca_pcc_dev_user_init()at startup anddoca_pcc_dev_user_algo()per event on the RP side.[1] - Rollback.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -y s USER_PROGRAMMABLE_CC=0, graceful shutdown, power cycle, thendiff <(sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q) pcc-before.txtand expect no differences.[1]
Complete the runbook for the Notification Point:
- Pre-flight: device must be ____ or later; link type must be ____; firmware at or above ____.[1]
- Baseline:
sudo mlxconfig -d /dev/mst/<dev> q > ____. - Change:
sudo mlxconfig -d /dev/mst/<dev> -y s ____=____.[1] - Then: graceful ____ and a ____, because a warm reboot leaves the setting inactive.[1]
- Verify by grepping the query output for the field; the NP device callback to look for in the sample is
____.[1] - Rollback: restore the field and repeat step 4, then ____ against the baseline file.
- Which release added injecting response timestamps in NP operations? ____.[1]
A Dell customer running an H200 pod on RA v2.3.1 wants to trial a custom congestion control algorithm on twelve BlueField-3 hosts in one rack, while the other ninety hosts keep the platform behaviour.
Produce: (a) the enablement runbook for one host with pre-flight, change, power cycle, verification and rollback, every step citing its rule; (b) the two questions you would ask before agreeing to the trial at all; (c) the paragraph explaining which of the three loops changes and which two do not; (d) an explicit statement of any step you are treating as unverified and how you would test it first. Acceptance: no step relies on a warm reboot, the firmware floor and link type are checked before any write, and the unverified legacy-DCQCN command, if you mention it, is labelled as such.[1][6][5]
Firmware: Dell DUP vs NVIDIA mlxfwmanager / bf-fwbundle — which one wins?
The whiteboard at the end of the hour
They leave with a decision tree rather than a project. The telemetry-based rate control comes with the platform[6]; PCC is for the team with a research reason to write their own algorithm and the staff to carry device code across DOCA releases[1]. You draw the boundary explicitly: PCC replaces the sender-side rate loop, not in-switch adaptive routing and not plane load balancing on the SuperNIC[6]. And you give the architect the sentence he writes down: “You can own the algorithm, but you still need the switch and the SuperNIC underneath it.” At ten the acceptance run goes green on eligible uplinks. By noon the infrastructure director has booked a briefing and used a word nobody has scoped for him: quad-plane.
Lab
Goal: enable the PCC reaction point on your Dell-lab BlueField-3, prove it took effect, and roll it back cleanly. This lab changes persistent adapter configuration.
Pre-flight inventory (read-only). sudo mst start; sudo mst status -v; sudo flint -d /dev/mst/mt41692_pciconf0 q (part number, PSID, firmware); ibv_devinfo | grep -E 'link_layer|fw_ver'; ofed_info -s or the DOCA-Host version. Stop here if firmware is below 32.38.1000 or the link layer is not Ethernet — the library requires both.[1]
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q > pcc-before.txt— expected: a full configuration dump. This file is the rollback reference for every later step. If the command fails, MFT is not installed; install it from the DOCA-Host repository first.grep -i -E 'USER_PROGRAMMABLE_CC|PCC_INT_EN' pcc-before.txt— expected: both fields present with their current values. If a field is absent, this firmware does not expose it; record that and stop.- Mutating.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -y s USER_PROGRAMMABLE_CC=1— expected: the tool prints the field with its next-boot value.[1] Rollback for this step: the same command with=0, followed by another power cycle, then a diff againstpcc-before.txt. - Mutating. Graceful shutdown of the host, then a full power cycle at the PDU or through iDRAC — not a warm reboot.[1] Rollback: none needed for a power cycle, but do not skip it; the verification in step 5 is meaningless without it.
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep -i USER_PROGRAMMABLE_CC— expected: current value 1. If it still reads 0, you warm-rebooted, or the device path in step 3 was not the device you just queried.- Build and run a DOCA PCC sample from the installed DOCA-Host, and watch the device-side callbacks fire on the DPA. Expected: initialisation through
doca_pcc_dev_user_init()and per-event calls todoca_pcc_dev_user_algo().[1] If the sample refuses to start, re-check the link type and the firmware floor before debugging the code. - Rollback (mandatory before you leave the lab).
sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -y s USER_PROGRAMMABLE_CC=0, graceful shutdown, power cycle, thensudo mlxconfig -d /dev/mst/mt41692_pciconf0 q > pcc-after.txt; diff pcc-before.txt pcc-after.txt— expected: no differences. - Optional, only on a card you own: test the unverified
ROCE_CC_LEGACY_DCQCN=0pattern with the same baseline-change-power-cycle-verify-rollback discipline, and record whether the field exists on your firmware at all. Report the result as your own observation, not as documentation.[1]
Goal: write the complete enablement runbook without touching hardware, and place PCC on the three-loop diagram.
- Build the pre-flight table: device family, link type, firmware floor, and the RA row for the customer’s track. Expected values: BlueField-3 or later; ETHERNET only; 32.38.1000 or higher; v2.3.1 pins DOCA-Host 3.5.0-082 and BlueField-3 firmware 32.50.1002 on the H200 track.[1][5]
- Write the two
mlxconfiglines exactly, one for RP and one for NP, and mark the shutdown requirement between the write and the verification.[1] Expected:USER_PROGRAMMABLE_CC=1andPCC_INT_EN=0, each followed by graceful shutdown and power cycle. - Write the verification command and the rollback, and state what the verification output must show for the runbook to be considered complete. If your rollback does not include a second power cycle, it is not a rollback.
- In two sentences, explain why a warm reboot silently leaves the setting inactive, in terms an operator can repeat to their change board.[1]
- Take the three-loop diagram from lesson 3 and mark which loop PCC replaces and which two it leaves untouched.[6] Expected: sender-side rate control is programmable; adaptive routing and plane load balancing are not.
- List the three probe mechanisms and what each provides, then choose one for a fabric where the customer wants per-hop attribution and justify it in one sentence.[1]
- Mark every step you could not source. Expected: at least the
ROCE_CC_LEGACY_DCQCN=0pattern, which is unverified.[1]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell lab engineer, in four sentences, what you are actually changing when you enable DOCA PCC on a BlueField-3, and why the change needs a power cycle.
Sources
Facts in this lesson were checked against DOCA PCC SDK page re-fetched 2026-09-07; DOCA-Host 3.5.0-082 per Spectrum-X validated stack v2.3.1; NVIDIA doRoCE.sh. Dates are when each page was fetched.
- DOCA PCC (Programmable Congestion Control) | DOCA SDK · fetched 2026-09-07 · DOCA 3.5.0
- Enhancing AI Cloud Data Centers and NVIDIA Spectrum-X with NVIDIA DOCA 2.7 (NVIDIA Technical Blog) · fetched 2026-09-07
- NVIDIA/doroce-linux: doRoCE.sh host-side RoCE configuration script · fetched 2026-09-07
- Powering Next-Generation AI Networking with NVIDIA SuperNICs (NVIDIA Technical Blog) · fetched 2026-09-07
- NVIDIA Spectrum-X Validated Solution Stack · fetched 2026-09-07
- Giga-Scale AI and the Ethernet Evolution: How Spectrum-X Ethernet Rewrites the Rules (NVIDIA Technical Blog) · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) certification page · fetched 2026-09-07
- Introduction | NVIDIA ConnectX-8 SuperNIC User Manual · fetched 2026-09-07
- NVIDIA ConnectX-8 SuperNIC Firmware Release Notes v40.48.1000 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.