Zero Touch RoCE, ZTR-RTT CC and DOCA PCC

1The product answer to 'we will not configure PFC'

Zero Touch RoCE is a claim about deployment: it “enables data centers to seamlessly deploy RDMA over Converged Ethernet (RoCE) without requiring any special switch configuration”.^[3] The mechanism behind that claim is ZTR-RTT CC, which NVIDIA describes as “a congestion control (CC) mechanism within the DOCA Programmable Congestion Control (PCC) framework”.^[1] Everything in this lesson therefore lives on the NIC. If a customer asks which switch feature to enable, the honest answer is none — that is the point.

The algorithm watches two indicators. RTT is the “Real-time measurement of the delay between sending a packet and receiving its acknowledgment”; CNPs are the “Explicit notifications sent by the network or receiver when congestion is encountered”.^[1] Rate control is “Additive Increase” and “Multiplicative Decrease”, and the RTT measurement uses “a highly synchronized, hardware-assisted timestamping sequence” with four timestamps: request-send, request-receive, response-send and final-receive.^[1] Four timestamps rather than two is what lets the NIC subtract the responder’s own turnaround time and attribute the remainder to the fabric.

Contrast that with the previous lesson’s loop. DCQCN reacts to a switch’s opinion of its own queue, expressed as a CE mark, and requires the switch to be configured with a threshold. ZTR-RTT reacts to a measurement the sender makes itself. A switch that was never told about RoCE will not mark, but it will still add queueing delay, and delay is what ZTR-RTT reads. That is the whole argument for the “zero touch” name.

The published performance framing is from the 2021-12-14 launch blog by Aviv Barnea, Itay Ozery and Barbara Claman: “ZTR-RTTCC extends DCQCN in RoCE networks with a hardware RTT-based congestion control algorithm”, claiming “superb RoCE performance at data center scale, without any special configuration of the switch infrastructure” and without enabling PFC, at 99 percent of conventional RoCE performance at 256 connections and 98 percent of conventional RoCE throughput at over 16,000 connections.^[3] Note what the blog does not say: it names no ConnectX generation.^[3]

mlxconfig -d <dev> s USER_PROGRAMMABLE_CC=1 PCC_INT_EN=0 \
  FLEX_PARSER_PROFILE_ENABLE=10 REAL_TIME_CLOCK_ENABLE=1 ROCE_CC_RTT_TIMESTAMP_FORMAT=2
mlxconfig -d <dev> q | grep -E "USER_PROGRAMMABLE_CC|ROCE_CC"

2The parameter table, and how to reason about it

NVIDIA publishes seventeen parameters, indexes 0 through 16. The ones that matter for a tuning conversation, re-read on 2026-09-07:^[2]

Two of these carry the whole tuning story. CONGESTION_DELAY_THRESHOLD is the floor below which measured delay is treated as the fabric’s baseline rather than as congestion; its default of 15000 ns is 15 microseconds, and its documented tuning range starts at 2 microseconds.^[2] On a tight single-rail fabric whose base RTT is a few microseconds, a 15-microsecond floor means the algorithm tolerates queues several times the fabric’s own latency before it does anything — which is a defensible default for a large multi-tier network and a poor one for a small tight pod. MAX_DELAY, at 250000 ns, is the point above which the algorithm reacts more aggressively.^[2]

TOPOLOGY_AWARE is the parameter that makes the threshold question partly self-solving, since it compares measured RTT to the minimum RTT the NIC has actually observed rather than to a fixed number.^[2] It defaults to off, so on a fabric with a base RTT far from the default you are choosing between moving the threshold and turning this on.

3Reading it: PPCC, cmd_type, and the counters that say blind

Everything is read and written through one firmware register, PPCC, with mlxreg. Setting a parameter:^[2]

mlxreg -d <dev> -y --set "cmd_type=8,value=<parameter value>" --reg_name PPCC \
  --indexes "local_port=1,pnat=0,lp_msb=0,algo_slot=1,algo_param_index=<parameter index>"

The counters are a debug-mode facility, so the session has a shape. Enable debug mode and counters with cmd_type=1,counter_en=1, reset with cmd_type=13, query with cmd_type=12, all on algo_slot=1; a separate cmd_type=2 on algo_slot=0 disables deployment mode.^[2] The mt4129 in NVIDIA’s examples is a ConnectX-7 MST device id — substitute your own from sudo mst status -v.^[2]

sudo mlxreg -d /dev/mst/mt4129_pciconf0 -y --set "cmd_type=1,counter_en=1" --reg_name PPCC \
  --indexes "local_port=1,pnat=0,lp_msb=0,algo_slot=1,algo_param_index=0"

Sixteen counters are published. The rate-action group is ZTR_CC_CNP_HANDLE_COUNTER, ZTR_CC_NACK_HANDLE_COUNTER, ZTR_CC_AI_INC_COUNTER, ZTR_CC_HAI_INC_COUNTER, ZTR_CC_DEC_COUNTER, ZTR_CC_HYPER_DEC_COUNTER and ZTR_CC_TX_DEC_COUNTER.^[2] The measurement group is ZTR_CC_MAX_RTT, ZTR_CC_MIN_RTT, ZTR_CC_SUM_RTT, ZTR_CC_NUM_RTT and ZTR_CC_NOT_VLD_RTT_COUNTER.^[2] The rate and diagnostic group is ZTR_CC_MAX_RATE, ZTR_CC_MIN_RATE, ZTR_CC_EMPTY_SYS_RTT_COUNTER for global minimum RTT detections and ZTR_CC_RTT_TIMEOUT_COUNTER for RTT probe timeouts.^[2]

The diagnosis that matters is the blind state. If ZTR_CC_NUM_RTT is flat, the algorithm is taking no measurements; if ZTR_CC_RTT_TIMEOUT_COUNTER is also flat, probes are not being sent rather than failing. A loaded algorithm with no RTT input is running open-loop on whatever CNPs happen to arrive, which is the one configuration strictly worse than either DCQCN or ZTR-RTT alone — you have given up the switch-side threshold and gained no delay signal.^[2] The counter that says the opposite — a healthy loop — is ZTR_CC_NUM_RTT climbing with a plausible ZTR_CC_MIN_RTT and a non-trivial ZTR_CC_DEC_COUNTER under load.

4The framework underneath, and its verified floor

ZTR-RTT is one algorithm inside DOCA PCC, and PCC is where the verified support statements live. “The DOCA PCC library offers a high-level programming interface that enables users to implement customized congestion control (CC) algorithms”; “DOCA PCC-based applications can run on either the host machine or the NVIDIA BlueField-3 Platform (or later) target”; “Currently, DOCA PCC is supported only for the ETHERNET link type”; and “The library requires firmware version 32.38.1000 and higher.”^[4] Two mode rules ride along and matter in a Dell lab: for BlueField-3 in DPU mode the NP process may not be executed on the x86 host, and running from an x86 host in NIC mode requires privileged permissions.^[4] Two roles are modelled, matching the DCQCN vocabulary from the previous lesson: the Reaction Point “Monitors network conditions actively, dynamically adjusting data transmission rates”, with RP context global per NIC; the Notification Point “Passively receives congestion notifications from external sources”, with NP context global per e-switch owner.^[4]

The firmware prerequisites are mlxconfig settings, staged for next boot:^[4][6]

mlxconfig -d <mlx_device> -y s USER_PROGRAMMABLE_CC=1        # RP: enable programmable CC
mlxconfig -d <mlx_device> -y s PCC_INT_EN=0                  # NP: disable the legacy notification point
mlxconfig -d <mlx_device> -y s REAL_TIME_CLOCK_ENABLE=1      # RP and NP: real-time-clock timestamps
mlxconfig -d <mlx_device> -y s ROCE_CC_RTT_TIMESTAMP_FORMAT=2 # RP: RTC timestamp format
mlxconfig -d <mlx_device> -y s FLEX_PARSER_PROFILE_ENABLE=10 # NP: response timestamp injection

The two timestamp settings are what make ZTR-RTT’s RTT probes measurable, so a card missing them is the loaded-but-blind case in segment 3; all five are staged for next boot and NVIDIA requires a graceful shutdown then a power cycle.^[4]

Custom algorithms are built with DPACC, which “compiles and links user algorithms and device code with the DOCA PCC device library to create loadable applications”, and the device code runs on the DPA while a host library manages the context.^[4] For an FAE the relevant point is not that you will write one, but that ZTR-RTT is delivered through the same machinery a customer could use to write their own — which is what “programmable” means in the product name.

Now the honest gap. Neither the ZTR-RTT algorithm page nor the parameter page states supported devices or a minimum firmware version, re-checked on 2026-09-07.^[2][1] The only verified floor is PCC’s, above, and it belongs to the framework. Extending “BlueField-3 or later” silently to a ConnectX part, in either direction, is an unsupported claim; the correct move in front of a customer is to read their cards.

The strongest available evidence that NVIDIA ships this by default on Spectrum-X hosts is circumstantial but specific. The NIC Configuration Operator’s Spectrum-X profile sets programmable congestion control true and DCQCN false, with CC slot 0 parameters of bandwidth 400, responsiveness alpha 6553, maximum decrease factor 63570, base round-trip time 15000 and maximum queuing delay 250000.^[5] Those four values are exactly the published defaults for ALPHA, MAX_DEC, CONGESTION_DELAY_THRESHOLD and MAX_DELAY.^[2] The same profile sets trust dscp, a ToS of 96 and a PFC bitmap of "00010000".^[5]