Host QoS with mlnx_qos: trust state, prio2tc, ETS

1One command, four blocks, and the line that decides everything

mlnx_qos is “a centralized tool used to configure QoS features of the local host. It communicates directly with the driver thus does not require setting up a DCBX daemon on the system.”^[1] Its syntax is mlnx_qos -i <interface> [options], and run with no options it prints the host’s entire QoS state — which is why it is the first command on any RoCE ticket.^[1][2]

The canonical output has four blocks. NVIDIA’s own sample, abridged:^[1][8]

DCBX mode: OS controlled
Priority trust state: dscp
dscp2prio mapping:
        prio:0 dscp:07,06,05,04,03,02,01,00,
        prio:1 dscp:15,14,13,12,11,10,09,08,
        prio:2 dscp:23,22,21,20,19,18,17,16,
        prio:3 dscp:31,30,29,28,27,26,25,24,
        prio:4 dscp:39,38,37,36,35,34,33,32,
        prio:5 dscp:47,46,45,44,43,42,41,40,
        prio:6 dscp:55,54,53,52,51,50,49,48,
        prio:7 dscp:63,62,61,60,59,58,57,56,
Cable len: 7
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   0   0   0   0   0
tc: 0 ratelimit: unlimited, tsa: vendor

Read them in this order. DCBX mode says who owns the configuration, firmware or the OS.^[1] Priority trust state says which packet field the NIC prioritizes on, and it is the line that decides whether an application’s DSCP has any effect at all — “The default trust state is PCP. Ethernet packets are prioritized based on the value of the field (PCP/DSCP).”^[1] The dscp2prio mapping block is the DSCP-to-priority table. The PFC configuration row and the eight tc: lines say what can be paused and how each class is scheduled.

The default map deserves a second look because operators keep rebuilding it by hand. It groups DSCP in eights, so DSCP 24-31 already lands on priority 3 and DSCP 48-55 on priority 6, with no --dscp2prio at all.^[1] That is exactly the split the switch side uses: Cumulus defaults RoCE data to DSCP 26 on switch priority 3 and CNP to DSCP 48 on switch priority 6.^[5] Host and switch agree out of the box on everything except the trust state.

ib_write_bw -T 104 -R        # CM path
ib_write_bw --tclass=104      # AV path, no CM
# no code change at all:
echo "tclass=104" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
echo "tclass=16,src_ip=1.1.1.2,dst_ip=1.1.1.0/24" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
echo "tclass=-1" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class   # delete

2The option surface, exactly as documented

Every flag below is quoted from the Ethernet QoS option table.^[1] -f LIST, --pfc=LIST: “Set priority flow control for each priority. LIST is a comma separated value for each priority starting from 0 to 7. Example: 0,0,0,0,1,1,1,1 enable PFC on TC4-7”.^[1] -p LIST, --prio_tc=LIST: “Maps UPs to TCs. LIST is 8 comma-separated TC numbers. Example: 0,0,0,0,1,1,1,1 maps UPs 0-3 to TC0, and UPs 4-7 to TC1”.^[1] -s LIST, --tsa=LIST takes strict, ets or vendor per class.^[1] -t LIST, --tcbw=LIST sets minimum guaranteed percentages that “must sum to 100”, and values on non-ETS classes “are ignored but must be present”.^[1] -r LIST, --ratelimit=LIST is Gbps per class, where “For an unlimited ratelimit, set the ratelimit to 0”, with a documented tolerance of roughly 10 percent.^[1] --trust=TRUST is “set priority trust state to pcp or dscp”; --dscp2prio=DSCP2PRIO takes 'set,30,2' to map DSCP 30 to priority 2 and 'del,30,2' to reset it; --cable_len=CABLE_LEN sets “cable_len for buffer’s xoff and xon thresholds”; -d DCBX chooses fw or os.^[1]

Two semantics matter more than the syntax. Strict priority: “TC 7 is the highest priority, TC 0 is lowest”, strict classes also outrank all ETS classes, and “This property needs to be used with care, as it may easily cause starvation of other TCs.”^[1] ETS is the opposite kind of promise — it “exploits the time periods in which the offered load of a particular Traffic Class (TC) is less than its minimum allocated bandwidth by allowing the difference to be available to other traffic classes”, so the percentages are floors, not ceilings.^[1] That is the pairing the switch side uses too: CNP strict, RoCE and default traffic at 50 percent each under weighted round robin.^[5] vendor is the third legal value and the one NVIDIA’s own sample output shows on every class, but the option table names it without defining its scheduling behaviour, re-checked 2026-09-07.^[1][2] Read a tsa: vendor line as “driver default, not a decision anyone made”, and if the class matters, set the algorithm you intend explicitly with -s.

3Applying it: the DSCP-trust host recipe and its rollback

The host half of a DSCP-based lossless RoCE deployment is three settings: trust DSCP, map the fabric’s RoCE DSCP to its priority, and enable PFC on that priority.^[1][5] A Juniper validated design that configures NVIDIA hosts uses exactly this shape, mlnx_qos -i <interface> --pfc <0/1>,<0/1>,... followed by a bare mlnx_qos -i <interface> to verify.^[7]

mlnx_qos -i <if> > /tmp/qos-before.txt   # this file is your rollback plan
mlnx_qos -i <if> --trust dscp
mlnx_qos -i <if> --dscp2prio set,26,3
mlnx_qos -i <if> --pfc 0,0,0,1,0,0,0,0
mlnx_qos -i <if>                          # diff against /tmp/qos-before.txt

Two cautions about that block. First, the exact three-line combination is widely circulated but the NVIDIA knowledge-base article carrying it as a canonical recipe could not be fetched for this course, so treat the strings as assembled from the documented option table rather than as a quoted NVIDIA recipe; the DSCP 26 to priority 3 default underneath it is confirmed.^[5][1] Second, --dscp2prio set,26,3 is redundant on a stock host, because the default map already puts 24-31 on priority 3 — it is worth running anyway when a previous operator may have edited the map, and worth checking rather than assuming.^[1]

Ordering has one hard rule: “Setting the Trust State mode shall be done before enabling SR-IOV in order to propagate the Trust State to the VFs.”^[1] On a host that is already running VFs, changing trust on the PF does not retroactively fix them, which is why this step belongs in provisioning and not in a live-troubleshooting session. NVIDIA’s own Spectrum-X host profile encodes the same decisions as declarative fields — Trust: dscp, ToS 96, and a PFC bitmap of "00010000" — so the recipe above is the manual form of a profile the NIC Configuration Operator applies for you.^[6]

4Where mlnx_qos is the wrong tool

On BlueField the tool has a documented trap: “When working in Embedded Host mode, using mlnx_qos on both the host and Arm will result with undefined behavior. Users must only use mlnx_qos from the Arm.”^[4] This is not a warning about conflicting values; it is a statement that the result is undefined. On a Dell PowerEdge with a BlueField-3 in DPU mode, the QoS conversation belongs to the Arm side, and a customer’s x86 runbook that sets host QoS needs to be rewritten before it is run, not after it misbehaves.

Rate limiting on the DPU is a different subsystem entirely. DOCA documents devlink port function rate add|del|set|show with tx_max and tx_share, plus set parent for grouping functions into a shared rate group — not mlnx_qos -r.^[4] Reaching for mlnx_qos to rate-limit a DPU function is a category error that will silently do nothing useful.

The third boundary is the switch. Everything in this lesson stops at the NIC: user priorities, traffic classes and PFC enables on the host. The switch has its own vocabulary — switch priority, traffic class, buffer pool — and its own defaults, and the two only meet on the wire through DSCP or PCP.^[5] When a customer says “we set priority 3”, always ask which side of the wire they set it on.