Skip to content
nodeza.learn
RoCE course/2. Congestion/Lesson 2.1

ToS, DSCP, PCP, SL: where marking gets lost

S2·E1The capture that ends the argument · Hotel lobby, 06:40, five days before the acceptance test

S2·E1Analyze~30 minsources checked todaylab mutates hardwareverified against DOCA-Host RoCE and Ethernet QoS pages (HTML + generated PDF), perftest master man page, Cumulus Linux 5.18 RoCE page, NCCL 2.31.2 env reference, 2026-09-07

Builds on: RoCEv1 vs RoCEv2: the encapsulation, GIDs: how a connection picks its RoCE version

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Trace one RoCE flow through the full marking chain from rdma_set_option ToS to the switch trust state and name the field that carries the marking at each hop.
  • Compute ToS from DSCP and SL and PCP from ToS using NVIDIA's stated arithmetic and check the result against a capture.
  • Analyze a customer symptom and decide which of the three loss points applies: no ToS set, no VLAN tag, or a trust-state mismatch.
  • Apply the Force DSCP sysfs override to re-mark an unmodified application and roll it back.

Episode 1 — The capture that ends the argument

The situation · Hotel lobby, 06:40, five days before the acceptance test

Six-forty in a hotel lobby, five days before the customer’s 64-node pod in Building D goes to acceptance, and the overnight dry run failed exactly as it did on Friday. Their network lead sent the capture with one sentence: we configured DSCP 26 everywhere and RoCE still lands in the default queue. Across the table the Dell SE is drafting a mail blaming the leaf switches, beside an airport coffee he has not touched and a spreadsheet whose row 41 reads “lossless: yes”.

You open the capture and filter UDP port 4791. The outer IP ToS byte reads zero. Trust state, the dscp2prio map, the leaf’s queues — all beside the point, because the application never marked anything: NVIDIA states that if a TOS is not set, the default value (0) is used.[1] Their jobs run under NCCL, whose NCCL_IB_TC defaults to 0, so an unset variable is an unmarked training run.[7]

This is why the marking chain exists at all. QoS on Ethernet is not a property a packet has; it is a value one layer hands to the next — application ToS, socket priority, user priority, traffic class — until a field on the wire reaches a switch that has been told which field to read.[2] Each handoff belongs to a different team, which is how all of them report success while the mark is gone.

A chain reports success at every link that still works. You have ninety minutes and one sysfs file that re-marks the flows without touching their binary.[1] Start where the chain starts.

1Marking is a chain, and every link can break

NVIDIA documents RoCE QoS as a four-step chain, not a setting. The application sets a ToS with rdma_set_option(..., RDMA_OPTION_ID_TOS, value); a fixed kernel table converts that ToS into a socket priority; tc (or the VLAN egress map) converts the socket priority into a user priority; and mlnx_qos or lldpad converts the user priority into a traffic class on the NIC.[2] The plain-Ethernet chain is the same shape and differs only at step 1, where the application calls setsockopt(IP_TOS, value), and at step 3, where the VLAN egress_map does the work.[2]

The first link is the one that fails most often, and it fails silently: “If a TOS is not set, the default value (0) is used”.[1] A flow with ToS 0 is DSCP 0, priority 0, and it will sit in the default queue on a fabric where every switch was configured perfectly for priority 3. NCCL behaves the same way — NCCL_IB_TC defaults to 0 and NVIDIA’s guidance is only to “set NCCL_IB_TC per vendor documentation”, so an unset variable is an unmarked training job.[7]

The second link has a hard ceiling that surprises people. The ToS-to-socket-priority table is fixed: TOS 0 <=> sk_prio 0, TOS 8 <=> sk_prio 2, TOS 24 <=> sk_prio 4, TOS 16 <=> sk_prio 6.[2] Because that table is fixed, “With RoCE, there can only be 4 predefined ToS values for the purpose of QoS mapping.”[2] The documented escape hatch is to skip the ToS entirely: “Socket applications can use setsockopt (SK_PRIO, value) to directly set the sk_prio of the socket. In this case, the ToS to sk_prio fixed mapping is not needed.”[2]

Rendering diagram…
Diagram source (Mermaid)
flowchart LR; A["1. App: rdma_set_option ID_TOS - default 0"] --> B["2. Fixed table: ToS 0/8/24/16 to sk_prio 0/2/4/6"]; B --> C["3. sk_prio to UP: tc filter or VLAN egress_map"]; C --> D["4. UP to TC: mlnx_qos -p or lldpad"]; D --> E["Wire: DSCP always present - PCP only if VLAN tagged"]; E --> F["Switch trust state picks which field it reads"]; A -. "setsockopt SK_PRIO skips the four-value limit" .-> C; G["Force DSCP: sysfs traffic_class"] -. "re-marks without an app change" .-> E
The documented RoCE QoS chain. Two dashed edges are the bypasses: SK_PRIO from inside the app, Force DSCP from outside it.

2The arithmetic, quoted exactly

Three formulas carry the whole chain, and all three are on NVIDIA’s RoCE page. DSCP to ToS: “The ToS field is 8 bits, while the DSCP field is 6 bits. To set a DSCP value of X, you need to multiply this value by 4 (SHIFT 2). For example, to set DSCP value of 24, set the ToS bit to 96 (24x4=96).”[1][3] ToS to SL: “SL = TOS >> 5 (e.g., take the 3 most significant bits of the TOS field)”.[1] SL to PCP: “PCP = SL & 7 (take the 3 least significant bits of the TOS field)”.[1]

Run the numbers for the three DSCP values you will meet on an AI fabric. DSCP 26 is the Cumulus default for RoCE data and gives ToS 104; DSCP 48 is the default for CNP and gives ToS 192; DSCP 24 is what the Spectrum-X host profile uses and gives ToS 96.[5][8][1] Neither ToS 96 nor ToS 104 is an entry in the fixed table - its only entries are ToS 0, 8, 24 and 16 - so on the host neither DSCP 24 nor DSCP 26 reaches a socket priority through the ToS-to-sk_prio path at all.[2] Both depend instead on DSCP trust plus the NIC’s dscp2prio map, which is why the Spectrum-X host profile sets Trust: dscp alongside its ToS of 96; the profile states the values and gives no rationale for choosing 24 over 26.[8]

The last formula only matters on tagged frames. NVIDIA states it twice: “SL affects the PCP only when the traffic goes over tagged VLAN frames”, and separately, “VLAN tagged Ethernet frames carry a 3-bit priority field. The value of this field is derived from the IB SL field by taking the 3 least significant bits of the SL field”.[1] On an untagged routed fabric — which is what a rail-optimized AI network normally is — PCP is not a weak signal, it is an absent one.

DSCP
ToS 104 · SL 3 · PCP 3 · prio 3NVUE default for RoCE data → switch priority 3
wire order — first byte on the wire at the topEthernet MAC headerethertype = IPv4 / IPv614 BIPv4 headerDSCP 26 · ECN bits 6–720 BDSCP (6 bits) + ECN (2 bits)UDP headersport = flow hash · dport 47918 BBTH ⚠opcode · dest QP · PSN · FECN/BECN12 BRDMA payloadpath MTU 256…4096 (RC/UC)≤ MTUICRC ⚠rx_icrc_encapsulated4 BEthernet FCSrecomputed at every hop4 B
step 1 / 6marking survives
marking chain · RoCEv2 · trust dscp

1. App sets a ToS

Carried by: nothing on the wire yet — this lives in the address vector

RDMA-CM apps set it per connection with rdma_set_option(..., RDMA_OPTION_ID_TOS, value); "If a TOS is not set, the default value (0) is used." The CM default can also be moved without touching code: cma_roce_tos -d <ib_device> -t <TOS>, or the RDMA-CM configfs entry. An app that calls rdma_set_option still overrides both.

ib_write_bw -T 104 -R        # CM path
ib_write_bw --tclass=104      # AV path, no CM
# no code change at all:
echo "tclass=104" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
echo "tclass=16,src_ip=1.1.1.2,dst_ip=1.1.1.0/24" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
echo "tclass=-1" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class   # delete
FAE angle

Failure mode #1 of three: the app never set a ToS, so the traffic is DSCP 0 / priority 0 no matter how good the switch config is. Force DSCP via .../tc/<port>/traffic_class proves it in one command — global wins over rules, negative values disable, valid range 0–255. The sibling file is ttl.

Verdict for this frame

trust dscp with DSCP 26 on the wire: the NIC classifies on the IP header and the default dscp2prio table puts it on priority 3. This is the configuration NVUE expects on the switch side.

Counter that proves it: ethtool -S <netdev> | grep prio3 climbing, and rx_prio3_pause / tx_prio3_pause telling you which side is pausing.

Arithmetic for DSCP 26
ToS = DSCP × 4 = 104 · SL = ToS >> 5 = 3 · PCP = SL & 7 = 3 (no VLAN tag → not on the wire)
default dscp2prio → priority 3 · sk_prio: n/a — ToS 104 is not one of the four table values (0, 8, 16, 24)
DOCA — Ethernet QoS · Cumulus — RoCE defaults

⚠ marks a number no fetched NVIDIA or IETF page states: the GRH, BTH and ICRC byte sizes (the IBTA spec is members-only) and whether a switch classifies RoCEv1 on the GRH traffic-class byte.

Marking overlay on a RoCEv2 frame. Turn the VLAN tag off and on to watch the PCP field appear and disappear, then flip trust between dscp and pcp and see which marking the switch is willing to read.

3The three places the mark disappears

Every “the switch is not putting my RoCE in the right queue” ticket lands on one of three causes. First, the application never set a ToS, so the flow is DSCP 0 and priority 0 no matter what the fabric does.[1] Second, the host or the switch is in trust pcp while the fabric is DSCP-based, so the DSCP the application set is ignored on the way in — PCP is the default trust state on the NIC, so this is the state you get by not choosing.[2] Third, there is no VLAN tag, so PCP never appears on the wire at all and a PCP-trust device has nothing to read.[1]

Those three are distinguishable with one capture and one command. A capture of UDP port 4791 shows the DSCP the frame actually carries, which separates cause one from causes two and three; mlnx_qos -i <interface> prints the host’s Priority trust state line, which separates cause two from cause three.[2] Note the ordering: the capture is the cheap test and it is decisive for the most common cause, so it goes first.

The default DSCP-to-priority table on the NIC matters here because it is already sensible and people re-invent it. Reading it off the canonical mlnx_qos output: DSCP 0-7 map to priority 0, 8-15 to 1, 16-23 to 2, 24-31 to 3, 32-39 to 4, 40-47 to 5, 48-55 to 6, and 56-63 to 7.[2] That is why DSCP 26 lands on priority 3 and DSCP 48 lands on priority 6 with no --dscp2prio configuration at all, and it is exactly the split Cumulus uses on the switch side: RoCE data on switch priority 3 with DSCP 26, CNP on switch priority 6 with DSCP 48.[5]

4Re-marking without touching the application

When the application cannot be changed — a customer binary, a container image under change control, a benchmark you did not write — NVIDIA exposes a driver-level override. A global traffic class is written to /sys/class/infiniband/<dev>/tc/<port>/traffic_class; “Negative values indicate that the feature is disabled” and “Valid values range between 0 - 255”.[1] The rule form takes source and destination IPs, verbatim from the page:[1]

echo "tclass=16,src_ip=1.1.1.2,dst_ip=1.1.1.0/24" > /sys/class/infiniband/mlx5_0/tc/1/traffic_class

The tclass= prefix is optional, a mask may be given on the destination IPv4, and "tclass=-1" removes a rule from the database.[1] Precedence is stated and is not what most engineers assume: a global traffic class wins over all rules; otherwise a rule matching both source and destination IP applies; rules matching only one of the two “have no defined precedence”; “The rule precedence is not affected by the order in which rules are inserted”; and “Overlapping rules are entirely up to the administrator.”[1] A sibling file, .../tc/<port>/ttl, forces the TTL on RC QPs with the same 0-255 range.[1]

For test traffic you have two marking paths in perftest and they are not interchangeable. -T, --tos=<tos value> is documented as “Set <tos_value> to RDMA-CM QPs. available only with -R flag”, so it exercises the CM path; --tclass=<value> is “Set the Traffic Class in GRH (if GRH is in use)” and marks on the address-vector path without the CM.[4] Running both against the same peer is a clean way to show a customer that their application’s problem is the CM’s default ToS and not the fabric.

Prove which link broke, then fix it without a code change

Symptom: a customer’s inference service pushes RoCE at line rate but the leaf’s priority-3 queue counters never move. You have SSH on both hosts and read access to the switch.

  1. Capture on the sender and read the outer IP ToS byte. A RoCEv2 frame is UDP with destination port 4791, so the filter is unambiguous:[1] 
    sudo tcpdump -i ens1f0 -nn -v -c 20 udp port 4791
    Expected on a correctly marked flow: tos 0x68 in the IP header, which is 104 decimal, which is DSCP 26.[1] If you see tos 0x0, the application never marked and you can skip every switch question.
  2. Read the host trust state and the DSCP map:[2] 
    mlnx_qos -i ens1f0
    Expected: a Priority trust state: line reading dscp or pcp, and the eight-line dscp2prio mapping: block. On defaults, DSCP 24-31 maps to priority 3.[2]
  3. Decide from the two artifacts. ToS 0 on the wire means cause one - the application. Correct ToS with trust pcp on an untagged frame means causes two and three together - the mark is present but nothing is reading it. Correct ToS with trust dscp means the host is clean and the investigation moves to the switch’s own DSCP map.[2][5]
  4. Prove it without a code change. Set a global traffic class of 104 on the device and re-run the same unmodified workload:[1] 
    cat /sys/class/infiniband/mlx5_0/tc/1/traffic_class      # record the before value first
echo 104 | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
    Expected: the capture in step 1 now shows tos 0x68 for the same application. That single line converts an argument into a measurement.
  5. Roll back. Restore the recorded value, or disable the feature entirely by writing a negative value, which NVIDIA defines as “disabled”:[1] 
    echo -1 | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
  6. Report the fix in the customer’s own terms: either the application must call rdma_set_option with ToS 104 (or set NCCL_IB_TC if it is NCCL), or the override stays as a documented host setting.[1][7]

Episode 1 — Ninety minutes later

How it ended

You write a global traffic class into /sys/class/infiniband/<dev>/tc/<port>/traffic_class on both hosts, re-run the capture, and the outer ToS byte reads 104 — DSCP 26, which the default host map puts on priority 3.[1][2] The leaf’s priority-3 counters move for the first time; no switch changed and their binary was never touched. What you say in the room: the fabric was configured correctly, the application never marked, and the mark belongs in the code or in NCCL_IB_TC before acceptance.[7] The network lead writes one line in her grid-paper notebook, then asks why one direction of that same pair still runs at half rate.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Goal: on the Dell-lab ConnectX or BlueField-3 pair, prove the override path end to end. Pre-flight first; every mutating step names its rollback.

  1. Pre-flight inventory, read-only. Record all of this into the ticket before changing anything: 
    ibdev2netdev
ibv_devinfo -d mlx5_0 | grep -E "link_layer|active_mtu|fw_ver"
mlnx_qos -i <if> > /tmp/qos-before.txt
cat /sys/class/infiniband/mlx5_0/tc/1/traffic_class
cat /sys/class/infiniband/mlx5_0/tc/1/ttl
    Expected: link_layer: Ethernet, and a traffic_class value that is negative (feature disabled) on a stock host.[1]
  2. Baseline capture with an unmarked run: server ib_write_bw -d mlx5_0, client ib_write_bw -d mlx5_0 <server-ip>, and sudo tcpdump -i <if> -nn -v -c 10 udp port 4791 alongside. Expected tos 0x0.[1]
  3. Mutating - global Force DSCP. Rollback: rewrite the value recorded in step 1, or -1 if it was negative.[1] 
    echo 104 | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
# re-run the SAME unmodified ib_write_bw command and capture again
# rollback:
echo -1 | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
    Expected: tos 0x68 on an application you did not change. If it does not change, check that you wrote to the port that carries the traffic — the path is tc/<port>/, and port 1 is not always the one in use.[1]
  4. Mutating - rule-based Force DSCP, scoped to one peer. Rollback is the same rule with tclass=-1:[1] 
    echo "tclass=104,src_ip=<local-ip>,dst_ip=<peer-ip>/32" | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
# rollback:
echo "tclass=-1,src_ip=<local-ip>,dst_ip=<peer-ip>/32" | sudo tee /sys/class/infiniband/mlx5_0/tc/1/traffic_class
    Expected: only traffic to that peer is re-marked. Confirm the documented precedence by leaving the global value disabled during this step — a global value would win over the rule.[1]
  5. Compare the two marking paths on the same pair, read-only: ib_write_bw -R -T 104 <server-ip> (CM path) against ib_write_bw --tclass=104 <server-ip> (AV path).[4] Expected: the same ToS on the wire by two different mechanisms. Note which one your customer’s application actually uses — an application on RDMA CM inherits the CM’s ToS, not the AV’s.[1]
  6. Close out: confirm cat /sys/class/infiniband/mlx5_0/tc/1/traffic_class is back to the step-1 value and that mlnx_qos -i <if> still matches /tmp/qos-before.txt.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell platform engineer, in five sentences, why a RoCE application that 'set QoS' can still land in the default switch queue, and name the one command you would run first to prove which link in the chain broke.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA-Host RoCE and Ethernet QoS pages (HTML + generated PDF), perftest master man page, Cumulus Linux 5.18 RoCE page, NCCL 2.31.2 env reference, 2026-09-07. Dates are when each page was fetched.

  1. RDMA over Converged Ethernet (DOCA-Host) - generated PDF · fetched 2026-09-07 · DOCA 3.5.0
  2. Ethernet QoS (DOCA-Host) - generated PDF · fetched 2026-09-07 · DOCA 3.5.0
  3. RDMA over Converged Ethernet (DOCA-Host, current) · fetched 2026-09-07 · DOCA 3.5.0
  4. perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
  5. RDMA over Converged Ethernet - RoCE (Cumulus Linux 5.18) · fetched 2026-09-07
  6. Quality of Service (QoS) - MLNX_OFED 5.9-0.5.6.0 · fetched 2026-09-07
  7. Environment Variables - NCCL 2.31.2 documentation · fetched 2026-09-07
  8. Spectrum-X Configuration - NVIDIA NIC Configuration Operator · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.