How a RoCE packet gets classified

1RoCEv2 is UDP, and the switch does not classify on the port

RoCEv2 rides on UDP. On a Cumulus switch with RoCE enabled the LLDP application TLV advertises UDP protocol 4791 together with the RoCE priority, so an attached NIC can learn which priority this fabric expects.^[1] That TLV is an advertisement, not a classifier: the switch’s own data-path decision is made on the packet’s marking, not on the UDP port.^[1][2]

The marking lives in one byte. In IPv4 that is the TOS octet, in IPv6 the Traffic Class octet, and RFC 3168 splits it: the top six bits are DSCP, and “bits 6 and 7” are the ECN field.^[4] The four ECN codepoints are 00 Not-ECT, 01 ECT(1), 10 ECT(0) and 11 CE, and the RFC is explicit about who sets the last one: “The CE codepoint ‘11’ is set by a router to indicate congestion to the end nodes”, and “For a router, the CE codepoint of an ECN-Capable packet SHOULD only be set if the router would otherwise have dropped the packet”.^[4]

Those two fields are independent. A switch that marks CE on a congested queue rewrites the low two bits and leaves the six DSCP bits alone, so classification upstream and downstream is unchanged. Engineers who have only ever seen the byte as a single number reliably get this wrong and report that “ECN changed our QoS marking”.

The practical consequence for a fabric: RoCE classification is an agreement between the host and the switch about DSCP values, enforced independently at every hop. Nothing in the packet says “I am RDMA”; the DSCP says “I belong in the lossless class”, and every switch on the path must agree what that means.^[1][7]

nv set qos roce state enabled
nv set qos roce mode lossless
nv config apply

nv set qos mapping default-global trust both
nv set qos mapping default-global dscp 26 switch-priority 3
nv set qos mapping default-global port-default-sp 0

2Trust: which marking the switch believes

Trust mode is the switch’s answer to “whose marking do I read?”. NVUE sets it per mapping profile: nv set qos mapping <profile-id> trust l2|l3|port|both, with pcp <value> switch-priority <sp>, dscp <value> switch-priority <sp> and port-default-sp <sp> to change the tables themselves.^[3] The semantics are: 802.1p (L2) maps PCP 0-7 directly onto switch priorities 0-7; DSCP (L3) groups the 64 DSCP values into eight ranges that map to switch priorities 0-7, so DSCP 0-7 becomes switch priority 0; port ignores markings entirely and classifies every frame to the port’s default priority.^[2]

The fourth value is the one that catches people. With both configured, the switch “prioritizes DSCP for IP traffic while applying 802.1p to non-IP frames” — verified on the 5.18 QoS page at write time.^[2] That is exactly what you want on a mixed fabric, and exactly what makes a PCP-only lab test look like it did nothing.

Enabling RoCE sets trust for you, and it is worth being precise about what it sets. The RoCE defaults table lists “Port trust mode: YES” for both lossy and lossless mode - that row means trust is enabled on the port, not that the mode is port - and nv show qos roce reports the resulting setting as trust-mode pcp,dscp: the switch trusts DSCP for IP traffic and PCP for the rest, which is the opposite of the port mode described above.^[1] The per-interface status view reports the same trust-mode field so you can read back what the one-liner did rather than infer it.^[6]

3The RoCE default tables

When RoCE is enabled, Cumulus installs a specific set of mappings. The DSCP table sends DSCP 24-31 to switch priority 3 (RoCE data) and DSCP 48-55 to switch priority 6 (congestion notification packets).^[1] The switch-priority table then sends SP3 to traffic class 3 and SP6 to traffic class 6, and every other switch priority to traffic class 0.^[1]

The two RoCE classes are scheduled differently on purpose. Traffic class 3 gets WRR 50% and traffic class 6 gets strict priority, in both lossy and lossless mode.^[1] Strict priority for CNP is the point: a congestion notification that queues behind the congestion it is reporting is useless, so it jumps every other class.

Outside RoCE the mapping is trivial: “By default switch priority values map directly to the matching egress queue”, so switch priority 0 leaves on egress queue 0.^[2] The default egress scheduler outside RoCE gives 12% to queues 0, 2, 4 and 6 and 13% to queues 1, 3, 5 and 7.^[2] You can move any of this with nv set qos egress-queue-mapping <profile> switch-priority <sp> traffic-class <tc> and nv set qos egress-scheduler <profile> traffic-class <tc> mode dwrr|strict.^[3]

Two numbers are worth memorising as a pair, because they are the fastest classification check you can do in a customer’s terminal: 3 for data, 6 for CNP, DSCP ranges 24-31 and 48-55. NCP-AIN objective 2.1 is “configure Spectrum-X switches for RoCE”, and this table is the part of it you will be asked to reproduce from memory.^[9]

4From traffic class to pool to queue

A traffic class is not yet buffer. Each class draws from a traffic pool, and pools are sized as a percentage of switch buffer memory: nv set qos traffic-pool <pool-id> memory-percent <value> and nv set qos traffic-pool <pool-id> switch-priority <sp>.^[3] Out of the box, and this is the fact that surprises people, “Cumulus Linux allocates 100% of the buffer memory to the default-lossy traffic pool by default”.^[2] A lossless class with no lossless pool behind it is a mapping with nowhere to store a burst.

Enabling RoCE creates the pools, and the documented tuning pattern splits them explicitly — 40% to default-lossy and 60% to roce-lossless — which lesson 3.2 runs as a change window.^[1] The nv show interface <if> qos roce status view reports the pool map alongside the mode and the mappings, so the pipeline is inspectable end to end from one command.^[6]

The last stage is the egress queue, where the scheduler decides service order and where ECN marking and WRED drops are counted. nv show interface <if> counters qos egress-queue-stats breaks out per-traffic-class transmitted frames and bytes, unicast buffer discards and WRED drops — the read that tells you whether the class you built is actually carrying the traffic you think it is.^[6]

One scope warning for lab work: a virtual Cumulus switch does not implement QoS shaping, buffer management or packet marking.^[8] In simulation you can build and read the mapping tables, and that is genuinely useful, but any “verification” of buffering or marking there is verifying nothing.^[8]

5The host side, and the arithmetic that bites

The switch’s table is only half the agreement. NVIDIA’s own host-side script doRoCE.sh sets the NIC’s trust mode to dscp with mlxreg -y -d {bdf} --reg_name QPTS -i "local_port=1" --set "trust_state={trust_val}", and writes the verbs traffic class to /sys/class/infiniband/{dev}/tc/1/traffic_class, with RDMA-CM defaults under /sys/kernel/config/rdma_cm/ and default_roce_tos.^[5]

Its documented defaults are TOS/DSCP 106, GID index 3, trust dscp.^[5] Read that carefully. 106 is a TOS value, and TOS is DSCP shifted left by two bits to make room for the ECN field. So 106 >> 2 = DSCP 26 — which sits inside the switch’s default 24-31 range and therefore lands on switch priority 3.^[5][1][4] The number is correct; the label on it is the trap. An engineer who reads “106” as a DSCP and configures the switch for DSCP 106 has configured an out-of-range value, and one who “fixes” the host to TOS 26 has just marked DSCP 6, which lands in traffic class 0 with the bulk traffic.

Field procedure: never trust either side’s configured value on its own. Capture the flow and read the byte. On the receiver, tcpdump -v on the UDP 4791 flow prints the TOS octet, and the DSCP is that value divided by four. That capture is the only artefact that proves host and switch agree.^[5][1]