Packet trimming and `lossy-multi-tc`

1A third answer to a full buffer

Lessons 3.2 and 3.3 gave you two responses to a congested queue: pause the upstream port so nothing is lost, or mark CE and let the endpoints slow themselves down. Packet trimming is a third, and it only exists on newer silicon. NVIDIA’s description: “The Spectrum switch implements a packet trimming mechanism, with which the switch trims and forwards packets that are about to be discarded due to unavailable buffer space.”^[1]

The trimmed header still arrives. That is the whole idea. A dropped packet is invisible to the receiver until a retransmission timer expires — the receiver cannot distinguish “lost” from “late” and must wait. A truncated packet arrives with an obviously incomplete payload, so the receiver knows now that data was lost and can ask for it again immediately. Recovery moves from timeout scale to roughly one round trip.

That matters most for the traffic this course is about. A collective operation over an AI fabric runs at the speed of its slowest participant; one retransmission timeout on one flow stalls the whole step. NVIDIA’s Spectrum-X framing is that congestion should be absorbed by rerouting first, signalled by ECN second, and that dropping is the outcome to avoid — trimming is what makes a drop cheap when it becomes unavoidable.^[8]

Trimming arrived in Cumulus Linux 5.14, in the same release as the lossy-multi-tc RoCE mode that pairs with it.^[5][2] They are designed together: trimming is the mechanism, lossy-multi-tc is the RoCE configuration that puts the mechanism in the traffic’s path.

nv set qos roce mode lossy-multi-tc
nv set system forwarding packet-trim state enabled
nv set system forwarding packet-trim remark dscp port-level
nv set interface swp1-16  qos remark profile lossy-multi-tc-host-group
nv set interface swp17-32 qos remark profile lossy-multi-tc-network-group
nv config apply

nv set qos congestion-control default-global traffic-class 3 ecn enabled
nv set qos congestion-control default-global traffic-class 3 min-threshold 150000
nv set qos congestion-control default-global traffic-class 3 max-threshold 1500000
nv set qos congestion-control default-global traffic-class 3 probability 100
nv show interface swp16 counters qos pfc-stats

2Eligibility: the four rules that decide whether it applies at all

Trimming is narrower than it first appears, and every constraint is a design question rather than a tuning question.

ASIC. “Cumulus Linux supports packet trimming on Spectrum-4 and later.”^[1] A Spectrum-3 SN4700 will not trim on any release. This is the rule that ends most customer conversations, because it makes trimming a purchasing decision.

Traffic type. “Cumulus Linux supports packet trimming for known unicast IPv4 and IPv6 traffic. Packet trimming does not support ISSU, VXLAN packets or flooding and multicast packets.”^[1] Read the VXLAN exclusion twice. A tenant fabric that carries RoCE inside a VXLAN overlay gets no trimming for that traffic, which puts trimming and EVPN-based multi-tenancy in tension on the same leaf.

Traffic class. Trimming is enabled per port per traffic class: nv set interface <ports> packet-trim egress-eligibility traffic-class <0-7>, and the default profile enables traffic classes 1, 2 and 3 on all ports.^[1] Traffic class 0 is not in the default set, so bulk traffic is not trimmed unless you say so.

A port. Enabling trimming consumes one service port — the subject of Segment 5.^[1]

Before promising trimming to a customer, ask four questions in this order: which ASIC, is the RoCE traffic encapsulated, which traffic classes carry it, and which port can you afford to give up. Three of those four are answered by the design, not by the switch.

3The command surface and the default profile

The profile that ships is packet-trim-default: DSCP remark 11, truncation size 256 bytes, switch priority 4, and eligibility on all ports for traffic classes 1, 2 and 3.^[1] The configurable ranges are size 256-1024 bytes in multiples of 4, dscp 0-63, switch-priority 0-7 and egress-eligibility traffic-class 0-7.^[1]

nv set system forwarding packet-trim state enabled
nv set system forwarding packet-trim profile packet-trim-default
nv set system forwarding packet-trim dscp <0-63>
nv set system forwarding packet-trim size <256-1024>
nv set system forwarding packet-trim switch-priority <0-7>
nv set interface <port> packet-trim egress-eligibility traffic-class <0-7>
nv config apply

^[1] Monitoring and clearing:

nv show system forwarding packet-trim
nv show system forwarding packet-trim counters
nv show interface <port> counters packet-trim
nv action clear system forwarding packet-trim counters
nv action clear interface <port> packet-trim counters

Two of the default values are worth reasoning about rather than memorising. 256 bytes is enough for Ethernet, IP, UDP and the RoCE transport headers with room to spare, which is what the receiver needs to identify the flow and the sequence position; making it larger buys nothing for recovery and costs bandwidth on an already congested port. Switch priority 4 puts trimmed packets on a class of their own, away from both the untrimmed RoCE traffic on priority 3 and the CNP traffic on priority 6 — so a burst of trimmed headers cannot itself congest the class that produced them.^[1][2]

4lossy-multi-tc: putting trimming in the RoCE path

lossy-multi-tc is the RoCE mode built for trimming. The documented Spectrum-X pattern from the 5.18 RoCE page:

nv set qos roce mode lossy-multi-tc
nv set system forwarding packet-trim remark dscp port-level
nv set interface swp1-16  qos remark profile lossy-multi-tc-host-group
nv set interface swp17-32 qos remark profile lossy-multi-tc-network-group
nv config apply

One thing the block does not show is an explicit enable, because it does not need one: NVIDIA states that the lossy multi TC profile “provides QoS settings for lossy RoCE traffic and enables Packet trimming with the default packet trimming profile settings”.^[2] nv set system forwarding packet-trim state enabled before the mode is belt-and-braces, not a requirement — which is why the documented five-line set is the one to reproduce in an exam or a runbook.

^[2] Read it as four decisions. The mode selects a lossy RoCE configuration with more than one traffic class in play. remark dscp port-level switches trimmed-packet remarking away from the single global DSCP value so that the per-interface profiles are consulted instead. Then two predefined profiles are applied to two port ranges: lossy-multi-tc-host-group sets DSCP 21 and lossy-multi-tc-network-group sets DSCP 11, both operating at switch priority 4 — re-verified on the Packet Trimming page at write time.^[1]

The reason the two directions differ is topology. A trimmed packet heading toward a host must be recognisable to the receiving NIC as a trim notification; a trimmed packet heading deeper into the fabric must be recognisable to the next switch. Different consumers, different marking, hence a profile per port role rather than one switch-wide value. The port ranges in the example — swp1-16 host-facing, swp17-32 network-facing — are the example’s, not a rule; substitute the customer’s actual port roles.^[2]

Note what lossy-multi-tc gives up: it is a lossy mode, so PFC is not pausing on switch priority 3 and the fabric relies on ECN plus trimming to keep flows recovering fast. That is the trade the mode exists to make. It is also worth checking what your validation tooling supports before committing to it: netq check roce test 0 checks “whether RoCE is configured for lossy or lossless mode” and test 1 checks classification consistency, so a mixed fleet where one leaf runs lossy-multi-tc and the rest run lossless will show up there as an inconsistency — which is what you want.^[9]

5What it costs: a port and a counter you do not get

Enabling trimming consumes a physical port. NVIDIA states plainly: “When you enable packet trimming, one service port is used. By default, this is the last service port on the switch” — a designated port, not whichever port happens to be free — and it is movable with nv set system forwarding packet-trim service-port <interface-id>. NVIDIA lists what must not be configured on it: packet trimming port eligibility, port security, adaptive routing, QoS, ACLs, PTP, VRR, PBR, telemetry and histograms.^[1] That list was re-read at write time and is longer than older summaries suggest: adaptive routing and port security are on it.

For a design review this is concrete. On an SN5600 the customer loses whichever port the switch designates as its service port — read the service-port field out of nv show system forwarding packet-trim rather than assuming it is the highest-numbered data port — and anything they had planned for it. If that port carried an uplink with adaptive routing enabled, or an ACL, or a PTP boundary clock, that configuration has to move somewhere else before trimming is enabled. Discovering it afterwards is a second maintenance window.

The counter limitation is the other half. “Spectrum-4 switches display only the total number of trimmed packets per port and do not provide counters for the number of trimmed packets that were sent successfully on a port.”^[1] So nv show interface <port> counters packet-trim answers “how much congestion reached the trimming threshold on this port”, not “did the recovery mechanism work”. Proving recovery works is a host-side measurement — throughput held up, retransmission behaviour sane — not a switch counter.^[4]

Finally the lab boundary. None of this is exercisable in simulation: a virtual Cumulus switch implements no QoS shaping, buffer management or packet marking, so a DSX Air node will accept the configuration and show it back to you without ever trimming anything.^[7] That makes this lesson read-and-reason by design, and it is worth being explicit with a customer about which of your claims come from documentation and which from measurement. NCP-AIN objective 2.2 covers enabling and verifying QoS on Spectrum switches; trimming is inside that scope and the counter limitation is exactly the kind of detail that separates configuration from verification.^[10]