`nv set qos roce`: the one command and its five modes
S3·E2Five minutes, low risk · Hotel lobby near the customer's campus, nine days out
Builds on: How a RoCE packet gets classified
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Enable RoCE on a Cumulus 5.18 switch and select the correct mode from lossless, lossy, lossless-single-ipool, lossy-multi-tc and dci-1.
- List what the single command configures - buffer pools, classification and priority mappings, PFC and ECN thresholds, and scheduling for regular, RoCE and CNP traffic.
- Split buffer memory between the default-lossy and roce-lossless traffic pools and justify the split you chose.
- Write a change-window script that applies RoCE with an automatic rollback and names the risks of a mode change.
Episode 2 — Five minutes, low risk
Pod A is racked in the customer’s cage, and at nine at night the network lead circulates tonight’s change record: “Enable RoCE on the remaining three leaves. Duration: five minutes. Risk: low.” The Dell SE forwards it with one word — “ok?” — and, under it, the procurement person’s only question, which is whether any of this moves the lead time on the second order.
It is not ok, and the reason is worth being precise about. The typing really is one line, and nv set qos roce with no mode is identical to nv set qos roce mode lossless, because lossless is the default.[1] But that line sets buffer pool settings, traffic classification and priority mappings, PFC and ECN thresholds, and queuing and scheduling for regular, RoCE and CNP packets, all at once.[4] Rewriting how a live switch divides its buffer memory is not a low-risk five-minute slot.
The command exists in that shape on purpose. Those five things are only correct together — classification without a lossless pool sends traffic to a class with no buffer — so NVIDIA ships the consistent set as one verb and leaves the individual knobs underneath.[4]
Two things are missing from the record: no confirm window, and no buffer split, which leaves every leaf at 100% of buffer memory in the default-lossy pool.[3] One line of typing is not one line of change. You rewrite it in the lobby. Segment 1 is the first thing you check.
1Off by default, lossless when you turn it on
Two defaults do most of the damage in the field. nv set qos roce state takes enabled or disabled and its default is disabled — RoCE QoS is off on a factory switch, and has been since Cumulus Linux 5.0.0.[2] And nv set qos roce with no mode is identical to nv set qos roce mode lossless, because lossless is the default mode.[1][2]
Put together: a customer whose leaf has never had nv set qos roce run on it has no RoCE QoS at all, not a partial configuration. Their RoCE traffic still forwards, classified into traffic class 0 with everything else, which is why “it works, it’s just slow under load” is the usual presenting symptom.
The full surface is small:
nv set qos roce # lossless (default mode)
nv set qos roce mode lossy
nv set qos roce mode lossless-single-ipool
nv set qos roce mode lossy-multi-tc
nv set qos roce mode dci-1 # Beta, Spectrum-4 through Spectrum-6
nv unset qos roce
nv config applyThat block is quoted from the 5.18 RoCE page.[1] One documentation wrinkle worth knowing before you write a runbook: the NVUE reference lists the mode values as lossy, lossless, lossless-single-ipool and dci-1, while the 5.18 RoCE page additionally documents lossy-multi-tc.[2][1] Both pages are current; cite both when you list the modes for a customer.
Traffic pool — roce-lossless
Configure the roce-lossless pool when you are using RoCE, otherwise configure default-lossless. Cumulus Linux allocates 100% of the buffer memory to the default-lossy traffic pool by default, so enabling RoCE without splitting the pools leaves the lossless class with nothing to reserve. Current split in this panel: default-lossy 100% / roce-lossless 0%.
FAE angle Rising `no-buffer-discard` on a lossless fabric usually means the pool was never split — the RoCE class is drawing from a pool sized for best-effort traffic. Second read: `cable-length` is still 100 m on a rack of 2 m DACs, reserving headroom nobody needs.
PFC on switch priority 3 plus ECN on TC3. The buffer must never overflow for RoCE: the switch pauses upstream before it would drop.
nv set qos roce state enabled nv set qos roce mode lossless nv config apply
nv set qos traffic-pool default-lossy memory-percent 100 nv set qos traffic-pool roce-lossless memory-percent 0 nv set qos traffic-pool roce-lossless switch-priority 3 nv show qos buffer
- Cumulus Linux allocates 100% of the buffer memory to the default-lossy traffic pool by default; the total across pools must not exceed 100%. source
- Typical RoCE split: nv set qos traffic-pool default-lossy memory-percent 40 plus roce-lossless memory-percent 60. source
- Configure the roce-lossless pool when you are using RoCE, otherwise configure default-lossless. source
- PFC defaults: xoff 10,000 bytes, xon delta 2,000 bytes, port buffer 25,000 bytes, recommended cable length 50 m — against `nv set qos roce cable-length` default 100 m. source
- PFC RX and TX are enabled in lossless mode and disabled in lossy mode. source
- One command sets buffer pools, classification and priority mappings, PFC and ECN thresholds, and scheduling for regular, RoCE and CNP traffic. source
RoCE (CL 5.18) · Quality of Service · Packet Trimming · NVUE QoS set
2The five modes, and what each one gives up
lossless is the default: PFC on switch priority 3 plus ECN on traffic class 3. PFC receive and transmit are enabled in lossless mode and disabled in lossy mode, which is the single line that separates the two.[1] The switch pauses the upstream port rather than dropping a RoCE packet.
lossy keeps the same classification and the same traffic-class mapping but sends no pause frames. Congestion is signalled by ECN only; past the buffer the switch drops and the NIC retransmits. The defaults table shows the ECN threshold on traffic class 3 is the same in both modes, and the schedulers are identical — strict for traffic class 6, WRR 50% for traffic class 3.[1]
lossless-single-ipool is a lossless variant using a single shared ingress pool. Its operational cost is not on the switch: NetQ supports lossless and lossy RoCE modes but does not support lossless-single-ipool, so a customer running it gets no netq check roce validation at all.[5] That is a design-review question, not a troubleshooting one.
lossy-multi-tc is the packet-trimming partner, added in Cumulus Linux 5.14 alongside trimming itself; lesson 3.4 is entirely about it.[8][1]
dci-1 is the Data Center Interconnect profile: NVIDIA documents it as “a Beta feature … supported on Spectrum-4 through Spectrum-6”, running mixed classes — traffic class 3 with PFC for lossless traffic inside the data centre, traffic class 4 with ECN for lossy traffic between data centres.[1] Beta means Beta, and the support window has an upper bound as well as a lower one — say both in a customer conversation.
3What the one command actually does
NVIDIA’s Ethernet Storage Fabric guide states it plainly: “Cumulus Linux simplifies the RoCE configuration to a single command and automatically applies best practice settings for optimal performance”, and that single command sets “buffer pool settings, traffic classification and priority mappings, PFC and ECN thresholds, and queuing and scheduling settings for regular, RoCE, and CNP packets”.[4]
Concretely, on a lossless switch that means: trust mode on; DSCP 24-31 to switch priority 3 and 48-55 to switch priority 6; switch priority 3 to traffic class 3 and 6 to traffic class 6 with everything else to traffic class 0; traffic class 3 scheduled WRR 50% and traffic class 6 strict; PFC receive and transmit on switch priority 3; and an ECN absolute threshold on traffic class 3 of 1,501,500 bytes.[1] The LLDP application TLV starts advertising UDP protocol 4791 with the RoCE priority.[1]
One knob rides along with the mode and is easy to forget: nv set qos roce cable-length, in metres, range 1 to 100000, default 100.[2] It sizes PFC headroom — the buffer reserved to absorb what is already in flight when an XOFF is sent. On a rack of 2 m DACs the default reserves headroom for cable that does not exist; on a 300 m campus run it under-reserves and the fabric drops anyway. Lesson 3.3 does the arithmetic.
Read back what you did rather than assuming: nv show qos roce for the system view, nv show interface <if> qos roce status per interface, and nv show interface qos-roce-status for all of them at once.[1][6]
4Splitting the buffer
Buffer memory is divided into traffic pools, each sized as a percentage: nv set qos traffic-pool <pool-id> memory-percent <value>, with nv set qos traffic-pool <pool-id> switch-priority <sp> to say which priorities draw from it. Both were introduced in Cumulus Linux 5.3.0.[2] The starting point is lopsided: “Cumulus Linux allocates 100% of the buffer memory to the default-lossy traffic pool by default”, and the total across pools must not exceed 100%.[3]
The documented RoCE split is:
nv set qos traffic-pool default-lossy memory-percent 40
nv set qos traffic-pool roce-lossless memory-percent 60
nv config apply[1] The pool name matters: NVIDIA’s guidance is “Configure the roce-lossless pool when you are using RoCE, otherwise configure the default-lossless pool”.[1] Getting that wrong produces a valid configuration that reserves memory for a pool your RoCE traffic does not use.
60/40 is a starting point, not a law. The right split depends on how much of the fabric’s traffic is RDMA: a dedicated backend GPU fabric carries almost nothing else and can push the lossless share higher, while a converged fabric that also carries storage and tenant traffic needs the lossy pool. What you cannot do is leave it at 100/0 and expect lossless behaviour under burst, because no-buffer-discard will climb on traffic class 3 while every mapping looks correct.[6]
5Applying it without losing the fabric
NVUE stages changes: nv set writes to pending, nv config apply moves pending to applied, nv config save overwrites the startup configuration in /etc/nvue.d/startup.yaml.[7] Read the version note before you rely on that split: in Cumulus Linux 5.9 and later auto-save is on by default and nv config apply “automatically saves the configuration and the configuration persists after a reboot”, so on a 5.18 switch a reboot is not a rollback. nv config save matters only where auto-save has been turned off with nv set system config auto-save enable off.[7] Three verbs make a QoS change survivable.
nv config diff shows the differences between configurations, so you read what you are about to commit before committing it.[7] nv config apply --confirm “applies the configuration change, but you must confirm the applied configuration. If you do not confirm within ten minutes, the configuration rolls back automatically”; the window is adjustable with a time argument in seconds, minutes or hours, and nv config apply --confirm-status “shows the amount of time left before the automatic rollback”.[7] nv config detach discards everything staged since the last apply.[7]
The order that matters: apply with --confirm, verify from a different session that the fabric is still healthy, then confirm, then save. With the 5.18 default of auto-save on, the confirm timer is the only rollback you have, which is why --confirm is not optional on a remote leaf.
Two more scope facts before you write the runbook. RoCE configuration is ASIC-specific and not transferable between Spectrum generations, so a startup.yaml from an SN4700 is not a migration path to an SN5600.[1] And a virtual Cumulus switch implements no QoS shaping, buffer management or packet marking, so DSX Air validates your syntax and your revision workflow but tells you nothing about buffering.[9] NCP-AIN objective 2.1 asks you to configure Spectrum-X switches for RoCE; the syntax half of that is exactly what a simulation can rehearse.[11]
Task: enable lossless RoCE with a 60/40 buffer split on leaf01, in a change window, with a rollback.
# 1. Evidence before. Save this output; it is your before-picture and your rollback reference.
nv show qos roce
nv show interface swp1 qos roce status
nv config show > /tmp/pre-roce-$(date +%F-%H%M).yaml
# 2. Stage the change (nothing has happened on the ASIC yet).
nv set qos roce state enabled
nv set qos roce mode lossless
nv set qos roce cable-length 3
nv set qos traffic-pool default-lossy memory-percent 40
nv set qos traffic-pool roce-lossless memory-percent 60
# 3. Read what you are about to commit.
nv config diff
# 4. Commit with an automatic rollback if you do not confirm.
nv config apply --confirm 5m
# 5. From a SECOND session, ideally over the OOB management port:
nv show qos roce
nv show interface swp1 qos roce status
nv show interface swp1 counters qos pfc-stats
nv config apply --confirm-status
# 6. Only when the fabric is verified healthy:
nv config apply --confirm # confirm the pending apply
nv config saveReasoning: cable-length 3 because these are in-rack DACs and the 100 m default would reserve headroom for cable that does not exist. The 40/60 split is NVIDIA’s published example and a defensible starting point. --confirm 5m shortens the ten-minute default because the verification above takes under a minute; if the change cost you management access, five minutes of outage is better than ten. nv config save is explicit here because auto-save can be disabled; on a 5.18 switch with the default auto-save on, the apply has already written /etc/nvue.d/startup.yaml and the confirm timer is your only rollback.
Rollback if verification fails: do nothing and let the confirm timer expire, or run nv config detach if you have not yet applied. If you already saved, restore by re-applying the captured /tmp/pre-roce-*.yaml and applying again.
Same change on leaf02, which uses 30 m fibre to a spine and carries both RDMA and tenant traffic.
- Capture the before-picture:
nv show ____ ____andnv config show > /tmp/pre-roce.yaml. - Enable and choose the mode:
nv set qos roce state ____andnv set qos roce mode ____. - Set headroom for the real cable:
nv set qos roce ____ 30. - Split the buffer, remembering which pool name RoCE uses:
nv set qos traffic-pool ____ memory-percent 50andnv set qos traffic-pool ____ memory-percent 50. - Read the staged change with
nv config ____. - Commit with a rollback:
nv config apply ____ 5m. - From a second session verify with
nv show qos roceand check the remaining time withnv config apply ____. - Confirm, then persist with
nv config ____. - State in one line why you chose 50/50 here and 60/40 on leaf01.
A Dell customer has four SN5600 leaves in a GPU backend fabric and one SN4700 leaf left over from a previous cluster, carrying two storage nodes on the same fabric. They send you the working startup.yaml from an SN5600 and ask you to “just apply the same thing to the SN4700 tonight”.
Produce: (a) your answer on whether the file can be reused, with the reason; (b) the change-window script for the SN4700, including which mode you choose and why the storage traffic affects the buffer split; (c) the exact verification commands you will run from the second session and the specific values you expect each one to show; (d) the rollback procedure at each of the three points where it differs (before apply, after apply but before confirm, after save).
Acceptance criteria: your answer to (a) cites the ASIC-portability rule rather than a guess about versions; the script uses --confirm with a stated window; the verification list includes at least one command from the per-interface status view and one counter read; and the rollback for “after save” is different from the rollback for “before confirm”.
Case update — the window holds
The rewritten record stages the mode, sets cable-length 3 for the in-rack DACs the night-shift operator has already been round labelling with his label maker, and splits the buffer 40/60 between default-lossy and roce-lossless — NVIDIA’s published example, not a number you invented.[1] It commits with nv config apply --confirm 5m and is verified from a second session on the management port before anyone confirms, because the session that breaks the fabric is the one that cannot tell you so.[7]
What you say to the network lead: “It is one command, and it re-lays the buffer on a live switch — so it gets a window and a rollback timer.” Pod A is clean at midnight. At 02:10 the operator calls: a job has stopped, and nothing is dropping.
Lab
Goal: configure the host end of the same fabric on a Dell-lab ConnectX or BlueField-3. Mutating — every step names its rollback.
Pre-flight inventory. Run all of it and save the output; it is the rollback baseline:
ibstat; ibdev2netdev
lspci | grep -i mellanox
sudo mlxreg -d <bdf> --reg_name QPTS -i "local_port=1" --get
sudo mlxreg -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --get
cat /sys/class/infiniband/<dev>/tc/1/traffic_class- Confirm the device and its netdev with
ibdev2netdev. Expected:mlx5_0 port 1 ==> ens1f0np0 (Up). If Down, stop. - Record the current trust state from the
QPTS --getoutput above. This is your rollback value. NVIDIA’sdoRoCE.shusestrust_val=2fordscp. - Set trust to DSCP:
sudo mlxreg -y -d <bdf> --reg_name QPTS -i "local_port=1" --set "trust_state=2". Expected: the command returns without error. Verify with--get. Rollback: re-run--setwith the value recorded in step 2. - Record the current PFC state from the
PFCC --getoutput. This is your rollback value. - Enable PFC on priority 3 (bit 3 of the mask, so
0x08):sudo mlxreg -y -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --set "prio_mask_rx=0x08,prio_mask_tx=0x08,pfctx=1,pfcrx=1,pprx=0,pptx=0". Expected: no error. Rollback: re-run--setwith every field restored to the step-4 values. - Verify it stuck:
sudo mlxreg -d <bdf> --reg_name PFCC -i "local_port=1,pnat=0,dcbx_operation_type=0" --get. Expected: the priority masks and pfctx/pfcrx read back what you set. If they read back zero, the NIC firmware may have DCBX in control — checkdcbx_operation_type. - Set the RoCE traffic class as in lesson 3.1:
echo 106 | sudo tee /sys/class/infiniband/<dev>/tc/1/traffic_class. Rollback: restore the pre-flight value. - Run a bandwidth test between the two hosts: receiver
ib_write_bw -d <dev> -F, senderib_write_bw -d <dev> -F <receiver-ip> --duration 20. Expected: a stable bandwidth figure. Record it; lesson 3.3 compares against it under congestion. - Read the NIC’s pause counters:
ethtool -S <ifname> | grep -i pause. Expected on an uncongested link: counters near zero. Non-zero pause with a single stream means something upstream is already pausing you.
Optional (customer lab, mutating, maintenance window only): apply lossless mode on the leaf facing these hosts using the worked change-window script above, including nv config apply --confirm 5m and the second-session verification. Do not run this on a production leaf without a change record.
Goal: rehearse the change window in DSX Air and document exactly what a simulation can and cannot tell you. Nothing here touches hardware.
- Create a simulation at
https://dsx-air.nvidia.com/simulationswith at least two Cumulus nodes and a link between them. Expected: both nodes boot andnv show systemreturns a version. If the nodes will not start, reduce the topology size. - On one node capture the before-state:
nv show qos roce. Expected: RoCE reported as disabled. Record the output. - Apply the change:
nv set qos roce state enabled,nv set qos roce mode lossless,nv set qos traffic-pool default-lossy memory-percent 40,nv set qos traffic-pool roce-lossless memory-percent 60, thennv config diffandnv config apply. - Read it back:
nv show qos roce,nv show interface swp1 qos roce status,nv show interface qos-roce-status. Expected: the mode, mappings and pool map render. - Now build the honesty table. For each field the show output prints — mode, enabled traffic classes, PFC state, trust mode, LLDP app TLV, min and max threshold, pool map, buffer usage, counters — mark it configuration (meaningful in simulation) or data plane (meaningless without the ASIC). A virtual Cumulus switch supports no QoS shaping, buffer management or packet marking, so every counter and occupancy field belongs in the second column.
- Exercise the rollback:
nv set qos roce mode lossy,nv config apply --confirm 1m, then do nothing. Expected: after the timer the configuration reverts to lossless; confirm withnv show qos roce. Watch the countdown withnv config apply --confirm-status. nv unset qos roceandnv config applyto return the node to its starting state.
Deliverable: the change-window script for a customer leaf, with the exact rollback command at each stage, plus the two-column honesty table you can hand to anyone who claims they “tested PFC in Air”.
Retrieval check
9 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell account SE, in five sentences, why 'RoCE is one command' is both true and misleading when they are quoting a change window to a customer.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 RoCE and QoS pages, NVUE 5.x QoS and Config references, NetQ 5.1 validation checks; re-fetched 2026-09-07. Dates are when each page was fetched.
- RDMA over Converged Ethernet - RoCE | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference - QoS set/unset commands · fetched 2026-09-07
- Quality of Service | Cumulus Linux 5.18 · fetched 2026-09-07
- Cumulus Linux Configuration Guide for Ethernet Storage Fabrics · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- NVUE Reference - QoS show commands · fetched 2026-09-07
- NVUE Reference - Config commands · fetched 2026-09-07
- What's New | Cumulus Linux 5.14 · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA/doroce-linux - doRoCE.sh host-side RoCE configuration script · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.