The switch half: Cumulus NVUE and Dell Enterprise SONiC
S4·E4Does this switch do RoCE, or only PFC? · Design review, NVIDIA briefing room, nine days before acceptance
Builds on: PFC, lossless and the case for lossy, Triage in order: symptom to counter to cause
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Configure lossless and lossy RoCE on Cumulus Linux with NVUE and verify it with the documented show commands.
- Configure RoCE on Dell Enterprise SONiC in the correct order and express the same policy as an Ansible task.
- Map the NVUE default priority and DSCP values onto the host-side marking so both halves agree.
- Check a Dell PowerSwitch model against the compatibility matrix before promising RoCEv2 support.
Episode 4 — Does this switch do RoCE, or only PFC?
The design review stalls on one line of the customer’s own summary: “switch supports PFC and ECN, therefore RoCE”. The leaf bill of materials already names a Dell PowerSwitch model. Procurement asks only whether changing it costs lead time; the NVIDIA PM offers a roadmap slide and the words “not announced”. Nobody has opened the compatibility matrix.
You open it in front of them: the two questions have different answers on the same page. The Dell Enterprise SONiC matrix lists RoCEv2 on four of the five platform families, and in the per-model feature table RoCE V2 is a yes on four of ten models while WRED, ECN and priority flow control are a yes on all ten.[4] Confirming PFC has confirmed nothing about RoCE.[4]
The switch side is one command precisely because the policy is many pieces. On Cumulus, nv set qos roce turns on port trust, maps switch priority 3 to traffic class 3 for RoCE data and priority 6 to TC6 for CNP, sets an ETS scheme with TC6 strict, sets an ECN threshold on TC3, and advertises an LLDP application TLV for UDP 4791 — with PFC and the lossless buffer pool added only in lossless mode.[2][1] On Dell Enterprise SONiC the same intent is roce enable pfc-priority 3,4, and the enable has to come first because the PFC priority field is “only configurable when RoCE is enabled”.[3]
Two vocabularies, one policy — and support for it is a row you look up, never infer. So you take the model number and the release, and check first.
1Cumulus: one command, a whole policy
On NVIDIA Spectrum switches running Cumulus Linux the RoCE configuration is a single NVUE command followed by a commit: nv set qos roce then nv config apply.[1] That plain form is equivalent to nv set qos roce mode lossless, which is the default; nv unset qos roce removes the configuration.[1]
The current page documents five modes:[1]
| Mode | What it is for |
|---|---|
lossless |
The default: classification plus ECN plus PFC on the RoCE priority |
lossy |
ECN-only signalling with no PFC |
lossless-single-ipool |
Lossy and lossless share one ingress buffer pool |
lossy-multi-tc |
Packet trimming across multiple traffic classes |
dci-1 |
Beta on Spectrum-4 and later: TC3 PFC-lossless plus TC4 ECN-lossy |
What the command actually programs is worth memorising, because it is the answer to “what did you change on my switch”.[2] In both lossless and lossy modes it turns port trust mode on, maps switch priority 3 to traffic class 3 for RoCE data and switch priority 6 to traffic class 6 for CNP with everything else to TC0, sets an ETS scheme with TC6 strict and TC3 and TC0 at 50 percent weighted round robin, sets a port ECN absolute threshold on TC3, and advertises an LLDP application TLV for UDP 4791 at priority 3.[2][1] Only lossless adds the last two pieces: PFC on switch priority 3, and switch priority 3 mapped to the lossless buffer pool.[2]
The default markings are the numbers every host-side lesson in this course has been pointing at: DSCP 26 into switch priority 3 for RoCE, DSCP 48 into switch priority 6 for CNP, and PCP 0 to 7 mapping one to one onto switch priorities.[1][2] The ECN threshold documented for traffic class 3 is 1501500 bytes; the page also mentions a smaller interface-level value arising from Spectrum-4 ASIC programming, so confirm which surface you are reading before quoting a number to a customer.[1]
Diagram source (Mermaid)
flowchart LR; A["App marks DSCP 26"] --> B["NIC: trust dscp - priority 3"]; B --> C["Switch ingress: DSCP 26 to switch priority 3"]; C --> D["TC3 - ETS 50 percent WRR"]; D --> E["ECN absolute threshold on TC3 marks CE"]; D -. "lossless mode only" .-> P["PFC on switch priority 3 + lossless buffer pool"]; E --> F["Receiver: np_ecn_marked_roce_packets"]; F --> G["CNP: DSCP 48 to switch priority 6 - TC6 strict"]; G --> H["Sender: rp_cnp_handled"]
2Verifying the Cumulus side, and the mode-switch trap
Three show commands and one clear command carry the verification.[1]
nv show qos roce # buffers, DSCP and ETS configuration
nv show interface swp1 qos roce status # per-interface state
nv show interface swp1 qos roce counters # per-interface RoCE counters
nv action clear interface swp1 qos roce counters # reset before a test runThe clear action is the one that turns a switch into a measuring instrument: clear, run a bounded ib_write_bw, then read.[1] Without it you are looking at counters that have been accumulating since the last reload and cannot attribute anything to the test you just ran.
One documented behaviour catches people during change windows. If you have enabled roce mode lossy, then “configuring nv set qos roce without a mode does not change the RoCE mode”.[1] The plain command is only equivalent to lossless on a switch that has not already been put into another mode, so a rollback plan that says “just run nv set qos roce again” silently does nothing.[1] Name the mode explicitly in both directions.
3Dell Enterprise SONiC: enable first, then priorities
On Dell Enterprise SONiC the starting point is one command: roce enable pfc-priority 3,4.[3] The Dell-authored Ansible collection models the same thing as two fields and supplies the semantics: roce_enable is “Enable or disable RoCEv2 default buffer configuration”, and the PFC priority field is documented as “Only configurable when RoCE is enabled”.[3] That ordering rule is the whole operational content of the command: the buffer profile arrives with the enable, and PFC priorities configured before it do not have the buffers they need.[3]
- name: Enable RoCE for PFC priorities
dellemc.enterprise_sonic.sonic_roce:
config:
roce_enable: true
pfc_priority: '3,4'
state: mergedThe module’s own EXAMPLES block — and every rendering of it, NVIDIA-hosted and Dell-hosted — prints pfc_priorities; the argument spec accepts only pfc_priority, so the published example does not run. Check with ansible-doc dellemc.enterprise_sonic.sonic_roce before a CI pipeline depends on it.[3]
merged is the only supported state for this module, so there is no declarative delete and no override; removing the configuration is a separate operation on the switch.[3] That is worth knowing before an automation-minded customer designs a pipeline around it, and it is the concrete example behind the NCP-AIN automation objective about writing Ansible playbooks for RoCE configuration.[9]
Two vocabularies for one policy, side by side:[1][3]
| Intent | Cumulus NVUE | Dell Enterprise SONiC |
|---|---|---|
| Turn RoCE on | nv set qos roce then nv config apply |
roce enable pfc-priority 3,4 |
| Choose lossless or lossy | nv set qos roce mode lossless or mode lossy |
PFC priorities are named in the enable command; the rest of QoS is configured separately |
| Priorities used by default | RoCE priority 3 and CNP priority 6 | the priorities you list in pfc-priority |
| Verify | nv show qos roce and per-interface qos roce status and counters |
switch show output plus the running configuration |
| Automate | NVUE templates | dellemc.enterprise_sonic.sonic_roce with state: merged |
4The matrix question, and proving both halves agree
The question a Dell FAE actually gets is not how to type the command. It is “does this switch do RoCE”. The Dell Enterprise SONiC compatibility matrix answers it per model, and the answer is uncomfortable: RoCEv2 is supported on four of the five platform families listed, and in the per-model feature table RoCE V2 is a yes on four of ten models while WRED, ECN and priority flow control are a yes on all ten.[4] So a customer who has confirmed that their switch does PFC and ECN has confirmed nothing about RoCE.[4] Check the matrix for the exact model, in the exact release, before it becomes a commitment.[4]
Proving the two halves agree is a two-command exercise once both sides are configured. On the host, mark deliberately: perftest’s --tclass sets the traffic class in the GRH, and because the ToS byte is the DSCP value multiplied by four, DSCP 26 is --tclass=104.[8][10] The host must also be trusting DSCP, because the NIC’s default trust state is PCP and a DSCP marking is otherwise ignored on ingress.[6] Then read both ends of the same event: nv show interface swp1 qos roce counters on the switch, and the per-priority counters on the host with ethtool -S <netdev> | grep -E "prio3_(bytes|packets|pause)".[1][7] Bytes on priority 3 on both sides means the classification chain is intact end to end; bytes on priority 0 on the host means the trust state or the marking is wrong before the packet ever left.[7][6]
| PowerEdge | BF-3 offering | DPU mode | NIC mode | Aux power | vSphere DSE | Notable KB |
|---|---|---|---|---|---|---|
R660 16G | ||||||
R760 16G | ||||||
R760XA 16G (GPU) | ||||||
XE9680 16G HGX H100/H200 | ||||||
XE9680L 16G liquid | ||||||
R7725 17G (AMD) | ||||||
R770 17G (Intel) | ||||||
XE9780 / XE9785 17G HGX B300 — Dell AI Factory |
✓ yes · ✗ no · ◐ conditional · ? unknown · — n/a. Click a cell for the evidence.
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
Request. “Two hosts, one leaf. Put RoCE data on priority 3 with PFC, keep CNP on priority 6, and prove it. We have a Cumulus lab switch and a Dell Enterprise SONiC lab switch.”
Cumulus side.
nv set qos roce mode lossless
nv config apply
nv show qos roce
nv show interface swp1 qos roce status
nv action clear interface swp1 qos roce countersReasoning: lossless is the default but naming it is what makes the change reversible in both directions, given that a plain nv set qos roce will not move a switch already in lossy mode.[1] The defaults do the rest: DSCP 26 to switch priority 3 to TC3 for data, DSCP 48 to switch priority 6 to TC6 for CNP, ETS with TC6 strict, an ECN threshold on TC3, PFC on switch priority 3 and the lossless pool.[1][2] Rollback: nv set qos roce mode lossy plus nv config apply, or nv unset qos roce to remove it.[1]
Dell Enterprise SONiC side.
roce enable pfc-priority 3,4Reasoning: the enable installs the RoCEv2 default buffer settings and has to precede the PFC priority configuration.[3] Capture show output and the running configuration before and after; there is no state: deleted in the Ansible module, so the reverse operation is a switch-side one you must write down explicitly.[3]
Host side, both cases. Trust DSCP on the NIC so DSCP 26 lands on priority 3, then generate marked traffic with ib_write_bw --tclass=104 using identical flags on both hosts.[6][8][10]
Proof. nv show interface swp1 qos roce counters on Cumulus and ethtool -S <netdev> | grep prio3 on both hosts. Priority-3 bytes on the host and RoCE counters moving on the switch means the chain is intact.[1][7]
Fill the blanks and name the rollback for each command you write.
- Cumulus:
nv set qos roce mode ________then________. - Verify with
nv show ________andnv show interface swp1 ________. - Reset the measurement window with
nv action ________ interface swp1 qos roce counters. - Dell Enterprise SONiC:
roce enable ________ 3,4, which must run ________ the PFC priority configuration because it installs ________. - Host: trust ________ so DSCP 26 lands on priority ________, and mark the benchmark with
--tclass=________. - Proof: priority-3 bytes appear in
ethtool -Son ________ and RoCE counters move on ________.
New request. A Dell customer has a PowerSwitch model you have not checked, running Enterprise SONiC, and a mixed workload: RoCE storage traffic plus an AI training job. They ask you to put both on priority 3 “because that is the RoCE priority”, and they want an Ansible task they can run in CI.
Write the response. Acceptance criteria:
- The first action is a matrix check on the exact model and release, and you say what you will do if RoCEv2 is not listed for it.[4]
- You give the
sonic_rocetask, state thatmergedis the only supported state, and say what that implies for the CI pipeline’s rollback story.[3] - You argue for or against putting both workloads on one priority using PFC behaviour rather than preference, and name the counter that would show the coupling.[7]
- You state the host-side change without which the switch configuration will not see the traffic it expects.[6]
- You name one thing about this platform you cannot state from a citable page and how you would resolve it.[5]
The line item, not the quarter
The chosen model is not in the RoCEv2 column, so the review changes a line item instead of a quarter, and the supported part lands inside the window — the only thing procurement asked about, and this time the right question. On the lab pair you configure both vocabularies to the same intent, clear the interface counters, run a bounded ib_write_bw with --tclass=104 for DSCP 26, and read priority 3 bytes on the switch and on the host.[1][8][7] Both sides agree.
What you say to the customer: “Give me the model and the release and I will look it up with you now, because ‘it does PFC’ and ‘it does RoCE’ are different rows on the same page.”[4]
Overnight, the platform team builds Kubernetes on the fabric you just proved.
Lab
Dell-lab hosts through the lab PowerSwitch. Switch changes are mutating and happen in a window with a written rollback.
- Pre-flight. Confirm the switch model and release against the compatibility matrix and record the result.[4] On both hosts:
ibdev2netdev,mlnx_qos -i <netdev> | tee ~/qos-before.txt,ethtool -S <netdev> > ~/eth-before.txt. On the switch: capture the full running configuration.[6] - Baseline traffic, unmarked.
ib_write_bw -d mlx5_0 -x <gid> -s 1048576 -n 20000 --report_gbitswith identical flags both sides and no--tclass. Readethtool -S <netdev> | grep -E "prio[0-9]_bytes"on both hosts. Expected: the bytes are on the default priority, not priority 3.[7] - Host half (mutating). Set the NIC trust state to DSCP and confirm the DSCP-to-priority mapping puts 26 on priority 3.[6][10] Rollback: re-apply everything captured in
~/qos-before.txt. - Switch half (mutating). Apply
roce enable pfc-priority 3,4on the Enterprise SONiC switch.[3] Rollback: restore the running configuration captured in step 1. - Marked traffic. Repeat step 2 with
--tclass=104on both sides, then read per-priority bytes and pause counters on both hosts and the switch’s own PFC and RoCE counters.[8][7] Expected: priority-3 bytes on the hosts and switch counters that move. If the bytes are still on priority 0, the host trust state is the fault, not the switch. - Optional, customer lab only. Repeat against a Spectrum switch running Cumulus:
nv set qos roce mode lossless,nv config apply,nv action clear interface swp1 qos roce counters, run the same benchmark, thennv show interface swp1 qos roce countersand compare it with the hosts’ethtool -Sview.[1][7] Rollback: the named reverse mode plusnv config apply, never a barenv set qos roce.[1] - Restore and verify. Re-apply the host QoS from
~/qos-before.txtand the switch running configuration, then diff both against the captures. Deliverable: one page showing the same policy in both vocabularies with the counter evidence from each side.
Two switches, both free: a Dell Enterprise SONiC 4.5.1 pair in ~/containerlab, and a Cumulus topology in NVIDIA Air.
- Bring up two Dell Enterprise SONiC 4.5.1 nodes back to back in containerlab with a Linux host container on each side. Expected: both nodes reach a CLI prompt and the link is up. If not: check the image tag and the containerlab kind before continuing.
- Capture the running configuration and the QoS-related
showoutput before any change; this is your rollback reference. - Apply
roce enable pfc-priority 3,4and capture the same outputs again.[3] Expected: the running configuration gains the RoCE stanza. Rollback: restore the captured configuration. - Record whether the virtual image enforces anything. It is unconfirmed whether the containerlab image accepts this command, what it prints, and whether any of it is programmed without a real ASIC.[3] If the command is accepted but no buffers or PFC state change, write that down explicitly: a config-only lab is still a useful lab, but only if the note says so.
- Write the Ansible task from Segment 3 and run it with
--checkagainst the node. Expected: a changed or ok result with no error;mergedis the only state available, so there is no delete task to write.[3] - In NVIDIA Air, build the same policy on Cumulus:
nv set qos roce mode lossless,nv config apply, thennv show qos roceandnv show interface swp1 qos roce status.[1] Rollback:nv unset qos roceplusnv config apply. - Deliverable: a two-column diff of the vocabularies, one row per intent, with the exact command from each platform and the exact verification command from each platform. Add a third column for anything the virtual images accepted but did not enforce.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE in five sentences why 'we turned on RoCE on the switch' is not the same statement on Cumulus and on Enterprise SONiC, and what you would ask for to check either claim.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 RoCE page re-fetched 2026-09-07 (five modes plus defaults plus verification commands confirmed), dellemc.enterprise_sonic.sonic_roce module, Dell Enterprise SONiC Compatibility Matrix 4.4.0, DOCA Ethernet QoS 3.5.0. Dates are when each page was fetched.
- RDMA over Converged Ethernet (RoCE): Cumulus Linux 5.18 · fetched 2026-09-07
- RDMA over Converged Ethernet (RoCE): Cumulus Linux 5.9 · fetched 2026-09-07
- dellemc.enterprise_sonic.sonic_roce: manage RoCE QoS configuration on SONiC · fetched 2026-09-07
- Dell Enterprise SONiC Distribution Compatibility Matrix Release 4.4.0 · fetched 2026-09-07
- Configure RoCE on the switch: Dell SmartFabric OS10 User Guide 10.5.4 (body not machine-fetchable) · fetched 2026-09-07
- Ethernet QoS (DOCA SDK 3.5.0) · fetched 2026-09-07 · DOCA 3.5.0
- Ethtool counters (mlx5): the Linux kernel documentation · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking Exam Study Guide (doc 4417000 MAY26) · fetched 2026-09-07
- Ethernet QoS (DOCA-Host): generated PDF · fetched 2026-09-07 · DOCA 3.5.0
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.