MTU, RoCE LAG, and what has to be installed
S4·E3The bond that took the RDMA device with it · Dell staging lab, Round Rock, the afternoon before the other 48 nodes ship
Builds on: RoCEv1 vs RoCEv2: the encapsulation, What ib_write_bw actually measures
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Read max_mtu and active_mtu and choose a perftest -m value that tests what you intend.
- State the Ethernet MTU you would request for a 4096-byte RoCE MTU and label it as arithmetic rather than vendor specification.
- Configure a bond that keeps RoCE LAG engaged and recognise the change that silently drops out of it.
- Select the DOCA-Host profile that matches a stated customer requirement and run the mandatory post-install steps.
Episode 3 — The bond that took the RDMA device with it
Node sixteen’s symptom is now on the bench in front of you, in the lab, on the image that ships tomorrow. A lab engineer spent the week comparing bonding modes on the two ConnectX ports, and on this node every pinned application fails to find its device. Nothing logged an error. The bond is up, both slaves are up, link state is perfect. The SE has already opened the promise spreadsheet to the row that says the rack ships Friday.
You run ibv_devices and the answer is on the screen: mlx5_bond_0 is gone and mlx5_0 and mlx5_1 are back. RoCE LAG exists to mimic Ethernet bonding for RDMA devices on dual-port cards, and when it engages the two IB devices collapse into one named mlx5_bond_0.[1][10] It engages only under active-backup, balance-xor or 802.3ad, and “any change of bonding configuration that negates one of the above rules … will result in exiting RoCE LAG mode”.[1] An afternoon of balance-rr silently renamed the device that every config file, every -d flag and every counter path names.[1]
Then the benchmark comes back and does not match last night’s, because perftest defaults -m to the port’s active_mtu and nobody forced it on either side.[3] Two invisible variables, forty-eight clones, one night.
Link state proves the cable, never the device your applications name. Both are packaging questions before they are fabric questions, so you start there.
1MTU: two numbers, one of which is not on the wire
ibv_devinfo prints two MTUs per port and they mean different things: max_mtu: 4096 (5) is the ceiling the hardware supports and active_mtu: 1024 (3) is what the port is actually using.[1] The RDMA path MTU is a small enumeration — 256, 512, 1024, 2048, 4096 — and the man page marks it “valid for RC/UC QPs only”, so a UD QP has no path MTU at all and behaves differently when the two ends disagree.[2]
perftest defaults -m to the port’s active_mtu, with a documented range of 256 to 4096 for non-raw-Ethernet tests.[3] That default is where a lot of confusion starts: a benchmark that quietly ran at 1024 while the customer believes the fabric is jumbo produces a number nobody can reproduce.[3] Since perftest also requires the same options on both sides, forcing -m identically on server and client is the cheapest way to remove MTU from a discussion.[3]
RoCEv1 is simpler and is a useful contrast: it is a Layer-2 protocol with its own ethertype and “the regular Ethernet MTU applies on the RoCE frame”.[1] RoCEv2 adds IP and UDP, so the frame that carries a 4096-byte RDMA MTU also carries Ethernet, IP, UDP, BTH, ICRC and FCS.[1]
Now the honest part. No NVIDIA page states the Ethernet MTU required to carry a 4096-byte RoCE MTU. The DOCA RoCE material gives the RoCEv1 statement and the header structure but no number, and the Cumulus Linux RoCE page — the switch-side reference for lossless defaults — contains no MTU specification at all.[1][4] So 4200, 9000 and 9216 are operator practice and arithmetic, and any lesson, runbook or customer email that attributes them to NVIDIA is inventing a citation.[1][4]
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2RoCE LAG: one device, three legal bond modes, silent exit
RoCE LAG “is a feature meant for mimicking Ethernet bonding for IB devices and is available for dual port cards only”, supported on kernel 4.9 and above.[1] It engages when both Ethernet interfaces of the card are bonded in active-backup (mode 1), balance-xor (mode 2) or 802.3ad (LACP) (mode 4).[1][10]
The visible effect is the device name: “Once RoCE LAG is enabled, instead of having two IB devices; mlx5_0 and mlx5_1, there will be one device named mlx5_bond_0”.[1][10] Everything that names a device changes with it — ibv_devinfo -d mlx5_bond_0, -d mlx5_bond_0 in perftest, NCCL_IB_HCA, and every counter path under /sys/class/infiniband/mlx5_bond_0/.[1]
The operational trap is the exit condition: “Any change of bonding configuration that negates one of the above rules … will result in exiting RoCE LAG mode and the return to normal IB device per port configuration.”[1] There is no error and no log line that says “you have left LAG”. An engineer testing balance-rr for an afternoon, or removing one slave, hands back a machine where mlx5_bond_0 no longer exists and every pinned application fails to find its device.[1]
Two more facts belong to the same paragraph of the manual. The mlx5_ib driver holds a reference to the netdev for link-state notifications and for resolving IP addresses to MACs when building address vectors, but “RoCE traffic does not go through the mlx5_core driver; it is completely offloaded by the hardware” — which is why bonding changes the RDMA device identity without the RDMA data path ever traversing the bond driver.[1] And the modules loaded at boot are listed in /etc/infiniband/openib.conf, which is where a bond that must survive reboot gets its ordering problem.[1]
3Packaging: five profiles, two mandatory restarts, one timeline
DOCA-Host replaces MLNX_OFED, and the profile you install decides what a host can do.[5] The contents are published verbatim:[5]
| Profile | Contents | Use it for |
|---|---|---|
doca-all |
All DOCA libraries and drivers, MLNX_OFED, DOCA Core, MLNX-DPDK, OVS-DOCA, DOCA Flow | Complete BlueField implementations needing the whole SDK |
doca-networking |
MLNX_OFED, DOCA Core, DOCA-PCC, OVS-DOCA, DOCA Flow, DOCA Perftest | Accelerated networking without the full library overhead |
doca-ofed |
MLNX_OFED drivers and tools | “Driver-only installation equivalent to MLNX_OFED” |
doca-roce |
rdma-core, ofed-scripts, mlnx-tools, mlnx-ofa_kernel, perftest | “Lightweight RDMA-over-Ethernet (RoCE) functionality” |
doca-host-basic |
rdma-core, mlnx-tools, mlnx-ofa_kernel, perftest, mlnx-ethtool, mlnx-iproute2 | “Lightweight Ethernet only functionality” |
For a pure RoCE compute node, doca-roce is the whole answer: it carries rdma-core for the verbs, mlnx-tools for show_gids and the IRQ-affinity scripts, and perftest for the proof.[5] NVIDIA’s own guidance for a like-for-like replacement of an MLNX_OFED fleet is doca-ofed, described as adding no DOCA functionality.[5]
Installation is a repository package then a profile package: sudo apt install -y doca-all on Debian family, yum or zypper on RPM family, after installing the repo file from the DOCA Downloads page with dpkg -i or rpm -Uvh.[6] Then two steps that are not optional: sudo /etc/init.d/openibd restart followed by sudo mst restart.[6] Skipping them leaves the previous kernel modules loaded and produces a host whose tools and drivers disagree. Verify with ofed_info -s.[1]
The timeline matters for planning conversations rather than for commands. October 2024 was the last standalone MLNX_OFED release; the period from then to October 2027 is an LTS window with “critical bug fixes and security updates”; October 2027 is end of life.[7] That is a migration plan, not a fire. The transition guide does not publish a package-name mapping table from the old mlnx-ofed-all names, and it does not say what replaced mlnxofedinstall, so those two questions have to be answered on the host rather than quoted.[7]
Host does not see the BlueField
4mlnx_tune is an auditor, not a fixer
Customers ask for a tuning profile. The source of mlnx_tune says there is not one for RoCE. Its allowed profiles are exactly HIGH_THROUGHPUT, IP_FORWARDING_MULTI_STREAM_THROUGHPUT, IP_FORWARDING_MULTI_STREAM_PACKET_RATE, IP_FORWARDING_MULTI_STREAM_0_LOSS, IP_FORWARDING_SINGLE_STREAM, IP_FORWARDING_SINGLE_STREAM_0_LOSS, IP_FORWARDING_SINGLE_STREAM_SINGLE_PORT, LOW_LATENCY_VMA and MULTICAST.[8] None of them is about RDMA.
What the tool is good for is the audit. -r, --report reports hardware and software status and issues without setting a profile, and -d, --debug_info dumps system debug information; the report covers OS, CPU, memory, hyper-threading, IRQ balancer, firewall and per-device status, and writes to /tmp/mlnx_tune_<YYMMDD_HHMMSS>.log by default.[8] Applying a profile requires root; reporting does not have to change anything.[8]
This is worth saying out loud to a customer because the vendor documentation is silent in an unhelpful way: MLNX_OFED’s own “Performance Tuning” page contains no commands and no settings, only the statement that it may be necessary to modify the default configuration, with a pointer to a community post.[9] So an FAE who arrives with a list of settings is quoting folklore. The defensible move is to hand over the audit output, the NUMA fact (/sys/class/net/<if>/device/numa_node), and an A/B measurement, and to change only what the A/B moves.[8]
Request. “Two Dell R760 nodes, dual-port ConnectX-7, no OVS and no DOCA applications. They will run NCCL over RoCE and we want an LACP bond across both ports. What do we install and what MTU do we ask the network team for?”
1. Profile. No OVS-DOCA, no DOCA Flow, no DPDK. doca-roce covers it: rdma-core, ofed-scripts, mlnx-tools, mlnx-ofa_kernel, perftest.[5] If the customer’s change-control prefers a like-for-like MLNX_OFED replacement, doca-ofed is the alternative and NVIDIA names it as such.[5]
sudo dpkg -i doca-host_<ver>_all.deb # repo package from the DOCA Downloads page
sudo apt update && sudo apt install -y doca-roce
sudo /etc/init.d/openibd restart
sudo mst restart
ofed_info -s2. Bond, then check the device name. Bond both ports in mode 4, then verify LAG actually engaged rather than assuming it:
ibv_devices # expect mlx5_bond_0, NOT mlx5_0 and mlx5_1
ibv_devinfo -d mlx5_bond_0 | grep -E "state|active_mtu|link_layer"3. MTU, stated honestly. Ask ibv_devinfo for max_mtu and active_mtu.[1] Then the email sentence: “For a 4096-byte RDMA path MTU the interface must carry 4096 bytes of RDMA payload plus Ethernet, IP, UDP, BTH, ICRC and FCS. Jumbo frames (9000) are the common operator choice and give ample headroom. NVIDIA does not publish a required interface MTU for this, so treat 9000 as our engineering decision, not a vendor specification.”[1][4]
4. Prove it. Same command both sides, MTU forced, and compare against the default:
ib_write_bw -d mlx5_bond_0 -x <gid> -m 4096 -s 1048576 -n 20000 --report_gbits [server-ip]
ib_write_bw -d mlx5_bond_0 -x <gid> -m 1024 -s 1048576 -n 20000 --report_gbits [server-ip]5. Audit rather than tune. sudo mlnx_tune -r | tee ~/mlnx_tune-report.txt and cat /sys/class/net/<if>/device/numa_node, and change nothing that an A/B does not justify.[8][9]
Same request. Fill the blanks.
- Profile:
________because the node needs rdma-core plus perftest and no OVS-DOCA. The MLNX_OFED-equivalent alternative is________. - Post-install, in order:
________then________, verified with________. - Bond mode
________keeps LAG; confirm withibv_devicesshowing________. - The MTU sentence must label the number as ________ because ________ publishes no required interface MTU.
- The proof is the same perftest command on both sides with
-mforced, because perftest otherwise defaults to ________.
New request. “One Dell XE9680 with eight ConnectX-7 cards. It runs OVS-DOCA for the north-south tenant network and NCCL over RoCE east-west. The team wants active-backup bonding on the north-south pair only, and they have an existing MLNX_OFED 24.10 fleet they cannot upgrade this quarter.”
Write the recommendation. Acceptance criteria:
- One profile is chosen and justified against the published contents, and you state what would be missing if you had chosen
doca-roceinstead.[5] - You state whether the east-west cards should be bonded and why the answer differs from the north-south pair.[1]
- You state which RDMA device names the NCCL configuration must use after bonding, and how you would verify them.[1]
- You place the MLNX_OFED fleet on the published timeline and give the date that actually forces action.[7]
- You give one sentence about MTU that a customer could paste into a change request without misattributing it.[1][4]
Three lines in the build sheet
The image goes back to 802.3ad, mlx5_bond_0 returns, and the build sheet gains three lines: the profile is doca-roce because these hosts run no OVS-DOCA and no DOCA application, the install ends with openibd restart and mst restart and is verified with ofed_info -s, and every benchmark carries the same -m on both sides.[5][6][3] The mlnx_tune report goes into the handover as evidence, not as a set of settings to copy.[8]
What you say to the customer: “Check the RDMA device name after every bonding change. Link state will look perfect while the device your applications name no longer exists.”[1]
Forty-eight nodes ship. Then the network lead turns his notebook to the leaf bill of materials and asks whether that switch model does RoCE at all.
Lab
Dell-lab ConnectX host with two ports. Steps 3 to 5 are mutating; capture first.
- Pre-flight, read-only.
ofed_info -s;ibv_devices;ibv_devinfo | grep -E "hca_id|state|max_mtu|active_mtu|link_layer";ip -d link show <if0>and<if1>;cat /proc/net/bonding/*if any bond exists. Save all of it to~/preflight.txt. This file is the rollback reference.[1] - Audit, read-only.
sudo mlnx_tune -r | tee ~/mlnx_tune-report.txtandcat /sys/class/net/<if0>/device/numa_node.[8] Note the NIC’s NUMA node; you will want it in the next module. - MTU pair (mutating). Set the netdev MTU to 1500, run
ib_write_bw -d mlx5_0 -x <gid> -m 4096 -s 1048576 -n 20000on both sides, and recordlocal_ack_timeout_errandreq_transport_retries_exceededbefore and after.[3] Then set the MTU to 9000 and repeat. Rollback: restore the original MTU recorded in step 1. - Bond into LAG (mutating). Create a mode-4 bond over both ConnectX ports, then verify with
ibv_devices. Expected:mlx5_bond_0replacesmlx5_0andmlx5_1.[1][10] If it does not appear, check that both slaves are the two ports of the same card and that the mode is 1, 2 or 4. Rollback: delete the bond and restore the interface configuration from~/preflight.txt. - Break the rules on purpose (mutating). Change the bond to
balance-rr, or remove one slave, and runibv_devicesagain. Expected: the card splits back tomlx5_0andmlx5_1with no error anywhere in the logs.[1] Write down every place in your environment that would have to change if this happened silently in production: perftest device flags,NCCL_IB_HCA, counter paths, monitoring. Rollback: restore mode 4 and confirmmlx5_bond_0returns. - Close. Restore everything from
~/preflight.txt, re-runofed_info -sandibv_devices, and diff against the pre-flight capture. Deliverable: a one-page note giving the profile you would install on this host, the bond mode, the RDMA device name that results, and one MTU sentence written so a customer could paste it into a change request without misattributing it to NVIDIA.[5][4]
Containerlab plus Soft-RoCE. Nothing here needs an NVIDIA card, and the failures are the point.
- Two nodes, different MTUs:
sudo ip link set <ifA> mtu 1500andsudo ip link set <ifB> mtu 9000. Record both. Rollback: restore the original values noted before you start. - Run
ib_write_bw -d rxe0 -m 2048 -s 1048576 -n 2000on both sides, then repeat with-m 4096.[3] Expected: one of them degrades or fails. Record which, and the first counter or completion status that moved. If neither fails, your veths are both large enough — lower one further and repeat. - Do the arithmetic yourself and write it down as arithmetic: 4096 bytes of RDMA payload plus Ethernet header and FCS, IPv4 header, UDP header, BTH and ICRC. State the interface MTU you would request and label the number “my arithmetic, not a vendor figure”.[1][4]
- Build a bond over two veth pairs in mode 4 (
sudo ip link add bond0 type bond mode 802.3ad, enslave both, bring up) and observe what therxedevice does: does a*_bond_0RDMA device appear? Expected: it does not, because RoCE LAG is a hardware feature of dual-port NVIDIA cards.[1] Record that as a fidelity limit of the software device. Rollback:sudo ip link delete bond0. - Profile mapping exercise, no installation: for each of five one-line customer requirements (RoCE-only compute node; OVS-DOCA on ConnectX; MLNX_OFED replacement with no DOCA; BlueField host running DOCA services; plain Ethernet host), name the profile and quote its published contents.[5] Check yourself against the Profiles tab of the CompatMatrix above.
- Read
mlnx_tune’s allowed profile list from the source and write the sentence you would say to a customer who asks for the RoCE profile.[8]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
In four sentences, explain to a Dell platform engineer why you will not tell them 'NVIDIA requires MTU 9000 for RoCE', and what you will tell them instead.
Sources
Facts in this lesson were checked against DOCA-Host Profiles page re-fetched 2026-09-07 (doca-all / doca-networking / doca-ofed / doca-roce / doca-host-basic contents confirmed verbatim), DOCA-Host RoCE PDF, ibv_modify_qp(3), perftest master man page, mlnx-tools master, MLNX_OFED 24.10 LTS. Dates are when each page was fetched.
- RDMA over Converged Ethernet (DOCA-Host): generated PDF · fetched 2026-09-07 · DOCA 3.5.0
- ibv_modify_qp(3): rdma-core man page · fetched 2026-09-07
- perftest(1) man page (linux-rdma/perftest master) · fetched 2026-09-07
- RDMA over Converged Ethernet (RoCE): Cumulus Linux 5.18 · fetched 2026-09-07
- DOCA Profiles: DOCA-Host · fetched 2026-09-07 · DOCA 3.5.0
- DOCA-Host Installation and Upgrade · fetched 2026-09-07 · DOCA 3.5.0
- MLNX_OFED to DOCA-OFED Transition Guide (DOCA 3.4.0) · fetched 2026-09-07
- mlnx_tune source (Mellanox/mlnx-tools master) · fetched 2026-09-07
- Performance Tuning: MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
- RDMA over Converged Ethernet (RoCE): MLNX_OFED 24.10-5.1.6.1 LTS · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- ibdiagnet: the command you run before escalatingInfiniBand course · Same ground: tuning, packaging and select
- Source routing, segments, and the two data planesSRv6 course · Same ground: timeline, mtu and active
- DOCA-Host install and profilesDOCA course · Same ground: DOCA-Host profile, recognise and matches