Layer 2 that survives a reload: bridges, bonds, MLAG and VRR
S2·E3Both bonds down, one host, 02:40 · Bridge call with the customer's NOC, you at the kitchen table, six days from acceptance
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Build a VLAN-aware bridge with access ports and SVIs, respecting the reserved VLAN ranges.
- Configure an 802.3ad bond and an MLAG pair with a legal system MAC, peer link and backup IP.
- Diagnose a protodown bond from the LACP partner MAC mismatch symptom and clear it.
- Explain why VRR is the anycast gateway for an MLAG pair and what it cannot coexist with.
Episode 3 — Both bonds down, one host, 02:40
One GPU host is isolated: both members of its bond are protodown on an MLAG pair, every other host on that pair is fine, and a server engineer is already on the bridge running tcpdump on the switch and getting nothing for it. Nobody changed anything, they say, except that the second MLAG pair went in on Saturday — cabled beautifully, every port tagged, because the night-shift operator had the label maker out again. The labels are perfect. What is underneath one of them is a copy.
Two facts end the call quickly. The switch log carries “lacp partner mac mismatch”, and the recovery is nv action clear mlag lacp-conflict — but clearing before you fix the duplicate MLAG system MAC just protodowns the bonds again seconds later.[4] And the capture is not evidence: on a Cumulus switch tcpdump sees only control-plane traffic to and from the CPU, never the data plane.[9]
MLAG exists because a server team wants two switches for redundancy and one bond for simplicity, and those wishes contradict each other. The pair answers a host as a single LACP identity, using a system MAC from a reserved range, and it synchronizes dynamic state while never synchronizing configuration.[4] Everything protective in that design, protodown included, is there to stop two switches answering as two different partners on one host bond — which is what a duplicated system MAC quietly creates.
One pair, one identity: the reserved-range MAC is per pair, never per template.
Layer 2 that survives a reload is what the rest of this lesson builds.
1One bridge, many VLANs
NVIDIA is unambiguous: “NVIDIA recommends that you use VLAN-aware mode bridges instead of traditional mode bridges.”[2] The default bridge is br_default and it ships “with no ports assigned”, so nothing is bridged until you say so.[1] The canonical block is four lines:[1]
nv set interface swp1-2 bridge domain br_default
nv set bridge domain br_default vlan 10,20
nv set bridge domain br_default untagged 1
nv config applyAn access port is nv set interface swp1 bridge domain br_default access 10; an SVI on the single default bridge is nv set interface vlan10 ip address 10.1.10.2/24.[1] When more than one VLAN-aware bridge exists you must name the SVI explicitly with type svi, vlan, base-interface and an address - but note the two hard limits that usually end that idea: “You cannot use MLAG with multiple VLAN-aware bridges” and “You cannot use the same port with multiple VLAN-aware bridges.”[1]
Three VLAN ranges get confused constantly, so learn them as a set. User VLANs are 2-4094. The internal reserved range defaults to 3725-3999 and moves with nv set system global reserved vlan internal range 4064-4094. L3 VNI VLANs default to 4000-4064. And VLAN 4094 belongs to the MLAG peer link.[1][4] Scale is per ASIC generation: “Spectrum 1: 10,000 VLAN elements maximum”; “Spectrum 2 & later: 15,996 elements (warm reboot off) or 7,934 (warm reboot on)”.[1]
MTU is the one interface setting the documentation says “you must set manually”, with a range of 552 to 9216 and a default of 9216 on switch ports; eth0 defaults to 1500.[8][7] Two other facts save time later: “A bridge cannot contain multiple subinterfaces of the same port”; and the MAC table view is nv show bridge domain br_default mac-table.[2] Cumulus Linux 5.18 adds nv set bridge default-vlan <vlan-id> and nv set bridge domain <domain-id> arp-inspection.[12]
2Bonds, and the LACP details that decide whether a host boots
nv set interface bond1 bond member swp1-4 builds a bond, where swp1-4 is range notation for swp1 through swp4.[3] The defaults are sensible for an AI fabric and worth quoting from memory: mode is “IEEE 802.3ad link aggregation mode” (bond mode static selects balance-xor instead), LACP rate is fast, MII monitor interval is 100 ms, minimum active links is 1, and transmit hash policy is layer3+4.[3]
LACP bypass is disabled by default and exists for one real case: it “allows a bond configured in 802.3ad mode to become active and forward traffic even when there is no LACP partner” - a server PXE-booting before its OS bonding driver exists.[6] With MLAG and all-active bonds, “all the slaves of the bond are active on both the primary and secondary MLAG nodes”, and NVIDIA recommends pinning PXE to one physical NIC in the boot configuration so simultaneous boot requests do not fail.[6] All-active mode “is only supported on bonds specified as bridge ports on the switch”.[6]
Diagram source (Mermaid)
flowchart TB H["host bond0 - LACP fast - layer3+4 hash"] H --- L1["leaf01 bond1 - mlag id 1"] H --- L2["leaf02 bond1 - mlag id 1"] L1 === PL["peerlink bond carrying peerlink.4094"] PL === L2 L1 -. backup IP over eth0 .- L2 L1 --- S["spine01"] L2 --- S V["vlan10 SVI with VRR virtual MAC live on both leaves"] --- L1 V --- L2
3MLAG: the six settings and the one that is not obvious
MLAG makes two switches present one LACP identity to a dual-attached host. Six settings do the work:[4]
nv set mlag mac-address 44:38:39:FF:00:AA
nv set mlag backup 10.10.10.2
nv set mlag peer-ip linklocal
nv set mlag priority 2084
nv set mlag init-delay 100
nv set interface bond1 bond mlag id 1
nv config applyThe system MAC must come from the reserved range 44:38:39:ff:00:00 - 44:38:39:ff:ff:ff, must differ per MLAG pair, and cannot be multicast.[4] The peer link must be the bond named peerlink carrying the subinterface peerlink.4094, and “Do not add VLAN 4094 to the bridge VLAN list”.[4] The MLAG id (clag-id) runs 1-65535, 0 disables MLAG on that bond, and it must match on both peers for the same dual-connected bond.[4] Priority defaults to 32768 over a 0-65535 range, lower wins primary, and if priorities tie the role is decided by MAC comparison.[4] init-delay defaults to 180 seconds with “Recommendation for scaled environments: 300 seconds”, and it resets to 0 when the peer is unreachable past the reload timeout, when the peer sends a goodbye, or when both MLAG sessions start simultaneously.[4]
The setting that surprises people migrating from Dell OS10 VLT is the backup IP. It “must differ from peer link IP address”, should be a loopback or management address, and takes an optional VRF form nv set mlag backup 10.10.10.2 vrf mgmt.[4] A VLT engineer expects the keepalive to be a routed link near the peer link; in Cumulus the backup path exists precisely so that it fails independently of the peer link.
Verify with nv show mlag, which returns state, peer-ip, local-role, peer-alive, backup-active and mac-address.[4] Two operational rules complete the picture: “MLAG synchronizes dynamic state… but does not synchronize switch configurations”, and “Always enable STP in your layer 2 network and BPDU Guard on host-facing bond interfaces.”[4] For sizing: “Determine how much bandwidth is traveling across single-connected interfaces and set half of that bandwidth to the peer link.”[4]
4The two calls you will actually take
Call one: my bonds are protodown. The log strings are “lacp partner mac mismatch”, duplicate-partner-mac and partner-mac-mismatch, and the recovery action is nv action clear mlag lacp-conflict.[4] The clear is the last step, not the first: the two causes are a host bonded into two different MLAG pairs, and one MLAG system MAC reused across two pairs. Clear before fixing and the bonds protodown again in seconds.
Call two: traffic black-holed for three minutes after a reload. That is init-delay, 180 seconds by default, doing its job.[4] The fix is a scheduling conversation, not a configuration change - and at scale the documented recommendation moves it up to 300 seconds, not down.[4]
VRR is the anycast gateway that makes the pair usable at layer 3. It works by “activating the virtual MAC on both MLAG devices, allowing either device to handle frames it receives”.[5] Configuration is two lines per SVI:[5]
nv set interface vlan10 ipv4 vrr address 10.1.10.1/24
nv set interface vlan10 ipv4 vrr state enabled
nv config applyThe default virtual MAC is 00:00:5E:00:01:01, custom values must sit inside 00:00:5E:00:01:00-00:00:5E:00:01:FF, and there is one absolute constraint: “You cannot configure both VRR and VRRP on the same switch.”[5]
BlueField-3 on a Dell PowerEdge: what is the symptom?
5Building it, and the SONiC contrast a Dell account will ask for
Design. leaf01 and leaf02, host server01 dual-attached with one LACP bond. VLAN 10 with a VRR gateway at 10.1.10.1/24. Peer link swp49-50. Backup over eth0.
Step 1 - both switches, identically except for addresses. MLAG never synchronizes configuration, so every line below runs twice.[4]
# leaf01
nv set interface swp49-50 bond member peerlink # build the peer-link bond
nv set mlag mac-address 44:38:39:FF:00:AA # reserved range, unique per pair
nv set mlag peer-ip linklocal
nv set mlag backup 192.0.2.42 # leaf02 eth0 - NOT the peerlink address
nv set mlag priority 2084 # lower wins primary
nv set mlag init-delay 180Step 2 - the bridge and the host bond, same MLAG id on both peers.[1][3][4]
nv set interface bond1 bond member swp1
nv set interface bond1 bond mlag id 1
nv set interface bond1 bridge domain br_default access 10
nv set bridge domain br_default vlan 10,20Step 3 - the anycast gateway.[5]
nv set interface vlan10 ip address 10.1.10.2/24 # .3/24 on leaf02
nv set interface vlan10 ipv4 vrr address 10.1.10.1/24 # identical on both
nv set interface vlan10 ipv4 vrr state enabled
nv config applyStep 4 - verify in the documented order. nv show mlag for state, local-role, peer-alive and backup-active; then nv show bridge domain br_default mac-table to confirm the host MAC is learned; then a ping from the host to 10.1.10.1 from each member in turn.[4][2]
Step 5 - break it deliberately. Configure the same 44:38:39:FF:00:AA on a second pair the host also touches. Expected: bonds protodown with “lacp partner mac mismatch”. Fix the MAC on the second pair, then nv action clear mlag lacp-conflict.[4]
Rollback for each mutating step. nv config apply --confirm <time> for the whole block, plus the explicit inverses: nv unset mlag mac-address, nv unset interface bond1 bond mlag id, nv unset interface vlan10 ipv4 vrr state.
nv set interface swp49-50 bond member ____
nv set mlag mac-address ____________ # which range, and how does it relate to the other pair?
nv set mlag backup ____________ # which address must it NOT be?
nv set interface bond1 bond mlag id ____ # what must be true on the peer?
nv set bridge domain br_default vlan ____
nv set interface vlan10 ipv4 vrr address ____________ # same or different on the two peers?
nv set interface vlan10 ipv4 ____ state enabled
nv config applyAnswer in one line each: which of the values above must be identical on both peers, which must differ, and which must be absent from the bridge VLAN list.
A Dell customer runs an MLAG pair. After a scheduled reload of leaf02, dual-attached hosts lose traffic for about three minutes, then recover. A week later the same pair loses one host permanently: its bonds are protodown while every other host is fine. The customer wants one root cause for both.
Produce: (1) the two distinct causes and the evidence command for each; (2) why “one root cause” is the wrong frame here; (3) the exact remediation order for the protodown host, naming what must happen before nv action clear mlag lacp-conflict; (4) the configuration change you would and would not make for the three-minute event, with the documented reason.
Acceptance: your answer refuses to change init-delay downward and says why in one sentence a customer will accept.
Franco’s own lab runs Dell Enterprise SONiC 4.5.1, and the mapping question comes up on every Dell account. Three concepts port directly: a peer link, a keepalive, and a LAG id per dual-attached host. One does not: Cumulus’s backup IP is deliberately out of band on the management or loopback path, whereas a VLT engineer expects a routed keepalive adjacent to the peer link.[4] Two commercial facts belong in the same conversation. Dell’s Enterprise SONiC compatibility matrix for release 4.4.0 lists only Broadcom-NPU PowerSwitch platforms - Tomahawk 5, Tomahawk 4, Trident 4 and older - with no NVIDIA Spectrum platform on the list.[11] And Dell Enterprise SONiC is bundle-gated: VXLAN and VXLAN EVPN appear only in the Enterprise Standard, Enterprise Premium and Lite bundles, not the Cloud bundles.[11] So “we will just run SONiC on the Spectrum box” is a question about which box and which bundle, not a preference about CLI.
What the NOC writes in the ticket
The duplicate system MAC on the new pair goes first; only then nv action clear mlag lacp-conflict, and the bonds come back and stay back.[4] You ask for nv show mlag from both peers rather than one, because local-role, peer-alive and backup-active from a single side is half a picture.[4] The three-minute black hole they saw after last week’s reload was init-delay working as designed, and at scale the documented number moves up to 300 seconds, not down.[4]
What you say: “Your gateway was never at risk — VRR keeps it live on both leaves. The bond was the only thing failing.”[5]
Monday the hosts run their first full training job. By the afternoon the network lead has a chart: spine01 near 80 percent, spine02 near 20.
Lab
Goal. There is no MLAG pair in the Dell lab, so verify the host half of the contract: the LACP partner MAC that the switch side depends on.
Pre-flight inventory:
ip -br link ; ip -br addr > before.txt
nmcli connection show 2>/dev/null >> before.txt # or your distro's equivalent
cat /proc/net/bonding/* 2>/dev/null >> before.txt- On a host with two ConnectX ports, create an 802.3ad bond across both, matching the switch defaults you learned: LACP rate fast, transmit hash policy layer3+4.[3] Expected:
ip -br linkshows the bond and both members. Rollback: delete the bond and restore the two interfaces frombefore.txt. - Read the host’s view:
cat /proc/net/bonding/bond0. Expected: an aggregator id and, for each member, a partner MAC. If your switch ports are on the same MLAG pair, the partner MAC is identical on both members - that is the exact condition whose absence protodowns the switch side.[4] - Now create the failure in the safe direction: move one member to a port on a different switch that is not in the same MLAG pair. Expected:
/proc/net/bonding/bond0shows two different partner MACs and the aggregator splits. Rollback: move the cable back. - Write the one-paragraph explanation you would give a customer’s server team about why the switch protodowns a bond it could technically forward on.
- Bonus, no mutation:
ethtool <iface>andip -d link show bond0to record speed, duplex and the bonding parameters actually in force, so you can compare them with what the switch believes.
Optional, customer lab. Bond the same host to a real MLAG pair and compare nv show mlag on both peers - state, local-role, peer-alive, backup-active - against the host’s partner MAC view.[4] Read-only on the switches; take no action beyond nv show.
Goal. Build an MLAG pair, break it the way customers break it, and recover. VLANs, bonds and MLAG are control plane plus kernel data plane, both of which a virtual Cumulus switch does honestly.[13]
- In DSX Air, build
leaf01,leaf02and one host, with two links between the leaves for the peer link and one link from each leaf to the host.[13] Expected: all nodes boot. Remember you cannot add or edit nodes after the simulation first starts. - Configure the peer link, MLAG system MAC, backup IP and priority on both leaves as in the worked example. Apply with
nv config apply --confirm <time>so a mistake rolls back. Expected:nv show mlagshowspeer-alivetrue and onelocal-roleprimary, one secondary.[4] Ifbackup-activeis false, your backup IP is unreachable - check that it is not the peer-link address. - Add VLAN 10, the host bond with
bond mlag id 1on both leaves, and the VRR gateway. Expected: the host bond comes up and the host can ping 10.1.10.1.[5] - Break it: configure the same MLAG system MAC on a second pair the host can reach, or simply set the host’s bond to a second pair. Expected: bonds protodown and the log carries “lacp partner mac mismatch”.[4]
- Recover in the right order: fix the duplicate MAC first, then
nv action clear mlag lacp-conflict. Expected: bonds return. If they protodown again within seconds, you cleared before fixing.[4] - Reload
leaf02and time the outage on the host. Expected: roughlyinit-delayseconds. Record the number and compare with the 180 s default.[4] - In
~/containerlab, build the same logical topology on Dell Enterprise SONiC 4.5.1 and write the mapping table: peer link, keepalive, LAG id, and the one Cumulus concept with no direct equivalent - the out-of-band backup IP.[4] Use your Dell Enterprise SONiC documentation for the MC-LAG command syntax; the compatibility matrix in this lesson’s sources covers platforms and bundles, not commands.[11]
Rollback. Delete the Air simulation; containerlab destroy -t <topology> for the SONiC lab.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell engineer who knows VLT, in five sentences, how a Cumulus MLAG pair keeps a dual-attached host alive, and which single mistake protodowns both bonds.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 documentation, fetched 2026-09-07. Dates are when each page was fetched.
- VLAN-aware Bridge Mode | Cumulus Linux 5.18 · fetched 2026-09-07
- Ethernet Bridging - VLANs | Cumulus Linux 5.18 · fetched 2026-09-07
- Bonding - Link Aggregation | Cumulus Linux 5.18 · fetched 2026-09-07
- Multi-Chassis Link Aggregation - MLAG | Cumulus Linux 5.18 · fetched 2026-09-07
- Virtual Router Redundancy - VRR | Cumulus Linux 5.18 · fetched 2026-09-07
- LACP Bypass | Cumulus Linux 5.18 · fetched 2026-09-07
- Interface Configuration and Management | Cumulus Linux 5.13 · fetched 2026-09-07
- Switch Port Attributes | Cumulus Linux 5.18 · fetched 2026-09-07
- Network Troubleshooting | Cumulus Linux 5.18 · fetched 2026-09-07
- Upgrading Cumulus Linux | Cumulus Linux 5.18 · fetched 2026-09-07
- Dell Enterprise SONiC Distribution Compatibility Matrix Release 4.4.0 · fetched 2026-09-07
- New and Removed NVUE Commands | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA DSX Air User Guide · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.