BGP-EVPN symmetric IRB for tenant isolation
S2·E5Tenant BLUE, seven hours out · Hotel lobby, 06:20, acceptance review the same afternoon
Builds on: Layer 2 that survives a reload: bridges, bonds, MLAG and VRR, The underlay: eBGP unnumbered, ECMP and VRFs
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish symmetric from asymmetric IRB by the lookups each PE performs and the routes each uses.
- Configure EVPN with L2 VNIs, tenant VRFs and L3 VNIs on a Cumulus leaf pair.
- Diagnose the four documented causes of broken tenant routing in the order that finds them fastest.
- Assess what a tenant VRF costs a Spectrum-X fabric in adaptive-routing eligibility.
Episode 5 — Tenant BLUE, seven hours out
The PoC leaves ran clean when you left the customer’s lab last night. This morning tenant RED still works across leaves and tenant BLUE does not: its hosts reach each other inside a leaf and nothing crosses between them. show bgp l2vpn evpn summary is healthy, BLUE’s Type-2 routes are sitting in the BGP table on both leaves, and this afternoon the network lead will ask you, in the room where the order gets signed, whether tenant isolation on this fabric is real.
What you have is a short list, not a packet capture. A layer 3 VNI whose id collides with a layer 2 VNI creates nothing and errors nowhere.[5] A VNI MTU that is not 50 bytes below the physical MTU passes ping and fails large transfers.[9] On an MLAG pair the anycast VTEP address has to be identical on both peers.[7] And routes can sit in BGP indefinitely if the route targets are not imported and exported symmetrically.[10]
EVPN exists so an overlay stops learning by flooding. A standards-based MP-BGP control plane advertises MACs, IP addresses and prefixes between VTEPs, and symmetric IRB has both ends route into a per-tenant layer 3 VNI instead of every leaf carrying every tenant’s VLANs.[1][4] That is what turns isolation from a promise into a route table.
When the control plane looks healthy and traffic does not move, suspect an id, an MTU or an import — in that order.
Which of the four it is, is the work of this lesson.
1What EVPN actually standardises
EVPN is “a standards-based control plane for VXLAN defined in RFC 7432 and RFC 8365”, using MP-BGP and building on the BGP/MPLS IP VPN model of RFC 4364.[1] Three documents matter and each contributes something you will be asked about.
RFC 7432 defines the route types: Type 1 Ethernet Auto-Discovery, Type 2 MAC/IP Advertisement, Type 3 Inclusive Multicast Ethernet Tag, Type 4 Ethernet Segment.[2] The NLRI uses AFI 25 (L2VPN) / SAFI 70 (EVPN), and the Ethernet Segment Identifier is a 10-octet value - one type octet plus nine value octets - where “ESI 0 denotes a single-homed site” and all-0xFF is reserved as MAX-ESI.[2]
RFC 8365 is the overlay half: “VXLAN encapsulation is based on UDP, with an 8-byte header following the UDP header. VXLAN provides a 24-bit VNI”, and IANA assigns BGP Tunnel Encapsulation Attribute value 8 = VXLAN Encapsulation.[3]
RFC 9135 defines integrated routing and bridging, and the distinction that decides your design. Asymmetric IRB: “the ingress PE performs three lookups, whereas the egress PE performs a single lookup — i.e., the ingress PE performs a MAC lookup, followed by an IP lookup, followed by a MAC lookup again. The egress PE performs just a single MAC lookup.”[4] Symmetric IRB: “the lookup operation is symmetric at both the ingress and egress PEs — i.e., both ingress and egress PEs perform lookups on both MAC and IP addresses.”[4] Symmetric uses Type 2 with the Label2 field populated plus Type 5 for prefixes; asymmetric uses Type 2 without Label2.[4]
Cumulus says the same in operational words: symmetric means “both the ingress VTEP and egress VTEP route the packets”, while asymmetric means “Only the ingress VTEP performs routing, the egress VTEP only performs bridging”.[5] Type-5 routes “carry the layer 3 VNI and router MAC address and follow the symmetric routing model”.[5]
2The Cumulus configuration, in the order that works
EVPN and its BGP address family go on in the default VRF - the VRF documentation is explicit that “You cannot configure EVPN address families within a VRF”.[16][6]
nv set evpn state enabled
nv set vrf default router bgp address-family l2vpn-evpn state enabled
nv set vrf default router bgp neighbor swp51 address-family l2vpn-evpn state enabled
nv set vrf default router bgp neighbor swp52 address-family l2vpn-evpn state enabledThen the overlay objects. The L2 side maps a VLAN to a VNI; the L3 side maps a tenant VRF to an L3 VNI; the VTEP source is a loopback address:[6][5]
nv set bridge domain br_default vlan 10 vni 10
nv set bridge domain br_default vlan 20 vni 20
nv set vrf RED evpn vni 4001
nv set vrf BLUE evpn vni 4002
nv set nve vxlan source address 10.10.10.1
nv set nve vxlan arp-nd-suppress enabled
nv config applyNVUE builds the plumbing for you: it “automatically creates a single VXLAN device (vxlan48), adds the VXLAN device to the default bridge br_default”.[6] The MAC-VRF service model is VLAN-based, which “requir[es] a one-to-one mapping between L2 VNIs and VLANs” - so a VLAN belongs to exactly one VNI and vice versa.[6]
On an MLAG pair there is one more mandatory object. The VTEP address must be an anycast IP on loopback, set with nv set nve vxlan mlag shared-address <IP>; “The anycast virtual IP address for VXLAN termination must be the same on both switches in the MLAG pair”, and NVIDIA “recommends that you configure the anycast IP address before you configure the MLAG interfaces”.[7] For symmetric mode specifically: “If you use VXLAN active-active with EVPN symmetric mode, you must set the anycast MAC address on both switches in the MLAG pair.”[7] clagd adds the anycast address on successful peering and withdraws it during failover when the secondary’s peer link fails.[7]
Diagram source (Mermaid)
flowchart LR A["tenant RED host 10.1.10.5 on VLAN 10"] --> L1["leaf01 ingress VTEP"] L1 -- "route into L3 VNI 4001" --> V["VXLAN over UDP 4789 - underlay ECMP in default VRF"] V --> L2["leaf02 egress VTEP"] L2 -- "route out of L3 VNI 4001" --> B["tenant RED host 10.1.20.9 on VLAN 20"] X["tenant BLUE - L3 VNI 4002"] -.- V
3The four things that break it, in the order to check them
Every EVPN escalation in this material reduces to four causes, and they are ordered by how cheap they are to rule out.[5][9][7][10]
- An L3 VNI colliding with an L2 VNI id. “A layer 3 VNI and a layer 2 VNI cannot have the same ID. If the VNI IDs are the same, Cumulus Linux does not create the layer 2 VNI.”[5] Nothing errors. The symptom is a tenant VLAN that stopped bridging shortly after somebody added a VRF. Check with
nv show evpn vniand look at the type column.[10] - A VNI MTU that is not 50 bytes below the physical MTU. “The MTU for a virtual network interface (VNI) must be 50 bytes smaller than the MTU of the physical interfaces on the switch.”[9] Small packets work, large ones do not, and the application team reports it as a storage problem. NetQ exists here for a reason: it runs link-MTU tests 0, 1 and 2 for physical interfaces, SVIs and bridge interfaces plus BGP test 7 for interface MTU.[11]
- An MLAG pair without a matching anycast VTEP - and, in symmetric mode, without the anycast MAC.[7] The symptom is traffic that works through one leaf and not the other, or that breaks on failover only.
- Asymmetric route targets.
nv show evpn vni <vni> route-targetshows[both]when route targets are auto-derived and the same RT is imported and exported;[10] anything else means routes can sit in the BGP table and never be imported into the tenant VRF.
The 5.18 troubleshooting page gives an order of operations for the general case: show bgp l2vpn evpn summary for sessions and prefix counts, then show evpn vni detail / nv show evpn vni for VNI presence and typing, then show evpn mac vni <vni> vtep <remote-ip> for learning from the expected VTEP, then show bgp l2vpn evpn route rd <rd> for the route with its RD and extended communities, then show ip route vrf <vrf-id> for installation in the tenant VRF.[10]
For symmetric routing specifically, four more commands earn their place: nv show vrf <vrf-id> evpn for L3 VNI state, SVI and router MAC; show evpn rmac vni all for remote router MACs - the page frames it as the symmetric-routing check, and a missing RMAC leaves the route installed but the packet unbuildable, so the failure raises no error anywhere;[10] show evpn next-hops vni <l3-vni> for remote VTEP IPs and their router MACs; and show ip route vrf <vrf-id> to confirm remote prefixes have correct onlink next hops.[10] Data-plane confirmation is ip -d link show type vxlan, which shows the VNI, tunnel endpoints and UDP port 4789.[10]
BlueField-3 on a Dell PowerEdge: what is the symptom?
4Building it, and what tenancy costs on a Spectrum-X fabric
Design. leaf01 and leaf02 over the m2-04 unnumbered underlay. Tenant RED: VLAN 10 (VNI 10) and VLAN 20 (VNI 20), L3 VNI 4001. Tenant BLUE: VLAN 30 (VNI 30), L3 VNI 4002. VTEP source is the loopback; the leaves here are standalone, so no anycast address is needed - the MLAG variant is step 6.
Step 1 - underlay first, and prove it. From lesson 4: eBGP unnumbered, loopbacks reachable, ECMP present. Do not proceed until ip route show <remote loopback> shows two next hops. Every EVPN problem that is actually an underlay problem wastes an afternoon.[10]
Step 2 - turn on EVPN in the default VRF.[6]
nv set evpn state enabled
nv set vrf default router bgp address-family l2vpn-evpn state enabled
nv set vrf default router bgp neighbor swp51 address-family l2vpn-evpn state enabled
nv set vrf default router bgp neighbor swp52 address-family l2vpn-evpn state enabledStep 3 - tenant VRFs and L3 VNIs. Pick L3 VNI ids far away from every L2 VNI id you will ever use; 4001 and 4002 against L2 VNIs 10, 20 and 30 leaves no room for a collision.[5]
nv set vrf RED evpn vni 4001
nv set vrf BLUE evpn vni 4002Step 4 - L2 VNIs, SVIs in the right VRF, and the VTEP. An interface joins a VRF with nv set interface <id> vrf <vrf>.[6][8][16]
nv set bridge domain br_default vlan 10 vni 10
nv set bridge domain br_default vlan 20 vni 20
nv set bridge domain br_default vlan 30 vni 30
nv set interface vlan10 vrf RED
nv set interface vlan20 vrf RED
nv set interface vlan30 vrf BLUE
nv set nve vxlan source address 10.10.10.1
nv set nve vxlan arp-nd-suppress enabled
nv config apply --confirm <time>Step 5 - verify in the documented order.[10]
sudo vtysh -c "show bgp l2vpn evpn summary"
nv show evpn vni
sudo vtysh -c "show evpn rmac vni all"
sudo vtysh -c "show evpn next-hops vni 4001"
sudo vtysh -c "show ip route vrf RED" # expect onlink next hops
ip -d link show type vxlan # expect VNI list and UDP 4789
nv show evpn vni 10 route-target # expect [both]Step 6 - the MLAG variant. If these leaves are an MLAG pair, add the anycast VTEP before the MLAG interfaces, identical on both, and set the anycast MAC because this is symmetric mode.[7]
nv set nve vxlan mlag shared-address 10.10.10.100Step 7 - MTU. Set the physical interface MTU deliberately and make the VNI MTU 50 bytes lower.[9]
Rollback. The block was applied with --confirm, so an unreachable leaf recovers by itself. Explicit inverses: nv unset vrf RED evpn vni, nv unset nve vxlan source address, nv unset evpn state.
nv set evpn state ________
nv set vrf ________ router bgp address-family l2vpn-evpn state enabled # which VRF, and why not RED?
nv set vrf RED evpn vni ________ # what must this number NOT collide with?
nv set bridge domain br_default vlan 10 vni ________
nv set nve vxlan source address ________ # standalone leaf
nv set nve vxlan ________ shared-address 10.10.10.100 # MLAG pair - and configured before what?
nv config apply --________ <time>
nv show evpn vni ________ route-target # what value proves symmetric auto-derived RTs?Then write, in one line each, the symptom you would expect from getting the L3 VNI id wrong and from getting the VNI MTU wrong.
A Dell customer runs two tenants on an MLAG leaf pair. Tenant RED works between leaves. Tenant BLUE has hosts that reach each other within a leaf but not across leaves; show bgp l2vpn evpn summary is healthy and BLUE’s Type-2 routes are visible in the BGP table on both leaves. Large NFS reads inside RED fail intermittently while ping and SSH are fine.
Produce: (1) which of the four documented causes explains BLUE and the single command that confirms it; (2) which explains RED’s intermittent large transfers and the arithmetic you would show the customer; (3) the order you would run the five general troubleshooting commands and why you would not start with show ip route vrf BLUE; (4) one NetQ check that would have caught each fault before a human did.
Acceptance: your answer names a different cause for BLUE and for RED, and each is confirmed by one command rather than by a capture.
Multi-tenancy on a Spectrum fabric is NCP-AIN objective 2.3 verbatim: “Configure multi-tenancy Border Gateway Protocol Ethernet VPN (BGP-EVPN) to isolate tenant workloads.”[14] There is one tension to hold honestly in front of an architect. Adaptive routing is documented for RoCEv2 unicast and VXLAN-encapsulated RoCE on layer 3 next-hop router interfaces in the default VRF, and is not supported on SVIs, subinterfaces, bonds or bond members.[15] So the Spectrum-X load balancing lives in the underlay that carries the tunnels, not inside the tenant VRF - and how a given reference architecture reconciles that is a deployment-guide question, not something to assert from the Cumulus page.[15]
Two 5.18 items belong in any tenancy design review, both labelled: Layer 3 VXLAN device mode (Beta), exposed as nv set evpn l3vxi state and nv show evpn l3vxi, and EVPN Multihoming without a unique SVI IP address.[12] Say “Beta” out loud; a customer who discovers it later will treat everything else you said as marketing.
Case closed
BLUE’s layer 3 VNI had been retyped into a collision with a layer 2 VNI, so that L2 VNI was never created.[5] RED’s slow large reads were the missing fifty bytes of MTU.[9] Both fixed before coffee, proved by counters.
What you tell the lead: “Isolation is real, and it is a trade: adaptive routing is documented for layer 3 next-hop interfaces in the default VRF, so load balancing lives in the underlay carrying your tunnels, not the tenant VRF.”[15]
The order releases. Procurement asks her only question, and is right to: lead time. The PM says “Beta” out loud about the 5.18 layer 3 VXLAN device mode, and “not announced” for the rest.[12] The SE closes his row. The fabric is correct; nobody has asked whether it is fast.
Lab
Goal. There is no Spectrum switch locally, so run the contrast case that a Dell FAE actually needs: the same overlay on Dell Enterprise SONiC, and the command-mapping table you will use when the customer’s other fabric is SONiC.
Pre-flight inventory: capture the containerlab topology file and containerlab inspect output before you start.
- In
~/containerlab, bring up two Dell Enterprise SONiC 4.5.1 nodes as leaves with an IP underlay between them, and configure VXLAN EVPN between them using your Dell Enterprise SONiC documentation for syntax. Expected: an EVPN session and a working L2 stretch. Rollback:containerlab destroy -t <topology>. - Build the mapping table an FAE needs. Left column: the Cumulus command from this lesson. Right column: the Dell Enterprise SONiC equivalent you just used. At minimum cover: EVPN session state (
show bgp l2vpn evpn summary), VNI list and typing (nv show evpn vni), MAC learning per VNI, route targets, and the VXLAN device view (ip -d link show type vxlan).[10] - Add two commercial rows that are not commands. Dell Enterprise SONiC gates VXLAN and VXLAN EVPN to the Enterprise Standard, Enterprise Premium and Lite bundles, not the Cloud bundles; and its 4.4.0 compatibility matrix lists only Broadcom-NPU platforms, with no NVIDIA Spectrum platform.[13] Note that your lab runs 4.5.1, so state the release your matrix row came from rather than implying it is current.
- Write the two-sentence answer to “can we just run SONiC on the Spectrum switch?” that names which box, which bundle and which support path - and stop there, because the RA question belongs to module 1.
Optional, customer lab, read-only. On a live leaf pair, repeat the five-step verification from segment 3 and cross-check with netq check evpn, which runs tests 0-6.[11] Read-only: no nv set, no nv action, and no NetQ configuration changes on a customer system during a fact-find.
Goal. Bring up symmetric IRB with two tenant VRFs on the m2-04 underlay, then inject each of the four faults and record which command first shows the symptom. VXLAN bridging and routing both work on a virtual Cumulus switch, so this lab is faithful.
- In DSX Air, build two leaves and two spines and reproduce the m2-04 unnumbered underlay. Expected: each leaf’s loopback reachable from the other over two paths. Do not continue until this is true.
- Enable EVPN and the L2VPN EVPN address family in the default VRF on both leaves.[6] Expected:
show bgp l2vpn evpn summaryshows the sessions with the EVPN address family active. - Create VRFs RED and BLUE with L3 VNIs 4001 and 4002, L2 VNIs 10, 20 and 30, SVIs in the right VRF, and the VTEP source address.[6][5] Apply with
nv config apply --confirm <time>. - Verify along the documented order:
show bgp l2vpn evpn summary,nv show evpn vni,show evpn rmac vni all,show evpn next-hops vni 4001,show ip route vrf RED(expectonlinknext hops),ip -d link show type vxlan(expect UDP 4789),nv show evpn vni 10 route-target(expect[both]).[10] Record each output before you break anything. - Fault 1 - VNI collision. Change RED’s L3 VNI to 10 so it collides with an L2 VNI. Expected: the L2 VNI is not created.[5] Record which command shows it first. Rollback: set it back to 4001 and apply.
- Fault 2 - MTU. Set a VNI MTU that is not 50 bytes below the physical MTU and test with large packets.[9] Expected: small packets pass, large ones do not. Rollback: restore the MTU.
- Fault 3 - MLAG VTEP. If you built the leaves as an MLAG pair, set different anycast addresses on the two peers.[7] Expected: asymmetric behaviour that follows which leaf the traffic entered. Rollback: make them identical again.
- Fault 4 - route targets. Configure a non-auto-derived route target on one leaf only. Expected: routes present in BGP, absent from the tenant route table, and
nv show evpn vni <vni> route-targetno longer[both].[10] Rollback: remove the manual RT. - Write the four-row table: fault, first command that shows it, and the time it took you to find it.
Rollback for the whole lab. Delete the simulation, or nv config apply startup on each node.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell customer's platform architect, in five sentences, how BGP-EVPN keeps two AI tenants apart on one Spectrum fabric, and one thing tenancy costs them.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 documentation, RFC 7432/8365/9135 and NetQ 5.1 validation reference, fetched 2026-09-07. Dates are when each page was fetched.
- Ethernet Virtual Private Network - EVPN | Cumulus Linux 5.18 · fetched 2026-09-07
- RFC 7432 - BGP MPLS-Based Ethernet VPN · fetched 2026-09-07
- RFC 8365 - A Network Virtualization Overlay Solution Using EVPN · fetched 2026-09-07
- RFC 9135 - Integrated Routing and Bridging in Ethernet VPN (EVPN) · fetched 2026-09-07
- Inter-subnet Routing (EVPN) | Cumulus Linux 5.18 · fetched 2026-09-07
- EVPN Basic Configuration | Cumulus Linux 5.18 · fetched 2026-09-07
- VXLAN Active-Active Mode | Cumulus Linux 5.18 · fetched 2026-09-07
- VXLAN Routing | Cumulus Linux 5.18 · fetched 2026-09-07
- Switch Port Attributes | Cumulus Linux 5.18 · fetched 2026-09-07
- Troubleshooting EVPN | Cumulus Linux 5.18 · fetched 2026-09-07
- Validation Tests Reference | Cumulus NetQ 5.1 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- Dell Enterprise SONiC Distribution Compatibility Matrix Release 4.4.0 · fetched 2026-09-07
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) certification page · fetched 2026-09-07
- Equal Cost Multipath Load Sharing (incl. Adaptive Routing) | Cumulus Linux 5.18 · fetched 2026-09-07
- Virtual Routing and Forwarding (VRF) | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.