The underlay: eBGP unnumbered, ECMP and VRFs
S2·E4Two spines, eighty-twenty · NVIDIA briefing room, design review, three days from acceptance
Builds on: The NVUE object model, revisions and snippets, Layer 2 that survives a reload: bridges, bonds, MLAG and VRR
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Configure eBGP unnumbered peering in NVUE and read its vtysh equivalent.
- Explain how RFC 8950 extended next-hop encoding lets an IPv4 prefix carry an IPv6 next hop.
- Analyze an ECMP imbalance and choose between hash fields, a hash seed and resilient hashing.
- Scope a tenant VRF correctly, including what a VRF costs you in adaptive-routing eligibility.
Episode 4 — Two spines, eighty-twenty
The customer’s network lead turns his laptop around: during the training run spine01 sits near 80 percent and spine02 near 20, and the underlay you are proposing has no IPv4 addresses on the fabric links at all. His notebook is four pages fuller than it was on the pilot night. Behind him sit his IPAM team, the Dell SE, and an NVIDIA PM with a roadmap slide queued and a lawyer’s vocabulary for everything on it. “Show me the counter,” the lead says. “Not the theory.”
Two answers, and they have nothing to do with each other. The addressing one is a standard: unnumbered peering forms over IPv6 link-local and carries IPv4 prefixes with an IPv6 next hop, because the receiver decides the next-hop family from the length field rather than from the prefix’s AFI.[3] The imbalance one is arithmetic: BGP multipath is on by default, the default hash includes the L4 ports, and one RoCE queue pair is one UDP source port toward 4791 — one 5-tuple, one hash result, one path.[1][4]
The underlay is built this way on purpose. Everything the customer is buying above it assumes a set of equal-cost layer 3 paths in the default VRF that simply work, and addressing fabric links buys nothing except addresses to maintain and monitor.[4] Boring is the design goal, not a compromise.
One flow is one path — count the flows in the test before you change a switch.
You have an hour before the break, and the whole answer is in this lesson.
1An underlay with no addresses on it
Cumulus Linux runs FRR as its routing control plane, so everything here has an nv form and a vtysh form that are two views of the same daemon.[1] The NVUE side of an eBGP unnumbered leaf is short:[2]
nv set router bgp autonomous-system 65101
nv set router bgp router-id 10.10.10.1
nv set vrf default router bgp neighbor swp51 remote-as external
nv set vrf default router bgp neighbor swp52 remote-as external
nv set vrf default router bgp address-family ipv4-unicast enable on
nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32
nv config applyThe equivalent in vtysh is what you will see in a cl-support bundle or a customer’s older runbook: router bgp 65101, bgp router-id 10.10.10.1, neighbor swp51 interface remote-as external, address-family ipv4 unicast, neighbor swp51 activate, write memory.[2] NVUE also accepts the auto forms nv set router bgp autonomous-system leaf and … spine.[2] One documentation wrinkle to expect: the same NVUE reference page shows both address-family ipv4-unicast enable on and address-family ipv6-unicast state enabled; state enabled/state disabled is the newer normalized form and both appear in current material.[2]
The standard underneath is worth getting right because customers audit it. Cumulus states that “The BGP unnumbered standard in RFC 5549, uses ENHE and does not require that you advertise an IPv4 prefix together with an IPv4 next hop.”[1] RFC 5549 has since been obsoleted by RFC 8950, same title, Standards Track.[3] The Extended Next Hop Encoding capability code is 5, and RFC 8950’s table allows NLRI of AFI 1 with SAFI 1, 2 or 4 to carry a next hop of length 16 or 32 bytes - an IPv6 address.[3] The receiver rule is the one to teach: receivers “determine the protocol type by examining the ‘Length of Next Hop Address field’” rather than assuming it matches the NLRI AFI.[3]
Diagram source (Mermaid)
flowchart TB SP1["spine01 - AS 65199"] SP2["spine02 - AS 65199"] L1["leaf01 - AS 65101 - lo 10.10.10.1/32"] L2["leaf02 - AS 65102 - lo 10.10.10.2/32"] L1 -- swp51 unnumbered --- SP1 L1 -- swp52 unnumbered --- SP2 L2 -- swp51 unnumbered --- SP1 L2 -- swp52 unnumbered --- SP2 H1["GPU host - RoCEv2 UDP 4791"] --- L1 H2["GPU host"] --- L2
2ECMP: on by default, and the three knobs behind it
“Cumulus Linux enables BGP multipath by default”, so a correct unnumbered fabric load shares without extra configuration.[1] The default hash fields are IP protocol, source IP, destination IP, source port, destination port and IPv6 flow label; ingress interface, GTP TEID and inner packet headers are off.[4] Fields change through NVUE, or through /etc/cumulus/datapath/traffic.conf when NVUE is not in use:[4]
nv set system forwarding ecmp-hash source-port off
nv set system forwarding ecmp-hash destination-port off
nv config applyThere is a consistency rule with a silent failure mode: “Make sure that the settings for the source IP and destination IP fields match, and that the settings for the source port and destination port fields match; otherwise Cumulus Linux disables symmetric hashing automatically.”[4] Nothing errors - symmetric hashing just stops being symmetric.
On the BGP side, FRR’s maximum-paths (1-128) sets “the maximum-paths value used for ecmp calculations for this bgp instance”, and the ceiling “can be limited by the ecmp cli for bgp or if the daemon was compiled with a lower ecmp value”.[7] The FRR page states no default value for maximum-paths, so do not quote one to a customer - read it back from the running instance instead.[7] The knob that actually matters in a one-AS-per-leaf design is bgp bestpath as-path multipath-relax: it makes “BGP decision process… consider paths of equal AS_PATH length candidates for multipath computation. Without the knob, the entire AS_PATH must match for multipath computation.”[7] Two spines with different ASNs produce two different AS_PATHs of equal length, so without multipath-relax you get one path and a customer who believes ECMP is broken.
Resilient hashing is a separate mechanism: next hops are assigned “in round robin fashion to a fixed number of buckets”, with all ECMP groups sharing 65,536 buckets by default, and it prevents disruption when a next hop is removed while adds may still disrupt flows.[4] One hard interaction to memorise: adaptive routing does not use resilient hashing - they are not complementary and must not be enabled together as a belt-and-braces configuration.[4]
3Polarization, and why one RoCE flow proves nothing
Two different complaints sound identical and have opposite fixes.
“My single test flow only uses one spine.” That is not a fault. The default hash includes L4 ports, and a RoCE queue pair is one UDP source port toward destination port 4791 - one 5-tuple, therefore one hash result, therefore one path.[4] A customer running one ib_write_bw stream or one iperf3 connection and concluding ECMP is broken is measuring the definition of a flow. The answer is more flows, not more configuration.
“Many flows, but they all pile onto the same spine at the next tier.” That is polarization: every switch computes the same hash over the same fields, so a decision made at the leaf is repeated identically at the spine. The documented fix is a per-switch hash seed:[4]
nv set system forwarding hash-seed 50
nv config applyThe range is 0-4,294,967,295, and the Linux form is ecmp_hash_seed = 50 in /etc/cumulus/datapath/traffic.conf followed by sudo systemctl restart switchd.service - and “Restarting switchd interrupts all network services.”[4] Treat a seed change as a maintenance-window item on every switch that needs one, not a live tweak on one box.
The third possibility is that the customer expected adaptive routing to be doing this and it is not eligible. Adaptive routing is a Spectrum-X feature with narrow gates: Spectrum-4 at 400G or 200G, RoCEv2 unicast or VXLAN-encapsulated RoCE, on layer-3 next-hop router interfaces in the default VRF - not on subinterfaces, SVIs, bonds or bond members - and there is no 800G link support on Spectrum-4.[11][4] Enabling it is two commands, globally and per interface:[11]
nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled
nv config apply
nv show router adaptive-routingBurst armed: 8 packets, one RDMA QP
Every packet of this QP carries the same 5-tuple: same source and destination IP, UDP, destination port 4791, and one source port picked by the QP. The default ECMP hash reads exactly those fields, so all 8 packets choose the same uplink — swp51, which is already carrying a background flow.
- Adaptive routing "improves network utilization by selecting forwarding paths dynamically based on the state of the switch, such as queue occupancy and port utilization" — per packet.
- Both the global and the per-interface knob default to disabled. link-utilization-threshold is also off by default; turned on, the default threshold is 70% and the per-interface override range is 1–100.
- Cumulus Linux 5.14 replaced the old profile-1 / profile-2 options with a unified default profile — a runbook that still names them is stale.
- File form in a cl-support bundle: /etc/cumulus/switchd.d/adaptive_routing.conf, where the global knob is the inverted key link_utilization_threshold_disabled.
nv set router adaptive-routing state enabled nv set interface swp51 router adaptive-routing state enabled nv config apply nv show router adaptive-routing nv show interface swp51 router adaptive-routingNVUE reference: adaptive routing
- ok Traffic type. RoCEv2 unicast is on the eligible list. “AR-eligible traffic is RoCEv2 unicast and VXLAN-encapsulated RoCE.”
- ok Interface type. A routed swp is a layer-3 next-hop router interface — the only eligible kind. “Adaptive routing does not support layer 3 subinterfaces, SVIs, bonds, or bond members.”
- ok VRF. Default VRF — the documented scope. “Adaptive routing applies to layer-3, default-VRF next-hop router interfaces.”
- ok ASIC. Spectrum-4 is the supported ASIC from Cumulus Linux 5.6 on. “Cumulus Linux 5.6 supports adaptive routing on Spectrum-4. Cumulus Linux 5.5 and earlier supports adaptive routing on Spectrum-3 and Spectrum-2 as a Beta feature for evaluation.”
- ok Link speed. 400G is one of the two documented speeds. “NVIDIA Spectrum-4 at 400G/200G.”
- ok Far-end NIC. A SuperNIC on a Spectrum-X switch: the NIC firmware arms AR, and it reorders the sprayed packets at the receiver. “Adaptive Routing is supported with Spectrum-X solution only.”
- always Resilient hashing. Always: AR and resilient hashing are not a belt-and-braces pair. If the customer enabled resilient_hash_enable in traffic.conf as a load-sharing fix, that is a separate mechanism, not an AR helper. “Adaptive routing has no integration with resilient hashing.”
FAE angle: check in this order — ASIC and speed, then interface and VRF, then the traffic itself, then the far-end NIC. Almost every “adaptive routing does nothing” escalation ends on one of these rows, not in a bug.
Why the hash collides (ECMP defaults)
- On by default: IP protocol, source IP, destination IP, source port, destination port, IPv6 flow label.
- Off by default: ingress interface, GTP TEID, inner packet fields.
- Polarization fix is a per-switch seed: nv set system forwarding hash-seed 50 (range 0–4,294,967,295).
- Resilient hashing shares 65,536 buckets by default — and adaptive routing does not use resilient hashing.
Numbers you can quote
- Failover 2.68 ms, described as 400× faster than the 1.08 s of software solutions.
- Tail latency P99 8–9 µs versus 22 µs on traditional Ethernet.
- Capacity-proportional degradation: 11% bandwidth loss for 10% link failures.
- DeepSeek-V3 step time 668 ms under isolation versus 1.18 s (1.6× degradation) on traditional Ethernet.
⚠ The tick counts and queue depths in this simulation are a teaching model, not measurements. Quote the numbers above, never the ticks.
4VRFs and the size of the table you are filling
A VRF is nv set vrf BLUE table auto or an explicit id, and “The table ID range must be between 1001 to 1255”.[5] Cumulus supports up to 255 VRFs; names are at most 15 characters and “you cannot use the name mgmt”.[5] Route leaking is a BGP operation:[5]
nv set vrf RED router bgp address-family ipv4-unicast route-import from-vrf list BLUE
nv set vrf RED router bgp address-family ipv4-unicast route-import from-vrf route-map BLUEtoRED
nv config applyTwo documented limits shape designs: “You cannot reach the loopback address of a VRF… from another VRF”, and “You cannot configure EVPN address families within a VRF.”[5] The second one is why the next lesson enables the L2VPN EVPN address family in the default VRF and maps tenant VRFs to L3 VNIs instead.
Forwarding profiles decide how the ASIC’s shared table space is carved up. nv set system forwarding profile <profile-name> then a switchd restart applies it, and on Spectrum-2 and later the published allocations are: default 50k MAC / 41k IPv4 neighbors / 82k IPv4 LPM; l2-heavy 115k / 74k / 16k; v4-lpm-heavy 16k / 41k / 124k; lpm-balanced 16k MAC with 16k+12k IPv6 neighbors and 124k+124k IPv6 LPM.[6] The documentation cautions these are “maximum values that Cumulus Linux software allocates” and that “theoretical hardware limits might be higher”.[6]
Finally, two 5.18 items to raise before a customer standardises. nv set vrf <vrf-id> router bgp plane-id arrived in 5.18 as the multiplane hook in BGP.[9] And 5.18.1 carries two open issues that read exactly like an underlay fault: “BGP peer sessions might repeatedly flap and re-establish” due to nexthop handling failures, and “Traffic might not be forwarded on some interfaces in an ECMP group”.[8] If a customer on 5.18.1 reports either symptom, check the release notes before you spend a day on their hash configuration.
5Proving the mechanism, not the output
Design. Two leaves, two spines. One AS per leaf (65101, 65102), one AS for the spines (65199). Loopbacks 10.10.10.1/32 and 10.10.10.2/32. No IPv4 on any fabric link.
Step 1 - leaf01.[2]
nv set interface lo ip address 10.10.10.1/32
nv set interface swp51,swp52
nv set router bgp autonomous-system 65101
nv set router bgp router-id 10.10.10.1
nv set vrf default router bgp neighbor swp51 remote-as external
nv set vrf default router bgp neighbor swp52 remote-as external
nv set vrf default router bgp address-family ipv4-unicast enable on
nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32
nv config apply --confirm <time>Step 2 - prove ENHE, not just adjacency. A session that is Established proves TCP and BGP agreed on something; capability 5 proves they agreed on this.[3]
sudo vtysh -c "show bgp neighbor swp51" | grep -iA3 "extended nexthop"Expected: the Extended Next Hop Encoding capability advertised and received.
Step 3 - prove the forwarding shape.
ip route show 10.10.10.2/32
sudo vtysh -c "show ip route 10.10.10.2/32"Expected: an IPv4 prefix whose next hops are IPv6 link-local addresses on swp51 and swp52, two of them, because multipath is on by default.[1]
Step 4 - prove it on the wire. Capture the UPDATE and read the next-hop length field - 16 bytes for a single IPv6 next hop.[3] Remember that tcpdump on a switch sees control-plane traffic only, which is exactly what BGP is, so this capture is legitimate where a data-plane capture would not be.[10]
Step 5 - make ECMP measurable. Generate at least 16 flows with different L4 source ports and count per-interface transmit counters. Then repeat with one flow and observe that it uses one path. Record both numbers - the second one is the answer to the customer question you will get.[4]
Step 6 - troubleshooting on an unnumbered fabric. Enable extended traceroute, which implements RFC 5837 and shows interface details where hops have no unique address:[10]
nv set system global icmp ipv4 errors-extension ingress-interface
nv config applyRollback for every mutating step above. The block was applied with --confirm, so doing nothing rolls it back; explicit inverses are nv unset vrf default router bgp neighbor swp51, nv unset router bgp autonomous-system and nv unset system global icmp ipv4 errors-extension.
nv set router bgp autonomous-system ______
nv set vrf default router bgp neighbor swp51 remote-as ________
nv set vrf default router bgp address-family ipv4-unicast ______ on
nv config apply --________ <time>
sudo vtysh -c "show bgp neighbor swp51" | grep -i "____________" # which capability?
ip route show 10.10.10.2/32 # how many next hops, and of what family?
nv set system forwarding ______ 50 # break polarization; what does it cost?Then answer: which of these steps proves the mechanism and which merely proves the session came up?
A Dell customer’s AI pod has two spines. Their monitoring shows spine01 at 80% utilisation and spine02 at 20% during training runs. BGP is unnumbered, multipath is default, the hash fields are default, and all leaves have the same hash seed. Adaptive routing is configured globally but the uplinks are 800G on Spectrum-4.
Produce: (1) the two independent causes present in this description and the command that confirms each; (2) which one you would fix first and why; (3) the exact eligibility sentence that disqualifies adaptive routing here; (4) the maintenance impact of your recommended change, quoted from the documentation; (5) one sentence you would say to the customer’s architect about what the fabric can and cannot do at 800G today.
Acceptance: your answer does not recommend enabling resilient hashing alongside adaptive routing, and it states the switchd-restart consequence explicitly.
The slide you redraw at the break
Their measurement was a single stream, so 80/20 is the definition of a flow rather than a fault; you ask for many flows before anyone changes a setting.[4] If the pile-up survives that, the fix is a per-switch hash seed, and you name the cost out loud: applied through /etc/cumulus/datapath/traffic.conf it restarts switchd, which interrupts all network services — a maintenance-window item, never a live tweak.[4] Adaptive routing is not the answer here either: their uplinks are 800G on Spectrum-4, which the eligibility rules exclude.[11]
The lead closes the notebook and books the tenant test for the morning. At 06:20 he messages you: tenant RED crosses leaves, tenant BLUE does not.
Lab
Goal. Two Dell-lab hosts with ConnectX ports, running FRR, become the leaf and spine. You are testing hashing behaviour on real 100/200G links.
Pre-flight inventory:
ip -br addr ; ip -br link > before.txt
sudo systemctl is-active frr >> before.txt
sudo vtysh -c "show running-config" >> before.txt- Install and enable FRR on both hosts, and configure unnumbered eBGP between them over two links, advertising a loopback from each.[2] Expected: two sessions Established and each loopback reachable over two paths. Rollback: restore
before.txt’s running-config withvtysh -c "configure terminal"and stop FRR if it was not previously running. - Confirm capability 5 and the link-local next hops exactly as in the no-hardware lab, steps 2 and 3.[3]
- Generate load with
iperf3using several client streams on different source ports, and watch distribution withss -tanplus per-interface counters fromip -s link. Expected: multiple flows spread across both links. - Now the RoCE case: run a single
ib_write_bwstream and observe that one link carries it. Write the explanation in terms of the 5-tuple: one queue pair is one UDP source port toward 4791, so the hash has nothing to spread.[4] No mutation is needed for this step. - Record, in two lines, what this experiment can and cannot tell you about a Spectrum-4 fabric: Linux ECMP hashing is not the switch ASIC’s hash, and adaptive routing is a switch-side, Spectrum-X-only mechanism you cannot exercise here.[11]
Optional, customer lab, read-only. On a live leaf: nv show vrf default router bgp, nv show system forwarding ecmp-hash, and nv show router adaptive-routing. Take no nv set action on a customer switch during a fact-find.
Goal. Build the underlay twice - once in FRR containers where you can capture packets, once in NVUE where you get the production syntax - and confirm the mechanism rather than the output.
- In
~/containerlab, wire four FRR containers as two leaves and two spines, each leaf with one link to each spine. Configure unnumbered eBGP invtyshusing the forms from segment 1.[2] Expected: four Established sessions. - Prove ENHE:
show bgp neighbor <iface>and look for the Extended Next Hop Encoding capability, code 5.[3] Expected: advertised and received. If missing, the neighbor was configured with an address rather thaninterface remote-as external. - Prove the forwarding shape:
show ip routefor a remote loopback. Expected: an IPv4 prefix with IPv6 link-local next hops out of two interfaces.[3] - Prove it on the wire:
tcpdump -i <link> -nvv tcp port 179while you clear the session, and read the next-hop length in the UPDATE. Expected: a 16-byte next hop.[3] - Repeat the same topology in DSX Air using the NVUE syntax. Expected:
nv show vrf default router bgp neighborshows the same two sessions. Apply withnv config apply --confirm <time>so a wrong ASN rolls back instead of isolating the node. - Generate 16 flows with different L4 source ports between two hosts and read per-interface counters. Expected: both uplinks carry traffic. Then repeat with a single source port. Expected: one uplink only. Write down both results side by side - this is the polarization demonstration that motivates the hash seed.[4]
- Read the current hash configuration with
nv show system forwarding ecmp-hashand predict, before running it, what asource-port offchange would do to your 16-flow test.[4]
Rollback. containerlab destroy -t <topology> for the FRR lab; delete the Air simulation. On the Air nodes, any applied change is reversible with nv config apply startup.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a customer's IPAM team, in four sentences, why an AI fabric underlay needs no IPv4 addresses on its links, and what their monitoring must understand instead.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 documentation, RFC 8950 and docs.frrouting.org, fetched 2026-09-07. Dates are when each page was fetched.
- Border Gateway Protocol - BGP | Cumulus Linux 5.18 · fetched 2026-09-07
- Basic BGP Configuration | Cumulus Linux 5.18 · fetched 2026-09-07
- RFC 8950 - Advertising IPv4 NLRI with an IPv6 Next Hop · fetched 2026-09-07
- Equal Cost Multipath Load Sharing | Cumulus Linux 5.18 · fetched 2026-09-07
- Virtual Routing and Forwarding (VRF) | Cumulus Linux 5.18 · fetched 2026-09-07
- Forwarding Table Size and Profiles | Cumulus Linux 5.18 · fetched 2026-09-07
- FRRouting BGP documentation · fetched 2026-09-07
- Cumulus Linux 5.18 Release Notes · fetched 2026-09-07
- New and Removed NVUE Commands | Cumulus Linux 5.18 · fetched 2026-09-07
- Network Troubleshooting | Cumulus Linux 5.18 · fetched 2026-09-07
- NVUE Reference - Adaptive Routing set/unset commands · fetched 2026-09-07
- Cumulus Linux in a Virtual Environment | Cumulus Linux 5.18 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.