Skip to content

HBN: BGP/EVPN on the server

S5·E1The fabric that stops one hop short · Hall 2 of a colocation operator's new building, a design review at the whiteboard

S5·E1Analyze~40 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs (HBN 3.5.0), 2026-09-06

Builds on: E-switch, representors, and switchdev, DOCA Ethernet and OVS-DOCA, rshim and the BFB path

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain how nl2docad, FRR, ifupdown2 and the br-hbn bridge cooperate to turn a BlueField-3 into an L3 leaf inside the server.
  • Configure the bf.cfg keys and the static-pod files that bring HBN up, and verify each layer with crictl and supervisorctl.
  • Write an nv (NVUE) configuration for BGP unnumbered plus EVPN-VXLAN and verify offload from the kernel routing tables.
  • Analyze what a Dell PowerSwitch OS10 leaf must be configured with to peer with HBN over unnumbered eBGP.
  • Recognize the scale, memory and mode constraints that decide whether HBN fits a customer design.

Episode 1 — The fabric that stops one hop short

The situation · Hall 2 of a colocation operator's new building, a design review at the whiteboard

The whiteboard has spines along the top, leaves under them, and a gap where the last hop should be. In thirty-one days a paying tenant moves into Hall 2, and the operator’s network lead will not sign the design until somebody answers her question: does the VXLAN fabric end at the top-of-rack, or can it end inside the server? Her host admins want a VLAN per tenant. Her network team says no, permanently. Behind you the Dell SE reheats his coffee for the third time, the elevation for 192 PowerEdge nodes open on his laptop, and the procurement person wants one thing: which BlueField-3 SKU goes on the order, and when.

This is the argument HBN was built to settle. Fabrics went L3 because L3 scales, but the last hop stayed inside the server, where the network team has no reach. HBN encapsulates a set of network functions inside a container deployed as a service pod on BlueField’s Arm cores, so the server itself becomes the first routed hop.[1] FRR pushes the EVPN MACs and routes into the kernel through netlink; nl2docad watches that state and programs the matching packet-processing rules into BlueField hardware through DOCA APIs, so the Arm cores carry the control plane and the exceptions, not the packets.[1]

She is ahead of you: her leaves run OS10, and OS10 documents symmetric IRB with unnumbered BGP peering on its own side.[9] A tenant boundary you cannot route to comes back as a VLAN request. First, though, you have to know what the pod is and what it costs.[2]

1What HBN is: a leaf switch that lives on the DPU

HBN (Host-Based Networking) “encapsulates a set of network functions inside a container, which is deployed as a service pod on BlueField’s Arm cores”, and its design is L3-centric: the server becomes the first routed hop.[1] The services index summarizes HBN as hardware-accelerated BGP/EVPN routing, ECMP, L4 firewall and NAT, and lists it at GA quality in DOCA 3.5.0.[6]

Five parts cooperate. FRR is the routing stack; it “pushes all the control states (EVPN MACs and routes) to kernel via netlink”. nl2docad (“Netlink-to-DOCA”) listens to that kernel state and “accelerates Linux networking using DOCA APIs to program specific packet processing rules in BlueField hardware”. ifupdown2 is the interface manager. br-hbn is an OVS bridge that serves as the “indirection layer between the SF and the actual ports” and is “automatically configured when the BFB image is installed on BlueField with HBN enabled”. The Linux kernel holds the state and carries the punt path for packets hardware cannot handle.[1]

The consequence for an FAE: HBN is not a replacement for OVS-DOCA. It sits on top of an OVS bridge and adds a routing daemon plus an offload translator. When a customer asks whether their VXLAN fabric “ends at the switch or at the server”, HBN is the answer that ends it at the server.[1]

Host (x86)BlueField-3 Arm (DPU or zero-trust mode)Leaf pairpf0hpf_ifnetlinkDOCA FloweBGP · EVPNeBGP · EVPNPF / VFs (host)sees a NICpf0hpf · pf0vf0representorsbr-hbnOVS SFC bridgeSFs pf0dpu1 / pf0dpu3→ *_if inside podHBN containerFRR 10.03 · ifupdown2 · nvnl2docadnetlink → DOCAe-switch tablesroutes · ECMP · VXLANp0 → p0_ifBGP unnumberedp1 → p1_ifBGP unnumberedDell PowerSwitch / OS10leaf-1 · BGP unnumbered · EVPNDell PowerSwitch / OS10leaf-2 · BGP unnumbered · EVPN

HBN: BGP/EVPN router in a container on the Arm

HBN "encapsulates a set of network functions inside a container, which is deployed as a service pod on BlueField's Arm cores": FRR (BGP/EVPN), ifupdown2, nl2docad and the NVUE nv CLI. The DPU becomes the server's first-hop router and VTEP; the host keeps seeing a plain NIC.

Why it matters. HBN 3.5.0 is GA, ships FRR 10.03 on DOCA 3.5.0 (BSP 4.16.0), and "requires BlueField to work in either DPU mode or zero-trust mode". It runs as a kubelet static pod (/etc/kubelet.d/doca_hbn.yaml).

FAE note. BF-3 32 GB: HUGEPAGE_COUNT=5120; qualified for at most 16 VFs alongside other DOCA services; default scaling profile rp_4k_16k (4K VTEPs, 16K L3 EVPN routes). Since 3.5.0 the "container will fail to start if it detects an invalid startup configuration".

Source: networking-docs.nvidia.com

Mode hbn: follow a packet from uplink p0 through br-hbn and the pf0dpu1/pf0dpu3 SFs into the HBN pod and out to pf0hpf. Toggle offload to see which hops nl2docad removes from the software path.

2Service function chaining, SFs and the _if names

HBN is “a ‘bump-in-the-wire’ service and requires specific network configuration on BlueField called service function chaining (SFC)”. SFC makes traffic between the uplinks and the host representors pass through the pod. Inside the container every port carries an _if suffix: p0_if, pf0hpf_if, pf0vf0_if, and the chain SFs pf0dpu1_if and pf0dpu3_if.[1][2] Outside the container the names are the ones from M4: uplinks p0/p1, host PF representor pf<port>hpf, VF representors pf0vf0…, and SF representors such as pf0sf0 or the long form en3f0pf0sf4.[7] The DPF HBN configuration repeats the rule for its own interfaces list: “Interfaces inside the HBN pod must have the _if suffix”.[8]

The chain is created at BFB install time from bf.cfg. The current keys are ENABLE_BR_HBN=yes (mandatory), BR_HBN_UPLINKS="p0,p1", BR_HBN_REPS for the PF/VF ports that join the chain, BR_HBN_SFS='pf0dpu1,pf0dpu3', and HUGEPAGE_COUNT=2048 (5120 for a 32 GB BlueField-3). Legacy files use ENABLE_SFC_HBN=yes with NUM_VFs_PHYS_PORT0=12 and NUM_VFs_PHYS_PORT1=2. Install with bfb-install -c bf.cfg -r rshim0 -b <BFB-image>.[2] Two hard preconditions: “HBN requires BlueField to work in either DPU mode or zero-trust mode” and “HBN requires SFC configuration to be activated on the BlueField before running the HBN service container”.[2]

To regenerate the chain without a reinstall, run /opt/mellanox/sfc-hbn/install.sh -c -r on the DPU; it “generates a new set of sfc.conf and mlnx-sf.conf and reloads the DPU”. The generated files are /etc/mellanox/hbn.conf, /etc/mellanox/sfc.conf and /etc/mellanox/mlnx-sf.conf.[2] A prep script ships in the NGC resource at scripts/doca_hbn/<hbn_version>/hbn-dpu-setup.sh with -m|--mtu, -u|--username, -p|--password and -e|--enable-rest-api-access.[2]

The pod itself follows the common DOCA service model: a kubelet with containerd watches /etc/kubelet.d/; you copy doca_hbn.yaml (from <resource>/configs/<doca_version>/) into that directory and “Kubelet automatically pulls the container image from NGC and spawns a pod”. Verify with sudo crictl pods, sudo crictl ps, sudo crictl logs <id> and sudo journalctl -u kubelet --since -5m; stop by deleting the YAML; a pod “Takes up to 20 seconds”; the YAML must use 2-space indentation; hugepage resource type hugepages-2Mi must match the request and a change needs sudo systemctl restart kubelet.service.[5][2] To get a shell: sudo crictl exec -it $(crictl ps | grep hbn | awk '{print $1;}') bash; inside, supervisorctl status frr, supervisorctl status nl2doca, supervisorctl status neighmgr; logs live under /var/log/hbn.[2][3]

3Configuring routing with nv: BGP unnumbered, EVPN, ECMP

HBN is configured with NVUE’s nv command set. The configuration guide’s reference block is worth memorizing as a shape, not as text:[3]

nv set interface lo ip address 10.10.10.1/32
nv set interface p0_if,p1_if type swp
nv set interface pf0hpf_if bridge domain br_default access 100
nv set interface vlan100 type svi
nv set interface vlan100 vlan 100
nv set interface vlan100 ip address 30.30.30.1/24
nv set vrf default router bgp router-id 10.10.10.1
nv set vrf default router bgp autonomous-system 65501
nv set vrf default router bgp enable on
nv set vrf default router bgp neighbor p0_if type unnumbered
nv set vrf default router bgp neighbor p0_if remote-as external
nv set evpn enable on
nv set nve vxlan enable on
nv set nve vxlan source address 27.0.0.4
nv set bridge domain br_default vlan 200 vni 2000
nv config apply
nv config save
nv show interface ; nv show vrf <vrf-name> router bgp

Applied config lands in /etc/network/interfaces, /etc/frr/frr.conf, /etc/frr/daemons and the auto-generated /etc/nvue.d/startup.yaml.[3] DPF drives the same surface declaratively: its startupYAMLJ2 NVUE document carries the header model: BLUEFIELD, nvue-api-version: nvue_v1, and sets evpn.enable, nve.vxlan.source.address, bridge.domain.br_default.vlan.'10'.vni, router.bgp.autonomous-system and vrf.default.router.bgp.neighbor.p0_if.type: unnumbered with a peer group remote-as: external.[8]

Documented features: “BGP and ECMP for IPv4 and IPv6 traffic” with “Up to 64 paths for ECMP”, BGP unnumbered, VRF route leaking (but not between the default and a non-default VRF), multi-hop eBGP for EVPN, EVPN-VXLAN with L2 bridging and L3 symmetric routing, a single VXLAN device, downstream VNI (DVNI), Type-2 and Type-5 routes, VLAN subinterfaces such as p0_if.100 on uplinks and on host-facing PF/VF ports, and a REST API that HBN enables by default with localhost-only access.[3] nv set system api listening-address 0.0.0.0 (or an explicit IPv4/IPv6 address) is what exposes it to the network and nv set system api state disabled turns it off; the default credentials are nvidia:nvidia and the guide strongly recommends replacing them, which you do before you widen the listening address.[3] Verify offload from the kernel side with ip route show <prefix> and ip nexthop show.[3]

Two documented limits matter in design reviews: “Ping or other IP traffic from a locally connected host in vrfX to a local interface IP address on the BlueField/HBN in vrfY does not work, even if VRF route-leaking is enabled”, and DVNI works only with L3 VNIs on the single VXLAN device.[3]

4Scale, memory and what changed in HBN 3.5.0

HBN 3.5.0 pairs with “DOCA 3.5.0 (BSP 4.16.0) on Ubuntu 22.04 OS”.[4] Note the conflict: the DOCA general-support page states “The default operating system provided with the BlueField bundle (for DPU and SuperNIC) is Ubuntu 24.04 64k” (lessons 1.5 and 2.3), while these HBN 3.5.0 release notes name Ubuntu 22.04.[11][4] Quote both, and confirm with NVIDIA which BFB variant an HBN deployment requires before you push a 24.04 image; record the answer in the qualification plan (lesson 8.3). The release notes list: “Upgraded FRR to version 10.03”; “Added support for B3220-SH Storage Controller”; “Introduced resource scaling profiles”; and a fail-closed rule: “HBN container will fail to start if it detects an invalid startup configuration”.[4] Supported hardware spans BlueField-2 25 GbE/100 GbE variants, BlueField-3 B3210E/B3220/B3240, and the SuperNICs B3210L/B3220L/B3140L/B3140H (running in DPU mode).[4]

Scale item (HBN 3.5.0) BlueField-2 BlueField-3
VTEP peers 8K 16K
L2 VNIs 20 per BlueField 20 per BlueField
EVPN type-5 routes 32K 128K
[4]

Resources: a 32 GB BlueField-3 has a recommended limit of 10 GB with HUGEPAGE_COUNT=5120; the HBN container takes 3 to 6 GB and OVS another 3 to 6 GB depending on profile; and “When running HBN alongside other DOCA services, the system is qualified to support a maximum of 16 VFs”.[2] The default scaling profile is rp_4k_16k (4K VTEPs, 16K L3 EVPN routes); a change in hbn.conf triggers hbn-profile.service and may need a manual container restart.[2] The DPF template for the same service requests memory: 6Gi and nvidia.com/bf_sf: 4 and pins image tag 3.4.0-doca3.4.0 from nvcr.io/nvidia/doca/doca_hbn, a reminder that the fleet tooling lags the newest service release.[8]

5Peering HBN with a Dell PowerSwitch running OS10

NVIDIA’s HBN pages do not name a validated Dell leaf, so treat this as a protocol-level pairing that you verify in the lab, not as a supported matrix entry. OS10 documents eBGP over unnumbered interfaces in its VXLAN/EVPN guide: the fabric interface gets no switchport, an MTU of 1650, and IPv6 router advertisements (ipv6 nd send-ra with min-ra-interval 3 / max-ra-interval 4); under router bgp the neighbor is declared with neighbor interface ethernet1/1/1 and inherits a template with inherit-type ebgp, send-community extended and address-family l2vpn evpn activated; the overlay uses evpn with auto-evi and disable-rt-asn, an nve whose source-interface is loopback0, and virtual-network <id> mapped to vxlan-vni <id>.[9] On the HBN side the matching lines are type unnumbered and remote-as external on p0_if, nv set evpn enable on, and a bridge domain br_default vlan <v> vni <vni> mapping.[3]

Worked → faded → problem: HBN on a B3220 peering with an OS10 leaf

Design. DPU p0 cabled to leaf ethernet1/1/1. HBN in AS 65501 with loopback 10.10.10.1/32 as VTEP source. Leaf in AS 201 with loopback0 192.168.1.1/32. One L2 VNI 2000 bridging host VLAN 200.

Step 1: HBN side (inside the HBN container). Reasoning: declare the uplink as a switch port, the host PF as an access port in the bridge domain, then BGP unnumbered and EVPN.[3]

nv set interface lo ip address 10.10.10.1/32
nv set interface p0_if,p1_if type swp
nv set interface pf0hpf_if bridge domain br_default access 200
nv set bridge domain br_default vlan 200 vni 2000
nv set vrf default router bgp router-id 10.10.10.1
nv set vrf default router bgp autonomous-system 65501
nv set vrf default router bgp enable on
nv set vrf default router bgp neighbor p0_if type unnumbered
nv set vrf default router bgp neighbor p0_if remote-as external
nv set evpn enable on
nv set nve vxlan enable on
nv set nve vxlan source address 10.10.10.1
nv config apply
nv config save

Step 2: OS10 leaf side. Reasoning: the unnumbered session needs router advertisements on the port; the template activates the EVPN address family; disable-rt-asn derives route targets without the ASN so two different ASNs can match on the VNI.[9]

interface loopback0
  no shutdown
  ip address 192.168.1.1/32
nve
  source-interface loopback0
interface ethernet1/1/1
  no shutdown
  no switchport
  mtu 1650
  ipv6 nd max-ra-interval 4
  ipv6 nd min-ra-interval 3
  ipv6 nd send-ra
virtual-network 2000
  vxlan-vni 2000
interface vlan200
  virtual-network 2000
router bgp 201
  router-id 192.168.1.1
  address-family ipv4 unicast
    redistribute connected
  template ebgp_unified
    send-community extended
    address-family l2vpn evpn
      activate
  neighbor interface ethernet1/1/1
    inherit template ebgp_unified inherit-type ebgp
    no shutdown
evpn
  auto-evi
  disable-rt-asn

Step 3: verify. On HBN: nv show vrf default router bgp for the session, then ip route show 192.168.1.1/32 and ip nexthop show to confirm the leaf loopback arrived and was programmed.[3] On OS10, the user guide’s show ip bgp neighbors topic is the counterpart (not re-verified on a fetched page for this lesson). Confirm MTU: the leaf runs 1650 on the fabric port; set the HBN side with hbn-dpu-setup.sh --mtu before the pod starts.[2]

Step 4: the open question. How HBN derives EVPN route targets is not shown on the fetched configuration page. If type-2 routes arrive but are not imported, route-target derivation across the two ASNs is the first suspect; fix it on the leaf (disable-rt-asn is already there) or add explicit RTs on HBN once you have confirmed the syntax in the 3.5.0 guide.[9]

End of Episode 1 — What stays on the whiteboard

How it ended

You draw one last line: the routed boundary moves into the server, and the leaf stays where it is, peering unnumbered with the DPU.[1] The SE writes three numbers on row 41 of his promise spreadsheet, because they decide the SKU: 3 to 6 GB for the pod, roughly the same again for OVS, and a qualified maximum of 16 VFs when other services share the card.[2] What you tell the network lead: HBN speaks standard BGP unnumbered and EVPN-VXLAN, OS10 documents both, and since nobody publishes a Dell-specific validation we prove the pairing in containerlab first.[9] Procurement, for once, is right about the lead time. Wave one racks next week — and then twenty-four cards come up green on a monitoring wall that shows nothing at all.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Pre-flight on the Dell-lab BlueField-3 (all read-only):

  1. Host: sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q INTERNAL_CPU_OFFLOAD_ENGINE must show ENABLED(0) (DPU mode); HBN cannot run in NIC mode.[2] DPU: cat /etc/mellanox/hbn.conf /etc/mellanox/sfc.conf 2>/dev/null shows whether SFC already exists; sudo crictl pods lists any running pods; free -g confirms the 32 GB SKU. Copy /etc/kubelet.d/ and /etc/mellanox/ to a backup directory before you change anything.

Mutating steps, each with its rollback:

  1. Enable SFC via bf.cfg (mutating: reinstalls the Arm OS). Write bf.cfg with ENABLE_BR_HBN=yes, BR_HBN_UPLINKS="p0,p1", BR_HBN_SFS='pf0dpu1,pf0dpu3', HUGEPAGE_COUNT=5120, then from the host sudo bfb-install -c bf.cfg -r rshim0 -b <3.5.0 BFB>.[2] Expected after reboot: /etc/mellanox/sfc.conf exists and ovs-vsctl show lists br-hbn.[1] Rollback: re-run bfb-install with your previous bf.cfg (without ENABLE_BR_HBN), or on the DPU /opt/mellanox/sfc-hbn/install.sh -c -r to regenerate the chain after editing the keys.[2]

  2. Deploy the pod (mutating). Fetch the doca_hbn NGC resource, run scripts/doca_hbn/<ver>/hbn-dpu-setup.sh --mtu 9000 if the leaf uses jumbo frames, then sudo cp configs/<doca_version>/doca_hbn.yaml /etc/kubelet.d/.[2] Expected within about 20 seconds: sudo crictl pods shows the HBN pod Ready; sudo crictl exec -it $(crictl ps | grep hbn | awk '{print $1;}') supervisorctl status frr shows RUNNING.[5][2] If not: sudo journalctl -u kubelet --since -5m and sudo crictl logs <id>; a restarting pod in 3.5.0 usually means an invalid startup configuration.[4] Rollback: sudo rm /etc/kubelet.d/doca_hbn.yaml.[5]

  3. Configure BGP unnumbered and EVPN (mutating inside the pod). Enter the container and apply the HBN block from the Worked example, then nv config apply and nv config save.[3] Rollback: nv unset each line you added (NVUE’s inverse verb) and re-apply, or remove the pod YAML from step 3, which discards the running configuration.

  4. Configure the OS10 leaf. On the PowerSwitch facing p0, apply the OS10 block from the Worked example (no switchport, mtu 1650, ipv6 nd send-ra, neighbor interface ethernet1/1/1 with inherit-type ebgp, evpn auto-evi disable-rt-asn).[9] Rollback: no router bgp 201 and no evpn, then restore the port’s previous switchport config from the running-config you saved.

  5. Verify. DPU: nv show vrf default router bgp shows the neighbor p0_if established; ip route show 192.168.1.1/32 and ip nexthop show show the leaf loopback and its programmed next hop.[3] Leaf: show ip bgp neighbors (OS10 user-guide topic; verify the exact output on your OS10 release). If the session is up but VNI 2000 carries no traffic, revisit route-target derivation as described in the Worked block.[9]

  6. Record and clean up. Save nv show interface and nv show vrf default router bgp output to the lab log, then decide whether to leave HBN running; if not, rollback steps 4, 3 and 2 in that order.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why a customer would run BGP/EVPN on the BlueField instead of on the leaf switch alone, and what that costs them.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs (HBN 3.5.0), 2026-09-06. Dates are when each page was fetched.

  1. DOCA HBN Service Guide (landing) · fetched 2026-09-06 · DOCA 3.5.0
  2. HBN Service Deployment · fetched 2026-09-06 · DOCA 3.5.0
  3. HBN Service Configuration · fetched 2026-09-06 · DOCA 3.5.0
  4. HBN Service Release Notes (HBN 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  5. DOCA Container Deployment Guide · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  7. DOCA Switching (e-switch, switchdev, representors) · fetched 2026-09-06 · DOCA 3.5.0
  8. doca-platform v26.4.1: dpuserviceconfiguration_hbn.yaml (HBN NVUE startup template) · fetched 2026-09-06
  9. Dell SmartFabric OS10 10.5.1 VXLAN/BGP EVPN guide: symmetric IRB with unnumbered BGP peering (example) · fetched 2026-09-06
  10. containerlab kind dell_ftosv (Dell FTOS10v / OS10) · fetched 2026-09-06
  11. DOCA General Support (default Arm OS for the BlueField bundle) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.