Skip to content
nodeza.learn
DOCA course/4. Data path/Lesson 4.1

E-switch, representors, and switchdev

S4·E1The VF that OVS cannot see · Dell PowerEdge lab, Round Rock, day two of the clean-pipes PoC

S4·E1Understand~30 minsources checked todayverified against DOCA 3.5.0 docs, 2026-09-06

Builds on: Modes of operation: DPU, NIC, Zero-Trust, Switching modes safely

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Explain who owns the embedded switch in DPU mode versus NIC mode and what that means for the host.
  • Name every representor on a BlueField-3 (p0/p1, pf0hpf, pf0vfN, SF forms) and map one to its function with ip -d link.
  • Describe the switchdev enable and revert sequence on a host-owned card and the preconditions it has.
  • Compare SR-IOV VFs and Scalable Functions and choose one for a given Arm-side or host-side need.

Episode 1 — The VF that OVS cannot see

The situation · Dell PowerEdge lab, Round Rock, day two of the clean-pipes PoC

It is 09:40 on day two and the customer’s network lead has the green notebook open: no counter, no belief. Their tenant VM is up with an SR-IOV VF attached, and the OVS bridge on the BlueField-3 shows nothing for it. The Dell SE, holding coffee he stopped drinking an hour ago, offers to put the host PF into switchdev with devlink — row 4 of his promise spreadsheet says tenant traffic on the card by Wednesday. You stop him: on a BlueField in DPU mode switchdev is configured by default for all PFs on the Arm and that configuration must not be changed, and the host PFs operate in legacy mode only.[1] The ports he wants already exist, on the other side of the card.

That is why the embedded switch exists. Virtual switching used to cost host cores and host trust; here BlueField owns the NIC resources, and all host traffic flows through a virtual switch on the Arm subsystem before reaching the host, with host drivers loading only after BlueField finishes NIC configuration.[2] So every host function gets a mirror port on the Arm — p0 for the wire, pf0hpf for the host PF, pf0vf0 for its first VF — and those representors are steering handles, not endpoints; they cannot host an IP server and cannot be pinged.[2][1] When the Arm needs an address of its own, you create a Scalable Function, which needs no SR-IOV.[3]

A missing representor means a missing function. Before either of you types anything, go and look at who owns the switch.

1One embedded switch owns every port

In DPU mode the card, not the host, is the network device. The DPL system guide states it plainly: BlueField owns NIC resources, and all host traffic flows through a virtual switch on the Arm subsystem before reaching the host; host-side NIC drivers load only after BlueField completes NIC configuration, and BlueField allocates all NIC memory (ICM) and configures the embedded switch.[2]

That embedded switch is the e-switch. Once a PF is in switchdev mode, the PF acts as an e-switch and manages all switch ports; it controls VF and SF traffic, and software drives it through the netdev and RDMA core interfaces.[1] On a BlueField in DPU mode, switchdev mode is configured by default for all PFs on the Arm and that configuration must not be changed; the host cannot put its PFs into switchdev and they operate in legacy mode only.[1]

Two limits follow. The e-switch netdev (for example pf0) looks like an ordinary port, but it is the upstream representor: it cannot host an IP server, so configuring an address on it is not useful and it cannot be pinged.[1] Its RDMA instance (for example mlx5_0) cannot host an RDMA connection and cannot have a GID; it exists to configure the e-switch, and the documented way around both limits is to create SFs.[1]

OVS-DOCA depends on the same fact: in switchdev mode the physical port and all supported functions share a single general domain, the e-switch, to execute offloaded flows.[7] Every later lesson in this module, DOCA Flow pipes, OVS-DOCA bridges and DPL programs, is a different way of programming that one switch.

Arm active — embedded Arm owns the NIC (ECPF)
Host (x86 / Arm server) — mlx5 netdevsArm subsystem — representors + bridgeUplinks (Arm netdevs p0 / p1)PF netdevhost pf0 (mlx5)VF netdevsSR-IOV pf0vf0…SF netdevsen3f0pf0sf4 …PCIe Gen5 x16e-switch (embedded switch, switchdev)inside the NIC ASIC — flow tables programmed by the e-switch managerp0uplink 0p1uplink 1Wire — QSFP112 400G / 200Gpf0hpfpf0vf0pf0sf0OVS-DOCA bridge ovsbr1 (p0 + pf0hpf)or HBN br-hbnArm appsovs-vswitchd · DOCA Flow · tcpdump -i pf0hpf
hop 1 / 6
DPU mode · Host → wire

1. Host app transmits on the PF netdev

The host PF is an ordinary mlx5 netdev (VFs and SFs likewise). In DPU mode the host does not own the e-switch — the embedded Arm system does.

Mode configuration
sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE   # ENABLED(0) = DPU mode
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

NIC → DPU on BlueField-3 (host side). BlueField-2 additionally needs INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0.

  • "For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended."
  • Default for DPU SKUs B3240 / B3220 / B3210 / B3210E.
  • Socket Direct is not supported in DPU mode.
BlueField Modes of Operation · Virtual Switch on BlueField · OVS-DOCA
Step a packet from a host VF through pf0vf0, the e-switch, and out of p0. Notice that the Arm kernel never touches an offloaded packet.

2Names: p0, pf0hpf, pf0vfN and the SF forms

The naming is regular once you see the pattern. The DPL system guide gives the table: uplink representor p<port_number>, PF representor pf<port_number>hpf, VF representor pf<port_number>vf<function_number>, SF representor en<domain>f<function>pf<port_number>sf<function_number>.[2] It adds the orientation: pf0hpf and pf1hpf are the representors facing the x86 host, and pf0 and pf1 are the representors facing the network.[2] The switching guide uses the same pf0 name for the e-switch netdev itself, so read p0 and pf0 as the wire side and pf0hpf as the host side.[1] The SF guide’s own example, pfnum 0 sfnum 4, prints the netdev en3f0pf0sf4; the leading en<domain>f<function> encodes the PCI domain and function, and that is the only SF representor form the 3.5.0 guides print.[3]

To map a representor to the function behind it, run ip -d link show <netdev> and read two fields: switchid, which maps the representor to the device (both PFs of one card report the same switchid), and portname, which is pf<X>vf<Y> where X is the PF number and Y the VF number.[1] The guide’s example line ends portname pf0vf0 switchid f4ab580003a1420c.[1]

Each VF or SF gets a representor port. The function itself is what a VM or container uses to reach the network; the representor is what virtual-switch software (on a bare-metal host, or on the Arm) uses to steer that function’s traffic.[1] The representor has a netdev usable with Linux tools and OVS, but no RDMA instance of its own; it is an RDMA port of the e-switch manager’s device.[1]

One sizing rule surprises people: the host function’s MTU must be smaller than that of both its uplink and its representor.[2] The DPF production flavor sets p0 and pf0hpf to mtu_request=9216 for exactly that reason, so the host can run large frames underneath them.[5]

Netdev on the Arm What it represents
p0, p1 uplink (wire) ports
pf0hpf, pf1hpf the host’s PF0 and PF1
pf0vf0 … pf0vfN host VFs of PF0
en3f0pf0sf4 SF number 4 on PF0 (en<domain>f<function>pf<port>sf<n>)

3switchdev when the host owns the card

Which side runs switchdev depends on the operating mode. In NIC mode the Arm cores are inactive and the device functions as a ConnectX network adapter; in DPU mode the Arm is active and runs the services that manage NIC resources and the data path; the zero-trust (restricted) variant is DPU mode with host privileges removed.[4] BlueField-3 moves to NIC mode with mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1 and back with =0; the change needs an Arm and NIC reset (a power cycle is recommended), and NIC mode combined with the zero-trust restrictions is not supported.[4]

On a ConnectX, or on a BlueField in NIC mode, switchdev is a host decision and the switching guide gives the sequence.[1] First unbind all VFs, for example echo 0000:3d:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind; VMs with attached VFs must be powered off to allow this.[1] Then run devlink dev eswitch set pci/0000:3d:00.0 mode switchdev, which creates the VF/SF representor ports in the host OS; mode legacy removes them again.[1] Kernels without devlink use echo switchdev > /sys/class/net/pf0/compat/devlink/mode.[1] The OVS-DOCA guide adds the step people forget: after the mode change, rebind the VFs (echo 0000:04:00.2 > /sys/bus/pci/drivers/mlx5_core/bind) before attaching representors to a bridge.[7]

On the Dell host one constraint frames all of this: BlueField-3 devices are not supported with MLNX_OFED as the host driver; they must use DOCA-Host.[8]

4VF or SF: two ways to slice a function

A VF is an SR-IOV PCIe function; an SF is not. The SF guide states that SFs do not require PCIe SR-IOV to be enabled and share PCIe-level resources with the parent PF.[3] On the Arm you create one with /opt/mellanox/iproute2/sbin/mlxdevm port add pci/0000:03:00.0 flavour pcisf pfnum 0 sfnum 4, which returns an SF port index (the guide’s example is 229409); then configure and activate it with mlxdevm port function set pci/<pci_address>/<sf_port_index> hw_addr <mac_address> trust on state active.[3] The SF is then bound to a driver over the auxiliary bus: echo mlx5_core.sf.<serial> > /sys/bus/auxiliary/drivers/mlx5_core.sf_cfg/unbind, followed by the same string written to /sys/bus/auxiliary/drivers/mlx5_core.sf/bind.[3] SF numbers 1000 and later are reserved for the virtio-net controller, and upstream kernels without mlxdevm use devlink for the same steps.[3]

Firmware decides how many of each can exist. The DPF production flavor sets PER_PF_NUM_SF=1, PF_TOTAL_SF=20, PF_SF_BAR_SIZE=10, SRIOV_EN=1, NUM_OF_VFS=46, LAG_RESOURCE_ALLOCATION=1, INTERNAL_CPU_MODEL=1 and INTERNAL_CPU_OFFLOAD_ENGINE=0, and turns on ENABLE_ESWITCH_MULTIPORT="yes" in /etc/mellanox/mlnx-bf.conf.[5]

The trade-off for an FAE: VFs are what a host hypervisor or bare-metal tenant consumes through SR-IOV, so their count is a host-facing capacity decision; SFs are the host-invisible functions the Arm side uses for its own endpoints, which is why the switching guide points to SFs whenever the e-switch netdev or RDMA device cannot terminate traffic.[1][3] DOCA applications on the Arm follow the same pattern: the Simple Forward VNF is launched as ./doca_simple_fwd_vnf -- -a aux/4 -a aux/5 -l 60, that is, on two SFs, and its guide requires encap mode to be set to none on the PF FDB before the SFs are created.[6] In DOCA Flow code a representor is attached to a port with doca_flow_port_cfg_set_dev_rep, while a plain device uses doca_flow_port_cfg_set_dev.[9]

VF SF
Needs SR-IOV yes (SRIOV_EN=1, NUM_OF_VFS) no
Typical consumer host OS, VM, tenant Arm-side apps, OVS, DOCA samples
Created by firmware SR-IOV settings on the PF mlxdevm port add … flavour pcisf
Representor pf0vfN en<domain>f<function>pf0sf<N>
Reserved numbers none documented sfnum 1000+ for virtio-net

What the switchid said

How it ended

On the Arm, ip -br link lists p0 p1 pf0hpf pf1hpf and nothing else: no pf0vf0, because the VF was never bound on the host. One bind later the representor appears, ip -d link show pf0vf0 prints portname pf0vf0 with the same switchid as pf1hpf, and the bridge finally sees the VM.[1] The night-shift operator squeezes his label maker and sticks a strip under the port: NOT A PING TARGET. The network lead writes the counter down, then asks the question that owns tomorrow: their security team’s own Arm-side application has to run a deny-list, a route lookup and an inspection hand-off in one pass — and what becomes of a packet that matches none of them?

Lab

Pre-flight (nothing below writes): on the Arm, sudo mst start && sudo mst status -v and record the MST device name.

  1. ip -br link | grep -E '^(p[01]|pf[01]hpf|pf0vf|pf0sf|en[0-9]+f)'. Expected: p0 p1 pf0hpf pf1hpf plus one pf0vfN per bound host VF.[2] If pf0hpf is missing, the card may be in NIC mode; check step 3 before anything else.

  2. ip -d link show pf0vf0 | grep -o 'portname [^ ]* switchid [^ ]*'. Expected: portname pf0vf0 switchid <16 hex digits>. Repeat for pf1hpf and confirm the switchid is identical.[1] If there is no pf0vf0: the host has not created or bound any VF yet.

  3. sudo mlxconfig -d /dev/mst/<dev> q INTERNAL_CPU_OFFLOAD_ENGINE SRIOV_EN NUM_OF_VFS PER_PF_NUM_SF PF_TOTAL_SF PF_SF_BAR_SIZE. Record the values. Expected on a DPU-mode card: INTERNAL_CPU_OFFLOAD_ENGINE 0.[4][5]

  4. ls /sys/bus/auxiliary/devices/ | grep mlx5_core.sf. Expected: zero or more mlx5_core.sf.<serial> entries, one per existing SF.[3] If empty, no SFs exist yet; lesson 4.2’s hardware lab creates them.

  5. Stop here. Do not run devlink dev eswitch set on the Arm PFs: the guide says that configuration must not be changed in DPU mode.[1]

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in four sentences, why a VM's VF shows up on the DPU as pf0vf0 and why that interface cannot be pinged.

11 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs, 2026-09-06. Dates are when each page was fetched.

  1. DOCA Switching (e-switch, switchdev, representors) · fetched 2026-09-06 · DOCA 3.5.0
  2. DPL System (representor naming, DPU-mode ownership) · fetched 2026-09-06 · DOCA 3.5.0
  3. BlueField Scalable Functions · fetched 2026-09-06 · DOCA 3.5.0
  4. BlueField Modes of Operation · fetched 2026-09-06 · DOCA 3.5.0
  5. DPF v26.4.1 DPUFlavor (nvconfig and OVS rawConfigScript) · fetched 2026-09-06
  6. DOCA Simple Forward VNF Application Guide · fetched 2026-09-06 · DOCA 3.5.0
  7. OVS-DOCA Hardware Acceleration · fetched 2026-09-06 · DOCA 3.5.0
  8. DOCA Release Notes 3.5.0 · fetched 2026-09-06 · DOCA 3.5.0
  9. doca-samples 3.5.0: samples/doca_flow/flow_common.c · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.