Skip to content
nodeza.learn
DOCA course/4. Data path/Lesson 4.4

DOCA Ethernet and OVS-DOCA

S4·E4Prove it is in hardware · Customer colo cage, Dallas, a 04:00 maintenance window

S4·E4Evaluate~45 minsources checked todaylab mutates hardwareverified against DOCA 3.5.0 docs + DPF v26.4.1 manifests, 2026-09-06

Builds on: E-switch, representors, and switchdev, The DOCA Flow model

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Describe the DOCA Ethernet RXQ/TXQ programming sequence and the 3.5.0 additions (shared mempool, CQE compression, inline data, header data-split).
  • Decide when the DOCA DPDK Bridge is the right integration path and what the 3.5.0 DPDK-dependency change means.
  • Enable OVS-DOCA with every other_config knob in the documented order and explain what each one controls.
  • Evaluate an OVS-DOCA deployment: which features are offloaded, how to size CT and megaflow limits, and how to verify offload with dpctl/offload-stats-show.
  • Read the DPF production OVS script and justify each deviation from the documentation's quick-start sequence.

Episode 4 — Prove it is in hardware

The situation · Customer colo cage, Dallas, a 04:00 maintenance window

The window opens at four and the network lead arrives with the printed competitor slide and the notebook, now thick enough to need a rubber band. He is polite about it: show me it is in hardware, on my bridge, tonight. The bridges are already up on the BlueField and traffic is flowing, but nobody standing in the cage can yet say whether a single packet is being switched by the card or by PMD threads on the Arm. The Dell SE, who wrote “line rate” on row 14, is standing very still.

OVS-DOCA exists for exactly that argument. It is the datapath offloading layer that uses the DOCA Flow library for offloading, GA in 3.5.0, and NVIDIA’s own guidance is to use it rather than OVS-Kernel or OVS-DPDK to maximize efficiency, performance, scalability and feature support.[4][8] It works because of the e-switch from Episode 1: in switchdev mode the physical port and all supported functions share a single general domain to execute offloaded flows.[7] Turning it on is other_config:doca-init=true plus hw-offload=true, a daemon restart, then bridges and ports of type=doca.[4] Proving it is ovs-appctl dpctl/offload-stats-show, plus a coverage/show counter that must not move.[4]

Offload is not a setting you trust, it is a counter you read. He wants the answer bottom-up, so you start one layer below OVS, at the queues the card hands an application.

1DOCA Ethernet: raw queues without DPDK

DOCA Ethernet (Beta in 3.5.0) gives an application direct receive and transmit queues on a device, using the DOCA Core objects from module 3.[9][1] Receive queues come in four types: DOCA_ETH_RXQ_TYPE_REGULAR (CPU, application-owned buffers), DOCA_ETH_RXQ_TYPE_CYCLIC (GPU only), DOCA_ETH_RXQ_TYPE_MANAGED_MEMPOOL (CPU, library-managed), and new in 3.5.0 DOCA_ETH_RXQ_TYPE_SHARED_MEMPOOL for several queues drawing from one pool.[1]

The RXQ sequence is: doca_eth_rxq_create(dev, max_burst_size, max_packet_size, &rxq), doca_eth_rxq_set_type, buffer sizing with doca_eth_rxq_estimate_packet_buf_size() and doca_eth_rxq_set_pkt_buf() (or doca_eth_rxq_shared_mempool_create() plus doca_eth_rxq_set_shared_mempool()), optional headroom and tailroom setters, then task or event registration (doca_eth_rxq_task_recv_set_conf(rxq, cb, err_cb, n), doca_eth_rxq_event_managed_recv_register, doca_eth_rxq_event_batch_managed_recv_register, doca_eth_rxq_event_batch_shared_mempool_recv_register), and finally ctx = doca_eth_rxq_as_doca_ctx(rxq); doca_pe_connect_ctx(pe, ctx); doca_ctx_start(ctx); doca_eth_rxq_apply_queue_id(rxq, queue_id);.[1] Transmit mirrors it: doca_eth_txq_create(dev, max_burst_size, &txq), doca_eth_txq_set_type(txq, DOCA_ETH_TXQ_TYPE_REGULAR), doca_eth_txq_task_send_set_conf, checksum offloads via doca_eth_txq_set_l3_chksum_offload() and _l4_chksum_offload(), LSO via doca_eth_txq_set_mss(), doca_eth_txq_set_max_lso_header_size() and doca_eth_txq_task_lso_send_set_conf(), then doca_eth_txq_apply_queue_id().[1]

Three 3.5.0 additions matter for performance conversations. Inline data: doca_eth_txq_set_max_inline_data_size() with doca_eth_txq_cap_get_max_inline_data_size() lets small payloads ride inside the WQE.[1][2] CQE compression: doca_eth_rxq_set_cqe_format() guarded by doca_eth_rxq_cap_is_cqe_format_supported().[1] Header data-split is announced in the release changes; the exact setter name is not on the 3.5.0 guide page fetched for this lesson, so check the header before quoting it.[2]

The rule that trips first-time users: DOCA Flow is required to ensure packets are steered correctly to the ETH queues, so you create pipes whose fwd targets the queue id you applied; a TXQ needs no pipes.[1] Timestamps need mlxconfig -d <device_id> s REAL_TIME_CLOCK_ENABLE=1 and root.[1] For GPU pipelines the control path always runs on the CPU while the data path can run on the GPU after doca_ctx_set_datapath_on_gpu(), with handles from doca_eth_rxq_get_gpu_handle() and doca_eth_txq_get_gpu_handle().[1] The CPU samples at tag 3.5.0 are eth_rxq_regular_receive, eth_rxq_managed_mempool_receive, eth_rxq_batch_managed_mempool_receive, eth_rxq_batch_shared_mempool_receive, eth_txq_send_ethernet_frames, eth_txq_lso_send_ethernet_frames, eth_txq_batch_send_ethernet_frames, eth_txq_batch_lso_send_ethernet_frames.[10][12]

2DPDK Bridge: for code that already exists

The DOCA DPDK Bridge exists to add DOCA to an application that already runs on DPDK. doca_dpdk_port_probe and doca_dpdk_port_as_dev bind a DPDK port id to a doca_dev, the library converts between rte_mbuf and doca_buf, doca_dpdk_cap_is_rep_port_supported reports representor support, and the pkg-config module is doca-dpdk-bridge.[3] NVIDIA’s own guidance is that the bridge exists for the interop case, not the start-fresh case; new applications should use DOCA Ethernet.[3]

You have already seen the bridge in use: init_doca_flow_vnf_ports in flow_common.c calls doca_dpdk_port_as_dev(i, &dev_arr[i]) for each DPDK port before starting DOCA Flow ports, and the file compiles those paths only when FLOW_NO_DPDK is not defined (#ifndef FLOW_NO_DPDK #include <doca_dpdk.h> #include <dpdk_utils.h>).[11] That conditional is the visible edge of a 3.5.0 change: DOCA Bridge removes the DOCA dependency on DPDK for customers who do not need DPDK.[2] The same release enables DOCA Ethernet for the OVS-DOCA datapath and adds an API to configure floating-CPU behaviour for PMD threads, which is how OVS itself is moving off a hard DPDK requirement.[2]

3OVS-DOCA: what it is and how you turn it on

OVS-DOCA is the datapath offloading layer (DPIF) that uses the DOCA Flow library for offloading; NVIDIA advises using it, rather than OVS-Kernel or OVS-DPDK, to maximize efficiency, performance, scalability and feature support, and it is GA in 3.5.0.[5][8] Its design rests on the e-switch dependency from lesson 4.1: in switchdev mode the physical port and all supported functions share a single general domain, the e-switch, to execute offloaded flows.[4]

Host (x86)BlueField-3 Arm + e-switch (DPU mode)match · CT · metersoffloaded flowsmiss → upcallencap · mirrorPF (host)lspci -d 15b3:VF0 … VFnSRIOV_EN=1pf0hpfrepresentorpf0vf0 … nrepresentorsbr-ovsdatapath_type=docae-switch HW tablesDOCA Flow · hwsArm: ovs-vswitchd PMDslow pathp0uplinkp1uplinkLeaf switchToR

OVS-DOCA: the e-switch programmed through DOCA Flow

OVS-DOCA is the OVS datapath interface (DPIF) that offloads flows with the DOCA Flow library instead of TC (OVS-Kernel) or rte_flow (OVS-DPDK). Host PFs/VFs show up on the Arm as representors; a bridge with datapath_type=doca joins them to the uplinks p0/p1.

Why it matters. In DPU mode "BlueField owns NIC resources, and all host traffic flows through a virtual switch on the Arm subsystem before reaching the host". The bridge is the policy point; offloaded flows never touch an Arm core.

FAE note. GA in DOCA 3.5.0. New in 3.5.0: IPv6 connection tracking and DOCA Ethernet for the OVS-DOCA datapath. BlueField-3 hosts must run DOCA-Host, not MLNX_OFED. Toggle offload and click hops to see what lands in hardware.

ovs-vsctl --no-wait set Open_vSwitch . other_config:doca-init=true
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
systemctl restart openvswitch-switch
ovs-vsctl add-br br0 -- set bridge br0 datapath_type=doca
ovs-vsctl add-port br0 pf0hpf -- set Interface pf0hpf type=doca

Source: networking-docs.nvidia.com

An OVS-DOCA bridge on the Arm with p0, pf0hpf and a VXLAN port. Toggle offload to see which path a packet takes.

The documented enable sequence on a host-owned card is: unbind the VFs, echo switchdev > /sys/class/net/enp4s0f0/compat/devlink/mode (on a DPU the Arm PFs are already in switchdev), rebind the VFs (echo 0000:04:00.2 > /sys/bus/pci/drivers/mlx5_core/bind), configure hugepages (mkdir -p /hugepages; mount -t hugetlbfs hugetlbfs /hugepages; echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages), systemctl start openvswitch, then:[4][7]

ovs-vsctl --no-wait set Open_vSwitch . other_config:doca-init=true
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
systemctl restart openvswitch-switch      # Debian; RPM systems: systemctl restart openvswitch
ovs-vsctl --no-wait add-br br0-ovs -- set bridge br0-ovs datapath_type=doca
ovs-vsctl add-port br0-ovs enp4s0f0   -- set Interface enp4s0f0   type=doca   # PF
ovs-vsctl add-port br0-ovs enp4s0f0_0 -- set Interface enp4s0f0_0 type=doca   # representor

other_config:doca-init enables DOCA Flow hardware offload, defaults to false, is only relevant for the userspace datapath and requires a daemon restart; hw-offload is the generic OVS offload switch.[4] Setting other_config:default-datapath-type=doca lets later add-br and add-port commands omit the type; an unsupported type falls back to system.[4] Optional per-port settings: ovs-vsctl set interface enp4s0f0 mtu_request=9000 (representors inherit from the e-switch manager) and options:dpdk-vf-mac=00:11:22:33:44:55 to set a VF/SF MAC, which needs the function unbound and rebound.[4]

Production looks different. The DPF DPUFlavor rawConfigScript deletes the default bridges, sets doca-init=true, dpdk-max-memzones=50000, hw-offload=true, pmd-quiet-idle=true, max-idle=20000, max-revalidator=5000, doca-congestion-threshold=60, flow-limit=500000 and hw-offload-ct-unidir-udp-enabled=true, removes any default-datapath-type, restarts openvswitch-switch (or openvswitch), and then creates br-sfc, br-hbn, br-dpu and br-ovn with datapath_type=netdev, adds p0 and pf0hpf as type=dpdk with mtu_request=9216, and patches bridges together; /etc/mellanox/mlnx-ovs.conf carries OVS_DOCA="yes" and CREATE_OVS_BRIDGES="no".[6] Read the two together: doca-init=true is what selects the DOCA offload provider, whether the bridge is declared doca or netdev.[4][6]

4What gets offloaded, and how to size it

The OVS-DOCA page lists the offloaded feature set and the knobs that bound it.[4]

  • Connection tracking: other_config:hw-offload-ct-size (default 250000; maximum 2M; do not exceed 1M for best performance; 0 disables CT offload; restart required); hw-offload-ct-ipv6-enabled (default false; IPv6 CT is new in 3.5.0); hw-offload-ct-unidir-udp-enabled (default false); limitation: ct-ct-nat is unsupported.[4][2]
  • Tunnels: VXLAN (options:local_ip, remote_ip, key, dst_port; GBP via options:exts=gbp, all VXLAN ports must agree), Geneve (only one option, maximum length 7; changing the matched option needs port removal or restart), GRE.[4]
  • Meters: OpenFlow meters including chained meters (ovs-ofctl add-flow … meter=<id>); slow-path options:sw-meter=pps:10k:6k and sw-meter-dry-run=true; Guaranteed Packet Rate via other_config:gpr-mode=<disabled|rep-only|all-ports> with per-core-meter-rate and per-port-meter-rate.[4]
  • Mirror (ovs-vsctl create mirror; 128 mirror configurations by default, raised with other_config:doca-mirror-max, whose example value is 2048), sFlow (partial offload; on a physical-plus-internal topology only on the internal bridge with VXLAN), hairpin (actions=in_port), DP-HASH select groups (IP only).[4]
  • Pre-miss rules: options:doca-pre-miss-rules on a PF punts listed EtherTypes straight to the kernel, bypassing the software datapath; defaults are LACP, LLDP and 802.1X; " " clears them and ovs-vsctl remove interface <pf> options doca-pre-miss-rules restores defaults; maximum 16 EtherTypes.[4]
  • SR-IOV VF LAG (active-backup, XOR, LACP) with options:dpdk-lsc-interrupt=true on the bond, and multiport e-switch via devlink dev param set pci/<bdf> name esw_multiport value 1 cmode runtime on both PFs.[4]

Sizing knobs: other_config:doca-congestion-threshold (30 to 90, default 80) is the occupancy percentage that triggers a resize of DOCA offload structures; ctl-pipe-size and ctl-pipe-infra-size preset control-pipe sizes (root, post-hash, post-CT, post-meter, split, miss); dpdk-max-memzones must grow with function count (two PFs plus 127 VFs each require 6500); flow-limit (megaflow cache, default 200000) and max-revalidator (default 250 ms) bound the software cache; pmd-cpu-mask=0x6 pins PMD threads to cores 1 and 2; pmd-quiet-idle and pmd-sleep-max let PMDs rest.[4] The DPF script’s choices (doca-congestion-threshold=60, flow-limit=500000, max-revalidator=5000, dpdk-max-memzones=50000) are exactly these knobs tuned for a Kubernetes DPU with many functions.[6]

5Verifying offload and bringing up VXLAN

The DOCA-native check is ovs-appctl dpctl/offload-stats-show, which the page presents under connection tracking as the way to view offload statistics.[4] Second is ovs-appctl coverage/show, whose non-zero counters include doca_async_queue_full, doca_async_queue_blocked, doca_async_add_failed, doca_pipe_resize and doca_pipe_resize_over_10_ms; a rising doca_async_add_failed means rules are not reaching hardware.[4] Third, ovs-appctl vlog/set <topics>:file:dbg raises logging per topic, and ovs-doca-tcpdump captures the software datapath (one instance at a time), with hooks such as -i pf0vf0:rx_pre_restore,rx,tx+pf0vf1:tx, -w capture.pcap, --list-interfaces and --list-hooks.[4]

Two commands you may know from other OVS datapaths are not on the 3.5.0 OVS-DOCA page: ovs-appctl dpctl/dump-flows type=offloaded is the OVS-Kernel and OVS-DPDK idiom and is not documented for OVS-DOCA, so do not use it as your evidence of offload; the generic ovs-appctl dpctl/dump-flows -m (verbose flow dump) is standard OVS tooling and will list datapath flows, but the page does not describe its OVS-DOCA fields, so treat its per-flow annotations as a secondary signal and offload-stats-show as the primary one.[4]

Worked: an OVS-DOCA bridge with a VXLAN port

The page’s VXLAN recipe uses two bridges: a physical bridge holding the uplink, and an integration bridge holding the representor and the tunnel port; the underlay IP lives on the uplink side.[4]

# 1. offload provider on, restart (once per host)
ovs-vsctl --no-wait set Open_vSwitch . other_config:doca-init=true
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
systemctl restart openvswitch-switch

# 2. physical bridge with the PF
ovs-vsctl add-br br-phy -- set Bridge br-phy datapath_type=doca \
  -- br-set-external-id br-phy bridge-id br-phy -- set bridge br-phy fail-mode=standalone
ovs-vsctl add-port br-phy enp4s0f0 -- set Interface enp4s0f0 type=doca
# underlay address on br-phy (56.56.67.1/24 in the example) and bring it up

# 3. integration bridge with the representor and the tunnel
ovs-vsctl add-br br-ovs -- set Bridge br-ovs datapath_type=doca \
  -- br-set-external-id br-ovs bridge-id br-ovs -- set bridge br-ovs fail-mode=standalone
ovs-vsctl add-port br-ovs enp4s0f0_0 -- set Interface enp4s0f0_0 type=doca
ovs-vsctl add-port br-ovs vxlan0 -- set interface vxlan0 type=vxlan \
  options:local_ip=56.56.67.1 options:remote_ip=56.56.68.1 options:key=45 options:dst_port=4789

# 4. verify
ovs-appctl dpctl/offload-stats-show
ovs-appctl coverage/show | grep doca_

Every ovs-vsctl line is the page’s own; local_ip, remote_ip, key and dst_port are the four VXLAN options it shows.[4] On a BlueField in DPU mode substitute p0 for enp4s0f0 and pf0hpf or pf0vf0 for the representor; the Arm PFs are already in switchdev.[7] Expected after traffic: offloaded flow and packet counters in offload-stats-show rise; doca_async_add_failed stays at zero.[4]

The three numbers

How it ended

Before and after a traffic run you show him dpctl/offload-stats-show, then coverage/show | grep doca_ with doca_async_add_failed still at zero, then top on the Arm where the PMD does not scale with throughput.[4] You also volunteer the caveat his competitor’s slide does not mention: connection tracking with NAT is not offloaded in 3.5.0.[4] Three numbers and one limitation go into the notebook, and it is the limitation that makes him believe the numbers. Then he turns the page to what he has been saving: between two encapsulations their platform inserts a four-byte transport tag of its own design, and nothing in this cage knows it exists.

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Pre-flight (read-only) on the Arm: ovs-vsctl show and ovs-vsctl get Open_vSwitch . other_config (record the current bridges and knobs, they are your rollback state); ip -br link | grep -E '^(p0|pf0hpf|pf0vf0)'; cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages; on a host-owned card only, devlink dev eswitch show pci/<bdf>.

  1. Mutating (host-owned card only; skip on a DPU where the Arm PFs are already in switchdev): power off VMs, unbind VFs, devlink dev eswitch set pci/<bdf> mode switchdev, rebind VFs. Expected: representors appear. Rollback: devlink dev eswitch set pci/<bdf> mode legacy after unbinding again.[7]

  2. Mutating: ovs-vsctl --no-wait set Open_vSwitch . other_config:doca-init=true && ovs-vsctl set Open_vSwitch . other_config:hw-offload=true && systemctl restart openvswitch-switch. Expected: ovs-vsctl get Open_vSwitch . other_config shows both keys and the daemon is running. Rollback: ovs-vsctl remove Open_vSwitch . other_config doca-init, same for hw-offload, restart.[4]

  3. Mutating: ovs-vsctl add-br br-phy -- set Bridge br-phy datapath_type=doca -- br-set-external-id br-phy bridge-id br-phy -- set bridge br-phy fail-mode=standalone and ovs-vsctl add-port br-phy p0 -- set Interface p0 type=doca; assign the underlay address to br-phy. Expected: ovs-vsctl show lists br-phy with p0. Rollback: ovs-vsctl --if-exists del-br br-phy.[4]

  4. Mutating: ovs-vsctl add-br br-ovs -- set Bridge br-ovs datapath_type=doca -- br-set-external-id br-ovs bridge-id br-ovs -- set bridge br-ovs fail-mode=standalone, ovs-vsctl add-port br-ovs pf0vf0 -- set Interface pf0vf0 type=doca, ovs-vsctl add-port br-ovs vxlan0 -- set interface vxlan0 type=vxlan options:local_ip=<local> options:remote_ip=<peer> options:key=45 options:dst_port=4789. Expected: ovs-vsctl show lists both ports. Rollback: ovs-vsctl --if-exists del-port br-ovs vxlan0 then del-br br-ovs.[4]

  5. Verify: generate traffic from the VF toward the peer, then ovs-appctl dpctl/offload-stats-show and ovs-appctl coverage/show | grep doca_. Expected: offloaded counters increase; doca_async_add_failed does not. If offload counters stay at zero: check step 2 took effect (other_config and restart), and that p0’s MTU allows the encapsulated frame (mtu_request=9216 in production).[4][6]

  6. Rollback in reverse order (steps 4, 3, 2, then 1 if applicable) and confirm ovs-vsctl show matches your pre-flight record.

Retrieval check

10 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell SE, in five sentences, why NVIDIA tells customers to run OVS-DOCA instead of kernel OVS on a BlueField-3, and how you would prove the offload is working.

12 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs + DPF v26.4.1 manifests, 2026-09-06. Dates are when each page was fetched.

  1. DOCA Ethernet (programming guide) · fetched 2026-09-06 · DOCA 3.5.0
  2. Changes and New Features (DOCA 3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  3. NVIDIA/skills: doca-dpdk-bridge SKILL.md · fetched 2026-09-06
  4. OVS-DOCA Hardware Acceleration · fetched 2026-09-06 · DOCA 3.5.0
  5. OpenvSwitch Acceleration - OVS in DOCA (landing) · fetched 2026-09-06 · DOCA 3.5.0
  6. DPF v26.4.1 DPUFlavor (OVS rawConfigScript) · fetched 2026-09-06
  7. DOCA Switching (e-switch, switchdev, representors) · fetched 2026-09-06 · DOCA 3.5.0
  8. DOCA Services (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  9. DOCA Libraries (index with quality levels) · fetched 2026-09-06 · DOCA 3.5.0
  10. doca-samples 3.5.0: samples/doca_eth/README.md · fetched 2026-09-06 · DOCA 3.5.0
  11. doca-samples 3.5.0: flow_common.c (FLOW_NO_DPDK) · fetched 2026-09-06 · DOCA 3.5.0
  12. doca-samples 3.5.0 full git tree (doca_eth sample inventory) · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.