NVIDIA NIC and DPU: SRH push with DOCA Flow

1A plugin, not an action: how SRv6 is wired into DOCA Flow

DOCA Flow’s capability list includes matching on “SRv6 (segments_left field)” next to the usual MAC, VLAN, IP, TCP, UDP, GRE, VXLAN, GTP-U, ESP and PSP items, and its action list includes “SRv6 actions: Insert SRv6 Segment Routing Header (SRH); Decrement SRv6 segments left; Copy active SID to IPv6 destination based on segments left field”.^[1] But the section that governs how you use it says something more specific: “DOCA Flow provides hardware-accelerated SRv6 support for IPv6 Segment Routing. SRv6 is implemented as an external action; that is, a plugin that extends the standard DOCA Flow action set.”^[1]

That framing has one enormous consequence. The plugin must be registered before the library starts:

doca_flow_external_action_srv6_register();
/* ... then, and only then ... */
doca_flow_init(&cfg);

NVIDIA states the prerequisite and the failure mode together: “SRv6 is supported in both vnf and switch modes with hardware steering (HWS) enabled. The SRv6 external action must be registered before calling doca_flow_init()” and “Calling this function after doca_flow_init() results in undefined behavior.”^[1] Undefined behavior, not an error return. An application that gets the order wrong may work in a unit test, fail on a different card, or corrupt an unrelated pipe — and there is no log line telling you why.

The second setup cost is memory, and it is easy to under-budget: “Each distinct SRH configuration consumes DOCA_FLOW_SRV6_ACTION_MEM_SIZE (128) bytes. This must be added to the total actions memory size set via doca_flow_port_cfg_set_actions_mem_size()”, with the documented pattern uint32_t mem_size = ACTIONS_MEM_SIZE(1024) + SRV6_ACTIONS_MEM_SIZE(16);.^[1] Read “each distinct SRH configuration” carefully: it is per unique header content, so a policy set with sixteen different segment lists costs sixteen allocations, not one.

Finally, the match and action fields must be registered through the Definitions API before init as well, with the extended structs putting base first: doca_flow_definitions_add_field(definitions, "match.packet.outer.srv6.segments_left", …) and the matching actions.packet.… string, where “The base member must be the first field in the extended structure.”^[1]

2The four capabilities, and what they add up to

DOCA Flow documents four SRv6 capabilities, verbatim: insert an SRH after the IPv6 base header with a user-provided segment list; match on the segments_left of an existing SRH; decrement segments_left; and “Copy the active SID (determined by the current segments_left value) to the IPv6 destination address.”^[1]

The push semantics are stated precisely: the hardware will “Insert a user-inputted SRH (header fields that are not set by HW must contain valid values). Set the IPv6 next_header field to 43. Set the SRH next_header field to the original IPv6 packet’s next_header value.”^[1] That is the RFC 8754 header chain built for you: Routing Type 4 announced by next_header 43 on the IPv6 header the packet already had, with the original upper-layer protocol preserved in the SRH’s own next_header field.^[6]

The decrement is an arithmetic trick rather than a dedicated action: “Use the ADD action with a per-entry value of UINT8_MAX (which is the equivalent of subtracting 1 in unsigned 8-bit arithmetic)”, with width = 8 and entry_actions.segments_left = UINT8_MAX;.^[1] And the copy is a field-to-field operation:

desc.type = DOCA_FLOW_ACTION_COPY;
desc.field_op.src.field_string = "outer.srv6.segments";
desc.field_op.dst.field_string = "outer.ipv6.dst_ip";
desc.field_op.width = 128;

^[1] The field table says outer.srv6.segments_left supports Add and Copy and holds the “Segments remaining count”, while outer.srv6.segments supports Copy as a source only and holds the “Active segment (128-bit SID)”.^[1]

Now put the last two together. Decrement segments_left, then copy the active segment into the destination address: those are steps S13 and S14 of the RFC 8986 End pseudocode, executed in NIC hardware.^[7] The same pair with a compressed container is what Cumulus calls uN on a Spectrum-4 switch.^[9][10] So a ConnectX or BlueField can be an SR source node and perform the endpoint arithmetic — but note what is not in the list: there is no decapsulation primitive, so full service-SID termination (End.DT46, USD) is not offered here in 3.5.0.^[1][7]

0000  60 00 00 00 05 a0 2b 40 fc bb bb bb 01 00 00 00 
0010  00 00 00 00 00 00 00 01 fc bb bb bb 00 01 00 00 
0020  00 00 00 00 00 00 00 00 29 04 04 01 01 00 00 00 
0030  fc bb bb bb 00 02 00 00 00 00 00 00 00 00 00 00 
0040  fc bb bb bb 00 01 00 00 00 00 00 00 00 00 00 00 
…… 1400 B of inner packet not shown

ip -6 route add 2001:db8:1::/64 encap seg6 mode encap segs A,B,C dev eth1

3Building the SRH by hand: four numbers DOCA will not check for you

The pipe action fixes the SRH size once:

#define NB_SIDS 2
struct doca_flow_external_action_srv6_pipe_action_cfg push_cfg = {0};
void *push_action = NULL;
push_cfg.op = DOCA_FLOW_EXT_ACT_SRV6_OP_PUSH;
push_cfg.port_id = port_id;
push_cfg.srh_size = sizeof(struct doca_flow_header_ipv6_srh_base)
                  + NB_SIDS * sizeof(struct doca_flow_ipv6_addr);
result = doca_flow_external_action_srv6_pipe_action_create(&push_cfg, &push_action);

^[1] and it is destroyed with doca_flow_external_action_srv6_pipe_action_destroy(push_action);.^[1] The constraint attached to srh_size is strict: it “Defines the total SRH header size in bytes. All entries added to this pipe action must provide an SRH of this exact size.”^[1] A two-segment policy and a three-segment policy therefore need two pipe actions, not one — a design consequence worth catching at the whiteboard rather than at runtime.

Per entry, the application builds the header itself:

uint32_t entry_len = sizeof(struct doca_flow_external_action_srv6_entry)
                   + NB_SIDS * sizeof(struct doca_flow_ipv6_addr);
struct doca_flow_external_action_srv6_entry *srv6_entry = calloc(1, entry_len);
struct doca_flow_header_ipv6_srh *srh = &srv6_entry->srh;
srh->base.hdr_ext_len   = (NB_SIDS * sizeof(struct doca_flow_ipv6_addr)) / 8;
srh->base.routing_type  = 4;
srh->base.segments_left = NB_SIDS - 1;
srh->base.last_entry    = NB_SIDS - 1;

^[1] The structure is “a fixed 8-byte base header followed by a flexible array of 128-bit segment addresses” — the same 8 bytes RFC 8754 specifies as Next Header, Hdr Ext Len, Routing Type 4, Segments Left, Last Entry, Flags and Tag.^[1][6]

Then the warning that makes this segment necessary: hdr_ext_len “Must be non-zero (at least one segment is required) and must be an even number, as each segment occupies 16 bytes (2 units of 8). DOCA Flow does not validate this field; the application must ensure it correctly reflects the total SRH size.”^[1] Nothing checks your arithmetic. A wrong hdr_ext_len produces a packet that a receiving endpoint will reject under RFC 8754’s bounds check — max_last_entry = (Hdr Ext Len / 2) - 1, and a violation yields an ICMP Parameter Problem Code 0 pointing at Segments Left — which is a long way from the NIC that built it.^[6]

So learn the four numbers as a set. For n segments: srh_size = 8 + 16n, hdr_ext_len = 2n, last_entry = n - 1, segments_left = n - 1 at the moment of push. Two segments: 40, 4, 1, 1. Three segments: 56, 6, 2, 2. The self-check that catches most mistakes is hdr_ext_len == (srh_size - 8) / 8, and it must be even.^[1][6]

One field you must not set: next_header. “When a PUSH operation executes, DOCA Flow automatically saves the original IPv6 next_header value before overwriting it with 43 (IPv6 Routing Header). After the SRH is inserted and the packet is reparsed, the saved value is restored into the SRH next_header field. Any manual application settings applied to srh->base.next_header are ignored.”^[1]

The plumbing that connects the two halves is one ordered-list element with two meanings: DOCA_FLOW_ORDERED_LIST_ELEMENT_EXTERNAL_ACTIONS “serves a dual role” — at pipe creation external_actions points at the pipe-level action handle; at entry creation it points at the per-entry struct doca_flow_external_action_srv6_entry holding that entry’s SRH.^[1]

4Four limitations, one contrasting layer, and some archaeology

NVIDIA lists four limitations and every one of them changes a design:^[1]

1. “Currently, only the PUSH operation is supported in hardware. SRH removal (POP) must be performed in software (future releases will introduce hardware POP support).”
2. “The total SRH size (base header + segment list) must not exceed the hardware’s insert-header maximum. The API will return a descriptive error message in such cases.”
3. “doca_flow_external_action_srv6_register() MUST be called before doca_flow_init().”
4. “SRv6 PUSH strictly requires an internal scratch register on the port. If this register is unavailable (which is dependent on the specific hardware model and firmware configuration), the pipe creation will fail.”

Limitation 4 is the one that will cost you a day. It means the same code can create a pipe on one card and fail on another with a different firmware configuration, and the guide gives no list of which models or settings qualify. That is why the first artefact to collect on any SRv6 pipe failure is mlxconfig -d <dev> q from both the working and the failing card.^[1] It also means which BlueField and ConnectX generations expose the SRv6 external action is [UNVERIFIED] — no NVIDIA page fetched states it, and the guide phrases the hardware dependency only as “dependent on the specific hardware model and firmware configuration”.^[1]

Underneath DOCA sits the DPDK mlx5 driver, and it does not tell the same story. In DPDK 26.07.0 the flow item SRv6 is supported in “sync HWS” and “template async HWS” but not “sync SWS”, and the actions “IPv6 ext push” and “IPv6 ext remove” appear in the same two columns: “There are more flow items for IPv6 extensions (like SRv6) which can be pushed or removed thanks to the flow offload actions RTE_FLOW_ACTION_TYPE_IPV6_EXT_PUSH and RTE_FLOW_ACTION_TYPE_IPV6_EXT_REMOVE.”^[5] The driver also documents matching constraints worth knowing: matching on an IPv6 routing extension header “requires HW steering, and supports the following fields: type, next_hdr, segments_left”; it is “not supported in flow template relaxed mode”; push and remove require HW steering, a non-zero group, “Supports TCP or UDP as next layer”, “IPv6 routing header must be the only present extension”, and are “Not supported on guest port”.^[5]

So IPV6_EXT_REMOVE exists in the DPDK driver while DOCA Flow 3.5.0 says hardware POP is not available. Both statements are true; they are different API layers over the same silicon. For a customer writing a DOCA application, the DOCA guide binds — and the honest answer is to state both facts and name which one governs their code.^[5][1]

Finally, the archaeology, because it is a transferable skill. The DOCA Flow guide has an SRv6 section in 3.3.0, 3.4.0 and 3.5.0 and zero occurrences of “srv6” in 3.2.0, so the feature landed in DOCA 3.3.0.^{[2][3][1][11]} And it was never a release highlight: the DOCA 3.5.0 “Changes and New Features” page does not mention SRv6 at all.^[4] If you index NVIDIA features by release notes, this feature does not exist.

5What this makes a BlueField in an SRv6 domain

Put the pieces in RFC terms. The NIC or DPU acts as an SR source node, but by insertion rather than encapsulation: DOCA Flow’s PUSH “inserts an SRH immediately after the IPv6 base header” and sets that same header’s next_header to 43 — it never builds an outer IPv6 header, so it matches none of RFC 8986’s four headend behaviors (H.Encaps, H.Encaps.Red, H.Encaps.L2, H.Encaps.L2.Red), every one of which encapsulates.^[1][7] It can additionally do the endpoint arithmetic — decrement segments_left, copy the active SID into the destination address — which is the End and uN primitive.^[1][7] What it does not offer as a DOCA Flow primitive in 3.5.0 is full endpoint decapsulation: no End.DT46, no USD, because hardware POP is not there.^[1]

That maps cleanly onto NVIDIA’s own switching story: the NICs originate and terminate SRv6 while the switches act as SRv6-aware nodes.^[9] Origination is DOCA Flow PUSH; the fabric executes uN and uA on Spectrum-4; termination on the far side is either software on the receiving host or a Linux seg6local endpoint.^[8]

flowchart LR
  A[App on BF-3 Arm] --> B[DOCA Flow pipe PUSH srh_size 40]
  B --> C[Wire: IPv6 nh 43 plus SRH 2 SIDs]
  C --> D[Spectrum-4 uN advances the list]
  D --> E[Linux host seg6local End.DT46]
  E --> F[Inner packet in VRF]