The DOCA Flow model

1Bring-up: cfg, init, ports

DOCA Flow is the GA steering library in 3.5.0; OVS-DOCA and the samples all sit on it.^[14] Every program starts the same way, and flow_common.c at tag 3.5.0 is the reference for the exact names.^[2]

struct doca_flow_cfg *flow_cfg;
doca_flow_cfg_create(&flow_cfg);
doca_flow_cfg_set_pipe_queues(flow_cfg, nb_queues);            /* one pipe_queue per core */
doca_flow_cfg_set_mode_args(flow_cfg, "vnf,hws");              /* or "switch,hws" */
doca_flow_cfg_set_cb_entry_process(flow_cfg, check_for_valid_entry);
doca_flow_cfg_set_cb_pipe_process(flow_cfg, pipe_process_cb);
doca_flow_cfg_set_resource_mode(flow_cfg, DOCA_FLOW_RESOURCE_MODE_PORT);
/* system-level alternative: doca_flow_cfg_set_nr_counters / _set_nr_meters / _set_nr_shared_resource */
/* CT only: doca_flow_cfg_set_ct_queues / doca_flow_cfg_set_ct_ctrl_queues */
doca_flow_init(flow_cfg);
doca_flow_cfg_destroy(flow_cfg);

The guide lists the remaining setters: doca_flow_cfg_set_queue_depth, doca_flow_cfg_set_rss, doca_flow_cfg_set_definitions, doca_flow_cfg_set_cb.^[1] simple_fwd_vnf additionally attaches the Flow Tune server with doca_flow_tune_cfg_create, doca_flow_tune_cfg_set_profile(…, DOCA_FLOW_TUNE_PROFILE_FULL) and doca_flow_cfg_set_tune_cfg.^[6]

Ports come next. For each port the helper creates a doca_flow_port_cfg, sets doca_flow_port_cfg_set_port_id, _set_actions_mem_size, and one doca_flow_port_cfg_set_nr_resources(port_cfg, DOCA_FLOW_RESOURCE_<COUNTER|METER|COUNTER_CT|RSS|PSP|ENCAP|DECAP|IPSEC_SA|EVENT_RSS|TRIM>, n) per resource, then binds the device with doca_flow_port_cfg_set_dev(port_cfg, dev) (or doca_flow_port_cfg_set_dev_rep for a representor), sets devargs such as "th_win_us=0" or "no_wire_to_wire,th_win_us=0", and calls doca_flow_port_start(port_cfg, &port); the cfg is destroyed right after.^[2] simple_fwd_vnf also sets doca_flow_port_cfg_set_priv_data_size to hang per-port state off the port.^[6] In vnf mode the ports are paired both ways, doca_flow_port_pair(ports[1], ports[0]) and doca_flow_port_pair(ports[0], ports[1]), so each port’s wire traffic egresses on its partner.^[2] In 3.5.0 you can query what a device supports before starting with doca_flow_port_capabilities_create(doca_dev, &caps) and _destroy.^[1]

Teardown is the mirror image and the order matters: doca_flow_port_pipes_flush on every port in reverse order, then doca_flow_port_stop in reverse order because in switch mode port 0 is the proxy port and must stop last, then doca_flow_destroy().^[2]

Two prerequisites sit outside the API: hugepages (echo '1024' | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages and a hugetlbfs mount), and, when DPDK probes the ports, devargs of the form -a <pci_id>,dv_flow_en=2,dv_xmeta_en=4,representor=[pf0vf0-2].^[1]

2Modes: vnf, switch, and the expert suffix

The mode string decides what happens to a packet that misses every pipe and where pipes live.^[1]

The guide documents two modes, vnf and switch, plus the expert switch variant; those rows are its wording.^[1] In switch mode mode_args also carries hairpin tuning options — hairpinq_num=[n], use_huge_mem, lock_rx_mem — when an internal wire-to-wire hairpin is created; there is no hairpin mode and no isolated suffix on the 3.5.0 page.^[1] The hws observation comes from the samples themselves: flow_acl, flow_lpm and flow_rss_meta call init_doca_flow(nb_queues, "vnf,hws", …), flow_switch calls it with "switch,hws".^[3][4][8] Switch samples expose the expert variant as the --expert-mode (-exp) flag.^[16]

Switch mode changes where you create pipes. flow_switch initializes ports with init_doca_flow_switch_ports and then creates its pipe on doca_flow_port_switch_get(ports[0]), the switch-manager port, with a pipe-level fwd.type = DOCA_FLOW_FWD_PORT; fwd.port_id = 0xffff (changeable) and per-entry fwd.port_id = port_base + 1 + entry_index.^[8] The connection-tracking samples run in "switch,hws" too, because CT needs the whole e-switch, not a pair of ports.^[12]

simple_fwd_vnf takes the mode from the command line and pairs ports as vnf does; its HAIRPIN_PIPE forwards to another pipe and its RSS_PIPE forwards to software queues, so the application chooses per flow whether to hairpin in hardware or receive on the Arm.^[6][7]

3Seven pipe types, seven add-entry calls

enum doca_flow_pipe_type has seven members: DOCA_FLOW_PIPE_BASIC, _CONTROL, _LPM, _ACL, _HASH, _ORDERED_LIST, _CT, chosen with doca_flow_pipe_cfg_set_type(pipe_cfg, type).^[1] Every pipe is built the same way: doca_flow_pipe_cfg_create(&pipe_cfg, port), then name, type and root flag (doca_flow_pipe_cfg_set_name, _set_type, _set_is_root, wrapped by the samples’ set_flow_pipe_cfg), optional _set_nr_entries, _set_match, _set_actions, _set_monitor, _set_domain, and finally doca_flow_pipe_create(pipe_cfg, &fwd, &fwd_miss, &pipe).^[2][3] What differs is the add-entry call.

The BASIC form is what flow_acl’s one-entry root and simple_fwd_vnf use; the first argument is the pipe queue, the fourth an action index.^[3][6] A CONTROL pipe is created with _set_type(DOCA_FLOW_PIPE_CONTROL) and _set_is_root(true), and its entries carry a match, a mask the samples leave NULL, four further optional pointers also left NULL, a monitor, a per-entry priority, fwd, user context and entry.^[9] LPM entries pass explicit masks such as DOCA_HTOBE32(0xffffffff), 0xffff0000 and 0x00000000 for the default route.^[4] ACL entries take a priority where 0 ≤ priority ≤ 1024 and the lowest value is the highest priority; a port mask equal to the port means exact match, 0 means any, and a mask larger than the port means a range.^[3] In a HASH pipe the entry index is hash_func(destination IPv4 address) mod nb_flows, and software can compute the same value with doca_flow_pipe_calc_hash(pipe, &match, &hash).^[10] An ORDERED_LIST entry is a struct doca_flow_ordered_list whose elements are DOCA_FLOW_ORDERED_LIST_ELEMENT_ACTIONS, _MONITOR or _NAT64.^[11] CT is covered in lesson 4.3.^[12]

4Match classes, action order, monitor, fwd, completion

A pipe-level match classifies every field into one of three classes. Ignored: field value zero, no comparison performed. Constant: a fixed value across all entries, 0 < V < 0xffff, mask NULL. Changeable: per-entry values, V = 0xffff, mask NULL.^[1] Passing doca_flow_pipe_cfg_set_match(pipe_cfg, &match, NULL) uses this implicit convention; passing an explicit &match_mask overrides it.^[1] struct doca_flow_match groups outer, inner, tun, parser_meta and meta; parser-meta selectors such as match.parser_meta.outer_l3_type = DOCA_FLOW_L3_META_IPV4, outer_l4_type = DOCA_FLOW_L4_META_TCP or tun_type = DOCA_FLOW_TUN_META_VXLAN let a pipe test the parsed protocol rather than header bytes.^[1] The samples’ 5-tuple template is exactly that: outer.l3_type = DOCA_FLOW_L3_TYPE_IP4, outer.ip4.src_ip = 0xffffffff, outer.ip4.dst_ip = 0xffffffff, outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP, outer.tcp.l4_port.src_port = 0xffff, outer.tcp.l4_port.dst_port = 0xffff.^[5]

Actions execute in a fixed order regardless of how you fill the struct: Crypto (decrypt), Decapsulation, Pop, Meta, Outer, Tun, Push, Encapsulation, Crypto (encrypt), then action descriptors (desc_array).^[1] enum doca_flow_action_type offers DOCA_FLOW_ACTION_AUTO, _ADD, _COPY, and actions are attached with doca_flow_pipe_cfg_set_actions(pipe_cfg, actions_arr, actions_mask_arr, descs, nb).^[1] A changeable action follows the same convention as a match: flow_rss_meta sets actions.meta.pkt_meta = UINT32_MAX at pipe level and DOCA_HTOBE32(10) in the entry.^[5]

The monitor carries per-entry counters, meters, aging and mirroring. monitor.counter_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED creates a counter per entry automatically; read it with doca_flow_resource_query_entry(entry, &query_stats) and query_stats.counter.total_pkts / total_bytes.^[1][3] Meters and aging get their own lesson.

Forwarding is struct doca_flow_fwd with enum doca_flow_fwd_type. The guide uses DOCA_FLOW_FWD_PIPE (next_pipe), _PORT (port_id), _RSS, _DROP, _CHANGEABLE, _TARGET (to a target such as the kernel, doca_flow_get_target()), _TRIM (fwd.trim.trim_id, the trim shared resource) and _HASH_PIPE (dynamic hash-algorithm selection); a fwd_miss handler accepts only _PIPE, _DROP and _TARGET.^[1] _ORDERED_LIST_PIPE is what an ordered-list pipe is forwarded to in the sample.^[11] RSS forwarding sets fwd.rss_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED, a queues_array, nr_queues, and hash flags such as DOCA_FLOW_RSS_IPV4_SRC or DOCA_FLOW_RSS_TCP_DST in rss.inner_flags or outer_flags; the port must have been started with nr_rss resources.^[5][1]

Completion is asynchronous. Entries carry DOCA_FLOW_ENTRY_FLAGS_WAIT_FOR_BATCH or DOCA_FLOW_ENTRY_FLAGS_NO_WAIT; the application then calls doca_flow_entries_process(port, pipe_queue_id, timeout_usec, max_entries) or doca_flow_pipe_queue_drain(port, queue, NULL), and the callback void cb(struct doca_flow_pipe_entry*, uint16_t pipe_queue, enum doca_flow_entry_status, enum doca_flow_entry_op, void *user_ctx) reports DOCA_FLOW_ENTRY_STATUS_SUCCESS or not; doca_flow_pipe_entry_get_status(entry) reads it later.^[1] The samples’ check_for_valid_entry sets status->failure on any non-success and increments nb_processed, and flow_process_entries compares that count with the number added.^[2] The one rule that bites in production: each core must use its own dedicated pipe_queue ID when calling entry addition to prevent race conditions.^[1]

5Worked: ACL then LPM then RSS, in code

simple_fwd_vnf shows the shape of a real pipeline: a root CONTROL_PIPE classifies the tunnel type, per-tunnel DOCA_FLOW_PIPE_BASIC pipes match the inner 5-tuple with a counter and monitor.aging_sec = 0xffffffff (changeable), each with fwd_miss pointing at the RSS_PIPE, and a VXLAN_ENCAP_PIPE in the egress domain forwards to the wire.^[6] Its loop is rte_eth_rx_burst, classify, on a miss add an entry, doca_flow_entries_process, optionally doca_flow_aging_handle, then rte_eth_tx_burst.^[7] Before running it you need hugepages and sudo mlxconfig -d <pcie_address> s FLEX_PARSER_PROFILE_ENABLE=3 plus a reboot, so the GRE, VXLAN and GTP parsers are enabled.^[7] The exercise below builds a smaller pipeline from the same parts.

struct flow_resources resource = {0};
uint32_t nr_shared_resources[SHARED_RESOURCE_NUM_VALUES] = {0};
struct doca_flow_port *ports[2];
uint32_t actions_mem_size[2];
struct entries_status status = {0};

resource.mode = DOCA_FLOW_RESOURCE_MODE_PORT;
resource.nr_counters = 64;     /* one per entry we plan to add */
resource.nr_rss = 1;           /* the RSS leaf needs an RSS resource */
init_doca_flow(nb_queues, "vnf,hws", &resource, nr_shared_resources);
ARRAY_INIT(actions_mem_size, ACTIONS_MEM_SIZE(64));
init_doca_flow_vnf_ports(2, ports, actions_mem_size, &resource);

static doca_error_t create_rss_pipe(struct doca_flow_port *port, struct doca_flow_pipe **pipe)
{
	struct doca_flow_pipe_cfg *pipe_cfg;
	struct doca_flow_match match = {0};
	struct doca_flow_fwd fwd = {0}, fwd_miss = {0};
	uint16_t rss_queues[1] = {0};

	match.parser_meta.outer_l3_type = DOCA_FLOW_L3_META_IPV4;   /* constant selector */

	doca_flow_pipe_cfg_create(&pipe_cfg, port);
	set_flow_pipe_cfg(pipe_cfg, "RSS_PIPE", DOCA_FLOW_PIPE_BASIC, false);
	doca_flow_pipe_cfg_set_match(pipe_cfg, &match, NULL);

	fwd.type = DOCA_FLOW_FWD_RSS;
	fwd.rss_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED;
	fwd.rss.queues_array = rss_queues;
	fwd.rss.nr_queues = 1;
	fwd.rss.inner_flags = DOCA_FLOW_RSS_IPV4_SRC | DOCA_FLOW_RSS_TCP_SRC;
	fwd_miss.type = DOCA_FLOW_FWD_DROP;

	doca_error_t r = doca_flow_pipe_create(pipe_cfg, &fwd, &fwd_miss, pipe);
	doca_flow_pipe_cfg_destroy(pipe_cfg);
	return r;
}
/* one entry, everything constant, so an empty match: */
doca_flow_pipe_basic_add_entry(0, rss_pipe, &empty_match, 0, NULL, NULL, NULL,
			       DOCA_FLOW_ENTRY_FLAGS_NO_WAIT, &status, &rss_entry);

struct doca_flow_fwd fwd = {.type = DOCA_FLOW_FWD_CHANGEABLE};
struct doca_flow_monitor counter = {.counter_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED};
match.outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match.outer.ip4.dst_ip = 0xffffffff;                       /* changeable */
doca_flow_pipe_cfg_create(&pipe_cfg, port);
set_flow_pipe_cfg(pipe_cfg, "LPM_PIPE", DOCA_FLOW_PIPE_LPM, false);
doca_flow_pipe_cfg_set_nr_entries(pipe_cfg, FLOW_COMMON_PIPE_RULES);
doca_flow_pipe_cfg_set_match(pipe_cfg, &match, NULL);
doca_flow_pipe_cfg_set_monitor(pipe_cfg, &counter);
doca_flow_pipe_create(pipe_cfg, &fwd, NULL, &lpm_pipe);
doca_flow_pipe_cfg_destroy(pipe_cfg);

/* /32 host route to the paired wire port */
match.outer.ip4.dst_ip = BE_IPV4_ADDR(10, 0, 0, 7);
match_mask.outer.ip4.dst_ip = DOCA_HTOBE32(0xffffffff);
fwd.type = DOCA_FLOW_FWD_PORT; fwd.port_id = port_id ^ 1;
doca_flow_pipe_lpm_add_entry(0, lpm_pipe, &match, &match_mask, 0, NULL, NULL, &fwd,
			     DOCA_FLOW_ENTRY_FLAGS_WAIT_FOR_BATCH, &status, &e[0]);
/* /16 prefix goes to software for inspection */
match_mask.outer.ip4.dst_ip = DOCA_HTOBE32(0xffff0000);
fwd.type = DOCA_FLOW_FWD_PIPE; fwd.next_pipe = rss_pipe;
doca_flow_pipe_lpm_add_entry(0, lpm_pipe, &match, &match_mask, 0, NULL, NULL, &fwd,
			     DOCA_FLOW_ENTRY_FLAGS_WAIT_FOR_BATCH, &status, &e[1]);
/* default route: drop */
match_mask.outer.ip4.dst_ip = DOCA_HTOBE32(0x00000000);
fwd.type = DOCA_FLOW_FWD_DROP;
doca_flow_pipe_lpm_add_entry(0, lpm_pipe, &match, &match_mask, 0, NULL, NULL, &fwd,
			     DOCA_FLOW_ENTRY_FLAGS_NO_WAIT, &status, &e[2]);

struct doca_flow_fwd fwd = {.type = DOCA_FLOW_FWD_CHANGEABLE};
struct doca_flow_fwd fwd_miss = {.type = DOCA_FLOW_FWD_PIPE, .next_pipe = lpm_pipe};
match.parser_meta.outer_l3_type = DOCA_FLOW_L3_META_IPV4;
match.outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match.outer.ip4.src_ip = 0xffffffff;  match.outer.ip4.dst_ip = 0xffffffff;
match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP;
match.outer.tcp.l4_port.src_port = 0xffff;  match.outer.tcp.l4_port.dst_port = 0xffff;
doca_flow_pipe_cfg_create(&pipe_cfg, port);
set_flow_pipe_cfg(pipe_cfg, "ACL_PIPE", DOCA_FLOW_PIPE_ACL, false);
doca_flow_pipe_cfg_set_nr_entries(pipe_cfg, 10);
doca_flow_pipe_cfg_set_domain(pipe_cfg, DOCA_FLOW_PIPE_DOMAIN_DEFAULT);
doca_flow_pipe_cfg_set_match(pipe_cfg, &match, NULL);
doca_flow_pipe_cfg_set_monitor(pipe_cfg, &counter);
doca_flow_pipe_create(pipe_cfg, &fwd, &fwd_miss, &acl_pipe);
doca_flow_pipe_cfg_destroy(pipe_cfg);

/* deny 1.2.3.4 -> 8.8.8.8 TCP any ports, priority 10 */
match.outer.ip4.src_ip = BE_IPV4_ADDR(1, 2, 3, 4);  match_mask.outer.ip4.src_ip = DOCA_HTOBE32(0xffffffff);
match.outer.ip4.dst_ip = BE_IPV4_ADDR(8, 8, 8, 8);  match_mask.outer.ip4.dst_ip = DOCA_HTOBE32(0xffffffff);
match.parser_meta.outer_l4_type = DOCA_FLOW_L4_META_TCP;  match_mask.parser_meta.outer_l4_type = UINT32_MAX;
match_mask.outer.tcp.l4_port.src_port = 0;  match_mask.outer.tcp.l4_port.dst_port = 0;   /* any port */
fwd.type = DOCA_FLOW_FWD_DROP;
doca_flow_pipe_acl_add_entry(0, acl_pipe, &match, &match_mask, 0, NULL, 10, &fwd,
			     DOCA_FLOW_ENTRY_FLAGS_NO_WAIT, &status, &acl_entry);

doca_flow_pipe_queue_drain(port, 0, NULL);
if (status.nb_processed != 6 || status.failure)   /* 1 root + 1 rss + 3 lpm + 1 acl */
	return DOCA_ERROR_BAD_STATE;
doca_flow_resource_query_entry(acl_entry, &q);  /* q.counter.total_pkts */