Domains, connection tracking, aging, meters

1Domains: where a pipe lives and the one-way rule

Every pipe belongs to a domain, set with doca_flow_pipe_cfg_set_domain. enum doca_flow_pipe_domain has four values: DOCA_FLOW_PIPE_DOMAIN_DEFAULT (ingress, non-secure), DOCA_FLOW_PIPE_DOMAIN_SECURE_INGRESS (decrypt only), DOCA_FLOW_PIPE_DOMAIN_EGRESS (encap), DOCA_FLOW_PIPE_DOMAIN_SECURE_EGRESS (encrypt only).^⁠1 The guide then states four constraints: DEFAULT and SECURE_INGRESS share one ingress root slot; EGRESS and SECURE_EGRESS share one egress root slot; ingress-to-egress forwarding requires a root pipe target; and no egress-to-ingress transitions are allowed.^⁠1 That last pair is the unidirectional rule: a packet travels ingress, then egress, then wire, and a pipe graph that tries to loop back is rejected.

simple_fwd_vnf shows the egress side. Its VXLAN_ENCAP_PIPE is created with doca_flow_pipe_cfg_set_domain(pipe_cfg, DOCA_FLOW_PIPE_DOMAIN_EGRESS) and doca_flow_pipe_cfg_set_is_root(pipe_cfg, true), carries an encap action (actions.encap_cfg.encap.tun.type = DOCA_FLOW_TUN_VXLAN, vxlan_tun_id = 0xffffffff, is_l2 = true, encap_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED), and forwards with fwd.type = DOCA_FLOW_FWD_PORT to the wire port.^⁠8 The CT UDP sample does the same for its encap pipe.^⁠3

flow_acl shows the ingress side and a subtle port rule. It keeps a static domain variable defaulting to DOCA_FLOW_PIPE_DOMAIN_DEFAULT with the comment that egress uses DOCA_FLOW_PIPE_DOMAIN_EGRESS; in DEFAULT the ACL pipe is created on ports[port_id] and allowed traffic goes to port_id ^ 1, whereas in EGRESS the pipe is created on ports[port_id ^ 1] and allowed traffic goes to port_id itself, and the sample then drains both ports’ queues because the entries were enqueued on different ports.^⁠7 Read that as: an ingress pipe belongs to the port the packet arrived on; an egress pipe belongs to the port the packet will leave from.

The no_wire_to_wire devargs hint from flow_common.c explains the cost of the boundary: without it, ingress-to-egress forwarding must check the direction bit and loop Rx traffic through an internal pre-egress table before entering the egress domain; with the hint it goes directly to egress.^⁠10

2Aging: hardware timers, software removal

Aging is a monitor property. monitor.aging_sec = N on an entry asks the e-switch to flag the entry after N idle seconds; at pipe level 0xffffffff marks the field changeable, which is what simple_fwd_vnf does so each flow can carry its own timeout.^⁠1⁠8 The aging sample sets monitor.aging_sec = 5 per entry, adds N-1 entries with DOCA_FLOW_ENTRY_FLAGS_WAIT_FOR_BATCH and the last with DOCA_FLOW_ENTRY_FLAGS_NO_WAIT, then doca_flow_pipe_queue_drain(port, 0, NULL).^⁠4

Collecting aged entries is an explicit loop:

uint64_t quota_time = 20; /* max handling aging time in ms */
int n = doca_flow_aging_handle(port, 0, quota_time, 0);
while (n != -1) {            /* -1 = full cycle complete */
	if (n > 0) total += n;
	n = doca_flow_aging_handle(port, 0, quota_time, 0);
}

Each aged entry arrives in the entry callback with op == DOCA_FLOW_ENTRY_OP_AGED, and the application frees it there with doca_flow_pipe_remove_entry(pipe_queue, DOCA_FLOW_ENTRY_FLAGS_NO_WAIT, entry).^⁠4 The sample stores a per-entry user_data struct (entry number, port id, status pointer) as the user_ctx so the callback knows which entry aged.^⁠4 simple_fwd_vnf runs the same handle call from an optional aging thread (-g/--age-thread).^⁠8

The aging sample also demonstrates the pipe miss counter: doca_flow_pipe_cfg_set_miss_counter(pipe_cfg, true) at creation, then doca_flow_resource_query_pipe_miss(pipe, &query_stats) to read query_stats.counter.total_pkts, which tells you how many packets fell through to fwd_miss.^⁠4 3.5.0 adds an API to disable the reset logic in DOCA-CT aging flows, for connections whose timer should not restart on every packet.^⁠9

3Meters: three RFCs, one struct

Meters are also monitor properties. enum doca_flow_meter_algorithm offers DOCA_FLOW_METER_RFC2697 (srTCM), DOCA_FLOW_METER_RFC2698 (trTCM) and DOCA_FLOW_METER_RFC4115 (trTCM without peak-rate dependency); enum doca_flow_meter_limit_type selects DOCA_FLOW_METER_LIMIT_TYPE_BYTES or _PACKETS.^⁠1 The meter sample uses the changeable pattern once more: at pipe level monitor.meter_type = DOCA_FLOW_RESOURCE_TYPE_NON_SHARED; monitor.non_shared_meter.cir = 0xffffffffffffffff; monitor.non_shared_meter.cbs = 0xffffffffffffffff;, and at entry level cir = 1024; cbs = 1024;, passed as the &monitor argument of doca_flow_pipe_basic_add_entry.^⁠5 The port must have been started with meter resources; the sample sets resource.nr_meters and the port helper translates that into doca_flow_port_cfg_set_nr_resources(port_cfg, DOCA_FLOW_RESOURCE_METER, n).^⁠5⁠10

Shared meters and counters exist for the case where many entries must be policed as one aggregate: reserve them with doca_flow_cfg_set_nr_shared_resource (system level) or doca_flow_port_cfg_set_nr_resources, obtain an id with doca_flow_port_shared_resource_get(port, DOCA_FLOW_RESOURCE_METER, &id), configure with doca_flow_port_shared_resource_set_cfg, and read with doca_flow_port_shared_resources_query.^⁠1 The ordered-list sample references a shared meter through meter.shared_meter.shared_meter_id and shared counters through counter.shared_counter.shared_counter_id, mixing them with non-shared counters in different list entries.^⁠6

Choosing an RFC: srTCM (2697) has one rate and two burst sizes, which suits a single committed rate with tolerance; trTCM (2698) adds a peak rate, so packets are yellow between committed and peak; RFC 4115 keeps two rates but colors excess traffic independently of the peak-rate dependency of 2698, the guide’s own description of the difference.^⁠1

4Connection tracking: prepare, add, process

The CT pipe (DOCA_FLOW_PIPE_CT) tracks a connection in both directions inside the e-switch.^⁠1 Initialization differs in three places. First, CT queues are declared at init: the samples call init_doca_flow_with_ct(nb_queues, "switch,hws", &resource, nr_shared_resources, &ct_cfg) with ct_cfg = { .ct_queues = 1, .ct_ctrl_queues = 1 }, which flow_common.c turns into doca_flow_cfg_set_ct_queues and doca_flow_cfg_set_ct_ctrl_queues; the CT queue id the sample then uses is nb_queues, one past the ordinary pipe queues.^⁠3⁠10 Second, the device must support CT: doca_flow_ct_cap_is_dev_supported(dev_info).^⁠2 Third, each port gets a struct ct_port_cfg with ct_flags = DOCA_FLOW_CT_FLAG_NO_AGING | DOCA_FLOW_CT_FLAG_NO_COUNTER and zone-match and modify masks for origin and reply, applied through doca_flow_port_cfg_set_ct_flags, doca_flow_port_cfg_set_ct_direction(port_cfg, is_reply, match_inner, zone_match_mask, modify_mask) once per direction, and optionally doca_flow_port_cfg_set_ct_actions_mem_size and doca_flow_port_cfg_set_ct_aging_core.^⁠3⁠10 Resources add resource.nr_ct_counters = nb_ipv4_sessions + nb_ipv6_sessions.^⁠3

The pipe itself is configured with doca_flow_pipe_cfg_set_ct_connections(cfg, nb_ipv4_sessions, nb_ipv6_sessions, 0), doca_flow_pipe_cfg_set_ct_max_connections_per_zone, doca_flow_pipe_cfg_set_ct_dup_filter_size, and an explicit doca_flow_pipe_cfg_set_match(cfg, &match, &mask); hits forward to one pipe, misses to another (fwd.type = fwd_miss.type = DOCA_FLOW_FWD_PIPE).^⁠3 The guide summarizes CT configuration as session pools, zone limits and a duplication filter.^⁠1

A connection is installed in three calls, all on the CT queue:

doca_flow_ct_entry_prepare(ct_queue, pipe, prepare_flags, match_origin, hash_origin,
			   match_reply, hash_reply, entry, &conn_found);
if (!conn_found)
	doca_flow_ct_add_entry(ct_queue, pipe, entry_flags, match_origin, match_reply,
			       actions_origin, actions_reply, fwd_origin, fwd_reply,
			       timeout_s, ct_status, *entry);
doca_flow_ct_entries_process(port, ct_queue, room, room, NULL);

prepare_flags are DOCA_FLOW_CT_ENTRY_FLAGS_ALLOC_ON_MISS | DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_ORIGIN | DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_REPLY; entry_flags are DOCA_FLOW_CT_ENTRY_FLAGS_NO_WAIT | DOCA_FLOW_CT_ENTRY_FLAGS_DIR_ORIGIN | DOCA_FLOW_CT_ENTRY_FLAGS_DIR_REPLY; if add fails after an ALLOC_ON_MISS prepare the sample calls doca_flow_ct_entry_prepare_rollback(ct_queue, pipe, *entry).^⁠2⁠3 The hashes come from the sample’s flow_ct_hash_6tuple over struct doca_flow_ct_match (5-tuple plus zone).^⁠2 The 3.5.0 tree ships thirteen CT samples, from flow_ct_udp and flow_ct_tcp to flow_ct_aging, flow_ct_tcp_ipv4_ipv6 and flow_ct_udp_tunnel_asymmetric.^⁠13

5NAT and what OVS-DOCA does with all this

DOCA Flow expresses NAT in two places. NAT64 is an ordered-list element, DOCA_FLOW_ORDERED_LIST_ELEMENT_NAT64, alongside _ACTIONS and _MONITOR.^⁠6 Four-tuple NAT for tracked connections goes through the CT entry’s actions_origin and actions_reply (struct doca_flow_ct_actions), which rewrite headers per direction; the exact field names inside that struct are not on the pages fetched for this lesson, so consult doca_flow_ct.h before quoting them to a customer.^⁠2⁠1

The same machinery is what OVS-DOCA uses when you type an OpenFlow ct() action, and its knobs are the operational face of this lesson: other_config:hw-offload-ct-size (default 250000, maximum 2M, best performance up to 1M), hw-offload-ct-ipv6-enabled (default false; IPv6 CT is new in 3.5.0), hw-offload-ct-unidir-udp-enabled (default false), and the stated limitation that ct-ct-nat, connection tracking with NAT, is unsupported.^⁠11⁠9 Meters map the same way: OVS-DOCA offloads OpenFlow meters, including chained meters, with a slow-path options:sw-meter=pps:10k:6k fallback.^⁠11

struct flow_resources resource = {0};
resource.mode = DOCA_FLOW_RESOURCE_MODE_PORT;
resource.nr_counters = 1;
resource.nr_ct_counters = nb_ipv4_sessions + nb_ipv6_sessions;  /* 1024 + 0 */
resource.nr_rss = 1;
resource.nr_encap = 1;
struct ct_init_cfg ct_cfg = { .ct_queues = 1, .ct_ctrl_queues = 1 };
init_doca_flow_with_ct(nb_queues, "switch,hws", &resource, nr_shared_resources, &ct_cfg);
uint16_t ct_queue = nb_queues;

struct doca_flow_meta o_zone_mask = {0}, r_zone_mask = {0};
struct doca_flow_ct_meta o_modify_mask = {0}, r_modify_mask = {0};
struct ct_port_cfg ct_port_cfg = {
	.ct_flags = DOCA_FLOW_CT_FLAG_NO_AGING | DOCA_FLOW_CT_FLAG_NO_COUNTER,
	.zone_match_mask = { &o_zone_mask, &r_zone_mask },
	.modify_mask = { &o_modify_mask, &r_modify_mask } };
init_doca_flow_switch_ports_with_ct(devs_manager, nb_devs, ports, 1, actions_mem_size, &resource, &ct_port_cfg);

doca_flow_pipe_cfg_create(&cfg, ports[0]);
set_flow_pipe_cfg(cfg, "CT_PIPE", DOCA_FLOW_PIPE_CT, false);
doca_flow_pipe_cfg_set_ct_connections(cfg, 1024, 0, 0);
doca_flow_pipe_cfg_set_ct_max_connections_per_zone(cfg, CT_DEFAULT_MAX_ZONE_SESSIONS);
doca_flow_pipe_cfg_set_ct_dup_filter_size(cfg, DUP_FILTER_CONN_NUM);
doca_flow_pipe_cfg_set_match(cfg, &match, &mask);          /* both zeroed: library-defined 5-tuple */
fwd.type = DOCA_FLOW_FWD_PIPE;      fwd.next_pipe = count_pipe;
fwd_miss.type = DOCA_FLOW_FWD_PIPE; fwd_miss.next_pipe = rss_pipe;
doca_flow_pipe_create(cfg, &fwd, &fwd_miss, &ct_pipe);
doca_flow_pipe_cfg_destroy(cfg);

doca_flow_ct_entry_prepare(ct_queue, ct_pipe,
	DOCA_FLOW_CT_ENTRY_FLAGS_ALLOC_ON_MISS | DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_ORIGIN | DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_REPLY,
	&match_o, hash_o, &match_r, hash_r, &entry, &conn_found);
doca_flow_ct_add_entry(ct_queue, ct_pipe,
	DOCA_FLOW_CT_ENTRY_FLAGS_NO_WAIT | DOCA_FLOW_CT_ENTRY_FLAGS_DIR_ORIGIN | DOCA_FLOW_CT_ENTRY_FLAGS_DIR_REPLY,
	&match_o, &match_r, &actions_o, &actions_r, NULL, NULL, 0 /* timeout_s */, &ct_status, entry);
doca_flow_ct_entries_process(ports[0], ct_queue, CT_DEFAULT_QUEUE_DEPTH, CT_DEFAULT_QUEUE_DEPTH, NULL);