Adaptive routing, SHIELD and hash-based forwarding

1Adaptive routing needs both ends to agree

NVIDIA describes adaptive routing plainly: when unexpected traffic patterns cause some paths to be overloaded, AR can automatically move traffic to less congested paths according to the current temporal state of the network.^[2] NVIDIA’s own SuperPOD reference architecture names it alongside two siblings: InfiniBand provides additional features to optimize performance including adaptive routing (AR), collective communication with SHARP and dynamic network healing with SHIELD.^[14]

The operational fact that resolves most tickets is that AR has two ends. On the switch, MLNX-OS says AR support is enabled by default on system profile ib-single-switch, that you disable it with either system profile ib-no-adaptive-routing-single-switch or system profile ib with the no-adaptive-routing parameter, and then adds the sentence to quote back to a customer: the AR option needs to be enabled in the SM for it to take effect.^[2] On the SM side, UFM ships routing_engine = ar_updn and ar_mode = 3, where 3 is auto mode in which adaptive routing is determined by the routing engine; 0 disables it, 1 enables local adaptive routing in which switches select the exit port based on local buffer utilisation, and 2 is adaptive routing with notifications and is deprecated.^[1]

Before any of that, there is a question that can end the call in one line. If the customer runs the switch-embedded subnet manager, adaptive routing is not merely off - it is not implemented: subnet manager running via MLNX-OS does not support adaptive routing, fault routing (SHIELD or FRN), congestion control, or SHARP.^[2] Ask who the master SM is before you look at a single parameter.^[2]

ibdiagnet -r --fast_recovery    # AR validation + SHIELD info
# evidence files: ibdiagnet2.ar  ibdiagnet2.far  ibdiagnet2.rn  ibdiagnet2.rnc2
# SM side: dump_ar = TRUE writes the AR data to file

2The SM knob surface, and the two masks that silently exclude traffic

Once AR is genuinely available, four parameters decide how much of the customer’s traffic it touches.

ar_sl_mask defaults to 0xFFFF, a 16-bit bitmask indicating which SLs should be configured for AR; the UFM appliance CLI exposes it as ib sm ar-sl-mask <mask> over the range 0x0000 to 0xffff, and show ib sm ar-sl-mask prints the current value.^[1][3] ar_transport_mask defaults to 0x000A, and its description reads bit 0 = UD, bit 1 = RC, bit 2 = UC, bit 3 = DCT, bits 4-7 reserved - so the shipped default sets bits 1 and 3, enabling RC and DCT while leaving UD and UC out.^[1] That is a deliberate exclusion worth stating to a customer before they measure with the wrong traffic type.

enable_ar_by_device_cap is TRUE and read-only: enable adaptive routing only to devices that support packet reordering.^[1] That is the HCA half of the contract - AR reorders packets by design, so the SM will not arm it towards an adapter that cannot cope. Around those sit enable_ar_group_copy = TRUE, cache_ar_group_id = TRUE and dump_ar = FALSE, the last being the switch to write adaptive routing data to file when you need evidence.^[1] For trees that are not balanced there is a small family of their own: ar_tree_asymmetric_flow = 1, ar_tree_asymmetric_flow_threshold = 0 and ar_tree_asymmetric_flow_threshold_limit = 2.^[1]

The messages AR and pFRN exchange between switches are protected by their own key family. NVIDIA lists C_Key and N2N_Key as securing node management communication for the IB infrastructure features that propagate status and messages between nodes without directly involving the SM, and says this is used in adaptive routing algorithms and Proactive Fault Routing Notification (pFRN); N2N_Key granularity is per link, port to port.^[9] On a hardened fabric, key configuration and AR are therefore the same conversation.

3SHIELD and pFRN: the SL that must be mapped

SHIELD, NVIDIA’s self-healing fabric feature, appears in the SM parameter table as shield_mode, default 3 and read-only, described as advanced routing and fast link fault recovery: 0 disables fast link fault recovery, 1 enables local fast link fault recovery, 2 enables the legacy form, and 3 is auto mode in which the enhanced form is used with ar_updn, ar_ftree and dfp2 and the local form otherwise; changing the default value is not recommended.^[1] Note the coupling: the enhanced behaviour is available only under the adaptive engines, so the routing-engine choice made in the previous lesson decides which SHIELD you get.^[1]

pFRN, Proactive Fault Routing Notification, is how switches tell each other about a failure. Its parameters are pfrn_sl = 4 with the instruction to make sure pfrn_sl is properly mapped in the SL2VL QoS settings, pfrn_mask_clear_timeout = 180 seconds, pfrn_mask_force_clear_timeout = 720 seconds, and pfrn_over_router_enabled = 2 where 0 ignores, 1 disables and 2 enables.^[1]

That instruction is the whole trap. In an OpenSM SL2VL template, VL15 used there means drop this SL.^[7] The shipped qos_swe_sl2vl default is 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7, which maps SL 4 to VL 4 and is fine.^[7] But a customer who writes their own eight-class QoS design, or who inherits one, can easily land SL 4 on VL15 - and then pFRN notifications are dropped by the switch with no counter that says so, while shield_mode still reads 3 and every screenshot looks correct.^[1][7] The second half of the same trap is max_op_vls = 2 in UFM: an eight-lane design collapses onto two lanes, so the VL you carefully assigned may not exist operationally at all.^[1]

qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl      0,1,2,3,15,5,6,7,8,9,10,11,12,13,14,7
max_op_vls        2

qos_ca_max_vls 15
qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

4Hash-based forwarding is a third mechanism, not a synonym

Adaptive routing reacts to congestion. SHIELD and pFRN react to failure. Hash-based forwarding reacts to neither - it spreads flows deterministically across the members of an AR group by hashing header fields, which is why it appears in the same parameter block and means something different.

The knobs are hbf_sl_mask = 0xFFFF, hbf_hash_type = 0 where 0 is CRC and 1 is XOR, hbf_seed_typ = 0 where 0 is configurable and 1 is random, and hbf_seed = 0xFFFFFFFF, where that value stands for taking the 32 LSB of the node GUID as the HBF hash seed.^[1] The field selection is hbf_hash_fields = 0x0000000040F00C0F, whose documented bits are 1 DLID, 2 SL, 3 VL, 4 LRH LNH, 10 SGID, 11 DGID, 20 BTH destination QP, 21 DETH source QP, 22 DCETH V1 source QP, 23 DCETH V1 ISID and 30 input port, with the note that changing the value is not recommended.^[1] Weighted HBF adds hbf_weights, default auto, which configures the ratio between three subgroups of the adaptive routing group as a tuple of three values.^[1]

Two neighbouring features are easy to promise by accident. Congestion control is not on: upstream OpenSM marks --congestion_control as experimental and disabled by default, and UFM ships mlnx_congestion_control = 0, meaning ignore.^[11][1] And the UFM SDN documentation page for adaptive routing does not document the setup inline at all - it points readers to a community how-to, which is a useful thing to know before you promise a customer a manual page.^[10]

On the switch side of a newer platform, the AR resource pool is a system profile setting in NVOS: nv action change system profile adaptive-routing-groups 2048, and the profile change warning is explicit - the operation will reset the system configuration and initiate a reboot.^[8] The valid range is where the sources disagree: the profile page’s own example uses 2048 while a search summary claims 128 to 1792 in multiples of 128, so this course does not quote a range. Confirm it against the exact NVOS build in front of you before you put a number in a design document.^[8]

5Proving it, and what a two-node lab cannot show

Evidence, in the order a customer can actually produce it. ibdiagnet -r provides a report of the fabric qualities and validates adaptive routing configuration against up-down min-hop tables; --fast_recovery displays Fast Recovery information, which is the SHIELD and FRN family.^[6][4] The files to ask for are ibdiagnet2.ar for adaptive routing tables, ibdiagnet2.far for AR with SHIELD settings, ibdiagnet2.far_flid for non-local FLIDs only, ibdiagnet2.rn for SHIELD configuration tables and ibdiagnet2.rnc2 for SHIELD, SHIELDv2 and HBF counters.^[5] Remember that ibdiagnet2.ar is one of the files disabled by default and needs --enable_output.^[5] From the SM side, dump_ar = TRUE writes the adaptive routing data to file.^[1]

ibdiagnet -r --fast_recovery --enable_output default,ar
ibdiagnet -r --r_opt static_ca2ca     # static CA-to-CA checks even with AR on
ibdiagnet -r --r_opt check_sl         # no SL2VL mapping routes to VL15

The --r_opt sub-options matter here for two different reasons: static_ca2ca forces static CA-to-CA checks even with AR on, and check_sl validates that all SL2VL mappings do not route to VL15 - the direct machine check for the pFRN trap in segment 3.^[6]

routing_engine ar_updn
ar_mode 3
ar_sl_mask 0x0001
ar_transport_mask 0x000A
shield_mode 3
pfrn_sl 4
qos_swe_sl2vl 0,1,2,3,15,5,6,7,8,9,10,11,12,13,14,7
max_op_vls 2

routing_engine minhop
ar_mode 3
ar_sl_mask 0xFFFF
shield_mode 3