QoS: service levels, virtual lanes, SL2VL and VL arbitration

1From service level to lane to bandwidth

The mechanism is three steps and each one is a table someone has to program. “Each port has its own set of buffers. The buffering is channeled through virtual lanes (VL) where each VL has its own flow control. There may be up to 16 VLs.”^[4] “The actual VL that a packet uses is configured by the SM in the switch/channel adapter tables and is determined based on the Service Level (SL) specified in every packet. There are 16 possible SLs.”^[4] And one lane is spoken for: “All ports must support VL15 which is reserved exclusively for subnet management datagrams.”^[4]

That reservation has a consequence inside the configuration language. In OpenSM’s SL2VL template, “Note that VL15 used here means drop this SL.”^[1] So the value 15 in a mapping table is not lane 15 — it is a discard, and it produces exactly the symptom customers describe as “the fabric is up and this one class of traffic disappears”.

Bandwidth comes from the two VL arbitration tables. Entries are VL and weight pairs, where a weight is “a VL number (values from 0-14), and a weighting value (values 0-255), indicating the number of 64 byte units (credits) which may be transmitted from that VL when its turn in the arbitration occurs”, with weight 0 skipping the entry.^[1] The unit conversion is the part to memorise: “for 4KB MTU a single packet will require 64 credits, so … the weighting values for each VL should be multiples of 64.”^[1] A weight of 4 is 256 bytes — a fraction of one packet.

The high and low tables are separated by a limit. “the number of bytes that can be sent is high_limit times 4K bytes. A high_limit value of 255 indicates that the byte limit is unbounded. Note: if the 255 value is used, the low priority VLs may be starved. A value of 0 indicates that only a single packet from the high-priority table may be sent before an opportunity is given to the low-priority table.”^[1] The two IBA anchors the documentation itself gives are SL2VL mapping at IBA 7.6.6 and VL arbitration at IBA 7.6.9 — useful when a customer’s own spec extract needs matching to what OpenSM writes.^[1][2]

2Where the tables live, and the section that is thrown away

QoS is off by default upstream: -Q/--qos — “This option enables QoS setup. It is disabled by default.”^[2] UFM ships qos = TRUE, enabled by “setting the qos flag to TRUE in the /opt/ufm/files/conf/opensm/opensm.conf file”.^[3][10] So the very first question about any customer’s QoS is not what they configured, it is whether QoS is on at all and who decided.

The tables are configured in the SM’s options file through four parameter sets: qos_ca_* for channel adapters, qos_swe_* for switch external ports, qos_sw0_* for switch port 0 and qos_rtr_* for routers.^[1] The shipped defaults, verbatim:^[1]

qos_ca_max_vls 15
qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
qos_swe_max_vls 15
qos_swe_high_limit 0
qos_swe_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_swe_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_swe_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

Read the SL2VL line: identity for SL0 through SL14, and SL15 mapped to VL7 rather than 15 — because 15 would mean drop.^[1] Read the weights: 4 credits is 256 bytes, well under one 4 K packet, so the shipped arbitration is a placeholder rather than a design.^[1]

Now the trap that wastes whole engagements. The QoS policy file has four sections — port-groups, qos-setup, qos-levels, qos-match-rules — and the second one does nothing: “This section describes how to set up SL2VL and VL Arbitration tables on various nodes in the fabric. However, this is not supported in OpenSM currently - the section is parsed and ignored. SL2VL and VLArb tables should be configured in the OpenSM options file”.^[1] A customer whose SL2VL design lives in qos-policy.conf has a design that has never once been applied, and nothing in the logs says so.

What the policy file does do is assign SLs to traffic. A QoS level carries an SL plus optional mtu-limit, rate-limit, pkey and packet-life, and “One QoS level that is mandatory to define is a DEFAULT QoS level.”^[1] The shortest legal file is therefore:^[1]

qos-levels
    qos-level
        name: DEFAULT
        sl: 0
    end-qos-level
end-qos-levels

Match rules are ordered — “Rules are scanned in order of appearance in the QoS policy file such as the first match takes precedence” — matching on source port group, destination port group, PKey, QoS class or Service ID, and the policy is re-parsed “during fabric initialization and at every heavy sweep”.^[1] The ordering gotcha worth writing on the change ticket: put the SRP or target-port-GUID rule last, because any ULP query may carry a target port GUID and would otherwise be misclassified as SRP.^[1]

qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
max_op_vls        15

qos_ca_max_vls 15
qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low  0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl      0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

3max_op_vls, and the eight-class design that becomes two

Here is the second way a careful QoS design does nothing. UFM ships max_op_vls = 2, described as the “Limit of the maximum operational VLs”, with the table’s own warning: “Note, SM will flap all fabric links to deploy the configuration upon parameter change.”^[3]

Two operational lanes means every SL in a sixteen-entry SL2VL table lands on VL0 or VL1 no matter what the table says. The classes still exist as labels; they no longer exist as separate buffers or separate flow control, which is the entire point of the exercise. So an eight-class design on a stock UFM fabric behaves as a two-class design, and it does so consistently enough that customers report “QoS makes no difference” rather than “QoS is broken”.

The order of checks that follows from this is short and worth internalising: qos enabled at all, then max_op_vls, then qos_*_sl2vl, then the VLArb weights.^[3][1] Reversing it means auditing tables whose lanes do not exist.

The warning attached to the parameter is equally important commercially. Raising max_op_vls flaps all fabric links, so it is a maintenance-window change on a production training cluster, not a live tune.^[3] Plan it with the customer’s job scheduler, not during a bridge call.

4Verifying a QoS design against the fabric

Four read-backs, from narrowest to widest.

Per port, straight from the SMA: smpquery SL2VL <lid> <port> and smpquery VLArb <lid> <port> return the programmed tables, using the operation aliases SL2VLTable (SL2VL) and VLArbitration (VLArb).^[7] This is the ground truth for one port and needs no SA.

Fabric-wide, ibdiagnet --qos “Displays qos config sl” and the SL and VL tables of switches land in ibdiagnet2.slvl; routing validation additionally dumps ibdiagnet2.vl2vl.^[9][6][5] The check that matters most is ibdiagnet -r --r_opt check_sl, which validates that “all SL2VL mappings don’t route to VL15” — the direct test for the silent-drop case.^[5]

Under traffic, perfquery reads per-SL data counters with -X/--xmtsl (PortXmitDataSL) and -S/--rcvsl (PortRcvDataSL), plus optional per-VL counters --vloppackets, --vlopdata and --vlxmitcounters.^[8] Remember the unit trap when you quote a number: “components that represent Data (e.g. PortXmitData and PortRcvData) indicate octets divided by 4 rather than just octets.”^[8]

To generate that traffic on a chosen SL, perftest takes -S/--sl with a default of 0, and both sides must be given the same options — “The SAME OPTIONS must be passed to both server and client.”^[11] That gives you the closed loop: set an SL, move bytes, and watch the counter for that SL rather than the aggregate.