SHARP: in-network reduction

1What SHARP is, and the two pieces that must both exist

SHARP improves the performance of MPI and machine-learning collective operations by offloading collective operations from CPUs and GPUs to the network and eliminating the need to send data multiple times between endpoints.^[1] With the third generation, multiple aggregation trees can be built over the same topology, so many parallel jobs get the in-network computing benefit over the same infrastructure.^[1]

There are exactly two software pieces: sharp_am, which runs in UFM, and libsharp, which is linked to client applications running on the compute nodes; the recommended setup is to enable SHARP in UFM and install libsharp on the compute nodes using HPC-X.^[2][7] The aggregation manager has a placement rule that ends several design conversations early: it is a central entity running on a dedicated server with a master subnet manager, and this dedicated server cannot serve as a compute node.^[3] On a Dell quote, that is a server line item, not a footnote.

Two facts about packaging trip people up. SHARP binaries ship as part of HPC-X, MLNX_OFED and UFM, where UFM includes the Aggregation Manager only; with MLNX_OFED the default install path is /opt/mellanox/sharp and HPCX_SHARP_DIR points at it.^[3] And on the DOCA side, the DOCA-Host page lists opensm, ibutils2, hcoll and sharp as proprietary packages requiring separate installation - installing a doca-* profile alone gives you neither a subnet manager nor SHARP.^[14]

sharp_hello -d mlx5_0:1 -v 3   # expect  type:LLT …  and  Test Passed.
SHARP_COLL_ENABLE_SAT=1        # adds a type:SAT streaming tree
ibdiagnet --sharp              # Fabric Summary → IB Aggregation Nodes

2Turning it on, and the prerequisites that are not switches

Enabling SHARP in UFM Enterprise is a config edit plus a restart. The gv.cfg block is [Sharp] with sharp_enabled = true, and optionally sharp_allocation_enabled = true; on the UFM Appliance the equivalent is the CLI enable command, and in both cases a UFM restart is required after making the change.^[8][2] The current documentation adds a version note worth remembering on an upgrade call: starting UFM v6.23.0, sharp_enabled is set to true by default, and a UFM Enterprise upgrade maintains the previous configuration.^[2]

Separately, the subnet manager has its own parameter: sharp_enabled defaults to 2, where 0 means ignore SHARP with no SHARP support, 1 means disable SHARP on all supporting switches, and 2 means enable SHARP on all supporting switches.^[9] On the appliance CLI these are ib sm sharp enable, no ib sm sharp enable and ib sm sharp ignore, the last of which does not change the current switch configuration; and it is not possible to modify the SHARP support parameter while OpenSM is running.^[10]

The prerequisites list from the environment page is the one to work through with a customer, because most of it is not about switches at all. Run sharp_am as root. Make sure the onboard subnet manager is disabled in the managed switches. The AM needs a dedicated server with a master SM that cannot be a compute node. Configure TCP/IP, because the AM and the clients talk over TCP/IP. And enable IPoIB on the compute servers in order to enable using UD multicast for result distribution in SHARP.^[3] On the transport itself: by default sharp_am communicates with libsharp clients over IPoIB using TCP port 6126, and setting smx_protocol = 1 in sharp_am.cfg switches to native IB via UCX; on a UFM Appliance Gen 3.x the firewall blocks 6126 until you open it.^[7]

The routing requirement is quoted verbatim because it is a design constraint, not a tuning knob: tree based topologies use updn, ar_updn, ftree or ar_ftree; DragonFly+ topology uses dfp; hypercube topologies use the dor routing engine with dor_hyper_cube_mode enabled.^[3] Combine that with the previous lesson: an engine chosen for deadlock-freedom on an irregular fabric, such as LASH or Nue, is not on this list.^[3]

3What the ASIC and the adapter allow

Two ceilings decide how many jobs can use SHARP at once, and they are per generation. NVIDIA Quantum supports up to 126 aggregation trees in the subnet, 63 low-latency trees and 63 streaming aggregation trees, and the number of SHARP streaming aggregation operations is limited to one active tree per switch. NVIDIA Quantum-2 supports up to 1023 aggregation trees in the subnet, 511 low-latency and 511 streaming, and the limit is one active tree per port.^[3] The generation mapping from the product side is that QM9700 and Quantum-2 introduce the third generation of SHARP, SHARPv3.^[15]

The adapter matters as much as the switch: ConnectX-5 supports SHARP low latency operation only, while ConnectX-6 and above support both SHARP low latency and streaming aggregation operations.^[3] On a mixed customer pool, the oldest adapter in the job sets what that job can use.

Tree allocation has changed default once, and the last explicit statement found is in the 3.8 documentation: in SHARP v3.3 and earlier, sharp_am used to operate in static trees mode by default, and as of SHARP v3.4 the default operation mode is dynamic trees, which is recommended as the preferred option.^[4] Dynamic trees are created per job and deleted when the job ends.^[4] Their parameters are dynamic_tree_allocation, dynamic_tree_algorithm (FatTree-oriented by default, with a QFT or Dragonfly variant), and the note that if the number of root switches in the fabric is larger than 126 when using the FatTree-oriented algorithm, it is desirable to modify max_trees_to_build to equal the number of root switches; a sharp_am restart is required.^[4] The limitation is explicit: dynamic trees are available for Fat-Tree, Quasi-Fat-Tree and Dragonfly topologies and are not supported for Hypercube topologies.^[4] Treat the SHARP 3.16 behaviour as most likely unchanged but state it as of the 3.8 documentation, which is the last page that says it outright.^[4][1]

A useful definition to carry to a customer call: a SHARP tree is a set of switches and their connected links to be used by one or more SHARP jobs; a single tree can be used by multiple jobs as long as they use different areas of the tree, and a single job can use multiple trees when it operates on multiple rails, each rail using a different tree.^[4]

4Reading a sharp_hello transcript field by field

sharp_hello verifies that a SHARP client can communicate with sharp_am and successfully create a SHARP job; it checks job creation only and does not perform data aggregation.^[2] Its usage is sharp_hello <-d | --ib_dev> <device> [OPTIONS] with -v setting the libsharp collective verbosity level, default 2, on a scale of 0 fatal, 1 error, 2 warn, 3 info, 4 debug, 5 trace.^[2] The documented invocation and its output:

$ sharp_hello -d mlx5_0:1 -v 3
[thor001:0:15042 - context.c:751] INFO tree_info: type:LLT tree idx:0 treeID:0x0 caps:0x6 quota: ( osts:167 user_data_per_ost:1024 max_groups:167 max_qps:1 max_group_channels:1)
Test Passed.

^[2] Field by field: type:LLT is a low-latency tree, and type:SAT is the streaming aggregation tree that appears as a second line when you set SHARP_COLL_ENABLE_SAT=1, in the documented example as type:SAT tree idx:1 treeID:0x3f caps:0x16.^[2] treeID identifies which tree the AM handed this job. The quota block is the resource grant - osts, user_data_per_ost, max_groups, max_qps, max_group_channels - and is what shrinks when the fabric’s trees are already busy. The log lines also expose rail_idx per rail and an mlid per group, the multicast LID over which results are distributed.^[2] Test Passed. means one thing only: a job was created.^[2]

Two expected-output markers from the 3.0 testing page make good grep targets in a script: INFO job (ID: ...) and INFO tree_info: type:LLT, followed by Test Passed.^[6] The next rung of proof is sharp_coll_test, which measures bandwidth between the node’s HCA and its connected switch and also verifies SHARP data transfer, and then $HPCX_SHARP_DIR/sbin/sharp_benchmark.sh, whose mandatory environment is sharp_ib_dev="mlx5_0:1" plus sharp_hostlist, with sharp_ppn defaulting to 1 and sharp_test_iters to 10000.^[2][6]

One environment gotcha that costs an afternoon: by default sharp_am blocks libsharp from running on management nodes, including the active and standby UFM machines; to run a client there you set ignore_sm_guids = False in conf/sharp/sharp_am.cfg and restart sharp_am.^[2] So testing from the UFM box itself is the one place where a correct setup still fails.

5The five-step triage, in fixed order

The order matters because the first two steps are answered from the SM configuration, before anyone touches a compute node.

Fabric-side, ibdiagnet --sharp collects SHARP configuration, with --sharp_control_version <0|1|2> defaulting to 0 meaning all, and --sharp_opt accepting csc, dsc, dscp, ad_hoc, trees; the results land in ibdiagnet2.sharp.^[12][13] The documented worked command on the SHARP side is ibdiagnet --sharp --fabric_summary followed by cat /var/tmp/ibdiagnet2/ibdiagnet2.sharp | grep -c TreeID.^[6] Two cautions before you paste that into a customer runbook: NVIDIA pages disagree on ibdiagnet’s default output directory, so read the tool’s own tail line rather than assuming a path; and --fabric_summary appears in the SHARP guide’s worked command but not in the IBUtils2 flag table, so check ibdiagnet -h on the build in front of you.^[6][12] The Fabric Summary itself is what you are after, and its aggregation-node line is the number to read:

Fabric Summary
Total Nodes            : 24
IB Switches            : 4
IB Channel Adapters    : 16
IB Aggregation Nodes   : 4
IB Routers             : 0

$ sharp_hello -d mlx5_0:1 -v 3
[node07:0:20114 - context.c:751] INFO tree_info: type:LLT tree idx:0 treeID:0x0 caps:0x6 quota: ( osts:167 user_data_per_ost:1024 max_groups:167 max_qps:1 max_group_channels:1)
Test Passed.