Skip to content
nodeza.learn
DOCA course/6. Acceleration/Lesson 6.2

Compress, SHA, Erasure Coding, AES-GCM

S6·E2The slide with an engine that is not there · NVIDIA briefing room, two days before the archive's quote goes out

S6·E2Apply~40 minsources checked todayverified against DOCA 3.5.0 docs and doca-samples tag 3.5.0, 2026-09-06

Builds on: DMA, RDMA, and Verbs

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • State exactly which compress and decompress operations BlueField-2 and BlueField-3 support and choose the correct task for each.
  • Explain what the absence of a SHA engine on BlueField-3 means for a customer design and name the alternatives.
  • Configure an Erasure Coding create/recover flow within the BlueField-3 block and count limits.
  • Run an AES-GCM encrypt sample with a valid key, IV and tag size, and read back its buffer constraints.
  • Measure any of these engines with DOCA Bench instead of quoting an unsourced throughput number.

Episode 2 — The slide with an engine that is not there

The situation · NVIDIA briefing room, two days before the archive's quote goes out

Slide six is on the screen and the Dell SE is proud of it: for the archive’s cold tier, the DPU offloads compression, hashing for deduplication, erasure coding and encryption. Their storage architect wants it confirmed so procurement can price the SKU; procurement only wants to know whether any of it moves the lead time. You confirm two of the four. The Compress guide lists deflate compress for BlueField-2; for BlueField-3 it lists deflate decompress and LZ4 decompress.[1] The SHA guide is blunter: “NVIDIA BlueField-3 does not support this library because it has no SHA acceleration engine.”[3] Erasure Coding and AES-GCM do run on BlueField-3, inside documented block and key limits.[4][6] The NVIDIA PM says everything else is “not announced”.

The room goes quiet, so you explain the asymmetry instead of defending it. These are fixed-function blocks in silicon, chosen per generation for the traffic the part was built to carry, and a firmware setting can expose a block but never add one.[3] A read-heavy archive decompresses far more often than it compresses, which is the shape the BlueField-3 engine matches.[1]

Then the network lead asks how fast: show me the counter. There is no per-engine throughput table on the 3.5.0 pages to quote, only DOCA Bench and a workload of their own.[8]

If a slide claims an offload, name the engine and the generation, or delete the line.

Segment 1 goes engine by engine, starting with compress.

1Compress and decompress: what each BlueField actually does

DOCA Compress is GA in 3.5.0, but the hardware behind it differs by generation and the guide is precise about it: “For BlueField-2 devices, this library supports: Compress operation using the deflate algorithm; Decompress operation using the deflate algorithm. For BlueField-3 devices, this library supports: Decompress operation using the deflate algorithm; Decompress operation using the LZ4 algorithm.”[1][7] Read that twice: on BlueField-3 there is no hardware deflate compress and no LZ4 compress. The datasheet’s storage feature list says “Decompression engine”, not compression, which matches.[10]

The task API mirrors the table. Configuration calls are doca_compress_task_compress_deflate_set_conf, doca_compress_task_decompress_deflate_set_conf, doca_compress_task_decompress_lz4_stream_set_conf and doca_compress_task_decompress_lz4_block_set_conf; capability probes are doca_compress_cap_task_<type>_is_supported, _get_max_buf_size, _get_max_buf_list_len and doca_compress_get_max_num_tasks.[1] Each task also yields checksums through getters: “Adler – produced by the deflate compress and decompress tasks”, “CRC – produced by all tasks”, “xxHash – produced by the LZ4 decompress tasks”.[1] The 3.5.0 sample tree contains compress_deflate, decompress_deflate and decompress_lz4_stream; the README notes the old generic “decompress LZ4 task” was removed in favor of the stream and block tasks.[2][15]

Samples take -p/--pci-addr, -f/--file, -o/--output and -c/--output-checksum; the deflate samples add -wf/--with-frame to write or read a zlib-framed file “compatible with default zlib settings”, and the LZ4 sample adds -bc/--has-block-checksum, -bi/--are-blocks-independent and its own -wf for LZ4 frames.[2][15] Without the frame flag DOCA Compress “handles payload only”.[2] A design consequence for storage stacks: on BlueField-3 the read path (decompress) can be offloaded while the write path (compress) stays on CPU or uses another engine — say so before the customer budgets cores.

2SHA: the engine BlueField-3 does not have

DOCA SHA exposes SHA1, SHA2-256 and SHA2-512 through doca_sha_task_hash_set_conf and doca_sha_task_partial_hash_set_conf, with capability queries doca_sha_cap_task_hash_get_supported, doca_sha_cap_task_partial_hash_get_supported, doca_sha_cap_get_max_src_buf_size, doca_sha_cap_get_min_dst_buf_size and doca_sha_cap_get_partial_hash_block_size.[3] Then the sentence that decides designs: “NVIDIA BlueField-3 does not support this library because it has no SHA acceleration engine.”[3] The library is GA in the index, but GA on BlueField-2 hardware only; on BF-2, PF, VF and SF all reach the same engine.[7][3]

What this means in practice. First, doca_sha device discovery on a BlueField-3 finds no capable device — the sha_create sample cannot run there.[11][3] Second, the DOCA Bench doca_sha step exists, but on BF-3 it will not enumerate; the NVIDIA bench skill’s advice is that empty enumeration means the library is not usable on that install, not a bench failure.[8][9] Third, integrity hashing for a BF-3 design lands on the host CPU, on the Arm cores in software, or on a BlueField-2. AES-GCM authenticates ciphertext with a tag, which is not a substitute for a standalone digest.[6]

The two samples show the API shape you would use on BF-2: sha_create sets a doca_sha_task_hash pool, populates two buffers, submits once and reads the digest; sha_partial_create chops the source into segments sized to the algorithm block size, feeds them through one doca_sha_task_partial_hash with doca_sha_task_partial_hash_set_src, and marks the last with doca_sha_task_partial_hash_set_is_final_buf.[11] The sha_create sample registers exactly one program flag, -d/--data (“user data”).[18]

3Erasure Coding: parity math on BlueField-3

DOCA Erasure Coding is GA and runs on “BlueField-3 with some limitations”, on any PF, VF or SF, with representors for host-DPU memory.[4][7] Matrices come from doca_ec_matrix_create(type) with DOCA_EC_MATRIX_TYPE_CAUCHY or DOCA_EC_MATRIX_TYPE_VANDERMONDE, plus doca_ec_matrix_create_from_raw, doca_ec_matrix_create_update and doca_ec_matrix_create_recover; tasks are doca_ec_task_galois_mul, doca_ec_task_create (encode), doca_ec_task_update and doca_ec_task_recover, each with *_set_conf, and capabilities doca_ec_cap_task_*_is_supported, doca_ec_cap_get_max_block_size and doca_ec_cap_get_max_buf_list_len.[4]

The documented BlueField-3 limits: “Data block count range: 1-128; Redundancy block count: 1-32; Block size: 64B-128MB”, and source data length must be a multiple of the block size and aligned to 64 bytes.[4] The erasure_coding_recover sample runs three phases: encode (split the input into -t/--data blocks, build a matrix by -x/--matrix cauchy or vandermonde, submit doca_ec_task_create, write -r/--rdnc redundancy blocks), delete (-d/--delete-index, comma-separated indices), and decode (read the survivors, detect missing blocks, doca_ec_matrix_create_recover() from the encoding matrix, submit doca_ec_task_recover, write the recovered file).[5][14] Defaults are -p 03:00.0, 2 data blocks, 2 redundancy blocks, cauchy, delete index 0, output under /tmp, with -b/--both to run encode and decode in one go.[14]

Scenario:

S3/NVMe-oF-heavy multi-tenant inference: SNAP and the storage initiator run on the Arm → DPU mode regardless of ROI (Dell ObjectScale pairs BF-3 with Spectrum-4 for S3 over RDMA). ⚠ 30% infra share is an estimate.

Features needed
Assumptions and sources (defaults are estimates — every field is editable)
  • Nodes = 200 — Fleet size.
  • Host cores per node = 64 cores — Physical cores across both sockets (e.g. 2× 32c = 64).
  • Cores consumed by infrastructure = 20 %estimate — Share of host cores spent on vSwitch/SDN agents, storage initiator, encryption, telemetry. NVIDIA marketing says "up to 30%"; hyperscaler blogs quote 25–40% — both secondary. ESTIMATE, measure your own with perf/top. source ↗
  • Server cost per core-year = $120estimate — e.g. $30k 2-socket server ÷ 64 cores ÷ 4-year life ≈ $117. ESTIMATE.
  • Per-core software licence per year = $0estimate — Hypervisor / SDN stacks licensed per core (VMware VCF is per-core). Default 0 — enter the customer's quote; this lever dominates in virtualization farms. ESTIMATE.
  • Watts per core = 6 Westimate — ≈ 350–400 W TDP ÷ 64 cores ≈ 6 W at load. ESTIMATE.
  • Electricity price = 0.12 $/kWhestimate — US commercial average ≈ $0.12–0.13/kWh (EIA). ESTIMATE — use the site tariff. source ↗
  • PUE = 1.3estimate — Facility overhead multiplier on IT watts. 1.3 is a typical enterprise value. ESTIMATE.
  • DPU price delta vs ConnectX = $2500estimate — Dell configurator snippets: BF-3 B3140H SuperNIC $6,213.80 (R7725), Partner DPU dual-port 400GbE $8,347.83 (R670) → ≈ $2.1k delta; ConnectX-7 400G list not fetched. UNVERIFIED ESTIMATE. source ↗
  • Extra watts per DPU vs NIC = 60 Westimate — B3220/B3240 are 150 W cards vs 75 W for the B3140H SuperNIC (Dell R7725 ISM). Real delta depends on load; 60 W is an ESTIMATE. source ↗
  • DPUs per node = 1 — HGX B300 RA uses one north-south BF-3 per node; VMware hosts typically one. source ↗
  • OVS / vSwitch datapath offload: frees 20% of infra cores in NIC mode, 35% in DPU mode estimate — ASAP² (tc-flower / OVS-DOCA) offloads the datapath on ConnectX too; DPU mode additionally moves ovs-vswitchd and the SDN agents to the Arm. source ↗
  • Connection tracking (CT) offload: frees 10% of infra cores in NIC mode, 15% in DPU mode estimate — CT offload is an ASIC feature (ConnectX-6 Dx and later); OVS-DOCA 3.5.0 adds IPv6 CT. source ↗
  • IPsec / TLS inline crypto: frees 15% of infra cores in NIC mode, 20% in DPU mode estimate — BF-3 datasheet: "MACsec/IPsec/TLS data-in-motion" inline. Crypto is SKU-dependent ("selective on SuperNICs") — check the OPN. source ↗
  • NVMe-oF / SNAP storage emulation: frees 0% of infra cores in NIC mode, 20% in DPU mode estimate — BlueField SNAP (NVMe / virtio-blk emulation) and the storage initiator run on the Arm cores → DPU mode only. source ↗
  • Tenant isolation / zero-trust: frees 0% of infra cores in NIC mode, 5% in DPU mode estimate — Zero Trust (Restricted) is a DPU-mode variant: the host admin loses port ownership, HW counters, tracer, RShim and FW flashing. source ↗
  • HBN (BGP/EVPN router on the DPU): frees 0% of infra cores in NIC mode, 5% in DPU mode estimate — HBN runs on the Arm side; not supported on 8 GB DDR SKUs. DPU mode only. source ↗
  • Model: incremental cores = infra cores × (DPU share − NIC share); savings = cores × ($/core-year + licence) + (cores × W/core − DPUs × extra W) × PUE × 8.76 kWh × $/kWh; payback = premium ÷ monthly savings. Recommendation rules: any Arm-only feature → DPU mode; no features + east-west AI → SuperNIC/ConnectX-8 in NIC mode; no features → plain ConnectX; else DPU mode only if payback ≤ 36 months.
Intro mode on the storage-heavy inference preset: the only two sliders are node count and the infrastructure core share, and the feature checkboxes are infrastructure services, not engines. Toggle NVMe-oF/SNAP emulation and IPsec/TLS crypto and watch the split between what a ConnectX already offloads and what DPU mode adds. The decompress, EC and AES-GCM decisions from this lesson sit inside the storage feature; SHA has no engine on BlueField-3, so it contributes nothing and the calculator does not offer it.

4AES-GCM: authenticated encryption with tight parameters

The AES-GCM guide states the library “is supported at alpha level and on NVIDIA BlueField-3 devices or higher”, while the 3.5.0 libraries index lists DOCA AES-GCM as GA — two NVIDIA pages that disagree, which you should quote together.[6][7] Keys are created with doca_aes_gcm_key_create; the sample’s -k/--key help spells out “32 characters for 128-bit key, and 64 for 256-bit key” in hex, and the code uses DOCA_AES_GCM_KEY_128 and DOCA_AES_GCM_KEY_256.[13] The IV is “0B-12B” long and the authentication tag is “only 12B and 16B”; additional authenticated data sits at the start of the source buffer, and on encrypt the tag is appended to the result while on decrypt it is verified and stripped.[6]

Tasks are doca_aes_gcm_task_encrypt and doca_aes_gcm_task_decrypt (*_set_conf, *_alloc_init, *_as_task), with capability calls doca_aes_gcm_cap_task_encrypt_is_supported, _get_max_buf_size and _get_max_list_buf_num_elem (and the decrypt equivalents).[13][6] Buffers may be local, PCIe-exported or linked lists. Task limitations: “The operation is not atomic” and “Once the task is submitted, the source and destination should not be read/written to”.[6]

Sample defaults from aes_gcm_common.c: device 03:00.0, output /tmp/out.txt, a 96-bit all-zero IV, tag size 12, AAD size 0, plus -ns/--num-src-buf and -nd/--num-dst-buf for list buffers.[13][12] Both samples follow the same eleven steps: locate device, init Core, set task conf, populate mmap with two buffers, allocate inventory elements, create the key, alloc-init the task, submit, retrieve, check, destroy.[12]

5Numbers: measure, do not quote

There is no official per-engine throughput table for these libraries on the fetched 3.5.0 pages, and NVIDIA’s own doca-bench skill deliberately ships no expected numbers.[8][9] The DOCA Bench guide shows an example block — “Throughput: 5712042 Operations/s, Ingress rate: 063.832 Gib/s” — as illustrative output with no stated workload; it is not a BlueField-3 spec.[8] The right artifact is a measured CSV with the command line, DOCA version, device and environment attached.[9]

DOCA Bench drives exactly these engines: pipeline steps doca_compress::compress|decompress, doca_sha, doca_ec::create|recover|update and doca_aes_gcm::encrypt|decrypt, tuned by attributes such as doca_compress.algorithm, doca_sha.algorithm and doca_ec.data_block_count.[8] The guide explains why the data provider matters: doca_dma and doca_sha “would execute in constant time regardless of the input data”, whereas doca_compress “would be faster with data with more duplication and slower for truly random data”.[8] Sweep support (--sweep core-count,1,8,*2), --csv-output-file (one line per sweep iteration) and --record-cpu-usage turn a run into the input M6.5 needs.[8] Note the table caveat that “Input remote memory is not supported for lz4 decompression” in Bench.[8]

Worked → faded → problem: decompress a zlib stream on BlueField-3
  1. Make a framed deflate input on any Linux box: python3 -c "import zlib,sys; sys.stdout.buffer.write(zlib.compress(open('in.txt','rb').read()))" > in.zz. Reasoning: zlib.compress writes a zlib header and Adler trailer, which is what -wf/--with-frame expects (“compatible with default zlib settings”).[2]
  2. Build: cd /opt/mellanox/doca/samples/doca_compress/decompress_deflate && meson /tmp/b_dd && ninja -C /tmp/b_dd.[2]
  3. Run on the Arm side against a BF-3 device: /tmp/b_dd/doca_decompress_deflate -p 03:00.0 -f in.zz -o out.txt -wf -c. Expected: the sample logs completion, writes out.txt, and prints the Adler and CRC checksums (_get_adler_cs, _get_crc_cs).[1][15]
  4. Verify: cmp in.txt out.txt returns nothing. If the sample reports the task type unsupported, run doca_caps for that device and confirm doca_compress_cap_task_decompress_deflate_is_supported.[1]
  5. Do not try doca_compress_deflate on this device: doca_compress_cap_task_compress_deflate_is_supported is false on BF-3 and the sample’s device filter will find nothing.[1]

After the whiteboard

How it ended

Slide six leaves with four rows instead of one: decompress yes on BlueField-3, deflate compress on BlueField-2 only, erasure coding and AES-GCM within the documented limits, SHA on the host.[1][3] The SE turns three spreadsheet rows green and one red, the first honest color in it all week. You take the archive’s sample corpus and a DOCA Bench command line agreed in the room.[8] “I would rather hand your CFO one number we measured on your data than four we copied off a slide.” At 02:10 their night operator calls: the ingest burst is hurting again, and the algorithm meant to fix it still will not load.

Lab

Pre-flight (read-only): on the Arm side doca_caps --list-devs, then for the PF: doca_caps -p <pci> | grep -iE "compress|sha|ec|aes". Record which task types report supported. Expected on BF-3: decompress deflate and LZ4 yes, compress deflate no, SHA absent, EC and AES-GCM yes.[1][3][4][6]

  1. Run the Worked block end to end on the Arm side; keep in.txt, in.zz, out.txt and the log.
  2. Bench the decompress engine by core count with real data: sudo /opt/mellanox/doca/tools/doca_bench --device <pci> --pipeline-steps doca_compress::decompress --data-provider file --data-provider-input-file in.zz --mode throughput --run-limit-seconds 10 --sweep core-count,1,8,*2 --csv-output-file /tmp/decompress_sweep.csv --record-cpu-usage. Expected: one CSV row per core count. If it exits with a device-capability error, re-check pre-flight.[8]
  3. Attempt --pipeline-steps doca_sha --attribute doca_sha.algorithm=sha256 on the same device and record the exact failure text; that is your evidence line for the customer.[8][3]
  4. Run erasure_coding_recover -p <pci> -i in.txt -o /tmp/ec -t 4 -r 2 -d 0,3 -b. Expected: blocks 0 and 3 recovered and the output file identical to the input (cmp). Rollback: rm -rf /tmp/ec. Nothing touches firmware.[14]
  5. Run aes_gcm_encrypt -p <pci> -f in.txt -o /tmp/enc.bin -k <64 hex chars> -i <24 hex chars> -t 16 then aes_gcm_decrypt with the same key/IV/tag on /tmp/enc.bin. Expected: decrypted output equals in.txt. If decrypt fails with a tag error, the IV or tag size differed between runs.[13][6]
  6. Copy all CSVs to the workstation; M6.5 consumes them.

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

Explain to a Dell storage SE, in four sentences, which of compression, hashing, erasure coding and encryption a BlueField-3 can take off the host CPU in DOCA 3.5.0, and which it cannot.

13 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against DOCA 3.5.0 docs and doca-samples tag 3.5.0, 2026-09-06. Dates are when each page was fetched.

  1. DOCA Compress (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  2. doca-samples 3.5.0: samples/doca_compress/README.md · fetched 2026-09-06 · DOCA 3.5.0
  3. DOCA SHA (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  4. DOCA Erasure Coding (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  5. doca-samples 3.5.0: samples/doca_erasure_coding/README.md · fetched 2026-09-06 · DOCA 3.5.0
  6. DOCA AES-GCM (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  7. DOCA Libraries index with quality levels (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  8. DOCA Bench (3.5.0) · fetched 2026-09-06 · DOCA 3.5.0
  9. NVIDIA/skills doca-bench SKILL.md · fetched 2026-09-06
  10. NVIDIA BlueField-3 DPU datasheet (PDF) · fetched 2026-09-06
  11. doca-samples 3.5.0: samples/doca_sha/README.md · fetched 2026-09-06 · DOCA 3.5.0
  12. doca-samples 3.5.0: samples/doca_aes_gcm/README.md · fetched 2026-09-06 · DOCA 3.5.0
  13. doca-samples 3.5.0: aes_gcm_common.c · fetched 2026-09-06 · DOCA 3.5.0
  14. doca-samples 3.5.0: erasure_coding_recover_main.c · fetched 2026-09-06 · DOCA 3.5.0
  15. doca-samples 3.5.0: compress_common.c · fetched 2026-09-06 · DOCA 3.5.0
  16. doca-samples 3.5.0: decompress_deflate_sample.c · fetched 2026-09-06 · DOCA 3.5.0
  17. doca-samples 3.5.0: decompress_lz4_stream_sample.c · fetched 2026-09-06 · DOCA 3.5.0
  18. doca-samples 3.5.0: sha_create_main.c · fetched 2026-09-06 · DOCA 3.5.0

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.