Keys, and locking a fabric down without locking yourself out
S2·E5The hardening plan that would blind the fabric it hardens · Change-board room with no windows, Friday afternoon, acceptance on Monday
Builds on: Partitions and PKeys: the most common mistake in IB
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish the IBA and NVIDIA key families by what each one protects and at what granularity.
- Evaluate an M_Key protection level against the diagnostics it disables and justify the level you recommend.
- Sequence a fabric hardening change so that every diagnostic you depend on keeps working or is knowingly given up.
- Judge worst-case recovery time from the documented lease-period formula before agreeing to a key rollout.
Episode 5 — The hardening plan that would blind the fabric it hardens
The board wants a signature before it goes home. The plan on the table runs Saturday at 22:00: every key family enabled, the SA key randomized, the SMP blocker pushed to every host in one pass. Procurement is in the room because Monday’s acceptance releases the expansion order. The network lead has not spoken for twenty minutes, and her notebook is down to its last clean page.
You start where this conversation usually goes wrong, because half the room thinks it is buying encryption. It is not: “IB keys are not involved in cryptographic operations but rather serve as access tokens.”[1] They exist because InfiniBand management traffic - LID assignment, table writes, diagnostic queries, and the node-to-node messaging behind adaptive routing and pFRN - crosses the fabric with no authentication of its own, and a key per class is how a subnet says who may ask and who may change.[1]
Then you ask what the plan does not answer: after Saturday, which host is key-aware? At M_Key protection level 2 the adapters ignore management queries that carry no valid key, so every un-keyed ibdiagnet and smpquery goes silent, and with the SA key randomized every saquery needs --smkey.[3][6] As written, the plan removes the room’s ability to see the fabric it is about to change, two days before it is handed over. You put an ordered version on the screen.
Harden bottom-up, one class at a time, and never past the last diagnostic you still need.
Segment 1 sorts the key families by what each one protects.
1Keys are access tokens, not crypto
Start with the sentence that prevents the wrong conversation: “IB keys are not involved in cryptographic operations but rather serve as access tokens.”[1] Nothing here encrypts anything. A key decides whether a device will answer a management question or accept a configuration change, and that is the entire model.
IBA defines four families: “Management keys (M_Keys), Partition keys (P_Keys), Memory keys (L Keys and R Keys), Communication Queue keys (Q Keys).”[1] P_Keys you configured in the previous lesson. The one that runs the control plane is the M_Key: it “administers the control of a master subnet manager (SM) and restricts the port from any configuration or setting changes… There is one M_Key for a switch”, and on mismatch “the packet is dropped, and an optional ‘Bad M_Key’ trap is sent to the SM.”[1]
On top of IBA, NVIDIA defines a set of MAD class keys, and their granularity is what decides the blast radius of a mistake:[1]
| Key | Owner | Granularity |
|---|---|---|
| M_Key | SM | Per port (port 0) |
| SA_Key | SA | Per subnet, per tenant |
| VS_Key | SM / Tools | Per port (port 0) |
| C_Key | SM (C_Mgr) | Per port (port 0) |
| N2N_Key | SM (C_Mgr) | Per link (port2port) |
| AM_Key | AM | Per subnet |
| CC_Key | SM | Per port (port 0) |
Two of those matter to you daily. VS_Key is the tooling key — “The VS Key is used by Nvidia IB diagnostic tools, such as ibdiagnet, to monitor and analyze various network attributes.”[1] C_Key and N2N_Key “secure node management communication for the IB infrastructure features that propagate status and messages between nodes, without directly involving the SM. This is used in adaptive routing algorithms and Proactive Fault Routing Notification (pFRN).”[1] Turning those on without arming the corresponding tools is how a hardening project takes out the fabric’s own observability.
Protect-bit semantics differ between M_Key and the rest. M_Key uses levels 0 to 3; for the others, “0: Full protection is provided. Class manager is allowed to read the key. 1: Full protection is provided. Class manager is not allowed to read the key.”[1] And the lease model is shared: “When a device receives a MAD with an incorrect key, it reports to the Subnet Manager and waits for the key_lease_period seconds for a response… If set to 0, the lease period is infinite.”[1]
- Profile doca-all is "other profiles" in the matrix → Level 2. Every component is in cycle 25 (Oct 2025 → Jul 2026 (3.2 → 3.5)) → supported until the next October GA.
- This is exactly the Spectrum-X validated stack v2.3.1 combination (Sep 2026).
- ⚠ Any FW or mode change on PowerEdge needs a full power cycle, not a warm reboot (Dell KB 000300192; NVIDIA modes page).
Matrix (policy): doca-ofed ↔ FW/BF-FW-Bundle = L1 · doca-ofed ↔ BF-Bundle = L1 · other profiles ↔ FW or BF-Bundle = L2 · DOCA-DPU ↔ BF-FW-Bundle = L2 · DOCA Services ↔ BF-Bundle/FW = L2. source ↗
⚠ = not confirmed on a fetched primary source (hover for why). Facts as of DOCA 3.5.0 (Sep 2026). Selections are saved.
2M_Key protection levels, and the ticket they generate
Upstream OpenSM implements m_key (IBA 14.2.4), m_key_protection_level (14.2.4.1) and m_key_lease_period (14.2.4.2). “OpenSM will configure all ports with the MKey specified by m_key, defaulting to a value of 0. A m_key value of 0 disables MKey protection on the subnet.”[3] UFM ships m_key = 0x0000000000000000 with m_key_per_port = FALSE and m_key_lease_period = 60 — that is, off by default in both worlds.[4]
The four levels, verbatim: 0 = “Queries return valid data, including MKey. Configuration changes are not allowed unless the request contains a valid MKey”; 1 = “Like level 0, but the MKey is set to 0 … in queries”; 2 = “Neither queries nor configuration changes are allowed, unless the request contains a valid MKey”; 3 = “Identical to 2. Maintained for backwards compatibility.” The default is level 0.[3]
Level 2 is the one that generates the ticket, and the man page warns about it in advance: “Setting a MKey may have a detrimental effect on diagnostic software run on the subnet… This is particularly true at protection level 2, where CAs will ignore queries for management information that do not contain the proper MKey.”[3] The infiniband-diags package carries the matching caveat from the other side: many of its tools “rely on the use of SMPs via QP0”, a mode of operation the package itself notes “is not technically in compliance with the InfiniBand specification”, and “For this reason many of these tools may require the use of an MKey”.[5]
So “we enabled M_Keys and ibdiagnet went blind” is documented behaviour, not a regression. The recovery flags exist and you should know them before you need them: saquery --smkey <val>, whose “Default (when not specified here or in <IBDIAG_CONFIG_PATH>/ibdiag.conf) is to use SM_Key == 0 (or ‘untrusted’)”, and ibdiagnet’s key flags --m_key, --vs_key, --cc_key, --am_key, --m2n_key and --security_keys <path>.[6][7]
Two more behaviours to plan around. Recovery time has a formula: “the total time to recover the subnet may be as long as the mkey lease period multiplied by the maximum number of hops between the SM and an endpoint, plus one.”[3] And OpenSM protects you from one bad combination on its own: “If OpenSM detects at startup that your sweep interval is greater than your MKey lease period, it will reset the lease period to be greater than the sweep interval.”[3] That self-correction is a reason to read the effective value back rather than trusting what you wrote.
One tool-level restriction worth remembering when you are trying to repair a keyed port by hand: in ibportstate, “mkey, mkeylease, and mkeyprot are only allowed on CAs, routers, or switch port 0 (An error is generated if attempted on external switch ports).”[9]
3NVIDIA's hardening order, and what each step costs
NVIDIA publishes the sequence with parameter names, and the useful way to read it is as a list of things that stop working.[2]
- Partitions. “Define all end nodes as limited members on the default PKEY” with
default_membership = limited.[2] Cost: any tool that assumed full membership on 0x7fff between end nodes stops — which is the whole point, and is why lesson 3 comes first. - SM MAD security.
m_key_per_port,global_m_key_seed = <random value>,m_key_lease_period 60,m_key_protection_level 2.[2] Cost: at level 2 every un-keyed diagnostic goes silent.[3] Restore withibdiagnet --m_keyand anibdiag.confentry.[7] - Tools MAD security.
vs_key_enable 2withvs_key_lease_period 60.[2] Cost: NVIDIA-specific diagnostic collection needs--vs_key.[1][7] - Node-to-node and congestion control.
n2n_key_enable 2,cc_key_enable 2, each with a 60-second lease.[2] Cost: these are the keys behind adaptive routing and pFRN messaging, so a mismatch here is a routing-behaviour problem, not a query problem.[1] - SA security.
randomize_sa_key = trueandsm_key = <random value>.[2] Cost: everysaqueryneeds--smkey.[6] - SM allowlist.
allowed_sm_guids <GUID A>,<GUID B>— orallowed_sm_guids 0to permit no other SM.[2] UFM ships it(null), disabled.[4] Cost: a standby SM you forgot to list will never take over. - Rotation.
periodic_key_update <minutes>— “Recommended value is 24 hours (1440/0x5A0 minutes). Minimum value is 10 minutes (any value below it will be addressed as 10 minutes)… This configuration will affect any per-class key that is enabled.”[2] - SMP blocker, last.
sudo mlxconfig -d <device> q SM_DISABLEto read,s SM_DISABLE=1to block,s SM_DISABLE=0to restore — and “SMP blocker configuration should be set for every non-SM host.”[2] For bare-metal tenants with root, “the Secure Host feature should be enabled” so the tenant cannot override the SMP firewall TLV.[2] Cost: it is an adapter firmware change, so it needs a reboot orflint -d <device> swresetto take effect, and setting it on the SM host itself is the way to lose the fabric.[8]
Two useful details. Setting a seed to 0xFFFFFFFFFFFFFFFF tells the SM to generate the key randomly.[2] And beyond keys there is static topology enforcement: topo_spec pins Switch GUID, Port, Neighbor GUID, Neighbor Port, Neighbor Type and Link State, the SM enforces it every sweep, and a discovered node absent from the file is reported and left unconfigured.[1] UFM exposes it as topo_config_file with topo_config_enabled = FALSE by default.[4]
Firmware: Dell DUP vs NVIDIA mlxfwmanager / bf-fwbundle — which one wins?
Case: a CSP customer with a three-tier fabric, one UFM host, five hops maximum from SM to endpoint, is onboarding a second tenant next month. They ask for “full security” in one maintenance window and have sent a change request containing m_key_protection_level 2, randomize_sa_key = true and SM_DISABLE=1 fabric-wide.
- Compute the number they did not include. Lease period 60 seconds, five hops: worst-case subnet recovery is 60 × (5 + 1) = 360 seconds, six minutes.[3] Put it in the change request as the expected blast time for a key mistake.
- Identify what the request removes. At level 2, CAs ignore management queries without a valid MKey, so every un-keyed
ibdiagnetandsmpquerygoes silent; withsa_keyrandomized everysaqueryneeds--smkey.[3][6] Ask which host is key-aware. If the answer is “none yet”, the window as written ends with a fabric nobody can inspect. - Check the allowlist consequence.
allowed_sm_guidsis(null)on stock UFM; if they set it, the standby UFM’s GUID must be in it or the HA pair cannot fail over after the change.[4][2][11] - Re-sequence into reversible steps, one verification each:
default_membership = limitedplus per-tenant partitions; verify withibdiagnet2.pkey.[2][10]m_key_per_port,global_m_key_seed, lease 60, protection level 0; verify every existing diagnostic still works untouched.[2][3]- raise to
m_key_protection_level 2; verifyibdiagnet --m_key <key>works and plainibdiagnetdoes not — if plain still works, the level did not apply.[3][7] vs_key_enable 2, thenn2n_key_enable 2andcc_key_enable 2, verifying the corresponding collection after each.[2][7]randomize_sa_keyand a randomsm_key; verifysaquery --smkey.[2][6]allowed_sm_guidswith both UFM GUIDs; verify a deliberate failover.[2][11]periodic_key_update 1440.[2]SM_DISABLE=1on non-SM hosts only, staged in batches with reboots; verify from a blocked host that it can no longer answer SMPs and from the SM that discovery is unchanged.[2][8]
- Name the two things you will not do in the same window: the SMP blocker on all hosts at once, and any
max_op_vlsor partition-enforcement change — one change class per window so an unexpected symptom has one plausible cause.[2] - Write the rollback per step, not for the whole window:
m_key 0disables MKey protection on the subnet; each*_key_enablereturns to its previous value;SM_DISABLE=0plus a reboot restores an adapter.[3][2][8] - Deliver the recommendation in one paragraph: the customer gets everything they asked for, in two windows instead of one, with a key-aware diagnostic host provisioned first and a six-minute worst-case recovery figure they have signed.
- Recovery arithmetic: lease ____ seconds times (____ hops + 1) equals ____ seconds.
- What level 2 removes: CAs ignore ____ without a valid MKey. What
randomize_sa_keyremoves: every____needs--smkey. allowed_sm_guidsdefault is ____ ; if you set it, the ____ SM’s GUID must be listed or failover stops working.- Re-sequence: partitions with
default_membership = ____→ M_Key at level ____ → raise to level ____ →vs_key_enable ____→n2n_key_enableandcc_key_enable→randomize_sa_keyand randomsm_key→allowed_sm_guids→periodic_key_update ____→SM_DISABLE=1on ____ hosts. - After each key step, re-run the diagnostic that step was supposed to ____ . If it still works, the step ____ .
- Rollbacks:
m_key ____disables MKey protection subnet wide;SM_DISABLE=____plus a ____ restores an adapter. - One-paragraph recommendation naming the number the customer signs.
Three weeks after a hardening rollout you are on a bridge call. Symptoms: ibdiagnet from the operations host returns almost nothing; saquery returns no records; UFM’s own topology view is complete and healthy; one leaf switch replaced yesterday shows as discovered but unconfigured; and the customer reports Bad M_Key traps in the SM log. Nobody can find the engineer who ran the rollout.
Produce the diagnosis and the recovery plan. Acceptance criteria: (a) explain which single configuration change accounts for the first two symptoms and why UFM is unaffected; (b) explain the replaced-switch symptom, naming the specific mechanism that leaves a discovered node unconfigured and how you would confirm it in one command or file; (c) explain what the Bad M_Key traps tell you and what they do not; (d) give the ordered recovery with the flag or parameter for each step and state which steps are safe on a live fabric and which need a window; (e) state the worst case you would quote for restoring a port whose key is genuinely lost, and the input you need from the customer to compute it.
Signed, in order - and the case closes
The plan goes out bottom-up, one class per step, each with its verification, the flag that keeps the diagnostic alive, and its own rollback: partitions first, then SM MAD security, tools, node-to-node, the SA key, and the SMP blocker last with a reboot in its rollback line.[2][8] ibdiagnet --m_key, ibdiagnet --vs_key and saquery --smkey keep every step observable.[7][6] At a 60-second lease and five hops, worst-case recovery is 360 seconds, and that number goes in the change request.[3]
Monday: one master you can name, four partitions that survive a reload, two lanes everybody agreed to, keys on, and a fabric you can still read. The co-op signs acceptance and procurement releases the expansion. The operator relabels the real SM host, correctly this time, and keeps the old label.
Lab
Goal: exercise the SMP blocker on one lab adapter and prove both directions. Step 4 is a firmware change requiring a reboot or a software reset; do it on a lab card only, never on the host running the subnet manager.
- Pre-flight, read-only, on both hosts:
Expected: the MST device path, the full current configuration saved,sudo mst start; sudo mst status sudo mlxconfig -d <device> q | tee /tmp/mlxconfig-before-$(hostname).txt sudo mlxconfig -d <device> q SM_DISABLE ibstat | grep -E "State:|Base lid" sminfoSM_DISABLEreported (normally 0 or False), active ports, and one SM master.[8] Write down which host runs the SM. That host is excluded from the rest of this lab. - Prove the fabric is visible before you change anything, read-only:
Expected: a normalibdiagnet smpquery PortInfo <lid> 1 | head saquery NR | headibdiagnetrun, a PortInfo block, and node records from the SA. Save all three outputs — they are your before state.[7][6] - Optional and instructive, customer lab only: set an
m_keyat protection level 0 first via the SM options file and repeat step 2. Expected: everything still works, because level 0 returns valid data including the MKey and restricts only configuration changes.[3] Rollback:m_key 0, which disables MKey protection on the subnet, then restart the SM.[3] - Mutating — enable the SMP blocker on the non-SM host. Rollback:
sudo mlxconfig -d <device> s SM_DISABLE=0followed by the same reset.
Expected: the query reflects the new value, and MFT tells you to reboot to load new configurations;sudo mlxconfig -d <device> s SM_DISABLE=1 sudo mlxconfig -d <device> q SM_DISABLE # apply: reboot, or sudo flint -d <device> swresetflint -d <device> swresetis the documented alternative.[2][8] - Verify the effect from both sides, read-only. From the SM host, re-run
ibdiagnetandsmpquery PortInfo <lid> 1against the blocked host’s LID and compare with step 2. From the blocked host, runibstatand confirm the port is stillActive— the data plane is untouched.[7] Expected:ibdiagnet,smpqueryandsaqueryrun FROM the blocked host now fail or return nothing, while from the SM host discovery is unchanged -ibdiagnetstill enumerates the node andsmpquery PortInfo <lid> 1still answers with the port Active and its LID. Record exactly which commands changed behaviour on which host; that list is your customer-facing evidence for what the blocker does. - Roll back and confirm:
sudo mlxconfig -d <device> s SM_DISABLE=0, reboot orflint -d <device> swreset, then re-run step 2 and diff against the saved outputs until they match.[2][8] - Full-configuration diff as the final check:
sudo mlxconfig -d <device> q | diff /tmp/mlxconfig-before-$(hostname).txt -. Expected: empty. A non-empty diff means something else changed during the lab and must be reconciled before the card goes back into use.[8] - Do not use
-ywithmlxconfigin any of these steps: the flag is documented formlxfwmanagerand not on the MFTmlxconfigpage this course could fetch, so treat it as unverified rather than assuming it means the same thing.[8]
Goal: produce the hardening runbook and the recovery arithmetic. All steps are documentation work; nothing touches hardware.
- Build the key table yourself with four columns: key, granularity, what it protects, and the diagnostic that stops working when it is enforced. Fill it only from this lesson’s sources.[1][2] Expected: seven rows, and at least three of them naming a specific command in the last column.
- Write the ordered hardening runbook as a change request, one step per line, each with: the parameter and value, the verification command, the expected result, and the rollback.[2] Expected: the SMP blocker is last and its rollback names a reboot.[8]
- For every step whose verification is a diagnostic, add the flag that restores it:
ibdiagnet --m_key,ibdiagnet --vs_key,saquery --smkey, and theibdiag.confdefault path.[7][6] Expected: no step in your runbook leaves a tool broken without a named remedy. - Compute worst-case recovery for three fabric shapes at a 60-second lease: two hops, five hops, and seven hops, using lease × maximum hops + 1.[3] Expected: 180, 360 and 480 seconds. Then answer the design question this raises: what would you change if the customer wanted worst case under two minutes, and what does that cost in exposure?
- Write the two-sentence answer you would give to “we enabled M_Keys and ibdiagnet went blind”, quoting the documented sentence rather than paraphrasing it.[3] Expected: no apology and no escalation - a citation and a flag.
- Walk the shipped troubleshooting tree’s firmware branch and note the shape of each node: check, command, expected, if-not. Then write your own four-node keys branch in the same shape, rooted at “diagnostics returned nothing after a security change”.[3][6]
- Record what this lesson does not cover, so you do not overstate it later:
topo_specstatic topology enforcement is described here from the security overview only, with UFM’stopo_config_enableddefaulting to FALSE, and no worked topology file was available.[1][4]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
A multi-tenant customer asks you to lock down their InfiniBand fabric before onboarding a second tenant. Explain in five sentences what you would turn on and, more importantly, what each step costs you in visibility.
Sources
Facts in this lesson were checked against NVIDIA InfiniBand Security practical-guidelines page re-fetched 2026-09-07 (hardening parameter list verbatim, SMP blocker mlxconfig commands, default_membership limited); Security in InfiniBand page re-fetched 2026-09-07 (key table and granularity); opensm(8) man page re-fetched 2026-09-07 (m_key_protection_level 0-3, lease recovery, diagnostics warning). Dates are when each page was fetched.
- Security in InfiniBand - NVIDIA InfiniBand Security Overview and Guidelines · fetched 2026-09-07
- NVIDIA InfiniBand Security: Practical Guidelines · fetched 2026-09-07
- opensm(8) man page source (linux-rdma/opensm master) · fetched 2026-09-07
- UFM Enterprise 6.24.1: UFM Subnet Manager Default Properties · fetched 2026-09-07
- infiniband-diags(8) overview man page (rdma-core master) · fetched 2026-09-07
- saquery(8) man page (rdma-core master) · fetched 2026-09-07
- ibdiagnet Basic Commands - IBUtils2 Utility Documentation · fetched 2026-09-07
- Using mlxconfig - NVIDIA Firmware Tools (MFT) v4.30.18 LTS · fetched 2026-09-07
- ibportstate(8) man page (rdma-core master) · fetched 2026-09-07
- ibdiagnet Dump Files - IBUtils2 Utility Documentation · fetched 2026-09-07
- Running UFM Server Software - UFM Enterprise 6.15.0 · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.