NCP-AIN prep: blueprint, drills and honest gaps
S5·E4The question before the account review · Hotel lobby across from the Dell campus, 07:20
Builds on: Proving the fabric: perftest and nccl-tests, Escalation bridge call: five InfiniBand tickets
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Recall the NCP-AIN logistics and the six weighted domains from the official certification page.
- Map the InfiniBand Networking and Troubleshooting Tools domains onto specific lessons in this course.
- Judge which exam objectives this course covers fully which it covers partially and which it does not touch.
- Rehearse the exam's named command list against the question each command actually answers.
- Produce a dated ninety-day ramp plan for the Dell OEM account that routes every gap to a named source.
Episode 4 - The question before the account review
Two coffees, hot for once, and ninety minutes before the account review. Your manager asks the question the room will ask: are you certified on this? Nobody is being pedantic - after a bridge call that ended at 03:00 and a fabric that came back, the account team wants to know which conversations route to you and which need an NVIDIA specialist.
The honest answer starts with a correction. The InfiniBand-specific credential is gone; its page carries the notice that the exam has been retired and points at the new professional AI networking exam.[2] The one a 2026 FAE books is NCP-AIN: 120 minutes, 70 to 75 questions, six weighted domains.[1] Two of them are your day job - InfiniBand Networking at 30 percent and Troubleshooting Tools at 20 percent, and the second names its command surface out loud, including ibstat, ibdiagnet, iblinkinfo, ibping, ibnodes, ib_write_bw and ib_write_lat.[1] That is the list the last three weeks of this account were fought with.
Weighted blueprints exist for exactly this conversation. The retired exam published five topic strings with no weights, so preparation was undirected and a claim of readiness meant nothing specific.[2] Weights let you say which half of an exam you own and which half belongs to somebody else - before a customer discovers the difference for you, in front of other people.
Claim the half you can prove; vagueness is the part customers remember.
So you spend the coffee building the map, objective by objective, gaps included.[1]
1Which exam, and what it actually asks
The InfiniBand-specific credential is gone. The NCP-IB page carries the notice “This certification exam has been retired. Please refer to our new Professional AI Networking Exam.”[2] Its published mechanics, while live, were 40 questions in a 90-minute limit for $220, English, Professional level, valid two years, with the prerequisite “A thorough understanding of data center infrastructure and networking” and exactly five unweighted topic strings: Introduction to InfiniBand, InfiniBand architecture, Fabric management, InfiniBand drivers, and InfiniBand utilities and tools.[2] Those five remain a good curriculum spine - this course is organised roughly along them - but they are not an exam you can book.
The exam a 2026 FAE books is NCP-AIN: 120 minutes, 70 to 75 questions, $400, English, valid two years from issuance, with the prerequisite “Two to three years of operational experience working in a data center with NVIDIA hardware solutions.”[1] Six weighted domains:
| Domain | Weight |
|---|---|
| AI Data Center Design and Optimization | 5% |
| NVIDIA Spectrum Networking | 30% |
| NVIDIA InfiniBand Networking | 30% |
| Kubernetes Integration | 5% |
| Troubleshooting Tools | 20% |
| Automation and Configuration | 10% |
The InfiniBand Networking domain’s objectives are initial configuration and provisioning with high availability, configuring partition keys for multi-tenancy, configuring QoS and adaptive routing, and using UFM for monitoring.[1] The Troubleshooting Tools domain names its command surface directly, including ib_write_lat, ib_write_bw, ibping, ibstat, ibdiagnet, ibnodes and iblinkinfo, alongside verifying low-latency interconnects and UFM system health.[1]
Add the two weights that matter to you: 30 plus 20 is half the exam, and both halves are the day job on a Dell OEM InfiniBand account.
2The 30% this course owns, objective by objective
Each InfiniBand Networking objective maps onto lessons you can re-run, not onto topics you can nod at.
| Objective | Where it lives |
|---|---|
| Perform initial configuration and provisioning with HA | infiniband-m2-01-opensm-bringup and infiniband-m2-02-sm-placement-ha |
| Configure partition keys for multi-tenancy | infiniband-m2-03-partitions-pkeys, with infiniband-m2-05-keys-security for hardening |
| Configure QoS and adaptive routing | infiniband-m2-04-qos-sl-vl and infiniband-m3-02-adaptive-routing |
| Use UFM for monitoring | infiniband-m2-02-sm-placement-ha and infiniband-m4-03-ibdiagnet, partially |
Two of those rows carry an honest asterisk, and you should know which before you book.
High availability is thin. The UFM HA documentation that was retrievable describes the older design - DRBD “synchronizes a replicated partition between the two servers”, Heartbeat “is responsible for starting UFM on master node and stopping it on the slave”, and a virtual IP that moves on failover, with readiness gates on management-network connectivity, DRBD state, disk space and cluster membership before a second failover is survivable.[13] The newer pacemaker-driven ufm_ha_cluster tooling could not be confirmed from a fetched page, so treat any subcommand list you see as unverified until you run it. What is solid is the SM side of HA: election by priority then GUID, with opensm’s -p running “from 0 (default and lowest priority) to 15 (highest)”, and SUBNET UP in /var/log/opensm.log as the proof that a new master actually configured the fabric.[11]
UFM monitoring is partial. What is documented and verifiable here is the System Health report: “create reports that run a series of checks on the UFM server”, covering UFM Configuration (release, license type and expiry, functionality level, device limits, running mode), UFM Processing (the status of OpenSM, ibpm, ModelMain, httpd and MySql), Memory, CPU, Disk and Fabric Interface.[9] That is server health. A UFM telemetry, REST or GUI walkthrough is not in this course’s sources, so if the exam asks about UFM dashboards rather than health checks, this course has not taught you that.
One more thing this domain rewards: knowing which defaults differ by who runs the SM. Upstream opensm ships sm_priority at 0 and QoS “disabled by default”, while UFM’s subnet-manager defaults table ships different values for sm_priority, qos and scatter_ports.[11][12] Quote them as a pair or read the running configuration; a single number is wrong half the time.
3Drilling the 20%: the exam's own command list
The Troubleshooting Tools domain is 20% and it names commands rather than concepts, which makes it the most drillable part of the exam. For each command the useful unit of recall is three-part: which question it answers, which flag makes it answer that, and which file or field holds the evidence.
ibstat “displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state” and is a binary rather than a script, with -l to list devices and -p to show port GUIDs.[3] ibnodes sits in the node-information group of infiniband-diags alongside ibswitches and ibhosts.[10] iblinkinfo “reports link info for each port in an IB fabric, node by node”, with -d for nodes having a Down port, -l for one line per link, and an exit status of 1 when check mode finds inconsistencies - which makes it usable directly in a health cron.[4] ibping “Uses vendor mads to validate connectivity between InfiniBand nodes”, run as sudo ibping -S on the remote node then sudo ibping <lid> locally.[10][8]
ibdiagnet is the one that produces evidence rather than an opinion. Its files are the vocabulary: ibdiagnet2.pkey for partition tables, ibdiagnet2.sm for subnet managers, ibdiagnet2.net_dump_ext for “Extended fabric link dump with FEC, BER, and phy data”, ibdiagnet2.nodes_info for firmware versions, ibdiagnet2.rails for the rail-optimization check.[5] Flags worth muscle memory: -r/--routing for the fabric-quality report, -P/--counter for counter thresholds, --get_phy_info for PHY data, and -f/--load_from_file to analyse a capture offline.[6]
ib_write_bw and ib_write_lat complete the list, and their two exam-relevant facts are the ones from module 5 lesson 1: a subnet manager must be running before any InfiniBand benchmark, and the same options must be passed to both server and client.[7]
One host, 60 seconds: you need LID, SMLID, port state, link width active and physical state straight from the local IB driver — from a binary, not a shell script.
Ten prompts drawn from all three rounds, in exam order.
Answer, then the panel gives the man-page wording, a working invocation, and where the evidence lands. Nothing here is trivia: every prompt is a sentence a customer has actually said.
ib_write_latib_write_bwibpingibstatibdiagnetibnodesiblinkinfo4The 45% this course does not teach
Being explicit about the gap is the point. No lesson here covers these, because the research corpus behind this course contains no material for them, and inventing coverage would be worse than naming a gap.
NVIDIA Spectrum Networking, 30%. RoCE switch configuration, QoS with ECN and PFC, BGP-EVPN multi-tenancy, NVIDIA Air, telemetry and WJH, NetQ, installing DOCA, and SuperNIC configuration.[1] This is the largest single block and it belongs to the sibling roce, spectrumx and doca courses.
Kubernetes Integration, 5%. Deploying and verifying the NVIDIA Network Operator.[1] Not here.
Automation and Configuration, 10%. Configuration templates and Ansible playbooks for network setup.[1] There is no InfiniBand automation material in this course’s sources at all.
Inside the Troubleshooting domain there are two more items that are not InfiniBand: cl-resource-query is a Cumulus tool and WJH is a Spectrum feature, both listed under the same 20% domain.[1] The UFM diagnostic-utility surface is partially covered - the tool list and its one-line descriptions are documented, and the ibdiagnet-side detail is the part this course teaches.[14]
So the arithmetic to say out loud: this course covers the 30% InfiniBand domain with two partial objectives, most of the 20% Troubleshooting domain, and part of the 5% design domain. That is roughly half the exam, precisely. The rest needs the sibling courses, and no amount of InfiniBand study substitutes for it.
5Ninety days on the Dell OEM account
A plan for this exam should serve the job first, because the job is what makes the material stick. Four blocks, dated, each with a deliverable rather than a reading.
Weeks 1 to 3 - own the half you have. Re-run the labs behind the four InfiniBand objectives and the troubleshooting command list on real Dell-lab hardware, and produce two artefacts: the timed drill score by round, and a one-page cheat sheet of defaults that differ between upstream opensm and UFM - sm_priority, qos, scatter_ports, the sweep interval, and the embedded-SM node ceiling, which the switch manual states as “up to 2,000 nodes” while the MLNX-OS page says “2048 nodes on x86 based systems”.[11][12][16][15]
Weeks 4 to 6 - close the two thin objectives. Get in front of a real UFM instance, run a System Health report and record the check groups it actually returns, then find a current HA source and confirm the cluster tooling rather than trusting a summary.[9][13] These are the two places where this course hands you a gap inside its own domain, and they are the two places a customer will ask about first.
Weeks 7 to 10 - the sibling material. Spectrum first because it is 30%, then Kubernetes and Automation.[1] The deliverable is the sibling course’s own labs, not a summary of them.
Weeks 11 to 12 - rehearsal and re-fetch. Re-fetch the certification page and diff the weights and objective wording against the table in segment 1; blueprints move and yours is dated.[1] Then run the full command list end to end in one session on hardware and record the real output of each into the cheat sheet.
The Dell-account layer runs alongside all of it. Every escalation you handle becomes a flashcard, every design review becomes a risks section, and the artefacts you build for customers - the evidence bundle request, the SDR KB reply, the perftest proof pack - are the same objects the exam is testing.
Brief — Back-to-back calls: a VMware farm (200× R760), an AI training pod (64× XE9780 on Spectrum-X) and a storage-heavy multi-tenant inference cluster. Each has objections.
Dell account team + three end customers: VMware farm: "200 R760s on vSphere 8; NSX eats about 20% of our cores. Should we go DPU?"
Starting point: modules 1 to 4 done, Dell-lab BlueField-3 and ConnectX hosts, containerlab with Dell Enterprise SONiC, no NVIDIA switch and no InfiniBand fabric at home.
Week 0 - re-fetch and diff. Pull the NCP-AIN page and check six items: duration, question count, price, validity, prerequisite wording and the six weights.[1] Deliverable: a one-page diff against the table in segment 1, even if it is empty.
Weeks 1 to 3 - the owned half. Re-run the module 2 labs for the four InfiniBand objectives, then the module 4 and module 5 labs for the command list. Timed drill of 40 items across ibstat, ibnodes, iblinkinfo, ibping, ibdiagnet, ib_write_lat and ib_write_bw, each requiring the command, the decisive flag and the file or field.[1][5][6] Deliverable: the score by round plus the defaults cheat sheet.[11][12]
Weeks 4 to 6 - the two thin objectives. UFM System Health run on a real instance with the check groups recorded, and a current HA source for the cluster tooling.[9][13] Deliverable: two pages, each citing a page you opened yourself.
Weeks 7 to 10 - sibling courses for Spectrum, Kubernetes and Automation, in weight order.[1] Deliverable: their labs, run.
Weeks 11 to 12 - full command list end to end in one session on hardware, real output pasted into the cheat sheet, then the re-fetch repeated before booking.[1]
Build the plan for a colleague with the opposite bench: they have a partner InfiniBand lab with UFM two days a month, but no Dell lab and no BlueField.
- Which two weeks move to the front and why? ____
- Which objective becomes their easiest rather than their thinnest? ____
- Which part of the command list can they not rehearse and what do they substitute? ____
- What is the deliverable for their scarcest resource - the two partner-lab days? ____
- What goes in their week 0 and why is it identical to yours? ____
- Which of the four blocks would you cut if they had only six weeks and why? ____
Produce your own plan.
Deliver: (a) a dated week-0 diff of the current certification page against the table in segment 1; (b) an objective-by-objective table for all six domains, marking each objective as covered here with a lesson id, partially covered with the reason, or a gap routed to a named sibling course; (c) a twelve-week plan with one named deliverable per week; (d) a defaults cheat sheet listing every value that differs between upstream opensm and UFM, with both numbers and the page each came from; (e) one paragraph you would actually say to a Dell account manager about what you can support today.
Acceptance: every “covered” row names a lesson id you can re-run; at least two rows are marked partial with the specific missing source; no week has “read” as its deliverable; the cheat sheet has both numbers for every entry; and the paragraph in (e) claims half the exam precisely rather than all of it vaguely.
Case closed - the pod, and the honest half
At the review you hand over one page: the two domains you own with the lessons behind each objective, and the two objectives inside your own domain that are still thin - the HA tooling, and UFM monitoring, where what is documented is the System Health report and the subnet-manager defaults rather than a telemetry walkthrough.[13][9][12] The other 45 percent belongs to the Spectrum, Kubernetes and automation material.[1] The pod passed acceptance on Friday, the second scalable unit is cabled to the full shape, and the operator has already labelled its spares. The network lead filled the notebook. Nobody asks for a logo. What you say to the account executive: “Route the InfiniBand and troubleshooting calls to me today, and give me ninety days for the rest.”
Lab
One session on the Dell lab, read-only throughout: every command below is a query and none changes fabric, firmware or configuration state. Pre-flight: confirm a subnet manager is running and record sminfo before you start, because two of these commands are meaningless without one.
ibstatandibstat -l. Expected: port state, physical state, LID, SMLID and link width active per device.[3] Paste the real output into your cheat sheet - not a paraphrase.ibnodes. Expected: the node list the fabric can see.[10] If it returns nothing, you have an SM problem, not a tooling problem.iblinkinfotheniblinkinfo -d. Expected: negotiated width and speed per link, and an empty result from-don a healthy fabric.[4] Note the exit-status rule for later use in a health cron.[4]sudo ibping -Son the remote node thensudo ibping <lid>locally. Expected: ping-like output.[10][8] Record the LID you used and where you read it from.ib_write_latthenib_write_bw, identical flags on both sides. Expected: a latency figure you can defend as half a round trip and a bandwidth figure you can compare against line rate.[7] Record the exact command line beside each number.ibdiagnetinto a fresh output directory, then openibdiagnet2.pkey,ibdiagnet2.sm,ibdiagnet2.net_dump_extandibdiagnet2.nodes_infoand write one line each on what that file answered.[6][5] Note where your build actually wrote the directory - the documentation is inconsistent about the default path, so record whatibdiagnetprinted rather than what you expected. Optional, in a customer or NVIDIA lab with UFM: run a System Health report and record the check groups it returns against the documented list.[9]
The timed drill and the plan. Nothing here touches hardware.
- Re-fetch the NCP-AIN certification page and check six facts: 120 minutes, 70 to 75 questions, $400, two-year validity, the prerequisite wording, and the six domain weights.[1] Expected: agreement with segment 1. If anything differs, that diff is your first study artifact.
- Re-fetch the NCP-IB page and confirm the retirement notice is still present.[2] Expected: the retirement sentence and a pointer to the AI Networking exam. This is the answer you will give a Dell colleague, so keep it current.
- Run the timed drill: 40 items across the exam’s named command list, each requiring the command, the decisive flag, and the file or field that holds the answer.[1][5][6] Expected: your weakest round is the file round; most people know the commands and not the evidence.
- Self-score by domain rather than by item. Expected: a percentage for the InfiniBand objectives and one for the Troubleshooting objectives, and nothing for the other four domains - which is the honest picture.
- Write the study plan that routes the uncovered 45% to the sibling courses in weight order: Spectrum, then Automation, then Kubernetes.[1]
- Build the one-page defaults cheat sheet -
sm_priority,qos,scatter_ports, sweep interval, embedded-SM node ceiling - with both the upstream and UFM values and the page each came from.[11][12][16][15] Expected: every line has two numbers. A line with one number is a line you will get wrong in front of a customer.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to your manager, in five sentences, what this course prepares you to sit and what it does not.
Sources
Facts in this lesson were checked against NCP-AIN and NCP-IB certification pages re-fetched 2026-09-07; UFM 6.25.1 Health Tab, UFM 6.24.1 SM default properties, opensm(8) and infiniband-diags man pages as fetched 2026-09-07. Dates are when each page was fetched.
- NVIDIA-Certified Professional: AI Networking (NCP-AIN) certification page · fetched 2026-09-07
- NVIDIA-Certified Professional: InfiniBand (NCP-IB) certification page - retired · fetched 2026-09-07
- ibstat(8) man page source - rdma-core master · fetched 2026-09-07
- iblinkinfo(8) man page source - rdma-core master · fetched 2026-09-07
- ibdiagnet Dump Files - IBUtils2 Utility Documentation · fetched 2026-09-07
- ibdiagnet Basic Commands - IBUtils2 Utility Documentation · fetched 2026-09-07
- perftest README - linux-rdma/perftest master · fetched 2026-09-07
- Networking Troubleshooting - NCCL user guide · fetched 2026-09-07
- UFM Health Tab - UFM Enterprise User Manual 6.25.1 · fetched 2026-09-07
- InfiniBand Fabric Utilities - MLNX_OFED 23.07-0.5.1.2 · fetched 2026-09-07
- opensm(8) man page source - linux-rdma/opensm master · fetched 2026-09-07
- UFM Subnet Manager Default Properties - UFM Enterprise 6.24.1 · fetched 2026-09-07
- High Availability - NVIDIA UFM Enterprise User Manual 6.11.2 · fetched 2026-09-07
- Diagnostic Utilities - UFM Enterprise User Manual 6.24.1 · fetched 2026-09-07
- Subnet Manager - NVIDIA MLNX-OS User Manual v3.12.6200 LTS · fetched 2026-09-07
- QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual: Introduction · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- NCP-AIN: what this course covers and what it does notRoCE course · Same ground: logistics, NCP-AIN and blueprint
- NCP-AIN: mapping the blueprint and closing the gapsRA course · Same ground: ncp-ib, gaps and tools
- NCP-AIN: blueprint map, gaps and a study planSpectrum-X course · Same ground: blueprint, gaps and NCP-AIN