RoCE and RDMA for AI fabrics
0 of 27 cases closed
Foundations
Congestion
Measurement
Operations
FAE and cert
RoCE and RDMA for AI fabrics
Verbs, RoCEv2 encapsulation, PFC/ECN/DCQCN, perftest and NCCL, and the counters that end the argument.
RDMA from the verbs object model up: how RoCEv2 puts InfiniBand transport on UDP/4791, how the ToS→DSCP→priority→traffic-class chain decides whether your traffic lands in the lossless queue, how PFC, ECN/DCQCN and ZTR-RTT congestion control actually close the loop, and how to prove any of it with perftest, nccl-tests and sysfs counters. Ends with Dell OEM field scenarios and an NCP-AIN coverage map. Interactive diagrams, two-variant labs (Soft-RoCE in containerlab / NVIDIA Air, or BlueField-3 and ConnectX in the Dell lab), checkpoints.
Who it is for: FAEs and SEs validating RoCE fabrics on Dell PowerEdge + NVIDIA ConnectX/BlueField.
- Not started
- In progress
- Complete, case open
- Case closed
- Locked
- Checkpoint
- A case closes when the lesson’s retrieval check reaches 80%.
Explain the verbs object model, pick the right transport, and read a RoCEv2 packet and a GID table well enough to say which version and which priority a connection is actually using.
- 1.1The verbs object model: device, PD, QP, CQ, MRThe card that says InfiniBand on an Ethernet cableUnderstand25 min
- 1.2Transports and opcodes: what RC, UC and UD can actually doThe one requirement that changed the transportAnalyze30 min
- 1.3RoCEv1 vs RoCEv2: the encapsulationWorks in the rack, dies between racksAnalyze25 min
- 1.4GIDs: how a connection picks its RoCE versionThe benchmark crosses the subnet and the application does notApply30 min
- 1.5QP state machine, retries, and connection setupForty minutes, every timeAnalyze30 min
- 1.6Multipath RC: a new transport in the annex, not in the APIThe annex he cannot readAnalyze30 min
- Module 1 checkpoint14 questions · pass mark 80% · 12 min
Build and defend the whole marking-and-congestion chain — application ToS through host trust state to switch queue, PFC vs ECN vs ZTR-RTT — and say which one a customer's symptom actually points at.
- 2.1ToS, DSCP, PCP, SL: where marking gets lostThe capture that ends the argumentAnalyze30 min
- 2.2Host QoS with mlnx_qos: trust state, prio2tc, ETSTwo identical nodes that are not identicalApply30 min
- 2.3PFC, lossless and the case for lossyThe architect who wants more pauseEvaluate30 min
- 2.4ECN, CNP and DCQCN: closing the loop02:10, and nothing is slowing downAnalyze35 min
- 2.5Zero Touch RoCE, ZTR-RTT CC and DOCA PCCThe line item they refuse to buyEvaluate35 min
- 2.6Packet spraying and the reordering it hands to the NICThe sixth word on the labelEvaluate40 min
- Module 2 checkpoint14 questions · pass mark 80% · 12 min
Produce and defend a number: run perftest and nccl-tests correctly, prove GPUDirect is actually on, and translate busbw into a statement about which piece of hardware is the bottleneck.
- 3.1What ib_write_bw actually measuresThe 180 that was never 180Apply30 min
- 3.2NUMA, data validation, and doca_perftestThe first word on the lineAnalyze30 min
- 3.3GPUDirect RDMA: proving it is actually onWhichever slot was easiest to reachAnalyze35 min
- 3.4NCCL environment variables on a RoCE fabricThe design was correct; the traffic never entered itApply35 min
- 3.5algbw, busbw, and reading a nccl-tests resultThe 1.75x he called marketingAnalyze30 min
- Module 3 checkpoint14 questions · pass mark 80% · 12 min
Run a RoCE fabric in production: know where every counter lives, work a symptom to a cause in a fixed order, configure the switch half on Cumulus and Dell SONiC, and plumb RDMA into Kubernetes.
- 4.1The counter map: sysfs, ethtool and rdma statisticThe dashboard that swears the rail is deadApply30 min
- 4.2Triage in order: symptom to counter to cause01:40, and the bridge wants to change a DCQCN parameterEvaluate35 min
- 4.3MTU, RoCE LAG, and what has to be installedThe bond that took the RDMA device with itApply30 min
- 4.4The switch half: Cumulus NVUE and Dell Enterprise SONiCDoes this switch do RoCE, or only PFC?Apply30 min
- 4.5RDMA in Kubernetes: the NVIDIA Network OperatorNobody proved the fabric, and acceptance is at nineAnalyze30 min
- Module 4 checkpoint13 questions · pass mark 80% · 12 min
Turn the technical material into what the job pays for: rail-aware designs on Dell hardware, defensible recommendations under customer pressure, and a mapped study plan for NVIDIA-Certified Professional: AI Networking.
- 5.1Rail-optimized fabrics on Dell hardwareThe two cards in the labelled slotsAnalyze30 min
- 5.2Scenario: "the 400G NIC only does 180 Gb/s"Two numbers, six people, and no command lineEvaluate35 min
- 5.3Scenario: "we do not want to configure PFC"The slide that promises no switch configurationCreate30 min
- 5.4NCP-AIN: what this course covers and what it does notEight exam seats in a draft emailEvaluate35 min
- 5.5Positioning multipath: what ships, what is a specificationA quarter later, the RFP paragraphCreate40 min
- Module 5 checkpoint14 questions · pass mark 80% · 12 min