Spectrum-X Ethernet AI fabrics with Cumulus Linux
0 of 29 cases closed
Foundations
Fabric build
RoCE QoS
Spectrum-X core
Operate
FAE + cert
Spectrum-X Ethernet AI fabrics with Cumulus Linux
Build, tune and troubleshoot an AI Ethernet fabric on NVIDIA Spectrum switches — and defend it in front of a Dell customer.
A hands-on course for a senior lab engineer moving into an NVIDIA Networking FAE seat on the Dell OEM account. It builds from the Spectrum ASIC portfolio and Cumulus Linux 5.18/NVUE fundamentals through RoCE QoS (PFC, ECN, buffer pools, packet trimming), the three Spectrum-X control loops (adaptive routing, telemetry-based congestion control, SuperNIC reordering and plane load balancing), telemetry and NetQ/WJH operations, and ends in FAE scenarios plus NCP-AIN blueprint prep. Every lesson ships two labs: one runnable with NVIDIA DSX Air, containerlab and Dell Enterprise SONiC 4.5.1, and one for the Dell lab's BlueField-3/ConnectX hosts, with customer-switch steps marked optional. Baselines: Cumulus Linux 5.18.1, Spectrum-X RA v2.3.1, NetQ 5.1.0, DOCA-Host 3.5.0-082.
Who it is for: Network engineers and FAEs designing and operating Spectrum-X fabrics with Cumulus Linux.
- Not started
- In progress
- Complete, case open
- Case closed
- Locked
- Checkpoint
- A case closes when the lesson’s retrieval check reaches 80%.
Explain what Spectrum-X actually is (a switch + SuperNIC system, not one box), pick the right Spectrum switch for a Dell AI pod against real airflow/power/cooling constraints, get a link up at the right speed and FEC, and read the RA version row that governs the whole fabric.
- 1.1What Spectrum-X is, and what it is notThe benchmark that measured nothingUnderstand25 min
- 1.2Choosing the switch: SN2201 through SN6800-LDThe switch that cannot be installedUnderstand30 min
- 1.3Links that come up: breakout, FEC and auto-negotiationTwenty-four up, eight refusingApply30 min
- 1.4The RA row: version pinning across switch, NIC and hostThe bridge call about one bug fixEvaluate25 min
- 1.5Dell PowerSwitch SN-series: who supports whatWe will just run SONiC on itAnalyze25 min
- Module 1 checkpoint14 questions · pass mark 80% · 12 min
Bring a Spectrum switch from ONIE to a working leaf-spine: the NVUE object model and its revisions, ZTP, L2 with MLAG and VRR, a BGP-unnumbered underlay with sane ECMP, and BGP-EVPN symmetric IRB for tenant isolation.
- 2.1The NVUE object model, revisions and snippetsThe line nobody deletedApply30 min
- 2.2Day-0: ONIE install, first boot and ZTPTwenty-nine of thirty-twoApply30 min
- 2.3Layer 2 that survives a reload: bridges, bonds, MLAG and VRRBoth bonds down, one host, 02:40Apply35 min
- 2.4The underlay: eBGP unnumbered, ECMP and VRFsTwo spines, eighty-twentyAnalyze30 min
- 2.5BGP-EVPN symmetric IRB for tenant isolationTenant BLUE, seven hours outAnalyze35 min
- Module 2 checkpoint14 questions · pass mark 80% · 12 min
Turn a switch into a lossless RoCE fabric with one command, then know precisely what that command did, how to tune it, and which four counters answer 'RoCE is slow'.
- 3.1How a RoCE packet gets classifiedThe DSCP that cannot existUnderstand25 min
- 3.2`nv set qos roce`: the one command and its five modesFive minutes, low riskApply30 min
- 3.3Tuning the lossless edge: PFC headroom, ECN thresholds, watchdogZero drops, and nothing is movingAnalyze35 min
- 3.4Packet trimming and `lossy-multi-tc`Enable trimming everywhere, next TuesdayAnalyze30 min
- 3.5Playbook: 'RoCE is slow'The ticket that says only 'RoCE is slow'Analyze30 min
- Module 3 checkpoint14 questions · pass mark 80% · 12 min
Explain and configure the three mechanisms that make Spectrum-X different from a stock RoCE fabric — per-packet adaptive routing, SuperNIC reordering, and telemetry-based congestion control — and know exactly when each is ineligible.
- 4.1Adaptive routing: per-packet spraying and its eligibility rulesThe register that says yes and means noAnalyze30 min
- 4.2Why spraying needs a SuperNIC: reordering and direct data placementThe line item the server team wants to changeUnderstand25 min
- 4.3Telemetry-based congestion control: three loops, not oneHigh pause, low marks, at ten past twoAnalyze30 min
- 4.4Programmable congestion control with DOCA PCCCoffee, a laptop, and a customer who wants their own algorithmApply30 min
- 4.5Multiplane fabrics and multi-tenant performance isolationQuad-plane on the hardware they already ownAnalyze30 min
- Module 4 checkpoint14 questions · pass mark 80% · 12 min
Run the fabric after day 1: name a drop instead of counting it, stream the right counters at the right interval, drive NVUE from templates and the REST API, and upgrade without losing the config or the data plane.
- 5.1What Just Happened: naming the dropThree nights of drops nobody can nameApply30 min
- 5.2NetQ 5.1: validation instead of inspectionTwo engineers on two leaves, both rightApply30 min
- 5.3Streaming telemetry: OTLP, high-frequency counters, latency measurementThe stall that never appears on the dashboardApply30 min
- 5.4Automating NVUE: REST API, config templates and AnsibleThe first question is not technicalCreate30 min
- 5.5Upgrades that do not lose the fabricFifty minutes before the change boardAnalyze35 min
- Module 5 checkpoint14 questions · pass mark 80% · 12 min
Convert the course into the two artifacts the job needs: a triage toolkit you can run on a customer call, and an honest map of the NCP-AIN blueprint showing what this course covers and what still has to be studied elsewhere.
- 6.1The triage toolkit: resources, optics and what tcpdump will not tell youThe empty capture at 02:40Analyze30 min
- 6.2Scenario: 'the fabric is slow and adaptive routing does nothing'Four claims and a cold coffeeEvaluate30 min
- 6.3Scenario: sizing, quoting and defending a Dell AI pod fabricThe airflow question that stopped the roomCreate35 min
- 6.4NCP-AIN: blueprint map, gaps and a study planBooking the exam is a design decisionEvaluate30 min
- Module 6 checkpoint13 questions · pass mark 80% · 12 min