Skip to content
nodeza.learn

Spectrum-X Ethernet AI fabrics with Cumulus Linux

A hands-on course for a senior lab engineer moving into an NVIDIA Networking FAE seat on the Dell OEM account. It builds from the Spectrum ASIC portfolio and Cumulus Linux 5.18/NVUE fundamentals through RoCE QoS (PFC, ECN, buffer pools, packet trimming), the three Spectrum-X control loops (adaptive routing, telemetry-based congestion control, SuperNIC reordering and plane load balancing), telemetry and NetQ/WJH operations, and ends in FAE scenarios plus NCP-AIN blueprint prep. Every lesson ships two labs: one runnable with NVIDIA DSX Air, containerlab and Dell Enterprise SONiC 4.5.1, and one for the Dell lab's BlueField-3/ConnectX hosts, with customer-switch steps marked optional. Baselines: Cumulus Linux 5.18.1, Spectrum-X RA v2.3.1, NetQ 5.1.0, DOCA-Host 3.5.0-082.

6 modules. Dots: grey not started, blue in progress, green complete with quiz ≥ 80%, amber complete but below 80%. Each module ends in a checkpoint that unlocks the next.

Spectrum-X Ethernet AI fabrics with Cumulus Linux

0%

0 of 29 cases closed

Foundations

Fabric build

RoCE QoS

Spectrum-X core

Operate

FAE + cert

The Spectrum-X platform: silicon, switches and the validated stack

Explain what Spectrum-X actually is (a switch + SuperNIC system, not one box), pick the right Spectrum switch for a Dell AI pod against real airflow/power/cooling constraints, get a link up at the right speed and FEC, and read the RA version row that governs the whole fabric.

  1. 1.1 What Spectrum-X is, and what it is notThe benchmark that measured nothing
  2. 1.2 Choosing the switch: SN2201 through SN6800-LDThe switch that cannot be installed
  3. 1.3 Links that come up: breakout, FEC and auto-negotiationTwenty-four up, eight refusing
  4. 1.4 The RA row: version pinning across switch, NIC and hostThe bridge call about one bug fix
  5. 1.5 Dell PowerSwitch SN-series: who supports whatWe will just run SONiC on it
  6. Checkpoint

Cumulus Linux and NVUE: building the fabric

Bring a Spectrum switch from ONIE to a working leaf-spine: the NVUE object model and its revisions, ZTP, L2 with MLAG and VRR, a BGP-unnumbered underlay with sane ECMP, and BGP-EVPN symmetric IRB for tenant isolation.

  1. 2.1 The NVUE object model, revisions and snippetsThe line nobody deleted
  2. 2.2 Day-0: ONIE install, first boot and ZTPTwenty-nine of thirty-two
  3. 2.3 Layer 2 that survives a reload: bridges, bonds, MLAG and VRRBoth bonds down, one host, 02:40
  4. 2.4 The underlay: eBGP unnumbered, ECMP and VRFsTwo spines, eighty-twenty
  5. 2.5 BGP-EVPN symmetric IRB for tenant isolationTenant BLUE, seven hours out
  6. Checkpoint

RoCE QoS: classification, PFC, ECN and buffers

Turn a switch into a lossless RoCE fabric with one command, then know precisely what that command did, how to tune it, and which four counters answer 'RoCE is slow'.

  1. 3.1 How a RoCE packet gets classifiedThe DSCP that cannot exist
  2. 3.2 `nv set qos roce`: the one command and its five modesFive minutes, low risk
  3. 3.3 Tuning the lossless edge: PFC headroom, ECN thresholds, watchdogZero drops, and nothing is moving
  4. 3.4 Packet trimming and `lossy-multi-tc`Enable trimming everywhere, next Tuesday
  5. 3.5 Playbook: 'RoCE is slow'The ticket that says only 'RoCE is slow'
  6. Checkpoint

Adaptive routing, congestion control and the SuperNIC

Explain and configure the three mechanisms that make Spectrum-X different from a stock RoCE fabric — per-packet adaptive routing, SuperNIC reordering, and telemetry-based congestion control — and know exactly when each is ineligible.

  1. 4.1 Adaptive routing: per-packet spraying and its eligibility rulesThe register that says yes and means no
  2. 4.2 Why spraying needs a SuperNIC: reordering and direct data placementThe line item the server team wants to change
  3. 4.3 Telemetry-based congestion control: three loops, not oneHigh pause, low marks, at ten past two
  4. 4.4 Programmable congestion control with DOCA PCCCoffee, a laptop, and a customer who wants their own algorithm
  5. 4.5 Multiplane fabrics and multi-tenant performance isolationQuad-plane on the hardware they already own
  6. Checkpoint

Telemetry, NetQ, automation and lifecycle

Run the fabric after day 1: name a drop instead of counting it, stream the right counters at the right interval, drive NVUE from templates and the REST API, and upgrade without losing the config or the data plane.

  1. 5.1 What Just Happened: naming the dropThree nights of drops nobody can name
  2. 5.2 NetQ 5.1: validation instead of inspectionTwo engineers on two leaves, both right
  3. 5.3 Streaming telemetry: OTLP, high-frequency counters, latency measurementThe stall that never appears on the dashboard
  4. 5.4 Automating NVUE: REST API, config templates and AnsibleThe first question is not technical
  5. 5.5 Upgrades that do not lose the fabricFifty minutes before the change board
  6. Checkpoint

FAE scenarios and NCP-AIN preparation

Convert the course into the two artifacts the job needs: a triage toolkit you can run on a customer call, and an honest map of the NCP-AIN blueprint showing what this course covers and what still has to be studied elsewhere.

  1. 6.1 The triage toolkit: resources, optics and what tcpdump will not tell youThe empty capture at 02:40
  2. 6.2 Scenario: 'the fabric is slow and adaptive routing does nothing'Four claims and a cold coffee
  3. 6.3 Scenario: sizing, quoting and defending a Dell AI pod fabricThe airflow question that stopped the room
  4. 6.4 NCP-AIN: blueprint map, gaps and a study planBooking the exam is a design decision
  5. Checkpoint