Scale-up is not scale-out: NVLink vs the fabric
S1·E3The commodity fabric · Video call with the customer's ML platform team, Wednesday, two days before the BOM freezes
Builds on: Which reference architecture governs this deal, Reading node nomenclature: 2-8-9-400
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Distinguish the NVLink scale-up domain from the RDMA scale-out fabric by semantics, size and bandwidth.
- Quote per-GPU NVLink bandwidth and NVLink domain size for Hopper, Blackwell and Rubin from the NVLink page.
- Compute the scale-up to scale-out bandwidth ratio for a platform and label it as derived rather than published.
- Decide from a job's coherent-GPU requirement whether the NVLink domain or the fabric is the binding constraint.
Episode 3 — The commodity fabric
The platform lead opens with the decision already made. They are looking at rack-scale NVL72 instead, so the east-west fabric is a commodity and the fabric line can go to whoever is cheapest. The network lead has his notebook open and says, mildly, “show me the counter.” The SE stops typing.
You ask one question instead. How many GPUs must a single forecasting job see coherently? The answer is 512.
That number moves the whole meeting. NVLink is memory-semantic and bounded by hardware: NVIDIA’s NVLink Switch table lists domains of 8 GPUs on NVLink 4 and 8 or 72 GPUs on NVLink 5, and inside an NVL72 rack nine fifth-generation NVLink switch trays deliver a total aggregated bandwidth of 130 TB/s.[1][3] Above 72 GPUs the job leaves that domain, and every crossing pair talks RDMA over a ConnectX-8 SuperNIC at 800 Gb/s per GPU across leaf and spine.[2] Each NVL72 rack is one scalable unit, and a fully tested system scales up to 8 SUs.[5]
The split exists because the two problems are genuinely different. Sharing memory coherently wants a bounded load/store interconnect with no headers; communicating at cluster scale wants a routable, message-semantic network with congestion control. Confusing them is how a 512-GPU training run ends up on a fabric nobody sized.
NVLink stops at the domain boundary, so the largest single job - not the GPU total - decides whether the fabric is plumbing or performance. Start with the semantics, not the bandwidth.
1Two networks with different semantics
Customers conflate NVLink and the fabric constantly, and the confusion is expensive because the two behave differently at every level.
NVLink is memory-semantic. A GPU reaches a peer GPU’s memory. The domain is bounded by hardware: NVIDIA’s NVLink Switch table lists domains of 8 GPUs on NVLink 4, and 8 or 72 GPUs on NVLink 5 and NVLink 6.[1] Inside an NVL72 rack, nine fifth-generation NVLink switch trays deliver a total aggregated bandwidth of 130 TB/s.[3]
The fabric is message-semantic. A GPU’s SuperNIC sends RDMA messages to another node across leaf and spine switches. On a B300 node each GPU has 800 Gb/s (2 x 400 Gb/s Ethernet) through its own ConnectX-8 SuperNIC.[2] Under dual plane those two 400 Gb/s interfaces land on two different leaf switches, each part of an independent fabric that scales to 1024 interfaces of 400 Gb/s.[6]
Diagram source (Mermaid)
flowchart LR
subgraph SU["Scale-up: NVLink domain, 8 or 72 GPUs"]
G1["GPU"] --- NVS["NVSwitch tray"]
G2["GPU"] --- NVS
G3["GPU"] --- NVS
end
G1 --> NIC["ConnectX-8 SuperNIC 800 Gb/s per GPU"]
NIC --> P1["plane A leaf 400G"]
NIC --> P2["plane B leaf 400G"]
P1 --> SP["spine"]
P2 --> SP2["spine plane B"]
SP --> REM["GPUs in another rack"]
SP2 --> REM2The numbers, with their units
Write them down once and stop guessing.
| Generation | NVLink per GPU | Links per GPU | Domain | Aggregate |
|---|---|---|---|---|
| NVLink 4 - Hopper | 900 GB/s | up to 18 | 8 GPUs | 7.2 TB/s[1] |
| NVLink 5 - Blackwell | 1800 GB/s | up to 18 | 8 or 72 GPUs | 130 TB/s in NVL72[1] |
| NVLink 6 - Rubin | 3600 GB/s | up to 36 | 8 or 72 GPUs | 260 TB/s in NVL72[1] |
The NVLink 6 row is marked as preliminary specifications that may be subject to change; repeat the disclaimer whenever you repeat the number.[1]
Two RA pages give the same figures from the platform side. The HGX B300 baseboard publishes a total aggregate bandwidth of 14.4 TB/s with GPU-to-GPU bandwidth of 1800 GB/s across its eight GPUs.[2] The GB300 NVL72 rack publishes 130 TB/s of total aggregated NVSwitch bandwidth and up to 1800 GB/s per GPU.[3]
Now the scale-out side, in different units. ConnectX-8 gives 800 Gb/s per GPU on both B300 and GB300.[2][3] One GB300 compute tray carries four ConnectX-8 Host Channel Adapters delivering 800 Gb/s plus one dual-port BlueField-3 B3240 at approximately 480 Gb/s aggregate.[3]
The derived ratio. 1800 GB/s is 14,400 Gb/s. Against 800 Gb/s of fabric per GPU that is about 18 to 1. Say that out loud as our arithmetic: both inputs are published on NVIDIA pages, the division is not.[1][2] An FAE who presents a derived ratio as a quoted figure loses the room the first time somebody searches for it.
Which RA governs this deal?
Four customer facts → one named document, its scope sentence and its SU arithmetic.
NVIDIA NVL72 AI Factory — Enterprise Reference Architecture (GB300)
“multi-user, single tenant workloads … users are all part of the same enterprise”
NVL72 AI Factory — overview- Power: this RA is written for DC busbar (MGX), the site has AC rack PDU.
"fully tested" up to "8 SUs (Scalable Units)"; larger clusters are "possible based on customer requirements". One SU is 18 trays = one NVL72 rack.
- Nodes at 4 GPUs each: 576 ÷ 4 = 144 nodes derived
- SUs: 576 ÷ 72 GPUs per SU = 8.00 → 8 SUs to quote derived
2-4-5-800 (dual plane). 2 Grace CPUs, 4 Blackwell Ultra GPUs, 5 NICs (4 ConnectX-8 Host Channel Adapters at 800 Gb/s + 1 BlueField-3 B3240 DPU at ~480 Gb/s from "two 400 Gb/s ports"), 800 Gb/s per GPU. The 4 CX-8 arrive as "two such boards, with two ConnectX-8 devices on each board". NVL72 AI Factory — overview- No BGP/EVPN, IP addressing, MTU, RoCE QoS (DSCP/PFC/ECN), adaptive-routing or congestion-control settings — those live in the Cumulus / Spectrum-X software docs, not in this RA.
- No multi-tenant design. The RA is scoped to a single tenant; a service-provider deal falls outside it.
- No AC-PDU variant: the rack is liquid-cooled MGX with 8 power shelves.
FAE angle: dual plane is not redundancy bonding. Each 800G ConnectX-8 port is broken out to 2x400G that land on two different leaf switches in two independent fabrics — leaf count doubles at the same per-GPU bandwidth and the customer must accept two IP fabrics. Ask whether their automation assumes one fabric per NIC.
3The sizing question that decides everything
There is one question that turns this into engineering: how many GPUs must a single job see coherently?
If the answer is 72 or fewer, an NVL72 rack keeps the whole job inside the NVLink domain and the fabric carries only what leaves the rack. Each scalable unit in the NVL72 AI Factory RA is a single NVL72 rack of 18 compute trays with 72 Blackwell Ultra GPUs and 36 Grace CPUs, and a fully tested system scales up to 8 SUs.[5]
If the answer is larger, every GPU pair that spans two racks talks over ConnectX-8 and the leaf-spine fabric, and the fabric technology becomes the performance decision rather than a plumbing choice. Both published options are large: Spectrum-X claims scaling up to 128,000 GPUs in two tiers with multiplane[8], and Quantum-X800 publishes 144 ports of 800 Gb/s per switch with SHARP v4, adaptive routing and telemetry-based congestion control.[9]
Note the asymmetry this creates in a conversation. You cannot buy your way out of a bad fabric with NVLink, because NVLink stops at the domain boundary. And you cannot buy your way out of a small NVLink domain with more fabric, because the fabric is roughly an order of magnitude slower per GPU and message-semantic besides.[1][2]
4Generations shift the numbers, not the framing
The framing survives every generation; the numbers do not.
GB200 NVL72 is a ConnectX-7 and NDR-era rack: each compute tray carries two GB200 Superchips, four ConnectX-7 NICs supporting InfiniBand NDR at 400 Gbps, and two BlueField-3 NICs supporting 2x200 Gbps connectivity.[10] GB300 NVL72 is ConnectX-8 and 800G: four ConnectX-8 at 800 Gb/s plus one BlueField-3 B3240 per tray.[3] Same rack shape, same NVLink 5 domain of 72 GPUs, double the fabric speed per adapter.
That is a two-times optics and leaf-port error waiting to happen. A customer holding a GB200 quote who then reads GB300 documentation - or the reverse - will size transceivers and leaf ports for the wrong generation, and the mistake surfaces at the cable delivery rather than at the design review.
On the scale-up side, NVLink 6 doubles per-GPU bandwidth again to 3600 GB/s with up to 36 links per GPU and 260 TB/s of aggregate in an NVL72-class domain, still on the preliminary label.[1] Note that the domain size in the published table does not grow between NVLink 5 and NVLink 6 - both list 8 or 72 GPUs.[1] Bandwidth per GPU grows; the coherent-GPU count that decides your design does not.
Request. “Build me the scale-up versus scale-out comparison for Blackwell Ultra so I can use it in a customer deck.”
1. Fix the platform. Blackwell Ultra appears in two shapes: HGX B300 (8 GPUs on a baseboard) and GB300 NVL72 (72 GPUs in a rack). Build both columns or you will be corrected.
2. Scale-up column, HGX B300. GPU-to-GPU bandwidth 1800 GB/s; total aggregate bandwidth across the baseboard 14.4 TB/s; domain size 8 GPUs.[2][1]
3. Scale-up column, GB300 NVL72. Up to 1800 GB/s per GPU; 130 TB/s total aggregated NVSwitch bandwidth from nine fifth-generation NVLink switch trays; domain size 72 GPUs.[3][1]
4. Scale-out column, both. 800 Gb/s per GPU through ConnectX-8, expressed by the RAs as 2 x 400 Gb/s Ethernet under dual plane, each interface landing on a different leaf in an independent fabric.[2][7]
5. Convert before comparing. 1800 GB/s x 8 bits = 14,400 Gb/s. 14,400 / 800 = 18.
6. Label the ratio. On the slide write: scale-up is roughly 18x scale-out per GPU (derived: 1800 GB/s vs 800 Gb/s; NVIDIA does not publish this ratio).[1][2]
7. Add the line that makes it useful. “Which means the design question is how many GPUs one job must see coherently, not which number is bigger.”[5]
Request. “Do the same for Hopper so we can show the customer their installed base next to the upgrade.”
- Scale-up per GPU on NVLink 4:
____GB/s with up to____links per GPU.[1] - NVLink 4 Switch domain size:
____GPUs, aggregate____TB/s.[1] - Scale-out per GPU on the H100/H200/B200 Enterprise RA generation: the east-west adapter is the BlueField-3
____SuperNIC. State the per-GPU east-west speed only if the RA page gives it; otherwise write not published on the page I read and say which page you read. - Convert the scale-up figure to Gb/s:
____. - Compute the ratio against your scale-out figure:
____. Mark it derived. - Write one sentence on what changes between the Hopper column and the Blackwell column, and one on what does not.[1]
Problem. A Dell account is comparing two quotes for the same 576-GPU training cluster: eight GB300 NVL72 racks, or 72 HGX B300 nodes.
Produce:
- The coherent-GPU domain size in each option, with a citation.[1][5]
- For a single job that must span all 576 GPUs, the fraction of GPU pairs that are inside an NVLink domain versus across the fabric, in each option. Show the arithmetic and label it as yours.
- The per-GPU fabric bandwidth in each option, with a citation.[2][3]
- One sentence stating which of the two options makes the fabric choice more consequential, and why.
Acceptance criteria. Every published number carries a page. Every computed number is visibly marked as derived. Your final sentence names the coherent-GPU domain as the reason, not a bandwidth comparison.
Two columns on the shared screen
You write the two numbers side by side - 1800 GB/s per GPU inside the domain, 800 Gb/s per GPU outside it - and say out loud that the roughly eighteen-to-one ratio between them is your own division and not a figure NVIDIA prints.[1][2] Then the sizing sentence: at 512 GPUs the job spans eight racks, so the fabric carries the gradients, and the RA is tested to 8 SUs.[5] What you actually say: “NVLink stops at 72 GPUs. Your largest job is 512, so the fabric is the performance decision, not the plumbing.” The fabric line survives. Overnight the customer’s network architect writes in: he has read the RA twice and believes he was sent an incomplete copy.
Lab
Goal: confirm the scale-out column against real hardware in the Dell lab. Every step is read-only.
- Pre-flight inventory. List the adapters with
mst status -vand map them to interfaces withibdev2netdev. Record the output before you change your terminal scrollback. - Read negotiated Ethernet link speed:
ethtool <iface> | grep -E 'Speed|Duplex|Link detected'. Expected: a Speed line in Mb/s, for exampleSpeed: 400000Mb/sfor a 400G port. If it readsUnknown!, the port has no link - note it and move on rather than touching configuration. - Read the RDMA side:
ibstatand, per device,ibstatus. Expected: arateline per port and astate: ACTIVEon any linked port. On an Ethernet-mode card the link layer readsEthernet. - Compare each port’s rate to the scale-out column you built. A dual-port 400G card that shows two 400 Gb/s ports is the 800 Gb/s per GPU figure in its physical form.[2]
- Read PCIe width and speed with
lspci -vv -s <bdf> | grep -E 'LnkCap|LnkSta'. Expected:LnkStamatchingLnkCapin width. A x16 card negotiated at x8 halves host bandwidth and is a common cause of a benchmark that misses the published number - note it, do not re-seat anything. - If a multi-GPU host is available, run
nvidia-smi topo -mand record the matrix. Expected:NV#entries between NVLink-connected GPU pairs andPIX,PXB,PHBorSYSfor pairs that cross PCIe. Write one sentence naming which pairs are inside the scale-up domain and which are not. - Optional and customer-lab only: on an NVL72 rack read the NVLink topology and compare the domain size to the published 72.[1] Do not attempt this on shared production hardware.
- Acceptance: for one real port you can say the negotiated rate, the PCIe width behind it, and which column of your table it belongs in.
Goal: build the two-column table yourself and then feel the difference between memory semantics and message semantics in containerlab.
- Open the NVLink and NVLink Switch page and copy both tables - the NVLink generation table and the NVLink Switch table - into a scratch file with the disclaimer text attached to the NVLink 6 rows.[1] Expected: 900 / 1800 / 3600 GB/s per GPU and domains of 8, then 8 or 72.
- Open the HGX AI Factory components page and record the baseboard aggregate and GPU-to-GPU numbers, then the east-west per-GPU figure.[2] Expected: 14.4 TB/s, 1800 GB/s, and 800 Gb/s expressed as 2 x 400 Gb/s Ethernet per GPU.
- Open the NVL72 AI Factory components page and record the NVSwitch aggregate, the per-GPU NVLink figure and the per-tray adapter counts.[3] Expected: 130 TB/s, up to 1800 GB/s, four ConnectX-8 plus one B3240.
- Build a table with one row per generation and two columns - scale-up per GPU and scale-out per GPU - with a third column for the ratio. Convert units explicitly in the cell, for example
1800 GB/s = 14400 Gb/s. Mark the ratio column derived in its header. - In containerlab, stand up two Linux hosts either side of an FRR router. A minimal topology file with three nodes and two point-to-point links is enough. Expected:
containerlab deploy -t <file>prints a node table with running containers. - From one host run
pingand theniperf3to the other and capture withtcpdump -i <iface> -c 5 -nn. Expected: every byte is inside an IP packet with a header and a checksum - there is no operation here that reads the other host’s memory. That absence is the point: this is what the fabric is, and it is what NVLink is not. - Tear down with
containerlab destroy -t <file>. - Acceptance: you can state, in one sentence and without notes, the domain size and per-GPU bandwidth on both sides for Blackwell, and you flag the ratio as your arithmetic when you say it.
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell SE in four sentences why a customer cannot buy their way out of a weak fabric with NVLink, using one number from each side.
Sources
Facts in this lesson were checked against NVIDIA NVLink and NVLink Switch page, HGX AI Factory components, NVL72 AI Factory components and overview and physical topologies, all re-fetched 2026-09-07. Dates are when each page was fetched.
- NVLink and NVLink Switch · fetched 2026-09-07
- Components - NVIDIA HGX AI Factory · fetched 2026-09-07
- System Hardware and Components - NVIDIA NVL72 AI Factory · fetched 2026-09-07
- NVIDIA GB300 NVL72 · fetched 2026-09-07
- Overview - NVIDIA NVL72 AI Factory · fetched 2026-09-07
- Networking Physical Topologies - NVIDIA HGX AI Factory · fetched 2026-09-07
- Networking Physical Topologies - NVIDIA NVL72 AI Factory · fetched 2026-09-07
- NVIDIA Spectrum-X Ethernet Platform · fetched 2026-09-07
- NVIDIA Quantum-X800 InfiniBand Platform · fetched 2026-09-07
- Key Components of the DGX SuperPOD - DGX GB200 Reference Architecture · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.
- Scenario: a 256-GPU Dell AI Factory design reviewElsewhere in this course · Same ground: derived, sizing and gpu
- Dual plane: two fabrics, not a bondElsewhere in this course · Same ground: gb300, sizing and Scale-out
- Topologies: fat-tree, rail-optimized, twin-planeInfiniBand course · Same ground: b300, sizing and Scale-out