Skip to content
nodeza.learn
RA course/2. Sizing/Lesson 2.4

The other three networks: converged, storage, out-of-band

S2·E4The call at 02:10 · The insurer's data hall, day two of the phase-two install

S2·E4Apply~25 minsources checked todaylab mutates hardwareverified against HGX AI Factory RA Networking Physical Topologies re-fetched 2026-09-09 (converged two 400 GbE ports to two separate switches up to 40 GB/s per node; OOB description; 1024 interfaces per fabric; 50 percent single-plane statement); HGX AI Factory NVIDIA-Certified Storage re-fetched 2026-09-09 (approximately 12.5 Gb/s per GPU scaling linearly; 16 GPU cluster around 200 Gb/s); DGX SuperPOD B300 Spectrum-4 and DC busbar Network Fabrics re-fetched 2026-09-09 (1:1 storage port to uplink; near 4:3 DGX oversubscription; I/O per node must exceed 40 GBps; SN5600/SN5600D storage; SN5600D in-band; SN2201 OOB; 2x100GbE DR1 edge); remaining figures as recorded in content/research/ra/part1.md fetched 2026-09-07

Builds on: The sizing chain: GPUs, NICs, planes, leaves, spines

Before you read: what do you already know?

3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.

After this lesson you can

  • Compute converged north-south port counts for a given node count using the published two-ports-per-node rule.
  • Apply the published storage floors to a GPU count and reconcile the bits-per-GPU figure against the bytes-per-node figure.
  • Count the out-of-band RJ45 ports a pod actually consumes and convert that into a number of SN2201 switches.
  • Name which storage and oversubscription ratios are published and which are your own arithmetic.

Episode 4 — The call at 02:10

The situation · The insurer's data hall, day two of the phase-two install

The night-shift operator calls at 02:10. The compute fabric is racked, cabled and a day ahead of schedule; he is halfway through the second rack of management cabling and has run out of RJ45 jacks. His label maker has already printed every port name on the design, which is how he knows the shortfall is exactly one jack per node rather than a miscount.

Somebody counted one out-of-band port per server. Every node has two. The BlueField-3 in each node is an independent computer with its own baseboard management controller, and the Quantum-X800 edition of the DGX B300 RA names both when it lists what the out-of-band fabric connects: “DGX B300 compute trays including system BMCs and BlueField-3 BMCs, switches, and management servers”.[9] That is a factor of two on the network with the cheapest ports and the least attention, against an SN2201 that offers 48 of them plus four 100 GbE uplinks in 1U.[10]

The three quiet networks exist because the compute fabric cannot carry their jobs. The HGX B300 RA defines compute east-west, CPU converged north-south and out-of-band management as three physical fabrics, with storage riding the converged one under VLAN isolation rather than getting a wire of its own.[1][2] Each has a published per-node number, and each number is small enough to forget.

Count management ports, not servers.

Segment 1 counts all three.

1Three networks, and the quote forgets all of them

The compute fabric takes the argument and the other three take the install-day emergency. The HGX B300 Enterprise RA defines three physical fabrics — compute east-west, CPU converged north-south, and out-of-band management — and the converged one carries several logical networks at once under VLAN isolation.[1][2] Storage is not a fourth wire in that design; it is traffic on the converged fabric, which is exactly why it gets left out of a switch count.[1]

Each of the three has a published per-node number, and each number is small enough to forget. Converged is two ports.[1] Storage is a per-GPU bandwidth floor.[3] Out-of-band is counted in RJ45 jacks, one per management controller.[2] None of them scale with your GPU count in the same way, so you cannot derive one from another — you size all three or you guess at two.

The DGX SuperPOD RAs treat the same three roles with named SKUs rather than ratios: SN5600 or SN5600D for Ethernet storage, SN5600D for in-band management, SN2201 for out-of-band, and a customer edge of “at least 2x 100GbE links with DR1 single-mode connectivity”.[4] That is the shape of the answer you want at the end of this lesson: a port count and a switch model for each of the three, derived out loud from published per-node figures.

GPUs
256
Planes
Uplink ratio
Compute nodes32 x 2 x 400 GbE = 64 portspublished
Management nodes7 x 2 x 400 Gb/s = 14 portspublished
Total converged ports78derived
Converged switches2 (two separate switches minimum)derived
Customer edge minimum2x 100GbE, DR1 single-mode, BGP to the customer networkpublished

Each compute and management node "is connected with two 400 GbE ports to two separate switches" for redundancy, delivering "up to 40 GB/s per node" on B300. On the H100/H200/B200 generation it is two 200 GbE ports. GB300 management nodes take four 200 Gb/s ports.

What the converged fabric carries

North-south: cluster service management, NFS storage access, uplink connectivity, and user access to Slurm or Kubernetes — RoCE-capable, terminated on the BlueField-3 B3240.

FAE angle

Nobody asks about the converged fabric until install day, and then it is two 400G ports per node that were never quoted. Count management nodes too: the HGX B300 RA supports up to eight control-plane nodes and the example deployment uses seven; the NVL72 RA assumes twelve.

HGX B300 Physical Topologies · NVL72 Physical Topologies
Start on the converged tab with the same 256-GPU cluster you sized in the previous lesson. Note that the converged port count follows nodes not GPUs.

2Converged: two ports, two switches, 40 GB/s

The sentence to memorise is verbatim: “Each compute and management node is connected with two 400 GbE ports to two separate switches to provide redundancy and high storage throughput that can reach up to 40 GB/s per node.”[1] Two facts hide in it. The second switch is mandatory — one port each on two boxes, not two ports on one — and the 40 GB/s is a capability of the path, not a requirement placed on the storage array.[1]

The port profile changes by generation and by platform, and this is where quotes get copied wrong. On the H100/H200/B200 Enterprise RA the same shape runs at half the speed: “two 200 GbE ports to two separate switches” per node, with a BlueField-3 B3220 rather than a B3240.[7] On GB300 NVL72 the compute tray takes dual 400 Gb/s ports while management nodes take four 200 Gb/s ports, over a fabric the RA describes as Ethernet with RoCE support delivered by the BlueField-3 B3240 in each tray.[6]

Management nodes are not a rounding error either. The HGX B300 RA supports “up to eight control plane nodes” and its example deployment uses seven.[12] Those seven take converged ports on exactly the same rule as the compute nodes.[1]

So for a 32-node pod with seven control-plane nodes: 39 nodes x 2 ports = 78 interfaces of 400 GbE, 39 landing on each of two switches. (Rule published, multiplication mine.) That fits comfortably inside one SN5600-class pair at 128 logical 400 GbE ports each, and does not fit inside an SN5400 pair at 64 x 400 GbE if you also want uplinks.[10]

3Storage: a bits floor and a bytes ceiling

The Enterprise RA gives one sizing rule: “approximately 12.5 Gb/s per GPU, scaling linearly as the cluster grows”, with the worked example that “a 16 GPU cluster would require around 200 Gb/s of aggregate storage bandwidth”.[3] Apply it to the pod you have been sizing: 256 GPUs x 12.5 Gb/s = 3,200 Gb/s, so 3.2 Tb/s of aggregate storage bandwidth. (Ratio published; the multiplication is mine.)

Now the trap. The storage figure is in bits per second per GPU; the converged figure is in bytes per second per node.[3][1] Convert before comparing: an eight-GPU node at 12.5 Gb/s per GPU wants 100 Gb/s, which is 12.5 GB/s. The converged fabric is specified at up to 40 GB/s per node.[1] The fabric therefore sits roughly three times above the Enterprise RA storage floor. (That reconciliation is mine — no fetched page performs it.)

The DGX SuperPOD RA is a different document with a harder statement: “the I/O per-node for the DGX SuperPOD must exceed 40 GBps”, storage devices connect “at a 1:1 port to uplink ratio”, and “The DGX B300 system connections are slightly oversubscribed with a ratio near 4:3”.[4] The GB200 SuperPOD publishes its own blocking factor of 5:3 on the DGX side with 16x 800 Gbps non-blocking to the storage appliances.[5] Quote whichever document governs the deal, and never mix the Enterprise RA’s 12.5 Gb/s with the SuperPOD’s 40 GBps as if they were the same rule.

Vendor selection has its own gate: storage must come from the “NVIDIA-Certified Storage program”, described as a validation framework for performance, quality and interoperability.[3] In a Dell deal that usually means PowerScale, which brings a hard constraint of its own — a private back-end with redundant switches where “Dell does not support connecting any other devices to the back-end switches”.[11]

4Out-of-band: the network you count in RJ45s

Out-of-band is 1 Gb RJ45 for all the nodes, connecting BMC ports, DPU and SuperNIC management ports and switch OOB ports into “NVIDIA SN2201 48-port 1Gb switches”, physically isolated from system users.[2][1] The SN2201 gives 48x 1 GbE plus four 100 GbE uplinks at 448 Gb/s in 1U.[10] Dell sells it alongside its own S3248T-ON in the same role from the same catalog page.[8]

The count is where people go wrong, because they count servers. The SuperPOD RA lists the clients explicitly: the OOB network “connects the management ports of all devices including DGX B300 compute trays, switches, and management servers, storage, networking gear, rack PDUs”.[4] The XDR edition adds the detail that decides the per-node number — OOB reaches “DGX B300 compute trays including system BMCs and BlueField-3 BMCs”, so the DPU has its own BMC on the OOB network and every node is at least two ports, not one.[9] Logically the whole thing is implemented as a dedicated VXLAN rolled into the aggregation or spine layer of each scalable unit.[4][5]

Do the arithmetic for the 32-node pod and the “one SN2201 is fine” instinct dies immediately: 64 node ports, plus a management port for each of the eight compute leaves and four spines, plus two converged switches, plus the OOB switches themselves, plus seven control-plane nodes and the rack PDUs. That is comfortably past 48 before storage is counted. (Counting rule published; the total is mine.)

GPUs
256
Planes
Uplink ratio
Per node1 system BMC + 1 BlueField-3 BMC = 2 x 1 GbE RJ45 (minimum)published
Node ports32 nodes x 2 = 64derived
Switch management ports14 (leaves 8 + spines 4 + converged 2)derived
Total OOB ports78derived
SN2201 switchesceil(78 / 48) = 2derived

SN2201 is 48x 1GbE RJ45 + 4x 100GbE, 448 Gb/s, 1U. The SuperPOD RA also puts rack PDUs on the OOB network, so add a port per PDU before you quote. OOB is implemented as "a dedicated VXLAN" rolled up into the aggregation layer, and is physically isolated from system users.

FAE angle

The BlueField-3 has its own BMC. Half the OOB port shortfalls in the field are a rack cabled for one management port per server and a DPU that needs a second. Count iDRAC plus DPU BMC per Dell node, add the switches and the PDUs, then divide by 48.

DGX B300 XDR Network Fabrics · HGX B300 Logical Architecture
On the OOB tab watch how the count crosses 48 and forces a second SN2201. Then switch to the storage tab and read the 3.2 Tb/s aggregate for the same cluster.
Size converged storage and OOB for a 32-node HGX B300 pod

Same pod as the previous lesson: 32 HGX B300 nodes, 256 GPUs, dual plane compute fabric with 8 leaves and 4 spines, plus seven control-plane nodes.[12]

  1. Converged ports. Rule: two 400 GbE ports per compute and management node to two separate switches.[1] Nodes = 32 + 7 = 39. Ports = 39 x 2 = 78 x 400 GbE, i.e. 39 per converged switch. (Rule published; arithmetic mine.)
  2. Converged switches. Two, minimum, by the rule itself.[1] An SN5600-class pair at 128 logical 400 GbE ports leaves ample room for storage-array ports and uplinks; an SN5400 pair at 64 x 400 GbE does not once uplinks are reserved.[10] (Derived.)
  3. Converged capability check. 40 GB/s per node is what the path can carry.[1] Do not turn it into a storage requirement.
  4. Storage floor. 256 GPUs x approximately 12.5 Gb/s = 3.2 Tb/s aggregate.[3] Per node that is 100 Gb/s = 12.5 GB/s, which sits about three times under the converged 40 GB/s capability. (Reconciliation mine.)
  5. Storage attachment. Storage devices connect at a 1:1 port-to-uplink ratio in the SuperPOD design; if the customer is on the Enterprise RA instead, the storage rides the converged fabric and the vendor must be in the NVIDIA-Certified Storage program.[4][3] If it is PowerScale, the back-end is private with redundant switches and nothing else attached.[11]
  6. OOB ports. 32 nodes x 2 (system BMC + BlueField-3 BMC) = 64.[9] Plus 7 control-plane nodes = 71. Plus switch management ports: 8 compute leaves + 4 spines + 2 converged + 2 OOB = 16, giving 87. Plus rack PDUs and any storage controllers.[4] (Counting rule published; total mine.)
  7. OOB switches. 87-plus ports ÷ 48 usable per SN2201 = 2 switches minimum, and two switches leave only 9 free ports before rack PDUs and storage controllers are added — the lab’s fuller count of 95 leaves one — so quote 3 if the pod is expected to grow.[10]
  8. Customer edge. At least 2x 100GbE DR1 single-mode uplinks peered with BGP.[4]
  9. Say the labels out loud. Published: two ports per node, 12.5 Gb/s per GPU, 40 GB/s per node capability, 48 ports per SN2201, 1:1 storage ratio. Mine: every total.

Episode 4 — Case note: the revised BOM

How it ended

The fix is arithmetic, not hardware: two out-of-band ports per compute node because the DPU has its own BMC,[9] one management port for every switch in every fabric, then the control-plane nodes and the rack PDUs, and only then divide by the 48 usable ports on an SN2201.[10][4] The pod needs three, not one, and the operator relabels a rack before his shift ends. What you say: “Nothing was wrong with the compute design — three networks were counted once each instead of port by port, and this is the cheapest week to fix that.” Then facilities gets a copy of the BOM and their power engineer asks one question: where are the optics?

Lab

Mutating steps ahead. Needs a maintenance window, out-of-band access (BMC/iDRAC/rshim console) and a rollback path. Record the pre-flight inventory before changing anything. Never on a production host.

Measure one real north-south path and count real RJ45s. Mutating: this mounts a filesystem and writes a test file, so every mutating step names its rollback.

  1. Pre-flight inventory. Read-only. 
    ip -br addr
ibdev2netdev
mount | grep -c nfs
df -h /mnt 2>/dev/null
    Record which interface belongs to the BlueField-3 and confirm the card is in NIC mode before you plan anything through it. Write the current mount list to a file so you can compare afterwards: mount > ~/premount.txt.
  2. Confirm the DPU is in NIC mode. Read-only. Do not change the mode in this lab; a mode change is a firmware operation with a power-cycle and belongs to the DOCA course, not here.
  3. Mount an NFS export across that path. MUTATING. 
    sudo mkdir -p /mnt/ra-lab
sudo mount -t nfs <server>:/<export> /mnt/ra-lab
    Expected: the mount appears in mount and df -h /mnt/ra-lab shows the server’s capacity. Rollback: sudo umount /mnt/ra-lab && sudo rmdir /mnt/ra-lab, then diff mount against ~/premount.txt to prove you left nothing behind. If the mount hangs, the path is wrong or the export is not permitted — do not force it with -o soft on a shared server.
  4. Measure achieved throughput. MUTATING - writes a file. 
    sudo fio --name=ra --directory=/mnt/ra-lab --rw=read --bs=1M --size=4G --numjobs=4 --time_based --runtime=60 --group_reporting
    Expected: an aggregate READ: bw= line in MB/s or GB/s. Rollback: sudo rm -f /mnt/ra-lab/ra.* removes every file fio created; verify with ls /mnt/ra-lab. Run a read-only workload first if the export is shared with anyone else.
  5. Convert and compare. Take the measured aggregate in GB/s and compute how many such paths a node would need to reach the RA’s converged capability of up to 40 GB/s per node.[1] Then compute what the Enterprise RA storage floor asks of an eight-GPU node — 100 Gb/s, or 12.5 GB/s — and say which of the two your measurement is closer to.[3]
  6. Count real out-of-band ports. Read-only. On one Dell node, physically count the management RJ45s in use: the iDRAC port and the BlueField-3 BMC port.[9] Multiply by 32 for a pod, add the switch management ports and PDUs, and compare against 48 usable ports on one SN2201.[10]
  7. Tear down and verify. MUTATING - this is the rollback step. sudo umount /mnt/ra-lab && sudo rmdir /mnt/ra-lab, then diff <(mount) ~/premount.txt and confirm no difference remains. Record the measured GB/s and the counted RJ45s in your notes; those two numbers are the ones you will quote from memory.
  8. Optional, customer lab only. Read the converged switch port allocation on a real cluster and check whether each node really has one port on each of two switches, as the RA requires.[1] Read-only; change nothing.

Retrieval check

9 questions from memory. Answer before looking anything up; misses become flashcards.

Explain it to a Dell SE

A Dell account manager says the customer only cares about the GPU fabric and wants to strip the other networks out of the quote to win on price. Explain in five sentences what the other three networks actually carry and what number you would size each one from.

14 flashcards for this lesson — 0 in deck. Spaced review lives at /review.

Sources

Facts in this lesson were checked against HGX AI Factory RA Networking Physical Topologies re-fetched 2026-09-09 (converged two 400 GbE ports to two separate switches up to 40 GB/s per node; OOB description; 1024 interfaces per fabric; 50 percent single-plane statement); HGX AI Factory NVIDIA-Certified Storage re-fetched 2026-09-09 (approximately 12.5 Gb/s per GPU scaling linearly; 16 GPU cluster around 200 Gb/s); DGX SuperPOD B300 Spectrum-4 and DC busbar Network Fabrics re-fetched 2026-09-09 (1:1 storage port to uplink; near 4:3 DGX oversubscription; I/O per node must exceed 40 GBps; SN5600/SN5600D storage; SN5600D in-band; SN2201 OOB; 2x100GbE DR1 edge); remaining figures as recorded in content/research/ra/part1.md fetched 2026-09-07. Dates are when each page was fetched.

  1. Networking Physical Topologies — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-09
  2. Networking Logical Architecture — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07
  3. NVIDIA-Certified Storage — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-09
  4. Network Fabrics — DGX B300 SuperPOD Spectrum-4 Ethernet and DC Busbar Power RA · fetched 2026-09-09
  5. Network Fabrics — DGX GB200 NVL72 SuperPOD Reference Architecture · fetched 2026-09-07
  6. Networking Physical Topologies — NVIDIA NVL72 AI Factory (GB300) Enterprise RA · fetched 2026-09-07
  7. Networking Physical Topologies — NVIDIA HGX AI Factory (H100 H200 B200) Enterprise RA · fetched 2026-09-07
  8. AI Networking Switches — Dell USA catalog · fetched 2026-09-07
  9. Network Fabrics — DGX B300 SuperPOD with Quantum-X800 InfiniBand and AC Power RA · fetched 2026-09-07
  10. NVIDIA Spectrum Ethernet Switches — product family port matrix · fetched 2026-09-07
  11. Dell PowerScale: Ethernet Back-End Network Overview (H16346.8) · fetched 2026-09-07
  12. Components — NVIDIA HGX AI Factory (B300) Enterprise RA · fetched 2026-09-07

The same idea elsewhere

Other lessons that cover this ground, sometimes from another course's angle.