Links that come up: breakout, FEC and auto-negotiation
S1·E3Twenty-four up, eight refusing · Dell lab, Round Rock, 02:10 on the pilot-rack bring-up
Before you read: what do you already know?
3 quick questions. Wrong answers are fine and expected; trying first makes the lesson stick.
After this lesson you can
- Configure a breakout port with NVUE and predict the sub-port names and the resulting usable port count.
- Apply the adjacent-even-port rule and write the /etc/cumulus/ports.conf equivalent of an NVUE breakout.
- Choose an FEC mode from the media type and the peer ASIC and justify it from the documented rule.
- Decide when auto-negotiation must be enabled and when speed must be forced instead.
- Verify a link with the NVUE status, BER and transceiver views.
Episode 3 — Twenty-four up, eight refusing
The night-shift operator meets you at the cage door with a label maker in one hand and the rack elevation in the other. Handover to the customer’s network lead is at 08:00 and eight of the thirty-two host links will not come up. The rest of the rack is clean, which is the detail that matters: same optics, same cage type, different configuration path. Somebody has already started swapping transceivers, and you ask them to stop.
The failing ports came from an 8x breakout of one cage, and on an SN5600 an 8x breakout requires the adjacent even-numbered port to be disabled - which is exactly why the documented maximum is 256 ports at 100G rather than 512.[1] The inherited runbook also pins fec auto explicitly on every port, and the documentation warns that doing so causes a link flap on every commit.[2] On the two spine links you find auto-negotiation forced off: a reflex from SN4700 QSFP-DD ports, where the speed must be forced, applied to a Spectrum-4 to Spectrum-4 PAM4 link, where auto-negotiation must be enabled.[2][1]
Layer 1 has rules like these because a modern link negotiates far more than speed. FEC mode, precoding and equalisation are settled during training between two ASICs that each hold several valid options, and if the two ends do not agree the link either never trains or trains and then errors.[1]
Swap the configuration before you swap the optic.
Segment 1 starts at the cage.
1Breakout: one cage, many ports
A breakout splits the lanes of one cage into several logical ports. In NVUE it is one setting: nv set interface <interface-id> link breakout <mode-id> where the mode is 1x, 2x, 4x or 8x, with 1x as the default; this syntax replaced the older 4x10G style in Cumulus Linux 5.4.0.[3][2] The sub-ports that appear are named by suffix - a 4x breakout of swp1 yields swp1s0, swp1s1, swp1s2 and swp1s3.[2]
The canonical sequence stages both the split and the port state, then applies once:
nv set interface swp1 link breakout 4x
nv set interface swp1s0-3 link state up
nv config applyWithout NVUE the same change is a file edit: put 1=4x in /etc/cumulus/ports.conf, disable the adjacent port where the platform requires it, then systemctl reload switchd.service.[1] The adjacent-port rule is the part that bites: “Certain switches, such as the SN2700, SN4600, and SN4600c, require that you disable the subsequent even-numbered port when you configure a breakout port for 4x or 8x”, and the SN5600 shares that requirement for 8x.[1] NVUE performs the disable automatically, and SN3700, SN3700c, SN2201, SN2010 and SN2100 do not have the restriction at all.[2]
That rule explains the port maths. An SN5600 gives 64 ports at 800G native, 128 at 400G with 2x, 256 at 200G with 4x, and 256 at 100G or 50G using 8x on only 32 cages because each 8x consumes its neighbour.[1] An SN5400 gives 64x400G native, 128x200G, 256x100G and 256x50G from 32 odd ports, plus its two SFP28 bonus ports at 25G and 10G.[1] An SN2201 gives 4x100G native, 8x50G, 4x40G and 16x25G or 16x10G from its four QSFP28 cages.[9][1] On Spectrum-6, Cumulus Linux 5.18 notes that on the new SN6600_LD platform “1x break out at 800G” is unsupported and “interfaces are 2x split by default”.[6]
link breakout 1x | 2x | 4x | 8x
- nv set interface <interface-id> link breakout <mode-id>, with mode-id one of 1x, 2x, 4x, 8x. Default is 1x.
- This syntax arrived in Cumulus Linux 5.4.0. Older releases used forms like 4x10G — a customer config that still says 4x10G predates 5.4.0 and will not paste in.
- The mode is a count, not a speed: 4x on an 800G OSFP cage gives 4 × 200G, 4x on a 400G QSFP-DD cage gives 4 × 100G.
- nv unset interface <id> link breakout returns the cage to 1x.
FAE angle: "the port disappeared after breakout" is usually the mode arriving without the sub-ports being brought up — swp1 stops existing the moment swp1s0-3 exist.
NVUE interface set/unset reference · Switch Port Attributes, Cumulus Linux 5.18
swp1s0 swp1s1 swp1s2 swp1s3nv set interface swp1 link breakout 4x nv set interface swp1s0-3 link state up # leave FEC unset — do NOT run: nv set interface swp1s0 link fec auto (link flap on commit) # media: DAC CA-25G-L (long, ≥5 m) — IEEE 802.3by long DAC class. Requires RS FEC. nv set interface swp1s0 link mtu 9216 # 552-9216, the one setting you must set manually nv config diff nv config apply
# /etc/cumulus/ports.conf 1=4x sudo systemctl reload switchd.service
Then verify (layer-1 triage)
nv show interface nv show interface swp1s0 link # speed, FEC, auto-negotiate state nv show interface status nv show interface --view ber # effective BER, raw BER, symbol errors nv show interface swp1s0 link phy health # per-lane SNR, lane0..lane7 nv show platform transceiver nv action reset platform transceiver <transceiver-id>
--view ber gives effective BER, raw BER and symbol errors; nv show interface --ber and the Beta --dom view are new in 5.18.0. Monitoring Interfaces and Transceivers with NVUE 5.18
2FEC: let the default do its job
Forward error correction on Spectrum switches has four settings - auto (the default), rs, baser and off.[2][3] The standards behind the names: RS FEC is “IEEE 802.3 Clause 108 (CL108) on individual 25G channels and Clause 91 on 100G”, and Base-R or FireCode follows “IEEE 802.3 Clause 74 (CL74)”.[1] IEEE 802.3by defines three 25G direct-attach classes with a FEC requirement each: CA-25G-L (long, requires RS FEC), CA-25G-S (short, requires Base-R) and CA-25G-N (no FEC).[1]
Three documented rules do most of the work in the field. First, the trap: “Do not explicitly configure the fec auto option on the switch as this leads to a link flap whenever you run net commit or ifreload -a.”[2] Auto is already the default, so a runbook line setting it turns every routine commit into a flap. Second, optics that override you: on 100GBASE-DR1 “you must either set the FEC setting to off or leave it unset for the link to function”, while on 400GBASE-DR4 “the switch automatically enables FEC if it is set to off”.[1] Third, the pairing rule: “If auto-negotiation is disabled on 100G and 25G interfaces, you must set FEC to off, RS, or BaseR to match the neighbor.”[1]
3Auto-negotiation: the reflex that has to be unlearned
Auto-negotiation is on by default and is disabled with nv set interface swp1 link auto-negotiate off.[2][3] Two platform rules point in opposite directions and both are documented. On Spectrum-4: “When you connect an NVIDIA Spectrum-4 switch to another NVIDIA Spectrum-4 switch with PAM4 modulation, you must enable auto-negotiation.”[1] On Spectrum-3: the SN4700 and SN4410 do “not support auto-negotiation on QSFP-DD 400G transceiver modules. You need to force set the speed.”[2] An engineer whose muscle memory comes from forcing 400G ports on an SN4700 will apply the same habit on an SN5600 spine link and break it.
When you do force, the SN5600 adds a caveat on ports 10 through 50: “Rx precoding setting must be the same between local and peer ports to get the optimal Signal-Integrity of the link”.[1] Two other settings round out the interface surface: nv set interface <id> link duplex accepts full or half with full the default, and nv set interface <id> link mtu accepts a value “between 552 and 9216. The default setting is 9216”.[3] MTU is called out as “the only interface setting that you must set manually”, eth0 defaults to 1500, and a VNI must be 50 bytes below the physical MTU.[1][4]
Finally, know what a protected port looks like. Link flap protection is on by default and logs a string of the form “link flapped more than 3 times in the last 60 seconds, setting protodown”, surfaced as protodown_reason <linkflap>.[4] A port in that state was not killed by the switch at random; something below it is flapping.
FEC and auto-negotiation
- FEC default is auto; the modes are rs (Reed-Solomon, IEEE 802.3 Clause 108/91), baser (Base-R / FireCode, Clause 74) and off.
- Do not explicitly configure fec auto: it flaps the link on every net commit / ifreload -a. Leave it unset instead.
- Auto-negotiation default is on; nv set interface swp1 link auto-negotiate off turns it off. With it off on 100G and 25G you must set FEC to off, RS or BaseR to match the neighbor.
- 100GBASE-DR1: FEC off or unset. 400GBASE-DR4: the switch enables FEC even if you set off.
- Spectrum-4 to Spectrum-4 over PAM4 requires auto-negotiation enabled — the opposite of the SN4700/SN4410 QSFP-DD 400G habit, where auto-negotiation is unsupported and you force the speed.
- MTU: 552-9216, default 9216 (eth0 is 1500), and the only interface setting you must set manually. A VXLAN VNI MTU must be 50 bytes below the physical MTU.
FAE angle: the OS10/SONiC reflex — force speed, force FEC — is the exact wrong move on a Spectrum-4 PAM4 switch-to-switch link. Check order: breakout/adjacent port, then explicit fec auto, then FEC matching with auto-negotiation off, then the PAM4 auto-negotiation rule, then Rx precoding on SN5600 ports 10-50.
Switch Port Attributes, Cumulus Linux 5.18 · Switch Port Attributes, Cumulus Linux 5.13 · NVUE interface set/unset reference · Interface Configuration and Management 5.13
swp1s0 swp1s1 swp1s2 swp1s3 swp1s4 swp1s5 swp1s6 swp1s7nv set interface swp1 link breakout 8x nv set interface swp1s0-7 link state up nv set interface swp1s0 link auto-negotiate off nv set interface swp1s0 link speed 100G # ⚠ speed token form not verified in the fetched docs nv set interface swp1s0 link fec rs # media: 100GBASE-DR1 optic — Set FEC to off, or leave it unset — anything else keeps the link down. nv set interface swp1s0 link mtu 9216 # 552-9216, the one setting you must set manually nv config diff nv config apply
# /etc/cumulus/ports.conf 1=8x 2=disabled # adjacent even port — NVUE does this for you sudo systemctl reload switchd.service
Then verify (layer-1 triage)
nv show interface nv show interface swp1s0 link # speed, FEC, auto-negotiate state nv show interface status nv show interface --view ber # effective BER, raw BER, symbol errors nv show interface swp1s0 link phy health # per-lane SNR, lane0..lane7 nv show platform transceiver nv action reset platform transceiver <transceiver-id>
--view ber gives effective BER, raw BER and symbol errors; nv show interface --ber and the Beta --dom view are new in 5.18.0. Monitoring Interfaces and Transceivers with NVUE 5.18
4Proving the link, not assuming it
Cumulus Linux 5.18 gives a layer 1 view that ends most arguments.[5]
nv show interface status
nv show interface --view counters
nv show interface --view ber
nv show interface swp1s0 link phy health
nv show platform transceiver
nv show interface swp1s0 transceiver
nv action clear interface counters[5] The --view ber output is a tabular summary with effective BER, raw BER and symbol errors; link phy health reports per-lane SNR across lane0 to lane7; the transceiver views give temperature, voltage, rx-power, tx-power, tx-bias-current and alarm thresholds.[5] The --ber PHY-health statistics and the --dom transceiver view (Beta) are new NVUE surfaces in 5.18.0.[6] On the SN5610 only, you can even set an optic temperature setpoint between 30 and 80 C with nv set platform transceiver <transceiver-id> temperature setpoint.[5]
One planning note that belongs here rather than in the telemetry module: high-frequency telemetry does not work on ports using 8 lanes, so on Spectrum-4 the native 800G ports swp1-swp64 need breakout before HFT can watch them.[7] Breakout is therefore not only a port-count decision; it decides what you can instrument later.[8]
Goal: an SN5600 leaf must present eight 100G ports to hosts using a CA-25G-class-equivalent copper break-out cable from cage swp1, with auto-negotiation left at its default.
- Decide the mode. 100G from an 800G OSFP cage is
8x, which on the SN5600 consumes the adjacent cage - so plan on odd cages only (swp1, swp3, swp5 …) and expect a maximum of 256 such ports across the switch.[1] - Stage the change:
NVUE disables the adjacent even port for you; the equivalent by hand would benv set interface swp1 link breakout 8x nv set interface swp1s0-7 link state up nv config apply1=8xin/etc/cumulus/ports.confwith port 2 set todisabled, followed bysystemctl reload switchd.service.[1] - Leave FEC alone. The default is
autoand explicitly writingfec autowould flap these ports on every future commit.[2] Leave auto-negotiation at its defaultonas well, since nothing here is an SN4700 QSFP-DD port.[2] - Verify:
Expect eightnv show interface status nv show interface --view ber nv show platform transceiverswp1sNentries up at 100G, low effective BER, and a transceiver entry for cage 1.[5] - If a sub-port stays down, check the peer’s FEC first: with auto-negotiation disabled on the host side, “you must set FEC to off, RS, or BaseR to match the neighbor”, and a DAC in class CA-25G-L requires RS while CA-25G-S requires Base-R.[1]
Rollback: nv unset interface swp1 link breakout then nv config apply returns the cage to a single 800G port and removes the sub-ports.[3]
Same goal on an SN5400 leaf, target speed 100G from cage swp5, peer is a 100GBASE-DR1 optic on a host.
- Mode: 100G on an SN5400 is a ____ breakout, and the documented maximum is ____ ports of 100G.[1]
- Commands:
nv set interface swp5 link breakout ____ nv set interface ____ link state up nv config apply - FEC: because the far end is 100GBASE-DR1 you must set FEC to ____ or leave it ____ for the link to function.[1]
- Verify with
nv show interface statusandnv show interface --view ____.[5] - Rollback command if the port map is wrong:
nv unset interface swp5 link ____thennv config apply.[3]
An SN5600 spine port swp30 connects to another SN5600 leaf at 400G over PAM4. The link trains, drops, and retrains; the log shows the port going protodown after several flaps. The customer’s runbook contains two lines applied to every uplink: nv set interface swp30 link auto-negotiate off and nv set interface swp30 link fec auto.
Produce: (a) which of the two lines violates a documented rule and which rule, (b) which additional SN5600-specific check applies to swp30 if the port stays in force mode, (c) the exact commands to return the port to a documented-safe state, (d) the command you would run to decide whether the media is marginal rather than mis-configured, and (e) what the protodown state tells you about who set it.
Acceptance criteria: both configuration lines are addressed with a cited reason; the fix uses nv unset rather than setting a new explicit value; you name nv show interface --view ber or link phy health for the media question; and you state that link flap protection - on by default - set protodown after roughly three flaps in 60 seconds.[1][2][4][5]
One change at a time, then read
The fec auto line comes out of the runbook - auto is the default and writing it converts every future commit into a flap.[2] Auto-negotiation goes back on for the Spectrum-4 spine links.[1] The port map is rebuilt on odd cages only, so each 8x has its neighbour to spend.[1] Then nv show interface --view ber before anyone declares victory, because a link can be up and still correcting symbol errors.[5] What you say at handover: “No part was faulty. The configuration described a switch that cannot exist.” The operator labels each disabled cage NOT A PORT. ASK THE FAE. and goes home. At 08:04 the network lead looks up from a full page of notes: which software version is this rack running, and can we upgrade just the switches?
Lab
Mutating on the host side. Every step names its rollback; do not touch switch firmware.
- Pre-flight inventory on the Dell lab host. Record before changing anything:
Expected: current speed, duplex, auto-negotiation state, driver and firmware version, and the active FEC encoding. Save this output - it is the rollback reference for step 3.ip -br link sudo ethtool ens1f0np0 | head -20 sudo ethtool -i ens1f0np0 sudo ethtool --show-fec ens1f0np0 - Read the installed optic and classify it:
Expected: identifier, connector, vendor name and part number, and for a DAC the cable length. Write down which 25G DAC class it corresponds to - CA-25G-L requires RS FEC, CA-25G-S requires Base-R, CA-25G-N requires none - and therefore which FEC a 25G peer would need.[1]sudo ethtool -m ens1f0np0 | head -30 - Optional and mutating: change the FEC mode on one host interface that is not carrying lab traffic, and put it back.
Expected: the link drops and re-trains. Rollback: re-apply the exact encoding you recorded in step 1, for examplesudo ethtool --set-fec ens1f0np0 encoding rs sudo ethtool --show-fec ens1f0np0sudo ethtool --set-fec ens1f0np0 encoding auto, and confirm the interface returns to its original state withethtool ens1f0np0. If the link does not come back, the peer is now mismatched - restore first, investigate second.[1] - Optional, customer or partner lab with a live SN5600 and a 400G breakout port. Read only:
Expected: effective and raw BER, per-lane SNR for lane0 to lane7, and module temperature and power. Compare the per-lane SNR spread across the four sub-ports; one outlying lane is a cable or connector story, all four equally poor is a media or FEC story.[5]nv show interface --view ber nv show interface swp1s0 link phy health nv show platform transceiver
Goal: perform a breakout and read the link attributes on a simulated switch, then contrast the mental model with Dell Enterprise SONiC. Cumulus VX is no longer released as a standalone image and containerlab has no working Cumulus kind, so the Cumulus half runs in DSX Air.[6]
- In a DSX Air simulation, open
leaf01and record the starting state:
Expected:nv show interface swp1 linkauto-negotiateon,fecnot explicitly set,mtu 9216. Write these down - this is your rollback target.[3] - Break out the port and bring the sub-ports up:
Expected:nv set interface swp1 link breakout 4x nv set interface swp1s0-3 link state up nv config apply nv show interfaceswp1s0throughswp1s3appear. If they do not, check thatnv config applyreturned without error and re-readnv show --pending.[1] Rollback (run it before moving on):nv unset interface swp1 link breakoutthennv config apply, and confirmswp1is back as a single port.[3] - Read the FEC and auto-negotiation state again and predict, in writing, what would change if you set
fec rs. Do not set it - a simulated node has no real PHY, so you would be reading your own configuration back, not a negotiated result.[2] - In
~/containerlab, boot a Dell Enterprise SONiC 4.5.1 node and perform the equivalent breakout withconfig interface breakouton a supported port, thenshow interface breakout. Expected: SONiC asks you to confirm the mode and reports the resulting sub-interfaces. If the command is not present on your image, captureshow versionand note it. - Deliverable: a two-column mental-model diff. Column one, what NVUE does for you (adjacent-port disable, one apply for both changes, defaults that are already correct). Column two, what SONiC makes you declare. Add one line stating which fabric behaviours you could not test here at all, and why - the virtual switch has no Spectrum ASIC.[6]
Retrieval check
10 questions from memory. Answer before looking anything up; misses become flashcards.
Explain it to a Dell SE
Explain to a Dell lab technician in five sentences why forcing speed and FEC - the habit that works on OS10 - is the wrong first move on a Spectrum-4 link that will not come up.
Sources
Facts in this lesson were checked against Cumulus Linux 5.18 Switch Port Attributes (re-fetched 2026-09-07), Cumulus Linux 5.13 Switch Port Attributes, NVUE interface reference, Cumulus Linux 5.18 What's New. Dates are when each page was fetched.
- Switch Port Attributes | Cumulus Linux 5.18 · fetched 2026-09-07
- Switch Port Attributes | Cumulus Linux 5.13 · fetched 2026-09-07
- NVUE Reference: Interface set/unset commands · fetched 2026-09-07
- Interface Configuration and Management | Cumulus Linux 5.13 · fetched 2026-09-07
- Monitoring Interfaces and Transceivers with NVUE | Cumulus Linux 5.18 · fetched 2026-09-07
- What's New | Cumulus Linux 5.18 · fetched 2026-09-07
- High Frequency Telemetry | Cumulus Linux 5.18 · fetched 2026-09-07
- NVIDIA Spectrum SN5600 Series Switches Datasheet (Dell-hosted) · fetched 2026-09-07
- NVIDIA Spectrum SN2201 Switch Datasheet (Dell-hosted) · fetched 2026-09-07
- Cumulus Linux 5.18 Release Notes · fetched 2026-09-07
The same idea elsewhere
Other lessons that cover this ground, sometimes from another course's angle.