The SRH on the wire

Episode 2 — Ping is fine, the backups are not

The situation · Bridge call at 02:40, customer NOC on one line, you on hotel Wi-Fi

The change window closed four hours ago and the night-shift operator is reading you the same two facts on a loop: every SID pings, and the replication job has not moved a study since the new policy went in. Someone wants the pilot rolled back before the stand-up, five weeks out from cutover. The Dell SE is on mute with a vending-machine coffee. The network lead says “show me the counter” and starts a fresh page. You ask what actually changed in the window. The answer is small: the fourth segment, the one that steers around the box past refresh.

The SRH exists so the list of instructions can ride inside a header IPv6 already knows how to carry. RFC 8754 gives it Routing Type 4 and an eight-byte base - Next Header, Hdr Ext Len, Routing Type, Segments Left, Last Entry, Flags, Tag - followed by the segments themselves as 128-bit addresses.^[1] A device that does not understand it still forwards the packet on the destination address, which is the whole reason the design is deployable.^[1]^[8] The price is arithmetic that never sleeps: 40 bytes of outer IPv6, 8 of SRH base, 16 per segment, spent against a floor where “IPv6 requires that every link in the Internet have an MTU of 1280 octets or greater.”^[1]^[2]

Ping proves the SID is reachable. It never proves the packet you care about fits.

Small packets fit, large ones do not, so you ask for a capture from the receiver and start reading bytes.^[1]

1Eight bytes, then addresses

The SRH is an IPv6 routing header. RFC 8754 says so plainly: “Routing headers are defined in [RFC8200]. The Segment Routing Header (SRH) has a new Routing Type (4).”^[1] That matters for two reasons. Any middlebox that already has an opinion about routing headers will have one about the SRH, and any device that does not understand it is still allowed to forward the packet on the destination address.^[1]^[8]

The fixed part is eight bytes, in this order: Next Header (8 bits), Hdr Ext Len (8 bits), Routing Type set to 4 (8 bits), Segments Left (8 bits), Last Entry (8 bits), Flags (8 bits), and Tag (16 bits).^[1] After that come the segment list entries, each a 128-bit IPv6 address, and after those, optional TLVs.^[1]

Three of those fields do arithmetic you can check by hand. Hdr Ext Len is the length of the header in 8-octet units not counting the first 8 octets, so for n segments and no TLVs it is exactly 2n.^[1] Last Entry “contains the index (zero based), in the Segment List, of the last element of the Segment List.”^[1] And Segments Left counts down as the packet is forwarded.^[1] The spec ties the first two together in the endpoint pseudocode with a line worth memorising: S09. max_last_entry = ( Hdr Ext Len / 2 ) - 1.^[1]

Mutability is spelled out and is a useful debugging constraint: “Next Header, Header Ext Len, and Routing Type are not mutable while Segments Left is mutable.”^[1] If a capture on hop three shows a different Hdr Ext Len than the same flow on hop one, something rewrote the header rather than processed it.

Here is a three-segment SRH base in hex, carrying an inner IPv6 packet:

29 06 04 02 02 00 00 00
|  |  |  |  |  |  +--+-- Tag = 0x0000
|  |  |  |  |  +-------- Flags = 0x00
|  |  |  |  +----------- Last Entry = 2   (3 segments, zero based)
|  |  |  +-------------- Segments Left = 2 (just left the headend)
|  |  +----------------- Routing Type = 4 (SRH)
|  +-------------------- Hdr Ext Len = 6  (3 segments x 2 units)
+----------------------- Next Header = 41 (0x29, IPv6 inner)

2The list is stored backwards

This is the sentence that costs engineers an afternoon: “The Segment List is encoded starting from the last segment of the SR Policy. That is, the first element of the Segment List (Segment List[0]) contains the last segment of the SR Policy, the second element contains the penultimate segment of the SR Policy, and so on.”^[1]

So for a policy A then B then C, the packet leaves the headend with the destination address set to A, Segment List[2] holding A, Segment List[1] holding B and Segment List[0] holding C, with Last Entry 2 and Segments Left 2.^[1] Every endpoint decrements Segments Left and reads the slot it now points at, which walks the array downwards toward index 0 while the packet walks the path forwards.

Linux hides the reversal from you, and that is exactly why people get confused when they then read a capture. The ip -6 route ... encap seg6 grammar documents segs SEGMENTS only as a “list of comma-separated IPv6 addresses”.^[6] What the man page does not say, and what you will confirm for yourself in the lab, is that you write them in SR Policy order and iproute2 reverses them into the SRH. The canonical example from the man page is:

ip -6 route add 2001:db8:1::/64 encap seg6 mode encap \
    segs 2001:db8:42::1,2001:db8:ffff::2 dev eth0

^[6] Type that, capture it, and you will see 2001:db8:ffff::2 at index 0 - the last segment - and 2001:db8:42::1 in the destination address. Nothing is wrong; the CLI speaks policy order and the wire speaks reverse order.

overhead 96 B1496 B fits MTU 1500

96 B overhead + 1400 B inner = 1496 B · red line = link MTU 1500 · grey line = the 1280-octet IPv6 floor · largest inner packet that fits: 1404 B

0000  60 00 00 00 05 b0 2b 40 fc bb bb bb 01 00 00 00 
0010  00 00 00 00 00 00 00 01 fc bb bb bb 00 01 00 00 
0020  00 00 00 00 00 00 00 00 29 06 04 02 02 00 00 00 
0030  fc bb bb bb 00 03 00 00 00 00 00 00 00 00 00 00 
0040  fc bb bb bb 00 02 00 00 00 00 00 00 00 00 00 00 
0050  fc bb bb bb 00 01 00 00 00 00 00 00 00 00 00 00 
…… 1400 B of inner packet not shown

SR policy — in policy order, first segment first (this is the order you type after segs)

inner packet 1400 Blink MTU

Full SRH

H.Encaps (RFC 8986 §5.1): push an outer IPv6 header carrying the whole segment list. The first segment is in the DA and again in the SRH.

ip -6 route add 2001:db8:1::/64 encap seg6 mode encap segs A,B,C dev eth1

Overhead: 40 B outer IPv6 + 8 B SRH base + 16 B × 3 = 96 B. The same 3-segment path as a full SRH costs 96 B; as one uSID carrier it costs 40 B.

The first segment appears twice: once in the outer DA and once at Segment List[Last Entry].

source

Predict before you look: Hdr Ext Len, Last Entry, Segments Left?

Destination Address — 16 B — the active SID

fcbb:bbbb:1:: (Segment List[Segments Left] will replace it at the next endpoint)

Always holds the segment the packet is being forwarded to right now.

This is why a transit node needs no SRv6 support at all: it does a plain IPv6 route lookup on this address (RFC 8754 §3.2). At an endpoint, line S15 decrements Segments Left and line S16 copies Segment List[Segments Left] into this field.

FAE angle: DOCA Flow's "copy the active SID to the IPv6 destination address" action — src field_string "outer.srv6.segments", dst "outer.ipv6.dst_ip", width 128 — is exactly line S16 done in hardware. That single copy is the uN/End primitive on a ConnectX or BlueField.

DOCA Flow 3.5.0 guide

Click any field in the packet (or tab to it and press Enter) to see what it does. DOCA Flow accounting: each distinct SRH configuration costs 128 B of actions memory (DOCA_FLOW_SRV6_ACTION_MEM_SIZE), and doca_flow_external_action_srv6_register() must be called before doca_flow_init().

Build the three-segment list, turn on the hex view, and find the last segment at index 0. Then drag the inner payload to 1400 bytes on a 1500-byte link and watch the packet cross the MTU.

3What an endpoint actually does, and the two ways it says no

An SR segment endpoint node is one whose local SID matches the packet’s destination address, and RFC 8754 section 4.3.1.1 gives its processing as numbered pseudocode.^[1] The lines to know are:

S09. max_last_entry = ( Hdr Ext Len / 2 ) - 1 - the bound the header must respect.^[1]
S15. Decrement Segments Left by 1.^[1]
S16. Copy Segment List[Segments Left] from the SRH to the destination address of the IPv6 header.^[1]

That pair is the SRv6 realisation of RFC 8402’s NEXT primitive, and it is the only thing that moves a packet from one segment to the next.^[8]

There are exactly two ways an endpoint refuses. If Last Entry > max_last_entry or Segments Left > (Last Entry + 1), the node must “Send an ICMP Parameter Problem, Code 0, message to the Source Address, pointing to the Segments Left field, and discard the packet.”^[1] That is a malformed-header answer: the sender built something inconsistent. If the Hop Limit is 1 or less, the node sends “ICMP Time Exceeded – Hop Limit Exceeded in Transit” and discards.^[1] That is a path-length answer: too many hops, or a loop.

Learn to read those two errors as different diagnoses. Parameter Problem Code 0 blames the headend that built the packet. Time Exceeded blames the topology. A third code appears later in the course and means something entirely different: RFC 8986 defines ICMP Parameter Problem Code 4, SR Upper-layer Header Error, for a permitted-behaviour question about the inner header, not a malformed SRH.^[3]

3 segments · 96 B overhead

hop 1 / 6

containerlab Linux/FRR · Full SRH

1. PE1 · Linux + FRR

SR source nodePUSH

Encapsulate: outer Destination Address = the first segment, fcbb:bbbb:2::1. The SRH carries 3 entries in reverse — Segment List[0] = fcbb:bbbb:4::100, the LAST segment of the policy.

The SR source node encapsulates: it pushes an outer IPv6 header whose Destination Address is the FIRST segment, and (for the uncompressed encodings) an SRH holding the rest. RFC 8402 calls this PUSH. The inner packet's Hop Limit is decremented once, here.

Packet state after this hop

Destination Address fcbb:bbbb:2::1

Segments Left 2

SRH Last Entry 2 · Hdr Ext Len 6 · Routing Type 4

Encapsulation overhead 96 B

Predict: what will the Destination Address be after the next hop? (optional, saved)

FAE angle: "SR supports per-flow explicit routing while maintaining per-flow state only at the ingress nodes to the SR domain" (RFC 8402 §1). The fabric holds no policy state — that is the whole scaling argument against RSVP-TE.

sysctl -w net.ipv6.conf.all.seg6_enabled=1 net.ipv6.conf.eth1.seg6_enabled=1
ip -6 route add 2001:db8:cafe::/64 encap seg6 mode encap \
  segs fcbb:bbbb:2::1,fcbb:bbbb:3::1,fcbb:bbbb:4::100 dev eth1
# mode encap.red for the reduced SRH. segs are written in SR-policy order;
# the kernel reverses them into the Segment List.

Full SRH

H.Encaps / ip -6 route … mode encap. All n segments ride in the SRH, stored in reverse: Segment List[0] holds the LAST segment of the policy. Segments Left = n-1, Last Entry = n-1, Hdr Ext Len = 2n. Overhead 40 + 8 + 16n.

Same path, three encodings: full SRH 96 B · reduced 80 B · uSID 40 B. The IPv6 floor is 1280 octets, so this is the number the MTU conversation turns on.

RFC 8986 §4–§5 — behaviors · encoding source · PUSH / NEXT / CONTINUE

Four FRR nodes around a vrnetlab/dell_sonic:4.5.1 node that is pure IPv6 transit. Three segments.

Predict the destination address and Segments Left before each step. Then switch the encoding to reduced and walk the same path - the field values change, the path does not.

4TLVs, HMAC, and what the Flags octet really says

After the segment list the SRH may carry TLVs.^[1] Two matter at this level. The Padding TLVs exist to satisfy alignment, and the HMAC TLV is type 5 with an 8n alignment requirement.^[1] The HMAC TLV carries a D-bit, where 1 means “Destination Address verification is disabled due to use of a reduced Segment List”, four octets of opaque HMAC Key ID, and the keyed HMAC itself “in multiples of 8 octets, at most 32 octets.”^[1] Note what that D-bit implies: reducing the SRH removes the first segment, so the verifier cannot reconstruct the original destination address, and the spec has to say so explicitly.^[1]

The Flags octet is a small trap. In RFC 8754 itself all eight bits are “U: Unused and for future use. MUST be 0 on transmission and ignored on receipt.”^[1] But RFC 9259 later claimed bit 2 of that octet as the O-flag for OAM, “used as a marking bit in user packets to trigger telemetry data collection and export at the segment endpoints.”^[5] Implementation of the O-flag is optional, and a node that does not support it simply ignores it.^[5] So a set flag bit in a 2026 capture is not automatically a broken sender - check which document the sender implements before you file a bug.

Security is the other reason to care about the SRH at this level. The deployment model in section 5 is titled “Intra-SR-Domain Deployment Model” with a subsection on securing the domain, and AH is explicitly discussed rather than adopted as the protection mechanism.^[1] The practical control is the two-level access control of section 5.1: at the domain edge, an inbound IACL on every external interface that drops any packet whose destination address falls in your SID block S/s; and inside the domain, a per-node IACL that drops packets addressed to that node’s local SIDs Sk/sk when the source address is not one of the domain’s own interface addresses A/a.^[1] Dropping inbound packets that carry Routing Type 4 is a common extra hardening step you will see in the field, but it is not what RFC 8754 specifies.

5The bill: 40 plus 8 plus 16n

Every SRv6 design decision is downstream of this arithmetic. An outer IPv6 header is 40 bytes, the SRH base is 8 bytes, and each segment in the list is 16 bytes.^[1] A three-segment encapsulation therefore costs 96 bytes of overhead, on every packet, forever. The floor it is spent against is fixed: “IPv6 requires that every link in the Internet have an MTU of 1280 octets or greater.”^[2] RFC 8754 devotes section 5.3 to MTU considerations for exactly this reason, and load balancing and ECMP get their own section right after.^[1]

The first relief is the reduced SRH. Section 4.1.1: “A reduced SRH does not contain the first segment of the related SR Policy (the first segment is the one already in the DA of the IPv6 header), and the Last Entry field is set to n-2.”^[1] One segment saved is 16 bytes, so the same three-segment path costs 80 bytes instead of 96. RFC 8986 gives the headend behaviour that produces it - “H.Encaps.Red reduces the length of the SRH by excluding the first SID in the SRH of the pushed IPv6 header. The first SID is only placed in the Destination Address field of the pushed IPv6 header” - and adds that the SRH push “MAY be omitted when the SRv6 Policy only contains one segment and there is no need to use any flag, tag, or TLV.”^[3] Linux exposes both as mode encap and mode encap.red on the same command.^[6]

The second relief is compression, and it is a bigger number: under RFC 9800 several segments share one 128-bit container, so a multi-hop path can ride entirely in the destination address with no SRH at all.^[4] That is lesson 4. Note the shape of the argument for a customer: the honest worst case is 96 bytes, the reduced case is 80, and the compressed case can be 40.^[1]^[3]^[4]

Compute the header fields for a three-segment policy

Policy: fcbb:bbbb:1:: then fcbb:bbbb:2:: then fcbb:bbbb:3::, carrying an inner IPv6 packet, full (non-reduced) SRH.

Segment count n = 3, so the SRH carries three 16-byte entries = 48 bytes of list.^[1]
Hdr Ext Len = length in 8-octet units excluding the first 8 octets = 48 / 8 = 6.^[1]
Check against the spec bound: max_last_entry = (Hdr Ext Len / 2) - 1 = (6 / 2) - 1 = 2.^[1]
Last Entry = index of the last element, zero based = 3 - 1 = 2, which must be less than or equal to max_last_entry. It is.^[1]
Segments Left at the headend = 2, because the first segment is already in the destination address and two remain in the list to be visited.^[1]
Segment List order, reversed: [0]=fcbb:bbbb:3::, [1]=fcbb:bbbb:2::, [2]=fcbb:bbbb:1::; the destination address is fcbb:bbbb:1::.^[1]
Next Header = 41, the inner packet is IPv6.^[1]
Overhead = 40 + 8 + (3 x 16) = 96 bytes. On a 1500-byte link the largest inner packet is 1404 bytes.^[1]^[2]

Base header on the wire: 29 06 04 02 02 00 00 00.

Episode 2 closes — Sixteen bytes, on the wire, at 03:20

How it ended

The receiver’s capture shows Hdr Ext Len of 8 where the pre-change flow had 6: one more segment, sixteen more bytes on every packet.^[1] One hop was never raised off 1500 when the rest were, and the replication payload sits exactly in the gap. You have them rebuild the policy as a reduced SRH, which leaves out the first segment because it is already in the destination address, and open a change for the missed interface.^[1] The operator labels that patch panel before he hangs up.

What you say on the call: “Nothing is broken. You added 16 bytes to every packet and one link never got the memo.”^[1]^[2]

By 08:00 the incumbent has sent the config they want replayed in the Dell lab. One line comes back refused.

Lab

Read-only: sysctls and routes only, no firmware writes and no mode switching. Rollback for the enable step is sysctl -w net.ipv6.conf.all.seg6_enabled=0 plus the same for default and the named interface.

Pre-flight inventory on both Dell-lab ConnectX hosts. Record kernel, interface names, current MTU and current seg6 state before changing anything:
```
uname -r
ip -6 link show
sysctl net.ipv6.conf.all.seg6_enabled
```
Expected: seg6_enabled = 0, the shipping default.^[7] If it is already 1, someone else is using the host - stop and ask.
Enable seg6 on the sender’s data interface only, then push a three-segment policy toward the receiver, exactly as in the no-hardware lab. Rollback: delete the route, then set the sysctls back to 0.
Capture on the receiver, not the sender:
```
sudo tcpdump -ni <ifname> -vvv -X ip6 and ip6[6] == 43
```
Expected: outer IPv6 Next Header 43 and an SRH with Routing Type 4, Segments Left 2, Last Entry 2 for a full three-segment SRH.^[1] The sender never sees an MTU drop, which is why the receiver is the interesting side.
Sweep ping6 -s around (interface MTU minus 96) to locate the real cliff for mode encap, then repeat for mode encap.red and record both numbers. Expected: the reduced run survives exactly 16 more bytes of payload.^[1]^[3]
Write the two numbers into your FAE notes with the link MTU beside them. That pair is the direct answer to “what does SRv6 cost me in MTU”, and having measured it on real ConnectX hardware is worth more than quoting the formula.
Optional, customer lab only: repeat step 4 across a path that includes a switch you do not control, and confirm that the cliff moves to that hop’s MTU rather than yours. Still read-only.

Evidence / notes (commands run, outputs, what surprised you)

Before you read: what do you already know?

After this lesson you can