Path MTU and fragmentation in routed networks
Explain the router's role in Path MTU Discovery (RFC 8201) and the Packet Too Big rules (RFC 4443). Calculate tunnel MTUs and TCP MSS clamping values for common encapsulations. Diagnose and fix PMTU black holes caused by filtered ICMPv6, ECMP and anycast (RFC 7690). Explain PLPMTUD (RFC 4821, RFC 8899) and plan MTUs in a provider core.
Take the quiz · pass mark 70% · 9 questions
Explanation
L3.08 looked at PMTU problems from the host's side. This lesson is about the routers in the middle, which cause most of them.
What routers do (and must not do)
An IPv6 router never fragments a packet it forwards (RFC 8200). If a packet is larger than the next link's MTU, the router:
- drops it
- sends an ICMPv6 Packet Too Big (PTB, type 2) to the packet's source, with the MTU of the next link (RFC 4443)
The PTB includes as much of the dropped packet as fits without the PTB exceeding the IPv6 minimum MTU of 1280 bytes, so the source can match it to a connection. Routers rate-limit ICMPv6 errors, so a router under heavy load may not answer every oversized packet.
The source host (RFC 8201) lowers its Path MTU for that destination, and retransmits in smaller packets (or fragments them itself, for UDP). It may try a larger PMTU again after at least 5 minutes, with 10 minutes recommended.
Only the source fragments, with the Fragment header. Every link must carry at least 1280 bytes; a link that cannot must do fragmentation and reassembly below IPv6.
Black holes
PMTUD fails when the PTB never arrives. The source keeps sending large packets that are dropped, so a TCP connection sets up (small packets) and then hangs as soon as data flows. Causes in routed networks:
- Filtering ICMPv6. A firewall or router ACL that drops ICMPv6, or drops PTB specifically. RFC 4890 lists PTB among the messages that must never be dropped. Rule: never block PTB.
- Routers that do not send PTB, because ICMP generation is disabled or rate-limited too hard, or because the router sources the PTB from an address that is itself filtered (for example a link address dropped by uRPF at the destination).
- ECMP and anycast (RFC 7690). A load balancer or ECMP path hashes the PTB on its own addresses, so it can arrive at a different server from the one that sent the large packet. Mitigations include hashing on the embedded packet inside the PTB, broadcasting PTBs to all servers behind the balancer, or lowering the server MSS.
Tunnel MTUs
Every encapsulation adds headers, so the tunnel's MTU is the underlying MTU minus the overhead:
| Encapsulation (on a 1500-byte link) | Overhead | Tunnel MTU | IPv6 TCP MSS |
|---|---|---|---|
| 6in4 (IPv6 in IPv4, protocol 41) | 20 | 1480 | 1420 |
| GRE over IPv4 | 24 | 1476 | 1416 |
| IPv6 in IPv6 (RFC 2473) | 40 | 1460 | 1400 |
| GRE over IPv6 | 44 | 1456 | 1396 |
| WireGuard over IPv4 | 60 | 1440 | 1380 |
| WireGuard over IPv6 | 80 | 1420 | 1360 |
| PPPoE | 8 | 1492 | 1432 |
(IPv6 TCP MSS = MTU − 40 (IPv6) − 20 (TCP).) A tunnel endpoint is a host for the outer packet: the tunnel interface must have the reduced MTU so the router sends PTBs for the inner packets, and RFC 2473 allows the encapsulator to fragment the outer packet if needed. Linux sets the right MTU automatically for most tunnel types; check with ip link show.
MSS clamping
Where you cannot rely on PMTUD (customer access links, tunnels to third parties), routers can rewrite the TCP MSS option in SYN packets so TCP never sends segments too large for the path:
table inet mangle {
chain forward {
type filter hook forward priority mangle; policy accept;
oifname "gre6-cpt" tcp flags syn tcp option maxseg size set rt mtu
}
}
set rt mtu uses the MTU of the route out of this router, so it only helps when this router owns the small link (a tunnel or PPPoE interface). For a bottleneck further along the path, set a fixed value (tcp option maxseg size set 1340 for a 1400-byte link). MikroTik RouterOS uses a mangle rule with action=change-mss, and FRR leaves this to the Linux firewall. MSS clamping only helps TCP; UDP, QUIC and tunnels inside tunnels still need working PMTUD or PLPMTUD.
PLPMTUD
Packetization Layer PMTUD (RFC 4821 for TCP and SCTP, RFC 8899 for datagram protocols such as QUIC) does not rely on ICMP at all: the sender probes with larger packets and treats a lost probe as "too big". It is slower but immune to black holes. On Linux, net.ipv4.tcp_mtu_probing=1 enables it for TCP after a black hole is detected (the setting applies to TCP over IPv6 as well), and QUIC stacks use RFC 8899 methods.
MTU in a provider core
- Run the core at a large MTU (for example 9000 or more) so customer 1500-byte packets fit even when encapsulated (MPLS, GRE, VXLAN, SRv6).
- Keep the MTU consistent on both ends of every link. OSPFv3 puts the interface MTU in its Database Description packets, and a neighbour with a different MTU sticks in ExStart or Exchange.
ipv6 ospf6 mtu-ignorehides the mismatch but does not fix it. - Give customers a full 1500 bytes, and keep 1280 as the floor for any link you control.
ictNetworks scenario
ictNetworks connects the Cape Town PoP to Johannesburg over a GRE-over-IPv6 tunnel across a third-party network with a 1500-byte MTU, while its own core uses 9000. The tunnel interface has MTU 1456, PTBs are allowed everywhere, and TCP MSS is clamped to 1396 on the tunnel as an extra safety net for customers behind broken firewalls.
Drawing diagram…
Worked examples
Example 1: GRE over IPv6
The underlay MTU is 1500. GRE over IPv6 adds 40 (outer IPv6) + 4 (GRE) = 44 bytes, so the tunnel MTU is 1456 and the IPv6 TCP MSS through it is 1456 − 60 = 1396.
Example 2: Sizing the core
To carry full 1500-byte customer packets over WireGuard over IPv6 (80 bytes of overhead), every underlay link must have an MTU of at least 1500 + 80 = 1580. Over IPv6-in-IPv6 it would be 1540.
Example 3: Clamping for a remote bottleneck
In the lab the bottleneck is the 1400-byte r2–r3 link, two hops from h1. Clamping on r1 must use a fixed value: 1400 − 60 = 1340. set rt mtu on r1 would use r1's own 1500-byte route MTU and clamp to 1440, which does not help.
Lab
Goal: Create a PMTU black hole in a routed path, diagnose it, and fix it three ways.
You need: containerlab with h1 – r1 – r2 – r3 – h2 (FRR routers with OSPFv3, Linux hosts), lab addresses from 2001:db8:1000:1LLL::/64 for links and 2001:db8:1001:1::/64, 2001:db8:1001:2::/64 for the hosts. Set MTU 1400 on the r2–r3 link (both ends). The small link is not adjacent to either host, so TCP's MSS negotiation does not hide it (L3.08).
- Verify that
ping -s 1452 -M dofrom h1 to h2 gets a PTB with MTU 1400, and show the cached PMTU on h1 (ip -6 route get). Flush it (ip -6 route flush cache). - On r1, drop PTB in the forward chain (
icmpv6 type packet-too-big drop). From h2 run a web server with a large file; from h1,curlit. Show the connection setting up and then hanging, and the capture on r2 showing the PTBs it sends. - Fix 1: remove the drop rule and show the transfer working and the PTB arriving at h1.
- Fix 2: put the drop rule back and clamp MSS on r1 to the right fixed value. Show the SYN before and after clamping and the transfer working. Explain why
set rt mtuon r1 would not help. - Fix 3: remove the clamp and enable
net.ipv4.tcp_mtu_probing=1on h2 (the sender). Show the transfer recovering after a delay. - Set the MTU on only one end of the r2–r3 link to 1500 and show the OSPFv3 adjacency stuck in ExStart or Exchange. Restore it.
- Build a GRE-over-IPv6 tunnel (
ip6gre) between r1 and r3, and show the tunnel MTU Linux chooses and that it matches the table.
What to submit: captures and outputs for each step, the clamp calculation, the OSPFv3 MTU-mismatch output and a short write-up of which fix you would use in production and why.
Summary
- Routers never fragment IPv6; they drop and send Packet Too Big (at most 1280 bytes) with the next-hop MTU.
- Black holes come from filtered PTB, routers that do not send PTB, and ECMP or anycast delivering PTB to the wrong host (RFC 7690).
- Tunnel MTU = underlay MTU − overhead; MSS = MTU − 60 for IPv6 TCP.
- MSS clamping fixes TCP only, and
rt mtuonly works on the router that owns the small link. - PLPMTUD (RFC 4821, RFC 8899) works without ICMP.
- Run a large, consistent core MTU; OSPFv3 adjacencies stall on MTU mismatch.
- Next lesson (L4.09): the integration lab, building an ISP core.
Quiz imported · 9 questions · pass mark 70%. Lab questions are not auto-marked.