Anycast services and large-scale design
Design anycast services with BGP (RFC 4786). Handle health checks and route withdrawal on failure. Explain anycast's effect on stateful protocols and PMTU (RFC 7094).
Take the quiz · pass mark 70% · 9 questions
Explanation
Anycast announces the same IP address from many locations and lets routing deliver each client to the nearest instance. It is how global DNS roots, public resolvers and CDNs scale and survive failures.
How anycast works (RFC 4786)
RFC 4786 describes operating anycast services. Each site advertises the same service prefix into BGP; a client's packets follow the best BGP path to one of the instances (usually the topologically nearest). There is one service address but many servers. Capacity and resilience scale by adding sites: more announcements of the same prefix mean more places to absorb load and more redundancy.
Health checks and withdrawal
The critical operational detail: an anycast node must stop announcing the prefix the instant it is unhealthy, or BGP will keep steering clients to a broken instance. Tie the BGP advertisement to a health check — the node advertises only while the local service passes, and withdraws on failure so routing reconverges to the next-nearest site. Get this wrong and anycast turns a single-site outage into a region-wide black hole.
Stateful protocols and PMTU (RFC 7094)
RFC 7094 covers the architectural considerations of anycast. The key caveat: anycast assumes a client's packets keep reaching the same instance, which is true for short stateless exchanges (a DNS query/response) but not guaranteed for long stateful sessions — a routing change mid-connection can move the flow to a different instance with no session state, breaking it. So:
- Anycast is ideal for stateless services (DNS over UDP) and short requests.
- For long-lived stateful sessions (long TCP, TLS), either keep sessions short, pin them (e.g. redirect to a unicast address of the chosen instance after the first contact), or ensure routing stability.
- PMTU needs care too: an ICMPv6 Packet Too Big generated for an anycast flow must reach the instance that holds the path state; anycast can misdeliver it, so instances should handle PMTUD robustly (and permit the right ICMPv6, L5.02).
ictNetworks scenario
ictNetworks runs anycast DNS from three sites: each announces the same resolver prefix into BGP only while a health check passes, and withdraws on failure so clients reconverge to the nearest healthy site. Because DNS over UDP is stateless, anycast is a clean fit; for its anycast TCP services it keeps sessions short and ensures robust PMTUD per RFC 7094. The service address comes from its documentation space and is filtered so only the intended prefix is anycast.
Drawing diagram…
Worked examples
Example 1: One address, many sites
An anycast service deployed at three sites uses 1 distinct service IP address; routing, not DNS, picks which instance a client reaches.
Example 2: Announcements of the prefix
Deploying anycast DNS at 3 sites produces 3 BGP announcements of the same prefix — add a site, add an announcement, and both capacity and resilience grow.
Example 3: The ideal anycast workload
Anycast suits stateless request/response, the classic case being DNS on port 53 over UDP, where each query is independent so it does not matter which instance answers.
Lab
Goal: Deploy anycast DNS at three sites and test failover.
You need: a containerlab topology with FRR at three sites, each running a resolver (BIND or Knot) announcing the same anycast prefix via BGP, a health-check mechanism, and test clients at different points. (FRR/BIND/Knot syntax varies by version — check the docs.)
- Configure three sites to announce the same anycast prefix into BGP, each with its own resolver.
- From clients at different topological locations, confirm each reaches the nearest instance.
- Tie the BGP announcement to a health check; kill the resolver at the nearest site and confirm the prefix is withdrawn and clients reconverge to the next site.
- Show that a stateless DNS query works seamlessly across the failover.
- Demonstrate the stateful caveat: start a long TCP flow and force a routing change; observe the break, and describe a mitigation (short sessions / pin to unicast) per RFC 7094.
What to submit: the BGP/resolver/health-check configuration, evidence of nearest-instance selection, the withdrawal-and-failover test, and the stateful-caveat demonstration with a mitigation.
Summary
- Anycast announces one service address from many sites (RFC 4786); routing delivers each client to the nearest instance.
- Tie the BGP announcement to a health check and withdraw on failure, or you black-hole clients to a broken node.
- Anycast suits stateless services (DNS/UDP); long stateful sessions can break on a routing change (RFC 7094) and need mitigation.
- Handle PMTUD robustly so Packet Too Big reaches the instance holding the flow.
- Next lesson (L8.06): automation for IPv6 networks.
Quiz imported · 9 questions · pass mark 70%. Lab questions are not auto-marked.