Download - ipv4-anycast
-
7/30/2019 ipv4-anycast
1/42
Best Practices in
IPv4 Anycast RoutingVersion 0.9
August, 2002
Bill WoodcockPacket Clearing House
-
7/30/2019 ipv4-anycast
2/42
What isntAnycast? Not a protocol, not a different version of
IP, nobodys proprietary technology.
Doesnt require any special capabilitiesin the servers, clients, or network.
Doesnt break or confuse existinginfrastructure.
-
7/30/2019 ipv4-anycast
3/42
What isAnycast? Just a configuration methodology.
Mentioned, although not described in detail,in numerous RFCs since time immemorial.
Its been the basis for large-scale content-distribution networks since at least 1995.
Its gradually taking over the core of the DNSinfrastructure, as well as much of theperiphery of the world wide web.
-
7/30/2019 ipv4-anycast
4/42
How Does Anycast Work? The basic idea is extremely simple:
Multiple instances of a service share thesame IP address.
The routing infrastructure directs any packetto the topologically nearest instance of the
service.
What little complexity exists is in the optionaldetails.
-
7/30/2019 ipv4-anycast
5/42
Example
Client
Server Instance A
Server Instance B
Router 1
Router 3
Router 2
Router 4
-
7/30/2019 ipv4-anycast
6/42
Example
Client
Server Instance A
Server Instance B
Router 1
Router 3
Router 2
Router 4
10.0.0.1
10.0.0.1
192.168.0.1
192.168.0.2
-
7/30/2019 ipv4-anycast
7/42
Client Router 1
Example
Server Instance A
Server Instance BRouter 3
Router 2
Router 4
10.0.0.1
10.0.0.1
192.168.0.1
192.168.0.2
DNS lookup for http://www.server.com/produces a single answer:
www.server.com. IN A 10.0.0.1
-
7/30/2019 ipv4-anycast
8/42
Router 1
Example
Client
Server Instance A
Server Instance BRouter 3
Router 2
Router 4
10.0.0.1
10.0.0.1
192.168.0.1
192.168.0.2
Routing Table from Router 1:
Destination Mask Next-Hop Distance192.168.0.0 /29 127.0.0.1 010.0.0.1 /32 192.168.0.1 110.0.0.1 /32 192.168.0.2 2
-
7/30/2019 ipv4-anycast
9/42
Router 1
Example
Client
Server Instance A
Server Instance BRouter 3
Router 2
Router 4
10.0.0.1
10.0.0.1
192.168.0.1
192.168.0.2
Routing Table from Router 1:
Destination Mask Next-Hop Distance192.168.0.0 /29 127.0.0.1 010.0.0.1 /32 192.168.0.1 110.0.0.1 /32 192.168.0.2 2
-
7/30/2019 ipv4-anycast
10/42
Router 1
Example
Client
Server Instance A
Server Instance BRouter 3
Router 2
Router 4
10.0.0.1
10.0.0.1
192.168.0.1
192.168.0.2
Routing Table from Router 1:
Destination Mask Next-Hop Distance192.168.0.0 /29 127.0.0.1 010.0.0.1 /32 192.168.0.1 110.0.0.1 /32 192.168.0.2 2
-
7/30/2019 ipv4-anycast
11/42
Router 1
Example
Client Server
Router 3
Router 2
Router 4
10.0.0.1
192.168.0.1
192.168.0.2
Routing Table from Router 1:
Destination Mask Next-Hop Distance192.168.0.0 /29 127.0.0.1 010.0.0.1 /32 192.168.0.1 110.0.0.1 /32 192.168.0.2 2
What the routers think the topology looks like:
-
7/30/2019 ipv4-anycast
12/42
Building an Anycast Server Cluster Anycast can be used in building either
local server clusters, or global networks,
or global networks of clusters,combining both scales.
F-root is a local anycast server cluster,for instance.
-
7/30/2019 ipv4-anycast
13/42
Building an Anycast Server Cluster Typically, a cluster of servers share a
common virtual interface attached to
their loopback devices, and speak anIGP routing protocol to an adjacentBGP-speaking border router.
The servers may or may not shareidentical content.
-
7/30/2019 ipv4-anycast
14/42
Example
Router
Eth0192.168.1.2/30
Lo010.0.0.1/32
Eth0192.168.2.2/30
Eth0192.168.3.2/30
Lo010.0.0.1/32
Lo010.0.0.1/32
Server Instance A
Server Instance B
Server Instance C
BGP IGPRedistribution
-
7/30/2019 ipv4-anycast
15/42
Router
Example
Eth0192.168.1.2/30
Lo010.0.0.1/32
Eth0192.168.2.2/30
Eth0192.168.3.2/30
Lo010.0.0.1/32
Lo010.0.0.1/32
Server Instance A
Server Instance B
Server Instance C
BGP IGPRedistribution
Destination Mask Next-Hop Dist0.0.0.0 /0 127.0.0.1 0192.168.1.0 /30 192.168.1.1 0192.168.2.0 /30 192.168.2.1 0192.168.3.0 /30 192.168.3.1 010.0.0.1 /32 192.168.1.2 110.0.0.1 /32 192.168.2.2 1
10.0.0.1 /32 192.168.3.2 1
-
7/30/2019 ipv4-anycast
16/42
Router
Example
Eth0192.168.1.2/30
Lo010.0.0.1/32
Eth0192.168.2.2/30
Eth0192.168.3.2/30
Lo010.0.0.1/32
Lo010.0.0.1/32
Server Instance A
Server Instance B
Server Instance C
BGP IGPRedistribution
Destination Mask Next-Hop Dist0.0.0.0 /0 127.0.0.1 0192.168.1.0 /30 192.168.1.1 0192.168.2.0 /30 192.168.2.1 0192.168.3.0 /30 192.168.3.1 010.0.0.1 /32 192.168.1.2 110.0.0.1 /32 192.168.2.2 1
10.0.0.1 /32 192.168.3.2 1
Round-robin load balancing
-
7/30/2019 ipv4-anycast
17/42
Building a Global Network of Clusters Once a cluster architecture has been
established, additional clusters can be
added to gain performance.
Load distribution, fail-over betweenclusters, and content synchronizationbecome the principal engineeringconcerns.
-
7/30/2019 ipv4-anycast
18/42
Example
Router 2
ServerI
nstanceD
ServerI
nstanceE
ServerI
nstanceF
Router
3Rout
er1
ServerInstanc
eA
ServerInsta
nceB
ServerInsta
nceC ServerInstanceG
ServerInstanceH
S
erverInstanceI
-
7/30/2019 ipv4-anycast
19/42
Example
Router 2
ServerI
nstanceD
ServerI
nstanceE
ServerI
nstanceF
Router
3Rout
er1
ServerInstanc
eA
ServerInsta
nceB
ServerInsta
nceC ServerInstanceG
ServerInstanceH
S
erverInstanceI
Region 1
Region 2
Region 3
-
7/30/2019 ipv4-anycast
20/42
Example
Router 2
ServerI
nstanceD
ServerI
nstanceE
ServerInstanceF
Router
3Rout
er1
ServerInstanc
eA
ServerInsta
nceB
ServerInsta
nceC ServerInstanceG
ServerInstanceH
S
erverInstanceI
BGP Announcements
10.0.0.1 /32192.168.0.0 /22192.168.0.0 /16
10.0.0.1 /32192.168.8.0 /22192.168.0.0 /16
10.0.0.1 /32192.168.4.0 /22192.168.0.0 /16
-
7/30/2019 ipv4-anycast
21/42
Example
Router 2
ServerI
nstanceD
ServerI
nstanceE
ServerInstanceF
Router
3Rout
er1
ServerInstanc
eA
ServerInsta
nceB
ServerInsta
nceC ServerInstanceG
ServerInstanceH
S
erverInstanceI
IGP 1 Announcements
10.0.0.1 /3210.0.0.1 /3210.0.0.1 /32
192.168.1.0 /30192.168.2.0 /30
192.168.3.0 /30
10.0.0.1 /3210.0.0.1 /3210.0.0.1 /32
192.168.9.0 /30192.168.10.0 /30
192.168.11.0 /3010.0.0.1 /32
10.0.0.1 /3210.0.0.1 /32
192.168.5.0 /30
192.168.6.0 /30192.168.7.0 /30
-
7/30/2019 ipv4-anycast
22/42
Example
Router 2
ServerI
nstanceD
ServerInstanceE
ServerInstanceF
Router
3Rout
er1
ServerInstanc
eA
ServerInsta
nceB
ServerInsta
nceC ServerInstanceG
ServerInstanceH
S
erverInstanceI
IGP 2 Announcements
10.0.0.1 /32192.168.1.0 /30192.168.2.0 /30
192.168.3.0 /30
10.0.0.1 /32192.168.9.0 /30192.168.10.0 /30
192.168.11.0 /30
10.0.0.1 /32
192.168.5.0 /30192.168.6.0 /30
192.168.7.0 /30
-
7/30/2019 ipv4-anycast
23/42
Performance-Tuning Anycast Networks Server deployment in anycast networks is
always a tradeoff between absolute cost andefficiency.
The network will perform best if servers arewidely distributed, with higher density in andsurrounding high demand areas.
Lower initial cost sometimes leadsimplementers to compromise by deployingmore servers in existing locations, which isless efficient.
-
7/30/2019 ipv4-anycast
24/42
ExampleGeographic plot of user population density
-
7/30/2019 ipv4-anycast
25/42
ExampleGeographic plot of user population density
Server deployment
-
7/30/2019 ipv4-anycast
26/42
ExampleGeographic plot of user population density
Server deployment
Traffic Flow
-
7/30/2019 ipv4-anycast
27/42
ExampleGeographic plot of user population density
Server deployment
Traffic Flow
-
7/30/2019 ipv4-anycast
28/42
ExampleGeographic plot of user population density
Server deployment
Traffic Flow
-
7/30/2019 ipv4-anycast
29/42
ExampleGeographic plot of user population density
Server deployment
Traffic Flow
-
7/30/2019 ipv4-anycast
30/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
31/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
32/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
33/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
34/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
35/42
ExampleDrawing traffic growth away from a hot-spot
Topological watershed
-
7/30/2019 ipv4-anycast
36/42
ExampleDrawing traffic growth away from a hot-spot
-
7/30/2019 ipv4-anycast
37/42
Caveats and Failure Modes DNS resolution fail-over
Long-lived connection-oriented flows
Identifying which server is giving anend-user trouble
-
7/30/2019 ipv4-anycast
38/42
DNS Resolution Fail-Over In the event of poor performance from a
server, DNS servers will fail over to the next
server in a list. If both servers are in fact hosted in the same
anycast cloud, the resolver will wind up
talking to the same instance again.
Best practices for anycast DNS server
operations indicate a need for two separate
overlapping clouds of anycast servers.
-
7/30/2019 ipv4-anycast
39/42
Long-Lived Connection-Oriented Flows
Long-lived flows, typically TCP file-transfers or interactivelogins, may occasionally be more stable than the underlyingInternet topology.
If the underlying topology changes sufficiently during the life ofan individual flow, packets could be redirected to a differentserver instance, which would not have proper TCP state, andwould reset the connection.
This is not a problem with web servers unless theyremaintaining stateful per-session information about end-users,
rather than embedding it in URLs or cookies.
Web servers HTTP redirect to their unique address wheneverthey need to enter a stateful mode.
Limited operational data shows underlying instability to be onthe order of one flow per ten thousand per hour of duration.
-
7/30/2019 ipv4-anycast
40/42
Identifying Problematic Server Instances
Some protocols may not include an easyin-band method of identifying the server
which persists beyond the duration of theconnection.
Traceroute always identifies the currentserver instance, but end-users may not
even have traceroute.
-
7/30/2019 ipv4-anycast
41/42
A Security Ramification Anycast server clouds have the useful
property of sinking DOS attacks at the
instance nearest to the source of theattack, leaving all other instancesunaffected.
This is still of some utility even whenDOS sources are widely distributed.
-
7/30/2019 ipv4-anycast
42/42
Bill Woodcock
www.pch.net/documents/tutorials/anycast