ipv4-anycast

7/30/2019 ipv4-anycast

1/42

Best Practices in

IPv4 Anycast RoutingVersion 0.9

August, 2002

Bill WoodcockPacket Clearing House


2/42

What isntAnycast? Not a protocol, not a different version of

IP, nobodys proprietary technology.

Doesnt require any special capabilitiesin the servers, clients, or network.

Doesnt break or confuse existinginfrastructure.


3/42

What isAnycast? Just a configuration methodology.

Mentioned, although not described in detail,in numerous RFCs since time immemorial.

Its been the basis for large-scale content-distribution networks since at least 1995.

Its gradually taking over the core of the DNSinfrastructure, as well as much of theperiphery of the world wide web.


4/42

How Does Anycast Work? The basic idea is extremely simple:

Multiple instances of a service share thesame IP address.

The routing infrastructure directs any packetto the topologically nearest instance of the

service.

What little complexity exists is in the optionaldetails.


5/42

Example

Client

Server Instance A

Server Instance B

Router 1

Router 3

Router 2

Router 4


6/42

Example

Client

Server Instance A

Server Instance B

Router 1

Router 3

Router 2

Router 4

10.0.0.1

10.0.0.1

192.168.0.1

192.168.0.2


7/42

Client Router 1

Example

Server Instance A

Server Instance BRouter 3

Router 2

Router 4

10.0.0.1

10.0.0.1

192.168.0.1

192.168.0.2

DNS lookup for http://www.server.com/produces a single answer:

www.server.com. IN A 10.0.0.1


8/42

Router 1

Example

Client

Server Instance A


Router 2

Router 4

10.0.0.1

10.0.0.1

192.168.0.1

192.168.0.2

Routing Table from Router 1:

Destination Mask Next-Hop Distance192.168.0.0 /29 127.0.0.1 010.0.0.1 /32 192.168.0.1 110.0.0.1 /32 192.168.0.2 2


9/42

Router 1

Example

Client

Server Instance A


Router 2

Router 4

10.0.0.1

10.0.0.1

192.168.0.1

192.168.0.2




10/42

Router 1

Example

Client

Server Instance A


Router 2

Router 4

10.0.0.1

10.0.0.1

192.168.0.1

192.168.0.2




11/42

Router 1

Example

Client Server

Router 3

Router 2

Router 4

10.0.0.1

192.168.0.1

192.168.0.2



What the routers think the topology looks like:


12/42

Building an Anycast Server Cluster Anycast can be used in building either

local server clusters, or global networks,

or global networks of clusters,combining both scales.

F-root is a local anycast server cluster,for instance.


13/42

Building an Anycast Server Cluster Typically, a cluster of servers share a

common virtual interface attached to

their loopback devices, and speak anIGP routing protocol to an adjacentBGP-speaking border router.

The servers may or may not shareidentical content.


14/42

Example

Router

Eth0192.168.1.2/30

Lo010.0.0.1/32

Eth0192.168.2.2/30

Eth0192.168.3.2/30

Lo010.0.0.1/32

Lo010.0.0.1/32

Server Instance A

Server Instance B

Server Instance C

BGP IGPRedistribution


15/42

Router

Example

Eth0192.168.1.2/30

Lo010.0.0.1/32

Eth0192.168.2.2/30

Eth0192.168.3.2/30

Lo010.0.0.1/32

Lo010.0.0.1/32

Server Instance A

Server Instance B

Server Instance C


Destination Mask Next-Hop Dist0.0.0.0 /0 127.0.0.1 0192.168.1.0 /30 192.168.1.1 0192.168.2.0 /30 192.168.2.1 0192.168.3.0 /30 192.168.3.1 010.0.0.1 /32 192.168.1.2 110.0.0.1 /32 192.168.2.2 1

10.0.0.1 /32 192.168.3.2 1


16/42

Router

Example

Eth0192.168.1.2/30

Lo010.0.0.1/32

Eth0192.168.2.2/30

Eth0192.168.3.2/30

Lo010.0.0.1/32

Lo010.0.0.1/32

Server Instance A

Server Instance B

Server Instance C


Destination Mask Next-Hop Dist0.0.0.0 /0 127.0.0.1 0192.168.1.0 /30 192.168.1.1 0192.168.2.0 /30 192.168.2.1 0192.168.3.0 /30 192.168.3.1 010.0.0.1 /32 192.168.1.2 110.0.0.1 /32 192.168.2.2 1

10.0.0.1 /32 192.168.3.2 1

Round-robin load balancing


17/42

Building a Global Network of Clusters Once a cluster architecture has been

established, additional clusters can be

added to gain performance.

Load distribution, fail-over betweenclusters, and content synchronizationbecome the principal engineeringconcerns.


18/42

Example

Router 2

ServerI

nstanceD

ServerI

nstanceE

ServerI

nstanceF

Router

3Rout

er1

ServerInstanc

eA

ServerInsta

nceB

ServerInsta

nceC ServerInstanceG

ServerInstanceH

S

erverInstanceI


19/42

Example

Router 2

ServerI

nstanceD

ServerI

nstanceE

ServerI

nstanceF

Router

3Rout

er1

ServerInstanc

eA

ServerInsta

nceB

ServerInsta


ServerInstanceH

S

erverInstanceI

Region 1

Region 2

Region 3


20/42

Example

Router 2

ServerI

nstanceD

ServerI

nstanceE

ServerInstanceF

Router

3Rout

er1

ServerInstanc

eA

ServerInsta

nceB

ServerInsta


ServerInstanceH

S

erverInstanceI

BGP Announcements

10.0.0.1 /32192.168.0.0 /22192.168.0.0 /16

10.0.0.1 /32192.168.8.0 /22192.168.0.0 /16

10.0.0.1 /32192.168.4.0 /22192.168.0.0 /16


21/42

Example

Router 2

ServerI

nstanceD

ServerI

nstanceE

ServerInstanceF

Router

3Rout

er1

ServerInstanc

eA

ServerInsta

nceB

ServerInsta


ServerInstanceH

S

erverInstanceI

IGP 1 Announcements

10.0.0.1 /3210.0.0.1 /3210.0.0.1 /32

192.168.1.0 /30192.168.2.0 /30

192.168.3.0 /30

10.0.0.1 /3210.0.0.1 /3210.0.0.1 /32

192.168.9.0 /30192.168.10.0 /30

192.168.11.0 /3010.0.0.1 /32

10.0.0.1 /3210.0.0.1 /32

192.168.5.0 /30

192.168.6.0 /30192.168.7.0 /30


22/42

Example

Router 2

ServerI

nstanceD

ServerInstanceE

ServerInstanceF

Router

3Rout

er1

ServerInstanc

eA

ServerInsta

nceB

ServerInsta


ServerInstanceH

S

erverInstanceI

IGP 2 Announcements

10.0.0.1 /32192.168.1.0 /30192.168.2.0 /30

192.168.3.0 /30

10.0.0.1 /32192.168.9.0 /30192.168.10.0 /30

192.168.11.0 /30

10.0.0.1 /32

192.168.5.0 /30192.168.6.0 /30

192.168.7.0 /30


23/42

Performance-Tuning Anycast Networks Server deployment in anycast networks is

always a tradeoff between absolute cost andefficiency.

The network will perform best if servers arewidely distributed, with higher density in andsurrounding high demand areas.

Lower initial cost sometimes leadsimplementers to compromise by deployingmore servers in existing locations, which isless efficient.


24/42

ExampleGeographic plot of user population density


25/42


Server deployment


26/42


Server deployment

Traffic Flow


27/42


Server deployment

Traffic Flow


28/42


Server deployment

Traffic Flow


29/42


Server deployment

Traffic Flow


30/42

ExampleDrawing traffic growth away from a hot-spot


31/42



32/42



33/42



34/42



35/42


Topological watershed


36/42



37/42

Caveats and Failure Modes DNS resolution fail-over

Long-lived connection-oriented flows

Identifying which server is giving anend-user trouble


38/42

DNS Resolution Fail-Over In the event of poor performance from a

server, DNS servers will fail over to the next

server in a list. If both servers are in fact hosted in the same

anycast cloud, the resolver will wind up

talking to the same instance again.

Best practices for anycast DNS server

operations indicate a need for two separate

overlapping clouds of anycast servers.


39/42

Long-Lived Connection-Oriented Flows

Long-lived flows, typically TCP file-transfers or interactivelogins, may occasionally be more stable than the underlyingInternet topology.

If the underlying topology changes sufficiently during the life ofan individual flow, packets could be redirected to a differentserver instance, which would not have proper TCP state, andwould reset the connection.

This is not a problem with web servers unless theyremaintaining stateful per-session information about end-users,

rather than embedding it in URLs or cookies.

Web servers HTTP redirect to their unique address wheneverthey need to enter a stateful mode.

Limited operational data shows underlying instability to be onthe order of one flow per ten thousand per hour of duration.


40/42

Identifying Problematic Server Instances

Some protocols may not include an easyin-band method of identifying the server

which persists beyond the duration of theconnection.

Traceroute always identifies the currentserver instance, but end-users may not

even have traceroute.


41/42

A Security Ramification Anycast server clouds have the useful

property of sinking DOS attacks at the

instance nearest to the source of theattack, leaving all other instancesunaffected.

This is still of some utility even whenDOS sources are widely distributed.


42/42

Bill Woodcock

[email protected]

www.pch.net/documents/tutorials/anycast

ipv4-anycast

Documents