A Measurement-based Deployment
Proposal for IP Anycast
Hitesh Ballani (Cornell University)Paul Francis (Cornell University)
Sylvia Ratnasamy (Intel-Research)
IMC 2006
What is IP Anycast?
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
Same IP Address assigned to the servers
making the anycast group
One-to-Any communicationwith no changes to Internet routing and clients
What is IP Anycast?
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
to2.1.1.1
CC Client
One-to-Any communicationwith no changes to Internet routing and clients
What is IP Anycast?
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
to2.1.1.1
CC Client
C
to2.1.1.1
One-to-Any communicationwith no changes to Internet routing and clients
What is IP Anycast?
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
to2.1.1.1
CC Client
C
to2.1.1.1
IP Anycast as a Service Discovery Primitive
I Distributes client load across servers
I Reduces access latency for clientsI Offers network-level resilience to DDoS attacks
IP Anycast Usage
I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)
I IPv4-to-IPv6 transition [RFC 3068]
I Rendezvous discovery for IP multicast [RFC 3446]
I Other usage scenarios[]
I AS112 Project [http://as112.net]
I Commercial CDNs [http://cachefly.net]
I DDos sinkholes [Greene et. al., NANOG’03]
IP Anycast Usage
I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)
I IPv4-to-IPv6 transition [RFC 3068]
I Rendezvous discovery for IP multicast [RFC 3446]
I Other usage scenarios[]
I AS112 Project [http://as112.net]
I Commercial CDNs [http://cachefly.net]
I DDos sinkholes [Greene et. al., NANOG’03]
IP Anycast Usage
I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)
I IPv4-to-IPv6 transition [RFC 3068]
I Rendezvous discovery for IP multicast [RFC 3446]
I Other usage scenarios[]
I AS112 Project [http://as112.net]
I Commercial CDNs [http://cachefly.net]
I DDos sinkholes [Greene et. al., NANOG’03]
IP Anycast Usage
I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)
I IPv4-to-IPv6 transition [RFC 3068]
I Rendezvous discovery for IP multicast [RFC 3446]
I Other usage scenarios[]
I AS112 Project [http://as112.net]
I Commercial CDNs [http://cachefly.net]
I DDos sinkholes [Greene et. al., NANOG’03]
In spite of growing usage, IP Anycast and itsinteraction with IP Routing is not well understood!
IP Anycast is not well understood
Are clients routed to close-by anycast servers?
What is the impact of the failure of an anycastserver?
Is the client load across the anycast server sitesbalanced?
Are subsequent packets from a client routed to thesame anycast site?
IP Anycast is not well understood
Are clients routed to close-by anycast servers?
What is the impact of the failure of an anycastserver?
Is the client load across the anycast server sitesbalanced?
Are subsequent packets from a client routed to thesame anycast site?
I Affinity offered by IP Anycast
I Load-distribution across deployments
I Failover properties of IP Anycast
I Proximity offered by IP Anycast
A sneak peek at the study’s results
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment
PlannedDeployment
A sneak peek at the study’s results
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment
PlannedDeployment
A sneak peek at the study’s results
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment
PlannedDeployment
Take Home Message
Talk Outline
I Introduction[]y
I Terminology[]y
I Deployments Measured[]y
I Methodology[]yI Measurements[]y
I ProximityI FailoverI Load-distributionI Affinity
I Conclusions
Terminology
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
Anycast Serversbelong to different domains
Study focusses on Inter-domain IP Anycast
Terminology
IP Address IP Address 2.1.1.12.1.1.1
AnycastServer
NetworkDomains
2.1.1.0/24 2.1.1.0/24
BGPAdvert.
Anycast Prefix = 2.1.1.0/24
Anycast Servers advertise the Anycast Prefix intoBGP through their Upstream Provider
Deployments Measured
External Deployments
F-Root : 27 serversJ-Root : 13 serversAS112 : 20 servers
Deployments Measured
External Deployments
F-Root : 27 serversJ-Root : 13 serversAS112 : 20 servers
Internal Deployment
Internal : 5 servers
IR Cambridge
UK
Internet
BGP Advert.for Anycast Prefix 204.9.168.0/22
Internal Deployment
ATT ggWCG
IRSeattle
US
Cornell Ithaca
US
IRBerkeley
US
IR Pittsburgh
US
ATT-World
Probing Methodology
Internet
F-Root (Palo Alto)
F-Root (Chicago)
F-Root (New York)
F-Root (Madrid)
F-Root (Osaka)
C C
F-Root Deployment: Anycast Servers are DNS Servers
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C C
Internal Deployment: Anycast Servers can run DNS Servers
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C C
DNS Query
Anycast Servers can be probedusing DNS queries
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C C
DNS Query DNS Response
Anycast Servers can be probedusing DNS queries
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
DNS Query DNS Response
Recursive DNS Servers can be used as Vantage Points [KING, IMW’02]
C
Recursive DNSServer used asCa Vantage Point
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
DNS Query DNS Response
C
Recursive DNSServer used asCa Vantage Point
StationMeasurement
such that it queries the Anycast Server Query the Recursive DNS Server
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
DNS Query DNS Response
C
Recursive DNSServer used asCa Vantage Point
StationMeasurement
Response from the Anycast Server isforwarded onto the Measurement Host
Probing Methodology
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
DNS Query DNS Response
C
Recursive DNSServer used asCa Vantage Point
StationMeasurement
Measurements: 1). Anycast Server being accessed by C 2). Latency from C to the Anycast Server
Probing Methodology
23,858 Recursive DNS Servers used as Vantage Points
Region No. of clients % of Total
North America 12931 54.827Central America 317 1.344South America 461 1.954
Europe 5585 23.680Asia 2402 10.184
S.E. Asia 566 2.400Oceania 1196 5.071Africa 187 0.792
Arctic Region 9 0.038Unknown 204 0.864
Total 23858 100.000
Talk Outline
I Introduction[]y
I Terminology[]y
I Deployments Measured[]y
I Methodology[]yI Measurements[]y
I ProximityI FailoverI Load-distributionI Affinity
I Conclusions
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Anycast Server is chosen by Inter-Domain Routing
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Ideally, Clients would be routedclose-by Anycast Server to a
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Poor choice of Anycast Server is possible!
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe Latency =
Probe the Deployment’s Anycast Address from the client
Anycast Latency
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
Latency to closest Anycast Server =Client probes individual Anycast Servers
Min. Unicast Latency
Proximity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
STRETCH = (Anycast Latency - Minimum Unicast Latency)
Proximity
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
F-RootJ-RootAS112
Internal
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
F-RootJ-RootAS112
Internal
Proximity
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
F-RootJ-RootAS112
Internal
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
F-RootJ-RootAS112
Internal
Internal 31%
30 msec
AS112 61%
All four deployments offer poor Proximity
Investigating Poor Proximity
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
Level3
Client at UC Berkeley
Client probes Anycast address of Internal Deployment
Investigating Poor Proximity
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
Level3
Client at UC Berkeley
Best path toany Server
Client probes Anycast address of Internal Deployment Routed to Ithaca (NY) instead of Berkeley (CA)
Investigating Poor Proximity
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
Level3
Best path toany Server
http://pias.gforge.cis.cornell.edu/trace.htmlSee this example at
Investigating Poor Proximity
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C
Probe toAnycast Address
Probe to Unicast Address
Level3
Best path toany Server
Level3 does not realize that these lead to different locationsLevel3 has two paths of 2 AS-hops: through ATT and WCG
Alleviating Poor Proximity
ISP X
US
US Europe
Asia Oceania (W. Coast)
(E. Coast)
Internet
Anycast Servers should have the same Upstream ISP
Alleviating Poor Proximity
ISP X
US
US Europe
Asia Oceania (W. Coast)
(E. Coast)
Internet
Anycast Servers should have the same Upstream ISP
Multiple Providers: Geographically cover all providers
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
ISP X
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
ISP Y
Internet
Alleviating Poor Proximity
ISP X
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
Internet
C
Alleviating Poor Proximity
ISP X
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
Internet
C
Anycast probes from client(s) routed to ISP X
Alleviating Poor Proximity
ISP X
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
Internet
C
Server chosen is based on X’s intra-domain routing
Verifying our hypothesis
Internet
ATT
WCGATT-World
StationMeasurement
C
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
A subset of the Internal Deployment
Verifying our hypothesis
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
InternalSubset
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
InternalSubset
31%
30 msec
5%
Verifying our hypothesis
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
InternalSubset
0
0.2
0.4
0.6
0.8
1
-50 0 50 100 150 200
CD
F
Stretch (msec)
InternalSubset
31%
30 msec
5%
Planned Anycast Deployment ⇒ good Proximity
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C C
Probe to Anycast Address (pre-failure)
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C C
Probe to Anycast Address (pre-failure)
Probe to AnycastAddress (post-failure)
Clients are re-routed to a different Anycast Server
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
StationMeasurement
C C
Probe to Anycast Address (pre-failure)
Probe to AnycastAddress (post-failure)
Clients are re-routed to a different Anycast ServerWhat is the failover time?
Failover
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
Failover
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
Ithaca and Cambridge servers have slow failover
Failover
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
20 40 60 80 100 120
CD
F
Failover Time (sec)
IthacaPittsburgh
CambridgeSeattle
Berkeley
Other servers have faster failover
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
BGP Withdrawal
BGP Withdrawal propagated beyond WCGGlobal Routing Event Slow convergence
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
BGP Withdrawal
BGP Withdrawal restricted to ATTLocal Routing Event Faster convergence
Failover
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
Planned Deployment Fast Failover
Load Distribution
Distribution of client-load across Anycast Servers
I Can operators control this load?I Used AS-Path Prepending for controlling load
AS-Path Prepending a BGP Advertisement
I Changing the advertisement’s AS-Path lengthI n-hop Prepending: Add n ASs to the AS-Pathg
Load Distribution
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
Load Distribution
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
Skewed distribution of clients
Load Distribution
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.0.34 0.23 0.18
Load Distribution
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.
0 0.1 0.2 0.3 0.4 0.5 0.6
43210
Load
Fra
ctio
n
Prepending at the Ithaca server (AS-hops)
IthacaPittsburgh
SeattleBerkeley
Cam.0.34 0.23 0.18
AS-Path Prepending provides coarse-grainedcontrol over client load
Affinity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C
IP Anycast is a network-layer service
Affinity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C
Client can to a different Anycast Serverflap
Affinity
Internet
ATT
WCGATT-World
Seattle, WA
Berkeley, CA Pittsburgh, PA
Ithaca, NY Cambridge, UK
C
Client can to a different Anycast ServerflapWhat is the offered by IP Anycast? Affinity
Affinity
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
Affinity
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
95% of clientsobserve less than 1 Flap per day
Affinity
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
Less than 1% of clients experience frequent flaps
Affinity
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
0
0.2
0.4
0.6
0.8
1
1001010
CD
F
Average Flaps per day
Internal
Less than 1% of clients experience frequent flaps
Frequent flaps can be attributed toload-balancing at the client site
Conclusions
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment Poor Slow
Conclusions
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment Poor Slow
PlannedDeployment Good Fast
Due to the planned deployment
Conclusions
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment Poor Slow
PlannedDeployment Good Fast
Skewed
Manipulatable
BGP Traffic Engineering techniques
Conclusions
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment Poor Slow
PlannedDeployment Good Fast
Skewed
Manipulatable
Good*
Good*
Conclusions
Property
AnycastIP
Proximity Failover Load Affinity
Ad-Hoc Deployment Poor Slow
PlannedDeployment Good Fast
Skewed
Manipulatable
Good*
Good*
Traceshttp://pias.gforge.cis.cornell.edu/measure.php
Alleviating Poor Proximity
ISP X
US
USEurope
Asia
Oceania(W. Coast)
(E. Coast)
Internet
C
Hot-Potato Routing
Least-CostRouting
Only Proximity measurements for the
External Deployments
Internet
F-Root (Palo Alto)
F-Root (Chicago)
F-Root (New York)
F-Root (Madrid)
F-Root (Osaka)
C
Probes to External DeploymentsCannot determine the identity of the responding server