CompSci 514: Computer Networks

Lecture 10: BGP ProblemsXiaowei Yang

1

2

Today

• Inferring BGP relationships– Opportunities for machine learning?– A candidate for the course project

• Known problems of BGP– Scaling issues caused by multi-homing – Instability– Delayed convergence

• Slow failover– Prefix hijacking

3

Background• Route import/export follow business

relationship between ASes

• Routing advertisements include AS path information

• à Use this information to infer AS-level topology

4

An annotated AS Graph

• Sibling: provide transit services for each other. – May belong to the same company

5

AS5

AS1

AS2 AS3

AS4

peer-to-peer

sibling-to-sibling edge

edgeAS6

AS7

provider-to-customer edge

What type of paths are legitimate?

7

The valley-free property

• Valley-free: After traversing a provider-to-customer or peer-to-peer edge, the AS path can not traverse a customer-to-provider or peer-to-peer edge

8

The basic algorithm

• Step 1: compute degree of each AS– The node with the largest number of neighbors

is the top-level provider on an AS path9

• Step 2: Assign pairwise AS relationships– AS pairs before the largest degree nodes

are customer-providers– AS pairs after the largest degree node are

provider-customers

10

11

Sources of errors

• Misconfigurations– An AS may erroneously provide transit

service between providers

• Partial view– An AS’s degree may be larger

12

Taken into configuration errors

• Counts the number of times an AS provides transit service for another AS

• Assign relationships if an AS is found only to provide transit service for another AS, or the number of times an AS providing transit service for another AS exceeds a threshold

13

14

Inferring peering relationships

• First identify all AS pairs that cannot be peers

• If the degrees between the two ASes do not differ too much, assign them to be peers

15

16

Evalution

• Verified using AT&T internal information

17

Comments

• Missing edges may affect the results

• Heuristics may or may not hold

• Interesting to conduct such a studyagain and compare results

18

19

Today

• Inferring BGP relationships– Opportunities for machine learning?– A candidate for the course project

• Known problems of BGP– Scaling issues caused by multi-homing – Instability– Delayed convergence

• Slow failover

– Prefix hijacking

20

Multi-homing

• Connect to multiple providers– Goal: Higher availability, more capacity

• Problems:– Provider-based addressing breaks– Everyone needs their own address space

21

Multi-homing increases routing table size

Mutil-home.com

128.0.0.0/8204.0.0.0/8

204.1.0.0/16

ISP2 ISP1

You can reach 128.0.0.0/8And 204.1.0.0/16 via ISP1

ISP3

204.1.0.0/16 ISP1204.1.0.0/16128.0.0.0/8 ISP1

204.1.0.0/16 ISP2204.0.0.0/8 ISP2

22

Global routing tables continue to grow

Source: http://bgp.potaroo.net/as6447/

24

Internet Routing Instability

• Goals: how often BGP sends updates to change routes

• Methodology:– Analyzing BGP logs for a long time

IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 5, OCTOBER 1998 515

Internet Routing InstabilityCraig Labovitz, Student Member, IEEE, G. Robert Malan, Student Member, IEEE, and Farnam Jahanian, Member, IEEE

Abstract—This paper examines the network interdomain rout-ing information exchanged between backbone service providersat the major U.S. public Internet exchange points. Internet rout-ing instability, or the rapid fluctuation of network reachabilityinformation, is an important problem currently facing the In-ternet engineering community. High levels of network instabilitycan lead to packet loss, increased network latency and time toconvergence. At the extreme, high levels of routing instabilityhave led to the loss of internal connectivity in wide-area, nationalnetworks. In this paper, we describe several unexpected trendsin routing instability, and examine a number of anomalies andpathologies observed in the exchange of inter-domain routinginformation. The analysis in this paper is based on data collectedfrom BGP routing messages generated by border routers at fiveof the Internet core’s public exchange points during a nine monthperiod. We show that the volume of these routing updates isseveral orders of magnitude more than expected and that themajority of this routing information is redundant, or pathological.Furthermore, our analysis reveals several unexpected trends andill-behaved systematic properties in Internet routing. We finallyposit a number of explanations for these anomalies and evaluatetheir potential impact on the Internet infrastructure.

Index Terms—Communication system, communication systemrouting, computer network, Internet, routing, stability.

I. INTRODUCTION

SINCE THE END of the NSFNet backbone in April of1995, the Internet has seen explosive growth in both

size and topological complexity. This growth has placedsevere strain on the commercial Internet infrastructure. Regularnetwork performance degradations, stemming from bandwidthshortages and a lack of router switching capacity, have leadthe popular press to warn of the imminent death of the Internet[12]. Routing instability, informally defined as the rapid changeof network reachability and topology information, has a num-ber of origins including router configuration errors, transientphysical and data link problems, and software bugs. Instability,also referred to as “route flaps,” significantly contributes topoor end-to-end network performance and degrades the overallefficiency of the Internet infrastructure. All of these sourcesof network instability result in a large number of routingupdates that are passed to the core Internet exchange pointrouters. Network instability can spread from router to routerand propagate throughout the network. At the extreme, route

Manuscript received August 8, 1997; revised May 8, 1998; approvedby IEEE/ACM TRANSACTIONS ON NETWORKING Editor C. Partridge. Thiswork was supported by the National Science Foundation under Grant NCR-9321060 and by the Intel Corporation. An early version of this paperappeared in Proceedings ACM SIGCOMM ’97; forwarded to TRANSACTIONSON NETWORKING by the SIGCOMM ’97 program committee.The authors are with the Department of Electrical Engineering and Com-

puter Science, University of Michigan, Ann Arbor, MI 48109-2122 USA (e-mail: labovit@merit.edu; rmalan@eecs.umich.edu; farnam@eecs.umich.edu).Publisher Item Identifier S 1063-6692(98)07739-5.

flaps have led to the transient loss of connectivity for largeportions of the Internet. Overall, instability has three primaryeffects: increased packet loss, delays in the time for networkconvergence, and additional resource overheard (memory,CPU, etc.) within the Internet infrastructure.The Internet is comprised of a large number of intercon-

nected regional and national backbones. The large publicexchange points are often considered the “core” of the Internet,where backbone service providers peer, or exchange trafficand routing information with one another. Backbone serviceproviders participating in the Internet core must maintain acomplete map, or default-free routing table, of all globally vis-ible network-layer addresses reachable throughout the Internet.The Internet is divided into a large number of different re-

gions of administrative control commonly called autonomoussystems. These autonomous systems (AS’s) usually have dis-tinct routing policies and connect to one or more remote AS’sat private or public exchange points. AS’s are traditionallycomposed of network service providers or large organizationalunits like college campuses and corporate networks. At theboundary of each autonomous system, peer border routersexchange reachability information to destination IP addressblocks [2], or prefixes, for both transit networks, and networksoriginating in that routing domain. Most AS’s exchange rout-ing information through the border gateway protocol (BGP)[11].Unlike interior gateway protocols, such as IGRP and OSPF,

that periodically flood an intradomain network with all knowntopological information, or link state entries, BGP is anincremental protocol that sends update information only uponchanges in network topology or routing policy. Moreover,BGP uses TCP as its underlying transport mechanism, incontrast to many interior protocols that build their own re-liability on top of a datagram service. As a path vector routingprotocol, BGP limits the distribution of a router’s reachabilityinformation to its peer, or neighbor routers. A path is asequence of intermediate autonomous systems between sourceand destination routers that form a directed route for packetsto travel. Router configuration files allow the stipulation ofrouting policies that may specify the filtering of specific routes,or the modification of path attributes sent to neighbor routers.Routers may be configured to make policy decisions basedon both the announcement of routes from peers and theiraccompanying attributes. These attributes, such as multi-exitdescriptor (MED), may serve as hints to help routers choosefrom alternate paths to a given destination.Backbone border routers at public exchange points com-

monly have thirty or more external, or interdomain, peers,as well as a large number of intradomain peering sessions

1063–6692/98$10.00 © 1998 IEEE

Background

• Another ACM SIGCOMM test of time award paper

• A first large scale study of BGP traffic• Motivated much improvement to BGP

25

Terms

• WADiff: withdrawal à announcement• AADiff: announcement à announcement• WADup: same route withdrawal à

announcement• AADup: same route announcement à

announcement• WWDup: same route withdrawal à

withdrawal26

Observed pathologies

• Repeated WWDup, WADup, AADup• Why are they pathologies?

27

28

• Majority of BGP updates are WWDup• WWDup belong to ASes that never announce them

Why?

• Many reasons. Not fully understood– Stateless BGP, does not remember what

have sent to peers– Send withdrawals to all peers– Physical link errors– Unjittered timers causing synchronization– IGP, BGP interactions– Conflicting routing policies

29

31

Data analysis techniques

• Time series analysis• Frequency analysis

– Fast Fourier transform – Maximum entropy spectral estimation

• Different estimation methods, but both find significant frequencies at seven days, and 24 hours

Main results• Much more updates than expected

– 99% is pathological. Impressive!– A taxonomy to analyze pathologies

• Root causes unknown– Worth a new study

• Motivated much follow-up work

32

33

IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 9, NO. 3, JUNE 2001 293

Delayed Internet Routing ConvergenceCraig Labovitz, Member, IEEE, Abha Ahuja, Member, IEEE, Abhijit Bose, and Farnam Jahanian

Abstract—This paper examines the latency in Internet pathfailure, failover, and repair due to the convergence properties ofinterdomain routing. Unlike circuit-switched paths which exhibitfailover on the order of milliseconds, our experimental mea-surements show that interdomain routers in the packet-switchedInternet may take tens of minutes to reach a consistent view of thenetwork topology after a fault. These delays stem from temporaryrouting table fluctuations formed during the operation of theBorder Gateway Protocol (BGP) path selection process on Internetbackbone routers. During these periods of delayed convergence, weshow that end-to-end Internet paths will experience intermittentloss of connectivity, as well as increased packet loss and latency.We present a two-year study of Internet routing convergencethrough the experimental instrumentation of key portions of theInternet infrastructure, including both passive data collectionand fault-injection machines at major Internet exchange points.Based on data from the injection and measurement of severalhundred thousand interdomain routing faults, we describe severalunexpected properties of convergence and show that the measuredupper bound on Internet interdomain routing convergence delayis an order of magnitude slower than previously thought. Ouranalysis also shows that the upper theoretic computational boundon the number of router states and control messages exchangedduring the process of BGP convergence is factorial with respectto the number of autonomous systems in the Internet. Finally,we demonstrate that much of the observed convergence delaystems from specific router vendor implementation decisions andambiguity in the BGP specification.

Index Terms—Failure analysis, Internet, network reliability,routing.

I. INTRODUCTION

I N A BRIEF number of years, the Internet has evolved froman experimental research and academic network to a com-modity, mission-critical component of the public telecommuni-cation infrastructure. During this period, we have witnessed anexplosive growth in the size and topological complexity of theInternet and an increasing strain on its underlying infrastruc-ture. As the national and economic infrastructure has becomeincreasingly dependent on the global Internet, the end-to-endavailability and reliability of data networks promises to havesignificant ramifications for an ever-expanding range of applica-

Manuscript received July 14, 2000; revised November 14, 2000; approvedby IEEE/ACM TRANSACTIONS ON NETWORKING Editor C. Diot. This workwas supported in part by the National Science Foundation under GrantsNCR-971017 and NCR-961276. Most of the work on this paper was carriedout while the first author was at Microsoft Research.C. Labovitz was with Microsoft Research, Redmond, WA 98052 USA. He

is now with Arbor Networks, Ann Arbor, MI 48104 USA and also with theUniversity of Michigan, Ann Arbor, MI 48109 USA.A. Ahuja is with Arbor Networks, Ann Arbor, MI 48104 USA.A. Bose is with the Center for Parallel Computing, University of Michigan,

Ann Arbor, MI 48109 USA.F. Jahanian is with Arbor Networks, Ann Arbor, MI 48104 USA and also with

the University of Michigan, Ann Arbor, MI 48109 USA.Publisher Item Identifier S 1063-6692(01)04725-2.

tions. For example, transient disruptions in backbone networksthat previously impacted a handful of scientists may now causeenormous financial loss and disrupt hundreds of thousands ofend users.Since its commercial inception in 1995, the Internet has

lagged behind the public switched telephone network (PSTN)in availability, reliability, and quality of service (QoS). Factorscontributing to these differences between the commercialInternet infrastructure and the PSTN have been discussedin various literature [26], [18]. Although recent advances inthe IETF’s Differentiated Services working group promise toimprove the performance of application-level services withinsome networks, across the wide-area Internet these QoSalgorithms are usually predicated on the existence of a stableunderlying forwarding infrastructure.The Internet backbone infrastructure is widely believed to

support rapid restoration and rerouting in the event of individuallink or router failures. At least one report places the latency ofinterdomain Internet path failover on the order of 30 secondsor less based on qualitative end user experience [16]. Thesebrief delays in interdomain failover are further believed to stemmainly from queuing and router CPU processing latencies [3,(message digests 11/98, 1/99)]. In this paper, we show that mostof this conventional wisdom about Internet failover is incorrect.Specifically, we demonstrate that the Internet does not supporteffective interdomain failover and that most of the delay in pathrestoral stems solely from the unexpected interaction of config-urable routing protocol timers and specific router vendor pro-tocol implementation decisions during the process of delayedBorder Gateway Protocol (BGP) convergence.The slow convergence of distance vector (DV) routing algo-

rithms is not a new problem [24]. DV routing requires that eachnode maintain the distance from itself to each possible desti-nation and the vector, or neighbor, to use to reach that des-tination. Whenever this connectivity information changes, therouter transmits its new distance vector to each of its neighbors,allowing each to recalculate its routing table.DV routing can take a long time to converge after a topolog-

ical change because routers do not have sufficient informationto determine if their choice of next hop will cause routing loopsto form. The count-to-infinity problem [24] is the canonical ex-ample used to illustrate the slow convergence in DV routing.Numerous solutions have been proposed to address this issue.For example, including the entire path to the destination, knownas the path vector approach, is used in the BGP, the interdo-main routing protocol in the Internet. Other attempts to solvethe count-to-infinity problem or accelerate convergence inmanycommon cases include techniques such as split horizon (withpoison reverse), triggered updates, and the diffusing update al-gorithm [9].

1063–6692/01$10.00 ©2001 IEEE

34

Delayed Internet Convergence

Measurement Problemdiscovery

Modeling& analysis Improvement

• Methodologies

35

Experiments setup

• Actively inject BGP faults– How is fault injected?

• Passively listen at peering sessions, and use NTP synchronized machines to calculate the convergence time

• Actively send probe packets to observe end-to-end packet loss and latency

• Much BGP work later uses similar measurement techniques

Types of routing events• Tup: A previously unavailable route is

announced as available.

• Tdown: A previously available route is withdrawn.

• Tshort: An active route with a long ASPathis implicitly replaced with a new route possessing a shorter ASPath.

• Tlong: An active route with a short ASPathis implicitly replaced with a new route possessing a longer ASPath.

36

37

Results show delayed convergence

• Bad news travels slow.

38

Slow routing convergence results in poor end-to-end performance

39

What causes the delayed routing convergence?

• A simple BGP convergence model reveals that in the worse case, all possible paths are explored before a prefix is withdrawn.

• No minimum advertisement timer: synchronized network, global message queue

0

1 2

R

(*0R, ∞,, 2R) (∞, ∞, *2R)(∞, ∞, *20R)

(*0R, 1R, ∞,)(01R, *1R, ∞)(*01R, 10R, ∞,)

(∞, *1R, 2R)(∞, ∞, *2R)(∞, ∞, ∞)

01R 01R

10R

10R

20R

20R

40

Failover• BGP is designed for scaling more than fast

failover– Many mechanisms favor this balance– Route flap damping, for example.

• If excess routing changes (�flapping�), ignore for some time.

• Has unexpected effects on convergence times.

– Route advertisement/withdrawal timers in the 30 second range

– Effect: tens of seconds to many minutes to recover from �simple� failures.

– 15-30 minute outages not uncommon.

BGP prefix hijacking attacks

• Security: There is almost none!– Some providers filter what announcements their

customers can make. Not all do.

41

How can it happen?

42

ISP

Duke UNC152.3.0.0/16

152.3.0.0/16

152.3.137.212

AS 7007 Incident

• Was a major disruption on April 25,

1997

• AS 7007 leaked part of its routing table

to the entire Internet

• Routes were deaggregated /24

43https://en.wikipedia.org/wiki/AS_7007_incident

44

• Pakistan aimed to censor YouTube

• AS 17557 (Pakistan Telecom) advertised a more specific YouTube prefix 208.65.153.0/24 to its neighbor AS 3491 (PCCW)– YouTube 208.65.153.0/22

• PCCW advertised to the rest of theInternet

• YouTube advertised more specific prefixes to regain control 45

Hard to Trace Spam

• If a source IP sends spam, it may get on ablacklist

• How to evade blacklisting?– Hijack a prefix (typically an unassigned one)– Send spam from that prefix– Withdraw

46

Summary• Infer AS relationships• Some known BGP problems

– Scaling– Pathological announcements– Delayed convergence– Prefix hijacking

• Re-run the experiments for your course projects!– Observe the IPv6 Internet

47