Failures in an Operational IP Backbone NetworkAthina Markopoulou

�, Gianluca Iannaccone � , Supratik Bhattacharyya � , Chen-Nee Chuah � , Yashar Ganjali

�,

Christophe Diot ��EE Department

Stanford Univ.,CA, USA

� Sprint ATL

Burlingame, CA, USA

� ECE Department

UC Davis, CA, USA

� Intel Research

Cambridge, UK

Abstract— As the Internet evolves into a ubiquitous commu-nication infrastructure and supports increasingly important ser-vices, its dependability in the presence of various failures becomescritical. In this paper, we analyze IS-IS routing updates from theSprint IP backbone network to characterize failures that affectIP connectivity. Failures are first classified based on probablecauses such as maintenance activities, router-related and opticallayer problems. Key temporal and spatial characteristics ofeach class are analyzed and, when appropriate, parameterizedusing well-known distributions. Our results indicate that 20%of all failures is due to planned maintenance activities. Of theunplanned failures, almost 30% are shared by multiple linksand can be attributed to router-related and optical equipment-related problems, while 70% affect a single link at a time. Ourclassification of failures according to different causes revealsthe nature and extent of failures in today’s IP backbones.Furthermore, our characterization of the different classes can beused to develop a probabilistic failure model, which is importantfor various network design and traffic engineering problems.

I. INTRODUCTION

The core of the Internet consists of several large networks(often referred to as backbones) that provide transit services tothe rest of the Internet. These backbone networks are usuallywell-engineered and adequately provisioned, leading to verylow packet losses and negligible queuing delays [1], [2]. Thisrobust network design is one of the reasons why the occurrenceand impact of failures in these networks have received littleattention. The lack of failure data from operational networkshas further limited the investigation of failures in IP back-bones. However, such failures occur almost everyday [3] andan in-depth understanding of their properties and impact isextremely valuable to Internet Service Providers (ISPs).

In this paper, we address this deficiency by analyzingfailure data collected from Sprint’s operational IP backbone.The Sprint network uses an IP-level restoration approach forsafeguarding against failures with no protection mechanismsin the underlying optical fiber infrastructure [4]. Therefore,problems with any component at or below the IP layer (e.g.,router hardware/software failures, fiber cuts, malfunctioningof optical equipment, protocol misconfigurations) manifestthemselves as the loss of connectivity between two directlyconnected routers, which we refer to as an IP link failure.

Email addresses: A. Markopoulou - amarko@stanfordalumni.org,G. Iannaccone - gianluca.iannaccone@intel.com, S. Bhattacharyya -supratik@sprintlabs.com, C. N. Chuah - chuah@ece.udavis.edu, Y. Ganjali -yganjali@stanford.edu C. Diot - christophe.diot@intel.com This work wasconducted when the authors were affiliated (or in collaboration) with SprintAdvanced Technologies Lab.

IS-IS [5] is the protocol used for routing traffic insidethe Sprint network. When an IP link fails, IS-IS automati-cally recomputes alternate routes around the failed link, ifsuch routes exist. The Sprint network has a highly meshedtopology to prevent network partitioning even in the event ofwidespread failures involving multiple links. However, linkfailures may still adversely affect packet forwarding. WhileIS-IS recomputes alternate routes around a failure, packetsmay be dropped (or caught in a routing loop) by routersthat lack up-to-date forwarding information. Moreover, whentraffic fails over to backup paths, links along that path mayget overloaded leading to congestion and eventually to packetloss [6]. Routing re-convergence may also impose significantburden on router processors. Futhermore, if failures happentoo frequently, route-flapping may lead to network instability.For all these reasons, failures are a major concern for anoperational network.

In this work, we collect IS-IS routing updates from theSprint network using passive listeners installed at geograph-ically diverse locations. These updates are then processed toextract the start-time and end-time of each IP link failure. Thedata set analyzed consists of failure information for all linksin the continental US from April to October 2002.

The first step in our analysis is to classify failures intodifferent groups according to their underlying cause, i.e. thenetwork component that is responsible. This is a necessarystep for developing a failure model where the faults of eachcomponent can be addressed independently. In our classifica-tion, we proceed as follows. First, link failures resulting fromscheduled maintenance activities are separated from unplannedfailures. Then, among the unplanned failures, we identifyshared failures, i.e. failures on multiple IP links at the sametime due to a common cause. Among shared link failures, wefurther distinguish those that have IP routers in common andthose that have optical equipment in common. The remainingfailures represent individual link failures, i.e. faults that affectonly one link at a time. For the individual failures, we furtherdifferentiate groups of links, based on the number of failureson each link.

The second step in our analysis is to provide the spatialand temporal characteristics for each class separately, e.g., thedistributions of the number of failures per link, time betweenfailures, time-to-repair, etc. When possible, we provide param-eters for these characteristics using well-known distributions.

Our results indicate that 20% of all failures can be attributed

to scheduled network maintenance activities. Of the remainingunplanned failures, 30% can be classified as shared. Half of theshared failures affected links connected to a common router,pointing to a router-related problem; the rest affect links thatshare optical infrastructure, indicating an optical layer fault.The remaining 70% of the unplanned failures are individuallink failures caused by a variety of problems. Interestingly,the failure characteristics of individual links vary widely- lessthan 3% of the links in this class contribute to 55% of allindividual link failures.

The contributions of this work are as follows:� We perform an in-depth analysis of IS-IS failure data

from a large operational backbone. This has not beenattempted before, largely due to the lack of availabilityof such data sets.

� We classify failures based on their causes. This methodol-ogy can enable an ISP to isolate failures attributable to acause and pinpoint areas of improvement. For example,this approach can help determine whether a significantnumber of failures are related to optical-layer problemsand identify optical components that should potentiallybe updated.

� We provide the statistical characteristics of each classof failures and, when appropriate, we approximate themwith well-known distributions. In addition, we study howthese characteristics vary with time. The parameters ob-tained can be used to generate realistic failure scenarios,which are important as input to various network designand traffic engineering problems that take failures intoaccount.

An earlier version of this work appeared in [7]. This journalpaper improves that work and extends it with additional mate-rials, including (i) an analysis of SONET alarms in conjuctionwith the IS-IS data and (ii) the analysis of shorter (one-month) time periods in order to understand how the statisticalcharacteristics vary with time.

The paper is organized as follows. Section II presents relatedwork in the area of failure analysis and fault management.Section III describes the data collection process in the Sprintbackbone and provides an overview of the data set understudy. Section IV describes our classification methodologyand evaluates its accuracy. Section V describes the resultsof our classification of failures and the characteristics ofeach identified class. Section VI applies the classification andcharacterization methodologies to shorter (one-month) timeperiods and discusses how the statistical characteristics of eachfailure class vary with time. Section VII discusses how ourcharacterization can be used to build a failure model, andidentifies open issues for further investigation. Section VIIIconcludes the paper.

II. RELATED WORK

The availability of spare capacity and sound engineeringpractices in commercial IP backbones makes it easy to achievetraditional QoS objectives such as low loss, latency and jitter.Recent results show that the Sprint network provides almost

no queuing delays [8], [2], negligible jitter [2] and is capableof supporting toll-quality voice service [9].

On the other hand, failures can degrade network per-formance by reducing available capacity and disrupting IPpacket forwarding. Common approaches for ensuring networksurvivability in the presence of failures include protection andrestoration at the optical layer or the IP layer [10], [11],[4]. A significant amount of effort has been investing inachieving sub-second convergence in IS-IS [12]. In addition,a number of new approaches have been proposed to accountfor backbone failures, including the selection of link weightsin the presence of transient link failures [13], [14], deflectionrouting techniques to alleviate temporary link overloads due tofailures [6], network availability-based service differentiation[15], and failure insensitive routing [16].

All of the above approaches require a thorough under-standing of the cause and characteristics of backbone failures.However, such an understanding has been limited partly bya lack of measurement data from operational networks, andpartly by a focus on traditional QoS objectives such as lossand delay. In some cases, traceroutes were used to study therouting behavior in the Internet. These include studies onrouting pathologies, stability and symmetry [17], stationarityof Internet path properties [18], and evaluation of routing-based and caching techniques to deal with failures [19]. To thebest of our knowledge, [20] is the only work that uses OSPFrouting updates for failure analysis, although its primary focusis on studying stability of inter-domain paths. More recently,[3] has performed a preliminary analysis of backbone linkfailures based on IS-IS routing updates. Our work builds on[3] and makes the following new contributions. We study alarger and more recent data set, classify failures accordingto probable causes, and provide characterization for eachclass that can be used to build a failure model. This paperimproves the earlier conference version [7], and also extendsit by additional materials, i.e., (i) the correlation of SONETalarms with our IS-IS data (Section IV.F) and (ii) the per-month statistical characterization (Section VI). Compared tothis work that focuses on understanding and characterizingfailures, ongoing and future work will focus on the impact offailures on network performance [21].

To put things in perspective, reliability is an aspect of de-pendable distributed systems and has been extensively studiedin the context of fault-tolerant distributed systems. A landmarkpaper on failures in Tandem systems and the techniques toprevent them is [22]. In parallel and even earlier, a mathemat-ical framework was developped in the Operations Researchworld to manage the reliability and risk in systems composedof various components [23]. In general, complex systemsthat have passed the stage of proof of concept and havematured into industrial grade systems, are expected to providehigh reliability/availability to their users. Telephone networksand power grids are examples of such dependable networks.Recently, there has been an increasing interest in the reliabilityof end-to-end services in the Internet. Examples of recent workinclude [24] where human errors and incorrect configurations

Fig. 1. The Sprint IP topology in the continental US

are identified as a main source of errors, [25] where giantscale web services are examined, and [26] where fast recoveryis compared to high reliability. Within the above context, ourwork targets failures that affect routing and availability acrossa single backbone network.

III. FAILURE MEASUREMENTS

In this section, we briefly describe the Sprint network, wediscuss the types of failures that impact IP connectivity, wepresent our methodology for extracting link failure informationfrom IS-IS routing updates and we summarize the data set.

A. The Design of the Sprint IP backbone

The Sprint IP topology in the continental US is shown inFig. 1. It consists of a collection of points of presence (PoPs),in various cities, connected via high speed links. Each PoP(shown as a node in Fig. 1, without details) consists itself ofa number of backbone and access routers: the first connect toother PoPs and the latter connect to clients and to backbonerouters. We refer to all the links between those routers as‘logical’ links, or IP links or just links. This logical IP net-work is layered over a dense wavelength-division multiplexing(DWDM) optical infrastructure with SONET framing.

It worths mentioning that the Sprint network is designedwith sufficient redundancy and careful engineering to be robustin case that some links fail. For example, as we see inFig. 1, there are multiple parallel links connecting a pairPoPs; furthermore these links are terminated on differentrouter within the same PoP. Inside a single POP, backbonerouters are connected in a fully meshed topology; each accessrouter connects customers to two different backbone routers.Among all PoPs, a full mesh would be prohibitively expensive;however there is still a large degree of connectivity and thelogical links are carefully chosen to use disjoint physical links.

Finally, sufficient capacity is provisioned to carry the re-routedtraffic in the case of failures.

B. Failures with an impact on IP connectivity

There are two main approaches for sustaining end-to-endconnectivity in IP networks in the event of failures: protec-tion and restoration. Protection is based on fixed and pre-determined failure recovery, with a working path set up fortraffic forwarding and an alternate protection path provisionedto carry traffic if the primary path fails. Restoration techniquesattempt to find a new path on-demand to restore connectivitywhen a failure occurs. Protection and restoration mechanismscan be provided either at the optical or at the IP layer, withdifferent cost-benefit tradeoffs [4], [11].

The Sprint IP network relies on IP layer restoration (via IS-IS protocol) for failure recovery. All failures at or below the IPlayer that can potentially disrupt packet forwarding manifestthemselves as the loss of IP links between routers. The failureor recovery of an IP link leads to changes in the IP-levelnetwork topology. When such a change happens, the routersat the two ends of the link notify the rest of the networkvia IS-IS. Therefore, the IS-IS update messages constitutethe most appropriate data set for studying failures that affectconnectivity.

Failures can happen at various protocol layers in the networkfor different reasons. At the physical layer, a fiber cut ora failure of optical equipment may lead to loss of physicalconnectivity. Hardware failures (e.g. linecard failures), routerprocessor overloads, software errors, protocol implementationand misconfiguration errors may also lead to loss of connec-tivity between routers. When network components (such asrouters, linecards, or optical fibers) are shared by multiple IPlinks, their failures affect all the links. Finally, failures maybe unplanned or due to scheduled network maintenance. Notethat at the IS-IS level, we observe the superposition of all theabove events. Inferring causes from the observed IS-IS failuresis a difficult reverse engineering problem.

C. Collecting and Processing ISIS Updates

We use the Python Routing Toolkit (PyRT)1 to collect IS-ISLink State PDUs (LSPs) from our backbone. PyRT includesan IS-IS “listener” that collects LSPs from an IS-IS enabledrouter over an Ethernet link. The router treats the listener in thesame way as other adjacent routers, hence it forwards to thelistener all LSPs that it receives from the rest of the network.Since IS-IS broadcasts LSPs through the entire network, ourlistener is informed of every routing-level change occurringanywhere in the network. However, the listener is passive inthe sense that it does not transmit any LSPs to the router. Thesession between the listener and the router is kept alive viaperiodic IS-IS keepalive (Hello) messages. Upon receiving anLSP, the listener prepends it with a header in MRTD format(extended to include timestamp of micro-second granularity)and writes it out to a file. The data presented in this paper

1The source code is publicly available at http://ipmon.sprint.com/pyrt

were collected from a listener at a Sprint backbone Point-of-Presence (POP) in New York.

Whenever IP level connectivity between two directly con-nected routers is lost, each router independently broadcasts a“link down” LSP through the network. When the connectivityis restored, each router broadcasts a “link up” LSP. We refer tothe loss of connectivity between two routers as a link failure.

The LSPs from the two ends of a link reporting loss orrestoration of IP connectivity may not reach our listener at thesame time. The start of a failure is recorded with the MRTDtimestamp of the first LSP received at our listener that reports“link down”. The end of each failure is recorded with theMRTD timestamp of the second LSP received at our listenerthat reports “link up”. This asymmetry is conformant withhow the IS-IS protocol reacts to routing updates. As soon asa router receives the first LSP reporting an “link down”, itconsiders the IP connectivity to be lost without waiting forthe second LSP. Hence, the first LSP is sufficient to trigger aroute re-computation, which may lead to a disruption in packetforwarding. However, in order to consider the IP connectivityrestored, a router waits until it receives LSPs reporting “linkup” from both ends of a link. In the rest of the paper, werefer to the time between the start and the end of a failure, asdefined above, as the time-to-repair for the failure.

D. The Failures Data Set

Using the steps described above, we can determine thestart and end times for failures on every link in the Sprintbackbone. The data are collected for the period between April1st and October 21st 2002, in the continental US. This dataset involves a large number of links, routers and POPs (inthe order of thousands, hundreds and tens respectively). Weconsider that link failures with time-to-repair longer than 24hours are due to a links being decommissioned rather than toaccidental failures, and therefore we exclude them from thefailures data set. Indeed, the usual time-to-repair for links inactive use is in the order of hours and not in the order of days.

Fig. 2 shows the failures in the data set under study, acrosslinks and time. A single dot at

���������is used to represent a

failure that started at time�, on link

�. One can see that failures

are part of the everyday operation and affect a variety of links.We also observe that the failures occurrence follows patterns,such as (more or less prominent) vertical and horizontal linesof different lengths. In the rest of the paper, we further usethese visual patterns as guidance for our failure classification.2

E. Additional Data Sets

The IS-IS logs described above, is the main data set weuse in this paper to identify and characterize failures at theISIS level. However, in our methodology Section IV, we alsouse two auxiliary data sets to confirm that some failures are

2The scale of the figure is chosen to emphasize the horizontal and verticalpatterns. Times-to-repair are not represented in the figure and the area coveredby the dots represents neither the total duration nor the impact of link failureson the Sprint backbone.

Apr May Jun Jul Aug Sep Oct Nov

200

400

600

800

1000

1200

Link

Num

ber

Time

Fig. 2. Data set under study: failures in continental US between April 1 andOctober 21, 2002.

optical-related. In Section IV-D, we use the IP-to-optical map-ping and in Section IV-F we use SONET alarm logs; see therespective sections for a detailed description. Unfortunately,these sets are not as complete (over the 7 months period understudy) as the main ISIS data set, and are used here only forsupplementary confirmation.

IV. CLASSIFICATION METHODOLOGY

This section describes our methodology for classifyingfailures according to their causes and properties. We attemptto infer the causes by leveraging patterns observed in theempirical data and by correlating them with the possiblecauses. We first give an overview of our classification andthen we discuss each class in detail.

A. Overview

Our approach is to use several hints obtained from the IS-IS failure data to identify groups of failures due to differentcauses. A visual inspection of Fig. 2 provides insights intohow to perform this classification. We observe that the failuresare not uniformly scattered- there are vertical and horizontallines. The vertical lines correspond to links that fail at thesame time (indicating a shared network component that fails)or to links that fail close in time (e.g. due to a maintenanceactivity) but appear almost aligned in the plot. The horizontallines correspond to links that fail more frequently than others.Apart from these lines, the remaining plot consists of roughlyuniformly scattered points.

Our classification of failures is summarized in Fig. 3 andconsists of the following steps. We first separate failures due toscheduled Maintenance from Unplanned failures. We analyzethe unplanned failures in greater depth since these are the onesthat an operator seeks to minimize. We distinguish betweenIndividual Link Failures and Shared Link Failures, dependingon whether only one or multiple links fail at the same time.Shared failures indicate that the involved links share a network

Individual Link Failures

Low Failure Links

High Failure Links

Shared Link Failures

Maintenance

Data Set

Optical Related Unspecified

Overlapping

Unplanned Failures

Simultaneous

(Router Related)

Fig. 3. Classification of failures

component that fails. This component can be located eitheron a common router (e.g. a linecard or route processor in therouter) or in the underlying optical infrastructure (a commonfiber or optical equipment). Therefore, we further classifyshared failures into three categories according to their cause:Router-Related, Optical-Related and Unspecified (for sharedfailures where the cause cannot be clearly inferred). We dividelinks with individual failures into High Failure and LowFailure Links depending on the number of failures per link.In Fig. 3, maintenance and shared failures correspond to thevertical lines, high failure links correspond to the horizontallines, and low failure links correspond to the roughly uniformplot that remains after excluding all the above classes offailures.

We now consider each class separately and describe (i) therules that we use to decide whether a failure belongs to thisspecific class and (ii) how we obtain partial confirmation forthe inferred cause.

B. Maintenance

Failures resulting from scheduled maintenance activities areunavoidable in any network. Maintenance is usually scheduledduring periods of low network usage, in order to minimize theimpact on performance. The maintenance window in the USSprint backbone network is Mondays 5am-2pm, UTC time. Itturns out that failures during this window are responsible forthe most prominent vertical lines in Fig. 2, including the onesin September - October.

C. Simultaneous Failures

In the shared failures class, we first identify failures thathappen simultaneously on two or more links. Failures onmultiple links can start or finish at exactly the same time,when a router reports them in the same LSP. For example,when a linecard fails, a router may send a single LSP to reportthat all links connected to this linecard are going down. Whenour listener receives this LSP, it will use the same MRTDtimestamp as the start for all the reported failures. Similarly,when a router reboots, it sends an LSP reporting that manyof the links connected to it are going up. When our listener

Tstart Tend

link 1

link 2

link 3

link 4

Wstart Wend

starting simultaneously Event of failures Event of

overlapping failures(a) (b)

Fig. 4. Example events of simultaneous and overlapping failures. The timeis according to the MRTD prepended at the ISIS listener.

receives this LSP, it will use the same MRTD timestamp asthe end for all the reported failures. (However, it still needsto receive an LSP from the other end to declare the end of afailure.)

In our data, we identify many such cases. An example isshown in Fig. 4(a): 4 links are going down at exactly thesame time

����������(and 3 out of 4 come up at the exactly

same time�����

). We refer to such failures as simultaneousfailures and we group them into events. Similtaneous failuresstart and/or finish at the exact same time, with an accuracyof microseconds. This indicates that they are more likely tobe due to a common cause rather than to a coincidence. Weconjectured that these links share a common component whichfails and causes the links to go down together. Furthermore,a router must have reported in the same LSP that these linksgo down together; our listener receives this LSP and prependsthe same MRTD timestamp on these failures, and that is whythey appear to fail simultaneously.

We then verified this conjecture. For every event of simul-taneous failures found in the data set, we verified that allinvolved links are indeed connected to a common router. Andconversely, there is no simultaneous failure event that doesnot involve a common router, which confirms our intuition.Therefore, we attribute these events (simultaneous failureson links connected to the same router) to problems on thecommon router, such a router crash or reboot, a linecardfailure or reset, or a CPU overload. In the rest of the paper,we refer to these failures as Router-Related. Unfortunately,based on the available data set, it is impossible to do anyfiner classification, i.e. we are unable to identify whether therouter-failure is due to high load, software or hardware erroror human misconfiguration.

Occasionally, a link in a router event may come up laterthan the others, as shown in Fig. 4(a). This can happen eitherbecause the link comes up later (e.g. router interfaces comingup one-by-one) or because the LSP from the other end of thelink reaches our listener later (either delayed or lost). However,in 50% of the router events identified in the data set, all linkscame up at the exact same time; in 90% of the cases the lastlink came up no later than 2 minutes after the first link.

D. Overlapping Failures

After excluding the simultaneous failures, we relax the timeconstraint from “simultaneous” to “overlapping”, i.e. we lookfor events where all failures start and finish within a timewindow of a few seconds. An example is shown in Fig. 4(b),failures on all 4 links start within

� ����� � �and finish within� ��

seconds from each other.Overlapping failures on multiple links can happen when

these links share a network component that fails and ourlistener records the beginning and the end of the failureswith some delays

� ����� � �and

� ��� . For example, a fiber cut

leads to the failure of all IP links over the fiber, but maylead to overlapping failures in our listener for several reasons.First, there are multiple protocol timers involved in the failurenotification and in the generation of LSPs by the routers atthe ends of the links. Most of these timers are typically onthe order of tens of milli-seconds up to a few seconds. Thedominant ones are the IS-IS carrier delay timer [3] with default2 seconds to report a link going down and 12 seconds to reporta link going up. The timers can be configured to have differentvalues on different routers. Finally, the LSPs from the two endsof the link can reach our listener through different paths in thenetwork and thus may incur different delays; or an LSP maybe lost, leading to an additional retransmission delay.

The choice of windows,� ��������

and� ���

, becomes im-portant for a meaningful definition of overlapping failures.If the windows are chosen too long, failures that overlap bycoincidence may be wrongly interpreted as shared failures.Windows that are too short may fail to detect some sharedfailures. We choose

� ����� � �and

� �� to be 2 and 12 sec to

match the default timers used to report a link down or uprespectively. We also varied

� ��������from 0.5 to 10 sec and� ��

from 0.5 to 20 sec and observed that the number ofoverlapping failures or events is relatively insensitive aroundthe chosen values.

We now focus on identifying the network component thatis responsible for the overlapping failures. Links can sharecomponents either at a router or in the optical infrastructure.

Optical-Related. Among all overlapping events, we identifythose that involve only inter-POP links and that do not sharea common router. It turns out that 75% of all overlappingevents and 80% of all overlapping failures are of this type, seeTable I. We consider those events to be Optical-Related forthe following reason. Since the links in the same event haveno router in common, an explanation for their overlappingfailures is that they share some underlying optical componentthat fails, such as a fiber or another piece of optical equipment.

To verify this conjecture, we use an additional database: theIP-to-Optical mapping of the Sprint network. This databaseprovides the mapping from the IP logical topology to theunderlying optical infrastructure. It provides the list of opticalequipment used by every IP link. The optical topology con-sists of sites (cities where optical facilities are located) andsegments (pair of sites connected with an optical fiber). IPlinks share necessarily some sites or segments.

TABLE I

SUMMARY OF OVERLAPPING EVENTS

Classification % %of event events failures

Overlapping 100% 100%Optical-Related 75% 80%

Unspecified 25% 20%

TABLE II

USING THE IP-TO-OPTICAL MAPPING TO CONFIRM THAT LINKS IN THE

SAME OPTICAL EVENT SHARE AN OPTICAL COMPONENT

Optical-Related Events %

Found in the database 93% of optical eventsAll links have common site(s) 96% of found events

All links have common segment(s) 98% of found events

Table II summarizes our findings in the IP-to-Opticaldatabase. Out of all overlapping events that we classify asoptical-related (i.e. inter-POP without a common router), wewere able to find 93% of them in the database (meaning thatall links in the same event were found in the database). Not alllinks are found in the database due to changes in the mapping.For each event found in the database, we check whether alllinks in the event share some optical component. We find that96% of the events found in the database, involve links that allshare at least one site; 98% of the found events involve linksthat all share at least one segment. In fact, links in the sameevent share even more than just one site or segment. Theyshare from 1 up to 30 sites (8.3 on average) and from 1 upto 27 segments (7.3 on average). These findings validate ourconjecture that the events classified as optical-related are mostlikely due to the failure of some optical component shared bymultiple IP links.

Unspecified. The overlapping failures that are not classifiedas optical-related fall in this class. These include overlappingfailures on inter-POP links connected to the same router,because the cause of the problem is ambiguous: it couldbe a problem at the router or at the optical infrastructure.They also include overlapping failures of links in differentPOPs, that clearly have no components in common and aredue to coincidence. For all these events, we are not able tosatisfactorily infer their causes and we call them Unspecified.We do not attempt to further analyze them as they account foronly 20% of the overlapping failures (see Table I), which isless than 3% of all the unplanned failures.

E. Individual Link Failures

After excluding all the above classes of failures from thedata set, we refer to the remaining failures collectively asIndividual Failures because they affect only one link at a time.

Let � �����be the number of individual failures for link

�,

where����� ����� �

. Let the maximum number of failures in asingle link be � ����� � � ����� � � �������

. For proprietary reasons,we show the normalized value ��� ��� ������������� � ������� � ����� ,instead of the absolute number � �����

. In Fig. 5, we plot ��� ��� �,

100

101

102

103

100

101

102

103

Low Failure Links

High Failure Links

Link (l), in decreasing number of failures

Nor

mal

ized

num

ber

of fa

ilure

s pe

r lin

k, n

n(l)

(2.5% of links, THR)

Fig. 5. Number of individual failures per link

for all links in decreasing order of number of failures. Thereare several interesting observations based on this graph. First,links are highly heterogeneous: some links fail significantlymore often than others, which motivates us to study themseparately. Second, there are two distinct straight lines in thislog-log plot in Fig. 5. We use a least-square fit to approximateeach one of them with a power-law: � ��� � � ������� ��

forthe left line and � ����� � ���� � ���

for the right line. Noticethat both the absolute ( � ��� �

) and the normalized ( ��� �����)

values follow a power-law with the same slope; therefore, theinterested reader is still able to use the normalized value tosimulate this behavior. The dashed lines in the figure, intersectapproximately at a point that corresponds to 2.5% of the linksand to a normalized number of failures ��� ������� �����

.We use this value as the threshold (

����� � �����) to

distinguish between two sub-classes: the High Failure Links( ��� ������� �����

) and the Low Failure Links (��� ��� �������

�����). High failure links represent only 2.5% of all links but

account for more than half of individual failures. All of themwere backbone links; most of them connected different POPs;half of them had a router in common with at least anotherhigh-failure link. High failure links may be in an advancedstage of their lifetime and their components fail frequently;or they may be undergoing an upgrade or testing operationfor a period of time. The remaining half of individual failureshappen on low failure links. Unlike all previous failure classes,low failure links do not have a prominent pattern either in timeor across links.

It is difficult to infer the cause of individual failures, eitheron high or on low failure links, based merely on the ISIS data;unlike the shared failures, there is no timing correlation withother links to exploit. They could be due to optical or anyother of the possible reasons. That is partially why we turnedto additional data sets.

F. SONET Alarms

In order to further confirm our classification methodology(and in particular to confirm the shared-optical related failuresand explore the individual failures), we tried to match the IS-IS

failures with failures reported in signals by the SONET layer,which we will refer to as the SONET alarms. Unfortunately,we only had SONET alarms available for the four later monthsof the measurement period, from July to October 2002.

A number of alarms from the SONET layer can be recordedand timestamped by routers in the Sprint network. Of thesealarms, the ‘Section Loss of Signal’ (SLOS) is the most criticalone and is triggered when a failure occurs in the optical layerand no bit transmissions are detected in the SONET signal forsome time (20-100us). When the failure is restored a ‘SLOScleared’ is sent. The IS-IS layer waits for some additionaltimeout, up to 10sec, before trigerring the beginning/end of afailure, in order to filter out transient effects. In view of thismechanism, we match an IS-IS failure with a SLOS signals asfollows: when an IS-IS link goes down/up at time

�according

to the IS-IS logs, we search for an SLOS alarm/cleared at anytime between

�! "���$#�%and

�.

Table III shows the percentage of IS-IS failures that werematched with SLOS signals for the failure classes definedby our classification methodology. Note that the percentagesshown in this table should be interpreted as qualitative ratherthan as quantitative arguments,3 for various reasons: firstbecause of the ambiguities in inferring the cause, mentionedbelow; second because there are various optical-layer logs inaddition to the SONET alarms; third, there are many SONETalarms, in addition to the critical SLOS signals.

We can make the following observations from Table III:� The most important observation is that the large ma-

jority (up to 87%) of the shared failures classified byour methodology as ‘optical-related’ matches the SLOSalarms, which is a confirmation of their cause. Thispercentage is clearly higher than in any of the otherclasses. We believe that the small remaining percentagethat is not matched to SLOS, could be matched toother SONET alarms or different optical-layer logs orcould indicate some false positives in the classification.However, even this first step of matching ISIS failuresto SLOS signals, sufficiently confirmed the cause of thelarge majority of shared optical-related failures.

� The shared failures classified as router-related have thelowest percentage matching the SLOS signals, as ex-pected. There is however a non-zero percentage of themmatching SLOS, because the optical layer could stillaffect two or more links of the same router sharing someoptical component.

� It is not possible to infer the causes of Individual Linkfailures based solely on the ISIS data. They can bedue to any of the possible reasons, including optical-layer problems. Therefore it is expected that some of theindividual failures (up to 52%) match the SLOS signals.The percentages are similar for low-failure links and for

3Note that the ISIS failures and not the SONET alarms are the focus ofthis paper. After the completion of this study, a more systematic collection ofthe SONET alarms got in place. The interested reader is referred to [21] fordetails on related ongoing work.

TABLE III

% OF IS-IS FAILURES MATCHED WITH SLOS ALARMS, FOR 4 MONTHS.

Failure Class Jul Aug Sep Oct

All failures 53% 54% 24% 35%

Shared Optical-related 87% 70% 60% 69%Shared Router-related 42% 40% 34% 27%

Individual Low-Failure Links 47% 52% 30% 49%All Individual (High+Low Failure) 41% 52% 20% 41%

TABLE IV

PARTITIONING FAILURES INTO CLASSES

Failure Class % of all % of unplanned

Data Set 100%Maintenance 20%Unplanned 80% 100%

Shared Router-Related 16.5%Shared Shared Optical-Related 11.4%

Unspecified 2.9%Individual High Failure Links 38.5%

Low Failure Links 30.7%

the remaining high-failure links, which do not seem tohave unusually high correlation with SLOS alarms.

V. FAILURE ANALYSIS

We now consider each class of failures separately and westudy its characteristics that are useful for re-producing itsbehavior. First, we apply the classification methodology ofthe previous section and count how many failures fall in eachclass. Table IV shows the contribution of each class to the totalnumber of failures.4 Then, we study the properties of each oneof the four classes and in particular: (i) the time-between-failures, (ii) the distribution of failures across components(links or routers), (iii) the time-to-repair, and (iv) the numberof links that fail together in a shared (router- or optical-related)event. Table V summarizes these properties for each class.Rows correspond to classes and columns correspond to proper-ties. In the rest of this section, we discuss each class separately,we provide empirical distributions for its properties and, whenpossible, we also fit them to well known-distributions. Fig. 6provides the empirical cumulative distribution function oftime-to-repair for each class of unplanned failures and willbe referenced repeatedly throughout this section.

A. Maintenance

20% of all failures happen during the window of 9-hoursweekly maintenance, although each such window accountsonly for 5% of a week. Fig.7 shows the occurrence of linkfailures due to scheduled maintenance. It turns out that thoseaccount for many of the vertical lines in Fig.2.

More than half of the failures during the maintenancewindow are also router-related (according to the definition

4For proprietary reasons, we provide a characterization in terms of per-centages and statistical properties, rather than in terms of absolute numbers.However, the information provided should be sufficient for the reader toreproduce realistic failure scenarios.

100

101

102

103

104

105

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time−to−Repair (sec)

Em

piric

al C

DF

router−related

optical−related

unplanned

high failure links

low failure links

(individual)

(individual)

(shared)

(shared)

maintenance

Fig. 6. Cumulative distribution function (CDF) of the time-to-repair for eachclass of unplanned failures

Apr May Jun Jul Aug Sep Oct Nov

200

400

600

800

1000

1200

Time

Link

Num

ber

Fig. 7. Failures during weekly maintenance windows

of Section IV-C). This is expected as maintenance opera-tions involve shutting down and (re)starting routers and in-terfaces. Also, Fig. 6 shows that the CDF of time-to-repairfor maintenance-related failures, is close to the CDF for therouter-related failures, which further supports the observationthat many of the maintenance failures are router-related. Atypical maintenance window for a given interface/router is onehour, although it can take less than that.

B. Router-Related Failures

Router-related events are responsible for 16.5% of theunplanned failures. They happen on 21% of all routers. 87%of these router events (or 93% of the involved failures) happenon backbone routers and the remaining 7% happens on accessrouters. An access router runs IS-IS only on two interfacesconnecting to the backbone but not on the customer side.

Router events are unevenly distributed across routers. Let� �

��

be the number of events in router � and ��� ��� �

����� � � ���� � ����� be its normalized value with respect to

its maximum value � ����� � � ��� � � � ��� �

. For proprietaryreasons, we present the normalized instead of the absolute

TABLE V

STATISTICAL CHARACTERIZATION OF EACH FAILURE CLASS

Time- Number of failures Time-to-repair Number of links-between-Failures per component failing together

Low-Failure Links Weibull Power-Law Empirical N/A(network-wide)

High-Failure Links Empirical Power-Law Empirical N/A(per-link)

Shared Router-Related Weibull Power-Law Empirical Empirical(network-wide)

Shared Optical-Related Weibull N/A Empirical Empirical(network-wide)

100

101

102

100

101

102

Routers (r), in decreasing number of router events

Nor

mal

ized

num

ber

of e

vent

s pe

r ro

uter

, nn(

r) EmpiricalFitted

Fig. 8. Normalized number of events per router, in decreasing order.

value. Fig. 8 shows the normalized number of events perrouter, for all routers ranked in decreasing number of events.Interestingly, the straight line in the log-log plot indicates that��� �

��

follows roughly a power-law. Both � ���

and ��� ���

follow a power-law with the same slope. An estimate of theparameters of the power-law using least-square method yields� �

�� � �

� � � �, which we plot as a dashed line in Fig. 8. One

can use the same figure to calculate the mean time betweenevents for different routers: it varies from 5 days up to severalmonths.

When a router event happens, multiple links of the samerouter fail together. The distribution of the number of links inan event is shown in Fig. 9. Events involve 2 to 20 links. Thisis related to the number of ports per linecard, which variestypically between 2 and 24. Most events involve two links;12% of these events are due to failures on the two links ofaccess routers.

The empirical CDF of time-to-repair for router-related fail-ures is shown in Fig. 6, together with those of the other classes.The CDF for the router and the maintenance-related classes areclose to each other, and shifted toward larger values comparedto other classes. This could be due to human intervention forrepair or due to the rebooting process that takes on the order ofseveral minutes for backbone routers. Repair times for failuresbelonging to the same event are roughly equal.

Another characteristic of interest is the frequency of such

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

Number of links in a router event

% o

f rou

ter

even

ts

Fig. 9. Empirical PDF for the number of links in a router event.

events. Because not all routers experience enough eventsfor a statistically significant derivation of per router inter-arrival times, we consider the time between any two routerevents, anywhere in the network. Fig. 10 shows the empiricalcumulative distribution of network-wide time between tworouter events. We observe that the empirical CDF is wellapproximated by the CDF of a Weibull distribution:

� � � � ��� �� ��� �� � � ��� �� ��� � � �. We estimate the Weibull param-

eters using maximum-likelihood estimation as� � ��� ����

and � � � �����. The fitted CDF is shown in dashed line

in Fig. 10. In addition, we notice that the autocorrelationfunction decreases fast beyond small values of the lag. Thismeans that, for practical purposes, one could use i.i.d Weibullrandom variables to simulate the time between router events.The appropriateness of the Weibull distribution for the timebetween failures, is discussed in Section VII.

C. Shared Optical-Related Failures

Shared optical failures have an important impact on the net-work operation, as they affect multiple links and are thereforemore difficult to recover from than individual link failures.Shared optical-related failures are responsible for 11.4% ofall unplanned failures.

Fig. 11 shows the histogram of the number of IP linksin the same optical event. There are at least two (as theirdefinition requires an overlap in time) and at most 10 links in

10−2

10−1

100

101

102

103

104

105

106

107

108

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Empirical CDFWeibull (shape=0.068, scale=0.299)

Time between router events (sec)

Em

piric

al C

DF

Fig. 10. Cumulative distribution function (CDF) for the network-wide timebetween router events.

2 3 4 5 6 7 8 9 100

5

10

15

20

25

30

35

40

45

50

number of links in an optical event

% o

f eve

nts

Fig. 11. Number of links in an optical event.

the same event. This is in agreement with sharing informationderived from the IP-to-Optical mapping. For example, the mostfrequent number of links sharing a segment according to themapping is 2 (which is also the case in optical events); themaximum number of links that share a segment according tothe mapping is 25 (larger than the maximum number of linksin any optical event).

The CDF of time-to-repair for optical-related failures isshown in Fig. 6. Short time-to-repair values are more likelydue to faults in the optical switches, while longer timescorrespond to fiber cuts or other failures that require humanintervention to be repaired. Similar to the previous classesof shared failures, the CDF is shifted towards larger values,compared to individual failures. By their definition, failures inthe same optical event happen within a few seconds from eachother.

Another characteristic of interest is the frequency of opticalfailures in the network. Fig. 12 shows the CDF for the time be-tween two successive optical events, anywhere in the network.The values range from 5.5 sec up to 7.5 days, with a mean of12 hours. We use maximum likelihood estimation to estimate

100

101

102

103

104

105

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time between optical events (sec)

Em

piric

al C

DF

Empirical CDFWeibull (shape=0.013, scale=0.445)

Fig. 12. Cumulative distribution function (CDF) for the network-wide timebetween optical events.

10−2

10−1

100

101

102

103

104

105

106

107

108

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time between failures (in sec) on a high failure link

Em

piric

al C

DF

Fig. 13. Time between failures on each high failure link.

the parameters of a Weibull distribution from the empiricaldata and we obtain

� � � � � � �and � � � ��� �

. The resultingCDF, shown in dashed line in Fig. 12, is an approximation ofthe empirical CDF. However, one can observe that there aremore distinct modes in this distribution (e.g. one from 0 upto 100 sec, a second from 100 sec up to 30 hours and a thirdone above that), hinting to more factors that could be furtheridentified. A closer look in the sequence of events revealsthat times between events below 100 sec correspond to manyclosely spaced events on the same set of links that could bedue to a persistent problem in the optical layer. However, theWeibull fit of the aggregate CDF sufficiently characterizes thefrequency of optical events network-wide.

D. High Failure Links

High failure links include only 2.5% of all links. However,they are responsible for more than half of the individualfailures and for 38.5% of all unplanned failures, which is thelargest contribution among all classes, see Table IV.

As we discussed earlier in Fig. 5, the number of failures� ��� �

per high failure link�

follows a power-law: � ��� � � ��� � � ���.

Each high failure link experiences enough failures to allow fora characterization by itself.

The empirical CDF of the time between failures on eachof the high failure links is shown in Fig. 13. Some of themexperience failures well spread across the entire period. Theycorrespond to the long horizontal lines in Fig. 2 and the smoothCDFs in Fig. 13. Some other high failure links are more bursty:a large number of failures happens over a short time period.They correspond to the short horizontal lines in Fig. 2 andto the CDFs with a knee in Fig. 13. The mean time betweenfailures varies from 1 to 40 hours for various links, i.e. ashorter range than for the other classes. Finally, the CDF ofthe time-to-repair for failures on high failure links is shownin Fig. 6. It is clearly distinct from all other classes- failureslast significantly shorter (up to 30% difference from the CDFof all unplanned failures and up to 70% from the CDF of theshared failures). The larger number of shorter failures is inaccordance with our conjecture that high failure links are inan advanced stage of their lifetime and their components areprobably subject to intermittent and recurring faults.

E. Low Failure Links

In Fig. 5, we have already defined low failure links arethose with less individual failures than the threshold

�����.

The number of failures � �����per link (l) follows roughly a

power-law: � ����� � � ��� � ��.

A statistically significant characterization is not possible forevery low failure link, as many of them do not experienceenough failures. We group all low failure links together andstudy the time between any two failures, i.e., the two failuresmay happen anywhere in the network and not necessarily onthe same link. Fig. 14(a) shows the empirical CDF for thenetwork-wide time between failures. It turns out that in thiscase too, the empirical CDF is well approximated by a Weibulldistribution with maximum-likelihood estimated parameters� � ��� ����

and � ��� ��� �; the fitted distribution is shown

in dashed line in the same figure. In Fig. 14(b), we also showthe autocorrelation function for the time between failures atthe 90% confidence interval. We notice that correlation in thetime between failures drops fast after a small lag. This meansthat, as a first approximation, we can use i.i.d. Weibull randomvariables with the fitted parameters to regenerate the network-wide time between individual failures on low failure links.

Finally, the empirical CDF for the time-to-repair in thisclass is shown in Fig. 6, together with the rest of the classes. Itis interesting to note that the CDF is very close to the CDF forall unplanned failures. This fact together with the observationthat low failure links correspond to the roughly random partof Fig. 2 indicate that, unlike the previous classes, failures inthis class have an “average” behavior and are the norm ratherthan the exception of the entire data set.

VI. FAILURE ANALYSIS OVER SHORTER TIME PERIODS

The classification and characterization of failures has beenconducted so far, considerering the entire measurements pe-riod. In addition, it is also important to understand how

10−2

10−1

100

101

102

103

104

105

106

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Network−wide time between failures (sec), for low failure links

Em

piric

al C

DF

Empirical DataWeibull (α= 0.046, β = 0.414)

(a) CDF of (network-wide) time between failures

0 50 100 150 200−0.2

0

0.2

0.4

0.6

0.8

1

Lag

Cor

rela

tion

coef

ficie

nt

(b) Autocorrelation of (network-wide) time between fail-ures

Fig. 14. Network-wide time between failures on low failure links

properties remain constant or change with time. For thispurpose, we now consider shorter, in particular one-month5,time periods and we investigate to what extent the failurecharacteristics identified, based on the entire measurementperiod, still hold or change with time.

For every one-month period, we now apply the sameclassification and characterization as we did for the entiretime period. As far as the classification is concerned, weshow how the total number of failures and their breakdownto the four failure classes (router-related, optical-related, high-

5As a first cut, we chose periods one-month long, because they aresignificantly shorter than the entire 7-months period to reveal time variabilitybut they are still long enough for a sufficient number of failures to occurand allow for a statistical characterization. In general, the entire measurementperiod might need to be partitioned into time intervals of variable length, inorder to better capture bursty behavior. This is a more general and difficultproblem and is not addressed here.

Apr May Jun Jul Aug Sep Oct0

5

10

15

20

25

Month

% o

f tot

al n

umbe

r of

failu

res

(shared) router−related(shared) optical−related(individual) high−failure links(individual) low−failure linksall failure classes

Fig. 15. Number of failures per month, per class and aggregate. (Forpropriatary reasons, instead of absolute numbers, we report the percentageof total number of failures over the entire period.)

failure links, low-failure links) varies with time; we find thatmost of the variability is due to the high-failure links. Asfar as the statistical characterization is concerned, we findthat the same type of distributions (weibull for the time-between-failures, power-laws for the heterogeneity of failuresper network element, as well as the empirical distributions forthe failure durations) still apply for each month. Furthermore,in the case of the low-failure links class, these distributionsare estimated to have roughly the same parameters.

A. Number of failures and classification

First, we break the entire period into seven months and lookat the number of failures and their causes for each monthseperately, see Fig. 15. (Note that, for proprietary reasons,we do not report the absolute number of failures; insteadwe report a normalized number, by converting the absolutenumber to the percentage of the total number of failures overthe entire measurement period). The top bold line shows the(normalized) number of failures in each month, including allcauses. The four bottom thin lines show the (normalized)number of failures per month for each class of failures.

We can make the following observations from this figure.� The low-failure links are consistently responsible for the

majority of failures. Furthermore, the number of failuresin this class does not vary much across months.

� The number of failures in the high-failure links variessignificantly from month to month. For example, in Julyand August there are no such failures, while in Juneand September they are the majority of failures. This isexpected, as this class of failures corresponds to excep-tionally rare events. A further explanation may be thatonce these failure prone links/interfaces are spotted bythe network operators, they get fixed which may explainthe dip after the initial surge period.

� The shared link failures (router-related and optical-related) have consistently a smaller and less variablecontribution.

TABLE VI

TIME-BETWEEN-FAILURES ON LOW FAILURE LINKS AND OVER

ONE-MONTH PERIODS. WEIBULL PARAMETERS ESTIMATED WITH 95%

CONFIDENCE INTERVAL.

Month Apr May Jun Jul Aug Sep Oct� 0.044 0.056 0.026 0.056 0.063 0.046 0.043�

0.450 0.421 0.490 0.403 0.388 0.423 0.438

� The top line is the sum of the four bottom lines andcorresponds to all failures per month, including all causes.Its variability is mainly due to the month-to-month vari-ability of high-failure links.

B. Statistical characterization

We now look at the statistical characteristics of three out ofthe four classes: low-failure links, high-failure links, router-related. We exclude the high-failure links class, because boththe small number of high failures per month and the variabilityfrom month-to-month do not allow for a sufficient statisticalcharacterization. For the remaining classes, we find that thedistributions describing their characteristics (time-between-failures, heterogeneity of network elements, time-to-repair)also hold for each month, with some small variation in thefitted parameters.

Let us first consider the low-failure links class. Fig. 16shows the empirical distributions for each month separately(a thin line for each month) and for the entire measurementperiod (shown in a bold line).

� Fig. 16(a) shows the empirical distributions for the time-between-failures. We can observe visually that the curvesfor each month have the same shape and are close to thecurve for the entire period. A Weibull distribution can befitted to each one of them, using maximum likelihood es-timation; the estimated parameters with 95% confidenceinterval are shown in Table VI. Roughly speaking, theseparameters are close to each other and to those estimatedfor the entire measurement period.

� Fig. 16(b) shows the number of failures in each link (withat least one failure in a month) in decreasing number offailures, for each month seperately. We see the curve foreach month can be roughly approximated by a straightline with roughly the same slope as the slope for theentire period.

� Finally, Fig. 16(c) shows the empirical CDF for the time-to-repair a failure for each month. All CDFs (with theexception of the month of August which seems to be anoutlier) have the same shape and are very close to theempirical CDF for the entire period.

Applying the same steps to the router-related and optical-related classes, we observe that the same type of distributionsalso hold in the per-month analysis. First, Weibull distribu-tions describe well both the time-between-router-related andthe time-between-optical-related events, network-wide and foreach month. Second, power-law distributions are appropriate

100

101

102

103

104

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time Between Failures (sec)

Em

piric

al C

DF Entire period

per month

(a) Time-between-failures

100

101

102

100

101

102

103

number of failures per low failure link

number of link

num

ber

of fa

ilure

s

entire period

per month

(b) Number of failures per low-failure link

100

101

102

103

104

105

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Failure Duration (sec)

Em

piric

al C

DF

Failure Durations for Low Failure links

Entire period

per month

Aug

(c) Failure durations

Fig. 16. Characterizing Low-Failure Links, Per Month.

for modeling the router events per router, for each month.Finally, the empirical distributions of time-to-repair for eachmonth have the usual consistent shape. For lack of space,we omit these plots here. However, it worths noting that,contrary to the low-failure class, the estimated parametersvary significantly from month-to-month. This can be partiallyexplained by the relatively small number of failures per monthin the router- and optical-related failure classes.

VII. DISCUSSION

This work offers a detailed characterization of link failuresand is useful in two ways. First, it reveals the nature and extentof failures in today’s IP backbones. Our methodology canbe used to identify failing network components and pinpointareas for improvement. Second, it is the first step towardbuilding a failure model, as we provide information about thecause of failures as well as their statistical properties. Sucha model would be useful as an input to various engineeringproblems that need to account for failures. In this section,we discuss how our classification and analysis can be usedtoward building a failure model, as well as open issues andfuture directions.

IP link failures occur due to several causally unrelatedevents at or below the IP layer. Accordingly, we have dividedfailures into a number of classes such that their underlyingcauses are unrelated. Therefore a backbone failure model canbe obtained by developing a model to characterize each classindependently, and then combining them. For each class, wehave identified a few key properties (such as the time betweenfailures, the time-to-repair and the distribution of failuresacross links and routers), provided their statistics and, whenpossible, fitted them using well-known distributions with asmall number of parameters.

Let us first discuss the validity of our classification and thenthe modeling of each class separately.

Our classification is based on hints from the ISIS data set,discussed in detail in Section IV. In the same section, we usedthe IP-to-Optical database and confirmed to a very satisfactorydegree the validity of our optical-related class of failures. Thefact that all simultaneous failures involved a common routerwas also a confirmation for the router-related class. When weapplied our classification methodology to the measurements,the statistics of the identified classes turned out to be quitedifferent from each other, which provides further assuranceabout our classification. For example, the CDF of time-to-repair in Fig. 6 are well separated from each other: the sharedfailures “pull” the CDF toward larger values, the high failurelinks “pull” it toward smaller values, while the low failurelinks are in the middle. A similar separation happens in theinitial Fig. 2: the maintenance and shared failures capture thevertical lines, the high failure capture the horizontal lines,the low failure links capture the remaining “random” plot.However, inferring the failures causes based solely on IS-ISlogs is a difficult reverse engineering problem and results toa coarse classification. In future work, we plan to correlatethe IS-IS data set with additional logs, such as SONET alarm

logs, router logs, maintenance activity schedules. Even withsuch cross examinations, it will still be difficult to uniquelyidentify the causes for all failures.

The characterization in Section V provides the basis formodeling each class separately. There are two interestingobservations from parameterizing the properties of variousclasses. First, we observe that the empirical CDF for thenetwork-wide time between failures (or events) for threeclasses of failures was well approximated by a Weibull distri-bution. These three classes are the router-related (Fig. 10), theoptical-related (Fig. 12) and the low failure links (Fig. 14(a)).The Weibull distribution has been found widely applicable inreliability engineering to describe the lifetime of components,primarily due to its versatile shape [27]. In addition, theWeibull distribution is derived as an extreme value distribution:for a large number of identical and independent components,the time to the first failure follows a Weibull, e.g. see [27].This provides a theoretical justification for the good fit in ourcase: there is a large number of components in each class andthe network-wide time between failures can be interpreted asthe time to the first failure, assuming a renewal process.

Our second finding is that power-laws describe well thedistribution of failures (or events) across components in thesame class. Indeed, power-laws fitted well the number ofrouter events per router (see Fig. 8) as well as for thenumber of individual failures per high or low failure link(see Fig. 5). Power-laws are often found to describe phe-nomena in which, small occurrences are extremely common,whereas large instances are extremely rare. Examples includeman-made or naturally occurring phenomena, such as wordfrequencies, income distribution, city sizes, and earthquakemagnitudes [28]. Recently, the Internet has been found todisplay quite a number of power-law distributions, such as thenode outdegrees and other properties of Internet topologies[29], the sizes and other attributes of objects in a web page[30] (Pareto distributions are another expression for power-laws).

Section V can be used as the basis of a failure model. Forexample, consider the low failure links in Section V-E and letus try to re-generate failures that have similar statistics withthe measured ones. In order to decide when the next failurehappens, one can pick a random number from the Weibulldistribution for the network-wide time between failures. Inorder to decide on which link this failure happens, one couldpick a link using the power-law distribution. We can use thepower-law to assign a failure to a link as follows. If we observea link

�over a long period of time (

� � � ��� � ����% ) andfind that it suffers � � failures, the failure probability � � isproportional to � � � � . If the links are independent and � � ,��� , ... ��� are given (here follow the power-law), so do � � ,��� , ...�� . Given that a failure happens, it happens on link

�with probability � � � � � �� ����� �� � � � � � � � �� ��� � ��� � whichcan now be easily calculated. Similar steps can be followedto reproduce the router events using the network-wide timebetween events and the distribution of events across routers.

We would like now to explicitly state the limitations of the

model so far:� Using the Weibull and Pareto distributions incorporates

the assumption that the arrival of successive failures isindependent of the distribution across links. Clustering offailures per link would break this assumption. Therefore,the dependence between the two dimensions, i.e. “timeof occurrence” and “link/router of occurrence”, needs tobe studied further. Furthermore, the independence amongcomponents in the same class is important. We foundnegligible correlation among low failure links; howeverthis is only a necessary and not a sufficient condition forthe independence among components.

� It also needs to be explored whether there is a dependencebetween the time-to-repair and the failure arrival process,in which case a marked Markov process (with the markof each point being the time-to-repair) could be a moreappropriate modeling approach.

� The characterization presented in this paper is specificto Sprint’s IP network and one should be careful beforeblindly applying it to any kind of network. In general,the failure behavior of a network is very related to itstopology, design, maintenance, technology, age and otherspecific traits. These traits may vary between networksand also for the same network in time, which is makethem inherently difficult to model in great precision. Inorder to develop a model that is applicable to an IPnetwork regardless of its topology, size and underlyinginfrastructure, we need a further characterization of eachcomponent and and of the spatial correlation of failures.

Therefore, this should only be considered as a first steptowards building a failure model.

Apart from understanding the failure characteristics andimproving the model itself, an important direction for futurework is to also understand their effect on network stability andavailability. For example, correlating failures with link weightscould reveal which failures had an important impact on traffic.Ongoing work in [21] focuses on better understanding the fail-ures impact on network operation and user perceived quality.The next step is to use our understanding of failures as input tovarious network design problems and make appropriate designchoices to minimize the impact of failures.

VIII. CONCLUSIONS

The availability of spare capacity and sound engineeringpractices in today’s IP backbone networks makes it easy toachieve traditional QoS objectives, such as low loss, delay andjitter. On the other hand, a variety of failures or at below theIP layer, can potentially result in loss of IP connectivity, andthus can reduce available capacity, cause netowrk instabilityand potentially disrupt packet forwarding.

In this paper, we analyze seven months of ISIS routingupdates from the Sprint’s IP backbone to characterize failuresthat affect IP connectivity. We classify failures according totheir cause and describe the key characteristics of each class.Our findings indicate that failures are part of the everydayoperation: 20% of them are due to scheduled maintenance

operation, while 16% and 11% of the unplanned failures areshared among multiple links and can be attributed to router-related and optical-related problems respectively. Our studynot only provides a better understanding of the nature andthe extent of link failures, but is also the first step towardsdeveloping a failure model which is necessary input to manynetwork design and traffic engineering problems.

Directions for future work include (i) the aspects discussedin Section VII regarding modeling and root-cause analysis, (ii)a better understanding of the impact of failures on networkperformance, and (iii) an automation of this analysis to makeit process failure logs online, raise alarms and prevent criticalfailures.

ACKNOWLEDGMENTS

We would like to thank Richard Mortier for contributinghis PyRT listener software; Richard Gass and Ed Kress fortheir help in the routing data collection; and the colleaguesat SprintLink Operations for allowing us to collect data onthe Sprint’s backbone as well as for providing their invaluablefeedback. Athina Markopoulou is grateful to Professor PeterGlynn from Stanford University for insightful discussions onthe data analysis and on failure problems.

REFERENCES

[1] C. Fraleigh, S. Moon, B. Lyles, C. Cotton, M. Khan, R. Rockell, D. Moll,T. Seely, and C. Diot, “Packet-level traffic measurements from the SprintIP backbone,” IEEE Network Magazine, vol. 17, no. 6, Nov. 2003.

[2] K. Papagiannaki, S. Moon, C. Fraleigh, P. Thiran, and C. Diot, “Mea-surement and analysis of single-hop delay on an IP backbone network,”IEEE Journal on Selected Areas in Communications, vol. 21, no. 6, Aug.2003.

[3] G. Iannaccone, C. Chuah, R. Mortier, S. Bhattacharyya, and C. Diot,“Analysis of link failures in an IP backbone,” in Proc. of ACM SigcommInternet Measurement Workshop, Nov. 2002.

[4] G. Iannaccone, C. Chuah, S. Bhattacharyya, and C. Diot, “Feasibility ofIP restoration in a tier-1 backbone,” in IEEE Network, Special Issue onProtection, Restoration and Disaster Recovery, Mar. 2004.

[5] D. Oran, “OSI IS-IS intra-domain routing protocol,” RFC 1142, Feb.1990.

[6] S. Iyer, S. Bhattacharyya, N. Taft, and C. Diot, “An approach to alleviatelink overload as observed on an IP backbone,” in Proc. of IEEE Infocom,Mar. 2003.

[7] A. Markopoulou, G.Iannaccone, S.Bhatacharyya, C.N.Chuah, andC. Diot, “Characterization of Failures in an IP Backbone,” in Proc. ofIEEE Infocom, 2004.

[8] C. Fraleigh, F. Tobagi, and C. Diot, “Provisioning IP backbone networksto support latency sensitive traffic,” in Proc. of IEEE Infocom, Mar. 2003.

[9] C. Boutremans, G. Iannaccone, and C. Diot, “Impact of link failures onVoIP performance,” in Proc. of NOSSDAV, May 2002.

[10] A. Fumagalli and L. Valcarenghi, “IP restoration versus WDM protec-tion: Is there an optimal choice?” IEEE Network Magazine, vol. 14,no. 6, pp. 34–41, Nov. 2000.

[11] L. Sahasrabuddhe, S. Ramamurthy, and B. Mukherjee, “Fault manage-ment in IP-over-WDM networks: WDM protection versus IP restora-tion,” IEEE Journal on Selected Areas in Communications, vol. 20, no. 1,Jan. 2002.

[12] S. C. A. Alaettinoglou, “Detailed analysis of isis routing pro-tocol on the qwest backbone,” Nanog presentation, Feb. 2002,http://www.nanog.org/mtg-0202/ppt/cengiz.pdf.

[13] A. Nucci, B. Schroeder, S. Bhattacharyya, N. Taft, and C. Diot, “IGPlink weight assignment for transient link failures,” in Proc. of IEEE ITCI8, Berlin, Germany, Sept. 2003.

[14] B. Fortz and M. Thorup, “Optimizing OSPF/IS-IS weights in a changingworld,” IEEE Journal on Selected Areas in Communications, Feb. 2002.

[15] M. Duvry, C. Diot, N. Taft, and P. Thiran, “Network availability basedservice differentiation,” in Proc. of IWQoS, June 2003.

[16] S. Nelakuditi, S. Lee, Y. Yu, and Z.-L. Zhang, “Failure insenstive routingfor ensuring service availability,” in Proc. of IWQoS, 2003.

[17] V. Paxson, “End-to-end routing behavior in the internet,” IEEE/ACMTransactions on Networking, vol. 5, no. 5, pp. 601–615, Oct. 1997.

[18] Y. Zhang, V. Paxson, and S. Shenker, “The stationarity of in-ternet path properties: Routing, loss and throughput,” available athttp://www.icir.org/, ACIRI, Tech. Rep., May 2000.

[19] M. Dahlin, B. Chandra, L. Gao, and A. Nayate, “End-to-end WANservice availability,” IEEE/ACM Transactions on Networking, vol. 11,no. 2, Apr. 2003.

[20] C. Labovitz, A. Ahuja, and F. Jahanian, “Experimental study of internetstability and wide-area network failures,” in Proc. of FTCS, June 1999.

[21] Y. Ganjali, S. Bhattacharyya, and C. Diot, “Limiting the impact offailures on network performance,” Research Report RR04-ATL-020666,available at http://www.ipmon.com/, Sprint ATL, Tech. Rep., 2003.

[22] J. Gary, “Why do computers stop and what can be done about it?” inSymposium on Reliability in Distributed Software and Database Systems,1986.

[23] R. Barlow and F. Proschan, Statistical Analysis of Reliability and LifeTesting Models. New York: Holt, Rinehart and Winston, 1975.

[24] D. Oppenheimer, A. Ganapathi, and D. Patterson, “Why do internetservices fail, and what can be done about it?” in 4th USENIX Symposiumon Internet Technologies and Systems (USITS’03), 2003.

[25] E. Brewer, “Lessons from giant-scale services,” in IEEE Internet Com-puting, vol. 5, no. 4, 2001.

[26] A. Fox and D. Patterson, “When does fast recovery trump high relia-bility?” in Proc. 2nd Workshop on Evaluating and Architecting SystemDependability, San Jose, CA, 2002.

[27] P. Tobias and D. Trindade, Applied Reliability, 2nd ed. ChapmanHall/CRC, 1995.

[28] L. Adamic, “Zipf, power-laws and pareto: a rankingtutorial,” Xerox Palo Alto Research Center, Palo Alto, CA,http://ginger.hpl.hp.com/shl/papers/ranking/ranking.html.

[29] M. Faloutsos, P. Faloutsos, and C. Faloutsos, “On power-law relation-ships of the internet topology,” in Proc. of ACM Sigcomm, Sept. 1999.

[30] A.Feldmann, A.Gilbert, P.Huang, and W.Willinger, “Dynamics of IPtraffic; a study of the role of variability and the impact of control,”in Proc. of ACM Sigcomm, 1998.