geometric approach to active packet loss measurement

8/8/2019 Geometric Approach to Active Packet Loss Measurement

1/14

1

A Geometric Approach to Improving Active PacketLoss Measurement

Joel Sommers, Paul Barford, Nick Dufeld, and Amos Ron

Abstract Measurement and estimation of packet loss char-acteristics are challenging due to the relatively rare occurrenceand typically short duration of packet loss episodes. While activeprobe tools are commonly used to measure packet loss on end-to-end paths, there has been little analysis of the accuracy of thesetools or their impact on the network. The objective of our studyis to understand how to measure packet loss episodes accuratelywith end-to-end probes. We begin by testing the capability of standard Poisson-modulated end-to-end measurements of lossin a controlled laboratory environment using IP routers andcommodity end hosts. Our tests show that loss characteristicsreported from such Poisson-modulated probe tools can be quiteinaccurate over a range of trafc conditions. Motivated by theseobservations, we introduce a new algorithm for packet lossmeasurement that is designed to overcome the deciencies instandard Poisson-based tools. Specically, our method entailsprobe experiments that follow a geometric distribution to ( 1)enable an explicit trade-off between accuracy and impact onthe network, and ( 2) enable more accurate measurements thanstandard Poisson probing at the same rate. We evaluate thecapabilities of our methodology experimentally by developingand implementing a prototype tool, called B ADABING . Theexperiments demonstrate the trade-offs between impact on thenetwork and measurement accuracy. We show that B ADABINGreports loss characteristics far more accurately than traditionalloss measurement tools.

Index Terms Active Measurement, B ADABING , NetworkCongestion, Network Probes, Packet Loss.

I. INTRODUCTION

Measuring and analyzing network trafc dynamics betweenend hosts has provided the foundation for the development of many different network protocols and systems. Of particularimportance is understanding packet loss behavior since losscan have a signicant impact on the performance of bothTCP- and UDP-based applications. Despite efforts of network engineers and operators to limit loss, it will probably never beeliminated due to the intrinsic dynamics and scaling propertiesof trafc in packet switched network [1]. Network opera-tors have the ability to passively monitor nodes within their

network for packet loss on routers using SNMP. End-to-endactive measurements using probes provide an equally valuableperspective since they indicate the conditions that applicationtrafc is experiencing on those paths.

The most commonly used tools for probing end-to-end pathsto measure packet loss resemble the ubiquitous PING utility.P ING -like tools send probe packets ( e.g., ICMP echo packets)to a target host at xed intervals. Loss is inferred by the senderif the response packets expected from the target host are notreceived within a specied time period. Generally speaking,an active measurement approach is problematic because of the discrete sampling nature of the probe process. Thus, the

accuracy of the resulting measurements depends both on thecharacteristics and interpretation of the sampling process aswell as the characteristics of the underlying loss process.

Despite their widespread use, there is almost no mentionin the literature of how to tune and calibrate [2] activemeasurements of packet loss to improve accuracy or how tobest interpret the resulting measurements. One approach issuggested by the well-known PASTA principle [3] which, ina networking context, tells us that Poisson-modulated probeswill provide unbiased time average measurements of a routerqueues state. This idea has been suggested as a foundation for

active measurement of end-to-end delay and loss [4]. However,the asymptotic nature of PASTA means that when it is appliedin practice, the higher moments of measurements must beconsidered to determine the validity of the reported results.A closely related issue is the fact that loss is typically arare event in the Internet [5]. This reality implies either thatmeasurements must be taken over a long time period, or thataverage rates of Poisson-modulated probes may have to bequite high in order to report accurate estimates in a timelyfashion. However, increasing the mean probe rate may lead tothe situation that the probes themselves skew the results. Thus,there are trade-offs in packet loss measurements between proberate, measurement accuracy, impact on the path and timeliness

of results.The goal of our study is to understand how to accurately

measure loss characteristics on end-to-end paths with probes.We are interested in two specic characteristics of packet loss:loss episode frequency , and loss episode duration [5]. Ourstudy consists of three parts: ( i) empirical evaluation of thecurrently prevailing approach, ( ii) development of estimationtechniques that are based on novel experimental design, novelprobing techniques, and simple validation tests, and ( iii) em-pirical evaluation of this new methodology.

We begin by testing standard Poisson-modulated probingin a controlled and carefully instrumented laboratory envi-ronment consisting of commodity workstations separated by

a series of IP routers. Background trafc is sent betweenend hosts at different levels of intensity to generate lossepisodes thereby enabling repeatable tests over a range of conditions. We consider this setting to be ideal for testingloss measurement tools since it combines the advantages of traditional simulation environments with those of tests in thewide area. Namely, much like simulation, it provides for ahigh level of control and an ability to compare results withground truth. Furthermore, much like tests in the wide area,it provides an ability to consider loss processes in actual routerbuffers and queues, and the behavior of implementations of thetools on commodity end hosts. Our tests reveal two important


2/14

2

deciencies with simple Poisson probing. First, individualprobes often incorrectly report the absence of a loss episode(i.e., they are successfully transferred when a loss episode isunderway). Second, they are not well suited to measure lossepisode duration over limited measurement periods.

Our observations about the weaknesses in standard Poissonprobing motivate the second part of our study: the developmentof a new approach for end-to-end loss measurement that in-cludes four key elements. First, we design a probe process thatis geometrically distributed and that assesses the likelihoodof loss experienced by other ows that use the same path,rather than merely reporting its own packet losses. The probeprocess assumes FIFO queues along the path with a drop-tail policy. Second, we design a new experimental framework with estimation techniques that directly estimate the meanduration of the loss episodes without estimating the durationof any individual loss episode. Our estimators are proved tobe consistent, under mild assumptions of the probing process.Third, we provide simple validation tests (that require noadditional experimentation or data collection) for some of the

statistical assumptions that underly our analysis. Finally, wediscuss the variance characteristics of our estimators and showthat while frequency estimate variance depends only on thetotal the number of probes emitted, loss duration variancedepends on the frequency estimate as well as the number of probes sent.

The third part of our study involves the empirical eval-uation of our new loss measurement methodology. To thisend, we developed a one-way active measurement tool calledBADABING . BADABING sends xed-size probes at speciedintervals from one measurement host to a collaborating targethost. The target system collects the probe packets and reportsthe loss characteristics after a specied period of time. We also

compare B ADABING with a standard tool for loss measurementthat emits probe packets at Poisson intervals. The resultsshow that our tool reports loss episode estimates much moreaccurately for the same number of probes. We also show thatBADABING estimates converge to the underlying loss episodefrequency and duration characteristics.

The most important implication of these results is thatthere is now a methodology and tool available for wide-areastudies of packet loss characteristics that enables researchersto understand and specify the trade-offs between accuracyand impact. Furthermore, the tool is self-calibrating [2] inthe sense that it can report when estimates are poor. Practicalapplications could include its use for path selection in peer-to-peer overlay networks and as a tool for network operatorsto monitor specic segments of their infrastructures.

I I . R ELATED W ORK

There have been many studies of packet loss behavior in theInternet. Bolot [6] and Paxson [7] evaluated end-to-end probemeasurements and reported characteristics of packet loss overa selection of paths in the wide area. Yajnik et al. evaluatedpacket loss correlations on longer time scales and developedMarkov models for temporal dependence structures [8]. Zhanget al. characterized several aspects of packet loss behavior [5].

In particular, that work reported measures of constancy of lossepisode rate, loss episode duration, loss free period durationand overall loss rates. Papagiannaki et al. [9] used a so-phisticated passive monitoring infrastructure inside Sprints IPbackbone to gather packet traces and analyze characteristics of delay and congestion. Finally, Sommers and Barford pointedout some of the limitations in standard end-to-end Poissonprobing tools by comparing the loss rates measured by suchtools to loss rates measured by passive means in a fullyinstrumented wide area infrastructure [10].

The foundation for the notion that Poisson Arrivals SeeTime Averages (PASTA) was developed by Brumelle [11], andlater formalized by Wolff [3]. Adaptation of those queuingtheory ideas into a network probe context to measure lossand delay characteristic began with Bolots study [6] and wasextended by Paxson [7]. In recent work, Baccelli et al. analyzethe usefulness of PASTA in the networking context [12]. Of particular relevance to our work is Paxsons recommendationand use of Poisson-modulated active probe streams to reducebias in delay and loss measurements. Several studies include

the use of loss measurements to estimate network propertiessuch as bottleneck buffer size and cross trafc intensity [13],[14] . The Internet Performance Measurement and Analysisefforts [15], [16] resulted in a series of RFCs that specifyhow packet loss measurements should be conducted. However,those RFCs are devoid of details on how to tune probeprocesses and how to interpret the resulting measurements.We are also guided by Paxsons recent work [2] in which headvocates rigorous calibration of network measurement tools.

ZING is a tool for measuring end-to-end packet loss inone direction between two participating end hosts [17], [18].ZING sends UDP packets at Poisson-modulated intervals withxed mean rate. Savage developed the S TING [19] tool to

measure loss rates in both forward and reverse directions froma single host. S TING uses a clever scheme for manipulatinga TCP stream to measure loss. Allman et al. demonstratedhow to estimate TCP loss rates from passive packet tracesof TCP transfers taken close to the sender [20]. A relatedstudy examined passive packet traces taken in the middle of the network [21]. Network tomography based on using bothmulticast and unicast probes has also been demonstrated to beeffective for inferring loss rates on internal links on end-to-endpaths [22], [23].

III . D EFINITIONS OF L OSS

CHARACTERISTICS

There are many factors that can contribute to packet lossin the Internet. We describe some of these issues in detailas a foundation for understanding our active measurementobjectives. The environment that we consider is modeled asa set of N ows that pass through a router R and competefor a single output link with bandwidth Bout as depicted inFigure 1a. The aggregate input bandwidth ( Bin ) must be greaterthan the shared output link ( Bout ) in order for loss to take place.The mean round trip time for the N ows is M seconds. Router R is congured with Q bytes of packet buffers to accommodatetrafc bursts, with Q typically sized on the order of M B [24],


3/14

3

inB

R

Q outB

N

(a) Simple system model. N ows on input links with aggregate bandwidth Bin compete for a single output link on router R with bandwidth Bout where Bin > Bout . The output link has Q seconds of buffer capacity.

Q

time

capacitybuffer

lengthqueue

c dba

(b) Example of the evolution of the length of a queue over time. The queuelength grows when aggregate demand exceeds the capacity of the outputlink. Loss episodes begin (points a and c) when the maximum buffer sizeQ is exceeded. Loss episodes end (points b and d ) when aggregate demandfalls below the capacity of the output link and the queue drains to zero.

1: Simple system model and example of loss characteristics

under consideration.

[25]. We assume that the queue operates in a FIFO manner,that the trafc includes a mixture of short- and long-lived TCPows as is common in todays Internet, and that the value of N will uctuate over time.

Figure 1b is an illustration of how the occupancy of thebuffer in router R might evolve. When the aggregate sendingrate of the N ows exceeds the capacity of the shared outputlink, the output buffer begins to ll. This effect is seen as apositive slope in the queue length graph. The rate of increaseof the queue length depends both on the number N and on

sending rate of each source. A loss episode begins when theaggregate sending rate has exceeded Bout for a period of timesufcient to load Q bytes into the output buffer of router R(e.g., at times a and c in Figure 1b) . A loss episode ends whenthe aggregate sending rate drops below Bout and the bufferbegins a consistent drain down to zero ( e.g., at times b andd in Figure 1b). This typically happens when TCP sourcessense a packet loss and halve their sending rate, or simplywhen the number of competing ows N drops to a sufcientlevel. In the former case, the duration of a loss episode isrelated to M , depending whether loss is sensed by a timeoutor fast retransmit signal. We dene loss episode duration as thedifference between start and end times ( i.e., b

a and d

c).

While this denition and model for loss episodes is somewhatsimplistic and dependent on well behaved TCP ows, it isimportant for any measurement method to be robust to owsthat do not react to congestion in a TCP-friendly fashion.

This denition of loss episodes can be considered a router-centric view since it says nothing about when any one end-to-end ow (including a probe stream) actually loses a packet orsenses a lost packet. This contrasts with most of the prior work discussed in II which consider only losses of individual orgroups of probe packets. In other words, in our methodology,a loss episode begins when the probability of some packetloss becomes positive. During the episode, there might be

transient periods during which packet loss ceases to occur,followed by resumption of some packet loss. The episode endswhen the probability of packet loss stays at 0 for a sufcientperiod of time (longer than a typical RTT). Thus, we offer twodenitions for packet loss rate :

Router-centric loss rate. With L the number of droppedpackets on a given output link on router R during a givenperiod of time, and S the number of all successfullytransmitted packets through the same link over the sameperiod of time, we dene the router-centric loss rate as L/ (S + L).

End-to-end loss rate. We dene end-to-end loss rate inexactly the same manner as router-centric loss-rate, withthe caveat that we only count packets that belong to aspecic ow of interest.

traffic generator traffic generatorhostshosts

DAG monitor host

probe sender

Si Si

hop identifier A B C D E

propagation delayemulator

(50 millisecondseach direction)

GE

GE OC12

OC12

GE

GE

GE

GEGEOC3 OC3

12000Cisco

12000Cisco

6500Cisco

SX14Adtech

12000Cisco Cisco

6500

2: Laboratory testbed. Cross trafc scenarios consisted of constant bit-rate trafc, long-lived TCP ows, and web-likebursty trafc. Cross trafc owed across one of two routers athop B, while probe trafc owed through the other. Opticalsplitters connected Endace DAG 3.5 and 3.8 passive packetcapture cards to the testbed between hops B and C, and hopsC and D. Probe trafc owed from left to right and the lossepisodes occurred at hop C.

It is important to distinguish between these two notions of loss rate since packets are transmitted at the maximum rate Bout during loss episodes. The result is that during a periodwhere the router-centric loss rate is non-zero, there may beows that do not lose any packets and therefore have end-to-end loss rates of zero. This observation is central to our studyand bears directly on the design and implementation of activemeasurement methods for packet loss.

As a consequence, an important consideration of our probeprocess described below is that it must deal with instanceswhere individual probes do not accurately report loss. Wetherefore distinguish between the true loss episode state andthe probe-measured or observed state . The former refers to therouter-centric or end-to-end congestion state, given intimateknowledge of buffer occupancy, queueing delays, and packetdrops, e.g. , information implicit in the queue length graph inFigure 1b. Ideally, the probe-measured state reects the truestate of the network. That is, a given probe Pi should accuratelyreport the following:

Pi =0 : if a loss episode is not encountered1 : if a loss episode is encountered (1)

Satisfying this requirement is problematic because, as notedabove, many packets are successfully transmitted during loss


4/14

4

episodes. We address this issue in our probe process in VIand heuristically in VII.

Finally, we dene a probe to consist of one or more veryclosely spaced ( i.e. , back-to-back) packets. As we will see in

VII , the reason for using multi-packet probes is that notall packets passing through a congested link are subject toloss; constructing probes of multiple packets enables a moreaccurate determination to be made.

IV. L ABORATORY TESTBED

The laboratory testbed used in our experiments is shownin Figure 2. It consisted of commodity end hosts connectedto a dumbbell-like topology comprised of Cisco GSR 12000routers. Both probe and background trafc were generated andreceived by the end hosts. Trafc owed from the sendinghosts on separate paths via Gigabit Ethernet to separate CiscoGSRs (hop B in the gure) where it transitioned to OC12(622 Mb/s) links. This conguration was created in orderto accommodate our measurement system, described below.Probe and background trafc was then multiplexed onto asingle OC3 (155 Mb/s) link (hop C in the gure) which formedthe bottleneck where loss episodes took place. We used ahardware-based propagation delay emulator on the OC3 link toadd 50 milliseconds delay in each direction for all experiments,and congured the bottleneck queue to hold approximately100 milliseconds of packets. Packets exited the OC3 link viaanother Cisco GSR 12000 (hop D in the gure) and passed toreceiving hosts via Gigabit Ethernet.

The probe and trafc generator hosts consisted of identicallycongured workstations running Linux 2.4. The workstationshad 2 GHz Intel Pentium 4 processors with 2 GB of RAM andIntel Pro/1000 network cards. They were also dual-homed, so

that all management trafc was on a separate network thandepicted in Figure 2.One of the most important aspects of our testbed was

the measurement system we used to establish the true lossepisode state (ground truth) for our experiments. Opticalsplitters were attached to both the ingress and egress linksat hop C and Endace DAG 3.5 and 3.8 passive monitoringcards were used to capture traces of packets entering andleaving the bottleneck node. DAG cards have been usedextensively in many other studies to capture high delitypacket traces in live environments ( e.g., they are deployed inSprints backbone [26] and in the NLANR infrastructure [27]).By comparing packet header information, we were able to

identify exactly which packets were lost at the congestedoutput queue during experiments. Furthermore, the fact thatthe measurements of packets entering and leaving hop Cwere time-synchronized on the order of a single microsecondenabled us to easily infer the queue length and how the queuewas affected by probe trafc during all tests.

We consider this environment ideally suited to understand-ing and calibrating end-to-end loss measurement tools. Lab-oratory environments do not have the weaknesses typicallyassociated with ns-type simulation ( e.g., abstractions of mea-surement tools, protocols and systems) [28], nor do they havethe weaknesses of wide area in situ experiments ( e.g., lack of

control, repeatability, and complete, high delity end-to-endinstrumentation). We address the important issue of testingthe tool under representative trafc conditions by using acombination of the Harpoon IP trafc generator [29] andIperf [30] to evaluate the tool over a range of cross trafcand loss conditions.

V. E VALUATION OF S IMPLE POISSON PROBING FORPACKET L OSS

We begin by using our laboratory testbed to evaluate thecapabilities of simple Poisson-modulated loss probe measure-ments using the ZING tool [17], [18]. ZING measures packetdelay and loss in one direction on an end-to-end path. TheZING sender emits UDP probe packets at Poisson-modulatedintervals with timestamps and unique sequence numbers andthe receiver logs the probe packet arrivals. Users specify themean probe rate , the probe packet size, and the number of packets in a ight.

To evaluate simple Poisson probing, we congured ZINGusing the same parameters as in [5]. Namely, we ran two

tests, one with = 100 ms (10 Hz) and 256 byte payloadsand another with = 50ms (20Hz) and 64 byte payloads. Todetermine the duration of our experiments below, we selecteda period of time that should limit the variance of the loss rateestimator X where Var ( X n) pn for loss rate p and number of probes n.

We conducted three separate experiments in our evaluationof simple Poisson probing. In each test we measured boththe frequency and duration of packet loss episodes. Again,we used the denition in [5] for loss episode: a series of consecutive packets (possibly only of length one) that werelost.

The rst experiment used 40 innite TCP sources with

receive windows set to 256 full size (1500 bytes) packets.Figure 3a shows the time series of the queue occupancy fora portion of the experiment; the expected synchronizationbehavior of TCP sources in congestion avoidance is clear. Theexperiment was run for a period of 15 minutes which shouldhave enabled ZING to measure loss rate with standard deviationwithin 10% of the mean [10].

Results from the experiment with innite TCP sources areshown in Table I. The table shows that ZING performs poorlyin measuring both loss frequency and duration in this scenario.For both probe rates, there were no instances of consecutivelost packets, which explains the inability to estimate lossepisode duration.

In the second set of experiments, we used Iperf to create aseries of (approximately) constant duration (about 68 millisec-onds) loss episodes that were spaced randomly at exponentialintervals with mean of 10 seconds over a 15 minute period.The time series of the queue length for a portion of the testperiod is shown in Figure 3b.

Results from the experiment with randomly spaced, constantduration loss episodes are shown in Table II. The table showsthat ZING measures loss frequencies and durations that arecloser to the true values.

In the nal set of experiments, we used Harpoon to createa series of loss episodes that approximate loss resulting from


5/14

5

web-like trafc. Harpoon was congured to briey increaseits load in order to induce packet loss, on average, every20 seconds. The variability of trafc produced by Harpooncomplicates delineation of loss episodes. To establish baselineloss episodes to compare against, we found trace segmentswhere the rst and last events were packet losses, and queuingdelays of all packets between those losses were above 90milliseconds (within 10 milliseconds of the maximum). Weran this test for 15 minutes and a portion of the time seriesfor the queue length is shown in Figure 3c.

Results from the experiment with Harpoon web-like trafcare shown in Table III. For measuring loss frequency, neitherprobe rate results in a close match to the true frequency. Forloss episode duration, the results are also poor. For the 10 Hzprobe rate, there were no consecutive losses measured, andfor the 20 Hz probe rate, there were only two instances of consecutive losses, each of exactly two lost packets.

I: Results from ZING experiments with innite TCP sources.frequency duration mean (std. dev.)

(seconds)true values 0.0265 0.136 (0.009)

ZING (10Hz) 0.0005 0 (0)ZING (20Hz) 0.0002 0 (0)

II: Results from ZING experiments with randomly spaced,constant duration loss episodes.

frequency duration mean (std. dev.)(seconds)

true values 0.0069 0.068 (0.000)ZING (10Hz) 0.0036 0.043 (0.001)ZING (20Hz) 0.0031 0.050 (0.002)

III: Results from ZING experiments with Harpoon web-liketrafc.

frequency duration mean (std. dev.)(seconds)

true values 0.0093 0.136 (0.009)ZING (10Hz) 0.0014 0 (0)ZING (20Hz) 0.0012 0.022 (0.001)

VI. P ROBE PROCESS M ODEL

The results from our experiments described in the previoussection show that simple Poisson probing is generally poor formeasuring loss episode frequency and loss episode duration.These results, along with deeper investigation of the reasonsfor particular deciencies in loss episode duration measure-ment, form the foundation for a new measurement process.

A. General Setup

Our methodology involves dispatching a sequence of probes, each consisting of one or more very closely spacedpackets. The aim of a probe is to obtain a snapshot of thestate of the network at the instant of probing. As such, therecord for each probe indicates whether or not it encountered

10 12 14 16 18 20 0 . 0

0

0 . 0

2

0 . 0

4

0 . 0

6

0 . 0

8

0 . 1

0

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

)

(a) Queue length time series for a portion of the experiment with 40 inniteTCP sources.

30 32 34 36 38 40 0 . 0 0

0 . 0

2

0 . 0

4

0 . 0

6

0 . 0

8

0 . 1

0

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

)

(b) Queue length time series for a portion of the experiment with randomlyspaced, constant duration loss episodes.

34 36 38 40 42 44 0 . 0 0

0 . 0

2

0 . 0

4

0 . 0

6

0 . 0

8

0 . 1

0

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

)

(c) Queue length time series for a portion of the experiment with Harpoonweb-like trafc. Time segments in grey indicate loss episodes.

3: Queue length time series plots for three different back-ground trafc scenarios.

a loss episode, as evidenced by either the loss or sufcientdelay of any of the packets within a probe ( c.f. VII ).

The probes themselves are organized into what we term ba-sic experiments , each of which comprises a number of probes

sent in rapid succession. The aim of the basic experiment is todetermine the dynamics of transitions between the congestedand uncongested state of the network, i.e. , beginnings andendings of loss episodes. Below we show how this enablesus to estimate the duration of loss episodes.

A full experiment comprises a sequence of basic experi-ments generated according to some rule. The sequence may beterminated after some specied number of basic experiments,or after a given duration, or in an open-ended adaptive fashion,e.g., until estimates of desired accuracy for a loss characteristichave been obtained, or until such accuracy is determinedimpossible.


6/14

6

We formulate the probe process as a discrete-time process.This decision is not a fundamental limitation: since we areconcerned with measuring loss episode dynamics, we needonly ensure that the interval between the discrete time slots issmaller than the time scales of the loss episodes.

There are three steps in the explanation of our loss measure-ment method ( i.e. , the experimental design and the subsequentestimation). First, we present the basic algorithm version. Thismodel is designed to provide estimators of the frequency of time slots in which loss episodes is present, and the durationof loss episodes. The frequency estimator is unbiased, andunder relatively weak statistical assumptions, both estimatorsare consistent in the sense they converge to their respectivetrue values as the number of measurements grows.

Second, we describe the improved algorithm version of ourdesign which provides loss episode estimators under weakerassumptions, and requires that we employ a more sophisticatedexperimental design. In this version of the model, we insert amechanism to estimate, and thereby correct, the possible biasof the estimators from the basic design.

Third, we describe simple validation techniques that can beused to assign a level of condence to loss episode estimates.This enables open-ended experimentation with a stoppingcriterion based on estimators reaching a requisite level of condence.

B. Basic Algorithm

For each time slot i we decide whether or not to commencea basic experiment; this decision is made independently foreach slot with some xed probability p over all slots. In thisway, the sequence of basic experiments follows a geometricdistribution with parameter p. (In practice, we make the

restriction that we do not start a new basic experiment whileone is already in progress. This implies that, in reality, therandom variables controlling whether or not a probe is sentat time slot i are not entirely independent of each other.) Weindicate this series of decisions through random variables { xi}that take the value 1 if a basic experiment is started in sloti and 0 otherwise.

If xi = 1, we dispatch two probes to measure congestion inslots i and i + 1. The random variable yi records the reportsobtained from the probes as a 2-digit binary number, i.e. , yi =00 means both probes did not observe a loss episode, while yi = 10 means the rst probe observed a loss episode whilethe second one did not, and so on. Our methodology is based

on the following fundamental assumptions, which, in view of the probe and its reporting design (as described in VII) arevery likely to be valid ones. These assumptions are required in both algorithmic versions. The basic algorithm requires astronger version of these assumptions, as we detail later.

1) Assumptions: We do not assume that the probes accu-rately report loss episodes: we allow that a true loss episodepresent during a given time slot may not be observed by anyof the probe packets in that slot. However, we do assume aspecic structure of the inaccuracy, as follows.

Let Y i be the true loss episode state in slots i and i + 1, i.e.,Y i = 01 means that there is no loss episode present at t = i

and that a loss episode is present at t = i + 1. As described in

III, true means the congestion that would be observed werewe to have knowledge of router buffer occupancy, queueingdelays and packet drops. Of course, in practice the value of Y i is unknown. Our specic assumption is that yi is correct,i.e. , equals Y i , with probability pk that is independent of i anddepends only on the number k of 1-digits in Y i . Moreover, if yi is incorrect, it must take the value 00. Explicitly,(1) If Y i = 00 (= no loss episode occuring) then yi = 00, too

(= no congestion reported), with probability 1.(2) If Y i = 01 (= loss episode begins), or Y i = 10 (= loss

episode ends), then P ( yi = Y i|(Y i = 01)(Y i = 10 )) = p1,for some p1 which is independent of i. If yi fails to matchY i , then necessarily, yi = 00.

(3) If Y i = 11 (= loss episode is on-going), then P ( yi = Y i|Y i =11 ) = p2 , for some p2 which is independent of i. If yi failsto match Y i, then necessarily, yi = 00.

As justication for the above assumptions we rst note thatit is highly unlikely that a probe will spuriously measure loss.That is, assuming well-provisioned measurement hosts, if no

loss episode is present a probe should not register loss. Inparticular, for assumptions (1) and (2), if yi = Y i , it follows that yi must be 00. For assumption (3), we appeal to the one-waydelay heuristics developed in VII: if yi = 00, then we holdin hand at least one probe that reported loss; by comparingthe delay characteristics of that probe to the correspondingcharacteristics in the other probe (assuming that the other onedid not report loss), we are able to deduce whether to assigna value 1 or 0 to the other probe. Thus, the actual networkingassumption is that the delay characteristics over the measuredpath are stationary relative to the time discretization we use.

2) Estimation: The basic algorithm assumes that p1 = p2 for consistent duration estimation, and p1 = p2 = 1 forconsistent and unbiased frequency estimation. The estimatorsare as follows:Loss Episode Frequency Estimation . Denote the true fre-quency of slots during which a loss episode is present by F .We dene a random variable zi whose value is the rst digitof yi . Our estimate is then

F = i zi/ M , (2)with the index i running over all the basic experiments weconducted, and M is the total number of such experiments.

This estimator is unbiased, E [

F ] = F , since the expected

value of zi is just the congestion frequency F . Under mildconditions ( i.e. , p1 = p2 = 1), the estimator is also consistent.For example, if the durations of the loss episodes and loss-freeepisodes are independent with nite mean, then the proportionof lossy slots during an experiment over N slots convergesalmost surely, as N grows, to the loss episode frequency F ,from which the stated property follows.Loss Episode Duration Estimation is more sophisticated.Recall that a loss episode is one consecutive occurrence of k lossy time slots preceded and followed by no loss, i.e., itsbinary representation is written as:

01 . . . 10 .


7/14

7

Suppose that we have access to the true loss episode state at allpossible time slots in our discretization. We then count all lossepisodes and their durations and nd out that for k = 1, 2, . . . ,there were exactly jk loss episodes of length k . Then, lossoccurred over a total of

A = k

k jk

slots, while the total number of loss episodes is

B = k

jk .

The average duration D of a loss episode is then dened as

D := A/ B.

In order to estimate D , we observe that, with the abovestructure of loss episodes in hand, there are exactly B timeslots i for which Y i = 01, and there are also B time slots i forwhich Y i = 10. Also, there are exactly A + B time slots i forwhich Y i = 00. We therefore dene

R := #{i : yi{01 , 10 , 11}},and

S := #{i : yi{01 , 10}}.Now, let N be the total number of time slots. Then P (Y i{01 , 10}) = 2 B/ N , hence P ( yi{01 , 10}) = 2 p1 B/ N .Similarly, P (Y i {01 , 10 , 11}) = ( A + B)/ N , and P ( yi {01 , 10 , 11}) = ( p2( A B) + 2 p1 B)/ N . Thus,

E ( R)/ E (S) =p2( A

B) + 2 p1 B

2 p1 B .

Denoting r := p2/ p1, we get then

E ( R)/ E (S) =r ( A B) + 2 B

2 B=

rA2 B r / 2 + 1.

Thus,

D =2r

E ( R)E (S) 1 + 1. (3)

In the basic algorithm we assume r = 1, the estimator D of D is then obtained by substituting the measured values of S

and R for their means:

D := 2 RS 1. (4)

Note that this estimator is not unbiased for nite N , due tothe appearance of S in the quotient. However, it is consistentunder the same conditions as those stated above for F , namely,that congestion is described by an alternating renewal processwith nite mean lifetimes. Then the ergodic theorem tells usthat as N grows, R/ N and S/ N converge to their expectedvalues (note, e.g. , E [ R/ N ] = p P [Y i{01 , 10 , 11}] independentof N ) and hence D converges almost surely to D .

C. Improved Algorithm

The improved algorithm is based on weaker assumptionsthan the basic algorithm: we no longer assume that p1 = p2.In view of the details provided so far, we will need, for theestimation of duration, to know the ratio r := p1/ p2. For that,we modify our basic experiments as follows.

As before, we decide independently at each time slot

whether to conduct an experiment. With probability 1/2, thisis a basic experiment as before; otherwise we conduct anextended experiment comprising three probes, dispatched inslots i, i + 1, i + 2, and redene yi to be the corresponding 3-digit number returned by the probes, e.g., y i = 001 means losswas observed only at t = i + 2, etc. As before Y i records thetrue states that our ith experiment attempts to identify. We nowmake the following additional assumptions.

1) Additional Assumptions: We assume that the probabilitythat yi misses the true state Y i (and hence records a string of 0s), does not depend on the length of Y i but only on thenumber of 1s in the string. Thus, P ( yi = Y i) = p1 whenever Y iis any of {01 , 10 , 001 , 100}, while P ( yi = Y i) = p2 whenever Y iis any of {11 , 011 , 110 }(we address states 010 and 101 below).We claim that these additional assumptions are realistic, butdefer the discussion until after we describe the reportingmechanism for loss episodes.

With these additional assumptions in hand, we denote

U := #{i : yi{011 , 110 }},and

V := #{i : yi{001 , 100}}.The combined number of states 011 , 110 in the full timeseries is 2 B, while the combined number of states of the form001 , 100 is also 2 B. Thus, we have

E (U )E (V )

= r ,

hence, with U / V estimating r , we employ (Eq. 3) to obtain

D :=2V U

RS 1 + 1.

D. Validation

When running an experiment, our assumptions require thatseveral quantities have the same mean. We can validate theassumptions by checking those means.

In the basic algorithm, the probability of yi = 01 is assumedto be the same as that of yi = 10. Thus, we can design astopping criterion for on-going experiments based on the ratiobetween the number of 01 measurements and the number of 10 measurements. A large discrepancy between these numbers(that is not bridged by increasing M ) is an indication thatour assumptions are invalid. Note that this validation doesnot check whether r = 1 or whether p1 = 1, which are twoimportant assumptions in the basic design.

In the improved design, we expect to get similar occurrencerate for each of yi = 01 , 10 , 001 , 100. We also expect toget similar occurrence rate for yi = 011 , 110. We can check those rates, stop whenever they are close, and invalidate the


8/14

8

experiment whenever the mean of the various events do notcoincide eventually. Also, each occurrence of yi = 010 or yi = 101 is considered a violation of our assumptions. A largenumber of such events is another reason to reject the resultedestimations. Experimental investigation of stopping criteria isfuture work.

E. ModicationsThere are various straightforward modications to the above

design that we do not address in detail at this time. Forexample, in the improved algorithm, we have used the triple-probe experiments only for the estimation of the parameter r .We could obviously include them also in the actual estimationof duration, thereby decreasing the total number of probes thatare required in order to achieve the same level of condence.

Another obvious modication is to use unequal weighingbetween basic and extended experiments. In view of the ex-pression we obtain for D there is no clear motivation for doingthat: a miss in estimating V / U is as bad as a correspondingmiss in R/ S (unless the average duration is very small). Basicexperiments incur less cost in terms of network probing load.On the other hand, if we use the reports from triple probesfor estimating E (S)/ E ( R) then we may wish to increase theirproportion. Note that in our formulation, we cannot use thereported events yi = 111 for estimating anything, since thefailure rate of the reporting on the state Y i = 111 is assumedto be unknown. (We could estimate it using similar techniquesto those used in estimating the ratio p2/ p1. This, however,will require utilization of experiments with more than threeprobes). A topic for further research is to quantify the trade-offs between probe load and estimation accuracy involved inusing extended experiments of 3 or more probes.

F. Estimator Variance

In this section we determine the variance in estimating theprobe loss rate F and the mean loss episode duration D thatarises from the sampling action of the probes. It is importantto emphasize that all the variation we consider stems from therandomness of the probing, rather than any randomness of theunderlying congestion periods under study. Rather, we viewthe congestion under study as a single xed sample path.

1) Assumptions on the Underlying Congestion: One couldrelax this point of view and allow that the sample pathof the congestion is drawn according to some underlyingprobability distribution. But it turns out that, under very weak

assumptions, our result holds almost surely for each suchsample path.

To formalize this, recall that during N measurement slotsthere are A congested slots distributed amongst B congestionintervals. We shall be concerned with the asymptotics of theestimators F and D for large N . To this end, we assume that A and B have the following behavior for large N , namely, forsome positive a and b

F = A/ N a and B/ N b, as N We also write d = a / b to denote the limiting average durationof a congestion episode.

For a wide class of statistical models of congestion, theseproperties will be obeyed almost surely with uniform a and b,namely, if A and B satisfy the strong law of large numbersas N . Examples of models that possess this propertyinclude Markov processes, and alternating renewal processeswith nite mean lifetimes in the congested and uncongestedstates.

2) Asymptotic Variance of F and D: We can write theestimators F and D in a different but equivalent way to thoseused above. Let there be N slots in total, and for the fourstate pairs y = 01 , 10 , 00 , 11 let y denote the set of slots i inwhich the true loss episode state was y. Let xi = 1 if a basicexperiment was commenced in slot i. Then X y = i

y xi is

the number of basic experiments that encountered the truecongestion state y. Note that since the y are xed sets, the X y are mutually independent. In what follows we restrict ourattention to the basic algorithm in the ideal case p1 = p2 = 1.Comparing with VI-B we have:

F = f F ( X 11 , X 10 , X 01 , X 00) :=

X 10 + X 11 X 00 + X 01 + X 10 + X 11

D = f D( X 11 , X 10 , X 01 ) := 1 + 2 X 11

X 10 + X 01

We now determine the asymptotic variances and covarianceof F and D as N grows using the -method; see [31].This supposes a sequence X ( N ) = ( X ( N )1 , . . . , X

( N )m ) of vector

valued random variables and a xed vector X = ( X 1 , . . . , X m)such that N 1/ 2( X ( N ) X ) converges in distribution as N to a multivariate Gaussian random variable of mean 0 =(0, . . . , 0) and covariance matrix c = ( ci j )i, j= 1,...m. If f is avector function R m R m that is differentiable about X , then N 1/ 2( f ( X ( N ) ) f ( X )) is asymptotically Gaussian, as N ,with mean 0 and asymptotic covariance matrix

ck =m

i, j= 1

f k ( X ) X i

ci j f ( X )

X j

In the current application we set f = ( f F , f D), X ( N ) = ( X 11 , X 10 , X 01 , X 00 )/ N and X = lim N E [ X

( N )] =lim N (#11 , #10 , #01 , #00) p/ N = ( a b, b, b, 1 a b) p. Since the X y are independent, the covariancematrix of N 1/ 2 X ( N ) is the diagonal matrix c with entriesVar ( X y)/ N = # y Var ( xi)/ N = p(1 p)# y/ N (1 p) X yas N . The derivatives of f F and f D are

f F ( X ) = ( 1 a , 1 a ,a ,a )/ p, f D( X ) = ( b, (b a )/ 2, (b a )/ 2, 0)/ ( pb2)

Thus using the -method we have shown that N 1/ 2(( F , D) (a , d )) is asymptotically Gaussian with mean 0 and covariance f F ( X ) c f F ( X ) f F ( X ) c f D( X ) f D( X ) c f F ( X ) f D( X ) c f D( X )

=1 p

pa (1 a ) (d 1)/ 2(d 1)/ 2 d (d 2 1)/ (2a ))

Note that positive correlation between F and D is expected,since with higher loss episode frequency, loss episodes willtender to be longer.


9/14

9

3) Variance Estimation: For nite N , we can estimate thevariance of F and D directly from the data by pluggingin estimated values for the parameters and scaling by N .Specically, we estimate the variances of F and D respectivelyby

V F = F (1 F )(1 p) N p and V D = D( D2 1)(1 p)2 N

F p

Thus, simple estimates of the relative standard deviations of

F and D are thus 1 / ( pN F ) and 1 / (2 pN B) respectively, where B = F / D is the estimated frequency of congestion periods.Estimated condence intervals for F and D follow in anobvious manner.

VII. P ROBE TOOL IMPLEMENTATIONAND EVALUATION

To evaluate the capabilities of our loss probe measurementprocess, we built a tool called B ADABING 1 that implementsthe basic algorithm of VI. We then conducted a series of experiments with B ADABING in our laboratory testbed withthe same background trafc scenarios described in V.The objective of our lab-based experiments was to validateour modeling method and to evaluate the capability of B AD -ABING over a range of loss conditions. We report results of experiments focused in three areas. While our probe processdoes not assume that we always receive true indications of loss from our probes, the accuracy of reported measurementswill improve if probes more reliably indicate loss. With this inmind, the rst set of experiments was designed to understandthe ability of an individual probe (consisting of 1 to N tightly-spaced packets) to accurately report an encounter witha loss episode. The second is to examine the accuracy of BADABING in reporting loss episode frequency and durationfor a range of probe rates and trafc scenarios. In our nalset of experiments, we compare the capabilities of B ADABINGwith simple Poisson-modulated probing.

A. Accurate Reporting of Loss Episodes by Probes

We noted in III that, ideally, a probe should providean accurate indication of the true loss episode state (Eq. 1).However, this may not be the case. The primary issue is thatduring a loss episode, many packets continue to be success-fully transmitted. Thus, we hypothesized that we might be ableto increase the probability of probes correctly reporting a lossepisode by increasing the number of packets in an individual

probe. We also hypothesized that, assuming FIFO queueing,using one-way delay information could further improve theaccuracy of individual probe measurements.

We investigated the rst hypothesis in a series of experi-ments using the innite TCP source background trafc andconstant-bit rate trafc described in V. For the inniteTCP trafc, loss event durations were approximately 150milliseconds. For the constant-bit rate trafc, loss episodeswere approximately 68 milliseconds in duration. We used a

1Named in the spirit of past tools used to measure loss including PING ,ZING , and S TING . This tool is approximately 800 lines of C++ and is availableto the community for testing and evaluation.

modied version of B ADABING to generate probes at xedintervals of 10 milliseconds so that some number of probeswould encounter all loss episodes. We experimented withprobes consisting of between 1 and 10 packets. Packets inan individual probe were sent back to back per the capabil-ities of the measurement hosts ( i.e., with approximately 30microseconds between packets). Probe packet sizes were setat 600 bytes 2.

2 4 6 8 10

0 . 0

0 . 2

0 . 4

0 . 6

0 . 8

1 . 0

bunch length (packets)

e m p

i r i c a

l p r o

b a

b i l i t y t h a

t a p r o

b e

b u n c

h

e x p e r i e n c e s n o

l o s s

d u r i n g a

l o s s e p

i s o

d e

infinite TCP trafficconstantbit rate traffic

4: Results from tests of ability of probes consisting of N packets to report loss when an episode is encountered.

Figure 4 shows the results of these tests. We see that forthe constant-bit rate trafc, longer probes have a clear impacton the ability to detect loss. While about half of single-packet probes do not experience loss during a loss episode,probes with just a couple more packets are much more reliableindicators of the true loss episode state. For the innite TCPtrafc, there is also an improvement as the probes get longer,but the improvement is relatively small. Examination of thedetails of the queue behavior during these tests demonstrateswhy the 10 packet probes do not greatly improve loss reportingability for the innite source trafc. As shown in Figure 5,longer probes begin to have a serious impact on the queuingdynamics during loss episodes.

This observation, along with our hypothesis regarding one-way packet delays, led to our development of an alternativeapproach for identifying loss events. Our new method consid-ers both individual packet loss with probes and the one-waypacket delay as follows. For probes in which any packet is lost,we consider the one-way delay of the most recent successfullytransmitted packet as an estimate of the maximum queue depth(OWD max ). We then consider a loss episode to be delimited

by probes within seconds of an indication of a lost packet(i.e. , a missing probe sequence number) and having a one-waydelay greater than (1 )OWD max . Using the parameters and , we mark probes as 0 or 1 according to Eq. 1 and formestimates of loss episode frequency and duration using Eqs. 2and 4, respectively. Note that even if packets of a given probeare not actually lost, the probe may be considered to haveexperienced a loss episode due to the and/or thresholds.

2 This packet size was chosen to exploit an architectural feature of the CiscoGSR so that probe packets had as much impact on internal buffer occupancy asmaximum-sized frames. Investigating the impact of packet size on estimationaccuracy is a subject for future work.


10/14

10

11.66 11.68 11.70 11.72 11.74

0 . 0

9 8 0

0 . 0

9 9 0

0 . 1

0 0 0

0 . 1

0 1 0

no probe traffic

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

)

x x xx xx x x xx xx x xxx xxxx x xxxxx

xcross traffic packetcross traffic loss

15.20 15.22 15.24 15.26 15.28 15.30

0 . 0

9 8 0

0 . 0

9 9 0

0 . 1

0 0 0

0 . 1

0 1 0

probe train of 3 packets

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

) oo

ooo oo

ooo

ooo

ooo o

ooo

ooo oo ooo

xx xx xx xxxxx xx x xx x x x x x x x xx xx x x xx x x x xx x xx xx+ + ++ +

xo+

cross traffic packetcross traffic lossprobeprobe loss

14.80 14.82 14.84 14.86 14.88 14.90

0 . 0

9 8 0

0 . 0

9 9 0

0 . 1

0 0 0

0 . 1

0 1 0

probe train of 10 packets

time (seconds)

q u e u e

l e n g

t h ( s e c o n

d s

)

x xx xxxxx x x xx xxxxx x xxx x x xx x x x xxxxxxxxxxxxxx xx xxxxxxxxxxxx xx

oooooooooo

oooooooo

ooooooooo

oooooooooo

oooooooo

oooooooooo

ooooooooo

ooooooooo o

oo

ooooooooo

oooooooooo

++ + ++ + + +++++++ +

xo+

cross traffic packetcross traffic lossprobeprobe loss

5: Queue length during a portion of a loss episode for differentsize loss probes. The top plot shows innite source TCP trafc

with no loss probes. The middle plot shows innite source TCPtrafc with loss probes of three packets, and the bottom plotsshows loss probes of 10 packets. Each plot is annotated withTCP packet loss events and probe packet loss events.

This formulation of probe-measured loss assumes that queu-ing at intermediate routers is FIFO. Also, we can keep anumber of estimates of OWD max , taking the mean whendetermining whether a probe is above the (1 ) OWDthreshold or not. Doing so effectively lters loss at end hostoperating system buffers or in network interface card buffers,since such losses are unlikely to be correlated with end-to-end

network congestion and delays.We conducted a series of experiments with constant-bit

rate trafc to assess the sensitivity of the loss thresholdparameters. Using a range of values for probe send probability( p), we explored a cross product of values for and .For , we selected 0.025, 0.05, 0.10, and 0.20, effectivelysetting a high-water level of the queue of 2.5, 5, 10, and 20milliseconds. For , we selected values of 5, 10, 20, 40, and80 milliseconds. Figure 6a shows results for loss frequencyfor a range of p, with xed at 80 milliseconds, and varying between 0.05, 0.10, and 0.20 (equivalent to 5, 10, and20 milliseconds). Figure 6b xes at 0.10 (10 milliseconds)

0.0 0.2 0.4 0.6 0.8 1.0 0

. 0 0 0

0 . 0

0 4

0 . 0

0 8

0 . 0

1 2

probe probability (p)

l o s s

f r e q u e n c y

true loss frequenc

alpha=0.05 (5 millisec.)alpha=0.10 (10 millisec.)alpha=0.20 (20 millisec.)

(a) Estimated loss frequency over a range of values for while holding xed at 80 milliseconds.

0.0 0.2 0.4 0.6 0.8 1.0 0

. 0 0 0

0 . 0

0 4

0 . 0

0 8

0 . 0

1 2

probe probability (p)

l o s s

f r e q

u e n c y

true loss frequenc

tau=20 millisec.tau=40 millisec.tau=80 millisec.

(b) Estimated loss frequency over a range of valuesfor while holding xed at 0.1 (equivalent to 10milliseconds).

6: Comparison of the sensitivity of loss frequency estimationto a range of values of and .

while letting vary over 20, 40, and 80 milliseconds. Wesee, as expected, that with larger values of either threshold,estimated frequency increases. There are similar trends forloss duration (not shown). We also see that there is a trade-off between selecting a higher probe rate and more permissivethresholds. It appears that the best setting for comes aroundthe expected time between probes plus one or two standarddeviations. The best appears to depend both on the proberate and on the trafc process and level of multiplexing, whichdetermines how quickly a queue can ll or drain. Consideringsuch issues, we discuss parameterizing B ADABING in generalInternet settings in

VIII.

B. Measuring Frequency and Duration

The formulation of our new loss probe process in VI callsfor the user to specify two parameters, N and p, where p is theprobability of initiating a basic experiment at a given interval.In the next set of experiments, we explore the effectivenessof BADABING to report loss episode frequency and durationfor a xed N , and p using values of 0.1, 0.3, 0.5, 0.7, and0.9 (implying that probe trafc consumed between 0.2% and1.7% of the bottleneck link). With the time discretizationset at 5 milliseconds, we xed N for these experiments at


11/14


12/14

12

VII: Comparison of loss estimates for p = 0.1 and twodifferent values of N and two different values for the threshold parameter.

1

N loss frequency loss duration(seconds)

true B ADABING true B ADABING180,000 40 0.0059 0.0006 0.068 0.021180,000 80 0.0059 0.0015 0.068 0.053720,000 40 0.0059 0.0009 0.068 0.020720,000 80 0.0059 0.0018 0.068 0.041

C. Dynamic Characteristics of the Estimators

As we have shown, estimates for a low probe rate do notsignicantly improve even with rather large N . A modestincrease in the probe rate p, however, substantially improvesthe accuracy and convergence time of both frequency andduration estimates. Figure 7 shows results from an experimentusing Harpoon to generate self-similar, web-like TCP trafcfor the loss episodes. For this experiment, p is set to 0.5.The top plot shows both the dynamic characteristics of bothtrue and estimated loss episode frequency for the entire 15minute-long experiment. B ADABING estimates are producedevery 60 seconds for this experiment. The error bars at eachBADABING estimate indicate a 95% condence interval forthe estimates. We see that even after one or two minutes,BADABING estimates have converged close to the true val-ues. We also see that B ADABING tracks the true frequencyreasonably well. The bottom plot in Figure 7 compares thetrue and estimated characteristics of loss episode duration forthe same experiment. Again, we see that after a short period,BADABING estimates and condence intervals have convergedclose to the true mean loss episode duration. We also see thatthe dynamic behavior is generally well followed. Except for

the low probe rate of 0.1, results for other experiments exhibitsimilar qualities.

D. Comparing Loss Measurement Tools

Our nal set of experiments compares B ADABING withZING using the constant-bit rate and Harpoon web-like trafcscenarios. We set the probe rate of ZING to match the link utilization of B ADABING when p = 0.3 and the packet sizeis 600 bytes, which is about 876 kb/s, or about 0.5% of the capacity of the OC3 bottleneck. Each experiment wasrun for 15 minutes. Table VIII summarizes results of theseexperiments, which are similar to the results of V. (Includedin this table are B ADABING results from row 2 of Tables IV

and VI. ) For the CBR trafc, the loss frequency measured byZING is somewhat close to the true value, but loss episodedurations are not. For the web-like trafc, neither the lossfrequency nor the loss episode durations measured by ZING aregood matches to the true values. Comparing the ZING resultswith B ADABING , we see that for the same trafc conditionsand probe rate, B ADABING reports loss frequency and durationestimates that are signicantly closer to the true values.

VIII. U SING BADABING IN PRACTICE

There are a number of important practical issues which mustbe considered when using B ADABING in the wide area:

0 200 400 600 800

0 . 0

0 5

0 . 0

1 0

0 . 0

1 5

0 . 0

2 0

time (seconds)

l o s s e p

i s o

d e

f r e q u e n c y

o

estimated loss frequencytrue loss frequency

0 200 400 600 800

0 . 0

2

0 . 0

4

0 . 0

6

0 . 0

8

0 . 1

0

0 . 1

2

0 . 1

4

time (seconds)

m e a n

l o s s e p

i s o

d e

d u r a

t i o n

( s e c o n

d s

)

o

estimated mean loss episode durationtrue mean loss episode duration

7: Comparison of loss frequency and duration estimates withtrue values over 15 minutes for Harpoon web-like cross trafcand a probe rate p = 0.5. BADABING estimates are producedevery minute, and error bars at each estimate indicate the 95%condence interval. Top plot shows results for loss episodefrequency and bottom plot shows results for loss episodeduration.

VIII: Comparison of results for B ADABING and ZING withconstant-bit rate (CBR) and Harpoon web-like trafc. Proberates matched to p = 0.3 for BADABING (876 kb/s) with probepacket sizes of 600 bytes. (B ADABING results copied fromrow 2 of Tables IV and VI. Variability in true frequencyand duration for Harpoon trafc scenarios is due to inherentvariability in background trafc source.)

trafc tool loss frequency loss durationscenario true measured true (sec) measured (sec)

CBR B ADABING 0.0069 0.0065 0.068 0.073Z ING 0.0069 0.0041 0.068 0.010

H arpoon B ADABING 0.0011 0.0011 0.113 0.143web-like Z ING 0.0159 0.0019 0.119 0.007

The tool requires the user to select values for p and N . Assume for now that the number of loss events isstationary over time. (Note that we allow the duration of the loss events to vary in an almost arbitrary way, andto change over time. One should keep in mind that inour current formulation we estimate the average durationand not the distribution of the durations.) Let B0 bethe mean number of loss events that occur over a unitperiod of time. For example, if an average of 12 lossevents occur every minute, and our discretization unit is5 milliseconds, then B0 = 12 / (60 200 ) = .001 (this is,of course, an estimate of the true the value of B0). Withthe stationarity assumption on B0, we expect the accuracy


13/14

13

of our estimators to depend on the product pNB 0, butnot on the individual values of p, N or B0 .3 . Indeed, wehave seen in VI-F.2 that a reliable approximation of therelative standard deviation in our estimation of durationis given by:

RelStdDev (duration ) 1

2 pNB0Thus, the individual choice of p and N allows a trade off between timeliness of results and impact that the useris willing to have on the link. Prior empirical studiescan provide initial estimates of B0 . An alternate designis to take measurements continuously, and to report anestimate when our validation techniques conrm that theestimation is robust. This can be particularly useful insituations where p is set at low level. In this case, whilethe measurement stream can be expected to have littleimpact on other trafc, it may have to run for some timeuntil a reliable estimate is obtained.

Our estimation of duration is critically based on correctestimation of the ratio B/ M (cf. VI). We estimate thisratio by counting the occurrence rate of yi = 01, as well asthe occurrence rate of yi = 10. The number B/ M can beestimated as the average of these two rates. The validationis done by measuring the difference between these tworates. This difference is directly proportional to the ex-pected standard deviation of the above estimation. Similarremarks apply to other validation tests we mention in bothestimation algorithms.

The recent study on packet loss via passive measurementreported in [9] indicates that loss episodes in backbonelinks can be very short-lived ( e.g., on the order of several microseconds). The only condition for our tool tosuccessfully detect and estimate such short durations isfor our discretization of time to be ner than the order of duration we attempt to estimate. Such a requirement mayimply that commodity workstations cannot be used foraccurate active measurement of end-to-end loss charac-teristics in some circumstances. A corollary to this is thatactive measurements for loss in high bandwidth networksmay require high-performance, specialized systems thatsupport small time discretizations.

Our classication of whether a probe traversed a con-gested path concerns not only whether the probe waslost, but how long it was delayed. While an appropriate parameter appears to be dictated primarily by thevalue of p, it is not yet clear how best to set for anarbitrary path, when characteristics such as the level of statistical multiplexing or the physical path congurationare unknown. Examination of the sensitivity of and inmore complex environments is a subject for future work.

To accurately calculate end-to-end delay for inferringcongestion requires time synchronization of end hosts.While we can trivially eliminate offset, clock skew is stilla concern. New on-line synchronization techniques such

3Note that estimators that average individual estimations of the duration of each loss episode are not likely to perform that well at low values of p.

as reported in [32] or even off line methods such as [33]could be used effectively to address this issue.

IX. S UMMARY , CONCLUSIONS ANDFUTURE W ORK

The purpose of our study was to understand how to measureend-to-end packet loss characteristics accurately with probesand in a way that enables us to specify the impact on thebottleneck queue. We began by evaluating the capabilities of simple Poisson-modulated probing in a controlled laboratoryenvironment consisting of commodity end hosts and IP routers.We consider this testbed ideal for loss measurement tool eval-uation since it enables repeatability, establishment of groundtruth, and a range of trafc conditions under which to subjectthe tool. Our initial tests indicate that simple Poisson probingis relatively ineffective at measuring loss episode frequency ormeasuring loss episode duration, especially when subjected toTCP (reactive) cross trafc.

These experimental results led to our development of ageometrically distributed probe process that provides more

accurate estimation of loss characteristics than simple Poissonprobing. The experimental design is constructed in such away that the performance of the accompanying estimatorsrelies on the total number of probes that are sent, but noton their sending rate. Moreover, simple techniques that allowusers to validate the measurement output are introduced. Weimplemented this method in a new tool, B ADABING , whichwe tested in our laboratory. Our tests demonstrate that B AD -ABING , in most cases, accurately estimates loss frequenciesand durations over a range of cross trafc conditions. For thesame overall packet rate, our results show that B ADABING issignicantly more accurate than Poisson probing for measur-ing loss episode characteristics.

While B ADABING enables superior accuracy and a betterunderstanding of link impact versus timeliness of measure-ment, there is still room for improvement. First, we intendto investigate why p=0.1 does not appear to work well evenas N increases. Second, we plan to examine the issue of appropriate parameterization of B ADABING , including packetsizes and the and parameters, over a range of realisticoperational settings including more complex multihop paths.Finally, we have considered adding adaptivity to our probeprocess model in a limited sense. We are also consideringalternative, parametric methods for inferring loss character-istics from our probe process. Another task is to estimatethe variability of the estimates of congestion frequency and

duration themselves directly from the measured data, undera minimal set of statistical assumptions on the congestionprocess.

ACKNOWLEDGMENTS

We thank the anonymous reviewers for their constructivecomments. This work is supported in part by NSF grantnumbers CNS-0347252, ANI-0335234, and CCR-0325653and by Cisco Systems. Any opinions, ndings, conclusions orrecommendations expressed in this material are those of theauthors and do not necessarily reect the views of the NSF orof Cisco Systems.


14/14

14

REFERENCES

[1] W. Leland, M. Taqqu, W. Willinger, and D. Wilson, On the self-similarnature of Ethernet trafc (extended version), IEEE/ACM Transactionson Networking , pp. 2:115, 1994.

[2] V. Paxson, Strategies for sound Internet measurement, in Proceedingsof ACM SIGCOMM Internet Measurement Conference 04 , Taormina,Italy, November 2004.

[3] R. Wolff, Poisson arrivals see time averages, Operations Research ,vol. 30(2), March-April 1982.

[4] G. Almes, S. Kalidindi, and M. Zekauskas, A one way packet lossmetric for IPPM, IETF RFC 2680, September 1999.

[5] Y. Zhang, N. Dufeld, V. Paxson, and S. Shenker, On the constancy of Internet path properties, in Proceedings of ACM SIGCOMM Internet Measurement Workshop 01 , San Francisco, November 2001.

[6] J. Bolot, End-to-end packet delay and loss behavior in the Internet, inProceedings of ACM SIGCOMM 93 , San Francisco, September 1993.

[7] V. Paxson, End-to-end Internet packet dynamics, in Proceedings of ACM SIGCOMM 97 , Cannes, France, September 1997.

[8] M. Yajnik, S. Moon, J. Kurose, and D. Towsley, Measurement andmodeling of temporal dependence in packet loss, in Proceedings of IEEE INFOCOM 99 , New York, NY, March 1999.

[9] D. Papagiannaki, R. Cruz, and C. Diot, Network performance monitor-ing at small time scales, in Proceedings of ACM SIGCOMM Internet Measurement Conference 03 , Miami, FL, October 2003.

[10] P. Barford and J. Sommers, Comparing probe- and router-based packet

loss measurements, IEEE Internet Computing , September/October2004.[11] S. Brumelle, On the relationship between customer and time averages

in queues, Journal of Applied Probability , vol. 8, 1971.[12] F. Baccelli, S. Machiraju, D. Veitch, and J. Bolot, The role of PASTA in

network measurement, in Proceedings of ACM SIGCOMM , Pisa, Italy,September 2006.

[13] S. Alouf, P. Nain, and D. Towsley, Inferring network characteristicsvia moment-based estimators, in Proceedings of IEEE INFOCOM 01 ,Anchorage, Alaska, April 2001.

[14] K. Salamatian, B. Baynat, and T. Bugnazet, Cross trafc estimationby loss process analysis, in Proceedings of ITC Specialist Seminar on Internet Trafc Engineering and Trafc Management , Wurzburg,Germany, July 2003.

[15] Merit Internet Performance Measurement and Analysis Project,http://nic.merit.edu/ipma/, 1998.

[16] Internet Protocol Performance Metrics,http://www.advanced.org/IPPM/index.html, 1998.

[17] A. Adams, J. Mahdavi, M. Mathis, and V. Paxson, Creating a scalablearchitecture for Internet measurement, IEEE Network , 1998.

[18] J. Mahdavi, V. Paxson, A. Adams, and M. Mathis, Creating a scalablearchitecture for Internet measurement, in Proceedings of INET 98 ,Geneva, Switzerland, July 1998.

[19] S. Savage, Sting: A tool for measuring one way packet loss, inProceedings of IEEE INFOCOM 00 , Tel Aviv, Israel, April 2000.

[20] M. Allman, W. Eddy, and S. Ostermann, Estimating loss rates withTCP, ACM Performance Evaluation Review , vol. 31, no. 3, December2003.

[21] P. Benko and A. Veres, A passive method for estimating end-to-endTCP packet loss, in Proceedings of IEEE Globecom 02 , Taipei, Taiwan,November 2002.

[22] M. Coates and R. Nowak, Network loss inference using unicast end-to-end measurement, in Proceedings of ITC Conference on IP Trafc, Measurement and Modeling , September 2000.

[23] N. Dufeld, F. Lo Presti, V. Paxson, and D. Towsley, Inferring link lossusing striped unicast probes, in Proceedings of IEEE INFOCOM 01 ,Anchorage, Alaska, April 2001.

[24] G. Appenzeller, I. Keslassy, and N. McKeown, Sizing router buffers,in Proceedings of ACM SIGCOMM 04 , Portland, OR, 2004.

[25] C. Villamizar and C. Song, High Performance TCP in ASNET,Computer Communications Review , vol. 25(4), December 1994.

[26] C. Fraleigh, C. Diot, B. Lyles, S. Moon, P. Owezarski, D. Papagiannaki,and F. Tobagi, Design and deployment of a passive monitoring infras-tructure, in Proceedings of Passive and Active Measurement Workshop ,Amsterdam, Holland, April 2001.

[27] NLANR Passive Measurement and Analysis (PMA),http://pma.nlanr.net/, 2005.

[28] S. Floyd and V. Paxson, Difculties in simulating the Internet, IEEE/ACM Transactions on Networking , vol. 9, no. 4, 2001.

[29] J. Sommers and P. Barford, Self-conguring network trafc gen-eration, in Proceedings of ACM SIGCOMM Internet Measurement Conference 04 , 2004.

[30] A. Tirumala, F. Qin, J. Dugan, J. Ferguson, and K. Gibbs,Iperf 1.7.0 the TCP/UDP bandwidth measurement tool,http://dast.nlanr.net/Projects/Iperf, 2007.

[31] M. Schervish, Theory of Statistics . New York: Springer, 1995.[32] A. Pasztor and D. Veitch, PC based Precision timing without GPS, in

Proceedings of ACM SIGMETRICS , Marina Del Ray, CA, June 2002.[33] L. Zhang, Z. Liu, and C. Xia, Clock Synchronization Algorithms for

Network Measurements, in Proceedings of IEEE Infocom , New York,NY, June 2002.

Joel Sommers received BS degrees in Mathematicsand Computer Science from Atlantic Union Collegein 1995, and a MS degree in Computer Sciencefrom Worcester Polytechnic Institute in 1997. He iscurrently a Ph.D. candidate in Computer Science atthe University of Wisconsin at Madison, where hehas been since 2001.

Paul Barford received his BS in electrical engineer-ing from the University of Illinois at Champaign-Urbana in 1985, and his Ph.D. in Computer Sciencefrom Boston University in December, 2000. He isan assistant professor of computer science at theUniversity of Wisconsin at Madison. He is thefounder and director of the Wisconsin AdvancedInternet Laboratory, and his research interests are inthe measurement, analysis and security of wide areanetworked systems and network protocols.

Nick Dufeld is a Senior Technical Consultant inthe Network Management and Performance Depart-ment at AT&T Labs-Research, Florham Park, NewJersey, where he has been since 1995. He previouslyheld postdoctoral and faculty positions in Dublin,Ireland and Heidelberg, Germany. He received aPh.D. from the University of London, UK, in 1987.His current research focuses on measurement andinference of network trafc. He was charter Chairof the IETF working group on Packet Sampling. Heis a co-inventor of the Smart Sampling technologies

that lie at the heart of AT&Ts scalable Trafc Analysis Service.

Amos Ron received his Ph.D. in Mathematics fromTel-Aviv University in 1988. He is currently a Pro-fessor of Computer Science and Mathematics and aVilas Associate at the University of Wisconsin. Hismain research area is Approximation Theory, andhe serves as the editor-in-chief of Journal of Ap-proximation Theory. Other research interests includedata representation (wavelets and Gabor), convexgeometry, and applications in areas like Internetmeasurements, NMR spectroscopy, medical MRI,and cell biology.
http://nic.merit.edu/ipma/http://www.advanced.org/IPPM/index.htmlhttp://pma.nlanr.net/http://dast.nlanr.net/Projects/Iperfhttp://dast.nlanr.net/Projects/Iperfhttp://pma.nlanr.net/http://www.advanced.org/IPPM/index.htmlhttp://nic.merit.edu/ipma/

geometric approach to active packet loss measurement

Documents