impact of le arrivals and departures on buffer

Impact of file arrivals and departures on buffersizing in core routers

Ashvin LakshmikanthaDepartment of Electrical and

Computer Engineeringand

Coordinated Science LaboratoryUniversity of Illinois,Urbana-Champaign

Email: [email protected]

R SrikantDepartment of Electrical and

Computer Engineeringand

Coordinated Science LaboratoryUniversity of Illinois,

Urbana-ChampaignEmail: [email protected]

Carolyn BeckDepartment of Industrial

and Enterprise Systems Engineeringand

Coordinated Science LaboratoryUniversity of Illinois,Urbana-Champaign

Email: [email protected]

Abstract—Traditionally, it had been assumed that the efficiencyrequirements of TCP dictate that the buffer size at the routermust be of the order of the bandwidth (C)-delay (RTT ) product.Recently this assumption was questioned in a number of papersand the rule was shown to be conservative for certain trafficmodels. In particular, by appealing to statistical multiplexingit was shown that on a router with N long-lived connections,buffers of size O(C×RTT√

N) or even O(1) are sufficient. In this

paper, we reexamine the buffer size requirements of core routerswhen flows arrive and depart. Our conclusion is as follows: ifthe core to access speed ratio is large, then O(1) buffers aresufficient at the core routers; otherwise, larger buffer sizes doimprove the flow-level performance of the users. From a modelingpoint of view, our analysis offers two new insights. First, it maynot be appropriate to derive buffer-sizing rules by studying anetwork with a fixed number of users. In fact, depending uponthe core-to-access speed ratio, the buffer size itself may affect thenumber of flows in the system, so these two parameters (buffersize and number of flows in the system) should not be treated asindependent quantities. Second, in the regime where the core-to-access speed ratio is large, we note that the O(1) buffer sizes aresufficient for good performance and that no loss of utilizationresults, as previously believed.

I. I NTRODUCTION

Traditionally, a buffer size ofC × RTT was considerednecessary to maintain high utilization (hereC denotes thecapacity of the router andRTT is the round trip time) forTCP type sources [21]. This buffer sizing rule implies that ifthere areN persistent connections, each requiring a throughputof c (C = Nc), then the buffer size should beNc × RTT orin other words the buffers should be scaled linearly with thenumber of flows, i.e.,O(N) or O(C). This traditional viewof buffer sizing was questioned in [2], [13], [22], [20], [11]and was shown to be outdated. By appealing tostatisticalmultiplexing, it was shown that buffer sizes that are scaledas O(

√N) or O( C√

N) are sufficient to maintain high link

utilization. Another extension to the above work shows thatbuffer sizes can be reduced to evenO(1) by smoothing thearrival process to the core [13]. In other words, accordingto [13] buffer sizes can be chosen independent of the linkcapacity and RTT, as long as the network operator is willing

to sacrifice some link utilization. In particular, it was shownthat a buffer size of about20 packets are largely sufficient tomaintain nearly80% link utilization (independent of the corerouter capacity).

All of the above results were obtained under the assumptionthat there areN long-lived flows in the network. The numberof long-lived flows in the network was not allowed to vary withtime. In reality, flows arrive and depart making the number offlows in the network random and time varying. The questionwe ask in this paper is the following: “Can the buffers onthe core routers be significantly reduced even when there areflow arrivals and departures, without compromising networkperformance?”.

The performance metric that we use to study the impactof buffer sizing on end-user performance is the average flowcompletion time (AFCT). When there are file arrivals anddepartures, AFCT is a better metric to use than link utilization,which is the commonly used metric when there are fixednumber of flows. For example, in an M/G/1 queue, smallchanges in traffic intensity can lead to large changes in meandelays (AFCTs) when the traffic intensity is close to 1. To seethis, note that the mean delay is proportional to

(

ρβ−ρ

)

, whereρ is the offered traffic andβ is the effective link capacity[5].In the context of TCP-type flows, the effective capacity isdetermined by the extent to which TCP can utilize the linkwith a given buffer size. Supposeρ = 0.95. Then a change inβ from 1 to 0.96 can increases the AFCT by a factor of5.

A. Main contributions

Our main contributions can be summarized as follows• We first study the impact of flow arrivals and departures

in networks that have been the motivation of recent buffersizing results. Such networks are characterized by a vastdisparity in the operating speeds of access routers and thecore routers (roughly three to four orders of magnitude).When there are flow arrivals and departures, we showthat the core routersare rarely congested even at highloads of98%. Since there is no congestion on the corerouter, the flows are largely limited by the access speeds.

Thus, the AFCT seen by an end user does not changesignificantly with the core router buffer size. While wearrive at our results from different considerations, ourresults agree with [13] in that, on such networks, the corerouter buffers should be scaled asO(log(Ca×RTT ))(In[13], the authors study a single link withN long-livedflows. They assume no access speed limitations butimpose a maximum window size constraint on TCP. Theirresult is that buffers on the core router should be scaledas O(log(Wmax)), whereWmax denotes the maximumwindow size of TCP. It is easy to see that our result isequivalent to this result) whereCa denotes the capacityof the access router. However, unlike [13], we further findthat we do not have to sacrifice link utilization to allowsuch small buffer sizes.

• We study the impact of small buffers on a single con-gested link where the access limitations are absent. Itis rather well known that TCP approximates processorsharing [15], when the file-sizes are large. Therefore, atany time, very few active flows are present in the networkeven at significantly high loads (for example, underprocessor sharing, even at90% loading, the probabilitythat more than50 flows are active is about0.005. ).Therefore, the assumption that a large number of usersexist in the system does not hold. Thus, large reductionsin buffer sizes due to statistical multiplexing effectsreported in previous work do not apply here. In fact,reducing buffer-sizes in these networks would result indramatic degradation in the overall performance. We haveobserved anorder of magnitude increase in the AFCT dueto the use of small buffers on such links. It turns out thatone would require buffers of sizeC ×RTT, or O(C) toobtain good end-user performance.

• All of the above conclusions can be obtained from asingle unifying model which is applicable to a large classof traffic scenarios. In particular, we argue that, givena particular access to core router ratio, there exists athreshold operating load below which small buffers seemsto be sufficient. Above this threshold, one would requirebuffers of sizeO(C × RTT ).

It is important to note that the results based on fixednumber of flows do not consider access speed limitations andmaximum window size limitations are removed, thus givingthe impression that the results are valid even if the core toaccess speed ratio is small. In other words, a model thatassumes a fixed number of flows would indicate that buffersizes can be reduced to1% of the bandwidth delay productif N = 10, 000, independent of the maximum window sizelimitations and access speed limitations. However, our resultsindicate that when access speed limitations and maximumwindow size limitations are removed, the number of activeflows in the system will be quite small (typically in tens).Since there are very few flows in the network, one cannotpresent an argument based on statistical multiplexing to reducebuffer size. If we were to design for the typical case in such

networks, we would require very large buffers to ensure goodperformance to the end users. This subtlety can be noticed onlywhen we study the system with file arrivals and departures.

We briefly comment on the similarities and differencesbetween our work and the results in [10]. In [10], it wasshown that on routers without any access speed limitations,O(C × RTT ) buffering is required when there are file ar-rivals and departures. The authors also suggest that back-bonerouters are lightly loaded due to over-provisioning, and thusone would require small buffers on such routers. While wealso note thatO(C × RTT ) buffering is required when thereare no access speed limitations, our approach and conclusionsare different from [10] in many respects. We show that, evenif the core router is heavily loaded (up to98%), we can stilloperate with very small buffers if the core to access speedratio is small. Further, we have developed a single unifyingmodel and also point out the key fact that the buffer size andthe typical number of users in the system are not independentquantities. This last observation seems to be the fundamentalreason why one should consider flow arrivals and departuresin sizing router buffers.

II. CORE ROUTERS IN ACCESS LIMITED NETWORKS

REQUIREO(1) BUFFERING

In this section, we will study networks where the core routerspeeds are several orders of magnitude larger than the accessrouter speeds. Before we model arrivals and departures, wefirst derive buffer requirements for a network with a fixednumber of long-lived flows, while using link utilization as theperformance metric. Using AFCT as the performance metric,we then consider file arrivals and departures and show that thesmall buffers do not increase the AFCT significantly, unlessthe traffic load is close to the instability region of the network.The reason is that, with access speed limitations, the corerouter is not congested (i.e., the number of packets droppedat the core is an order of magnitude smaller than the numberdropped at the access routers) unless the offered load is veryclose to the instability region of the network. Since the corerouter does not get congested, the core router buffer size hasno significant affect on the AFCT of the flows.

In prior literature, the congestion at the core is oftenmeasured using link utilization which we believe is incorrect.Our model indicates that even at very high levels of linkutilization, the core is not congested in the sense that corerouters will not be able to control the transmission rates ofthe end users. This is due to the fact that packet drops are soinfrequent on core routers that they contribute very littletothe overall packet loss probability.

DestinationSource

Ca C

Fig. 1. Access limited networks

Let us first consider a fixed number of flowsN accessing the

Internet via an access router and a core router. The capacityofthe access router isCa packets/sec and the core router capacityis C packets/sec. Letβ(B,N,K) denote thecore router linkutilization, which is a function of the buffer sizeB, numberof flows N and the core to access speed ratioK = C

Ca. Let

γC denote the mean packet arrival rate at the core router andpc denote the mean packet loss probability at the core router.It is straightforward to show that

β = γ(1 − pc). (1)

Assume thatNCa ≤ γC. In this case, by our assumption thatCa ≪ C, N can be fairly large. WhenN is large, by standardresults in stochastic processes, the arrival rate can be wellapproximated by a Poisson process [6]. Further assuming anM/M/1/B model for the queueing process at the core router,whereB is the buffer size at the core, we can compute thepacket loss probability at the core router to be [5]

pc = γB 1 − γ

1 − γB+1. (2)

This formula can be used to size the buffer i.e., to obtain anupper bound onB. To do this, we first need a specificationof the desiredpc. Due to the fact thatNCa ≤ γC, it isclear that the network is access speed limited and thereforewe should design the core buffer size such that it does notinduce significant packet loss compared to the access router.This is due to the fact that TCP throughput is approximatelygiven by,

X =

√1.5

RTT√

pa + pc

, (3)

whereX denotes TCP throughput andpa denotes the packetloss probability on the access router [19]. Here, we have usedthe approximation1 − (1 − pa)(1 − pc) ≈ pa + pc. Supposewe design the buffer size such thatpc = 0.1pa and the bufferson the access link are sized such that the access link is fullyutilized, then we get

Ca =1

RTT

√

1.5

pa + pc

=1

RTT

√

1.5

11pc

.

Substituting forpc from the above formula in (2), we get thedesired buffer size to beO(log 1

γ(Ca × RTT )). To illustrate

the importance of the above result, we consider the followingexample.

Example 1: Consider a core router which is accessed viaaccess routers of capacity2 Mbps. Let the packet size be1000bytes. If theRTT = 50ms, then to achieve a transmission rateof 2 Mbps, using (3), the loss probability on the access router(pa) can be no more than0.01. To ensure that the core routerdoes not affect the throughput of a flow, we choose the buffersize on the core router such thatpc = 0.1pa = 10−3. Theamount of buffering required to achieve a certain loss probabil-ity is given in Figure 2. Figure 2 is plotted using (2). Supposewe require90% link utilization (i.e., β(B,N,K) ≈ γ = 0.9and NCa < 0.9C ). From Fig. 2, it is clear that no morethan40 packets are required to maintain a loss probability ofpc = 10−3. Even at95% link utilization, we need no more

than about80 packets of buffering at the core router. The mostimportant thing to note is that this result is independent ofthecore router capacity.

0 10 20 30 40 50 60 70 80 90 10010

−12

10−10

10−8

10−6

10−4

10−2

100

Buffer Size (pkts)

Pac

ket L

oss

Pro

babi

lity

β = 0.95β=0.90β=0.80

Fig. 2. Packet Loss probability on an uncongested link

Thus, even at very high utilization levels at the core router,small buffers seem to be sufficient. Next, we consider the casewherepc ≫ pa. This situation would arise ifNCa ≫ C (whenNCa ≈ C, it there would be drops both on the access routerand on the core router. This scenario is particularly trickytohandle and we do not do so in this paper. As our simulationsdemonstrate, such a detailed model is not required for thecalculation of AFCT.). In this case, the per-flow throughputisapproximately given by

X ≈√

1.5

RTT√

pc.(4)

As before, sinceN is large, we can model the packet arrivalprocess at the core router by a Poisson process. Therefore, thepacket loss probability on the core router is still given by (2).If the per-flow throughput isX, then

γ =NX

C. (5)

Given a buffer sizeB, (1), (2), (4) and (5) reduce to a set offixed point equations. These equations can be solved to obtainβ.

Example 2: Consider a congested core of capacityC =100Mbps accessed via access routers of capacityCa =2Mbps. Let the packet size be1000 bytes and the RTT be50ms. The bandwidth delay productC × RTT = 625 packets.Suppose there are60 flows in the system. In this case, the corerouter becomes the bottleneck and hence it is the main sourceof congestion feedback. The amount of buffering required toobtain a certain link utilizationβ, is given in Fig. 3. Thishas been obtained by solving the set of fixed point equationsas described above. As seen from Fig. 3, even if we wereto operate at95% efficiency, we require no more than40packets buffering at the core router. As we increase the corerouter capacity, the amount of buffering required to maintainthe same efficiency increases but not by a significant amount.

When the link capacity of both the core router and the accessrouter is increased by a factor of100, the increase in the buffersize required is only a factor of5 (the increase in buffer sizeappears to be logarthmic in the core router capacityC).

0.6 0.7 0.8 0.9 110

1

102

103

β

Buf

fer

Siz

e (p

kts)

C× RTT = 625

C× RTT = 6250

C× RTT = 62500

Fig. 3. Amount of buffering required to maintain target link utilization

We have now provided a simple derivation of buffer-sizerequirements for a network where the number of long-livedflows is fixed, using link-utilization as the metric. However,as mentioned in Section I, when one considers arrivals anddepartures of flows, link utilization is not the appropriatemetric. We now study the impact of small buffers on the AFCTof flows, when flows arrive and depart.

Consider the following model of an access limited networkwhich is similar to the model in [9]. Suppose that files arriveaccording to a Poisson process of rateλ and the file sizesare from an arbitrary distribution with mean1

µ. Recalling the

definition of the core link utilizationβ(B,N,K), we notethat whenever there areN users in the system, the per-flowthroughput is β(B,N,K)C

N. Due to the insensitivity property

of our model to file-size distribution, N(t) can be representedas a Markov chain, specifically a birth-death process [17]. Itfollows from elementary analysis of birth-death processesthatλ

µC< limn→∞ β(B,n,K) is necessary for the Markov chain

to be stable (i.e., positive recurrent). Under this assumption,the exact stationary distribution of this Markov chain canbe characterized. For details on calculating the stationarydistribution we refer the reader to [17, Theorem 3.8].

The stationary distribution of the Markov chain is given by,

πi = Prob{N = i} =ρi∏i

j=01βj

∑∞k=0 ρk

∏kj=0

1βj

, (6)

whereβj = β(B, j,K) andρ = λµC

.Since the exact stationary distribution is known, AFCT

can be easily characterized. By Little’s law it follows thatAFCT = E[N ]

λand therefore,

AFCT =

∑∞i=1 iπi

λ.

While the above expression provides a closed-form solutionfor the AFCT, it provides very little intuition on the depen-dence of AFCT on the buffer size at the core router. To get

more insight, we rewrite the AFCT as follows:

AFCT =1

λ

γK∑

i=0

iπi +∞∑

γK+1

iπi

(7)

Recall that in our design we have assumed that the router iscongested ifN ≥ γK + 1. If πi is small ati = γK + 1 anddecreases exponentially fori > γK+1, then it is clear that thesystem rarely experiences congestion and therefore the AFCTis predominantly determined by the access router speed.

If K is sufficiently large (K should be large enough toremove synchronization effects as studied in [2]), thenβ is anincreasing function ofN is the region[γK,∞]. From (6), itfollows that if i > γK then

Prob(N = i) ≤ρi∏γK

j=01βi

(

1βγK+1

)i−(γK)

∑∞k=0 ρk

∏kj=0

1βj

= Prob(N = γK)

(

ρ

βγK+1

)i−γK

,

which means thatProb(N = i) decreases at least exponen-tially for i > γK. Therefore, ifProb(N = γK) is sufficientlysmall (and consequentlyProb(N > γK) is also very small),then

AFCT =1

µCa

.

The system is rarely in congestion and therefore the AFCTis dictated by the access speed limitations. In the followingexample, we verify that even at very high loads,Prob(N >γK) is quite small.

To evaluateProb{N > γK} using (6) we need the valueof β(B,N,K) for all values ofN. Note that

β(B,N,K) = NCa

Cif NCa ≤ γC,

where B is chosen to achieve a link utilization ofγ. Onthe other hand, ifNCa > γC, and K is sufficiently large,as described earlier,β(B,N,K) is an increasing function ofN in the region[γK,∞). Therefore, we can determine anupper-bound onProb{NCa > γC} (i.e., Prob{N > γK})by replacingβ(B,N,K) by β(B, γK,K) for all values ofN > γK. Based on this approximation, we present somenumerical results in the following example.

Example 3: Consider again a core router with capacityC = 100 Mbps being accessed by flows via an access routerwith capacityCa = 1Mbps. Therefore,K = 100. The totalpropagation delay is assumed to be50ms. If the mean packetsize is1000 bytes, it is clear that we requireC ×RTT = 625packets to achieve100% link utilization. On the other hand, toachieve95% link utilization, we need only about40 packetsof buffering.

The fraction of time the system is congested is plotted asa function of the offered load in Fig. 4. As seen from Fig.4 , the core router is congested less than10% of the timeeven at85% load. As the ratioK increases, the amount oftime spent in congestion further decreases. For example, if

K = 500 (i.e., C = 500 Mbps andCa = 1 Mbps), even at90% load, the core router is congested less than1% of thetime! Since the system spends a very small amount of time incongestion, the AFCT should not increase significantly withsmaller buffers. Simulations presented in Section IV show thatthis is indeed the case. In other words, core router buffer sizecan be chosen independently of the core router link capacityin networks with a large disparity between access speeds andcore router speeds.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Offered Load

Con

gest

ion

even

t pro

babi

lity

K=100

K=200

K=300

K=500

Fig. 4. Fraction of time system spends in Congestion

Remark 1: In today’s Internet, there exists networks wherecore routers operate about1000 − 10000 times faster thanthe access routers. For example consider a DSL user basewith each user accessing the network with a1 Mbps accessbandwidth and the aggregation point switching the packets at10 Gbps (K = 10000). Further suppose that the RTT of flowsis about50 ms. The traditional buffer provisioning guidelinessuggest that to achieve100% link utilization a buffer of sizeB = C × RTT = 62.5 MB is required. On the other hand,our analysis indicates that we can achieve a link utilization ofβ = 0.999 using a buffer of size250KB.

Assuming a buffer size of250KB, the probability that thecore router is congested at an extremely high load of98% isgiven by,

Prob{N > 9990} ≈ 0.007!

Thus, in such a network, even at98% load, there is verylittle congestion on the core router. This implies that theAFCT does not increase when small buffers are used, evenwhen the system is operating at98% load. Therefore buffersizes can be reduced dramatically, without degrading end-userperformance.

Our results in this section depend on the fact that therelevant performance metric is AFCT. As mentioned earlier,the AFCT is insensitive to the file-size distribution, thus theresults apply to heavy-tailed file-size distributions as well.However, heavy-tailed file-size distributions may affect thetime required by the system to reach its steady-state. Inparticular, if the network enters a heavily congested state, thenit may persist in this transient state for a long period of time.

Nevertheless, we note that Example 2 suggests that even whenthe core is heavily congested, small buffers may be sufficient.Defining a performance metric and studying the system inmore detail under such worst-case transient scenarios is anarea for future research that we do not undertake in this paper.

III. C ORE ROUTERS WITHO(C) BUFFERING

When there is a large disparity between the access speedsand the core router speeds, buffering is required primarilyto reduce the small variance of a Poisson process and thesystem is rarely in congestion. Therefore, small buffers aresufficient in these networks. However, as a network designer,it is important to study the impact of small buffers in networksthat get congested very often. Typically, these are the networkswhere there are no access speed limitations and each flow canpotentially use a large fraction of the capacity of the link.

In this section we study the impact of small buffers innetworks without access speed limitations. Our approach, asbefore, is based on time-scale separation. We first constructa detailed packet-level model of a single congested linkassumingN long-lived flows access the link. Using this modelwe characterize the link-utilizationβ(N,B) as a function ofthe buffer sizeB and the number of long-lived flowsN.Unlike in the previous section, note that the parameter betadoes not depend on K, the core to access speed ratio, sincein this section, there are no access speed limitations. Wethen study the dynamics of flow level arrivals and departures.The only parameter that we require from our packet-levelmodels to carry out flow level analysis is the link-utilizationβ(N,B), since we neglect the impact of other TCP dynamicsand packet-level dynamics such slow-start, fast retransmission,time outs, etc. A more detailed model may increase the nu-merical accuracy of our results, but we would lose our abilityto obtain qualitative insight into the congestion phenomenonat the core routers. As our simulations demonstrate, our simplemodel is quite accurate in predicting the AFCT of flows.

We consider a single-link of capacityC packets/sec ac-cessed byN long-lived flows. The round trip time of flowi is denoted byRTTi. In our earlier model of access-limitednetworks, we did not explicitly model the RTT, since we coulduse very small buffers at the core. The impact of RTT on TCPthroughput was irrelevant as we had designed the system sothat the throughput of TCP was roughly equal to the accessspeed. Now, since the access speed limitations are no longerpresent, the queueing delay at the core router affects the overallthroughput of a flow. Therefore, we explicitly break up theRTT into propagation delayτp and queueing delayτq. Weassume that the propagation delayτ i

p of a useri is uniformlydistributed between[a b]. The maximum window size of TCPis denoted byMWS. The packet loss probability at the corerouter is denoted bypc as before.

The average rate at which flowi will transmit data is givenby,

xi =1

RTTi

min

(√

1.5

bp,MWS

)

, (8)

whereb denotes the number of packets acknowledged per TCPack [19]. In the current TCP implementations,b is either 1or 2. In our analysis and in our simulations, we assume thatb = 1. Let τq denote the average queueing delay seen by eachuser. Therefore,

RTTi = τ ip + τq.

Taking expectations over all users, we find that the averagerate of transmission is

x̄ = E[xir] =

1

b − amin

(√

1.5

bp,MWS

)∫ b

a

dx

x + τq

=1

b − alog

(

b + τq

a + τq

)

min

(√

1.5

bp,MWS

)

.

The average packet arrival rate at the core router is

λc = Nx̄. (9)

To complete the setup, we require a model for packet lossprobability pc as a function of the arrival process. We presentsuch a model here. In Section II, we had assumed that thearrival process to the core router is Poisson. This is a validassumption when the access speeds are very small comparedto the core router speeds and a large number of flows arerequired to congest the core router. At very high access speeds,it takes only a few flows to cause congestion on the core router.Therefore the packet arrival process at the core router tendsto be bursty and one cannot use a Poisson approximation thatwas used earlier. In this case, we have to model the packetarrival process using a stochastic process with a larger inter-arrival time variance. We use a diffusion approximation tostudy the resulting queueing process. Let the load on the corerouter queue beρc = λc

C. Further, we denote the SCV (squared

coefficient of variance, i.e., variance divided by the square ofthe mean) of the inter-arrival times of the arrival process byc2a. Then, according to [4], the loss probability is given by

pc =θeθB

eθB − 1

1 +c2

a

ρ

2

, (10)

whereθ = 2(ρ−1)ρc2

a+1 . We use simulations to estimatec2a and use

this estimated value in (10).With small buffers, the queueing delay is negligible com-

pared to the propagation delay. As the buffer size increases,the round trip time increases due to an increase in the queueingdelay. To model this we calculate the average number ofpackets in the queue. According to [4], the average number ofpackets in the queue is given by,

q̄ =B

1 − e−θB− 1

θ.

Therefore, the average queueing delay (τq) is given by

τq =q̄

C. (11)

The set of fixed point equations (9)-(11) can be solved usingstandard fixed-point equation solvers. Then the overall linkutilization is given by

β(N,B) = ρp(1 − p).

We now study the impact of small buffers on the AFCT whenflows arrive and depart. We consider a single congested linkof capacityC being accessed by many flows. Flows arriveaccording to a Poisson process of rateλ. Each flow seeks totransfer a file which is taken from an exponential distributionwith mean 1

µ. Then the number of active flows in the system

N(t) forms a Markov chain. Note that this is the same Markovchain as in Section II, except that the expression forβ isdifferent. A similar analysis has been carried out in [15] wherethe authors justify the use of processor-sharing queues todescribe TCP flow level performance. However, in [15] it isassumed that buffers are large and consequentlyβ is chosen tobe unity. Our work can be considered a generalization, whichtakes into account the impact of buffer size.

Define the load on the system as

ρ =λ

µC.

Assuming thatρ < limN→∞ β(B,N) = β∗, the distributionof the number of flows in the system at equilibrium is givenby

Prob(N = i) =1

M

(

ρi

∏ij=1 β(i, B)

)

,

whereM is a normalization constant given by

M =∞∑

i=0

(

ρi

∏ij=1 β(i, B)

)

.

Using these equations, we can calculateNavg and AFCT(using Little’s law) as

Navg =

∞∑

i=0

i · Prob {N = i},

AFCT =Navg

λ.

We consider the following numerical example.Example 4: Consider a core router of capacity100 Mbps.

Flows use TCP for data transfer with a MWS of64. The RTTof each flow is chosen to be uniformly distributed between40 − 60 ms. The mean flow size is assumed to1.1 MB. Weassume that each packet is1000 bytes. For this problemC ×RTT = 625 packets.

As we mentioned earlier, it is difficult to determine the SCVof the arrival process analytically when there are very fewflows. Thus, we use simulations to determine the SCV of thearrival process. Our simulations indicate that as the buffer sizeincreases, the SCV varies asB0.63. In our analysis we use thefollowing empirical expression for the SCV:

c2a = B0.63.

Using the theory developed earlier in this section, we calculatethe AFCT as a function of the buffer size at various loads.These are plotted in Fig. 9. As indicated in these plots, it isclear that in networks with no access speed limitations it isimpossible to reduce buffer size without seriously degradingperformance. At a modest80% load, our analysis indicates

that the AFCT increases by nearly anorder of magnitude whensmall buffers are used! Even when the load is small, say50%,the overall AFCT doubles with the use of small buffers in thenetwork.Thus we conclude that, whenever core routers are severelycongested, it is not possible to use small buffers at the routers.In fact, we requireO(C ×RTT ) buffers in order to maintaingood performance to the end-users. Note that the fact we havean empirical value for c2

a from simulations is not a seriouslimitation of our model. The model primarily offers qualitativeinsight and allows us to compute the appropriate order forthe buffer size. In practice, precise buffer sizing rules mighthave to be perhaps obtained using simulations, but the modeloffers important insight into the physics of the congestionphenomenon.

Remark 2: Our conclusions are based on the fact that TCPis the protocol of choice even in networks where the coreto access speed ratio is smaller than today’s networks. Forexample, if the access speed were to become larger than whatTCP’s currentMWS can support, then we implicitly assumethat theMWS is increased correspondingly to support theaccess rate. However, one could argue that when access speedsincrease, we may use protocols other than TCP which are moreefficient in large access-speed regimes (Ex. RCP [12], FASTTCP [16], Scalable TCP [18], BIC-TCP [23] ). In such cases,similar analysis can be carried out as before, although thevalues ofβ should be modified to reflect the efficiency of thenew protocol. The conclusions depend on how the protocolefficiency (i.e.,β) varies with the number of flows and withthe amount of buffering.

IV. SIMULATION RESULTS

Our objective in this section is to test the accuracy of thevarious models described in Sections II and III. We haveconducted detailed packet level simulations using ns-2 [1]. Weconsider a dumb-bell shaped network topology as in Fig. 5.File transfer requests arrive according to a Poisson process.These flows access the network via the access routers, transmitdata and then leave the system once all the ack packets havebeen received by the sender. The core router capacity is variedfrom 100 Mbps-500 Mbps depending on the regime that wewould like to study. The access router capacity is varied fromto 2 Mbps-10 Mbps when we study the case with limitedaccess capacities and is set to30 Mbps when we study highaccess speed networks. The link delay at the core router is10ms. The access links have delays that are uniformly distributedbetween10 ms-20 ms. Thus, the two-way propagation delayτp is uniformly distributed between40 ms-60 ms. We fix thepacket size to be 1000 bytes.

There has been a lot of work on traffic characterization ofthe Internet [8], [7], [14]. While the exact numbers vary fromtime to time and from link to link, it is largely believed thatthe Internet traffic is heavy-tailed (see for example, [7], [8]);that is most of the files are very short and a few files tend tosend large amounts of data. Usually, about70 − 90% of theflows are short and they contribute to about10 − 30% of the

overall load. The long-flows which are about10− 30% of theflows make up for nearly70% of the overall traffic.

In our simulations, we neglect the effects of short-flows.Short-flows have very small transmission times. Since TCPstarts data transmission with a small window size, it is verylikely that short-flows do not last long enough to utilize theaccess speed capacity fully. As such, short-flows do not causecongestion either on the access router or on the core router.Therefore presence (or absence) of short-flows does not affectour buffer sizing results, since buffer sizing is based on flowsthat cause congestion on the router.

It has been suggested [3] that a bounded Pareto (b.p.)distribution can be used to capture the heavy tailed propertyof the Internet traffic. A b.p. distribution,B(c, d, α), has thefollowing c.d.f:

Prob {X < x} =c−α − x−α

c−α − d−α.

The b.p. distribution has the following property

Prob {X < x|X > y} =y−α − x−α

y−α − d−α

We refer to all flows whose size is greater than a particularvalue y as long-flows. Then the above observation indicatesthat the distribution of long-flows will also be a b.p. distribu-tion. Therefore in all our simulations, we assume that the long-flows are distributed according to a b.p. distribution. The long-

100Mbps

S

S

SR

R

R

10ms30Mbps

Avg 15ms

Fig. 5. Network Topology

flows arrive according to a Poisson process with rateλ. In allthe simulations, we assume that the long-flows are distributedaccording to a b.p. distribution with parametersα = 1.1,y = 200 KB and d = 200 MB. With these parameters, themean-flow-size is1.1 MB. By varying λ, we can change theoverall load on the system.

A. Access limited Networks

In these simulations, we study the effect of buffer sizingin networks where the access link capacity is very smallcompared to the core router capacity. We assume that TCPhas a MWS of 64 packets. This is consistent with the currentimplementations of TCP. The access link capacity is assumedto be2 Mbps. The core router can switch packets at the rateof 96 Mbps (We assume that the router can switch packetsat a raw bit rate of100 Mbps. However, each packet has a40 byte TCP header and therefore the maximum good put is10001040 ×100 = 96 Mbps.). We chose the access link buffer sizeto be13 packets, mainly to ensure that the overall utilizationof the access link is close to unity. The core router buffer size

is varied from20 packets to1000 packets. The load on thesystem is varied from0.6 to 0.8.

The results of the simulations and the corresponding theo-retical predictions are presented in Fig. 6. As seen from thefigures, our model is quite accurate and predicts the resultswith less than10% error. Further, even when the external loadis 80%, there is little degradation in throughput with smallbuffers. Why is this so? According to our model, when theaccess speeds are small, the core router will experience verylittle congestion. Therefore, very small buffers suffice. We nowneed to find out whether our reasoning is correct.

To justify our claim, we plot the packet drop probabilityat the core router and at the access router in Fig. 8. Asthe figure rightly demonstrates, losses on the core router areseveral orders of magnitude smaller than the losses on theaccess router. Since the transmission rate of TCP, is inverselyproportional to the total loss probability (i.e.,pa + pc), thepacket loss probability on the core router is too small toinfluence the transmission rate of the end-users. In otherwords, the core router is not congested.

Our theoretical analysis also indicates that core router buffersize can be chosen independently of its capacity. To verify thisclaim, the performance of the system was studied at differentcore router speeds without changing the traffic parameters orthe access speed limitations. The external load was chosen tobe0.8. The core router buffer size was varied from20 packetsto 1000 packets. The results of the simulations are presentedin Fig. 7. From the figure, it is quite clear that the flows donot suffer any performance degradation with small buffers.

Our analysis and simulations strongly suggest that in accesslimited networks core router buffer size can be reduced about100 packets, without affecting performance.

B. Networks with very fast edge routers

We study a scenario in which the edge routers do not limitthe transmission rate of the TCP. In this simulation we set theaccess speed to30 Mbps. Since current implementations ofTCP have a MWS of64 KB (which translates to64 packetsin our simulations), the maximum throughput achievable byTCP is,

xmax = 64/RTT ≈ 12.8Mbps.

Therefore, it is quite clear that in this setting, access speedlimitations do not limit TCP throughput. Furthermore, due towindow flow control mechanism, any TCP connection cannottransmit more than64 packets within a RTT. To avoid edgerouter imposing any kind of restriction on TCP, the edge routerbuffer size was set to64 packets. Similar to the previousexercise, the core router buffer size was varied from20 packetsto 1000 packets. The load on the system was varied from0.5 to 0.8. To validate our theoretical results of Section IIIusing the simulation results, we have to know how theSCVof the arrival process (c2

a) varies with the buffer size. Asdiscussed in Example 4, our simulations indicate thatc2

a varieswith the buffer sizeB roughly asB0.63. We use this valueof c2

a in our model. Using theoretical models developed in

5 5.2 5.4 5.6 5.8

6 6.2

10 100 1000

AF

CT

(se

c)

Core router Buffer Size (pkts)

Simulations

Scenario C=100Mbps, Ca = 2Mbps, Avg RTT = 50ms

ρ=0.6ρ=0.7ρ=0.8

5 5.2 5.4 5.6 5.8

6 6.2

10 100 1000

AF

CT

(se

c)


Theoretical results


ρ=0.6ρ=0.7ρ=0.8

Fig. 6. AFCT under access limited networks: Theory and Simulations

4.2

4.4

4.6

4.8

5

5.2

5.4

10 100 1000

AF

CT

(se

c)


Impact of O(1) buffering access limited networks: ρ=0.8

C=200MbpsC=500Mbps

C=1000Mbps

Fig. 7. Demonstration of O(1) buffering in access limited networks

Section III, we were able to predict the performance of thesystem for different buffer sizes. The simulation results and thetheoretical predictions are presented in Fig. 9. Our theoreticalresults match the simulation results consistently at all buffersizes and at all loads, thereby validating our theoretical model.Additionally, as seen from these results, small buffers in suchnetworks degrade the performance significantly. For examplewhen the system load is 0.8, the AFCT can be decreased bynearly 85% by increasing the buffer size from20 packets to1000 packets. Similarly, as seen in the simulations, the averagethroughput increases by about400% with the increase in thebuffer size.

V. CONCLUSIONS

In this paper, we have developed simple models to providebuffer sizing guidelines for today’s high-speed routers. Ouranalysis points out that the core-to-access speed ratio is thekey parameter which determines the buffer sizing guidelines.

0.02 0.025

0.03 0.035

0.04 0.045

0.05

10 100 1000

Pac

ket l

oss

prob


Access router


ρ=0.6ρ=0.7ρ=0.8

0

0.001

0.002

10 100 1000

Pac

ket l

oss

prob


Core router


ρ=0.6ρ=0.7ρ=0.8

Fig. 8. Packet loss probability at core and access routers

0 2 4 6 8

10 12

10 100 1000

AF

CT

(se

c)


Simulations


ρ=0.6ρ=0.7ρ=0.8

0 2 4 6 8

10 12

10 100 1000

AF

CT

(se

c)


Theoretical results


ρ=0.6ρ=0.7ρ=0.8

Fig. 9. Impact of the core router buffer size in networks with fast edgerouters: Theory and Simulations

In particular, this parameter along with the buffer size deter-mines the typical number of flows in the network. Thus, animportant message in this paper is that the number of flows andbuffer size should not be treated as independent parametersinderiving buffer sizing guidelines. Further, we also point outthat link utilization is not a good measure of congestion levelat a router. In fact, we show that even at98% utilization, thecore router may contribute very little to the overall packetlossprobability seen by a source if the core to access speed ratiois large.

VI. A CKNOWLEDGMENTS

We would like to thank Dr. Damon Wischik for his sugges-tions and comments on the earlier version of this paper.

The research reported here was supported by NSF grantsECS 04-01125 and CCF 06-34891.

REFERENCES

[1] The network simulator: NS-2. Available at http://www.isi.edu/nsnam/ns.[2] G. Appenzeller, I. Keslassy, and N. McKeown. Sizing router buffers. In

ACM/SIGCOMM, 2004.[3] N Bansal and M. Harchol-Balter. Analysis of SRPT scheduling:

Investigating unfairness. InProceedings of the 2001 ACM SIGMETRICSInternational conference on Measurement and modeling of computersystems, 2001.

[4] A. Berger and W. Whitt. Brownian motion approximations for the rate-controlled throttle and theG/G/1/C queue.Journal of Discrete-EventDynamic Systems, 2:7–60, 1992.

[5] D. Bertsekas and R. Gallager.Data Networks. Prentice Hall, EnglewoodCliffs, NJ, 1992.

[6] J. Cao and K. Ramanan. A Poisson limit for buffer overflow probabili-ties. In Proceedings of IEEE INFOCOM, June 2002.

[7] M. Crovella, M. Taqqu, and A. Bestavros.A Practical Guide to HeavyTails: Statistical Techniques for Analyzing Heavy-Tailed Distributions.Birkhauser, 1998.

[8] M.E. Crovella and A. Bestavros. Self-similarity in the World WideWeb traffic: Evidence and possible causes.IEEE/ACM transactions onNetworking, pages 835–846, 1997.

[9] A. Das and R. Srikant. Diffusion approximations for modelsofcongestion control in high-speed networks.IEEE Transactions onAutomatic Control, pages 1783–1799, October 2000.

[10] A. Dhamdhere and C. Dovrolis. Open issues in router buffer sizing.ACM/SIGCOMM Computer Communication Review, pages 87–92, Jan-uary 2006.

[11] A. Dhamdhere, H. Jiang, and Constantine Dovrolis. Buffer sizing forcongested Internet links. InProceedings of IEEE INFOCOM, March2005.

[12] N. Dukkipati and N. McKeown. Processor sharing flows in the Internet,June 2004. High Performance Networking Group Technical ReportTR04-HPNG-061604.

[13] M. Enachescu, Y. Ganjali, A. Goel, T. Roughgarden, and N. McKeown.Part III: Routers with very small buffers.ACM/SIGCOMM ComputerCommunication Review, 35(3):7, July 2005.

[14] C. Fraleigh, S. Moon, B. Lyles, C. Cotton, M. Khan, D. Moll, R. Rockell,T. Seely, and C. Diot. Packet-level traffic measurements from theSPRINT IP backbone.IEEE Network, 17(6):6–16, November-December2003.

[15] S. Ben Fredj, T. Bonald, A. Proutiere, G. Regnie, and J.W. Roberts.Statistical bandwidth sharing: a study of congestion at flowlevel. InProceedings of ACM/SIGCOMM, August 2001.

[16] C. Jin, D. X. Wei, S. H. Low, G. Buhrmaster, J. Bunn, D. H. Choe,R. L. A. Cottrell, J. C. Doyle, W. Feng, O. Martin, H. Newman,F. Paganini, S. Ravot, and S. Singh. FAST TCP: From theory toexperiments.IEEE Network, 19(1):4–11, January/February 2005.

[17] F. P. Kelly. Reversibility and Stochastic Networks. John Wiley, NewYork, NY, 1976.

[18] T. Kelly. Scalable TCP: Improving performance in highspeed wide areanetworks.Computer Communication Review, 32(2), April 2003.

[19] J. Padhye, V. Firoiu, D. Towsley, and J. Kurose. Modeling TCPthroughput: A simple model and its empirical validation.IEEE/ACMTransactions on Networking, 8(2), April 2000.

[20] G. Raina, D. Towsley, and D. Wischik. Part II: Control theory for buffersizing. ACM/SIGCOMM Computer Communication Review, pages 79–82, July 2005.

[21] C. Villamizar and C. Song. High performance TCP in ANSNET.ACMComputer Communications Review, 24(5):45–60, 1994.

[22] D. Wischik and N. McKeown. Part I: Buffer sizes for core routers.ACM/SIGCOMM Computer Communication Review, pages 75–78, July2005.

[23] L. Xu, K. Harfoush, and Injong Rhee. Binary increase congestion controlfor fast long-distance networks. InProceedings of the IEEE INFOCOM,2004.

impact of le arrivals and departures on buffer

Education