flow assignment and packet scheduling for multipath routing

230 JOURNAL OF COMMUNICATIONS AND NETWORKS, VOL. 5, NO. 3, SEPTEMBER 2003

Flow Assignment and Packet Scheduling forMultipath Routing

Ka-Cheong Leung and Victor O. K. Li

Abstract: In this paper, we propose a framework to study howto route packets efficiently in multipath communication networks.Two traffic congestion control techniques, namely, flow assignmentand packet scheduling, have been investigated. The flow assign-ment mechanism defines an optimal splitting of data traffic onmultiple disjoint paths. The resequencing delay and the usageof the resequencing buffer can be reduced significantly by prop-erly scheduling the sending order of all packets, say, according totheir expected arrival times at the destination. To illustrate ourmodel, and without loss of generality, Gaussian distributed end-to-end path delays are used. Our analytical results show that the tech-niques are very effective in reducing the average end-to-end pathdelay, the average packet resequencing delay, and the average rese-quencing buffer occupancy for various path configurations. Thesepromising results can form a basis for designing future adaptivemultipath protocols.

Index Terms: Computer communications, dispersity routing,flow assignment, high speed networks, multipath routing, packetscheduling, performance modeling, resequencing, traffic disper-sion, traffic engineering.

I. INTRODUCTION

Multipath routing or spatial traffic dispersion [1]–[6] is a loadbalancing technique in which the total load from a source to adestination is spatially distributed over several paths. It is usefulfor relieving congestion and delivering quality of service (QOS)guarantees in communication networks. Generally speaking,congestion describes the network state where the number ofpackets transmitted through a network approaches or exceedsthe network packet-handling capacity. The idea of congestioncontrol is to keep the number of packets within the network be-low the level at which network performance degrades dramati-cally [7].

The convergence of the computer, communications, entertain-ment, and consumer electronics industry is driving an explosivegrowth in multimedia applications [8]. Recent studies show thatmultimedia traffic exhibits variability or correlation on varioustime scales [9], [10]. Such long-range dependence property hasa considerably unpleasant impact on queueing performance, and

Manuscript received January 11, 2002; approved for publication by JaiyongLee, Division III Editor, July 16, 2003.

K.-C. Leung is with the Department of Computer Science, Texas Tech Uni-versity, Lubbock, TX 79409-3104, USA, email: [email protected].

V. O. K. Li is with the Department of Electrical and Electronic Engi-neering, the University of Hong Kong, Pokfulam Road, Hong Kong, China,email: [email protected].

This research was supported in part by the Areas of Excellence Scheme es-tablished under the University Grants Committee of the Hong Kong SpecialAdministrative Region, China (Project No. AoE/E-01/99). An earlier versionof this paper appeared in the Proceedings of the IEEE GLOBECOM 1999.

is a dominant characteristic for a number of packet traffic engi-neering problems [11].

Nowadays, the standard routing approach in communicationnetworks consists of finding a single shortest path from a sourceto a destination based on some heuristic link cost metrics, whichare updated periodically. Although such unipath routing pro-tocols can adapt very quickly to changing network conditions,they become unstable under heavy loads or bursty traffic as thelink cost metrics used in the routing algorithms are related todelays or congestion experienced over the network links [12],[14].

Multipath routing has recently been found to be an effectivemethod to alleviate the adverse effects of traffic bursts [3]. Inaddition, multipath routing protocols help to spread out conges-tion and thus minimize network delays [14]. The key to multi-path routing is how to allocate a proper portion of traffic to eachparticipating path so as to satisfy the desired objective, such asthe minimization of the average end-to-end path delay. This isknown as flow assignment or optimal routing [12]–[14]. Mostof the existing work considered the traffic arrival rate at everylink in the network, and found an optimal routing which directstraffic exclusively on least-cost paths with respect to some linkcosts that depend on the flows carried by the links. It is compu-tationally expensive to find an optimal flow assignment for eachsource-destination pair. The solution is generally not scalable interms of the size of the network considered.

Moreover, packets which travel along different paths may ar-rive out of order. Those packets arriving out of order may haveto be resequenced, i.e., stored in a buffer, called a resequenc-ing buffer, until they can be delivered to the end process inthe proper order. Broadly speaking, there are two different ap-proaches to recover the order at the receiver. The first and themost popular way to do so is to make use of or insert variousvariants of sequence numbers into packets [3], [15]. Anothermethod is to utilize synchronization markers and some prop-erties of routing protocols, like the use of the deficit countersin [16]. The latter approach can avoid the use of explicit se-quence numbers, but is very sensitive to packet losses as a packetloss can cause resequencing protocols to lose synchronization,which can be recovered only when a marker arrives. Neverthe-less, resequencing should be minimized [17].

Cyclic traffic dispersion is a method to evenly distribute pack-ets over all active paths. Many existing traffic dispersion strate-gies, such as dispersity routing [1], [5], [6], the string-mode pro-tocol [2], and the vector routing algorithm [18], utilize this idea.The major advantage of this approach is that it is quite simple toimplement. However, this method does not take the path hetero-geneity into account. With heterogeneous paths, the best wayto spread traffic along multiple paths may not be by cyclic dis-

1229-2370/03/$10.00 c© 2003 KICS

LEUNG AND LI: FLOW ASSIGNMENT AND PACKET SCHEDULING FOR MULTIPATH... 231

persion, since it may not achieve the desired objective, such asminimizing the end-to-end path delay. In addition, it may inducesubstantial packet resequencing delay when the end-to-end de-lays are quite different among these paths.

Adaptive routing schemes [19], [20] have been proposed tospread packets dynamically over multiple paths according to thenetwork load. These procedures require parameters to determinethe load distribution. Yet, the calculations of such parametersare either computationally intensive or done in an ad hoc man-ner. Thus, there is a need for new multipath routing schemesthat allow a rapid computation of the optimal load distributionparameters.

Since resequencing is due to packets arriving out of order atthe destination, instead of sending packets from the source ac-cording to their sequence numbers, the packet sending order canbe scheduled anew to minimize or to reduce the necessity forresequencing.

A. Our Contributions

The objective of this work is to propose a framework to studyhow to route packets efficiently in multipath communicationnetworks. Two traffic congestion control techniques, namely,flow assignment and packet scheduling, have been investigated.The flow assignment technique works for any routing strategies,such as deterministic routing (with a pre-determined routing se-quence to route packets to a set of paths) and probabilistic rout-ing (by sending a packet on a path at random), to distributetraffic on multiple disjoint paths. The packet scheduling tech-nique operates for any routing strategies with pre-determinedrouting sequences. These two techniques can also be appliedto both connectionless and connection-oriented communicationnetworks.

The flow assignment mechanism defines an optimal splittingof data traffic on multiple disjoint paths based on pre-definedobjectives, such as the minimization of the average end-to-endpath delay. Instead of formulating the problem based on linkcosts, we formulate it as an end-to-end optimal routing problem.This allows the problem to be solved in real time.

The resequencing delay and the usage of the resequencingbuffer can be reduced significantly by properly scheduling thesending order of all packets, say, according to their expectedarrival times at the destination. The difference in the averageend-to-end path delays are compensated, and hence the need forresequencing can be diminished.

Here, the words “source” and “destination” are quite general.A source can mean a source host, which generates traffic. It canalso mean a source router, from which network traffic departs.A destination can be defined similarly. It can mean a destinationhost, which takes and absorbs traffic. It can also mean a desti-nation router, to which network traffic arrives. Though our pro-posed techniques are end-to-end based for source-based routing,they can be applied to perform traffic engineering under variousscenarios, ranging from inter-router to inter-host traffic.

We will investigate the effectiveness of these control mecha-nisms for multipath routing by examining three basic questions:

• What is the optimal split of traffic to achieve the best per-formance?

• How effective is the packet scheduling technique in im-proving performance?

• Under what circumstances should we employ the optimalsplit of traffic instead of cyclic traffic dispersion, and thepacket scheduling technique, in order to achieve a signifi-cant performance improvement?

B. Organization of the Paper

This paper is organized as follows. Section II gives a flow as-signment model to compute an optimal splitting of data traffic onmultiple paths. Section III presents a packet scheduling modelto find a proper sending schedule so as to reduce the packet re-sequencing delay and the resequencing buffer occupancy. Sec-tion IV examines the analytical results derived from the frame-work and studies the effectiveness of the flow assignment andpacket scheduling techniques on multipath routing. Section Vconcludes and discusses some possible extensions to our work.

II. FLOW ASSIGNMENT MODEL

Our flow model consists of a disordering network connectingthe source to the destination, and flows of packets. The disorder-ing network consists of a set of N disjoint paths, namely Path 1,Path 2, · · · , Path N , connecting the source to the destination,such that packets may arrive at the destination in a different or-der as they are sent. Paths are disjoint if and only if they do notshare any directed links.

We assume the source has an unlimited supply of pack-ets, which are delivered to paths according to deterministicor probabilistic routing. The routing weight for Path i, pi,is defined as the portion of dispersed traffic to be routed toPath i, where

∑Ni=1 pi = 1. Denote the flow configuration

by p =(p1 p2 · · · pN

). Due to limited path capacity,

it may not be possible to route all packets to some of thesepaths. Therefore, the load distribution pi is feasible if and onlyif 0 ≤ pi < Mi ≤ 1, where Mi is the upper bound of pi, forall i = 1, 2, · · · , N . A flow configuration is said to be a feasi-ble flow configuration if and only if 0 ≤ pi < Mi ≤ 1 for alli = 1, 2, · · · , N . The following lemma states that the set of allfeasible flow configurations is convex.

Lemma 1: The set of all feasible flow configurations, S, isconvex.

Proof: Suppose f and g are two feasible flowconfigurations. Since fi and gi are non-negative for alli = 1, 2, · · · , N ,

α fi + (1 − α) · gi ≥ 0, (1)

where 0 ≤ α ≤ 1.Since fi and gi are both upper-bounded by Mi for all i =

1, 2, · · · , N ,

α fi + (1 − α) · gi < α Mi + (1 − α) · Mi = Mi. (2)

Combining, α f +(1−α) ·g is a feasible flow configuration.Thus, the set S is convex. �

Let Ci(pi) denote the non-negative monotonically increasingcost function of transmitting packets on Path i with the routingweight pi, where i = 1, 2, · · · , N . Ci(pi) is well defined for all


feasible flow configurations. The notion of ‘cost’ here is quitegeneral. It can represent any system and user costs, includingthe average end-to-end path delay, the average packet loss prob-ability, the average total buffer requirement for all nodes alongthe path, and so on. Hence, the expected cost for multipath rout-ing can be written as:

C(p) =N∑

i=1

pi Ci(pi). (3)

Let Ci(pi) be a non-negative monotonically increasing con-vex function in pi. The subsequent two lemmas show thatpi Ci(pi) and C(p) are also convex functions.

Lemma 2: For every i = 1, 2, · · · , N , pi Ci(pi) is a convexfunction in pi, where pi is a feasible load distribution.

Proof: Without loss of generality, suppose fi and gi aretwo feasible load distributions such that fi ≤ gi. In other words,0 ≤ Ci(fi) ≤ Ci(gi), which implies that

α · (1 − α) · (gi − fi) · [Ci(fi) − Ci(gi)] ≤ 0, (4)

where 0 ≤ α ≤ 1.Since the set of all feasible flow configurations is convex, the

set of all feasible load distributions is convex and thus hi =αfi + (1 − α) · gi is also a feasible load distribution. Using theabove expression,

hi Ci(hi)≤ [α fi + (1 − α) · gi] · [α Ci(fi) + (1 − α) · Ci(gi)]= αfi Ci(fi) + (1 − α) · gi Ci(gi)

+ α · (1 − α) · (gi − fi) · [Ci(fi) − Ci(gi)]≤ α fi Ci(fi) + (1 − α) · gi Ci(gi).

(5)

This means that pi Ci(pi) is a convex function in pi, wherei = 1, 2, · · · , N . �

Lemma 3: The expected cost C(p) is a convex function inp, where p is a feasible flow configuration.

Proof: Suppose f and g are two feasible flow configura-tions. By applying Lemma 2,

C(α f + (1 − α) · g)

=N∑

i=1

[α fi + (1 − α) · gi] · Ci(α fi + (1 − α) · gi)

≤N∑

i=1

[α fi Ci(fi) + (1 − α) · gi Ci(gi)]

= αN∑

i=1

fi Ci(fi) + (1 − α) ·N∑

j=1

gj Cj(gj)

= α C(f) + (1 − α) · C(g).

(6)

Thus, the convexity of the cost function C(p) follows. �

Since the expected cost C(p) is a convex function in p and allfeasible flow configurations p composes the convex set S, C(·)has an unique global minimum on S if S is non-empty [21]. Iter-ative feasible direction algorithms like the constrained gradient

projection method [12] can be adopted to compute the minimumexpected cost and the corresponding feasible flow configuration.

If the path cost function is not known in advance and it actu-ally represents the average end-to-end path delay, we can applythe technique described in [22] to perform on-line marginal de-lay estimation for each participating path. The method is basedon perturbation analysis that assumes no knowledge of networkparameters, such as arrival rates and link capacities. We alsobelieve that it is possible to apply the technique to other costfunctions as well.

To complete our flow assignment model, the cost functionCi(pi) is formulated as the average end-to-end delay on Path i,Di(pi), for our succeeding analytical studies. That is,

Ci(pi) = Di(pi), (7)

where i = 1, 2, · · · , N .

A. Error-Correcting Codes

Error correcting codes may be used to detect and correct anumber of missing or erroneous bits of data. Suppose the num-ber of additional bits required for the error correcting codesis proportional to the amount of user data. Given the routingweight for Path i is pi, the cost of transmitting packets on Path i,including error correcting codes, is:

Ci(vi) = Ci(pi · (1 + ηi)), (8)

where ηi is a constant and path specific, for alli = 1, 2, · · · , N .

Provided that v is a feasible flow configuration, all lemmasderived above are still valid. It is thus always possible to ap-ply the constrained gradient projection method [12] to find theoptimal split of traffic.

B. Non-Disjoint Paths

A pair of paths are said to be non-disjoint if and only if theyshare at least one directed link. A set of non-disjoint paths isone that has at least one pair of non-disjoint paths. Although theaforementioned problem formulation requires that all participat-ing paths have to be disjoint, the authors believe that the methodcan be extended to handle cases where some paths are not dis-joint with each other. The source can still easily check whether itis feasible to distribute traffic over a set of paths under a certainconfiguration, as the set of all feasible flow configurations overa set of paths (either disjoint or non-disjoint) can be shown to beconvex. Given a convex objective cost function, it is always pos-sible to apply the constrained gradient projection method [12] tofind the optimal split of traffic.

III. PACKET SCHEDULING MODEL

In Section II, we have proposed to employ the flow assign-ment method to minimize the expected cost for multipath rout-ing. However, this method merely distributes the load to Npaths. Given a sequence of W ordered packets to be trans-mitted, there are many ways to order the sending sequence forthese packets from the source. Each order of sending sequencefor a set of packets is called a sending schedule. A sending


schedule can have very different packet resequencing delay andresequencing buffer distributions when compared with anotherschedule. This means that care must be taken to determine thebest schedule. Given a pre-determined routing sequence to routethese packets to a set of paths, we propose a packet schedulingmechanism to minimize or to reduce the consumption of the re-sequencing buffer and the delay on resequencing.

Suppose pi is the routing weight for Path i, where∑N

i=1 pi =1. Denote p =

(p1 p2 · · · pN

). To achieve the best system

performance, packets are distributed to each path in a cyclicalfashion so that the arrival instances of any two packets to eachpath is as uniform as possible. Algorithms for constructing suchrouting sequences can be found in [23]. These routing sequencescontain subsequences that can be used to re-construct the wholesequence by the repeated applications of the same subsequence.Nevertheless, our packet scheduling model does not assume anyspecific routing sequences are used.

A round is defined as the minimal subsequence. It is possibleto construct a round of length Q such that qi packets are senton Path i, i = 1, 2, · · · , N , in each round, where pi ≈ qi

Q and

Q =∑N

j=1 qj . Thus, we can denote Pi(j) = k as the posi-

tion, in every round, of the jth packet to be sent on Path i, anddefine R(k) = i to be the identity of the path on which thepacket in Position k of every round will be transmitted. For ex-ample, consider the periodic routing sequence 121312 · · · suchthat six consecutive packets are transmitted on Paths 1, 2, 1,3, 1, 2, respectively, and so on. Therefore, p =

(12

13

16

),

Q =∑3

j=1 qj = 3 + 2 + 1 = 6, P1(2) = 3, and R(3) = 1.The sequence number of the packet sent at Round r,

r ≥ 1, and Position k is designated as O(r, k), whereO(r, k) ≥ 1. Denote by tm, dm, and am, respectively, the send-ing time of the first bit of the packet with the sequence numberm from the source, the end-to-end path delay experienced bythe packet, and the arrival time of the last bit of the packet at thedestination. The relationship among these three quantities is:

am = tm + dm. (9)

Let R(s) denote the resequencing cost function for multipathrouting under the sending schedule s. Our objective is to findan optimal sending schedule s∗ such that R(s) is minimized ats = s∗.

To complete our packet scheduling model, the resequencingcost function R(s) is expressed as:

R(s) =W∑

k=2

k−1maxl=1

(0, al − ak), (10)

where ak = E[ak].R(s) can be minimized to take on value 0 if it is possible

to obtain the optimal sending schedule such that the expectedarrival times for all packets are in ascending order of their se-quence numbers. Ties are resolved by sending the packet withthe smaller sequence number earlier. If all packets arrive at theirexpected arrival times, resequencing is not needed. The task isreduced to searching for such a schedule.

Denote by S(i, r, j) and A(i, r, j), respectively, the expectedsending time of the first bit of the packet at Round r and Position

Pi(j) from the source, and the expected arrival time of the lastbit of the packet at the destination. Without loss of generality,the first packet, i.e., at Round 1 and Position 1, is expected to besent at Time 0. That is,

S(R(1), 1, 1) = 0. (11)

Thus, the packet at Round r and Position Pi(j) is expected tobe transmitted at:

S(i, r, j) = [(r − 1) · Q + Pi(j) − 1] · ∆, (12)

where ∆ denotes the average inter-packet time, i.e., the meantime between two successive packets to be transmitted from thesource. This information can be inferred directly from the esti-mated average bandwidth consumed by the traffic source.

Suppose the average end-to-end delay for Path i is Di. Thisinformation may be obtained by inferring the network statis-tics from the underlying network protocols, if they find feasiblepaths based on the cost metrics, which involve the use of theaverage path delays. In addition, the source can estimate theaverage path delays by exchanging control information periodi-cally with the destination. The expected packet arrival time canbe expressed as:

A(i, r, j) = S(i, r, j) + Di. (13)

Packets are scheduled such that a packet with a smallersequence number O(r1,Pi1(j1)) is expected to arrive at thedestination no later than one with a larger sequence numberO(r2,Pi2(j2)). This schedule order relationship can be writtenas:

O(r1,Pi1(j1)) < O(r2,Pi2(j2)), (14)

if and only if

A(i1, r1, j1) < A(i2, r2, j2) (15)

or {A(i1, r1, j1) = A(i2, r2, j2)S(i1, r1, j1) < S(i2, r2, j2).

(16)

It is time to introduce the theorem that guides the constructionof the optimal schedule.

Theorem 1: By rearranging the packet sending order, thepacket sent at Round r, r = 1, 2, · · · , and Position Pi(j),i = 1, 2, · · · , N , j = 1, 2, · · · , qi, is the one with the sequencenumber:

O(r,Pi(j)) =N∑

u=1u�=i

qu∑

v=1

max(0, r − ξ(i, j, u, v))

+ (r − 1) · qi + j,

(17)

where

ξ(i, j, u, v)

=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(Du−Di)+[Pu(v)−Pi(j)]·∆Q·∆ + 1

if

{Du−Di

∆≡ Pi(j) − Pu(v) (mod Q)

Di > Du

;

� (Du−Di)+[Pu(v)−Pi(j)]·∆Q·∆ � otherwise.

(18)


Proof: Denote γ(i, j, u, v) as the number of packets senton Path u at Position Pu(v) with sequence number less thanO(r,Pi(j)), where u �= i. There are three different cases toconsider.

Case 1:{

Du−Di


Di > Du,(19)

O(γ(i, j, u, v),Pu(v)) < O(r,Pi(j))< O(γ(i, j, u, v) + 1,Pu(v)),

(20)

if and only if

A(i, r, j) = A(u, γ(i, j, u, v) + 1, v), (21)

if and only if

γ(i, j, u, v) =(Di − Du) + [Pi(j) − Pu(v)] · ∆

Q · ∆ +r−1. (22)

When (Di−Du)+[Pi(j)−Pu(v)]·∆Q·∆ ≤ 1 − r, O(r,Pi(j)) <

O(1,Pu(v)) and hence γ(i, j, u, v) = 0. Combining,

γ(i, j, u, v) = max(0, (Di−Du)+[Pi(j)−Pu(v)]·∆Q·∆ + r − 1).

Case 2:{

Du−Di


Di < Du,(23)


(24)

if and only if

A(i, r, j) = A(u, γ(i, j, u, v), v), (25)

if and only if

γ(i, j, u, v) =(Di − Du) + [Pi(j) − Pu(v)] · ∆

Q · ∆ + r. (26)

When (Di−Du)+[Pi(j)−Pu(v)]·∆Q·∆ ≤ −r, O(r,Pi(j)) <


γ(i, j, u, v) = max(0, (Di−Du)+[Pi(j)−Pu(v)]·∆Q·∆ + r).

Case 3:

Du − Di

∆�≡ Pi(j) − Pu(v) (mod Q), (27)


(28)

if and only if

A(u, γ(i, j, u, v), v) < A(i, r, j) < A(u, γ(i, j, u, v) + 1, v),(29)

if and only if

γ(i, j, u, v) = r − � (Du − Di) + [Pu(v) − Pi(j)] · ∆Q · ∆ �. (30)

λµ

b2

2

2

. . .b

L

bL

µLλ 1

µ

1b

b1

b

P

Fig. 1. The L-hop path example.

When � (Du−Di)+[Pu(v)−Pi(j)]·∆Q·∆ � ≥ r, O(r,Pi(j)) <


γ(i, j, u, v) = max(0, r − � (Du−Di)+[Pu(v)−Pi(j)]·∆Q·∆ �).

Combining the above three cases,γ(i, j, u, v) = max(0, r − ξ(i, j, u, v)).

Since O(r,Pi(j)) is equal to the total number of packets thatare expected to arrive at the destination before the tagged packet,

O(r,Pi(j)) =N∑

u=1u�=i

qu∑

v=1

γ(i, j, u, v) + (r − 1) · qi + j. (31)

Thus, the theorem is proved. �

Take δ(i, j) = ξ(R(1), 1, i, j). By using Theorem 1, it canbe shown that:

O(r + 1,Pi(j)) = O(r,Pi(j)) + Q, (32)

where r ≥ maxu,v

u�=R(1)

(1, δ(u, v)) − minx,y

x�=R(1)

(1, δ(x, y)) + 1.

The determination for the set of δ(i, j) characterizes thesteady state behavior of the sending schedule, and hence thepacket resequencing delay and resequencing buffer occupancydistributions.

IV. ANALYTICAL INVESTIGATION

This section discusses the numerical results based on the an-alytical expressions obtained in Sections II and III. To eval-uate the effectiveness of flow assignment and packet schedul-ing, we need to calculate the resequencing delay and the re-sequencing buffer occupancy. Those expressions can be foundfrom our proposed resequencing model in [24]. With the helpof some numerical examples, we can illustrate the effectivenessof these control mechanisms for multipath routing by answeringthe three basic questions posed in Section I.

The expressions we have obtained are applicable to generalpath delay distributions. To illustrate our multipath routingscheme, and without loss of generality, the following tandemqueue path delay model is used.

A path is modelled as an L-node M/M/1 tandemnetwork with a fixed delay line, as exhibited in Fig. 1. LM/M/1 queues and a delay line are connected in tandem. Theith queue receives input from two traffic sources: The taggeddispersed traffic of rate λ, and the interfering or background traf-fic of rate bi. The service rate of the ith server is µi. The delayline, which generally represents the total propagation delay ofthe path, is P . Denote by D and σ2

D the mean and the variance


of the end-to-end path delay. It can be shown [25] that:

D =L∑

i=1

1µi(1 − ρi)

+ P, (33)

σ2D =

L∑

i=1

1µ2

i (1 − ρi)2, (34)

where the utilization of the ith server, ρi = λ+bi

µi. L is set to 5

for our subsequent studies.The central limit theorem [26] suggests that the end-to-end

path delay, which is the sum of a large number of hop delays, isapproximately normally distributed. The mean and the varianceof the end-to-end path delay provide sufficient information togenerate an approximate distribution, which can then be utilizedto compute the resequencing delay distribution. This approach,which was used to solve the end-to-end percentile-type delayobjective allocation problem for networks supporting SwitchedMulti-megabit Data Service (SMDS), has been shown to providethe best approximation to the reference values [27]. Thus, theend-to-end path delay is assumed to be Gaussian or normallydistributed with mean D and variance σ2

D.

Since ∂D∂λ > 0 and ∂2D

∂λ2 > 0 when λ is evaluated at a valuebetween 0 and µi − bi, D is a non-negative monotonically in-creasing convex function, which can be employed as the costfunction to find the unique optimal flow configuration illustratedin Section II.

The inter-sending time between any two consecutive packetsfrom the tagged source, denoted as the inter-packet time, is fivetime units. There are two disjoint paths, namely, Path 1 andPath 2, connecting the source to the destination. Packets arerouted in a cyclical fashion so that the arrival instances of anytwo packets to each path is as uniform as possible. Each serveron Path i, i = 1, 2, serves a packet with an average service timeof φi time units, where φ1 = 1, and φ2 = 0.5, 1, 2, 5. Thiscaptures the cases where capacities of participating paths aredifferent from each other. The set of fixed delays for Path 1 andPath 2 consists of three different combinations, namely (0, 0),(0, 15), and (15, 0) time units, where the first and second entriesof each tuple are the fixed delay on Path 1 and Path 2, respec-tively. This captures the cases where the average end-to-end de-lays of participating paths differ significantly. The backgroundload on Path 1 is fixed to be 0.75 and that on Path 2 varies be-tween 0 and 1, at an increment of 0.02. This set of experimentscan then fully demonstrate why we need to find an optimal traf-fic split instead of cyclic traffic dispersion for multipath routing,and to apply our proposed packet scheduling technique to reducethe necessity for resequencing.

The results are provided in two sets. The first set studies theeffectiveness of the optimal path-splitting of traffic for multipathrouting. It provides the load distributions on Path 2, p2 (wherep1 = 1 − p2), and the average end-to-end path delays. Thesecond set studies the effectiveness of the flow assignment andpacket scheduling techniques on resequencing. It includes theaverage resequencing delays for various settings.

We examine the first set of results. Fig. 2 shows the load dis-tributions on Path 2 when the background load on Path 2 varies

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.2 0.4 0.6 0.8 1

Load

Dis

trib

utio

n on

Pat

h 2

Background Load on Path 2

Fixed Delays = (0, 0) time unit

Link Service Rates = (1, 0.5) packets / time unitLink Service Rates = (1, 1) packet / time unit

Link Service Rates = (1, 2) packets / time unitLink Service Rates = (1, 5) packets / time unit

(a)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.2 0.4 0.6 0.8 1

Load

Dis

trib

utio

n on

Pat

h 2


Fixed Delays = (0, 15) time units



(b)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.2 0.4 0.6 0.8 1

Load

Dis

trib

utio

n on

Pat

h 2


Fixed Delays = (15, 0) time units



(c)

Fig. 2. Load distribution plots: (a) When fixed delays are (0, 0) time unit,(b) when fixed delays (0, 15) time unit, and (c) when fixed delayswhen fixed delays (15, 0) time unit.

between 0 and 1. The link service rate tuple has two entries, cor-responding to the link service rate on Paths 1 and 2, respectively.The load distribution drops from 1 to 0 as the background loadincreases from 0 to 1. Initially, the background load on Path 1is much higher than that on Path 2. It is beneficial to transmita majority of packets on Path 2 instead of cyclic dispersion inorder to take advantage of a much lower average end-to-end de-lay on Path 2. Its superiority fades as the background load onPath 2 increases. At a certain background load on Path 2, cyclic


0

50

100

150

200

250

300

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 0) time unit, Link Service Rates = (1, 0.5) packets / time unit

OptimalCyclic Dispersion

0

20

40

60

80

100

120

140

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 0) time unit, Link Service Rates = (1, 1) packet / time unit


(a) (b)

0

20

40

60

80

100

120

140

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 0) time unit, Link Service Rates = (1, 2) packets / time unit


0

20

40

60

80

100

120

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t




(c) (d)

0

20

40

60

80

100

120

140

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 15) time units, Link Service Rates = (1, 1) packet / time unit


0

20

40

60

80

100

120

140

0 0.2 0.4 0.6 0.8 1

Ave

rage

End

-to-

end

Pat

h D

elay

/ T

ime

Uni

t




(e) (f)

Fig. 3. Average end-to-end path delay plots: (a) When fixed delays are (0, 0) time unit and link service rates are (1, 0.5) packets per time unit, (b)when fixed delays are (0, 0) time unit and link service rates are (1, 1) packets per time unit, (c) when fixed delays are (0, 0) time unit and linkservice rates are (1, 2) packets per time unit, (d) when fixed delays are (0, 0) time unit and link service rates are (1, 5) packets per time unit, (e)when fixed delays are (0, 15) time unit and link service rates are (1, 1) packets per time unit, and (f) when fixed delays are (15, 0) time unit andlink service rates are (1, 1) packets per time unit.

dispersion becomes the best strategy for traffic splitting. As thebackground load on Path 2 continues to rise, it is advantageousto distribute a higher load on Path 1 rather than Path 2. Eventu-ally, all packets should be sent on Path 1.

Besides, it is always possible to achieve optimal splitting witha higher load distribution on Path 2 with faster transmissionlinks. A faster transmission link means that it can take a higher

portion of traffic from the source to get the same offered load,given the total work feeding into the network from the source isfixed. Moreover, a higher propagation delay on Path 2 causesa greater end-to-end path delay, resulting in a reduction in theoptimal load distribution on Path 2.

Fig. 3 exhibits the average end-to-end path delays for boththe optimal split of traffic and cyclic dispersion when the back-


0

50

100

150

200

250

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 0) time unit, Link Service Rates = (1, 0.5) packets / time unit

Optimal with SchedulingOptimal without Scheduling

Cyclic Dispersion with SchedulingCyclic Dispersion without Scheduling

0

20

40

60

80

100

120

140

160

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t


Fixed Delays = (0, 0) time unit, Link Service Rates = (1, 1) packet / time unit



(a) (b)

0

20

40

60

80

100

120

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t





0

5

10

15

20

25

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t





(c) (d)

0

20

40

60

80

100

120

140

160

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t





0

20

40

60

80

100

120

140

160

0 0.2 0.4 0.6 0.8 1

Ave

rage

Res

eque

ncin

g D

elay

/ T

ime

Uni

t





(e) (f)

Fig. 4. Average resequencing delay plots: (a) When fixed delays are (0, 0) time unit and link service rates are (1, 0.5) packets per time unit, (b) whenfixed delays are (0, 0) time unit and link service rates are (1, 1) packets per time unit, (c) when fixed delays are (0, 0) time unit and link servicerates are (1, 2) packets per time unit, (d) when fixed delays are (0, 0) time unit and link service rates are (1, 5) packets per time unit, (e) whenfixed delays are (0, 15) time unit and link service rates are (1, 1) packets per time unit, and (f) when fixed delays are (15, 0) time unit and linkservice rates are (1, 1) packets per time unit.

ground load on Path 2 varies between 0 and 1. At low and mod-erate background load on Path 2, we see that the optimal split oftraffic yields a greater reduction on the average delay when thelink service rate for Path 2 becomes higher, since an increase inthe link service rate reduces the overall offered load and thus theaverage end-to-end delay on Path 2. In addition, the improve-ment decreases when the propagation delay on Path 2 is higher,and vice versa.

The second set of results demonstrates the effectiveness ofthe flow assignment and packet scheduling techniques on re-sequencing. Our results in Fig. 4 show that the optimal flowassignment always yields a lower average resequencing delayand hence a lower average resequencing buffer occupancy1 thancyclic dispersion of traffic. A further improvement can be ob-

1The average resequencing buffer occupancy is proportional to the averageresequencing delay by applying Little’s Theorem [25].


tained by applying the proposed packet scheduling technique.When the background load on Path 2 rises from 0 to 1, the end-to-end delay on Path 2 increases. This reduces the differencein path delays and hence the average packet resequencing de-lay when cyclic dispersion is used. At a certain backgroundload on Path 2, the average difference in path delays and alsothe average resequencing delay attain their minimum values.As the background load on Path 2 continues to rise, the dif-ference in path delays and hence the average resequencing de-lay increase till Path 2 becomes overloaded. By employing thepacket scheduling technique together with cyclic dispersion, theaverage resequencing delay is kept below the inter-packet timeuntil the offered load on Path 2 becomes very heavy. We seethat a substantial performance improvement can be achieved forcases where there is a large difference in path delays only whenthe packet scheduling technique is applied. This means that thepacket scheduling mechanism is very effective in providing per-formance improvement when there is a large difference in pathfixed delays.

V. CONCLUSIONS

In this paper, we have proposed a framework to study how toroute packets efficiently in multipath communication networks.Two traffic congestion control techniques, namely, flow assign-ment and packet scheduling, have been investigated. The flowassignment mechanism defines an optimal splitting of data traf-fic on multiple disjoint paths, so as to minimize the average costfor multipath routing, such as the average end-to-end path delayexperienced by a packet. Yet, packets may still experience a sub-stantial difference in average path delays if they are sent on dif-ferent paths. The packet scheduling mechanism can be utilizedto minimize or to reduce the consumption of the resequencingbuffer and delay on resequencing by reducing the chance for outof order packet arrivals at the destination, say, by schedulingthem according to their expected arrival times at the destination.This technique is particularly useful for transmitting archival in-formation, such as stored videos, since all archival packets areavailable and they can be sent in any desired order.

To illustrate our model, and without loss of generality, Gaus-sian distributed end-to-end path delays are used. Our analyti-cal results show that the techniques are very effective in reduc-ing the average end-to-end path delay, the average packet rese-quencing delay, and the average resequencing buffer occupancyfor various path configurations.

Now we re-visit the three questions posed in Section 1. As ex-pected, the optimal flow assignment aims to minimize the spec-ified objective cost, and shifts the offered load from one path toother paths when the background load of the path continues torise. It is always possible to achieve optimal traffic splitting bysending more traffic on a path with faster transmission links.

Compared with cyclic traffic dispersion, the optimal split oftraffic yields a greater reduction on the average delay whenthe background loads of participating paths differ substantially.However, a large difference in the end-to-end propagation de-lays generally causes a great difference in the average end-to-end path delays, resulting in a reduction in the performance im-provement. This means that the optimal split of traffic can yield

a significantly better network performance whenever the back-ground loads of participating paths differ substantially or thereexist large differences in end-to-end propagation delays. Our re-sults also show that the optimal flow assignment always yieldsa lower average resequencing delay and hence a lower averageresequencing buffer occupancy than cyclic dispersion of traffic.

A further performance improvement can be achieved by ap-plying the proposed packet scheduling technique, which helpsto effectively compensate for the difference in path delays andhence reduce the necessity for resequencing even when cyclicdispersion is used. By applying the packet scheduling techniquetogether with cyclic dispersion, the average resequencing delayis kept below the inter-packet time until some participating pathsbecome heavily loaded. Since the optimal split of traffic maynot reduce the differences in end-to-end path delays, the pro-posed packet scheduling technique can complement the flow as-signment technique by reducing the necessity for resequencing,thereby improving the effectiveness in using the resequencingbuffer. When transmitting archival and some delayed real timeinformation, it is always possible to reduce the necessity for re-sequencing by applying the packet scheduling technique. Thesepromising results can form a basis for designing and deployingfuture adaptive multipath protocols.

There are several possible extensions to our work, some ofwhich are listed below:

• Devise an adaptive multipath protocols for packet-switching networks, such as IP-based and ATM networks;

• Incorporate quality of service routing [17] with multipathrouting;

• Extend the framework to consider reliability and fault tol-erant issues.

ACKNOWLEDGEMENTS

The authors would like to thank the anonymous reviewers fortheir valuable comments and suggestions which assisted us inimproving the quality of the paper.

REFERENCES[1] S. N. Chiou and V. O. K. Li, “Diversity transmissions in a communica-

tion network with unreliable components,” in Proc. IEEE ICC ’87, vol. 2,Seattle, WA, USA, 7-10 June 1987, pp. 968–973.

[2] J. H. Dejean, L. Dittmann, and C. N. Lorenzen, “String mode – A newconcept for performance improvement of ATM networks,” IEEE J. Select.Areas Commun., vol. 9, no. 9, pp. 1452–1460, Dec. 1991.

[3] E. Gustafsson and G. Karlsson, “A literature survey on traffic dispersion,”IEEE Network, vol. 11, no. 2, pp. 28–36, Mar.-Apr. 1997.

[4] K.-C. Leung and V. O. K. Li, “Generalized load sharing for packet-switching networks,” in Proc. ICNP 2000, Osaka, Japan, 14-17 Nov. 2000,pp. 305–314.

[5] N. F. Maxemchuk, “Dispersity routing,” in Proc. IEEE ICC ’75, San Fran-cisco, CA, USA, June 1975, pp. 41-10 – 41-13.

[6] N. F. Maxemchuk, “Dispersity routing in high-speed networks,” ComputerNetworks and ISDN Systems, vol. 25, no. 6, pp. 645–661, Jan. 1993.

[7] W. Stallings, High-Speed Networks: TCP/IP and ATM Design Principles,Prentice-Hall International, Inc., 1998.

[8] V. O. K. Li and W. Liao, “Distributed multimedia systems,” Proc. IEEE,vol. 85, no. 7, July 1997, pp. 1063–1108.

[9] J. Beran et. al., “Long-range dependence in variable-bit-rate video traffic,”IEEE Trans. Commun., vol. 43, no. 2–4, pp. 1566–1579, Feb.–Apr. 1995.

[10] W. E. Leland, et. al., “On the self-similar nature of ethernet traffic (ex-tended version),” IEEE/ACM Trans. Networking, vol. 2, no. 1, pp. 1–15,Feb. 1994.


[11] A. Erramilli, O. Narayan, and W. Willinger, “Experimental queueing anal-ysis with long-range dependent packet traffic,” IEEE/ACM Trans. Net-working, vol. 4, no. 2, pp. 209–223, Apr. 1996.

[12] D. Bertsekas and R. Gallager, Data Networks, Second Edition. PrenticeHall, 1992.

[13] K.-C. Leung and V. O. K. Li, “Flow assignment and packet schedulingfor multipath networks,” in Proc. IEEE GLOBECOM ’99, vol. 1, Rio deJaneiro, RJ, Brazil, 5-9 Dec. 1999, pp. 246–250.

[14] S. Vutukury and J. J. Garcia-Luna-Aceves, “A simple approximationto minimum-delay routing,” Computer Communication Review, vol. 29,no. 4, pp. 227–238, Oct. 1999.

[15] D. A. Khotimsky, “A packet resequencing protocol for fault-tolerant mul-tipath transmission with non-uniform traffic splitting,” in Proc. IEEEGLOBECOM ’99, vol. 2, Rio de Janeiro, RJ, Brazil, 5-9 Dec. 1999,pp. 1283–1289.

[16] H. Adiseshu, G. Varghese, and G. Parulkar., “An architecture for packet-striping protocols,” ACM Trans. Computer Systems, vol. 17, no. 4,pp. 249–287, Nov. 1999.

[17] S. Chen and K. Nahrstedt, “An overview of quality of service routing fornext-generation high-speed networks: Problems and solutions,” IEEE Net-work, vol. 12, no. 6, pp. 64–79, Nov.-Dec. 1998.

[18] T. T. Lee, S. C. Liew, and Q.-L. Ding, “Parallel communications forATM network control and management,” Performance Evaluation, vol. 30,no. 4, pp. 243–264, Oct. 1997.

[19] S. Bahk and M. E. Zarki, “Preventive congestion control based routing inATM networks,” in Proc. IEEE ICC ’94, vol. 3, New Orleans, LA, USA,1-5 May 1994, pp. 1592–1599.

[20] R. Krishnan and J. A. Silvester, “An approach to path-splitting in mul-tipath networks,” in Proc. IEEE ICC ’93, vol. 3, Geneva, Switzerland,23-26 May 1993, pp. 1353–1357.

[21] D. G. Luenberger, Linear and Nonlinear Programming, Second Edition,Addison-Wesley, 1984.

[22] C. G. Cassandras, M. V. Abidi, and D. Towsley, “Distributed routing withon-line marginal delay estimation,” IEEE Trans. Commun., vol. 38, no. 3,pp. 348–359, Mar. 1990.

[23] Y. Arian and Y. Levy, “Algorithms for generalized roundrobin routing,” Operations Research Letters, vol. 12, no. 5,pp. 313–319, Nov. 1992.

[24] K.-C. Leung and V. O. K. Li, “A resequencing model for high speednetworks,” in Proc. IEEE ICC ’99, vol. 2, Vancouver, BC, Canada, 6-10 June 1999, pp. 1239–1243,.

[25] L. Kleinrock, Queueing Systems (Volume I: Theory), John Wiley & Sons,1975.

[26] W. W. Hines and D. C. Montgomery, Probability and Statistics in En-gineering and Management Science, Third Edition, John Wiley & Sons,1990.

[27] F. Y. S. Lin, “Allocation of end-to-end delay objectives for networks sup-porting SMDS,” in Proc. IEEE GLOBECOM ’93, vol. 3, Houston, TX,USA, Nov. 29–Dec. 2, 1993, pp. 1346–1350.

Ka-Cheong Leung was born in Hong Kong in 1972.He received the B.Eng. degree in Computer Sciencefrom the Hong Kong University of Science and Tech-nology, Hong Kong, in 1994, the M.Sc. degree inElectrical Engineering (Computer Networks) and thePh.D. degree in Computer Engineering from the Uni-versity of Southern California, Los Angeles, Califor-nia, USA, in 1997 and 2000, respectively. He workedas a senior research engineer at Nokia Research Cen-ter, Nokia Inc., Irving, Texas, USA from 2001 to 2002.Since August 2002 he has been with the Texas Tech

University, Lubbock, Texas, USA, where he is an Assistant Professor at theDepartment of Computer Science. His research interests include routing, con-gestion control, and quality of service guarantees in high-speed communicationnetworks, high-performance computing, and parallel applications.

Victor O. K. Li was born in Hong Kong in 1954.He received SB, SM, EE, and ScD degrees in Electri-cal Engineering and Computer Science from the Mas-sachusetts Institute of Technology, Cambridge, Mas-sachusetts, USA, in 1977, 1979, 1980, and 1981, re-spectively. He joined the University of Southern Cali-fornia (USC), Los Angeles, California, USA in Febru-ary 1981, and became Professor of Electrical Engi-neering and Director of the USC Communication Sci-ences Institute. Since September 1997 he has beenwith the University of Hong Kong, Hong Kong, where

he is Chair Professor of Information Engineering at the Department of Electri-cal and Electronic Engineering, and Managing Director of Versitech Ltd., thetechnology transfer and commercial arm of the University. He also serves onvarious corporate boards. His research is in information technology, includinghigh-speed communication networks, wireless networks, and Internet technolo-gies and applications. He is a Principal Investigator of the Area of Excellence inInformation Technology funded by the Hong Kong Government. Sought by gov-ernment, industry, and academic organizations, he has lectured and consultedextensively around the world. Prof. Li chaired the Computer CommunicationsTechnical Committee of the IEEE Communications Society 1987-1989, and theLos Angeles Chapter of the IEEE Information Theory Group 1983-1985. He co-founded the International Conference on Computer Communications and Net-works (IC3N), and chaired its Steering Committee 1992-1997. He also chairedvarious international workshops and conferences, and is most recently appointedGeneral Chair of IEEE INFOCOM 2004. Prof. Li has served as an editor ofIEEE Network, IEEE JSAC Wireless Communications Series, and Telecommu-nication Systems. He also guest edited special issues of IEEE JSAC, ComputerNetworks and ISDN Systems, and KICS/IEEE Journal of Communications andNetworks. He is now serving as an editor of ACM/Kluwer Wireless Networksand IEEE Communications Surveys and Tutorials. Prof. Li was recently ap-pointed to the Hong Kong Information Infrastructure Advisory Committee bythe Chief Executive of the Hong Kong Special Administrative Region. He alsoserves on the Innovation and Technology Fund (Electronics) Vetting Commit-tee, the Small Entrepreneur Research Assistance Programme Committee, theEngineering Panel of the Research Grants Council, and the Task Force for theHong Kong Academic and Research Network (HARNET) Development Fund ofthe University Grants Committee. He was a Distinguished Lecturer at the Uni-versity of California at San Diego, at the National Science Council of Taiwan,and at the California Polytechnic Institute. Prof. Li has also delivered keynotespeeches at many international conferences. He has received numerous awards,including, most recently, the Croucher Foundation Senior Research Fellowship,in March 2002, and the Bronze Bauhinia Star, Government of the Hong KongSpecial Administrative Region, China, July 1 2002. He was elected an IEEEFellow in 1992.

flow assignment and packet scheduling for multipath routing

Documents