de nition 1. a continuous random variable is said to have ... continuous random variable xis said to...

IMA BOOTCAMP: STOCHASTIC MODELING

JASMINE FOO AND KEVIN LEDER

Note: much of the material in these bootcamp lecture notes is adapted from the book‘Introduction to Probability Models’ by Sheldon Ross and lecture notes on the web byJanko Gravner for MAT135B.

1. Exponential random variables and their properties

We begin with a review of exponential random variables, which appear a lot in mathe-matical models of real-world phenomena because of their convenient mathematical prop-erties.

Definition 1. A continuous random variable X is said to have an exponential distributionwith parameter λ, λ > 0, if its probability density function is given by

f(x) =

{λe−λx, x ≥ 0

0, x < 0

1.1. Moments. The mean of the exponential distribution, E[X] is given by

E[X] =

∫ ∞−∞

xλe−λxdx =1

λ.

The moment generating function φ(t) of the exponential distribution is

φ(t) = E[etX ] =

∫ ∞0

etxλe−λxdx =λ

λ− t.

Using this we can derive the variance

V ar(X) = E[X2]− (EX)2 =d2

dt2φ(t)

∣∣∣∣t=0

− 1

λ2=

1

λ2.

1.2. Memoryless property. Exponential distributions have the memoryless property.Specifically, it satisfies the following:

P (X > s+ t|X > t) = P (X > s), for all s, t ≥ 0.

To see this, note that the property is equivalent to

P (X > s+ t,X > t)

P (X > t)= P (X > s)

or

P (X > s+ t) = P (X > s)P (X > t).1

2 JASMINE FOO AND KEVIN LEDER

For X exponentially distributed, we get

e−λ(s+t) = e−λse−λt

so the memoryless property is satisfied. It can be shown that the exponential distributionis the only distribution that satisfies this property.

Example 1. A store must decide how much product to order to meet next month’s demand.The demand is modeled as an exponential random variable with rate λ. The product costsc dollars per unit, and can be sold at a price of s > c dollars per unit. How much shouldbe ordered to maximize the store’s expected profit? Assume that any inventory left over atthe end of the month is worthless.

Solution. Let X be the demand. If the store orders t units of the product, the profit is

P = s min(X, t)− ctIn other words

P =

{sX − ct, X < t

(s− c)t, X ≥ tSo

E[P ] = (sE[X|X < t]− ct)P (X < t) + (s− c)tP (X ≥ t).To calculate this conditional expected value, we note that

1/λ = E[X] = E[X|X < t]P (X < t) + E[X|X ≥ t]P (X ≥ t)

= E[X|X < t](1− eλt) + (t+ EX)e−λt

where in the last line we have used the memoryless property: conditioned on having exceededt, the amount by which X exceeds t is distributed as an exponential variable with parameterλ. Thus we have

E[X|X < t] =1λ − (t+ 1

λ)e−λt

1− e−λtPlugging this into the expression for E[P ] we have

E[P ] = (s1λ − (t+ 1

λ)e−λt

1− e−λt− ct)(1− e−λt) + (s− c)teλt

= s/λ− se−λt/λ− ct

To maximize the expected profits, we take the derivative and set se−λt − c = 0, obtainingthat the maximal profit is obtained when

t =1

λlog (s/c).

Definition 2. The hazard rate or failure rate of a random variable with cumulative distri-bution function F and density f is:

h(t) =f(t)

1− F (t).

IMA BOOTCAMP: STOCHASTIC MODELING 3

Consider the following related quantity: conditioned on X > t, what is the probability thatX ∈ (t, t+ dt)?

P (X ∈ (t, t+ dt)|X > t) =P (X ∈ (t, t+ dt), X > t)

P (X > t)=P (X ∈ (t, t+ dt))

P (X > t)≈ f(t)dt

1− F (t)

So, if we think of X as the time to failure of some object, the hazard rate can be interpretedas the conditional probability density of failure at time t, given that the object has not faileduntil time t.

The failure rate of an exponentially distributed random variable is a constant:

h(t) = λe−λteλt = λ

1.3. Sum and minimums of exponential random variables. Suppose Xi, i = 1 . . . nare independent identically distributed exponential random variables with parameter λ. Itcan be shown (by induction, for example), that the sum

X1 +X2 + . . .+Xn

has a Gamma distribution with parameters n and 1/λ, with pdf

f(t) = λe−λt(λt)n−1

(n− 1)!.

Next, let’s define Y to be the minimum of n independent exponential random variables,X1, . . . , Xn with parameters λ1, . . . , λn.

P (Y > x) =n∏i=1

P (Xi > x)

=n∏i=1

e−λix

= e−(∑n

i=1 λi)x

Thus we see that the minimum, Y , is exponentially distributed with parameter equal tothe sum of the parameters λ1 + λ2 · · ·+ λn.

2. Poisson Processes

Definition 3. A counting process is a random process N(t), t ≥ 0 (e.g. number of eventsthat have occurred by time t) such that

(1) For each t, N(t) is a nonnegative integer(2) N(t) is nondecreasing in t(3) N(t) is right continuous.

Thus N(t)−N(s) represents the number of events in (s, t].

Definition 4. A Poisson process is a random process with intensity (or rate) λ > 0 is acounting process N(t) such that


(1) N(0) = 0(2) it has independent increments: if (s1, t1] ∩ (st, t2] = ∅ then N(t1) − N(s1) and

N(t2)−N(s2) are independent(3) number of events in any interval of length t is Poisson(λt)

Recall that a Poisson random variable Y with parameter λ takes values k = 0, 1, 2, . . .with probability

P (Y = k) =λke−λ

k!In particular, we have the probability mass function for the number of events in an intervalof size h:

P (N(t+ h)−N(t) = k) = e−λh(λh)k

k!, k = 0, 1, 2, . . .

Now, if h→ 0 we have that the number of arrivals in a small interval of size h satisfies

P (N(h) = 1) = λh

and

P (N(h) ≥ 2) = o(h).

Also, from the properties of a Poisson random variable we have that

E[N(t)] = λt.

2.1. Motivation for the Poisson process. Suppose we have a store where the customersarrive at a rate λ = 4.5 customers per hour on average, and the store owner is interested tofind the distribution of the average number X of customers who arrive during a particulartime period of length t hours. The arrival rate can be recast as 4.5/3600 seconds per hour= 0.00125 per second. We can interpret this probabilistically by saying that during eachsecond either 0 or 1 customers will arrive, and the probability during any single secondis 0.00125. Under this setup the Binomial(3600t, 0.00125) distribution describes X, thenumber of customers who arrive during a time period of length t. Using the Poissonapproximation to the Binomial we then have that X is approximated by a Poisson variablewith mean 3600t ∗ 0.00125 = 4.5t = λt, which is consistent with our definition of a Poissonprocess. Recall that property 3 of a Poisson process is that the number of arrivals in anygiven time period of length t is Poisson(λt). In addition, by construction we have that thenumber of arrivals in non overlapping time intervals is independent.

This example motivates the definition of a Poisson process, which is often used to modelarrival processes of events with a constant average arrival rate.

2.2. Interarrival and waiting time distributions. Consider a Poisson process withrate λ and let T1 denote the time of the first event, and for n > 1 let Tn denote the elapsedtime between the (n − 1)st and nth event. The sequence {Tn, n = 1, 2, . . .} is called thesequence of inter arrival times.

Proposition 1. Tn, n = 1, 2, . . . are independent identically distributed exponential randomvariables with parameter λ.


To see this, first consider T1.

P (T1 > t) = P (N(t) = 0) = e−λt

Thus T1 is exponentially distributed with parameter λ. Then

P (T2 > t) =

∫ ∞0

P (T2 > t|T1 ∈ (s, s+ ds))P (T1 ∈ (s, s+ ds))ds

=

∫ ∞0

P (no events in (s, s+t])λe−λsds

=

∫ ∞0

e−λtλe−λsds

= e−λt

So, T2 is also exponentially distributed with parameter λ. From this argument we can alsosee that T2 is independent of T1 since conditioning on the value of T1 has no impact on thepdf of T2. Repeating the argument for n ≥ 2 gives the result.

Intuitively, remember that the Poisson process has independent increments (so from anypoint onward it is independent of what happened in the past) and stationary increments(any increment of the same length has the same distribution), so the process has no memory.Hence exponential waiting times until the next arrival are expected.

Next we consider the waiting time until the nth event, Sn, and its distribution. Notethat

Sn =n∑i=1

Ti

and since the Ti are iid exponential(λ) variables, the variable Sn is Gamma(n, λ). In otherwords, Sn has pdf

fSn(t) = λe−λt(λt)n−1

(n− 1)!, t ≥ 0.

Note that we could also equivalently define a Poisson process by starting with iid ex-ponential inter arrival times with parameter λ, and then defining a counting process bysaying that the nth event occurs at time

Sn = T1 + T2 + . . .+ Tn.

The resulting counting process would be a Poisson process with rate λ.

3. Continuous time Markov Chains

Next we consider a class of probability models that are widely used in modeling real-world applications: continuous-time Markov Chains. These stochastic processes take placein continuous time and have the property that, given the present state, the future isindependent of the past. The Poisson process is an example of a continuous-time Markovchain.


Definition 5. Suppose we have a continuous time stochastic process {X(t), t ≥ 0} takingvalues in the nonnegative integers. The process X(t) is a continuous time Markov Chainif for all s, t ≥ 0 and nonnegative integers i, j, x(u), 0 ≤ u < s

P (X(t+ s) = j|X(s) = i,X(u) = x(u), 0 ≤ u < s} = P (X(t+ s) = j|X(s) = i).

If, additionally,

P (X(t+ s) = j|X(s) = i)

is independent of s, then X is said to have homogeneous transition probabilities.

In the following, we will always work with Markov chains with homogeneous transitionprobabilities. The Markovian property tells us that, no matter how long the chain hasbeen in a particular state, the time it waits until transitioning to a new state has the samedistribution (i.e. the waiting time between transitions has the memoryless property, andmust thus be exponentially distributed). Thus, we can think of a continuous-time Markovchain as moving according to a discrete time Markov chain except that the time betweenjumps is exponentially distributed.

Example 2. A population of size N has It infected individuals, St susceptible individualsand Rt recovered/removed individuals. New infections occur at an exponential rate βItStand infected individuals become removed/recovered at an exponential rate Y (i.e. the overallrate of leaving the infected state is Y It.) Here, the state of the system is represented by thenumber of people in each state at any time t.

3.1. Birth and Death processes. Let us consider a system whose state at any time isrepresented simply by the number of people in the system. Whenever there are n people inthe system, new arrivals enter at an exponential rate λn and people leave at an exponentialrate µn. This system is called a birth and death process, and it is a continuous time Markovchain with states 0, 1, . . .. The rates λn and µn are called the birth and death rates.

This chain can only move up or down by one unit at a time. If the system is in staten > 0, the time it takes until the next arrival is exponentially distributed with parameterλn, and the time it takes until the next death is exponentially distributed with parameterµn. The time the system stays in state n is the minimum of these two exponentials, anexponential with rate λn + µn. If n = 0 then it takes an exponential(λ0) amount of timefor the system to reach state 1. The transition probabilities Pi,j tell us the probability ofmoving from state i at the next transition if the system is in state j. Here we have

P0,1 = 1

and

Pi,i+1 =λi

λi + µiwhich is the probability that the next arrival/birth occurs before the next death, and canbe derived from the properties of the exponential distribution. Lastly

Pi,i−1 =µi

λi + µi


Example 3. A Poisson process is a birth and death process with λn = λ and µn = 0 forall n ≥ 0. This is also called a pure birth process.

Example 4. A Yule process is a birth-death process with λn = λn and µn = 0, for alln ≥ 0. This models the scenario in which each member of the population can give birthto a new member and takes an exponential(λ) amount of time to give birth, but no deathsoccur. This is also an example of a pure birth process.

Example 5. A linear birth-death process has λn = λn and µn = µn, for all n ≥ 0. Thismodels the scenario in which each member of the population can give birth to a new memberand takes an exponential(λ) amount of time to give birth. At the same time, each membercan die at exponential rate µn.

Example 6. Queueing system M/M/1. Suppose that customers arrive at a queue as aPoisson process with rate λ. If the queue is empty the customer goes directly into service;otherwise he/she goes to wait at the end of the queue. The server serves each customerin order of the queue, and the service times are independent exponential variables withparameter µ. This is the M/M/1 queueing system. It is a birth-death process with λn = λfor n ≥ 0 and µn = µ for n ≥ 1.

Let us return to a general birth and death process with birth and death rates λn andµn, where µ0 = 0. Let Ti be the time, starting from state i, that it takes the process to goto state i+ 1, for any i ≥ 0.

For i = 0, the only way for the system to reach state 1 is to get a birth, so

E[T0] =1

λ0

For i > 0 the time it takes depends on whether the first transition is a birth or a death.Let

Ii =

{1, if the first transition is to i+ 1

0, if the first transition is to i− 1

Now

E[Ti] = E[Ti|Ii = 1]P (Ii = 1) + E[Ti|Ii = 0]P (Ii = 0)

=1

λi + µi

λiλi + µi

+

(1

λi + µi+ E[Ti−1] + E[Ti]

)µi

λi + µi

=1

λi + µi+ (E[Ti−1] + E[Ti])

µiλi + µi

Here we have used the fact that the distribution of Ti conditioned on the first transitionbeing to i+ 1 or i− 1 is still exponential with parameter λi + µi. This can be seen by thefollowing:

P (Ti ∈ dt|I1 = 1) =P (Ti ∈ dt, I1 = 1)

λiλi+µi

=λ1e−λite−µit

λiλi+µi

= (λi + µi)e−(λi+µi)t.


Solving we obtain

E[Ti] =1

λi+µiλiE[Ti−1], i ≥ 1.

This equation can be solved to get estimates recursively for E[T1], E[T2], . . ..

3.2. Limiting probabilities.

Definition 6. Let

Pij(t) ≡ P (X(t+ s) = j|X(s) = i)

denote the probability that a process in state i will be in state j t time units later. Thesequantities are called the transition probabilities of the continuous-time Markov chain.

Example 7. Let us derive the transition probabilities for a pure birth process X(t) withbirth rates λn, for n ≥ 0. Consider the quantity, for i < j,

P (X(t) < j|X(0) = i)

which is the probability that the process has not yet reached state j by the time t. We canthink of the time before the process hits state j as the sum of the times the process has spentin each of the states k = 1, . . . , j− 1. Let Xi denote the amount of time the process spendsin state k before moving to k + 1. We are essentially interested in the equivalent quantity

P (

j−1∑k=i

Xk > t).

Now recall that the Xk are independent exponentials with with parameters λk. This quantitycan be derived directly from the properties of the exponential distribution (we will not showit here); it can be shown that

P (X(t) < j|X(0) = i) = P (

j−1∑k=i

Xk > t) =

j−1∑k=i

j−1∏l 6=k

λlλl − λk

e−λkt

We can also find

P (X(t) < j + 1|X(0) = i) = P (

j∑k=i

Xk > t) =

j∑k=i

j∏l 6=k

λlλl − λk

e−λkt

Thus we have, for i < j

Pij(t) = P (X(t) = j|X(0) = i) =

j−1∑k=i

j−1∏l 6=k

λlλl − λk

e−λkt −j∑k=i

j∏l 6=k

λlλl − λk

e−λkt

and of course

Pii(t) = P (Xi > t) = e−λit


These transition probabilities are governed by a set of differential equations called theKolmogorov forward and backward equations. In order to state them, we need to define afew quantities. For any pair of states i, j let vi be the rate at which the process makes atransition when in state i, and let

qij = viPij .

Then, qij are instantaneous transition rates; qij represents the rate, when in state i, atwhich the process makes a transition into state j. Note that

vi =∑j

viPij =∑j

qij

Now we are ready to state the Kolmogorov backward equations. We will not prove that Pijsatisfies these equations here, but the derivation can be found in Ross Chapter 6. For allstates i, j and t ≥ 0,

P ′ij(t) =∑k 6=i

qikPkj(t)− viPij(t)

Example 8. The backwards equations for the pure birth process are

P ′ij(t) = λiPi+1,j(t)− λiPij(t)

Similarly, it can be shown that the Pij satisfy the Kolmogorov forward equations, givenby:

P ′ij(t) =∑k 6=i

qkjPik(t)− viPij(t)

Next we will consider how to find the limiting probability that a continuous time Markovchain will be in state j:

Pj = limt→∞

Pij(t),

where we are assuming that the limit exists and is independent of the initial state i. Letus consider the forward equations, taking a limit as t → ∞ and assuming that the limitand summation can be interchanged gives us

limt→∞

P ′ij(t) = limt→∞

∑k 6=i

qkjPik(t)− viPij(t)

=∑k 6=i

qkjPk − vjPj

Since Pij(t) is a bounded function it follows that if P ′ij(t) converges it must converge tozero. So

0 =∑k 6=j

qkjPk − vjPj

i.e.

vjPj =∑k 6=j

qkjPk(1)


Solving this system along with the constraint that∑

j Pj = 1 allows us to solve for thelimiting probabilities. This equation balances the rate at which the process enters andleaves state j.

Example 9. Let us consider the limiting probabilities for a birth and death process. Usingequation (1) we have

(λj + µj)Pj = (λj+1 + µj+1)µj+1

(λj+1 + µj+1)Pj+1 + (λj−1 + µj−1)

λj−1(λj−1 + µj−1)

Pj−1

= µj+1Pj+1 + λj−1Pj−1

for j ≥ 1 and

λ0P0 = µ1P1

for j = 0. This set of equations can be simplified to

λ0P0 = µ1P1

λ1P1 = µ2P2

λ2P2 = µ3P3

...

Solving this we obtain

P1 =λ0µ1P0

P2 =λ1µ2P1 =

λ1λ0µ2µ1

P1

...

Pn =λn−1λn−2 · · ·λ0µnµn−1 · · ·µ1

P0

Combining with the fact that∑

j Pj = 1 we obtain

P0 =1

1 +∑∞

n=1λn−1λn−2···λ0µnµn−1···µ1

and for n ≥ 1

Pn =λn−1λn−2 · · ·λ0

µnµn−1 · · ·µ1(1 +∑∞

n=1λn−1λn−2···λ0µnµn−1···µ1 )

4. Renewal Process

Consider a lightbulb in a room. Each time it burns out we go and replace it, if we countthe number of times we replace it then we arrive at a stochastic process called a ‘renewalprocess’. Each time the bulb is replaced we say a renewal has occurred.


Definition of Renewal process. Let {Xn}n≥1 be an i.i.d sequence of positive randomvariables. Define S0 = 0 and for n ≥ 1 set

Sn =

n∑i=1

Xi

then for t ≥ 0 define

N(t) = max{n : Sn ≤ t}.The process {N(t); t ≥ 0} is called a renewal process. The sequence {Sn}n≥1 are the timesat which renewals occur and N(t) counts the number of renewals in the interval [0, t].

Since the i.i.d random variables Xi > 0 we know that µ = E[X1] > 0 and therefore bythe law of large numbers Sn → ∞ as n → ∞. This implies that N(t) < ∞ for all t < ∞.Also know that

N(∞) = limt→∞

N(t) =∞,

since

P (N(∞) <∞) = P (Xn =∞ some n) = P

( ∞⋃n=1

{Xn =∞}

)= 0.

Distribution of N(t) is obtained using the result that the two events

{N(t) ≥ n} ⇐⇒ {Sn ≤ t}.Therefore

P (N(t) = n) = P (N(t) ≥ n)− P (N(t) ≥ n+ 1)

= P (Sn ≤ t)− P (Sn+1 ≤ t).

If we can find CDF of Sn then we can find PMF of N(t). This can be difficult in generalso often study mean of N(t) and asymptotic properties of N(t).

The mean of renewal process is given by

m(t) = E[N(t)] =∞∑n=1

P (N(t) ≥ n) =∞∑n=1

P (Sn ≤ t).

If X1 has PDF f and CDF F then can show that m satisfies a renewal equation:

m(t) = F (t) +

∫ t

0m(t− x)f(x)dx,

can occasionally solve to get explicit formula for m(t). Can also show that

limt→∞

N(t)

t=

1

µ.(2)

Example 10. Suppose customers arrive at a server according to a Poisson process withrate λ. If customer sees server is busy customer leaves, if server is free the customer isserved. Service times are i.i.d random variables with mean µS. What is the rate of arrivalfor customers that stay?


Answer: Denote an arbitrary service time by S. By memoryless property of exponentialdistribution the time between arrival of staying customers is

S +X,

where X is an exponential random variable with mean 1/λ. Therefore the mean timebetween arrivals of customers that stay is

µ = µS + 1/λ =1 + µSλ

λ.

If N(t) is number of customer that stay that have arrived by time t then N(t) is a renewalprocess and from (2) we see that long run average arrival rate is 1/µ.

4.1. Renewal-Reward Processes. Let {N(t); t ≥ 0} be a renewal process with inter-arrival times {Xn}n≥1. At nth renewal we receive an award Rn, we assume that thesequence {(Xn, Rn)}n≥1 is an i.i.d sequence. The total reward earned by time t is given by

R(t) =

N(t)∑n=1

Rn.

A good analogy is to think of a taxi-cab. They drive around and pick up a customer, atthe end of the ride they are given a reward (note that the reward is correlated with thetime between pickups).

Renewal-Reward processes are useful because of the following theorem.

Theorem 11. If E[X1] <∞ and E[R1] <∞ then

(i) limt→∞R(t)t = E[R]

E[X] .

(ii) limt→∞E[R(t)]

t = E[R]E[X] .

Terminology: Say a ‘cycle’ occurs with every renewal, then this theorem says long runaverage reward is

E[reward during cycle]

E[length of cycle].

Example 12. Lifetime of a car is a random variable with PDF f and CDF F . Mr. Smithhas a policy to purchase a new car if

(i) old one breaks down(ii) new one reaches T years of age.

A new car costs C1 and a breakdown costs him an extra C2. What is Mr. Smith’s long runaverage cost?

Answer: From theorem, long run average cost is given by

E[reward during cycle]

E[length of cycle]=E[C(T )]

E[L(T )].


We first find expected cost during cycle, E[C(T )]. If X is lifetime of car during cycle then

C(T ) =

{C1, X > T

C1 + C2, X ≤ T.

Therefore

E[C(T )] = C1P (X > T ) + (C1 + C2)P (X ≤ T ) = C1 + C2F (T ).

We now calculate the length of the cycle:

L(T ) =

{X, X ≤ TT, X > T.

Therefore

E[L(T )] =

∫ T

0xf(x)dx+ T

∫ ∞T

f(x)dx =

∫ T

0xf(x)dx+ T (1− F (T )).

We can now calculate long run average cost as

C1 + C2F (T )∫ T0 xf(x)dx+ T (1− F (T ))

.

Example 13. Manufacturing process sequentially produces items. Goal is to minimizenumber of defective items shipped. Use following procedure for quality control:

(i) Initially inspect every item. Do this until k consecutive acceptable items are found.(ii) Afterwards inspect each item with probability α ∈ (0, 1). Continue until defective is

found.(iii) Return to (i).

Each item is defective with probability q ∈ (0, 1). Want to answer two questions:

(1) What fraction of items are inspected?(2) If defective items are removed, what fraction of remaining items are defective?

Answer this question using renewal-reward theorem. Say cycles begin each time 100%inspection begins, and reward is number of items inspected in cycle. Then from renewal-reward theorem

frac. of items inspected =E[no. inspected in cycle]

E[no. produced in cycle].

Let Nk be the number inspected until k consecutive successes. After Nk need a Geom(q)number of inspections to find defectives so

E[no. inspected in cycle] = E[Nk] + 1/q.

After Nk there will be Geom(αq) items produced until first defective found so

E[no. produced in cycle] = E[Nk] + 1/(αq).


It remains to find E[Nk]. Can set up a recursive equation to find that

E[Nk] =1

p+

1

p2+ . . .+

1

pk=

(1/p)k − 1

q,

where p = 1− q. Therefore

frac. of items inspected = PI =(1/p)k

(1/p)k − 1 + 1/α.

We now answer what fraction of non-removed items are defective. First note that in alarge number N of items roughly Nq are defective. Furthermore we inspect roughly NPIitems, of which NqPI are defective. Therefore we have roughly Nq(1 − PI) undiscovereddefectives. After our inspection we will have N(1−qPI) non-removed items. After N itemswe will have roughly the following fraction of defectives in our non-removed items

Nq(1− PI)N(1− qPI)

If we send N → ∞ the approximations become exact and the long run proportion of nonremoved items that are defective is

q(1− PI)(1− qPI)

.

5. Queueing Systems

5.1. Structure of Queueing Systems. Customers requiring service enter queueing sys-tem where they are either served or join a queue to wait for service. After service thecustomers may leave the system or may require additional service at another (or same)server.

• Customers - entity that enters system requiring service.– Time of customer arrivals can be deterministic or governed by a stochastic

process. If arrivals are stochastic, common model for arrivals times is a Poissonprocess. Sometimes use a general renewal process.

– Customers may arrive as individuals or in various sized batches. Batch sizescan be deterministic or random.

• Queue - line where customers wait.– Important property is finiteness of queue, is there a finite limit to the size of

the queue? Always some upper bound on possible size of queue, but can weassume it is infinite?

– Service discipline - how is the line ordered? Common choices - first come firstserve (FCFS), last come first serve, random, or priority.

• Service - what steps are necessary for customers to complete service and exit sys-tem?

– Simplest scenario - a single server and customer exits.


∗ Important property is how long does the service time take? Often as-sume service times are i.i.d. random variables from a non-negative dis-tribution. Most common choice is exponential, used for simplicity notrealism.

– More complex scenarios - multiple servers connected in a network, called aqueueing network.

Example 14. Emergency room at a hospital has one doctor on staff, patients arrive atrandom. Customers are patients, and server is doctor. Possible models for service disciplineare either FCFS or some form of priority discipline. Patients arrive according to a Poissonprocess with rate λ and service times are exponential with mean 1/µ. This is an exampleof the M/M/1 queue.

6. Queueing in Real Life

Queueing is used in real world everyday. For example they are used in commercialservice systems such as bank tellers, checkout stands at grocery stores, cafeteria, airportticketing, etc. Also used in transportation: tollbooths, traffic lights, ships unloading at adock, airplanes waiting to take off or land at airport, parking lots, etc. Also used to modelsystems inside companies: materials-handling systems, maintenance systems, inspectionsystems, etc.

Below is a list of some notable examples.

• In 1970’s Xerox had backlog for technicians to repair copying machines. Researchersused queueing theory to see how to redeploy technicians to reduce waiting times.Recommendations resulted in a substantial decrease in customer waiting times andan increase in technician utilization. See Interfaces, November 1975 Part 2.• in 1980’s United Airlines needed to figure out how to assign employees at its 11

reservations offices and 10 largest airports. Each checkin counter at an airport wasmodeled as a queueing system, and queueing theory was used to find minimumnumber of employees at each counter. Finding improved distribution of employeesresulted in annual savings of $6 million. (Interfaces, January 1986.• L.L. Bean used to do most of it’s business via phone orders based on its mail catalog.

Call center where orders are placed can be viewed as a queuing system. Customersare callers trying to order things, and servers are telephone agents taking orders.L.L. Bean used queueing theory to analyze their call center to answer questionslike 1) How many trunk lines should be provided for incoming calls? 2) How manyagents should be employed at various times? 3) How many hold positions shouldbe made available?• Queueing theory was used to analyze routing of air tankers in fighting forest fires

of Ontario. In this setting customers are forest fires and servers are air tankers, seeOR/MS Today Volume 40, Number 2.

6.1. Queueing: Fundamental Results. Interested in quantities for t > 0

• X(t) = # in system at time t


• Q(t) = # in queue at time t.

These stochastic processes can be quite difficult to study, so often settle for long runaverages (which we assume to exist):

L = avg. no. of cust. in system = limt→∞

1

t

∫ t

0X(s)ds,

and

LQ = avg. no. of cust. in queue = limt→∞

1

t

∫ t

0Q(s)ds.

Also interested in waiting times. Define

Wi = time cust. i spends in syst.

Wi,Q = time cust. i spends in queue.

Again, settle for long run averages:

W = limn→∞

1

n

n∑i=1

Wi

WQ = limn→∞

1

n

n∑i=1

Wi,Q.

Little’s Law is a very useful relationship that relates L and W . Define the arrival time ofcustomer i by Ti and denote the number of customers that have arrived by time t by N(t)then

1

t

∫ t

0X(s)ds =

1

t

∫ t

0

N(t)∑i=1

1{Ti≤s≤Ti+Wi}(s)ds ≈1

t

N(t)∑i=1

Wi

=N(t)

t

1

N(t)

N(t)∑i=1

Wi.(3)

If customers arrive according to a renewal process with inter arrival times with mean 1/λthen Renewal theorem tells us that

limt→∞

N(t)

t= λ.

Therefore taking t→∞ in (3) we see Little’s Law:

L = λW.(4)

Can also show that LQ = λWQ and

average no. cust. in service = λ ∗ E[S],

where S is random service time.


Another way to derive Little’s Law is the following. Assume customers arriving at thesystem are forced to pay $ to the system then

average earning rate of system = λ× average amount customer pays.(5)

Little’s Law follows by charging customers $1 per unit time in the system, number in queueversion follows by charging customers for time spent in queue.

6.2. PASTA Property. Long run fraction of time with k customers:

Pk = limt→∞

1

t

∫ t

01{X(s)=k}ds.

What about perspective of arriving customers:

ak = limn→∞

1

n

n∑i=1

1{X(Ti)=k}.

Does ak = Pk?

Example 15. Suppose a single server serves customers. All customers have service timeof 1 minute. Time between arrivals is uniformly distributed between 1 and 2 minutes. Iffirst customer arrives at t = 0 then they depart at t = 1. Second customer arrives at sometime t ∈ (1, 2) and finds system empty. Third customer finds system empty. Thereforea0 = 1, since all customers find the system empty, however P0 6= 1 since system is notalways empty.

If arrival process is Poisson then can rule out this situation which is the PASTA Property(Poisson Arrivals See Time Averages).

Theorem 16. If the arrival process is a Poisson process and the sequences {ak}k≥1 and{Pk}k≥1 exist then Pk = ak for all k ≥ 1.

Rough idea: Since Poisson process has independent increments knowing that an arrivaloccurred at time t tells us nothing about what happened prior to time t.

6.3. Exponential Models. Simplest queueing models have exponentially distributed ser-vice times and Poisson arrivals. Specifically, customers arrive at a single server accordingto a Poisson process with rate λ. Upon arrival customer enters service if server is available,if not customer joins the queue. After completing service the customer leaves the system.Service times are i.i.d exponential random variables with mean 1/µ. This system is calledan M/M/1 queue, M is for Markovian (or memoryless), and 1 represents that there is asingle server.

Will find Pn for n ≥ 0. General idea: the rate at which the system enters state n is thesame as the rate at which the system leaves state n. System enters state 0 at rate µP1, andleaves at rate λP0, so therefore

λP0 = µP1.


For n ≥ 1, system leaves state n at rate (λ+µ)Pn and enters at rate λPn−1 +µPn+1. Thusfor n ≥ 1 we have

(λ+ µ)Pn = λPn−1 + µPn+1.

Can iterate this system to obtain that

Pn =

(λ

µ

)nP0.

Solve for P0 with condition that probabilities must some to 1:

1 =∞∑n=0

Pn = P0

∞∑n=0

(λ

µ

)n=

P0

1− λ/µ,

or

P0 =µ− λµ

Pn =

(λ

µ

)n(µ− λµ

)n ≥ 1.

Note that these probabilities only exist if λ < µ, i.e., arrival rate is strictly less than servicerate. If arrival rate is greater than service rate number of customers in system will growwith time and can’t define Pn.

We can use {Pn}n≥1 to obtain the long run average number of customers in the system:

L =

∞∑n=1

nPn =

∞∑n=1

n

(λ

µ

)n(µ− λµ

)=

λ

µ− λ.

We can now use Little’s Law to find the average wait:

W = L/λ = 1/(µ− λ).

If we want instead WQ we use the fact that W = WQ + E[S] = WQ + 1/µ, this gives

WQ =λ

µ(µ− λ).

6.3.1. Birth Death Queueing Models. M/M/1 queue is a special case of a class of modelscalled birth-death queueing models. In these models the system state increases from n atrate λn and decreases from n at rate µn.Examples:

• M/M/1 queue with arrivals at rate λ and service at rate µ.

λn = λ n ≥ 0

µn = µ n ≥ 1


• M/M/1 queue with arrivals at rate λ, service at rate µ and balking. Specifically, ifa customer see n customers present then they leave (balk) with probability 1− αn

λn = λαn n ≥ 0

µn = µ n ≥ 1

• M/M/k queue with arrivals at rate λ, and k servers each with service rate µ. Inthis model arriving customers enter service if any of the k servers are free, otherwisethey wait in line.

λn = λ n ≥ 0.

Total service rate:

µn =

{nµ, n ≤ kkµ, n > k

.

Can find limiting probabilities for birth death queueing models using principle that rate in= rate out. In particular get the equations

λ0P0 = µ1P1

(λn + µn)Pn = λn−1Pn−1 + µn+1Pn+1 n ≥ 1.

Solve these equations to get that

P0 =

(1 +

∞∑n=1

λ0λ1 · · ·λn−1µ1µ2 · · ·µn

)−1,

and

Pn =λ0λ1 · · ·λn−1µ1µ2 · · ·µn

P0.

6.4. General Queueing Systems. Poisson process is often a reasonable model arrivalprocess, but exponential distribution is rarely appropriate for modeling service times.Therefore people often study the M/G/1 distribution, here G stands for general. Firstestablish a result about work in system

V (t) = time required to empty the system if no more arrivals

= work in system = virtual waiting time.

Denote the sequence of service, arrival times, and time in queue by {Si}i≥1 , {Ti}i≥1 and{Wi,Q}i≥1 respectively. Then

V (t) =

∞∑i=1

Si1{Ti<t<Ti+Wi,Q} +∞∑i=1

(Ti +Wi,Q + Si − t)1{Ti+Wi,Q<t<Ti+Wi,Q+Si}.(6)

Instead of studying V (t) study simpler quantity

V = limT→∞

1

T

∫ T

0V (s)ds.


Integrate RHS of (6) over [0, T ]:∫ T

0

[ ∞∑i=1

Si1{Ti<t<Ti+Wi,Q} +

∞∑i=1

(Ti +Wi,Q + Si − t)1{Ti+Wi,Q<t<Ti+Wi,Q+Si}

]dt

=∞∑i=1

∫ T

0

[Si1{Ti<t<Ti+Wi,Q} + (Ti +Wi,Q + Si − t)1{Ti+Wi,Q<t<Ti+Wi,Q+Si}

]dt

=

N(T )∑i=1

∫ T

0

[Si1{Ti<t<Ti+Wi,Q} + (Ti +Wi,Q + Si − t)1{Ti+Wi,Q<t<Ti+Wi,Q+Si}

]dt

≈N(T )∑i=1

(SiWi,Q + S2

i /2).

Therefore

1

T

∫ T

0V (t)dt ≈ 1

T

N(T )∑i=1

(SiWi,Q + S2

i /2)

=N(T )

T

1

N(T )

N(T )∑i=1

(SiWi,Q + S2

i /2).

Assume that {N(t); t ≥ 0} is a renewal process with rate λ and that

limn→∞

1

n

n∑i=1

SiWi,Q = ¯SWQ

limn→∞

1

n

n∑i=1

S2i = S̄2.

Then we conclude that

V = λ[

¯SWQ + S̄2/2],

and if we assume service times are i.i.d and independent of arrival process then can simplifyto

V = λ

[E[Si]W̄Q +

1

2E[S2]

].

If we assume that arrivals occur according to a Poisson process then we can use theprevious formula to derive explicit expression for W̄Q. Note if arrivals occur according toa Poisson process and services are i.i.d random variables independent of arrivals then thequeueing system is an M/G/1 queue.

By definition Wi,Q = V (Ti) and therefore

1

n

n∑i=1

Wi,Q =1

n

n∑i=1

V (Ti).


Since arrivals are Poisson can use the PASTA property to see that

W̄Q = limn→∞

1

n

n∑i=1

V (Ti) = limt→∞

∫ t

0V (s)ds = λ

[E[Si]W̄Q +

1

2E[S2]

].

Solve the above equation for W̄Q to get that

W̄Q =λE[S2]

2(1− λE[S]),

known as Pollaczek-Khinchine formula. Can use Little’s Law to get value of average numberin system, average wait time, and average number in queue:

W = W̄Q + E[S]

LQ = λW̄Q

L = λW.

de nition 1. a continuous random variable is said to have ... continuous random variable xis said to...

Documents