Lecture 13 – Continuous-Time Markov Chains

Topics• Markovian property• Exponential distribution• Rate matrix• ATM Example• Birth and death processes• Queuing systems

Markovian Property for CTMCsStochastic process: {Yt : t 0 }, where Yt is a nonnegative

integer

CTMC is similar to that of a DTMC except “one-step” has no meaning in continuous time so the Markovian property must hold for all future times instead of just for one step.

Interpretation: First equation says that the conditional distribution of the future Yt+s given the present Ys and the past Yu, 0 ≤ u ≤ s, depends only on the present and is independent of the past. Second equation says that Pr{Yt+s = j | Ys = i } is independent of s.

Definition 1: The process Y = {Yt : t ≥ 0 } with state space S is a CTMC if the following condition holds for all j S, and t, s ≥ 0

Pr{Yt+s = j | Yu , u ≤ s } = Pr{Yt+s = j | Ys }.

In addition, the chain is said to have stationary transitions if

Pr{Yt+s = j | Ys = i } = Pr{Yt = j | Y0 = i }.

Example of Definition 1• Problem: Suppose that a CTMC enters state i at, say, time 0

and does not leave during the next 15 minutes; i.e., a transition does not occur.

• Question: What is the probability that a transition will not occur in the next 5 minutes?

• Approach: Markovian property tells us that the probability that the process will remain in state i during the interval [15, 20] is just the unconditional probability that it stays in state i for at least 5 minutes.

• Solution: Let Ti denote the amount of time that the process stays in state i before making a transition into a different state. Then

Pr{ Ti > 20 | Ti > 15 } = Pr{ Ti > 5 }

or, in general, Pr{Ti > s + t | Ti > s } = Pr{ Ti > t }

for all s, t ≥ 0. Hence, the random variable Ti is memoryless and so is exponentially distributed.

Generalization of ExampleThe Markovian property gives

Pr{ Ti > s + t | Ti > s } = Pr{ Ti > t }

for all s, t ≥ 0.

Implication: The random variable Ti is memoryless and thus is exponentially distributed.

Alternative definition of CTMC: A stochastic process having the properties that each time it enters state i,

(i) the amount of time it spends in that state before making a transition into a different state is exponentially distributed with mean, say, 1/i , and

(ii) when the process leaves state i it next enters state j with some probability, say, pij , where pij must satisfy

pii = 0, for all i S

j pij = 1, for all i S

ATM Example (max of 5 in system)

Statistics

• Average time between arrivals = 30 sec (0.5 min): = 2/min

• Average service time = 24 sec (0.4 min): = 2.5/min

Design questions

• How many ATMs should there be?

• Should the foyer be expanded?

State-transition network

5

a

d

4

a

d

30 1 2

a a a

d d d

Exponential Distribution

pdf: f (t ) = e–t for t ≥ 0

CDF: F (t ) = 1 – e–t for t ≥ 0

Parameters:Mean = 1/Var = (1/ )2

f(t)

0

0.1

0.2

0.3

0.4

0.5

0.6

0 0.5 1 1.5 2 2.5 3 3.5 4

t

Exponential distribution with Mean = 0.5 ( = 2)

Poisson Process

Pr{ k arrivals in time t } = for k = 0, 1,…

For the ATM example with = 2, the expected number of arrivals in the interval [0, t ] is 2t.

( )!

k tt e

k

λλ −

When the duration of the time between events is exponentially distributed, the number of occurrences of the event in a given time interval has a Poisson distribution.

Rate Diagram

• For CTMCs, activities are better represented by their rate of occurrence, so rather than using a state-transition network we use a rate diagram or rate network.

• This network is easily constructed from the state-transition network by replacing the activity designation by the activity rate.

5430 1 2

µ µ µ

λ

µ

λ

µ

ATM Network

Transient analysis

Rate MatrixA computationally more convenient alternative to the rate diagram is the rate matrix R whose element rij is the transition rate from state i to j. In general,

rij = pij or rij = pij

0 0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0 0

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥

=⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

R

Rate matrix for ATM example

General rate matrix

Transient Analysis

• Determine the probability that the system will be in a particular state at time t.

• The transient probabilities are a function of the initial state.

• Unconditional probability vector:

q(t ) = (q0(t ), q1(t ), q2(t ),…,qm-1(t )) • Requirement:

1

0

( ) 1m

ii

q t−

=

=∑

Transient Analysis (cont’d)

• For some small interval of time ∆, let n = t/∆ be the number of steps or increments required to represent t.

• The transient solution of the process can be approximated at time t = n∆ with a DTMC by solving the following equation:

q(n∆) = q(0)P(n) or q(n∆+∆) = q(n∆)P

where P is a state-transition matrix determined from the rate matrix R.

Transition Matrix for Transient Analysis

Let i be the sum of all transition rates out of state i ; that is,

and let pij rij. Then

1

0

m

i ijj

r−

=

=∑

0 01 02 0, 1

10 1 12 1, 1

1,0 1,1 1,2 1

1

1

1

m

m

m m m m

r r r

r r r

r r r

−

−

− − − −

− … ⎡ ⎤⎢ ⎥ − … ⎢ ⎥=⎢ ⎥…⎢ ⎥ … −⎢ ⎥⎣ ⎦

PM M M M

Transition Matrix for ATM Example

1 0 0 0 0

1 ( ) 0 0 0

0 1 ( ) 0 0

0 0 1 ( ) 0

0 0 0 1 ( )

0 0 0 0 1

− ⎡ ⎤⎢ ⎥ − + ⎢ ⎥⎢ ⎥ − +

=⎢ ⎥ − + ⎢ ⎥⎢ ⎥ − + ⎢ ⎥

−⎢ ⎥⎣ ⎦

P

Rate network

1 2 2 0 0 0 0

2.5 1 4.5 2 0 0 0

0 2.5 1 4.5 2 0 0

0 0 2.5 1 4.5 2 0

0 0 0 2.5 1 4.5 2

0 0 0 0 2.5 1 2.5

− ⎡ ⎤⎢ ⎥ − ⎢ ⎥⎢ ⎥ −

=⎢ ⎥ − ⎢ ⎥⎢ ⎥ − ⎢ ⎥

−⎢ ⎥⎣ ⎦

Transient Analysis for ATM Example• Assume system is empty at t = 0.

• We wish to approximate the transient probabilities at t = 1 min.

• Initial probability vector: q(0) = (1, 0, 0, 0, 0, 0)

• Use equation q(n∆) = q(0)P(n)

• Number of steps: n = t/∆ = 1/∆

– Case 1: ∆ = 0.05 n = 20 steps (1 min)

q(20∆) = q(1) = (0.433, 0.291, 0.162, 0.075, 0.029, 0.011)

– Case 2: ∆ = 0.025 n = 40 steps (1 min)

q(40∆) = q(1) = (0.435, 0.291, 0.160, 0.073, 0.029, 0.011)

(almost identical)

Transient Solution for ATM Example (∆ = 0.05, 0 t 1)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2 4 6 8 10 12 14 16 18 20

Steps

State 0State 1State 2State 3State 4State 5

0

Transient probabilities

Steady-State Solutions• Definition: The probability that the system is in

state i is constant (independent of initial conditions).

• Steady-state probability for state i : iP = limtq(t )

• Vector:

• Calculations in Chapter 15: must solve m simultaneous linear equations in m unknowns.

• ATM example:

– After 1 min with ∆ = 0.25

q(1) = (0.435, 0.291, 0.160, 0.073, 0.029, 0.011)

– In the limit

P = (0.271, 0.217, 0.173, 0.139, 0.111, 0.089)

( )P P P P1 2 1, ,..., mπ π π π −=

Transient Computations for ATM Example with ∆ = 0.025

Steps, n Time (min)

q0

q1

q2

q3

q4

q5

0 0 1 0 0 0 0 0

40 1 0.435 0.291 0.160 0.073 0.029 0.011

80 2 0.348 0.258 0.175 0.110 0.066 0.042

120 3 0.311 0.239 0.175 0.124 0.087 0.063

160 4 0.292 0.228 0.175 0.131 0.098 0.075

200 5 0.282 0.223 0.174 0.135 0.104 0.082

240 6 0.277 0.220 0.174 0.137 0.107 0.085 M M M M M M M M Steady state ∞ 0.271 0.217 0.173 0.139 0.111 0.089

System Statistics • Provide managerial insights

• Evaluate system performance and quality of service

• Evaluate design options

• ATM Example:– Proportion of time ATM is idle:

– Efficiency (proportion of time busy):

– Proportion of customers rejected:

– Proportion of customers who wait:

– Expected number in system:

P0 0.271 =

P01 0.729− =

P5 0.089 =

P P0 51 0.64 − − =

5 P

01.868ii

i=

=∑

ATM Example (cont’d)

– Expected number in queue:

– Throughput rate (average number passing

through the system):

– Balking rate (average number of customers

lost):

– Average time in system (given by Little’s

law):

( )5 P

11 1.139ii

i π=

− =∑

( )P51 1.822λ π− =

P5 0.178 =

average number in system 1.8681.025 min

throughput rate 1.822= =

ATM Design Alternatives

• Performance summary (contradictory?)

– Busy 73% of time

– Space in foyer less than 40% utilized; that is,

(average no. in systems / 5) 100% = 37.36%

– 9% of customers lost

– Average wait in queue = 60(1.139/1.822) = 37 sec

• Options

– Add machines

– Expand size of foyer

– Add human teller

Add More ATMs

5430 1 2

µ 2µ

λ λ

3µ 3µ 3µ

Rate diagram for 3 ATMs:

= 2, = 2.5

Alternative

System measures ATM1 ATM2 ATM3

Number of machines 1 2 3

Capacity of foyer 5 5 5

Average number in system 1.8683 0.9187 0.8134

Average time in system 1.0252 0.4635 0.4078

Average number in queue 1.1394 0.1258 0.0156

Average time in queue 0.6252 0.0635 0.0078

Throughput rate 1.8224 1.9823 1.9946

Efficiency (utilization) 0.7289 0.3965 0.2659

Proportion who must wait 0.7289 0.224 0.0511

Proportion of customers lost 0.0888 0.0088 0.0027

Comparative analysis

Add Human Teller

• Performance– Average service rate for teller: 1 = 1/min– Average service rate for ATM: 2 = 2.5/min– Arrival rate: = 2/min

• Two-server queuing system– Indices: teller = 1; ATM = 2

• State variables: s = (s1, s2, s3)

Add Human Teller (cont’d)• Events

– Arrival = a

– Service completion for teller = d1

– Service completion for ATM = d2

• State-transition network

0 a

(000)

(100)

2

(010)

(110)

3

a

a(111)

4

(112)

5a a

d1, d2 d1, d2

d1

d2

d1

d2(113)

6

a

d1, d2

1

• Explanation: s = (110); teller and ATM are busy, no customers are waiting.

Add Human Teller (cont’d)

• Event rates– Arrival: = 2/min

– Service completion for teller: 1 = 1

– Service completion for ATM: 2 = 2.5

• Rate diagram

0

2

3 4 5

µ1

6

µ2 λ

λ

λ λ λµ1

µ2

µ1 µ2+ µ1 µ2+ µ1 µ2+

(100)

(010)

(000) (110) (111) (112) (113)1

Add Human Teller (cont’d)

Rate matrix R = (rij)

where rij = transition rate from state i to state j

(000) (010) (100) (110) (111) (112) (113)

(000) 0 0 2 0 0 0 0

(010) 2.5 0 0 2 0 0 0

(100) 1 0 0 2 0 0 0

R = (110) 0 1 2.5 0 2 0 0

(111) 0 0 0 3.5 0 2 0

(112) 0 0 0 0 3.5 0 2

(113) 0 0 0 0 0 3.5 2

Explanation: r43 = 1 + 2 = 1 + 2.5 = 3.5where state 4 = (111) and state 3 =

(110)

Comparisons For ATM Example

Alternative

System measures ATM1 ATM and teller ATM2

Average number in system 1.8683 1.580 0.9187

Average time in system 1.0252 0.8213 0.4635

Average number in queue 1.1394 0.3659 0.1258

Average time in queue 0.6252 0.1902 0.0635

Throughput rate 1.8224 1.9234 1.9823

ATM Efficiency (utilization) 0.7289 0.4731 0.3965

Teller efficiency (utilization) –– 0.7408 ––

Proportion who must wait 0.7289 0.4275 0.224

Proportion of customers lost 0.0888 0.0383 0.0088

Steady-state solution for human teller:

s = (000) (010) (100) (110) (111) (112) (113)

πP = (0.214, 0.046, 0.313, 0.205, 0.117, 0.067, 0.038)

Pure Birth Processes

2 3 4 …a a

1a

0a0 1 2 3

Example: Hurricanes

Rate matrix

(let i be arrival rate for state i )

0

1

2

0 0 0

0 0 0

0 0 0

0 0 0 0

…⎡ ⎤⎢ ⎥…⎢ ⎥⎢ ⎥= …⎢ ⎥…⎢ ⎥⎢ ⎥⎣ ⎦

R

M M M M O

Properties• Markov process if time between arrivals

has exponential distribution

• No steady state [transient probabilities

are governed by Poisson distribution:

pk(t ) = (t )ke-t/k !, k = 0, 1, 2, … ]

• Probability of N (t ) arrivals in time t is

n: Pr{ N (t ) n } = ( )0

!n k t

kt e kλλ −

=∑

Pure Death Processes

Examples• Delivery of packages

• Completion of 10 course study units

2 3 4 …d

1d

0d 41 2 3d

Rate matrix• Let i be completion rate for state i

• State space S = (0,1,…,10}

1

2

3

10

1 0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

…⎡ ⎤⎢ ⎥…⎢ ⎥⎢ ⎥…

=⎢ ⎥…⎢ ⎥⎢ ⎥⎢ ⎥

…⎢ ⎥⎣ ⎦

R

M M M M MSteady state probability vector:

πP = (1,0,…,0)

State 0 is an absorbing state

Pure Death Process Example

• Let = 1 completions per week

• Probability of completing k units in t = 14 weeks:

Completed units, k 10 9 8 7 6 5 4 ≤3

Probability,p10 k(14) 0.891 0.047 0.03 0.017 0.009 0.004 0.001 0.0005

• Assume all units have the same completion rate:

rk,k–1 = µk = µ, k = 1,…,10

• Then transient probabilities are:

p10–k(t ) = (t )ke-t/k !, 0 k < 10, and

p0(t ) = 1 – p1(t ) – · · · – p10(t )

Pure Death Process Example (cont’d)

• Let = 1 completions per week

• Probability of k units remaining in t = 14 weeks:

Incomplete units, k 0 1 2 3 4 5 6 ≥7

Probability,pk(14) 0.891 0.047 0.03 0.017 0.009 0.004 0.001 0.0005

• Transient probabilities for k units remaining:

pk(t ) = (t )10–ke-t/(10–k) !, 0 k < 10,

and

p0(t ) = 1 – p1(t ) – · · · – p10(t )

General Birth and Death Processes

Examples • Repair shop for a taxi company

• Intensive care unit in hospital (turnover of nurses)

2 3 4 …10

d4d1 d2 d3

a0a1 a2 a3

Rate matrix• Assume 7 states

• Typically, and depend on state

• Steady state probabilities, P, will exist

0

1 1

2 2

3 3

4 4

5 5

6

0 0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0 0

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

R

Queuing Systems

Queue Discipline: Order in which customers are served; FIFO, LIFO, Random, Priority

Five Field Notation:

Arrival distribution / Service distribution / Number of servers /Maximum number in the system / Number in the calling population

Inputsource Queue Service

mechanism

DeparturesCustomers

Queuing Notation

Distributions (interarrival and service times)M = ExponentialD = Constant timeEk = ErlangGI = General independent (arrivals only)G = General

Parameterss = number of serversK = Maximum number in system

N = Size of calling population

Characteristics of Queues

Infinite queue: e.g., Mail order company (GI/G/s)

2 s s +1 …a a

1

a

0

a

d 2d sd sd

s –1…

Finite queue: e.g., Airline reservation system (M/M/s/K)

…a

Ksd

K–1 a …a

K

sdK–1

a. Customer arrives but then leaves b. No more arrivals after K

Characteristics of Queues (continued)

Finite input source: e.g., Repair shop for taxi company (N vehicles)

with s service bays and limited capacity parking lot (K – s spaces).

Each repair takes 1 day (GI/D/s/K/N).

In this diagram N = K so we have GI/D/s/K/K system.

2

N

a

1

(N–1)a

0

Na

d 2d

sd

N–1…s

sd

s–1

…

s+1

(N–s)a

sd…

(N–s+1)a

Single Channel Queue – Two Kinds of Service

Bank teller: normal service (d ), travelers checks (c ), idle (i )

Let p = portion of customers who buy travelers checks after normal service

s1 = number in system, where s1 { 0, 1, 2, . . . }

s2 = status of teller, where s2 {i, d, c }

s = (s1, s2)State-transition network

(2,d)(0,i) (1,d) (3,d)

(1,c) (2,c) (3,c)

a

d, 1– p

d, pc

a

a

d, 1– p

c

a

d, 1– p

cd, p d, p

a …

…

Single Channel Queue for Bank (cont’d)

• State transitions w.r.t. customer departures from teller– Current state: s = ( j, d ), j = 1, 2,… (teller busy)– Next state: either

s' = ( j –1, d ), departure with probability 1 – p, ors' = ( j, c ), get checks with probability p

• State transitions w.r.t. customer departures after purchasing travelers checks– Current state: s = ( j, c ), j = 1, 2,… (customer buying checks)– Next state: s' = ( j –1, d ), departure with probability 1

• State transitions w.r.t. customer arrivals– Current state: s = ( j, d or c), j = 1, 2,… (teller or checks busy)– Next state: s' = ( j +1, d or c), arrival with probability 1

Single Channel Queue for Bank (cont’d)• Rate of transitions

– Event: x (arrival or departure)– Rate of event x : x (where a = , d = 1, c = 2)– Conditional probability: p(s,s' | x ) or p(i, j|x )– Computations: rij = xp(i, j |x )

• States -- assume limited no. customers at teller: K = 2s0 = (0,i ), s1 = (1,d ), s2 = (1,c ), s3 = (2,d ), s4 = (2,c )

• Rate matrix

1 1

2

1 1

2

0 0 0 0 (0, )

(1 ) 0 0 (1, )

0 0 0 (1, )

0 (1 ) 0 (2, )

0 0 0 (2, )

i

p p d

c

p p d

c

⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥=⎢ ⎥−⎢ ⎥⎢ ⎥⎣ ⎦

R

Part Processing with Rework• Consider a machining operation in which there is a

0.4 probability that upon completion, a processed part will not be within tolerance.

• Machine is in one of 3 states [ s = { (0), (1), (2) } ]:

0 = idle

1 = working on part for first time

2 = reworking part

Events

a = arrival

d1 = service completion from state 1

d2 = service completion from state 2

(0) (1) (2)

a d1, 0.4

d2

d1, 0.6

State-transition network

Classification of StatesAccessible: Possible to go from state i to state j (path exists in

the network from i to j).

2 3 4 …10

d4d1 d2 d3

a0a1 a2 a3

2 3 4 …a a

1a

0a0 1 2 3

Two states communicate if both are accessible from each other. A system is irreducible if all states communicate.

State i is recurrent if the system will return to it some time in the future after leaving it.

If a state is not recurrent, it is transient.

What You Should Know About Markov Chains

• Definition of a CTMC.• What the difference is between a DTMC

and a CTMC.• What the rate matrix and rate diagram are.• What is meant by a transient solution• What is meant by a steady-state solution.• What a birth-death process is.• Classification of the various types of

queuing systems.