st3236: stochastic process tutorial 5 ta: mar choong hock email: g0301492@nus.edu.sg exercises: 6

ST3236: Stochastic ProcessTutorial 5

TA: Mar Choong Hock

Email: g0301492@nus.edu.sg

Exercises: 6

Question 1Consider the MC with transition probability matrix

Starting in state 1, determine the mean time that the process spends in state 1 prior to absorption and the mean time that the process spends in state 2 prior to absorption. Verify that the sum of these is the mean time to the absorption.

Question 1Denote by wi1 the mean time the process spends in

state 1 starting in state i prior to the absorption. We have,

w01 = 0w11 = 1 + 0.1w01 + 0.2w11 + 0.5w21 + 0.2w31

w21 = 0 + 0.1w01 + 0.2w11 + 0.6w21 + 0.1w31

w31 = 0

The solution isw01 = 0, w11 = 1.8182, w21 = 0.9091, w31 = 0.

Question 1Similarly, denote by wi2 the mean time the process spends in state 2 starting in state i prior to the absorption. We have,

w02 = 0w12 = 0 + 0.1w02 + 0.2w12 + 0.5w22 + 0.2w32

w22 = 1 + 0.1w02 + 0.2w12 + 0.6w22 + 0.1w32

w32 = 0

The solution isw02 = 0, w12 = 2.2727, w22 = 3.6364, w32 = 0.

Question 1Denote by vi the mean time to the absorption

starting in state i prior to the absorption. We have,

v0 = 0v1 = 1 + 0.1v0 + 0.2v1 + 0.5v2 + 0.2v3

v2 = 1 + 0.1v0 + 0.2v1 + 0.6v2 + 0.1v3

v3 = 0

The solution is v0 = 0, v1 = 4.0909, v2 = 4.5455, v3 = 0.We have verified that,

v1 = w11 + w12

Question 2Consider the MC with transition probability matrix

Starting in state 1, determine the mean time that the process spends in state 1 prior to absorption and the mean time that the process spends in state 2 prior to absorption. Verify that the sum of these is the mean time to the absorption.

Question 2Denote by wi1 the mean time the process spends in state 1 starting in state i prior to the absorption. We have,

w01 = 0w11 = 1 + 0.5w01 + 0.2w11 + 0.1w21 + 0.2w31

w21 = 0 + 0.2w01 + 0.1w11 + 0.6w21 + 0.1w31

w31 = 0

The solution is,w01 = 0, w11 = 1.290, w21 = 0.3225, w31 = 0.

Question 2Similarly, denote by wi2 the mean time the process spends in state 2 starting in state i prior to the absorption. We have,

w02 = 0w12 = 0 + 0.5w02 + 0.2w12 + 0.1w22 + 0.2w32

w22 = 1 + 0.2w02 + 0.1w12 + 0.6w22 + 0.1w32

w32 = 0

The solution isw02 = 0, w12 = 0.3230, w22 = 2.5808, w32 = 0.

Question 2Denote by vi the mean time to the absorption starting in state i prior to the absorption. We have,

v0 = 0v1 = 1 + 0.5v0 + 0.2v1 + 0.1v2 + 0.2v3

v2 = 1 + 0.2v0 + 0.1v1 + 0.6v2 + 0.1v3

v3 = 0

The solution is v0 = 0, v1 = 1.613, v2 = 2.9033, v3 = 0.We have verified that,

v1 = w11 + w12

Question 3

Consider the MC in question 1. Starting in state 1, determine the probability that the process is absorbed into state 0. Compare this with the (1,0)th entry in the matrix powers P2, P4, P8 and P16.

Question 3

Denote by ui the probability that the MC is absorbed by 0 starting in state i. We have,

u0 = 1u1 = 0.1u0 + 0.2u1 + 0.5u2 + 0.2u3

u2 = 0.1u0 + 0.2u1 + 0.6u2 + 0.1u3

u3 = 0

The solution is,u0 = 1, u1 = 0.4091, u2 = 0.4545, u3 = 0.

Compare:u1 = 0.4091

Consider a (4 x 4) transition probability matrix,

By definition,

1

01 10t

t XXPu

Question 3

10

...

...

...

...

....

....

....

;

...

...

...

023013201210111000210

30

20

10

00

131211102

30

20

13121110

00

XXPppppppppp

p

p

p

p

pppp

p

p

pppp

p

P

P

But for our case, p00 = 1, p03 = 0. Question 3

2

10

2012101110

3013201210111000210

10t

t XXP

ppppp

ppppppppp

1 21 p11 p22

p10

p20

30 1

p21

p12

p13

p23

Let:

1.F(t) be the set of t-step first passage paths from state 1 to state 0

2.G(n-t) be the set of (n-t)-step paths from state 0 to state 0

3.H(t) be the set of paths that is formed jointly by F(t) followed by G(n-t). Note: paths are n-step.

Question 3-Optional

Question 3-Optional

01 0G(n-t)F(t)

H(t)

Let L(n) be the set of n-step paths from state 1

to state 0. s.t.

Question 3-Optional

n

ttn

n

t

n

t

n

t

n

t

n

t

n

t

n

t

XXPtf

tnGPtFPtnGPtFP

tnGtFPtHP

tHPtHPnLP

tHnL

10,1

11

11

11

1

00

,)(

)(

npnLP 10

f1,0(t) is the t-step first passage probability from

state 1 to state 0.

If state 0 is an absorbing state,

Also, trivially,

Question 3-Optional

100 tn XXP

10 00,1 XXPtf t

n

tt

n

ttn

n

XXP

XXPtf

pnLP

10

10,1

10

10

00

Question 3-Optional

10

101 10

p

XXPut

t

Question 4

Which of the following MC is regular:

a)

b)

Question 4

a) YES, because

(all entries are greater than 0)

b) NO, because it has absorbing states