st3236: stochastic process tutorial 4 ta: mar choong hock email: [email protected] exercises: 5
TRANSCRIPT
Question 1An urn initially contains a single red and single green ball. A ball is drawn at random, removed and replaced by a ball of the opposite color and this procedure repeats so that there are always two balls in the urn.
Let Xn be the number of red balls in the urn after n draws, with X0 = 1.
Specify the transitions probabilities for MC {X}.
Question 1Case (Xn=0): Both balls are green. One ball will certainly be replaced with red after a ball is drawn.
P(Xn+1 = 1 | Xn = 0) = 1, P(Xn+1 = 0 | Xn = 0) = 0, P(Xn+1 = 2 | Xn = 0) = 0
Case (Xn=2): Both balls are red. One ball will certainly be replaced with green after a ball is drawn.
P(Xn+1 = 1 | Xn = 2) = 1, P(Xn+1 = 0 | Xn = 2) = 0, P(Xn+1 = 2 | Xn = 2) = 0
Question 1Case (Xn = 1): One ball is red. Outcome dependent on the colour of the ball drawn.
Note: P(Green is drawn|Xn=1) = P (Red is drawn|Xn=1) = 0.5
P(Xn+1 = 0 | Xn = 1) = P(Red is drawn|Xn = 1) = 0.5 P(Xn+1 = 2 | Xn = 1) = P(Green is drawn|Xn = 1) = 0.5 P(Xn+1 = 1 | Xn = 1) = 0
Question 1
The transition matrix is given as follows:
Question 2Find the mean time to reach state 3 starting from state 0 for the MC whose transition probability matrix is
Question 2
Let T = min{n : Xn = 3} and vi = E(T | X0 = i). The mean time to reach state 3 starting from state 0 is v0. We apply first step analysis.
i0 3
1 step vi
Recursive Formula for average time step required:
i
iio vpv 01
Question 2Therefore,
v0 = 1 + 0.4v0 + 0.3v1 + 0.2v2 + 0.1v3
v1 = 1 + 0v0 + 0.7v1 + 0.2v2 + 0.1v3
v2 = 1 + 0v0 + 0v1 + 0.9v2 + 0.1v3
v3 = 0
Solving the equations, we havev0 = 10.
Question 3Consider the MC with transition probability matrix
(a) Starting in state 1, determine the probability that the MC ends in state 0
(b) Determine the mean time to absorption.
Question 3a
Let T = min{n : Xn = 0 and Xn = 2}, ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
u0 = 1u1 = 0.1u0 + 0.6u1 + 0.3u2
u2 = 0
we have u1 = 0.25.
Question 3b
Let T = min{n : Xn = 0 and Xn = 2}, ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
v0 = 0v1 = 1 + 0.1v0 + 0.6v1 + 0.3v2
v2 = 0
we have v1 = 2.5.
Question 4Consider the MC with transition probability matrix
(a) Starting in state 1, determine the probability that the MC ends in state 0
(b) Determine the mean time to absorption.
Question 4aLet T = min{n : Xn = 0 and Xn = 2}, ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
u0 = 1u1 = 0.1u0 + 0.6u1 + 0.1u2 + 0.2u3
u2 = 0.2u0 + 0.3u1 + 0.4u2 + 0.1u3
u3 = 0
we have,u1 = 0.3810, u2 = 0.5238
Starting in state 1, the probability that the MC ends in state 0 is u1 = 0.3810
Question 4b
Let T = min {n : Xn = 0 and Xn = 2}, ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
v0 = 0v1 = 1 + 0.1v0 + 0.6v1 + 0.1v2 + 0.2v3
v2 = 1 + 0.2v0 + 0.3v1 + 0.4v2 + 0.1v3
v3 = 0
we have v1 = v2 = 3.33.
Question 5
A coin is tossed repeatedly until two successive heads appear. Find the mean number of tosses required.
[Hint: Let Xn be the cumulative number of successive heads. Then the state space is 0,1,2 before stop]
Question 5 – Method 1Let Yn be the outcome {H, T} of each toss and(Yn-1, Yn) denotes the sample point for the sucessivetosses. There are 4 possible sample points.
HH
TH
HT
TT
Z
Denote
n
,3
,2
,1
,0
:
0(T,T)
1(T,H)
3(H,H)
P(Yn=T)=0.5
1
Denote: States = {(Yn-1, Yn)}Where Yn = H if head at nth toss and T if otherwise.
2(H,T)
P(Yn=T)=0.5
P(Yn=T)=0.5
P(Yn=H)=0.5
P(Yn=H)=0.5
P(Yn=H)=0.5
Question 5 – Method 1The transition probability matrix is
1000
005.05.0
5.05.000
005.05.0
P
Question 5 – Method 1Let vi denote the mean time to each state 3 starting from state i.
By the first step analysis, we have the following equations:
v0 = 1 + 0.5v0 + 0.5v1
v1 = 1 + 0.5v2 + 0.5v3
v2 = 1 + 0.5v0 + 0.5v1
v3 = 0 Therefore, v0 = 6 (v1 = 4, v2 = 6)
Question 5 – Method 2Let Xn be the cumulative number of successive heads. The 3-state state space now is {0,1,2}.
Example:
n 0 1 2 3 4
Yn T H T H H
Xn 0 1 0 1 2
Question 5 – Method 2
Case (Xn = 0) :Previous two tosses are tails Or a head followed by tail. Note: P(Head) = P(Tail) = 0.5
P(Xn+1 = 1 | Xn = 0) = P(Head) = 0.5, P(Xn+1 = 0 | Xn = 0) = P(Tail) = 0.5, P(Xn+1 = 2 | Xn = 0) = 0
Case (Xn = 1): A tail is followed by a headP(Xn+1 = 2 | Xn = 1) = P(Head) = 0.5P(Xn+1 = 0 | Xn = 1) = P(Tail) = 0.5 P(Xn+1 = 1 | Xn = 1) = 0
Question 5 – Method 2
Case (Xn = 2): Previous two tosses are heads. We make this state absorbing.
P(Xn+1 = 1 | Xn = 2) = 0P(Xn+1 = 2 | Xn = 2) = 1, P(Xn+1 = 0 | Xn = 2) = 0
Question 5 – Method 2
The transition probability matrix is
Question 5 – Method 2Let vi denote the mean time to each state 2 starting from state i.
By the first step analysis, we have the following equations:
v0 = 1 + 0.5v0 + 0.5v1
v1 = 1 + 0.5v0 + 0.5v2
v2 = 0
Thus, we have,v0 = 6, v1 = 4