Problems 10/3

1. Ehrenfast’s diffusion model:

P(Xn+1 =k−1Xn =k) =k2a

, k≥1

P(Xn+1 =k + 1Xn =k) =1−k2a

, k < 2a

P =

0 1 0 0 L 0 012a 0 1−1

2a 0 L 0 00 2

2a 0 1−22a L 0 0

L L L L L L L

0 0 0 0 L 1 0

⎛

⎝

⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟

Problems, cont.

2.

Discrete uniform on {0,...,n}

PX|U=u(s) =(1−u+us)n

PX (s) = PX|U=u(s)du0

1

∫ = (1+u(s −1))ndu0

1

∫=

(1+ u(s − 1))n+1

(n + 1)(s − 1)

⎡

⎣⎢

⎤

⎦⎥

u=0

1

=1

n + 1

sn+1 − 1

s − 1

=

1

n + 11+ s + s2 +L + sn

( )

Problems, cont.

3.

where

k=0?

P( Sn+1 =k + 1 Sn =k)

=P(Sn+1 =k + 1Sn =k or Sn =−k)

=P(Xn+1 = 1)P(Sn = k) + P(Xn+1 = −1)P(Sn = −k)

P(Sn = k) + P(Sn = −k)

=ppk

(n) + qp−k(n)

pk(n) + p−k

(n)

pk(n) =P(Sn =k) =

nn+k2

⎛

⎝⎜⎞

⎠⎟pn+k2 q

n−k2

ppk(n) +qp−k

(n)

pk(n) +p−k

(n) =pk+1 +qk+1

pk +qk

Classification of states

A state i for a Markov chain Xk is called persistent if

and transient otherwise.

Let

and .

j is persistent iff fjj=1.

Let

P(Xk =i for somek≥1X0 =i) =1

fij(n) =P(X1 ≠j,X2 ≠j, ...,Xn−1 ≠j,Xn =j X0 =i)

fij = fij(n)

n=1

∞

∑

Pij (s) = pij(n)sn

n=0

∞

∑ Fij (s) = fij(n)sn

n=0

∞

∑

Some results

Theorem:

(a)Pii(s)=1+Fii(s)Pii(s)

(b)Pij(s)=Fij(s)Pij(s) for i ≠ j.Proof:As for the random walk case we deduce from the Markov property that

Multiply both sides by sm, sum over m≥1 to get

pij(m) = fij

(r)pjj(m−r)

r=1

m

∑

Pij (s)−1(i=j) =Fij (s)Pij (s)

Some results, cont.

Corollary:

(a) State j is persistent if and then for all i.

(b) State j is transient if and then for all i.

Proof:

SInce we see that

But (by Abel’s thm)

p jj(n) =∞∑

pij(n) =∞∑

p jj(n) < ∞∑

pij(n) < ∞∑

Pjj (s) =1

1−Fjj (s)

Pjj (s)→ ∞ iff Fjj (1) =fjj =1

lims↑1

Pjj (s) = pjj(n)∑

A final consequence

If i is transient, then

Why?

Example: Branching process

What states are persistent? Transient?

State 0 is called absorbing, since once the process reaches 0, it never leaves again.

p jj(n) → 0

Mean recurrence time

Let Ti = min{n>0: Xn = i} and i = E(Ti|X0=i).

For a transient state i = ∞.

For a persistent state

We call a recurrent state positive persistent if i < ∞, null persistent otherwise.

Example: Simple random walk

positive recurrent = non-null persistent

i = nfii(n)∑

A model forradiation damage

Initial damage from radiation can either heal or get worse until it is visible.

0 is a healthy organism (absorbing)

3 visible damage (absorbing)

1 initial damage

2 amplified damage

P =

1 0 0 023 0 1

3 00 1

3 0 23

0 0 0 1

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

Radiation damage, cont.

Recovery probability 0 is probability of reaching 0 before 3.

Last step must go

Thus

1→ 00 = p10 p11

(n)

n=0

∞

∑

p11(2n) =( 13 ×1

3)n

0 =2

3

1

9⎛⎝⎜

⎞⎠⎟

n

=2

3

89n=0

∞

∑ =3

4

Communication

Two states i and j communicate, , if for some m.

i and j intercommunicate, , if and .

Theorem: is an equivalence relation.

What do we need to prove?

i→ jpij

(m) > 0

i ↔ j i→ jj→ i

↔

Equivalence classes of states

Theorem: If then

(a) i is transient iff j is transient

(b) i is persistent iff j is persistent

Proof of (a): Since there are m,n with

By Chapman-Kolmogorov

so summing over r we get

i ↔ j

pii(m+r+n) = pij

(m)pjj(r)pji

(n)

r=1

n

∑ ≥pjj(r)pij

(m)pji(n)

>01 24 34

i ↔ jpij

(m)p ji(n) > 0

pii(r ) < ∞ if pjj

(r) < ∞∑∑

Closed and irreducible sets

A set C of states is closed if pij=0 for all i in C, j not in C

C is irreducible if for all i,j in C.

Theorem:

S=T+C1+C2+...

where T are all transient, and the Ci are irreducible disjoint closed sets of persistent states

Note: The Ci are the equivalence classes for

i ↔ j

↔

Example

S={0,1,2,3,4,5}

{0,1},{4,5} closed irreducible persistent

{2,3} transient. Why?

P =

12

12 0 0 0 0

14

34 0 0 0 0

14

14

14

14 0 0

14 0 1

414 0 1

4

0 0 0 0 12

12

0 0 0 0 12

12

⎛

⎝

⎜⎜⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟⎟⎟

f11(n) =

p11 =34 if n=1

p10 (p00 )n−2p01 = 1

812( )

n−2if n > 1

⎧⎨⎪

⎩⎪

1 = nf11(n) =∑ 3

4 + 18 n 1

4( )n−2

=3

2n=2

∞

∑

Long-term behavior

Recall from the 0-1 process that

When does this not depend on n?

(a) p01 = p11

(b)

(c)

1(n) =

p01

1− (p11 − p01)+ μ1

(0) −p01

1− (p11 − p01)

⎛

⎝⎜⎞

⎠⎟(p11 − p01)

n

1(0) =

p01

1− (p11 − p01)

n→ ∞

Stationary distribution

Case (b) is the general one. Here is the idea: Recall that (n)= (0)Pn. In order to get the same distribution for all n, we use (0)=where solves P =

(1) = P = P2 = P = ...

Pn =

Snoqualmie Falls

so

or

P =.602 .398.166 .834⎛⎝⎜

⎞⎠⎟

0 = .602π0 + .166π1

π1 = .398π0 + .834π1

π0 + π1 = 1

0 =.166

1− .602 + .166= .295

1 = .705

P5 =.305 .695.290 .710⎛⎝⎜

⎞⎠⎟