lec18

Stat 150 Stochastic Processes Spring 2009

Lecture 18: Markov Chains: ExamplesLecturer: Jim Pitman

A nice collection of random walks on graphs is derived from random movementof a chess piece on a chess board. The state space of each walk is the set of8 8 = 64 squares on the board:

a b c d e f g h

1

2

3

4

5

6

7

8

Each kind of chess piece at a state i on an otherwise empty chess board hassome set of all states j to which it can allowably move. These states j arethe neighbours of i in a graph whose vertices are the 64 squares of the board.Ignoring pawns, for each of king, queen, rook, bishop and knight, if j can bereached in one step from i, this move can be reversed to reach i from j. Notethe pattern of black and white squares, which is important in the followingdiscussion.

The King

interior corner edge8 possible moves 3 possible moves 5 possible moves

!

"

# %%%&

'''(

%%%)

'''*

!"###$

!

"

#$$$%

&&&'

1

Lecture 18: Markov Chains: Examples 2

In the graph, i j means that j is a kings move from i, and i is a kings move from j.Observe that N(i) := #( possible moves from i) {3, 5, 8}. From generaldiscussion of random walk on a graph in previous lecture, the reversible

equilibrium distribution is pii =N(i)

where

:=j

N(j) = (6 6) 8 + (4 6) 5 + 4 3

Question: Is this walk regular? Is it true that m : Pm(i, j) > 0 for all i, j?Yes. You can easily check this is so for m = 7.

Rook and Queen

Very similar treatment. Just change the values of N(i).

Both walks are regular because P 2 is strictly positive: even the rook canget from any square to any other square in two moves.

Bishop

32 white squares, 32 black squares. Bishop stays on squares of one colour.

!!!"

###$

!!!%

###&

Is this chain regular? No, because the bishops walk is reducible: there aretwo disjoint sets of states (B and W ). For each i, j B, N ;PN(i, j) > 0.In fact, N = 3. Similarly for each i, j W,P 3(i, j) > 0, hence PN(i, j) > 0for all N 3. But B and W do not communicate.

!"#$!"#$1...

32

33...64

B

W

B W

The matrix is decomposed into two: B B,W W . If the bishop


starts on a black square, it moves as if a Markov chain with state space B,and the limit distribution is the equilibrium distribution for the random

walk on B, with pii =N(i) as before, but now = jB N(j), with 32

terms, rather than a sum over all 64 squares. This is typical of a Markovchain with two disjoint communicating classes of states.

Knight

Is it regular? (No)After an even number of steps n, P n(i, j) > 0 only if j is of the same coloras i. No matrix power of P is strictly > 0 at all entries = not regular.

Periodicity Fix a state i. Look at the set of n : P n(i, i) > 0. For the knightsmove {n : P n(i, i) > 0} = {2, 4, 6, 8, . . . }. You cannot return in an odd numberof steps because knights squares change colour B W B at each step.

Definition: Say P is irreducible if for all states i, j, n : P n(i, j) > 0.This condition is weaker than regular, which requires the n to work for all i, j.

Deal with periodic chains: Suppose P is irreducible, say state i has periodd {1, 2, 3, . . . } if the greatest common divisor of {n : P n(i, i) > 0} = d. E.g.,d = 2 for each state i for the knights walk.

Fact (Proof see Feller Vol.1): For an irreducible chain, every state i has thesame period.

d = 1: called aperiodic

d = 2, 3, . . . : called periodic with period d

In periodic case, d disjoint sets of states, C1, . . . , Cd (Cyclically moving sub-classes): P (i, j) > 0 only if i Cm, j Cm+1 (m+ 1 mod d).


C1!"#$

C2!"#$

C3!"#$

...

Note If i : P (i, i) > 0, then i has period 1. Then, assuming P is irreducible,all states have period 1, and the chain is aperiodic.

Fact An irreducible, aperiodic chain on finite S is regular.

Death and immigration chainPopulation story. State space S = {0, 1, 2, . . . }. Let Xn S represent thenumber of individuals in some population at time n.

Dynamics: Between times n and n+ 1,

each individual present at time n dies with probability p and remains withprobability q := 1 p.

add an independent Poisson() number of immigrants.

Problem Describe limit behavior of Xn as n . Natural first step: writedown the transition matrix.For i 0, j 0, condition on number of survivors:

P (i, j) =i

k=0

(i

k

)qkpike

jk

(j k)!1(k j)

Now try to solve the equations piP = pi. Very difficult!

Idea: Suppose we start at state i = 0, X0 = 0,X1 Poi()X2 Poi(q + ) by thinning and + rule for PoissonX3 Poi((q + )q + )

...

Xn Poi((1 + q + + qn1)) = Poi(

1 qn1 q

) Poi(/p)


So given X0 = 0, we see Xnd Poi(/p).

If we start with Xn > 0, it is clear from the dynamics that we can write

Xn = Xn + Yn

where Xn = survivors of the initial population

Yn = copy of the process given (Xn = 0)

Note that

Xn 0, Yn Poi(/p)

Conclusion: Xnd Poi(/p) no matter what X0 is.

Ad hoc analysis special properties of Poisson.General idea: What do you expect in the limit? Answer: stationary distributionpiP = pi. Here pi is Poi(/p).

Does piP = pi? That is, does X0 Poi(/p) imply X1 Poi(/p) . . . ?Check: Say X0 Poi(), then by thinning and addition rules for Poisson, asbefore X1 Poi(q + ). So

X1d= X0 = q + =

1 q =

p

Therefore, the unique value of which makes Poi() invariant for this chain is = /p. In fact, this is the unique stationary distribution for the chain. Thisfollows from the previous result that no matter what the distribution of X0, thedistribution ofXn converges to Poi(/p). Because if pi was some other stationarydistribution, if we started the chain with X0 pi, then Xn pi for every n, andthe only way this can converge to Poi(/p) is if in fact pi = Poi(/p).

lec18

Documents