lecture6a

7/28/2019 lecture6a

1/4

Lecture 6 : Markov Chains

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Alex Michalka

Markov Chains

Discrete time

Discrete (finite or countable) state spaceS

Process {Xn}

Homogenous transition probabilities

matrix P = {P(i, j); i, j S}

P(i, j), the (i, j)th entry of the matrix P, represents the probability of moving to statej given that the chain is currently in state i.

Markov Property:

P(Xn+1 = in+1|Xn = in, . . . , X 0 = i0) = P(in, in+1)

This means that the states of Xn1 . . . X 0 dont matter. The transition probabilitiesonly depend on the current state of the process. So,

P(Xn+1 = in+1|Xn = in, . . . , X 0 = i0) = P(Xn+1 = in+1|Xn = in)

= P(in, in+1)

To calculate the probability of a path, multiply the desired transition probabilities:

P(X0 = i0, X1 = i1, X2 = i2, . . . , X n = in) = P(i0, i1) P(i1, i2) . . . P(in1, in)

Example: iid Sequence {Xn}, P(Xn = j) = p(j), and

jp(j) = 1.

P(i, j) = j.

Example: Random Walk. S = Z (integers), Xn = i0 + D1 + D2 + . . . + Dn, whereDi are iid, and P(Di = j) = p(j).

P(i, j) = p(j i).

Example: Same random walk, but stops at 0.

P(i, j) =

p(j i) if i = 0;1 if i = j = 0;0 if i = 0, j = 0.

6-1

7/28/2019 lecture6a

2/4

Lecture 6: Markov Chains 6-2

Example: Y0, Y1, Y2 . . . iid. P(Y = j) = p(j). Xn = max(Y0, . . . , Y n).

P(i, j) =

p(j) if i j;ki

p(k) if i = j;

0 if i > j.

Matrix Properties:P(i, ) = [P(i, j); j S], the ith row of the matrix P, is a row vector. Each row ofP is a probability distribution on the state-space S, representing the probabilities oftransitions out of state i. Each row should sum to 1. (

jS

P(i, j) = 1 i )

The column vectors [P(, j);j S] represent the probability of moving into state j.

Use the notation Pi = probability for a Markov Chain that started in state i (X0 = i).In our transition matrix notation, Pi(X1 = j) = P(i, j).

n-step transition probabilities:

Want to find Pi(Xn = j) for n = 1, 2,...This is the probability that the Markov Chain is in state j after n steps, given thatit started in state i. First, let n = 2.

Pi(X2 = k) = jS

Pi(Xi = j, X2 = k)

=jS

P(i, j)P(j, k).

This is simply matrix multiplication.

So, Pi(X2 = k) = P2(i, k)

We can generalize this fact for n-step probabilities, to get:

Pi(Xn = k) = Pn(i, k)

Where Pn(i, k) is the (i, k)th entry of Pn, the transition matrix multiplied by itself ntimes. This is a handy formula, but as n gets large, Pn gets increasingly difficult andtime-consuming to compute. This motivates theory for large values of n.

Suppose we have X0 , where is a probability distribution on S, so P(X0 = i) =

7/28/2019 lecture6a

3/4


(i) for i S. We can find nth step probabilities by conditioning on X0.

P(Xn = j) =iS

P(X0 = i) P(Xn = j|X0 = i)

=iS

(i)Pn(i, j)

= (Pn)j

= the jth entry of Pn.

Here, we are regarding the probability distribution on S as a vector indexed byi S.

So, we have X1 P,X2 P2, . . . , X n P

n.

Note: To compute the distribution of Xn for a particular , it is not necessary tofind Pn(i, j) for all i,j,n. In fact, there are very few examples where Pn(i, j) canbe computed explicitly. Often, P has certain special initial distributions so thatcomputing Pn is fairly simple.

Example: Death and Immigration Process

Xn = the number of individuals in the population at time (or generation) n.

S = {0, 1, 2, . . .}

Idea: between times n and n + 1, each of the Xn individuals dies with probabilityp, and the survivors contribute to generation Xn+1. Also, immigrants arrive eachgeneration, following a Poisson() distribution.

Theory: What is the transition matrix, P, for this process? To find it, we conditionon the number of survivors to obtain:

P(i, j) =ijk=0

i

j

(1 p)k(p)ik e

jk

(jk)!

Here, i j = min{i, j}. This formula cannot be simplified in any useful way, butwe can analyze the behavior for large n using knowledge of the Poisson/Binomialrelationship. We start by considering a special initial distribution for X0.

Let X0 Poisson(0). Then the survivors of X0 have a Poisson(0q) distribution,where q = 1p. Since the number of immigrants each generation follows a Poisson()distribution, independent of the number of people currently in the population, we haveX1 Poisson(0q+ ). We can repeat this logic to find:

X2 Poisson((0q+ )q+ ) = Poisson(0q2 + q+ ), and

7/28/2019 lecture6a

4/4


X3 Poisson((0q2 + q+ )q+ ) = Poisson(0q

3 + q2 + q+ ).

In general,

Xn Poisson(0qn +

n1k=0

qk).

In this formula, 0qn represents the survivors from the initial population, qn1 rep-

resents the survivors from the first immigration, and so on, until q represents thesurvivors from the previous immigration, and represents the immigrants in the cur-rent generation.

Now well look at what happens as n gets large.

As we let n : 0qn 0, and 0qn + n1k=0

qk 1q

= p

.

So, no matter what 0, if X0 has Poisson distribution with mean 0,

limn P(Xn = k) =ek

k!, where =

1q=

p.

It is easy enough to show that this is true no matter what the distribution of X0.The particular choice of initial distribution for X0 that is Poisson with mean 0 =

gives an invariant (also called stationary, or equilibrium, or steady-state) distributionof X0. With 0 = , we find that each Xn will follow a Poisson() distribution. This

initial distribution (i) = ei

i!is special because it has the property that

iS

(i)P(i, j) = (j) for all j S.

Or, in matrix form, P = . It can be shown that this is the unique stationaryprobability distribution for this Markov Chain.

lecture6a

Documents