agenda: countable markov chainscandes/teaching/stats318/...k=1 n(t+1) k fx tgis a markov chain...
TRANSCRIPT
-
Stats 318: Lecture # 17
Agenda: Countable Markov Chains
I Branching process
I Recurrence and transience
I Classification of states
I Limit theorems
-
Examples of countable state space
Symmetric random walk
Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2
Branching processes
Originally considered by Galton & Watson in the 1870’s while seeking for an explanation forthe phenomenon of the disappearance of family names even in a growing population
-
Examples of countable state space
Symmetric random walk
Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2
Branching processes
Originally considered by Galton & Watson in the 1870’s while seeking for an explanation forthe phenomenon of the disappearance of family names even in a growing population
-
Examples of countable state space
Symmetric random walk
Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2
Branching processes
Originally considered by Galton & Watson in the 1870’s while seeking for an explanation forthe phenomenon of the disappearance of family names even in a growing population
-
Examples of countable state space
Symmetric random walk
Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2
Branching processes
Originally considered by Galton & Watson in the 1870’s while seeking for an explanation forthe phenomenon of the disappearance of family names even in a growing population
-
Branching processes
Problem: Each male in a given family has a prob. pk of having k sons
What is the probability that after n generations, an individual has no male descendants?
I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk
I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on
I Xt : size of population at time t
Xt+1 =
Xt∑k=1
N(t+1)k
{Xt} is a Markov chain
-
Branching processes
Problem: Each male in a given family has a prob. pk of having k sons
What is the probability that after n generations, an individual has no male descendants?
I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk
I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on
I Xt : size of population at time t
Xt+1 =
Xt∑k=1
N(t+1)k
{Xt} is a Markov chain
-
Branching processes
Problem: Each male in a given family has a prob. pk of having k sons
What is the probability that after n generations, an individual has no male descendants?
I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk
I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on
I Xt : size of population at time t
Xt+1 =
Xt∑k=1
N(t+1)k
{Xt} is a Markov chain
-
Branching processes
Problem: Each male in a given family has a prob. pk of having k sons
What is the probability that after n generations, an individual has no male descendants?
I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk
I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on
I Xt : size of population at time t
Xt+1 =
Xt∑k=1
N(t+1)k
{Xt} is a Markov chain
-
Branching processes
Problem: Each male in a given family has a prob. pk of having k sons
What is the probability that after n generations, an individual has no male descendants?
I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk
I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on
I Xt : size of population at time t
Xt+1 =
Xt∑k=1
N(t+1)k
{Xt} is a Markov chain
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulations
Prob. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations
First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1
P(Xt = 0) =∑k≥0
P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1
pkqkt−1
∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0
pksk (MGF of N)
= E(sN )
Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex
-
Two possibilities: q0 = 0
ϕ′(1) ≤ 1 ϕ′(1) ≥ 1
qt −→t→∞
1 qt −→t→∞
π0
π0 only fixed point in [0, 1] of ϕ(s) = s
-
Two possibilities: q0 = 0
ϕ′(1) ≤ 1 ϕ′(1) ≥ 1
qt −→t→∞
1 qt −→t→∞
π0
π0 only fixed point in [0, 1] of ϕ(s) = s
-
Two possibilities: q0 = 0
ϕ′(1) ≤ 1 ϕ′(1) ≥ 1
qt −→t→∞
1 qt −→t→∞
π0
π0 only fixed point in [0, 1] of ϕ(s) = s
-
Two possibilities: q0 = 0
ϕ′(1) ≤ 1 ϕ′(1) ≥ 1
qt −→t→∞
1 qt −→t→∞
π0
π0 only fixed point in [0, 1] of ϕ(s) = s
-
Two possibilities: q0 = 0
ϕ′(1) ≤ 1 ϕ′(1) ≥ 1
qt −→t→∞
1 qt −→t→∞
π0
π0 only fixed point in [0, 1] of ϕ(s) = s
-
More information
ϕXt(s) = E[sXt]= E
[ϕ(s)Xt−1
]= ϕXt−1(ϕ(s))
∴ ϕXt(s) = ϕ ◦ . . . ◦︸ ︷︷ ︸t times
ϕ(s)
-
More information
ϕXt(s) = E[sXt]= E
[ϕ(s)Xt−1
]= ϕXt−1(ϕ(s))
∴ ϕXt(s) = ϕ ◦ . . . ◦︸ ︷︷ ︸t times
ϕ(s)
-
More information
ϕXt(s) = E[sXt]= E
[ϕ(s)Xt−1
]= ϕXt−1(ϕ(s))
∴ ϕXt(s) = ϕ ◦ . . . ◦︸ ︷︷ ︸t times
ϕ(s)
-
Countable Markov chains
Two distinct situations
(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain
Ergodic theorem holds
Convergence
(2) The chain does not have an equilibrium distribution
(a) It is transient : each state is visited a finite number of times
(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)
-
Countable Markov chains
Two distinct situations
(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain
Ergodic theorem holds
Convergence
(2) The chain does not have an equilibrium distribution
(a) It is transient : each state is visited a finite number of times
(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)
-
Countable Markov chains
Two distinct situations
(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain
Ergodic theorem holds
Convergence
(2) The chain does not have an equilibrium distribution
(a) It is transient : each state is visited a finite number of times
(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)
-
Countable Markov chains
Two distinct situations
(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain
Ergodic theorem holds
Convergence
(2) The chain does not have an equilibrium distribution
(a) It is transient : each state is visited a finite number of times
(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)
-
Countable Markov chains
Two distinct situations
(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain
Ergodic theorem holds
Convergence
(2) The chain does not have an equilibrium distribution
(a) It is transient : each state is visited a finite number of times
(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Recurrence and transienceReturn times : X0 = x
τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}
Definition
State x is recurrent if q(x) = P(τx
-
Why?
(τ(1)x , τ
(2)x , . . . , τ
(k)x
)distributed as
(W
(1)x ,W
(1)x +W
(2)x , . . . ,W
(1)x + . . .+W
(k)x
)W
(k)x (waiting times) iid τx
I If P(τx
-
Why?(τ(1)x , τ
(2)x , . . . , τ
(k)x
)distributed as
(W
(1)x ,W
(1)x +W
(2)x , . . . ,W
(1)x + . . .+W
(k)x
)
W(k)x (waiting times) iid τx
I If P(τx
-
Why?(τ(1)x , τ
(2)x , . . . , τ
(k)x
)distributed as
(W
(1)x ,W
(1)x +W
(2)x , . . . ,W
(1)x + . . .+W
(k)x
)W
(k)x (waiting times) iid τx
I If P(τx
-
Why?(τ(1)x , τ
(2)x , . . . , τ
(k)x
)distributed as
(W
(1)x ,W
(1)x +W
(2)x , . . . ,W
(1)x + . . .+W
(k)x
)W
(k)x (waiting times) iid τx
I If P(τx
-
Why?(τ(1)x , τ
(2)x , . . . , τ
(k)x
)distributed as
(W
(1)x ,W
(1)x +W
(2)x , . . . ,W
(1)x + . . .+W
(k)x
)W
(k)x (waiting times) iid τx
I If P(τx
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possible
I All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possible
I All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possible
I All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possible
I All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possible
I All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possibleI All states are recurrent
I All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possibleI All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Analytical recurrence criterion
x recurrent ⇔ E (Nx|X0 = x) =∞
Nx =∑t≥1
It It =
{1 Xt = x
0 else
E (Nx|X0 = x) =∑t≥0
E(It|X0 = x) =∑t≥1
P(Xt = x|X0 = x) =∑t≥1
P t(x, x)
Proposition
State x is recurrent iff∑t≥1
P t(x, x) =∞
Proposition
{Xt} irreducible chain. Only two cases are possibleI All states are recurrentI All states are transient
Note : If X is finite, then all states are necessarily recurrent
-
Proof
Suppose one state x0 is recurrent, then we show that all states are recurrent
Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0
P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)
∴∑t≥1
P t(x, x) ≥∑
t≥t1+t2
P t(x, x) ≥
∑k≥0
P k(x0, x0)
P t1(x, x0)P t2(x0, x) =∞=⇒ x recurrent
-
Proof
Suppose one state x0 is recurrent, then we show that all states are recurrent
Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0
P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)
∴∑t≥1
P t(x, x) ≥∑
t≥t1+t2
P t(x, x) ≥
∑k≥0
P k(x0, x0)
P t1(x, x0)P t2(x0, x) =∞=⇒ x recurrent
-
Proof
Suppose one state x0 is recurrent, then we show that all states are recurrent
Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0
P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)
∴∑t≥1
P t(x, x) ≥∑
t≥t1+t2
P t(x, x) ≥
∑k≥0
P k(x0, x0)
P t1(x, x0)P t2(x0, x) =∞=⇒ x recurrent
-
Proof
Suppose one state x0 is recurrent, then we show that all states are recurrent
Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0
P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)
∴∑t≥1
P t(x, x) ≥∑
t≥t1+t2
P t(x, x) ≥
∑k≥0
P k(x0, x0)
P t1(x, x0)P t2(x0, x) =∞
=⇒ x recurrent
-
Proof
Suppose one state x0 is recurrent, then we show that all states are recurrent
Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0
P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)
∴∑t≥1
P t(x, x) ≥∑
t≥t1+t2
P t(x, x) ≥
∑k≥0
P k(x0, x0)
P t1(x, x0)P t2(x0, x) =∞=⇒ x recurrent
-
If all states are recurrent : 2 possibilities
I All null recurrent
I All positive
-
If all states are recurrent : 2 possibilities
I All null recurrent
I All positive
-
If all states are recurrent : 2 possibilities
I All null recurrent
I All positive
-
Example : SRW
I 1D null recurrence
I 2D null recurrence
I 3D+ transience
Why?
-
Example : SRW
I 1D null recurrence
I 2D null recurrence
I 3D+ transience
Why?
-
Example : SRW
I 1D null recurrence
I 2D null recurrence
I 3D+ transience
Why?
-
Example : SRW
I 1D null recurrence
I 2D null recurrence
I 3D+ transience
Why?
-
Recurrence and transience of SRW’s
I 1D : P(X2n = 0) =(2nn
) (12
)2n ∼ 1√πn
I 2D : P(X2n = 0) =[(
2nn
) (12
)2n]2 ∼ 1πnI 3D+ : P(X2n = 0) ∼ 3
√3
2(πn)3/2
P(τ0
-
Recurrence and transience of SRW’s
I 1D : P(X2n = 0) =(2nn
) (12
)2n ∼ 1√πn
I 2D : P(X2n = 0) =[(
2nn
) (12
)2n]2 ∼ 1πn
I 3D+ : P(X2n = 0) ∼ 3√3
2(πn)3/2
P(τ0
-
Recurrence and transience of SRW’s
I 1D : P(X2n = 0) =(2nn
) (12
)2n ∼ 1√πn
I 2D : P(X2n = 0) =[(
2nn
) (12
)2n]2 ∼ 1πnI 3D+ : P(X2n = 0) ∼ 3
√3
2(πn)3/2
P(τ0
-
Limit theoremsP irreducible : then there is at most one invariant distribution π
If π exists, then π(x) > 0 ∀ x ∈ X
Theorem (Ergodic theorem)
Irreducible chain with invariant dist. π
limT→∞
1
T
T∑t=1
1{Xt = x} = π(x) =1
Eτx
Shows that all states are positive recurrent
Theorem (Convergence theorem)
Irreducible & aperiodic chain with invariant dist. π
π(t) −→ π i.e. limt→∞
P(Xt = x) = π(x)
-
Limit theoremsP irreducible : then there is at most one invariant distribution π
If π exists, then π(x) > 0 ∀ x ∈ X
Theorem (Ergodic theorem)
Irreducible chain with invariant dist. π
limT→∞
1
T
T∑t=1
1{Xt = x} = π(x) =1
Eτx
Shows that all states are positive recurrent
Theorem (Convergence theorem)
Irreducible & aperiodic chain with invariant dist. π
π(t) −→ π i.e. limt→∞
P(Xt = x) = π(x)
-
Limit theoremsP irreducible : then there is at most one invariant distribution π
If π exists, then π(x) > 0 ∀ x ∈ X
Theorem (Ergodic theorem)
Irreducible chain with invariant dist. π
limT→∞
1
T
T∑t=1
1{Xt = x} = π(x) =1
Eτx
Shows that all states are positive recurrent
Theorem (Convergence theorem)
Irreducible & aperiodic chain with invariant dist. π
π(t) −→ π i.e. limt→∞
P(Xt = x) = π(x)
-
Limit theoremsP irreducible : then there is at most one invariant distribution π
If π exists, then π(x) > 0 ∀ x ∈ X
Theorem (Ergodic theorem)
Irreducible chain with invariant dist. π
limT→∞
1
T
T∑t=1
1{Xt = x} = π(x) =1
Eτx
Shows that all states are positive recurrent
Theorem (Convergence theorem)
Irreducible & aperiodic chain with invariant dist. π
π(t) −→ π i.e. limt→∞
P(Xt = x) = π(x)
-
Limit theoremsP irreducible : then there is at most one invariant distribution π
If π exists, then π(x) > 0 ∀ x ∈ X
Theorem (Ergodic theorem)
Irreducible chain with invariant dist. π
limT→∞
1
T
T∑t=1
1{Xt = x} = π(x) =1
Eτx
Shows that all states are positive recurrent
Theorem (Convergence theorem)
Irreducible & aperiodic chain with invariant dist. π
π(t) −→ π i.e. limt→∞
P(Xt = x) = π(x)
-
Stationary distribution criterion
Theorem
Irreducible (homogeneous) chain is positive recurrent iff there exists a stationary distribution
Moreover, inv. dist. π, when it exists, is unique & π > 0
-
Example : birth and death process
X = {0, 1, 2, . . .}
p(x) > 0 q(x) > 0 except q(0) = 0
Proposition
{Xt} positive recurrent (exists π) iff∑x≥1
λ(x)
-
Example : birth and death process
X = {0, 1, 2, . . .}
p(x) > 0 q(x) > 0 except q(0) = 0
Proposition
{Xt} positive recurrent (exists π) iff∑x≥1
λ(x)
-
Example : birth and death process
X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0
Proposition
{Xt} positive recurrent (exists π) iff∑x≥1
λ(x)
-
Example : birth and death process
X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0
Proposition
{Xt} positive recurrent (exists π) iff∑x≥1
λ(x)
-
Example : birth and death process
X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0
Proposition
{Xt} positive recurrent (exists π) iff∑x≥1
λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)
-
Proof
Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X
∑π(x)P (x, y) = π(y)⇔
{π(0)p(0) = q(1)π(1)
π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)
By induction
q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)
i.e. ∃ inv. dist. iff∑λ(x)