chapter ii: markov jump processes
TRANSCRIPT
![Page 1: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/1.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Chapter II: Markov Jump Processes
Jakub Cerny
Department of Probability and Mathematical Statistics
Stochastic Modelling in Economics and Finance
October 15, 20121 / 25
![Page 2: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/2.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Contents
1 Introduction
2 Intensity Matrix and Kolmogorov Differential Equations
3 Stationary Distribution
4 Time Reversibility
2 / 25
![Page 3: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/3.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Notation and Markov property
(Ω,F ,P) . . . probability spaceE . . . finite or countable state spaceS0,S1,S2, . . . . . . times of jumps, S0 := 0Tn = Sn+1 − Sn . . . holding times, n ∈ N
Y0,Y1,Y2 . . . . . . visited states from state space E
Definition (Markov Chain with Continuous Time)
The system of random variables Xt , t ≥ 0 defined on (Ω,F ,P) iscalled Markov chain with continous time and countable (finite)state space (Markov jump process) E if
P(Xt = j |Xs = i , . . . ,Xt1 = i1) = P(Xt = j |Xs = i)
for all j , i , i1, . . . , in ∈ E and 0 < t1 < · · · < tn < s < t whereP(Xs = i ,Xtn = in, . . . ,Xt1 = i1) > 0.
3 / 25
![Page 4: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/4.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Basic Characteristics
Assuming that Markov jump process is time-homogenous, i.e.
P(Xs+t = j |Xs = i) = P(Xt = j |X0 = i) = Pi(Xt = j) = ptij ,
let us denote the transition semigroup by the family
ptij , t ≥ 0,∑
j∈E
ptij = 1 =
Pt , t ≥ 0
.
Chapman-Kolmogorov equations are analogic to discrete timeMarkov chains
pt+sij =
∑
k∈E
psikptkj
and in the matrix form Pt+s = PtPs which can be also intepretedas usual matrix multiplication.
4 / 25
![Page 5: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/5.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example of Application in Insurance
healthy
ill
dead
S S S S1 2 3 4S0
T T T T1 2 3 4
5 / 25
![Page 6: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/6.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Absorbtion State
Two possible phenomena of the Markov jump process may occur.First,the process may be absorbed at some state, for eg. state i . Itmeans that there exists a last finite jump time Sn. ThenTj = ∞, j = n, n + 1, . . . and Yk = i , k = n, n+ 1, . . .
healthy
ill
dead
S S S S1 2 3 4S0
absorbtion state
last finite jump time
6 / 25
![Page 7: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/7.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Explosion Time
Second, the jumps of the process may accumulate, i.e. the processexplodes. Jump times of the process are defined as
S1 = inft > 0,Xt 6= X0
S2 = inft > S1,Xt 6= XS1
. . .
Sn = inft > Sn−1,Xt 6= XSn−1
and jump times can be rewritten by holding times as
Sn =n
∑
k=1
Tk , ξ = supSn =∞∑
k=1
Tk ,
Random variable ξ is called explosion time.
7 / 25
![Page 8: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/8.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example - Explosion Process
S S S S1 2 3 4S0
Y1
Y2
bbb
b b b ξ
8 / 25
![Page 9: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/9.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Intensity Matrix
Theorem
For every state i ∈ E exists a limit
limh→0+
1− phiih
:= Λi ≤ ∞
and for every i , j ∈ E, i 6= j exists a limit
limh→0+
phijh
:= Λij < ∞
and for every i ∈ E is∑
i 6=j
Λij ≤ Λi .
Nonnegative numbers Λij are called transition intensities from thestate i to the state j , Λii = −Λi , Λi is called total intensity. Thematrix Λ = Λij , i , j ∈ E is called intensity matrix. 9 / 25
![Page 10: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/10.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Intensity Matrix
When state space E is finite then
Λi =∑
i 6=j
Λij ⇒∑
j∈E
Λij = 0.
If state space E is infinite then
Λi ≥∑
i 6=j
Λij .
Theorem
If Λi = 0 then ptii = 1. If 0 < Λi < ∞ then the holding time instate i has exponential distribution with expected value equal 1/Λi .
10 / 25
![Page 11: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/11.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Reuter’s Explosion Condition
The following result is of gives a necessary and sufficient condition,known as Reuter’s condition, for a Markov jump process to beexplosive (nontrivial application are birth-death processes).
Theorem
A Markov jump process is nonexplosive if and only if the onlynonngeative bounded solution k = (ki )i∈E to the set of equations
Λk = k
is k = 0.
11 / 25
![Page 12: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/12.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Kolmogorov Differential Equations
Let Λ be an intensity matrix on E , Λi < ∞ and Xt , t ≥ 0 is theMarkov jump process defined on (Ω,F ,P), then E × E -matricesPt satisfy the backward equation, i.e.
(ptij)′
=∑
k∈E
Λikptkj = −Λip
tij +
∑
k 6=i
Λikptkj
(Pt)′
= ΛPt ,
and forward equation, i.e.
(ptij )′
=∑
k∈E
ptikΛkj = −ptijΛj +∑
k 6=i
ptikΛkj
(Pt)′
= PtΛ,
assuming thatphij
h→ Λij converges uniformly in i .
12 / 25
![Page 13: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/13.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Kolmogorov Differential Equations
Theorem
If E is finite and Λ = Λij , 0 ≤ i , j ≤ n is a matrix whereΛij ≥ 0, i 6= j and Λi =
∑
i 6=j
Λij . Then exists a unique solution of
both Kolmogorov differential equations which satisfies the initialcondition P0 = I. The solution in matrix form is
Pt = eΛt
where eΛt is an exponential matrix function defined as
eΛt =
∞∑
k=0
Λktk
k!
13 / 25
![Page 14: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/14.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example - Kolmogorov Differential Equations
Suppose that E has p = 2 states Y1,Y2 and intensities Λ(Y1),Λ(Y2) are not zero. Then Λ has eigenvalues 0 andΛ = −Λ(Y1)− Λ(Y2) with corresponding right eigenvectors(1, 1)T , (Λ(Y1),−Λ(Y2)). Hence
Λ =
(
−Λ(Y1) Λ(Y1)Λ(Y2) −Λ(Y2)
)
= B
(
0 00 Λ
)
B−1, B =
(
1 Λ(Y1)1 −Λ(Y2)
)
using eigendecomposition.
Pt = eΛt =
∞∑
n=0
tn
n!B
(
0 00 Λn
)
B−1 = B
(
0 00 eΛt
)
B−1
=1
Λ(Y1) + Λ(Y2)
(
Λ(Y2) + Λ(Y1)eΛt Λ(Y1)− Λ(Y1)e
Λt
Λ(Y2)− Λ(Y2)eΛt Λ(Y1) + Λ(Y2)e
Λt .
)
.
14 / 25
![Page 15: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/15.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Stationary Measure and Distribution
Forward and backward equations are quite limited utility even incases when state space is finite (for eg. complex eigenvalues). Onecommon application is to look for a stationary distribution.
Definition
Let Xt , t ≥ 0 be Markov jump process with transition matrix Pt .If vector π = πj , j ≥ 0 satisfies
πT = π
TPt
is called stationary measure of the process Xt , t ≥ 0 on E due toPt , t ≥ 0. If π is also probability distribution on E then it iscalled stationary distribution.
15 / 25
![Page 16: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/16.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Classification of States
Let Λ∗ = Λ∗ij , i , j ∈ E be matrix with
Λ∗ij =
Λij/Λi , Λi > 00, Λi = 0
, i 6= j
Λ∗ii =
0, Λi > 01, Λi = 0
and we know that jump times of Xt , t ≥ 0 are given by thesequence S0,S1, . . . . We define
Z0 = X0,Zn = XSn , n = 1, 2, . . .
It can be shown that Zn, n ∈ N0 is discrete Markov chain withtransition probabilities Λ∗
ij defined above.
16 / 25
![Page 17: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/17.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Classification of States
Theorem
The following properties are equivalent
(i) Zn, n ∈ N0 is irreducible,
(ii) for any i , j ∈ E we have ptij > 0 for some t > 0
(iii) for any i , j ∈ E we have ptij > 0 for all t > 0.
We define Xt , t ≥ 0 to be irreducible if one of the properties(i)-(iii) hold. Similarly it is seen that we can define i to betransient (reccurent) for Xt , t ≥ 0 if either (i) the sett : Xt = i is bounded (unbounded) Pi -a.s. (ii) i is transient(reccurent) for Zn, n ∈ N0 or (iii)Pi(inft > 0,Xt = i , lims→t Xs 6= i) < 1(= 1).
17 / 25
![Page 18: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/18.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Stationary Measure
Theorem
Suppose that Xt , t ≥ 0 is irreducible and recurrent on E. Thenthere exists one, and up to a multiplicative factor only one,invariant measure π. This has the property πj > 0 for all j and canbe found in either of the following ways:
(i) for some fixed but arbitrary state i , πj is the expected timespent in j between successive entrances to i . That is, withω(i) = inft > 0,Xt = i , lims↑t Xs 6= i
πj = Ei
ω(i)∫
0
I(Xt = j)dt
(ii) πj = µj/Λj where µ is stationary for Zn
(iii) as a solution of πΛ = 0.18 / 25
![Page 19: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/19.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Ergodicity
An irreducible recurrent process with the stationary measurehaving finite mass is called ergodic, and
Theorem
An irreducible nonexplosive Markov jump process is ergodic if andonly if exists a probability solution π (
∑
i∈E
πi = 1, πi ∈ [0, 1]) to
πΛ = 0. In that case π is the stationary distribution.
Theorem
If Xt , t ≥ 0 is ergodic and π is the stationary distribution, thenptij → πj , t → ∞ for all states i , j ∈ E.
19 / 25
![Page 20: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/20.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Ergodicity
As in discrete time, time-average properties like
1
T
T∫
0
f (Xt)dta.s.→ π(f ) = Eπ(f (Xt)) =
∑
i∈E
πi f (i)
hold under suitable conditions on f . It means that the timeaverage converges to the spatial average if the process is ergodic.
Corollary
If Xt , t ≥ 0 is irreducible recurrent but not ergodic (stationarymeasure does not have finite mass), then ptij → 0 for all statesi , j ∈ E.
20 / 25
![Page 21: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/21.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Time Reversibility
Time reversibility (or just reversibility) of a process means looselythat the process evolves in just the same way irrespective ofwhether time is read forward (as usual) or backward.
Definition (Reversibility)
Process Xt , t ∈ R is time reversible if for finite dimensionaldistributions and for all t is
(Xt1 ,Xt2 , . . . ,Xtn)D= (Xt−t1 ,Xt−t2 , . . . ,Xt−tn),
Lemma
If the process Xt , t ∈ R is time reversible then it is also timestationary.
21 / 25
![Page 22: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/22.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Detailed Balance Condition
Theorem
Stationary Markov jump process Xt , t ∈ R with intensity matrixΛ is time reversible if and only if exists probability distribution π onE satisfying
π(x)Λ(x , y) = π(y)Λ(y , x), x , y ∈ E .
In this case π is stationary distribution.
Previous theorem is also called detailed balance conditionbecause the flow between every two states is in balance. The termπ(x)Λ(x , y) is the probability flow from x to y .
22 / 25
![Page 23: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/23.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Full Balance Condition
In contrast to detailed balance condition the equilibrium equationπΛ = 0 gives the condition of full balance. More precisely,rewriting πΛ = 0
∑
x 6=y
π(x)Λ(x , y) =∑
x 6=y
π(y)Λ(y , x) for all states.
This condition loosely means that everything that flows into somestate also flows out of it. Left hand side is the inflow of the state xand right hand side is the outflow.
23 / 25
![Page 24: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/24.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
References
Asmussen S.: Applied Probability and Queues, Springer, NewYork, 2003.
Lachout P., Praskova Z.: Zaklady nahodnych procesu,Karolinum, Praha, 2005.
Uncovsky L.: Stochasticke modely operacnej analyzy, Alfa,Bratislava, 1980.
24 / 25
![Page 25: Chapter II: Markov Jump Processes](https://reader033.vdocuments.site/reader033/viewer/2022050302/626f0b1e1963ad39536ef1bf/html5/thumbnails/25.jpg)
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Thank you for your attention!
Jakub Cerny
25 / 25