chapter 4 - course.shufe.edu.cncourse.shufe.edu.cn/sxwh/semimarkov/semi-markov risk models for...

Chapter 4 DISCRETE TIME AND REWARD SMP AND THEIR NUMERICAL TREATMENT 1 DISCRETE TIME SEMI-MARKOV PROCESSES 1.1 Purpose This chapter will present both discrete time homogeneous (DTHSMP) and non-homogeneous (DTNHSMP) semi-Markov processes and the numerical methods to be used for applying semi-Markov models in real-life problems, furthermore the Semi-Markov ReWard Processes (SMRWP) will be presented. Although, in general, time in real-life problems is continuous, the real observation of the considered system is almost always made in discrete time even if the used time unit may in some cases be very small. The choice of this time unit depends on what we observe and what we wish to study. For example if we are studying the random evolution of the earthquake activity in a tectonic fracture zone, then it could be observed with a unitary time scale of ten years. If we are studying the behaviour of a disablement resulting from a job related illness, the unitary time could be one year, and so on. So it results that the phenomenon of time evolution is continuous, nevertheless usually, the observations are discrete in time. Consequently, if we construct a model to be fitted with real data, in our opinion, it would be better to begin with discrete time models. 1.2 DTSMP Definition Though DTHSMP and DTNHSMP definitions are similar to the continuous ones, for the sake of completeness, we will recall these definitions using directly the terminology used for discrete time models. Let I={1, 2, …, m} be the state space and let { }, , PΩ ℑ be a probability space. Let us also define the following random variables: :nJ IΩ → , :nT Ω → . (1.1) Definition 1.1 The process ( , )n nJ T is a discrete time homogeneous Markov renewal process or a discrete time non-homogeneous Markov renewal process if

132 Chapter 4

the kernels Q associated with the process are defined respectively in the following way: [ ] ( )[ ]1 1( ) , - | , ,ij n n n nQ t P J j T T t J i i j I t+ += = = ≤ = ∈ ∈Q , (1.2) [ ] ( )[ ]1 1( , ) , | , , , ,ij n n n nQ s t P J j T t J i T s i j I s t+ += = = ≤ = = ∈ ∈Q . (1.3) As in the continuous time case, it results that for the homogeneous case, we define: [ ] lim ( ) ; , ,ij ijt

p Q t i j I t→∞

⎡ ⎤= = ∈ ∈⎣ ⎦P . (1.4)

For the non-homogeneous case, we obtain: [ ]( ) lim ( , ) ; , , ,ij ijt

p s Q s t i j I s t→∞

⎡ ⎤= = ∈ ∈⎣ ⎦P , (1.5)

P being the transition matrix of the embedded Markov chain of the process. Furthermore it is necessary to introduce the probability that the process will leave the state i before or at a time t: [ ] ( )[ ]1( ) - | ,i n n nH t P T T t J i+= = ≤ =H (1.6) [ ] ( )[ ]1( , ) | , .i n n nH s t P T t J i T s+= = ≤ = =H (1.7) From the results of Chapter 3, we know that obviously:

1 1

( ) ( ) and ( , ) ( , )m m

i ij i ijj j

H t Q t H s t Q s t= =

= =∑ ∑ . (1.8)

The following probabilities only have sense in the discrete time case and to be concise, we present first the definition for the homogeneous case and then for the non-homogeneous one. Definition 1.2 Matrix B is defined as follows: [ ] ( )[ ]1 1( ) , - | ,ij n n n nb t P J j T T t J i+ += = = = =B (1.9) [ ] ( )[ ]1 1( , ) , | , .ij n n n nb s t P J j T t J i T s+ += = = = = =B (1.10) From Definition 1.1 it results that:

(0) 0 if 0,

1 if 1,2,...,ij

ij

Q tb (t) Q (t) Q (t ) tij ij

= =⎧⎪= ⎨ − − =⎪⎩ (1.11)

( , ) 0 if ,

( , ) ( , ) ( , 1) if .ij

ij

Q s s t sb s t Q s t Q s t t sij ij

= =⎧⎪= ⎨ − − >⎪⎩ (1.12)

Definition 1.3 The discrete time conditional distribution functions of the waiting times given the present and the next states, are given by: [ ] ( )[ ]1 1( ) - | , ,ij n n n nF t P T T t J i J j+ += = ≤ = =F (1.13) [ ] ( )[ ]1 1( , ) | , , .ij n n n nF s t P T t J i J j T s+ += = ≤ = = =F (1.14)

Discrete time SMP and numerical solution 133

Obviously the related probabilities can be obtained by means of the following formulas:

1

( ) if 0,( )

( ) if 0,ij ij ij

ij

Q t / p pF tij U t p

≠⎧= ⎨ =⎩

(1.15)

1

( , ) ( ) if ( ) 0,( , )

( , ) if ( ) 0,ij ij ij

ij

Q s t / p s p sF s tij U s t p s

≠⎧= ⎨ =⎩

(1.16)

where 1 1( ) ( , ) 1 ,U t U s t s t= = ∀ . Now, we can introduce the discrete time semi-Markov process ( )( ),Z Z t t= ∈ where { }( )( ) , ( ) max :N t nZ t J N t n T t= = ≤ representing the state occupied by the process at time t. For i,j=1,…,m, the transition probabilities are defined in the following way: ( )0( ) Pij tt Z j | Z iφ = = = (1.17) for the homogeneous case; for the non-homogeneous case, we have: ( )( , ) P .ij t ss t Z j | Z iφ = = = (1.18) They are obtained by solving the following evolution equations:

1 1

( ) (1 ( )) ( ) ( ),m t

ij ij i i jt H t b tβ ββ ϑ

φ δ ϑ φ ϑ= =

= − + −∑∑ (1.19)

1 1

( , ) (1 ( , )) ( , ) ( , ),m t

ij ij i i js

s t H s t b s tβ ββ ϑ

φ δ ϑ φ ϑ= = +

= − +∑ ∑ (1.20)

where, as usual, δij represents the Kronecker symbol. 2 NUMERICAL TREATMENT OF SMP In this section, we present the numerical solutions of the evolution equation of continuous time semi-Markov process in homogeneous and non-homogeneous cases. The proposed approach uses a general quadrature method and we will prove that the numerical solution tends to the solution of the evolution equation of the continuous time HSMP. It will also be shown that, using a very particular quadrature formula for the numerical solution of evolution equations of continuous time processes, it is possible to obtain the evolution equation of discrete time processes. These results were obtained in Corradi et al (2004) and Janssen-Manca (2001a) generalizing the classical results on integral equation numerical solutions (see Baker (1977)). Let us consider a continuous time SMP of kernel Q supposed to be differentiable. First of all we write down the evolution equations of the SMP (see Chapter 3, relation (10.2) and (21.18)) as follows:

134 Chapter 4

1 0

( ) (1 ( )) ( ) ( )tm

ij ij i i jt H t Q t dβ ββ

φ δ ϑ φ ϑ ϑ=

= − + −∑∫ , (2.1)

1

( , ) (1 ( , )) ( , ) ( , )tm

ij ij i i js

s t H s t Q s t dβ ββ

φ δ ϑ φ ϑ ϑ=

= − +∑∫ , (2.2)

where ijQ represents the derivative respect to time of ijQ . Each generic quadrature formula can be written as (see Evans (1993)):

,00

( ) ( ),kh k

k ll

f t dt w f lh=

≅ ∑∫ (2.3)

where h is the step length, ,, , , k lk N k N w≤ ∈ are the weights related to the quadrature formula (10.1); they are functions of both the end point and the point in which we compute the function value. If we set: ( ) (1 ( )) ,ij ij ijd t H t δ= − (2.4) ( , ) (1 ( , )) ,ij ij ijd s t H s t δ= − (2.5) we obtain respectively:

1 0

( ) ( ) ( ) ( ) ,m k

ij ij k lj ill

kh d kh w kh h Q hττ

φ φ τ τ= =

⎛ ⎞= + −⎜ ⎟

⎝ ⎠∑ ∑ (2.6)

, ,1

( , ) ( , ) ( , ) ( , )m k

ij ij u k lj ill u

uh kh d uh kh w h kh Q uh hττ

φ φ τ τ= =

⎛ ⎞= + ⎜ ⎟

⎝ ⎠∑ ∑ , (2.7)

where h is the step length, w the weights related to the quadrature formulas, 0 , , , ,u k N u k N≤ ≤ ≤ ∈ such that and [0, ]Nh Y Y= is the integration interval. Now we proceed showing only the homogeneous case but all the results given for the homogeneous case hold in both cases. The reader interested in acquiring more details can refer to Janssen and Manca (2001a) and Corradi et al (2004). We suppose we have already computed:

1 1 1 1

2 2 2 2

3 3 3 3

(0) ( ) (2 ) (( 1) )(0) ( ) (2 ) (( 1) )(0) ( ) (2 ) (( 1) )

(0) ( ) (2 ) (( 1) )

j j j j

j j j j

j j j j

mj mj mj mj

h h k hh h k hh h k h

h h k h

φ φ φ φφ φ φ φφ φ φ φ

φ φ φ φ

−−−

−

(2.8)

where 1 k N≤ ≤ and (0) (0) (1 (0)) , 1, , , 1, ,ij ij i ijH i m j mφ φ δ= = − = =… … . (2.9) Then from (2.6) it follows that for a fixed j and 1, ,i m∀ = … :

01 1 1

( ) ( ) (0) ( ) ( ) ( )m m k

ij k lj il ij k lj ill l

kh w kh Q d kh w kh h Q hττ

φ φ φ τ τ= = =

⎛ ⎞− = + −⎜ ⎟

⎝ ⎠∑ ∑ ∑ . (2.10)


From relation (2.8), we get

1 0 1 11

2 0 2 21

01

( ) ( ) (0)

( ) ( ) (0)

( ) ( ) (0)

m

j k lj llm

j k lj ll

m

mj k lj ml ml

kh w kh Q c

kh w kh Q c

kh w kh Q c

φ φ

φ φ

φ φ

=

=

=

− =

− =

− =

∑

∑

∑

(2.11)

where, for convenience, the mici ,,1, …= represent the second member of (2.10). The linear system (2.11) in the unknowns mikhij ,,1),(~ …=φ admits solution if the coefficient matrix is non-singular. The following theorem holds: Theorem 2.1 Assume that [ ] [ ]: 0, , : 0,ij ijQ Y Yφ→ → (2.12) and { }1, , ,q N N∈ ∈… , such that YNh ≤ . Furthermore let Nkkhkhh ijij

kij ,,2,1,0),()(~)( …=−= φφξ (2.13)

where )(khijφ is the solution of (2.1) and )(~ khijφ is the solution of (2.6) and

.)()(1∑=

=m

i

kij

k hh ξη (2.14)

Furthermore, let:

∞<==≤≤≤ h

www ku

NkuN

0max , (2.15)

01 1 0

( ) ( ) ( ) ( ) ( )m m kkhk

ij lj il ku lj ill l u

t h kh Q d w kh uh Q uhφ τ τ τ φ= = =

⎡ ⎤= − − −⎢ ⎥

⎣ ⎦∑ ∑ ∑∫ , (2.16)

,)()(1∑=

=m

i

kij

k hthσ (2.17)

),(max)()( hhh kNhq

N σττ≤≤

== (2.18)

1

0( ) ( ).

qu

uh hξ η

−

== ∑ (2.19)

Then, if 1( )ijQ t c≤ for [0, ],t Y∈ we have:

136 Chapter 4

11

1 1

( ) ( )( ) exp , , 1, ,1 1

NNk

N N

mw c khh mhw c hh k q q Nmhw c mhw c

τ ξη + ⎛ ⎞≤ = +⎜ ⎟− −⎝ ⎠… (2.20)

given that 1, , , 1Nm h w c < Proof This theorem is a particular case of the one given in Janssen and Manca (2001a). Remark 2.1 In general, equation (2.6) cannot be solved exactly. Then if the values ˆ ( )ij khφ give the approximate solution of (2.6) (for 0≥k ), then

1 0

ˆ ˆ( ) ( ) ( ) ( )m k

kij ij k lj il ij

lkh d kh w kh h Q hτ

τφ φ τ τ ς

= =

⎛ ⎞= + − +⎜ ⎟

⎝ ⎠∑ ∑ , (2.21)

if we suppose that: ˆ (0) (0) (0).ij ij ijdφ φ= = Then setting: ˆ( ) ( ) ( )k

ij ij ijh kh khπ φ φ= − (2.22) it follows that { }0 ˆ( ) (0) (0), , 1, , .ij ij ijh i j mπ φ φ= − ∀ ∈ … (2.23) Using relations (2.6) (2.21) and (2.22), we get for 1≥k ,

1 0

( ) ( ) ( ) .m k

k k kij k il lj ij

lh w Q h hτ

ττ

π τ π ς−

= =

⎛ ⎞= +⎜ ⎟

⎝ ⎠∑ ∑ (2.24)

Using (2.15) and summing up with respect to the first index, it results that:

11 1 0 1

( ) ( ) .m m k m

k klj N lj ij

l l lh m w h c hτ

τπ π ς

= = = =≤ +∑ ∑∑ ∑ (2.25)

If we set:

1

( ) ( ) ,m

k klj

lh hη π

=

= ∑ and 1

,m

k kij

lρ ς

=

= ∑ (2.26)

it follows that:

10

( ) ( ).k

k kNh mw hc hτ

τη ρ η

=

≤ + ∑ (2.27)

Given the following positions:

11 0max , ( ) ( )

kk k

N N Nk Nh mw hc hτ

ττ ρ η τ η

≤ ≤ =

= ≤ + ∑ , (2.28)

that is:

( )1

1 10

1 ( ) ( )k

k uN N N

umw hc h mw hc hη τ η

−

=

− ≤ + ∑ , (2.29)

it follows finally that:


1

1 1( ) exp , 1, ,

1 1NNk

N N

m w c k hh k N

mh w c mh w cτη ⎛ ⎞≤ =⎜ ⎟− −⎝ ⎠

… . (2.30)

Remark 2.2 In the non-homogeneous case, as s is fixed in the system of integral equations (2.2), τ is the only parameter, so for each [ ]0,s Y∈ the result of Theorem 2.1 holds. 3. DTSMP AND SMP NUMERICAL SOLUTIONS In the previous section, we gave general formulas for the discretization of continuous time HSMP and NHSMP with a finite number of states. With the most simple quadrature method (rectangle formula), we get:

1 1

( ) ( ) ( ) ( ) ,m k

ij ij lj ill

kh d kh h kh h Q hτ

φ φ τ τ= =

⎛ ⎞= + −⎜ ⎟

⎝ ⎠∑ ∑ (3.1)

1 1

( , ) ( , ) ( , ) ( , ) .m k

ij ij lj ill u

uh kh d uh kh h h kh Q uh hτ

φ φ τ τ= = +

⎛ ⎞= + ⎜ ⎟

⎝ ⎠∑ ∑ (3.2)

Here, the sum on the time starts from 1 (u+1) as, the probability of changing state with a waiting time 0 is 0. Substituting in relations (3.1) and (3.2) the differential with the increment and with h=1, it results that:

( )1 1

( ) ( ) ( ) ( ) ( 1) ,m k

ij ij lj il ill

k d k k Q Qτ

φ φ τ τ τ= =

⎛ ⎞≅ + − − −⎜ ⎟

⎝ ⎠∑ ∑ (3.3)

( )1 1

( , ) ( , ) ( , ) ( , ) ( , 1) .m k

ij ij lj il ill u

u k d u k k Q u Q uτ

φ φ τ τ τ= = +

⎛ ⎞≅ + − −⎜ ⎟

⎝ ⎠∑ ∑ (3.4)

Furthermore, taking into account relations (1.9) and (1.10) it results that

1 1

( ) ( ) ( ) ( )m k

ij ij lj ill

k d k k bτ

φ φ τ τ= =

≅ + −∑∑ , (3.5)

1 1

( , ) ( , ) ( , ) ( , ).m k

ij ij lj ill u

u k d u k k b uτ

φ φ τ τ= = +

≅ + ∑ ∑ (3.6)

In this way, the evolution equations of the (DTHSMP) and (DTNHSMP) as defined in relations (1.19) and (1.20) are obtained:

1 1

( ) ( ) ( ) ( ),m k

ij ij lj ill

k d k k bτ

φ φ τ τ= =

= + −∑∑ (3.7)

1 1

( , ) ( , ) ( , ) ( , ).m k

ij ij lj ill u

u k d u k k b uτ

φ φ τ τ= = +

= +∑ ∑ (3.8)

If the discretization step is h, then relations (3.7) and (3.8) become:

1 1

( ) ( ) ( ) (( ) )m k

h h h hij ij il lj

lkh d kh b h k h

τφ τ φ τ

= =

= + −∑∑ , (3.9)

138 Chapter 4

1 1

( , ) ( , ) ( , ) ( , ).m k

h h h hij ij il lj

l uuh kh d uh kh b uh h h kh

τφ τ φ τ

= = +

= +∑ ∑ (3.10)

Now the equations (3.9) and (3.10) can be rewritten in matrix form as follows:

1

( ) ( ) ( ) (( ) ),k

h h h hkh kh h k hτ

τ τ=

= + ∗ −∑Φ D B Φ (3.11)

1

( , ) ( , ) ( , ) ( , ),k

h h h h

uuh kh uh kh uh h h kh

ττ τ

= +

= + ∗∑Φ D B Φ (3.12)

or equivalently:

1

( ) ( ) (( ) ) ( ),k

h h h hkh h k h khτ

τ τ=

− ∗ − =∑Φ B Φ D (3.13)

1

( , ) ( , ) ( , ) ( , ), , .k

h h h h

uuh kh h kh h kh uh kh k u k

ττ τ

= +

− ∗ = ∈ ≤∑Φ B Φ D (3.14)

Taking into account that k ∈ , both equations (3.13) and (3.14) can be written more compactly as: .h h h∗ =U Φ D (3.15) For the homogeneous case, it results that:

( )

(2 ) ( )(3 ) (2 ) ( )

h

h h h

h h h

hh hh h h

⎡ ⎤⎢ ⎥−⎢ ⎥

= − −⎢ ⎥⎢ ⎥− − −⎢ ⎥⎢ ⎥⎣ ⎦

I 0 0 0B I 0 0

U B B I 0B B B I

, (3.16)

(0)( )

(2 )(3 )

h

h

h h

h

hhh

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

ΦΦ

Φ ΦΦ

,

(0)( )

(2 )(3 )

h

h

h h

h

hhh

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

DD

D DD

and in the non-homogeneous case:

(0, ) (0,2 ) (0,3 )( ,2 ) ( ,3 )

(2 ,3 )

h h h

h h

h h

h h hh h h h

h h

− − −⎡ ⎤⎢ ⎥− −⎢ ⎥⎢ ⎥= −⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

I B B B0 I B B

U 0 0 I B0 0 0 I

, (3.17)


(0, ) (0,2 ) (0,3 )( ,2 ) ( ,3 )

.(2 ,3 )

h h h

h h

h h

h h hh h h h

h h

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

I D D D0 I D D

D 0 0 I D0 0 0 I

The following theorem holds for both homogeneous and non-homogeneous cases. Theorem 3.1 Equations (3.13) and (3.14) admit a unique solution. Proof The determinant of matrix hU is absolutely convergent (Riesz (1913)); more precisely det( ) 1h =U and consequently matrix hU is invertible. In the homogenous case equation (3.13) can be seen as an infinite linear system with an infinite number of unknowns. It ensues from Theorem 3.1 that the system is solvable. To solve such a system it is usually necessary to apply the truncation method Riesz (1913); but in our case, it is very simple to find the solution. Obviously, (0)h =Φ I and, once (0)hΦ is known we get: ( ) ( ) (0) ( )h h h hh h h= ∗ +Φ B Φ D . (3.18) Once (0), ( ), , ( )h h hh khΦ Φ Φ… are known, then:

1

1(( 1) ) ( ) (( 1 ) ) (( 1) )

kh h h hk h h k h k h

ττ τ

+

=

+ = ∗ + − + +∑Φ B Φ D (3.19)

and it is not necessary to apply the truncation method. Furthermore, these results are obtained without any matrix inversion. As in the homogeneous case, equation (3.14) can be seen as an infinite linear system with an infinite number of unknowns, but the non-homogeneous matrix equation is different from the homogeneous one because the coefficient matrix is upper triangular in the non-homogeneous case and lower triangular in the homogeneous. Also in this case the truncation method should not be applied. In the homogeneous case this result is trivial. In the non-homogeneous case the result is not so immediate. We report, with more precision, the following result given in and Janssen and Manca (2001a). Proposition 3.1 The solution of the infinite order linear system (3.14) can be given explicitly step by step. Proof For the proof see Janssen-Manca (2001a)

140 Chapter 4

Remark 3.1 Here it should also be mentioned that the upper triangularity of the coefficient block matrix and the fact that the matrices on the main diagonal are identity matrices implies that the results are obtained with no matrix inversion. The following two theorems hold for both the homogeneous and non-homogeneous cases but only the homogeneous results will be given. A straightforward proof of the randomness of the matrix hΦ is possible by exploiting the following: Theorem 3.2 The matrices ( )h khΦ are stochastic. Proof The result is true for (0)h =Φ I. We suppose that it is true for

1, , kτ = … .We have to check what happens under these hypotheses for k+1. From (1.11) the equation (3.19) becomes:

( ) ( )( )

1

1(( 1) ) ( 1) (( 1 ) )

(( 1) ).

kh h h h

h

k h h h k h

k hτ

τ τ τ+

=

+ = − − + −

+ +

∑Φ Q Q Φ

D (3.20)

To prove that (( 1) )h k h+Φ is stochastic we have to show that:

1

(( 1) ) 1, 1, , .m

hij

jk h i mφ

=

+ = =∑ … (3.21)

An element of (3.20) is given by:

1

1 1

1

1 1

(( 1) ) (( 1) ) ( ) (( 1 ) )

(( 1) ) (( 1 ) ).

k mh h h hij ij il lj

l

k mh hil lj

l

k h d k h Q h k h

Q h k h

τ

τ

φ τ φ τ

τ φ τ

+

= =

+

= =

+ = + + + −

− − + −

∑∑

∑∑ (3.22)

Summing up with respect to j and taking into account relations (2.4), (1.8), (1.19) and the inductive hypothesis, the following result is obtained:

1

1 1 1 1 1

1

1 1 1

(( 1) ) 1 (( 1) ) ( ) (( 1 ) )

(( 1) ) (( 1 ) ) 1.

m m k m mh h h hij ij il lj

j j l j

k m mh hil lj

l j

k h Q k h Q h k h

Q h k h

τ

τ

φ τ φ τ

τ φ τ

+

= = = = =

+

= = =

+ = − + + + −

− − + − =

∑ ∑ ∑∑ ∑

∑∑ ∑(3.23)

Now let Z be a continuous time HSMP with Φ as evolution equation and { }nT as sequence of the state change times. If we set:


nhn

TT hh

⎢ ⎥= ⎢ ⎥⎣ ⎦ (3.24)

and ( )h

nZ t J= if 1h h

n nT t T +≤ < , (3.25) then hZ is a DTHSMP with evolution equations given by:

1 1

( ) ( ) ( ) (( ) )m k

h h h hij ij il lj

lkh d kh b h k h

τφ τ φ τ

= =

= + −∑∑ (3.26)

and it is defined on the same probability space { }, , PΩ ℑ of Z. Given Ω∈ω the following result holds P-almost. Theorem 3.3 The hZ process converges to Z for 0h → in the Skorohod topology1. Proof It must be shown that 0T∀ > there exists a time rescaling sequence { }hλ where hλ is a continuous, strictly increasing and surjective function from [0, )+∞ to [0, )+∞ such that: sup ( ) 0h

t Tt tλ

≤− → if 0h → (3.27)

and ( )sup ( ) ( ) 0h h

t TZ t Z tλ

≤− → if 0h → ; (3.28)

(see Ethier and Kurtz (1986)). Obviously, nT → +∞ if n → +∞ , (3.29) then it is sufficient to verify that the proposition holds for , .nT T n= ∀ If we set: 1 0min , 0h h h

k kl k nh T T T−≤ ≤< − = , (3.30)

let hλ be the linear function that transforms the intervals [ ]1,k kT T− into [ ]1,h h

k kT T− with 1 k n≤ ≤ and 0 0T = , given by: ( ) ( ),h n

h n nt T t T t Tλ = + − ≥ . (3.31) Then { }hλ verifies the proposition's conditions and: ( ( )) ( ) 0 .h h

nZ t Z t t Tλ − = ∀ ≤ (3.32) Then hZ converges to Z in the Skorohod topology with probability 1 and therefore, in particular, hZ converges in law, i.e. in the weak sense of stochastic processes. 1 Skorokhod topology is the topology defined on the set of trajectories of stochastic processes. For more details see (Billingsley (1968)).

142 Chapter 4

Remark 3.2 The weak convergence of stochastic processes is in reality the weak convergence of their laws, where the laws are probability measures induced from the processes on the space of their trajectories. In this case it is the space of right continuous functions with left limits, in which the Skorokhod topology holds. Finally it is to be observed that the hλ sequence depends on nT by means of (3.30). 4 SOLUTION OF DTHSMP AND DTNHSMP IN THE TRANSIENT CASE: A TRANSPORTATION EXAMPLE 4.1 Principle Of The Solution Generally speaking it is clear that, for the purpose of application, it is more worthwhile to solve systems (3.13) and (3.14) in a finite time horizon, which means that the process is solved in the transient case. In the following, an algorithm useful for solving both evolution equations is briefly presented. First of all, an epoch T is fixed and, in this light, equation (3.15) for both cases becomes: T T T∗ =U Φ D . (4.1) The algorithms solve the linear system (4.1), in the sense that for known matrices T F and P, it determines the matrix of the unknown TΦ by means of two iterative procedures. For the algorithms it is not necessary to compute the U matrix, but it is enough to construct the matrices B whose elements are defined respectively in (1.11) and (1.12). The variables involved in the algorithms are: INPUTS: m, T, P (embedded M.C.), T F (Matrix of the increasing distribution function of waiting times) RESULTS: T Q , T B , T S , T D , TΦ STEPS: Reads m, T Reads P Reads T F


Constructs T Q Constructs T B Constructs T S Constructs T D Solves the system and finds TΦ The steps are very general and hold in both cases. As we already said, the main difference is given by the fact that in the homogeneous case the matrices are lower triangular and upper triangular in the non-homogeneous case, see (3.16) and (3.17). For a complete description of algorithms respectively in homogeneous and non-homogeneous cases, we refer to Corradi et al (2004) and Janssen-Manca (2001a). 4.2 Semi-Markov Transportation Example 4.2.1 Homogeneous Case In this first example of a semi-Markov model, we extend to a semi-Markov environment the transportation problem presented in section 9.6 of Chapter 2. We consider that a taxicab driver will work for eight hours. So we will work in the transient case within 32 time periods, which means that a period is a time interval of 15 minutes. In this way we can consider the full working time of a driver. In this model, ( )ij tφ represents the probability that a driver who is in the state i will be in the state j after a time t and we have to solve the evolution equation. This example is really simple so all the steps that are necessary to solve the DTHSMP could be shown. The input is: m = 3, T = 32. The matrix P was given in the formula (9.108) of Chapter 2 but in real application it could be constructed by data in the following way. We are supposed to know all the runs that were driven in one month by all the taxicab drivers, so we know for each run the starting zone, the arriving zone and the course duration. We refer to our case study, so we have three states, and we should construct the transition matrix P by the known data. We could construct a matrix A where

ija = number of runs from the zone i to the zone j in the month. Then, it follows that:

144 Chapter 4

3 .ijij

ikk

ap

a=

∑ (4.2)

The matrix [ ]( )ijF t=F , the discrete time increasing probability distribution of the waiting time in each state i given that the next state to be successively occupied is j, should be constructed by the data. The way to construct these d.f. by the data is the following. We would construct for each i and j the related d.f. (0), (1), (2), , (32)ij ij ij ijF F F F… . (4.3) From our data we compute the vector ( (1), (2), , (32), (33))ij ij ij ij ijv v v v=v … (4.4) where (1)ijv represents the number of all the runs that have a duration (including also the waiting time of the taxi driver before beginning the run) less than or equal to 15 minutes, (2)ijv the number of all the courses that have a duration greater than 15 minutes and less than or equal to 30 minutes and so on. In (33)ijv there will be the number of all the runs from i and j that have a duration larger than eight hours if any. From the vector ijv we can construct the vector ijw :

1( ) ( ) 1, ,33

t

ij ijs

w t v s t=

= =∑ … . (4.5)

So, we finally get the elements of the matrix F: ( )

(0) 0, ( ) , 1, ,32(33)

ijij ij

ij

w tF F t t

w= = = … . (4.6)

To illustrate this method proposed for real data, we will here construct artifical data and find matrix F by means of pseudorandom generator numbers. Our example is very simple (only three states) but with 32 time periods, reporting of all the matrices involved in the computation will be too long and so we will report for each matrix the time periods 1, 5, 10, 20, 30 and 32. We get the following results:

Matrix F F(1) F(5) F(10)

0.0470 0.0396 0.0514 0.1835 0.1299 0.2349 0.2863 0.2581 0.3524 0.0200 0.0449 0.0009 0.1478 0.0847 0.0910 0.3083 0.2648 0.2924 0.0456 0.0168 0.0325 0.1314 0.1472 0.1106 0.2828 0.2491 0.3183

F(20) F(30) F(32) 0.5288 0.5628 0.6047 0.8268 0.8377 0.9236 0.8468 0.8574 0.9938 0.5877 0.4767 0.5842 0.8943 0.7982 0.8997 0.9423 0.8272 0.9382 0.5500 0.4956 0.6617 0.8721 0.8131 0.8376 0.9307 0.8478 0.9028


Note that, always, F(0)=0 because there are no movements in a time 0, and F(32)<1 because it is the last time period and therefore a truncated d.f. After the computation of matrix F, we can compute the kernel Q as follows (see relation (1.15)) ( ) ( )t t= ⋅Q F P , (4.7) where the symbol ⋅means matrix product element by element.

Matrix Q Q(1) Q(5) Q(10)

0.0235 0.0158 0.0051 0.0918 0.0520 0.0235 0.1432 0.1032 0.0352 0.0060 0.0269 0.0001 0.0443 0.0508 0.0091 0.0925 0.1589 0.0292 0.0091 0.0017 0.0227 0.0263 0.0147 0.0775 0.0566 0.0249 0.2228

Q(20) Q(30) Q(32) 0.2644 0.2251 0.0605 0.4134 0.3351 0.0924 0.4234 0.3430 0.0994 0.1763 0.2860 0.0584 0.2683 0.4789 0.0900 0.2827 0.4963 0.0938 0.1100 0.0496 0.4632 0.1744 0.0813 0.5863 0.1861 0.0848 0.6319 The next matrix to be computed is the matrix B using the following result:

if 0,

( )( ) ( 1) if 0.

tt

t t t=⎧= ⎨ − − >⎩

0B

Q Q (4.8)

Matrix B B(1) B(5) B(10)

0.0235 0.0158 0.0052 0.0194 0.0061 0.0046 0.0061 0.0079 0.0032 0.0060 0.0269 0.0001 0.0029 0.0153 0.0009 0.0095 0.0303 0.0040 0.0091 0.0017 0.0227 0.0110 0.0023 0.0257 0.0009 0.0006 0.0331

B(20) B(30) B(32) 0.0222 0.0151 0.0000 0.0083 0.0097 0.0034 0.0062 0.0059 0.0053 0.0033 0.0138 0.0045 0.0067 0.0251 0.0048 0.0128 0.0047 0.0034 0.0038 0.0011 0.0020 0.0115 0.0045 0.0040 0.0078 0.0013 0.0240 Let us just mention that the 0 in position 1,3 (20)b is a numerical zero in the sense that rounding this number at the fourth decimal gives 0. Then, we have to compute the estimation of matrix H, whose elements are the following, see relation (1.8):

1

0 if ,( )

( ) if .m

ijik

k

i jH t

Q t i j=

≠⎧⎪= ⎨

=⎪⎩∑ (4.9)

146 Chapter 4

0 0.0330 0 0 0.1043 0 0 0.2806 0 0 0 0.0335 0 0 0.1184 0 0 0.3043

H(20) H(30) H(32) 0.5500 0 0 0.8408 0 0 0.8658 0 0

0 0.5207 0 0 0.8372 0 0 0.8728 0 0 0 0.6228 0 0 0.8421 0 0 0.9028

We know that these elements represent the probability to leave the state i in a period less than or equal to the period t, and so have sense only in the main diagonal of each submatrix. The next matrix D,whose elements represent the probability of remaining in the state for t periods, is given by ( ) ( )t t= −D I H . (4.10) We get:

Matrix D D(1) D(5) D(10)

0.9555 0 0 0.8328 0 0 0.7184 0 0 0 0.9670 0 0 0.8957 0 0 0.7194 0 0 0 0.9665 0 0 0.8816 0 0 0.6957

D(20) D(30) D(32) 0.4500 0 0 0.1592 0 0 0.1342 0 0

0 0.4793 0 0 0.1628 0 0 0.1272 0 0 0 0.3772 0 0 0.1579 0 0 0.0972

The matrix we look for, that is Φ , is the solution of the evolution equation of the DTHSMP. Here, ( )ij tφ represents the probability that a taxicab driver being at time 0 in zone i will be after t periods, in the state j. From the results given below, any row of the submatrix ( )tΦ is indeed a probability distribution. The results are:

Matrix Φ Φ (1) Φ (5) Φ (10)

0.9790 0.0158 0.0052 0.9228 0.0530 0.0242 0.8553 0.1065 0.0382 0.0060 0.9939 0.0001 0.0437 0.9465 0.0098 0.0916 0.8767 0.0317 0.0091 0.0017 0.9892 0.0264 0.0154 0.9582 0.0582 0.0288 0.9130

Φ (20) Φ (30) Φ (32) 0.6973 0.2320 0.0707 0.5337 0.3490 0.1173 0.5132 0.3587 0.1281 0.1767 0.7548 0.0685 0.2745 0.6116 0.1139 0.2902 0.5882 0.1216 0.1220 0.0686 0.8094 0.2039 0.1278 0.6683 0.2196 0.1386 0.6418

Matrix H H(1) H(5) H(10)

0.0445 0 0 0.1672 0 0 0.2816 0 0


4.2.2 Non-Homogeneous Case As above in the homogoneous case, we consider that a taxicab driver will work for eight hours with time intervals of 15 minutes and so we will consider the transient case within 32 time periods for which ( , )ij s tφ represents the probability that a driver being in state i at time s will be in state j at time t. Though this example is one of the simplest that can be done, it will clearly confirm that non-homogeneity, as already shown, gives some intrinsic supplementary difficulties. Also, we will try to show all the steps that are necessary to understand the development of a DTNHSMP. This time, the input is: m = 3, T = 32, and the non-homogeneous Markov chain, reported in the following table.

P(0) P(5) P(10) 0.3 0.4 0.3 0.39 0.35 0.26 0.49 0.3 0.21 0.4 0.2 0.4 0.35 0.32 0.33 0.3 0.42 0.28

0.32 0.38 0.3 0.28 0.33 0.39 0.23 0.28 0.49 P(20) P(25) P(29)

0.54 0.35 0.11 0.5 0.4 0.1 0.5 0.4 0.1 0.3 0.62 0.08 0.3 0.6 0.1 0.3 0.6 0.1

0.23 0.18 0.59 0.24 0.13 0.63 0.2 0.1 0.7

Other input should be the matrix [ ]( , )ijF s t=F , the discrete time increasing probability distribution of the waiting time in each state i, given that the arrival time in the state i was at time s and that the next state to be successively occupied is j. From the data, we can construct these distribution functions, as in the homogeneous cases. To compute for each s, i and j the related d.f., ( , ), ( , 1), ( , 2), , ( ,32), 32,ij ij ij ijF s s F s s F s s F s s+ + ≤… (4.11) we first introduce the following quantities: ( ) ( ( , 1), ( , 2), , ( ,32), ( ,33))ij ij ij ij ijs v s s v s s v s v s= + +v … . (4.12)

( , 1)ijv s s + gives the number of all the runs for which the taxi driver arrived at time s in the state i and finished the new course in the state j in a time less than or equal to 15 minutes (including the waiting time before beginning the new course).

148 Chapter 4

Similarly ( , 2)ijv s s + gives the number of all the runs for which the taxi driver arrived at time s in the state i and finished the new run in the state j in a time more than 15 minutes and less than or equal to 30 minutes and so on. Finally, ( ,32)ijv s gives the number of all the courses from i and j that began at the arrival time s and finished after 7.45 hours but within the eight hours of the turn and ( ,33)ijv s the number of taxi drivers who arrived at time s in i and who will finish the next course in j, but after the eight hours. It is important to remark that in the semi-Markov environment the stopping time of the taxidriver before the beginning of another run is included in the duration of the course. From the vector ( )ij sv we can construct the vector ( )ij sw :

1

( , ) ( , ) 1, ,33t

ij ijh

w s t v s s h t s=

= + = +∑ … . (4.13)

We finally obtain the elements of the matrix F given by:

( , )

( , ) 0, ( , ) , 1, ,32.( ,33)

ijij ij

ij

w s tF s s F s t t s

w s= = = + … (4.14)

In place of real data not available here, we construct the F matrix by means of pseudorandom generator numbers as for the homogoneous case given above. We can then multiply matrices F and P and obtain the matrix Q with the relation ( , ) ( ) ( , )s t s s t= ⋅Q P F . (4.15) The next matrix to be computed is the matrix B; from the result:

0 if ,

( , )( , ) ( , 1) if ,

t ss t

s t s t t s≤⎧= ⎨ − − >⎩

BQ Q

(4.16)

we get: Using the relation

1

0 if ,( , )

( , ) if ,m

ijik

k

i jH s t

Q s t i j=

≠⎧⎪= ⎨

=⎪⎩∑ (4.17)

we finally obtain the matrix H. The elements of this matrix represent the probability of leaving the state i in the time that goes from s to t. They have sense only on the main diagonal of each submatrix. The matrix D representing the probabilites of remaining in the state from the time s up to the time t is obtained in the following way: ( , ) ( , )s t s t= −D I H . (4.18) For the last step, we have to compute the matrix Φ , the solution of the evolution equation of the DTHSMP whose element ( , )ij s tφ represents the probability that a taxicab driver being at time s in the zone i will be at time t in state j.


Here, it can be verified that any row of the submatrix ( , )s tΦ is indeed a probability distribution.

Matrix Φ

(0,1)Φ (0,10)Φ (0,20)Φ 0.9696 0.0255 0.0049 0.7936 0.1307 0.0757 0.6196 0.2089 0.1715 0.0123 0.9686 0.0191 0.1149 0.7845 0.1006 0.2473 0.5132 0.2395 0.0115 0.0046 0.9839 0.0862 0.1053 0.8085 0.2002 0.2142 0.5856

(5,6)Φ (5,15)Φ (5,25)Φ 0.9898 0.0084 0.0018 0.7812 0.1290 0.0898 0.5840 0.2421 0.1739 0.0213 0.9685 0.0102 0.1374 0.7324 0.1302 0.2551 0.5364 0.2085 0.0164 0.0157 0.9679 0.1153 0.1147 0.7700 0.2069 0.2375 0.5556

(10,11)Φ (10,21)Φ (10,31)Φ 0.9672 0.0242 0.0086 0.7275 0.1620 0.1105 0.5121 0.2971 0.1908 0.0201 0.9646 0.0153 0.1415 0.7095 0.1490 0.3241 0.4443 0.2316 0.0055 0.0057 0.9888 0.1254 0.1206 0.7540 0.2678 0.2726 0.4596

(20,21)Φ (20,26)Φ (20,31)Φ 0.9892 0.0034 0.0074 0.7926 0.1583 0.0491 0.5561 0.3145 0.1294 0.0186 0.9718 0.0096 0.1277 0.8174 0.0549 0.2883 0.6075 0.1042 0.0289 0.0095 0.9616 0.1287 0.0781 0.7932 0.2420 0.2192 0.5388

(25,26)Φ (25,28)Φ (25,32)Φ 0.9724 0.0096 0.0180 0.7794 0.1644 0.0562 0.4170 0.4360 0.1470 0.0715 0.9141 0.0144 0.1079 0.8313 0.0608 0.2924 0.5670 0.1406 0.0252 0.0009 0.9739 0.0552 0.0277 0.9171 0.2422 0.2194 0.5384

(29,30)Φ (29,31)Φ (29,32)Φ 0.8975 0.0661 0.0364 0.7097 0.2161 0.0742 0.4500 0.4351 0.1149 0.0246 0.9532 0.0222 0.1911 0.7463 0.0626 0.2827 0.5960 0.1213 0.1264 0.0287 0.8449 0.1378 0.0702 0.7920 0.1824 0.1675 0.6501

In the non-homogeneous case we report only the final results, the interested reader can find them in the internet site given in the introduction. 5 CONTINUOUS AND DISCRETE TIME REWARD PROCESSES In this part we will present undiscounted and discounted semi-Markov reward processes. Reward processes can be seen as a class of stochastic processes. In the non- homogeneous case it is possible to write more than 200 different evolution equations of Semi-Markov ReWard Processes (SMRWP). We develop only three

150 Chapter 4

cases, the simplest and the most general. For a wider approach the reader can refer to Janssen-Manca (2006). 5.1 Classification And Notation 5.1.1 Classification Of Reward Processes In this book we will apply semi-Markov processes mainly in finance, insurance and reliability problems. In all these fields, the association of a sum of money to a state of the system and to a state transition assumes great relevance. In general, this can be done by attaching a reward structure to the process. This structure can be seen as a random variable associated with the state occupancies and transitions (Howard (1971) vol. 2). The rewards can be of different kinds, but in this book we will, because of our kind of applications, consider only amounts of money. These amounts can be positive, if for the system they can be seen as a benefit and negative if they can be considered as a cost. The reward processes can be seen as classes of stochastic processes that we can classify in different cases. The following tables report the classification of the SMRWP. Process classification

Homogeneous Non-Homogeneous Continuous time Discrete time Non-discounted

Fixed interest rate Homogeneous interest law Discounted Variable interest rate Non-homogeneous interest law

Reward classification Time fixed rewards

Homogeneous rewards Time variable rewards Non-homogeneous rewards Transition (impulse) rewards

Immediate Discrete time Due

Permanence (rate) rewards

Independent on next transition Dependent on next transition


We will not present permanence rewards that depend on the next transition because in financial and insurance environments they do not have sense. In general the distinction between homogeneous and non-homogeneous cases is done for stochastic processes. Also an interest rate law can be defined as homogeneous if the discount factor is a function only of the length of the financial operation, non-homogeneous if the discount factor takes into account also the initial time of operation, not only the duration. In the same way, rewards can be fixed in time, can depend only on the duration or can be non-homogeneous in time. We will use the following notation:

, ( ), ( , )i i it s tψ ψ ψ : represent rewards given for permanence in state i; the first is time fixed, the second changes because of time and the third represents a time non-homogeneous permanence reward.

, ( ), ( , )ij ij ijt s tγ γ γ : represent the three different kinds of rewards given for the transition from state i to state j (impulse reward). In the discrete time case, the immediate case means that the reward is paid at the end of each period; in the due case the reward is paid at the beginning of the period. The impulse rewards γ represent lump sums that are paid at the transition instant. 5.1.2 Financial Parameters To study the process with discounting, let us recall some basic results for computing present values of amounts of money, annuities and also the related notation. For more details, refer to Volpe di Prignano (1985) or Kellison (1991). Fixed time interest rate:

1(1 )r e δν − −= + = : represents the one-period discount factor, where r is the interest rate and δ the corresponding intensity,

1 1 1; , ;1

t t t

t r t r tv r ea a d a

r d r

δ

δν

δ

−− − −= = = =

+ represent the present value of

respectively a unitary annuity-immediate, an annuity-due and a continuous time annuity,

1 1 1, ,r ra a ar d δ δ∞ ∞ ∞= = = represent the present value of infinite time unitary

annuities, also called perpetuities.

152 Chapter 4

Time variable interest rate: Now we suppose that the interest rates are variable and depend on the time period.:

0

( )1

1( ) (1 ) , ( )

k

k d

hh

k r k eδ τ τ

ν ν−

−

=

∫= + =∏ represent the k-period discount factor at

time 0 respectively in discrete and continuous time. They give, at time 0, the discounted value of one monetary unit to be paid at the end of period k,

1

1 0 0

( ) ( ), ( ) ( ), ( ) ( )tt t

i i ik k

k k k k dν ψ ν ψ ψ ϑ ν ϑ ϑ−

= =∑ ∑ ∫ represent the present value

respectively of an annuity-immediate, an annuity-due and a continuous annuity with variable rewards in the time and variable interest rate on a time horizon t. The infinite rates cases are given by the limit to ∞ of the three relations. They converge depending on the values of andψ ν .

( )1

1( , ) (1 ) , ( , )

t

s

t d

hh s

s t r s t eδ τ τ

ν ν−

−

= +

∫= + =∏ represent the t s− period discount

factor at time s, with homogeneous interest rates giving, at time s, the discounted value of one monetary unit to be paid at the end of period k, respectively in discrete and continuous time,

1

1( , ) ( , ), ( , ) ( , ), ( , ) ( , )

tt t

i i ik s k s s

s k s k s k s k s s dν ψ ν ψ ψ ϑ ν ϑ ϑ−

= + =∑ ∑ ∫ represent the present

value at time s respectively of an annuity-immediate, an annuity-due and a continuous annuity paid on the time interval ( ],s t with non-homogeneous rewards and variable interest rate.

( , )1

1( , ) (1 ) , ( , ) s

t s d

s hh s

s t r s e

ϑ

δ τ τ

ν ν ϑ−

−

= +

∫= + =∏ represent the t s− period discount

factors at time s, with non-homogeneous interest rates, 1

1( , ) ( , ), ( , ) ( , ), ( , ) ( , )

tt t

i i ik s k s s

s k s k s k s k s s dν ψ ν ψ ψ ϑ ν ϑ ϑ−

= + =∑ ∑ ∫ represent the present

value at time s respectively of an annuity-immediate, an annuity-due and a continuous time annuity paid on the time interval ( ],s t with non-homogeneous rewards and non-homogeneous interest rate. In the following sections, we will show that annuities are strongly related to the semi-Markov reward processes; for the Markov case see Janssen-Manca (2006). A reward structure can be considered as a very general structure that, given a financial and economic meaning can be very useful in stochastic modelling. For example, this behaviour is particularly efficient to construct models useful to follow the dynamic evolution of insurance problems. In this case, the


permanence in a state usually involves the payment of a premium or the receipt of a claim. Furthermore often the transition from one state to another induces some other cost or benefit. 5.2 Undiscounted SMRWP For each given case we will present the immediate, the due and the continuous cases, both in homogeneous and non-homogeneous environments. We will give first the simplest case (only with permanence rewards and fixed rate of interest and rewards) and after the general ones. The same cases will be given for discounted processes. 5.2.1 Fixed Permanence Rewards We assume that: a) rewards are fixed in time, b) rewards are given only for permanence in the state. First we present the immediate case.

( )iV t ( ( , )iV s t ) represents the Mean Total Reward (MTR) paid or received in t periods (from time s to time t), given that at time 0 (at time s) the system was in state i. At time 1 the evolution equation for the homogeneous immediate case is given by the following relation:

( )

1

1 1 0(1) 1 (1) (1) ( ) (1 )

m m

i i i ik i ik kk k

V H b b Vϑ

ψ ψ ϑ ϑ= = =

= − + + −∑ ∑∑ . (5.1)

To have a good understanding of the evolution equation, let us first say that relations (1.8) and (1.11) imply that (1) (1)ij ijb Q= and so relation (5.1) can be decomposed in the following way:

1

1 1 1 0(1) 1 (1) (1) ( ) (1 )

m m m

i ik i ik i ik kk k k

V Q Q Q Vϑ

ψ ψ ϑ ϑ= = = =

⎛ ⎞= − + + −⎜ ⎟⎝ ⎠

∑ ∑ ∑∑ (5.2)

where (0) 0, and (0) 0 ,k ikV k Q i k= ∀ = ∀ , and so:

(1) .i iV ψ= (5.3)

For the next step, we can write that:

2 2

1 1 1 1(2) (1 ( )) 2 ( ) ( ) (2 ).

m m

i i i ik i ik kk k

V H t b b Vϑ ϑ

ψ ϑ ψ ϑ ϑ ϑ= = = =

= − + + −∑∑ ∑∑ (5.4)

This time, two rewards must be paid but in different ways. We divide the evolution equation in three parts. - the term (1 ( )) 2i iH t ψ− represents the rewards obtained without state changes;

154 Chapter 4

- the expression 2

1 1( )

m

ik ik

bϑ

ϑ ψ ϑ= =∑∑ gives the rewards obtained before the change of

state. As (0) 0 ,ikb i k= ∀ , the sum on ϑ begins from 1;

- the double sum 2

1 1( ) (2 )

m

ik kk

b Vϑ

ϑ ϑ= =

−∑∑ gives the rewards paid or earned after the

transitions. For time t, we get the following general result:

1 1 1 1

( ) (1 ( )) ( ) ( ) ( ).m t m t

i i i ik i ik kk k

V t H t t b b V tϑ ϑ

ψ ϑ ψ ϑ ϑ ϑ= = = =

= − + + −∑∑ ∑∑ (5.5)

The general formula in the non-homogeneous case is:

1 1

1 1

( , ) (1 ( , ))( ) ( , )( )

( , ) ( , ).

m t

i i i ik ik s

m t

ik kk s

V s t H s t t s b s s

b s V t

ϑ

ϑ

ψ ϑ ϑ ψ

ϑ ϑ

= = +

= = +

= − − + −

+

∑ ∑

∑ ∑ (5.6)

In this simple case the due and the immediate processes correspond. So we report only the continuous cases.

( )0 01 1

( ) 1 ( ) ( ) ( ) ( )m mt t

i i i i ik ik kk k

V t H t t Q d Q V t dψ ψ ϑ ϑ ϑ ϑ ϑ ϑ= =

= − + + −∑ ∑∫ ∫ , (5.7)

( )

1

1

( , ) 1 ( , ) ( ) ( , ) ( )

( , ) ( , ) .

m t

i i i i iksk

m t

ik ksk

V s t H s t t s Q s s d

Q s V t d

ψ ψ ϑ ϑ ϑ

ϑ ϑ ϑ

=

=

= − ⋅ − + ⋅ −

+

∑∫

∑∫ (5.8)

5.2.2 Variable Permanence And Transition Rewards Here we assume that: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, Under these hypotheses, we get respectively for homogeneous and non-homogeneous environments, in the immediate cases the following results:

1 1 1 1

1 1 1 1

( ) (1 ( )) ( ) ( ) ( )

( ) ( ) ( ) ( ),

t m t

i i i ik ik

m t m t

ik ik ik kk k

V t H t b

b b V t

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= = = =

= = = =

= − +

+ + −

∑ ∑∑ ∑

∑∑ ∑∑ (5.9)


1 1 1 1

1 1 1 1

( , ) (1 ( , )) ( ) ( , ) ( )

( , ) ( ) ( , ) ( , ).

t m t

i i i ik is k s

m t m t

ik ik ik kk s k s

V s t H s t b s

b s b s V t

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= + = = + =

= = + = = +

= − +

+ +

∑ ∑ ∑ ∑

∑ ∑ ∑ ∑ (5.10)

In the due case we obtain:

1 1 1 1

1 1 1 1

( ) (1 ( )) ( 1) ( ) ( 1)

( ) ( ) ( ) ( ),

t m t

i i i ik ik

m t m t

ik ik ik kk k

V t H t b

b b V t

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= = = =

= = = =

= − − + −

+ + −

∑ ∑∑ ∑

∑∑ ∑∑ (5.11)

1 1 1

1 1 1 1 1

( , ) (1 ( , )) ( 1) ( , ) ( )

( , ) ( 1) ( , ) ( , ).

t m t

i i i ik iks k s

m t m t

ik i ik kk s k s

V s t H s t b s

b s b s V t

τ ϑ

ϑ

ϑ τ ϑ

ψ τ ϑ γ ϑ

ϑ ψ τ ϑ ϑ

= + = = +

= = + = = = +

= − − +

+ − +

∑ ∑ ∑

∑ ∑ ∑ ∑ ∑ (5.12)

The difference between immediate and due is given only by the time of payment of the rewards. The continuous cases are the following:

( )

( )

0 0 01

01

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ( ) ,

mt t

i i i ik ik

m t

ik k ikk

V t H t d Q d d

Q V t d

ϑψ τ τ ϑ ψ τ τ ϑ

ϑ ϑ γ ϑ ϑ

=

=

= − +

+ − +

∑∫ ∫ ∫

∑∫ (5.13)

( )

( )

1

1

( , ) 1 ( , ) ( ) ( , ) ( )

( , ) ( , ) ( ) .

mt t

i i i ik is s sk

m t

ik k iksk

V s t H s t d Q s d d

Q s V t d


ϑ ϑ γ ϑ ϑ

=

=

= − +

+ +

∑∫ ∫ ∫

∑∫ (5.14)

The presence of the lump sums given or taken at the moment of transition times is taken into consideration. 5.2.3 Non-Homogeneous Permanence And Transition Rewards In the last immediate case model, the rewards are non-homogeneous and so we have to consider only the non-homogeneous case. Assumptions are thus: a) rewards depend on the times s and t, b) permanence and transition rewards are non-homogeneous. Here, only the non-homogeneous case has sense and the evolution equations take the form:

156 Chapter 4

1 1 1 1

1 1 1 1

( , ) (1 ( , )) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) ( , ).

t m t

i i i ik is k s s

m t m t

ik ik ik kk s k s

V s t H s t s b s s

b s s b s V t

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= + = = + = +

= = + = = +

= − +

+ +

∑ ∑ ∑ ∑

∑ ∑ ∑ ∑ (5.15)

1 1 1 1

1 1 1 1

( , ) (1 ( , )) ( , 1) ( , ) ( , 1)

( , ) ( , ) ( , ) ( , ).

t m t

i i i ik is k s

m t m t

ik ik ik kk s k s

V s t H s t s b s s

b s s b s V t

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= + = = + =

= = + = = +

= − − + −

+ +

∑ ∑ ∑ ∑

∑ ∑ ∑ ∑ (5.16)

( )

( )

1

1

( , ) 1 ( , ) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) .

mt t

i i i ik is s sk

m t

ik k iksk

V s t H s t s d Q s s d d

Q s V t s d


ϑ ϑ γ ϑ ϑ

=

=

= − +

+ +

∑∫ ∫ ∫

∑∫ (5.17)

The other non-discounted cases can be treated in a similar way and are left to the reader, who can refer also to Janssen-Manca (2006). 5.3 Discounted SMRWP For the discounted case developed in this section, we assume that all the rewards are discounted at time 0 in the homogeneous case and at time s in the non-homogeneous case. Let us point out that these models, as we will see below, are very important for insurance applications. 5.3.1. Fixed Permanence And Interest Rate Cases In the first formulation of this case we suppose that: a) rewards are fixed in time, b) rewards are given only for permanence in the state, c) interest rate r is fixed. In this case ( )iV t represents the Rewards Mean Present Value (RMPV) paid or received in a time t, given that at time 0 the system is in state i. Under these hypotheses, a similar reasoning as before leads to the following result for the evolution equation, firstly for the homogeneous immediate case:

( )1

1 1 1 1

1 1 1(1) 1 (1) (1) ( ) (1 ) ,

m m

i i i ik i ik k ik k

V H b b Vϑ

ψ ν ψ ν ϑ ϑ ν ψ ν= = =

= − + + − =∑ ∑∑ (5.18)

1 1 1 1( ) (1 ( )) ( ) ( ) ( ) .

m t m t

i i i ik i ik kt r rk k

V t H t a b a b V t ϑϑ

ϑ ϑψ ϑ ψ ϑ ϑ ν

= = = =

= − + + −∑∑ ∑∑ (5.19)

For the non-homogeneous case, this last result becomes:


1 1

1 1

( , ) (1 ( , )) ( , )

( , ) ( , ) .

m t

i i i ik it s r srk s

m ts

ik kk s

V s t H s t a b s a

b s V t

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ ν

− −= = +

−

= = +

= − +

+

∑ ∑

∑ ∑ (5.20)

To explain these results, as for the continuous case, we divide the evolution equation in three parts. The meaning is the same given in the previous cases but we use annuity formulas. Let us just make the following comments: The term (1 ( , ))i i t s rH s t aψ −− represents the present value of the rewards obtained without state changes. More precisely (1 ( , ))iH s t− is the probability of remaining in the state i and i t s raψ −

is the present value of a constant annuity

of t s− payments iψ .

The term 1 1

( , )m t

ik i s rk s

b s aϑϑ

ϑ ψ −= = +∑ ∑ gives the present value of the rewards t obtained

before the change of state.

The term 1 1

( , ) ( , )m t

sik k

k sb s V t ϑ

ϑϑ ϑ ν −

= = +∑ ∑ gives the present value of the rewards paid

or earned after the transitions and as the change of state happens at timeϑ , it is necessary to discount the reward values at time s. In the due environment we obtain:

1

1 1 1 1( ) (1 ( )) ( ) ( ) ( ) ,

m t m t

i i i ik i ik kt r rk k

V t H t a b a b V t ϑϑ

ϑ ϑψ ϑ ψ ϑ ϑ ν −

= = = =

= − + + −∑∑ ∑∑ (5.21)

1 1

1

1 1

( , ) (1 ( , )) ( , )

( , ) ( , ) .

m t

i i i ik it s r srk s

m ts

ik kk s

V s t H s t a b s a

b s V t

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ ν

− −= = +

− −

= = +

= − +

+

∑ ∑

∑ ∑ (5.22)

At last the evolution equations in the continuous case are the following:

( )01

01

1 1( ) 1 ( ) ( )

( ) ( ) ,

mt t

i i i ik ik

m t

ik kk

e eV t H t Q d

Q e V t d

δ δϑ

δϑ

ψ ϑ ψ ϑδ δ

ϑ ϑ ϑ

− −

=

−

=

− −= − +

+ −

∑∫

∑∫ (5.23)

( )( ) ( )

1

( )

1

1 1( , ) 1 ( , ) ( , )

( , ) ( , ) .

mt s st

i i i ik isk

m t sik ksk

e eV s t H s t Q s d

Q s e V t d

δ δ ϑ

δ ϑ

ψ ϑ ψ ϑδ δ

ϑ ϑ ϑ

− − − −

=

− −

=

− −= − +

+

∑∫

∑∫ (5.24)

158 Chapter 4

5.3.2 Variable Interest Rate, Permanence And Transition Cases Now we make the following assumptions: a) rewards are variable in time, b) rewards are given for permanence in the state and at a given transition, c) the interest rate is variable. Under these hypotheses, in the immediate case we get the following relations:

( ) ( )

1 1 1 1

1 1 1 1

( ) (1 ( )) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ,

t m t

i i i ik ih k h

m t m t

ik ik ik kk k

V t H t h h b h h

b b V t

ϑ

ϑ

ϑ ϑ

ψ ν ϑ ψ ν

ϑ γ ϑ ν ϑ ϑ ϑ ν ϑ

= = = =

= = = =

= − +

+ + −

∑ ∑∑ ∑

∑∑ ∑∑ (5.25)

( ) ( )

1 1 1 1

1 1 1 1

( , ) (1 ( , )) ( ) ( , ) ( , ) ( ) ( , )

( , ) ( ) , ( , ) ( , ) , .

t m t

i i i ik ih s k s h s

m t m t

ik ik ik kk s k s

V s t H s t h s h b s h s h

b s s b s V t s

ϑ

ϑ

ϑ ϑ

ψ ν ϑ ψ ν

ϑ γ ϑ ν ϑ ϑ ϑ ν ϑ

= + = = + = +

= = + = = +

= − +

+ +

∑ ∑ ∑ ∑

∑ ∑ ∑ ∑ (5.26)

In the due case we get: 1 1

0 1 1 0

1 1 1 1

( ) (1 ( )) ( ) ( ) ( ) ( ) ( )

( 1) ( ) ( ) ( ) ( ) ( ),

t m t

i i i ik ik

m t m t

ik k ik ikk k

V t H t b

b V t b

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ν τ ϑ ψ τ ν τ

ν ϑ ϑ ϑ ν ϑ ϑ γ ϑ

− −

= = = =

= = = =

= − +

+ − − +

∑ ∑∑ ∑

∑∑ ∑∑ (5.27)

1 1

1 1

1 1 1 1

( , ) (1 ( , )) ( ) ( , ) ( , ) ( ) ( , )

( , 1) ( , ) ( , ) ( , ) ( , ) ( ).

t m t

i i i ik is k s s

m t m t

ik k ik ikk s k s

V s t H s t s b s s

s b s V t s b s

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ν τ ϑ ψ τ ν τ

ν ϑ ϑ ϑ ν ϑ ϑ γ ϑ

− −

= = = + =

= = + = = +

= − +

+ − +

∑ ∑ ∑ ∑

∑ ∑ ∑ ∑ (5.28)

The evolution equations in the continuous case are the following:

( )

( )

0

0 0

( )

01

( ) ( )

0 0 01

( ) ( ) ( ) ( )

1 ( ) ( ) ( ) ( ) ,

m t di ik k ik

k

mt td di i ik i

k

V t Q e V t d

H t e d Q e d d

ϑ

θ θ

δ τ τ

ϑδ τ τ δ τ τ

ϑ ϑ γ ϑ ϑ

ψ θ θ ϑ ψ θ θ ϑ

−

=

− −

=

∫= − +

∫ ∫+ − +

∑∫

∑∫ ∫ ∫ (5.29)

( )

( )

( )

1

( ) ( )

0 1

( , ) ( , ) ( , ) ( )

1 ( , ) ( ) ( , ) ( ) .

s

s s

m t di ik k iksk

mt td di i ik is sk

V s t Q s e V t d

H s t e d Q s e d d

ϑ

θ θ

δ τ τ

ϑδ τ τ δ τ τ

ϑ ϑ γ ϑ ϑ


−

=

− −

=

∫= +

∫ ∫+ − +

∑∫

∑∫ ∫ ∫ (5.30)


5.3.3 Non-Homogeneous Interest Rate, Permanence And Transition Case For our last case, we consider non-homogeneous rewards and interest rate. And so basic assumptions are: a) rewards are non-homogeneous, b) rewards are given for premanence and transitions, c) interest rate is non-homogeneous. It can easily be verified that the evolution equations take the form:

( )

( )

1 1 1

1 1 1 1 1

( , ) (1 ( , )) ( , ) ( , ) ( , ) ( , ) ,

( , ) ( , ) , ( , ) ( , ) ( , ),

t m t

i i i ik ks k s

m t m t

ik ik ik ik s k s s

V s t H s t s s b s V t s

b s s s b s s s

τ ϑ

ϑ

ϑ ϑ τ

ψ τ ν τ ϑ ϑ ν ϑ

ϑ γ ϑ ν ϑ ϑ ψ τ ν τ

= + = = +

= = + = = + = +

= − +

+ +

∑ ∑ ∑

∑ ∑ ∑ ∑ ∑ (5.31)

1

1 1

1

1 1 1 1 0

( , ) (1 ( , )) ( , ) ( , ) ( , ) ( , ) ( , )

( , 1) ( , ) ( , ) ( , ) ( , ) ( , ),

t m t

i i i ik iks k s

m t m t

ik k ik ik s k s

V s t H s t s s s b s s

s b s V t b s s s

τ ϑ

ϑ

ϑ ϑ τ

ψ τ ν τ ν ϑ ϑ γ ϑ

ν ϑ ϑ ϑ ϑ ψ τ ν τ

−

= = = +

−

= = + = = + =

= − +

+ − +

∑ ∑ ∑

∑ ∑ ∑ ∑ ∑ (5.32)

( ) ( )

( , )

1

( , ) ( , )

1

( , ) ( , ) ( , ) ( , ) 1 ( , )

( , ) ( , ) ( , ) .

s

s s

m t s di ik k ik isk

mt ts d s di ik is s sk

V s t Q s e V t s d H s t

s e d Q s s e d d

ϑ

θ θ

δ τ τ

ϑδ τ τ δ τ τ

ϑ ϑ γ ϑ ϑ


−

=

− −

=

∫= + + −

∫ ∫+

∑∫

∑∫ ∫ ∫ (5.33)

6 GENERAL ALGORITHMS FOR DTSMRWP In the previous section, we presented useful discrete time semi-Markov reward processes as well as general global models for which the evolution equations can be written in the matrix form * =U V C . (6.1) In the homogeneous case, U is an infinite order lower-triangular matrix whose elements are m m× matrices and V and C are infinite order vectors whose elements are m-dimensional vectors. In the non-homogeneous case in (6.1) U is an infinite order upper-triangular matrix whose elements are m m× matrices and V and C are infinite order matrices whose elements are m-dimensional vectors. Of course, matrices U and C depend on the particular models presented in the preceding section. For real life applications, it is generally sufficient to study the problem on a finite time horizon [0, ]T and then the infinite system (6.1) becomes a finite system *T T T=U V C (6.2)

160 Chapter 4

where T U is a square lower triangular block matrix of order 1T + in the homogeneous case and an upper triangular block matrix in the non-homogeneous case. ,T TC V are respectively 1T + -dimensional vectors, in the homogeneous case, and matrices, in the non-homogeneous case, whose elements are m-dimensional vectors. We will present briefly two general algorithms (homogenous and non- homogeneous) solving all possible reward cases. The main steps of these algorithms are the following: (i) Homogeneous case Input – selectors that choose among the SMRWP, the number of states and the number of periods, the permanence and transition rewards, the fixed or variable interest rate, the transition matrix P and the matrix T F of waiting time d.f. Construct - T Q Construct - T B Construct - T H Construct - T D Construct – the permanence rewards Construct – the transition rewards Construct – the vector discount factors Construct – T C , known terms Solve - the system and find T V (ii) Non-homogeneous case Input – selectors, the number of states and the number of periods, the permanence and transition rewards, the fixed, variable or non-homogeneous interest rate, the transition matrix P and the matrix T F waiting time d.f. Construct - T Q Construct - T B Construct - T H Construct - T D Construct – the permanence rewards Construct – the transition rewards Construct – the matrix discount factors Construct – T C , known terms Solve - the system and find T V


These algorithms are able to solve any DTSMRWP. They constitute a very important tool for the application of semi-Markov reward processes in many applied sciences and, in this book, mainly in Finance, Insurance and Reliability. 7 NUMERICAL TREATMENT OF SMRWP 7.1 Undiscounted Case Let us consider relations (5.7) and (5.13):

( )0 01 1

( ) 1 ( ) ( ) ( ) ( ) ,m mt t

i i i i ik ik kk k

V t H t t Q d Q V t dψ ψ ϑ ϑ ϑ ϑ ϑ ϑ= =

= − + + −∑ ∑∫ ∫ (7.1)

( )

( )

0 0 01

01

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ( ) .

mt t

i i i ik ik

m t

ik k ikk

V t H t d Q d d

Q V t d


ϑ ϑ γ ϑ ϑ

=

=

= − +

+ − +

∑∫ ∫ ∫

∑∫ (7.2)

For non-homogeneous models, the simplest and the most difficult cases are given by formulas (5.8) and (5.17), i.e:

( )

1

1

( , ) 1 ( , ) ( ) ( , ) ( )

( , ) ( , ) ,

m t

i i i i iksk

m t

ik ksk

V s t H s t t s Q s s d

Q s V t d

ψ ψ ϑ ϑ ϑ

ϑ ϑ ϑ

=

=

= − ⋅ − + ⋅ −

+

∑∫

∑∫ (7.3)

( )

( )

1

1

( , ) 1 ( , ) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) .

mt t

i i i ik is s sk

m t

ik k iksk


Q s V t s d


ϑ ϑ γ ϑ ϑ

=

=

= − +

+ +

∑∫ ∫ ∫

∑∫ (7.4)

Relations (7.2) and (7.4) can be written also as follows:

( )

0 0 01

0 01 1

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ,

mt t

i i i ik ik

m mt t

ik k ik ikk k

V t H t d Q d d

Q V t d Q d


ϑ ϑ ϑ ϑ γ ϑ ϑ

=

= =

= − +

+ − +

∑∫ ∫ ∫

∑ ∑∫ ∫ (7.5)

( )

1

1 1

( , ) 1 ( , ) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) ( , ) .

mt t

i i i ik is s sk

m mt t

ik k ik iks sk k


Q s V t d Q s s d



=

= =

= − +

+ +

∑∫ ∫ ∫

∑ ∑∫ ∫ (7.6)

and so both the homogeneous and non-homogeneous integral equations can be written as follows:

162 Chapter 4

01

( ) ( ) ( ) ( ) ,m t

i i ik kk

V t c t Q V t dϑ ϑ ϑ=

= + −∑∫ (7.7)

1

( , ) ( , ) ( , ) ( , ) ,m t

i i ik kskV s t c s t Q s V t dϑ ϑ ϑ

=

= +∑∫ (7.8)

where for relations (7.1) and (7.3), we have:

( )01

( ) 1 ( ) ( ) ,m t

i i i i ikk

c t H t t Q dψ ψ ϑ ϑ ϑ=

= − + ∑∫ (7.9)

( )1

( , ) 1 ( , ) ( ) ( , ) ( ) ,m t

i i i i ikskc s t H s t t s Q s s dψ ψ ϑ ϑ ϑ

=

= − ⋅ − + ⋅ −∑∫ (7.10)

and for relations (7.2) (7.4):

( )

0 0 01

01

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ,

mt t

i i i ik ik

m t

ik ikk

c t H t d Q d d

Q d


ϑ γ ϑ ϑ

=

=

= − +

+

∑∫ ∫ ∫

∑∫ (7.11)

( )

1

( , ) 1 ( , ) ( , ) ( , ) ( , )

( , ) ( , ) .

tt

i i i ik ikss

m t

ik is sk

c s t H s t s d Q s s d

Q s s d dϑ

ψ τ τ ϑ γ ϑ ϑ

ϑ ψ τ τ ϑ=

= − +

+

∫ ∫

∑∫ ∫ (7.12)

As it can be seen, these last four equations have known terms that differ substantially but the coefficients of the two homogeneous integral equations are the same as in the non-homogeneous case. Furthermore the integral equation (7.7) has the same coefficient as the equation (2.1) and the equation (7.8) the same as the equation (2.2). As for the discretization of the homogeneous and non-homogeneous semi-Markov processes presented in section 2, we can consider the generic quadrature formula (2.3). Let us recall that h is the step measure, , , , , ,u k N u k N≤ ∈ ,w the weights related to the quadrature formula. We also know that N is such that Nh Y= and [ ]0,Y is the integration interval. Now, relations (7.7) and (7.8) can be discretized in the following way:

1 0

( ) ( ) ( ) ( ) ,m k

i i k l ill

V kh c kh w V kh h Q hττ

τ τ= =

⎛ ⎞= + −⎜ ⎟

⎝ ⎠∑ ∑ (7.13)

, ,1

( , ) ( , ) ( , ) ( , ) ,m k

i ij u k il il u

V uh kh c uh kh w Q uh h V h khττ

τ τ= =

⎛ ⎞= + ⎜ ⎟

⎝ ⎠∑ ∑ (7.14)

but here, due to integral terms, the known term c should also be discretized. Relations (7.13) and (7.14) in matrix form become:


0

( ) ( ) ( ) ( ),k

kkh kh w h kh hττ

τ τ=

= + ∗ −∑V c Q V (7.15)

( , ) ( , ) ( , ) ( , ),k

uku

uh kh uh kh w uh h h khττ

τ τ=

= + ∗∑V c Q V (7.16)

or equivalently:

01

( ) (0) ( ) ( ) ( ) ( ),k

k kkh w kh kh w h kh hττ

τ τ=

− ∗ = + ∗ −∑V Q V f Q V (7.17)

1

( , ) ( , ) ( , )

( , ) ( , ) ( , ).

uu

k

uku

uh kh w uh uh uh kh

uh kh w uh h h kh

τ

ττ

τ τ= +

− ∗

= + ∗∑

V Q V

c Q V (7.18)

These last two relations take the form:

( )01

(0) ( ) ( ) ( ) ( ),k

k kw kh kh w h kh hττ

τ τ=

− ∗ = + ∗ −∑I Q V f Q V (7.19)

( )

1

( , ) ( , )

( , ) ( , ) ( , ).

uu

k

uku

w uh uh uh kh

uh kh w uh h h kh

τ

ττ

τ τ= +

− ∗

= + ∗∑

I Q V

c Q V (7.20)

Coefficient matrices of (7.19) and (7.20) are the same as those defined in systems (2.6) and (2.7) and so Theorem 2.1 holds also in these cases. The only difference is that the elements of the known terms are more difficult to construct. 7.2 Discounted Case In this financial environment, we will consider the most complicated homogeneous and non-homogeneous discounted cases. More precisely the CTHSMRWP related to relation (5.29) and the CTNHSMRWP formula (5.33) will be tackled. As before we report the two relations:

( )

( )

0

0 0

( )

01

( ) ( )

0 0 01

( ) ( ) ( ) ( )

1 ( ) ( ) ( ) ( ) ,

m t di ik k ik

k

mt td di i ik i

k

V t Q e V t d

H t e d Q e d d

ϑ

θ θ

δ τ τ

ϑδ τ τ δ τ τ

ϑ ϑ γ ϑ ϑ


−

=

− −

=

∫= − +

∫ ∫+ − +

∑∫

∑∫ ∫ ∫ (7.21)

( ) ( )

( , )

1

( , ) ( , )

1

( , ) ( , ) ( , ) ( , ) 1 ( , )

( , ) ( , ) ( , ) .

s

s s

m t s di ik k ik isk

mt ts d s di ik is s sk

V s t Q s e V t s d H s t

s e d Q s s e d d

ϑ

θ θ

δ τ τ

ϑδ τ τ δ τ τ

ϑ ϑ γ ϑ ϑ


−

=

− −

=

∫= + + −

∫ ∫+

∑∫

∑∫ ∫ ∫ (7.22)

Relations (7.21) and (7.22) can be written as follows:

164 Chapter 4

( )

0 0

0 0

( ) ( )

0 01 1

( ) ( )

0 0 01

( ) ( ) ( ) ( ) ( )

1 ( ) ( ) ( ) ( ) ,

m mt td di ik k ik ik

k kmt td d

i i ik ik

V t Q e V t d Q e d

H t e d Q e d d

ϑ ϑ

θ θ

δ τ τ δ τ τ

ϑδ τ τ δ τ τ



− −

= =

− −

=

∫ ∫= −

∫ ∫+ − +

∑ ∑∫ ∫

∑∫ ∫ ∫(7.23)

( )

( , ) ( , )

1 1

( , ) ( , )

1

( , ) ( , ) ( , ) ( , ) ( , )

1 ( , ) ( , ) ( , ) ( , ) .

s s

s s

m mt ts d s di ik k ik iks sk k

mt ts d s di i ik is s sk

V s t Q s e V t d Q s e s d

H s t s e d Q s s e d d

ϑ ϑ

θ θ

δ τ τ δ τ τ

ϑδ τ τ δ τ τ



− −

= =

− −

=

∫ ∫= +

∫ ∫+ − +

∑ ∑∫ ∫

∑∫ ∫ ∫ (7.24)

These equations take also the following form:

0( )

01( ) ( ) ( ) ( ),

m t di ik k i

kV t Q e V t d c t

ϑδ τ τ

ϑ ϑ ϑ−

=

∫= − +∑∫ (7.25)

( , )

1( , ) ( , ) ( , ) ( , ),s

m t s di ik k isk

V s t Q s e V t d c s tϑδ τ τ

ϑ ϑ ϑ−

=

∫= +∑∫ (7.26)

equations having the same structure. This is also true for all the other cases with a fixed or variable intensity of interest rate. We can thus proceed to the discretization procedure as before, to get the following relations:

( ) 1

1 0 1( ) ( ) ( ) ( ) 1 ( ) ,

m k

i i k l il hl

V kh c kh w V kh h Q h rτ

ττ θ

τ τ θ −

= = =

⎛ ⎞= + − +⎜ ⎟

⎝ ⎠∑ ∑ ∏ (7.27)

( ) 1

1

( , ) ( , )

( , ) ( , ) 1 ( , ) .

i i

m k

uk l il hl u u

V uh kh c uh kh

w V h kh Q uh h r uτ

ττ θ

τ τ θ −

= = =

=

⎛ ⎞+ +⎜ ⎟⎝ ⎠

∑ ∑ ∏ (7.28)

The functions ( )hr θ and ( , )hr u θ are the variable rates of interest obtained in this way:

( 1)

( 1)

( )

( , )

, 1(1 ( )) ,1, 0,

, 1(1 ( , )) ,1, .

h

h

h

h

d

h

u d

h

er

e ur uu

θ

θ

θ

θ

δ τ τ

δ τ τ

θθθ

θθθ

−

−

⎧ ∫⎪ ≥+ = ⎨⎪ =⎩⎧ ∫⎪ ≥ ++ = ⎨⎪ =⎩

(7.29)

In matrix form, the equations can be written in the following way:

( ) 1

0 0( ) ( ) 1 ( ) ( ) ( ),

k

k hkh kh w r h kh hτ

ττ θ

θ τ τ−

= =

= + + ∗ −∑ ∏V c Q V (7.30)


( ) 1( , ) ( , ) 1 ( , ) ( , ) ( , )k

uk hu u

uh kh uh kh w r u uh h h khτ

ττ θ

θ τ τ−

= =

= + + ∗∑ ∏V c Q V (7.31)

or equivalently:

( )

0

1

1 0

( ) (0) ( ) ( )

1 ( ) ( ) ( ),

k

k

k h

kh w kh kh

w r h kh hτ

ττ θ

θ τ τ−

= =

− ∗ =

+ + ∗ −∑ ∏

V Q V c

Q V (7.32)

( ) 1

1

( , ) ( , ) ( , )

( , ) 1 ( , ) ( , ) ( , ).

uu

k

uk hu u

uh kh w uh uh uh kh

uh kh w r u uh h h kh

τ

τ

ττ θ

θ τ τ−

= + =

− ∗

= + + ∗∑ ∏

V Q V

c Q V (7.33)

We can also write:

( )

( )

0

1

1 0

(0) ( )

( ) 1 ( ) ( ) ( ),

k

k

k h

w kh

kh w r h kh hτ

ττ θ

θ τ τ−

= =

− ∗

= + + ∗ −∑ ∏

I Q V

c Q V (7.34)

( )

( ) 1

1

( , ) ( , )

( , ) 1 ( , ) ( , ) ( , ).

uu

k

uk hu u

w uh uh uh kh

uh kh w r u uh h h kh

τ

τ

ττ θ

θ τ τ−

= + =

− ∗

= + + ∗∑ ∏

I Q V

c Q V (7.35)

So, we have shown that in the most difficult discounted cases, we get equations having the same coefficient matrices as the equations (7.19) and (7.20); consequently, here too, Theorem 2.1 holds. 8. RELATION BETWEEN DTSMRWP AND SMRWP NUMERICAL SOLUTIONS This section is related to the relation between the numerical solution of continuous and discrete time semi-Markov reward processes to show that, as in the SMP case, this approximation formula leads to the related discrete time process. For simplicity, we just prove this result in the simplest numerical approach, i.e. the rectangle formula. With this method, it is possible to evaluate the integral using the values of the function at the minimum of each of the discretization intervals or at the maximum Clearly, in the continuous case to distinguish between the due and the immediate cases is meaningless and here we assume that reward payment times correspond to the “times” of the evaluations so that we could obtain respectively, by means of discretization, the due and the immediate cases. Here we only consider the immediate case.

166 Chapter 4

8.1 Undiscounted Case Taking into account relations (7.7) and (7.8), the general numerical solutions of a non-discounted CTHSMRWP and CTNHSMRWP can be written respectively in the following way:

0 1

( ) ( ) ( ) ( )k m

i i il ll

V kh c kh h Q h V kh hτ

τ τ= =

= + −∑∑ , (8.1)

1

( , ) ( , ) ( , ) ( , ).k m

i i il lu l

V uh kh c uh kh h Q uh h V h khτ

τ τ= =

= + ∑∑ (8.2)

In the special case of relations (5.7) and (5.8), the equations (8.1) and (8.2) become:

( )

1 1

0 1

( ) 1 ( ) ( )

( ) ( ),

k m

i i i il il

k m

il ll

V kh H kh k h Q h

h Q h V kh h

τ

τ

ψ τ τψ

τ τ

= =

= =

= − +

+ −

∑∑

∑∑ (8.3)

( )

1 1

1

( , ) 1 ( , ) ( ) ( , )( )

( , ) ( , ),

k m

i i i il iu l

k m

il lu l

V uh kh H uh kh k u h Q uh h u

h Q uh h V h kh

τ

τ

ψ τ τ ψ

τ τ

= + =

= =

= − − + −

+

∑ ∑

∑∑ (8.4)

where iψ represents the constant permanent reward paid at the end of each period. Substituting the differential by means of the increment, we get:

( ) ( )

( )

1 1

0 1

( ) 1 ( ) ( ) (( 1) )

( ) (( 1) ) ( ),

k m

i i i il il il

k m

il il ll

V kh H kh k Q h Q h

Q h Q h V kh h

τ

τ

ψ τ τ τψ

τ τ τ

= =

= =

≅ − + − −

+ − − −

∑∑

∑∑ (8.5)

( )

( )

( )

1 1

1

( , ) 1 ( , ) ( )

( , ) ( ,( 1) ) ( )

( , ) ( , ( 1) ) ( , ).

i i i

k m

il il iu l

k m

il il lu l

V uh kh H uh kh k u

Q uh h Q uh h u

Q uh h Q uh h V h kh

τ

τ

ψ

τ τ τ ψ

τ τ τ

= + =

= =

≅ − −

+ − − −

+ − −

∑ ∑

∑∑

(8.6)

Using relations (1.11) and (1.12) and setting 1h = , these results can be written in the form:

( )1 1 0 1

( ) 1 ( ) ( ) ( ) ( ),k m k m

i i i il i il ll l

V k H k k b b V kτ τ

ψ τ τψ τ τ= = = =

≅ − + + −∑∑ ∑∑ (8.7)


( )

1 1

1

( , ) 1 ( , ) ( ) ( , )( )

( , ) ( , ).

k m

i i i il iu l

k m

il lu l

V u k H u k k u b u u

b u V k

τ

τ

ψ τ τ ψ

τ τ

= + =

= =

≅ − − + −

+

∑ ∑

∑∑ (8.8)

As we know that (0) 0ijb = and ( , ) 0ijb u u = , i,j=1,…,m, relations(8.7) and (8.8) correspond in fact to the evolution equations (5.5) and (5.6). As for the second case, we will now give the numerical solution of the evolution equations of (5.13) and (5.17), respectively for homogeneous and non-homogeneous cases. Let us recall these two equations:

( )

1 1 1

1 1 1 1 1

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ),

k k m

i i i il ill

k m k m

il i il ll l

V kh H kh h h h Q h h

h Q h h h h Q h V kh h

τ ϑ

ϑ

ϑ τ ϑ

ψ τ ϑ γ ϑ

ϑ ψ τ ϑ ϑ

= = =

= = = = =

= − +

⎛ ⎞+ + −⎜ ⎟⎝ ⎠

∑ ∑∑

∑∑ ∑ ∑∑ (8.9)

( )1

1 1 1

1 1

( , ) 1 ( , ) ( , )

( , ) ( , )

( , ) ( , ) ( , ) ( , ).

k

i i ilu

k m

il iu l u

k m k m

il l il ilu l u l

V uh kh H uh kh h uh h

h Q uh h h uh h

h Q uh h V h kh h Q uh h uh h

τ

ϑ

ϑ τ

ϑ ϑ

ψ τ

ϑ ψ τ

ϑ ϑ ϑ γ ϑ

= +

= + = = +

= = = =

= −

⎛ ⎞+ ⎜ ⎟⎝ ⎠

+ +

∑

∑ ∑ ∑

∑∑ ∑∑

(8.10)

Proceeding as for the first case, we obtain:

( )

1 1 1 1

1 1 1 1

( ) 1 ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ),

k k m

i i i il il

k m k m

il il il ll l

V k H k b

b b V k

ϑ

τ ϑ τ

ϑ ϑ

ψ τ ϑ ψ τ

ϑ γ ϑ ϑ ϑ

= = = =

= = = =

⎛ ⎞= − + ⎜ ⎟⎝ ⎠

+ + −

∑ ∑∑ ∑

∑∑ ∑∑ (8.11)

( )

1 1

1 1 1 1

( , ) 1 ( , ) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) ( , ),

k k m

i i i il ilu u l

k m k m

il i il lu l u u l

V u k H u k u b u u

b u u b u V k

τ ϑ

ϑ

ϑ τ ϑ

ψ τ ϑ γ ϑ

ϑ ψ τ ϑ ϑ

= + = =

= + = = + = =

= − +

⎛ ⎞+ +⎜ ⎟⎝ ⎠

∑ ∑∑

∑ ∑ ∑ ∑∑ (8.12)

so that relations (8.11) and (8.12) correspond in fact to relations (5.10) and (5.14).

168 Chapter 4

8.2 Discounted Case The change from continuous and discrete time in discounted cases implies that the financial discounting factors should change; in the case of constant intensity interest rate δ , it results that ( ) 11 h

hr e δ− −+ = , (8.13) and in the case of variable intensities respectively for homogeneous and non- homogeneous cases:

( )

( )

( 1)

( 1)

( )1

( , )1

1 ( ) , 1, , ,

1 ( , ) , 1, , .

h

h

h

h

d

h

u d

h

r e k

r u e u k

τ

τ

τ

τ

δ θ θ

δ θ θ

τ τ

τ τ

−

−

−−

−−

∫+ = =

∫+ = = +

…

…

(8.14)

(for more details on this topic see Volpe di Prignano (1985), Kellison(1991). Now we will present two cases in the discounted environment: the discretization of relations (5.23), (5.24) and then of relations (5.29) and (5.33). For the first case, the discretization method gives as relations:

( )

( )

1 1

1 1

( ) 1 ( ) ( )

( ) ( ) 1 ,

h h

k m

i i i il ik r rl

k m

il l hl

V kh H kh a h Q h a

h Q h V kh h r

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ

= =

−

= =

= − +

+ − +

∑∑

∑∑ (8.15)

( )

( )

1 1

( )

1 1

( , ) 1 ( , ) ( , )

( , ) ( , ) 1 .

h h

k m

i i i il ik u r url

k mu

il l hl

V uh kh H uh kh a h Q uh h a

h Q uh h V h kh r

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ

− −= =

− −

= =

= − +

+ +

∑∑

∑∑ (8.16)

Proceeding in the same way we use to get relations (8.7) and (8.8), the following results are obtained:

( )

( )

1 1

1 1

( ) 1 ( ) ( )

( ) ( ) 1 ,

k m

i i i il ik r rl

k m

il ll

V k H k a b a

b V k r

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ

= =

−

= =

= − +

+ − +

∑∑

∑∑ (8.17)

( )

( )

1 1

( )

1 1

( , ) 1 ( , ) ( , )

( , ) ( , ) 1 ,

k m

i i i il ik u r u rl

k mu

il ll

V u k H u k a b u a

b u V k r

ϑϑ

ϑ

ϑ

ψ ϑ ψ

ϑ ϑ

− −= =

− −

= =

= − +

+ +

∑∑

∑∑ (8.18)

corresponding to relations (5.19) and (5.23). For the second case, let us begin with the discretization of the equation (5.29) leading to:


( ) ( )

( )

( )

( )

1

1 1

1

1 1 1 1

1

1 1 1

1

1 1 1

( ) 1 ( ) ( ) 1 ( )

( ) ( ) 1 ( )

( ) ( ) 1 ( )

( ) ( ) 1 ( ) .

k

i i i h

k m

il i hl

k m

il il hl

k m

il l hl

V kh H kh h h r

h Q h h h r

h Q h h r

h Q h V kh h r

ϑ

ϑ τ

ϑ θ

ϑ θ τ

ϑ

ϑ τ

ϑ

ϑ τ

ψ ϑ τ

ϑ ψ θ τ

ϑ γ ϑ τ

ϑ ϑ τ

−

= =

−

= = = =

−

= = =

−

= = =

= − +

+ +

+ +

+ − +

∑ ∏

∑∑ ∑ ∏

∑∑ ∏

∑∑ ∏

(8.19)

Once more, proceeding in the same way we use to find relations (8.7) and (8.8), we get:

( )

( ) ( )

( ) ( )

1

1 1 1 1

1 1

1 1 1 1 1 1

1

1 1

( ) ( ) ( ) 1 ( )

( ) ( ) 1 ( ) ( ) ( ) 1 ( )

1 ( ) ( ) 1 ( ) ,

k m

i il il

k m k m

il il il ll l

k

i i

V k b r

b r b V k r

H k r

ϑ θ

ϑ θ τ

ϑ ϑ

ϑ τ ϑ τ

ϑ

ϑ τ

ϑ ψ θ τ

ϑ γ ϑ τ ϑ ϑ τ

ψ ϑ τ

−

= = = =

− −

= = = = = =

−

= =

= +

+ + + − +

+ − +

∑∑ ∑ ∏

∑∑ ∏ ∑∑ ∏

∑ ∏

(8.20)

this result corresponding to relation (5.25). Finally, we have to dicretize the equation (5.33):

( )

( ) ( )

( )

1

1 1 1 1

1

1 1

1

1 1 1

1

( , ) ( , ) ( , ) 1 ( , )

1 ( , ) ( , ) 1 ( , )

( , ) ( , ) 1 ( , )

( , ) ( , )

k m

i il i hu l u

k

i i hu u

k m

il il hu l u

m

il lu l

V uh kh h Q uh h h uh h r u

H uh kh h uh h r u

h Q uh h uh h r u

h Q uh h V h kh

ϑ θ

ϑ θ τ

ϑ

ϑ τ

ϑ

ϑ τ

ϑ

ϑ ψ θ τ

ψ ϑ τ

ϑ γ ϑ τ

ϑ ϑ

−

= + = = + =

−

= + = +

−

= + = = +

= + =

= +

+ − +

+ +

+

∑ ∑ ∑ ∏

∑ ∏

∑ ∑ ∏

∑ ( ) 1

1 11 ( , ) .

k

hu

r uϑ

ττ −

= +

+∑ ∏

(8.21)

And as above, we get the result corresponding to relation (5.31)

( )

( ) ( )

( )

( )

1

1 1 1 1

1

1 1

1

1 1 1

1

1 1 1

( , ) ( , ) ( , ) 1 ( , )

1 ( , ) ( , ) 1 ( , )

( , ) ( , ) 1 ( , )

( , ) ( , ) 1 ( , ) .

k m

i il iu l u

k

i ilu u

k m

il ilu l uk m

il lu l u

V u k b u u r u

H u k u r u

b u u r u

b u V k r u

ϑ θ

ϑ θ τ

ϑ

ϑ τ

ϑ

ϑ τ

ϑ

ϑ τ

ϑ ψ θ τ

ψ ϑ τ

ϑ γ ϑ τ

ϑ ϑ τ

−

= + = = + =

−

= + = +

−

= + = = +

−

= + = = +

= +

+ − +

+ +

+ +

∑ ∑ ∑ ∏

∑ ∏

∑ ∑ ∏

∑ ∑ ∏

(8.22)

chapter 4 - course.shufe.edu.cncourse.shufe.edu.cn/sxwh/semimarkov/semi-markov risk models for...

Documents