A Note on Stopped Feynman-Kac Funetionalsof Markov Chains

Josep tvl. FerrandizNetwork and System Management DepartmentIIPLIJ-NSMG-92-.jVersion 0.3June 17. 1992

Feynman-h:ac functionals,i\larkov chains, mart.ingales.stopping times

Consider two continuous additive rUllctionalsA(.) and H(.) or a continuous time }l'Iarkovchain on a discrete stat.e space such that H(.) isabsolut.ely continuous wit.h respect. to A(.). Let0/ be the right continuous inverse of A(.). Wederive the joillt. law or I::J(.) and t.he jumpsbetween 5tates of t.lle chain 5Lopped at time 0 1

and particularize the result. t.o hitting t.imes.

© Copyright. Hewlet.t.- Packard Company 1992

Internal Accession Date Only

1. Markov Additive Functionals

Let X/ be a l\'I<I. ..ko\, chain on a finite state space t:; with intensity matrix Q(qij)i,jE/~ such t.hat,

00 > q; = -qji :2: L qij-

j':I:i

The absol'p1.iol1 t.ime of X is denoted by 1'. i.e.: for t :2: 'J', X t f/: 1:) and for l <1', XI E 1::. Note t.hat if (ji = Ljf-i (/ij, (recurrent case) we have a.s. 'I' = 0c , andotherwise (transient case), we ha\"c a.S. T < 00.

An ;jX -adapted process 1~ such thai,

• the paths of V are llon-<Iccrcasing <\.1)(1 right continuous.

• \"0 = 0,

• V, is ;i;" -measurable.

• Vr+& = ~ + \~ 00", where Os be the shirl. operator.

is called a l\larkov additive funct.ional [2]. We consider bol.h continuous and discon­I.inuolls Markov additive functionals. The cont.inuous tl'larkov addil.i\"c fUllctionaJsconsidered arc of the forlll,

( l.l)

where 'U(') is a bounded non-negative mapping defined Oil 6', In particular, ifv('i) = l('i = j), \~ becomes,

f~j(l) = 1,' I(X, = j)ds, 120, j E e, (1.2)

whicli is the amount of t.ime spent in sta.te j by time l. The discontinuous Markova.dditive functionals considered are of the form,

Ut = L Il(Xs-, X s ), l ~ 0,O<s9

(IJ)

where tt( •.. ) is a bounded non-negat.ive mapping defined on J:; x J:; such (,haLnU, i) = 0, i E £. In particular, if u(i,j) = l(i = J.:)1(j = I), Ut becomes,

Jk/(I) ~ L I(X,- = k)I(X, = f), (IA)0<.59

which is equal to the number of jumps of X from k into I in }O, tl. k =I- 1. Since anynon-negati\'e continuous J\lal'kO\" additive functional of the form given by equa­tion (1.1) can be written in terms of the family of additive functionals (LAI)).

,EE

1

where J(.) is the point process of jumps of X. Taking expectations and using thefact that for jEt:, 1(1 < 1')I(X, ~ j) = I(X, = j) a.s. yields,

1';j(l) = )lij (O)+E[l(Xo =i) r e-LkECukLds) II =rtdr )(J1o.l ) k.lH

kt- I

L I(X.- ~ ",)I(X. = j):mj -l(X.- = j))J(ds)]rnamt-j

-Vj11

}~As)ds

)~j(O) +E [l(Xo = i) 101

e- LkEE ul·L.,{s) II =fil (.-)(

'dEl:kt-I

L l(X. = m)qmj:mj - I(X. = j)'/j)ds] - Vj 10' l';;(s)dsmEl:mt-i

\';;(0) + r' L Yim(s)qmj:mjds - Vj r'l,;,(s)ds.~ nlEE Jo

In vector form.

Y;(t) = Y;(O)+ 10' Y i(S)(Q 0 Z - V)ds,

where Z = (=ij)i,jEE. wil,h =ii =] and V = diag{ui' i E I.::}. Hence.

Yi(l) ~ Y;(O)exl'{ (Q 0 Z- V)I}.In particular,

lej( I) ~ l'ii(O) exl'{ (Q 0 Z- V) t} lij'

where Y;;(U) = P(Xo = i). Therefore,

(2.1 )

E[e- L", "Ld'l II :/t,(l)l(1 < ~')1( X, = j)IXo = i] = cxp{ (Q 0 Z - V)t }Iij'~·.IE£

k#

and the result is proved.

•

Thc prc"iolls result has an interesting corollary. For i E I::, if XI =J i for allt. 0 ::; l < '1', define 'i = 1', otherwise, let 'i = inf{O < l < l' : Xl = i}. In these<luel, if F denotes a subset of the state space 1::, the restriction of a ma.trix Cindexed in I.:: to the subset F will be denoted by CF. A similar notation will helIsed for \·cctors. We have.

and any non-negative discontinuous JVla,rkov additive functional of the form givenby equation (1.:3) can be written in terms of the ramily of additive functionals

(Jk/(l)) , we will limit. ourselves to the study of continuous iVlarkov additiveIdeE

fundionals of the form given by equations (1.2) and (lA).

2. Laws of Markov Additive Functionals

The joint law of lVlarkov additive functionals given by equations (1.1) and (1.11) is

derived in this section. The entrywise product of two matrices C = (Cij);,jEE andC 1= (cij);,jEH is denoted by Co e ', 'i.e., Co C 1= (c;jc;j)i,iE!;::'

We begin by present.ing a somewhat simpler proof of a result in [1].

Lemma 2.1 For l ~ 0, lel P(t) = (]Jij(l))i,jEE, 'where,

]>'jlt) ~ E [l(t < 1') exp{ - I>,L,(t)} IT z;t'!'ll (X, ~ j) IXo ~ ;],kEE k,IEE

k#

whe'I'e Vk 2: (J an(/O :S ;;1../ :S 1, /" ¥- 1. II 'we let Zj,-k = " /,: E 8, Z = (Zkl)k,I€E, andV = diag{tl;, i E J.:,.'}, then the IIwl'l'i.T P(l) is given by,

P(t)~exp{(QoZ-V)I},

Remark: Even if, for j E J~, 1(l < '1')1(X1 = j) = l(Xt = j) a.s.) introducing theindicil.t,or or the event. {I < T} <dlows treating the transient and recurrent casestogether. In particular, taking Vk = (J and Zk/ = 1, /':, lEE yields,

which is the conditional proba,bility that st.arting ill 'i E J!.;, the chain is by time l

in j E J~) <lIlel hClB not been absorbed.

U;j(t) = l(Xo = i)l(l < T)1(X1 = j) II zttl(l)e- LkEE "'kLdl),

k.IEEkf-l

whe'T Uk 2: 0 and 0 S z" S 1. Let Y,,(t) ~ E[U'j(t)] and Y,(t) (lej(i)) "lEb

Since the discontinuity jumps or U;j(') can only happen at jumps or x, we ha.vethe sample path eql\{dity,

where (J.i ~ 0 for all i E .£. Let also B be a. continuous additive functional of theform,

H t = 10/ bx.ds,

ahsolutely continuous with respect to i\. The absolute continuil,y condition is equiv­alent to.

ai = 0 ==:::} bi = 0, i E E.

For n ~ 0 definc,

e.. ={ inf{t>O:A,>n} ifn<AT,00 otherwise.

For posith'e reCllrrent i\larkO\" chains, AI' = ex:: a.S. and therefore we ha\-e a.s ..en < oc for n ~ O. On the other hand, for transient chains, AI' < oc a.s. andtherefore. thc distribution of 0 n is defective.

Let X~ = Xe " and Jkt(n) = At(8'1)' 11 2': O. The processes (X,~)ll>o alld (Ji/(n))- ,,>0are dcfined only on {0 n < oc}. We partition the intensity matrix Q in blocks, ac~

cording to f:). = {i E .£ : (Ii > OJ,

(Q"Q'lI

l-'rolll the strong Markov property, the process (X,~)n~O is a Markov chain. lts statespace is I:;~ and its intensity matrix is [2],

Similarly, tltc process (J~:1(1'I)) is only defined on {(k,f): J..~,IE b'R<lndk=j:l}."2: 0

Lemma 3.1 For n ,?: 0, lei P(n) = (Pij(n))i.ieE" 'where,

p;;(nj=E[l(e" <oojexp{-He,,} II zit'(e"ll(Xe" =j)IXo=i],k,IEE'k#

where 0 $: =kt $: I, k =I- I. If we let =kk = 1, k E Hw, Z = (=kllk,teE" Adiag{oi,iE t:"}. alld B=diag{bi.iE.t.'W}, 'hen /he ma'rixP(n) isgive1I by.

Lemma 2.2 Por l 2: 0, let P(t) = (Pij(t))i,jEEl whe're,

1'lj(1) = E [1(1 < '1')l(X, = j) II i(T, > I)exp{ - I: veLdt) }IXo ~ i].k<lf kEf;

The'll Ihe matrix P(l) is give'll by;

F

P( ) ~ F (eXP{(QF - V r)l]I b-F 0

·"<In "" 1 (J It) )z/:I = L..., zkl /:I = 11. ,,,>0

b-F

oo )

and fll: I(Jk/(t) = 0) = 1(;::1 >tL the choice,

Zk/ ={ ~ if I E b - F, k E fl,ir/EF,kEl';,

yield~,

II,ft'[<l ~ II 1( Tt > t).~.(EE I~.F

kl-I

The result follows from lemma 2.1 and taking into accollnt the sample path equalityfor j E b',

l(X, ~j) II l(T, >t)exr{- I:v,Ldt)} = l(X, =j) II l(T' > t)ex!,{- I:v[Ldt)},k~F kEE k~F kEE

where for i E b',if oj E F,ifi rt fi'.

•

3. Laws of Stopped Markov Additive Functionals

Consider a, continuous additive functional A or the form,

which implies Yii(O) = P(Xo = i) and the result is proved.

•

Lemma 3.2 For 'II 2: 0, let P(n) = (]JiiCt))i,iEE" wh.ere)

ami F is a subset. 0)"./;.,'''. Then/he Hwfri:cP(n) is gi'ven by,

P(n) ~

F

r (eXP1AF'(QF - BF)n)1':" - P 0

E"-Foo )

Proof: Let Tt = inr{n > 0: .Y,~ = i}. Since,

.1,,1"1 '" J (J" () .)::::~_/ = ~:/;Il . kI n = ./, ,j~ll

and TIl.- 1 (Jk/{l) = 0) = l(Tt > t), the choice,

{0 if lEE" - P, k E E",

"'"/;/ --' - 1 otherwise,

yields,

II .1:,1"1 II 1(' ()):/;/ = T/>-",Irt F

and the result follows rrom lemma 3.1.

References

•

[1] L. C. G. Hagel's and D. \Villiams. DiiTasions, Markov Frocesses a.nd Ma:rtin­gales, volume 2: lti:> Calculus. John Wiley a.nd Sons, Chichester, Hl87.

[:2] H. Syski. introduction 1.0 Congestion in Telephone Systems. North-l-lollClnd,Amsterdam, 2nd edition, 198.5.

7

Proof' As mention ned earlier, the process X~ = X e" is also a Markov chain on E~

with intensity matrix A-IQ~ [2] and absorption time AT. For k,f E E/, let,

Jk/(n) ~ I: l(X,;,_ ~ k)l(X;', = I)o<"'s"

be the number of jumps of X~ from ~: into I in ]0,17.]. Since AI is non-decreasing,we have on {8" < oo} = {n < AT} the sample path equalities,

Xe"Jkdn), k,1 E E',

and,

1e

" 1" 0"bx,ds = -'-'-"dOLo 0 a.x,~,

Let {,j E e', II'

U;;(n) = l(Xo =i)1(8" < oo)l(Xe" = j)exp{-1e" ox,ds} II Z;;,(0" 1,

o k,IEE'

I.:f-I

we have the sample equality,

U;j(n) = U;j('II.)n 20, ,j,j E E~.

But equation (2.1) and lemma 2.1 give for X~,

E[U;,(n)] = Y;,(O)exp{A-I (Q' 0 Z - B)n}I'f

where Y;·;(O) = P(.Y(i = 'i), Z = (:;ij)i,jEE with Zii = I, B = diag{bi, i E .t:~}, a.ndA = diag{a.;, i E .t:-}. Hence,

and,(U)

Finally, since 8 0 = 0 ,1..5., we ha.ve,

Xo = ){o, a.s.

6