7/24/2019 Dtmc Basic

1/185

Discrete Time Markov Chains, Definition

and classification

Bo Friis Nielsen1

1Applied Mathematics and Computer Science

02407 Stochastic Processes 1, September 1 2015

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

2/185

Discrete time Markov chains

Today:

Short recap of probability theory

Markov chain introduction (Markov property) Chapmann-Kolmogorov equations

First step analysis

Next week

Random walks First step analysis revisited

Branching processes

Generating functions

Two weeks from now Classification of states

Classification of chains

Discrete time Markov chains - invariant probability

distributionBo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

3/185

Basic concepts in probability

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

4/185

Basic concepts in probability

Sample space

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

5/185

Basic concepts in probability

Sample space

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

6/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

7/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

B i i b bili

7/24/2019 Dtmc Basic

8/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

B i t i b bilit

7/24/2019 Dtmc Basic

9/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

B i t i b bilit

7/24/2019 Dtmc Basic

10/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probabilit

7/24/2019 Dtmc Basic

11/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac =

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

12/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

13/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

14/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

15/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

16/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Intersection

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

17/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Intersection A B

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

18/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Intersection A B Outcome is inbothA and B

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

19/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Intersection A B Outcome is inbothA and B

(Empty) or impossible event

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Basic concepts in probability

7/24/2019 Dtmc Basic

20/185

Basic concepts in probability

Sample space set of all possible outcomesOutcome Event A, B

Complementary event Ac = \A

Union A B outcome in at least one ofAor B

Intersection A B Outcome is inbothA and B

(Empty) or impossible event

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

7/24/2019 Dtmc Basic

21/185

Probability axioms and first results

0 P(A) 1, P() =1

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

7/24/2019 Dtmc Basic

22/185

Probability axioms and first results

0 P(A) 1, P() =1

P(A B) = P(A) + P(B) forA B=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

7/24/2019 Dtmc Basic

23/185

y

0 P(A) 1, P() =1

P(A B) = P(A) + P(B) forA B=

Leading to

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

7/24/2019 Dtmc Basic

24/185

y

0 P(A) 1, P() =1

P(A B) = P(A) + P(B) forA B=

Leading toP() =0, P(Ac) =1 P(A)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Probability axioms and first results

7/24/2019 Dtmc Basic

25/185

y

0 P(A) 1, P() =1

P(A B) = P(A) + P(B) forA B=

Leading toP() =0, P(Ac) =1 P(A)

P(A B) = P(A) + P(B) P(A B)

(inclusion- exlusion)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

26/185

p y p

P(A|B) = P(A B)P(B)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

27/185

p y p

P(A|B) = P(A B)P(B)

P(A B) = P(A|B)P(B)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

28/185

p y p

P(A|B) = P(A B)P(B)

P(A B) = P(A|B)P(B)

(multiplication rule)

iBi=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

29/185

P(A|B) = P(A B)P(B)

P(A B) = P(A|B)P(B)

(multiplication rule)

iBi= Bi Bj= i=j

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

30/185

P(A|B) =P

(A B)P(B)

P(A B) = P(A|B)P(B)

(multiplication rule)

iBi= Bi Bj= i=j

P(A) =

i

P(A|Bi)P(Bi) (law of total probability)

Independence:

P(A|B) = P(A|Bc) = P(A)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Conditional probability and independence

7/24/2019 Dtmc Basic

31/185

P(A|B) =P

(A B)P(B)

P(A B) = P(A|B)P(B)

(multiplication rule)

iBi= Bi Bj= i=j

P(A) =

i

P(A|Bi)P(Bi) (law of total probability)

Independence:

P(A|B) = P(A|Bc) = P(A) P(A B) = P(A)P(B)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Discrete random variables

7/24/2019 Dtmc Basic

32/185

Mapping from sample space to metric space

(Read: Real space)

Probability mass function

f(x) = P(X=x) = P ({|X() =x})

Distribution function

F(x) = P(Xx) = P ({|X() x}) =tx

f(t)

Expectation

E(X) =

x

xP(X=x), E(g(X)) =

x

g(x)P(X=x) =

x

g(x)f(x)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Joint distribution

7/24/2019 Dtmc Basic

33/185

f(x1, x2) = P(X1=x1, X2=x2), F(x1, x2) = P(X1x1, X2 x2)

fX1

(x1) = P(X1=x1) = x2

P(X1=x1, X2=x2) = x2

f(x1, x2)

FX1 (x1) =

tx1,x2

P(X1=t1, X2=x2) =F(x1,)

Straightforward to extend tonvariables

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

34/185

We can define the joint distribution of (X0, X1)through

P(X0=x0, X1 =x1) = P(X0 =x0)P(X1=x1|X0 =x0) = P(X0=x0)Px0,x1

Suppose now some stationarity in addition that X2 conditioned

onX1 is independent onX0

P(X0 =x0, X1 =x1, X2=x2) =P(X0=x0)P(X1=x1|X0=x0)P(X2=x2|X0 =x0, X1=x1) =

P(X0=x0)P(X1=x1|X0=x0)P(X2=x2|X1=x1) =

px0 Px0,x1 Px1,x2

which generalizes to arbitraryn.

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property

7/24/2019 Dtmc Basic

35/185

P (Xn=xn|H)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property

7/24/2019 Dtmc Basic

36/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property

7/24/2019 Dtmc Basic

37/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property

7/24/2019 Dtmc Basic

38/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property

7/24/2019 Dtmc Basic

39/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov property(X |H)

7/24/2019 Dtmc Basic

40/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov propertyP (X |H)

7/24/2019 Dtmc Basic

41/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov propertyP (X |H)

7/24/2019 Dtmc Basic

42/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov propertyP (X |H)

7/24/2019 Dtmc Basic

43/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov propertyP (X x |H)

7/24/2019 Dtmc Basic

44/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix P={pij}

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Markov propertyP (X x |H)

7/24/2019 Dtmc Basic

45/185

P (Xn=xn|H)

= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)

= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix P={pij}

The joint probability of the first noccurrences is

P(X0 =x0, X1 =x1, X2=x2. . . , Xn=xn) =px0 Px0,x1 Px1,x2. . . Pxn1,xn

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

46/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

47/185

Storage/inventory models

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

48/185

Storage/inventory models Telecommunications systems

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

49/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

50/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n Xncould be

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

51/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n Xncould be

The number of cars in stock

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

52/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n Xncould be

The number of cars in stock The number of days since last rainfall

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

53/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n Xncould be

The number of cars in stock The number of days since last rainfall The number of passengers booked for a flight

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

A profuse number of applications

7/24/2019 Dtmc Basic

54/185

Storage/inventory models Telecommunications systems

Biological models

Xnthe value attained at time n Xncould be

The number of cars in stock The number of days since last rainfall The number of passengers booked for a flight

See textbook for further examples

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

55/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

56/185

P=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

57/185

P=

1 p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

58/185

P=

1 p p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

59/185

P=

1 p p 0 . . . 0 0 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

60/185

P=

1 p p 0 . . . 0 0 0q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

61/185

P=

1 p p 0 . . . 0 0 0q 1 p q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

62/185

P=

1 p p 0 . . . 0 0 0q 1 p q p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

63/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

64/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0...

... ...

... ...

... ...

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

65/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0... ...

... ...

... ...

...

0 0 0 . . .

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

66/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0... ...

... ...

... ...

...

0 0 0 . . . q 1 p q p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

67/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0... ...

... ...

... ...

...

0 0 0 . . . q 1 p q p0 0 0 . . . 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

68/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0... ...

... ...

... ...

...

0 0 0 . . . q 1 p q p0 0 0 . . . 0 q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 1: Random walk with tworeflecting barriers0andN

7/24/2019 Dtmc Basic

69/185

P=

1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0

0 q 1 p q . . . 0 0 0... ...

... ...

... ...

...

0 0 0 . . . q 1 p q p0 0 0 . . . 0 q 1 q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

70/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

71/185

P=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

72/185

P=

1 p p 0 0 0 . . .

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

73/185

P=

1 p p 0 0 0 . . .q 1 p q p 0 0 . . .

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

74/185

P=

1 p p 0 0 0 . . .q 1 p q p 0 0 . . .

0 q 1 p q p 0 . . .0 0 q 1 p q p . . .

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 2: Random walk with onereflecting barrier at0

7/24/2019 Dtmc Basic

75/185

P=

1 p p 0 0 0 . . .q 1 p q p 0 0 . . .

0 q 1 p q p 0 . . .0 0 q 1 p q p . . ....

... ...

... ... . . .

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

76/185

P=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

77/185

P=

1

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

78/185

P=

1 0 0 0 0 . . . 0 0 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

79/185

P=

1 0 0 0 0 . . . 0 0 0

q 1 p q p 0 0 . . . 0 0 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

80/185

P=

1 0 0 0 0 . . . 0 0 0

q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...

... ...

... ...

... ...

... ...

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

81/185

P=

1 0 0 0 0 . . . 0 0 0

q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...

... ...

... ...

... ...

... ...

0 0 0 0 0 . . . q 1 p q p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Example 3: Random walk with twoabsorbing barriers

7/24/2019 Dtmc Basic

82/185

P=

1 0 0 0 0 . . . 0 0 0

q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...

... ...

... ...

... ...

... ...

0 0 0 0 0 . . . q 1 p q p0 0 0 0 0 . . . 0 0 1

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

83/185

The matrix can be finite

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

84/185

The matrix can be finite (if the Markov chain is finite)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

85/185

The matrix can be finite (if the Markov chain is finite)

P=

p1,1 p1,2 p1,3 . . . p1,np2,1 p2,2 p2,3 . . . p2,np3,1 p3,2 p3,3 . . . p3,n

... ...

... ...

...

pn,1 pn,2 pn,3 . . . pn,n

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

7/24/2019 Dtmc Basic

86/185

The matrix can be finite (if the Markov chain is finite)

P=

p1,1 p1,2 p1,3 . . . p1,np2,1 p2,2 p2,3 . . . p2,np3,1 p3,2 p3,3 . . . p3,n

... ...

... ...

...

pn,1 pn,2 pn,3 . . . pn,n

Two reflecting/absorbing barriers

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

or infinite

7/24/2019 Dtmc Basic

87/185

or infinite

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

or infinite (if the Markov chain is infinite)

7/24/2019 Dtmc Basic

88/185

or infinite (if the Markov chain is infinite)

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

or infinite (if the Markov chain is infinite)

7/24/2019 Dtmc Basic

89/185

or infinite (if the Markov chain is infinite)

P=

p1,1 p1,2 p1,3 . . . p1,n . . .p2,1 p2,2 p2,3 . . . p2,n . . .

p3,1 p3,2 p3,3 . . . p3,n . . ....

... ...

... ...

...

pn,1 pn,2 pn,3 . . . pn,n . . ....

... ...

... ...

...

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

or infinite (if the Markov chain is infinite)

7/24/2019 Dtmc Basic

90/185

or infinite (if the Markov chain is infinite)

P=

p1,1 p1,2 p1,3 . . . p1,n . . .p2,1 p2,2 p2,3 . . . p2,n . . .

p3,1 p3,2 p3,3 . . . p3,n . . ....

... ...

... ...

...

pn,1 pn,2 pn,3 . . . pn,n . . ....

... ...

... ...

...

At most one barrier

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

91/185

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

92/185

the engine that drives the process

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

93/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

94/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

Bo Friis Nielsen Discrete Time Markov Chains Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

95/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions For eachiwe have a conditional distribution

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

96/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions For eachiwe have a conditional distribution

What is the probability of the next state being jknowing thatthe current state isi

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

97/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions For eachiwe have a conditional distribution

What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

98/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions For eachiwe have a conditional distribution

What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)

jpij=1

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The matrixPcan be interpreted as

7/24/2019 Dtmc Basic

99/185

the engine that drives the process

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions For eachiwe have a conditional distribution

What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)

jpij=1

We say thatPis a stochastic matrix

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

100/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

101/185

We have defined rules for the behaviour from one value

and onwards

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

102/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

103/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

104/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0 X0 could be certain

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

105/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

106/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

107/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random P (X0 =i) =pi

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More definitions and the first properties

7/24/2019 Dtmc Basic

108/185

We have defined rules for the behaviour from one value

and onwards

Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random P (X0 =i) =pi Once again we collect the possibly infinite many

parameters in a vectorp

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

7/24/2019 Dtmc Basic

109/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

7/24/2019 Dtmc Basic

110/185

P (Xn=j|X0=i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

111/185

P (Xn=j|X0=i) =P(n)ij

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

112/185

P (Xn=j|X0=i) =P(n)ij

the probability of being in jat thenth transition having

started ini

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

113/185

P (Xn=j|X0=i) =P(n)ij

the probability of being in jat thenth transition having

started ini

Once again collected in a matrix

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

114/185

P (Xn=j|X0=i) =P(n)ij

the probability of being in jat thenth transition having

started ini

Once again collected in a matrix P(n) ={P(n)ij }

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

115/185

P (Xn=j|X0=i) =P(n)ij

the probability of being in jat thenth transition having

started ini

Once again collected in a matrix P(n) ={P(n)ij }

The rows ofP(n) can be interpreted like the rows ofP

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

116/185

P (Xn=j|X0=i) =P(n)ij

the probability of being in jat thenth transition having

started ini

Once again collected in a matrix P(n) ={P(n)ij }

The rows ofP(n) can be interpreted like the rows ofP

We can define a new Markov chain on a larger time scale

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

nstep transition probabilities

(n)

7/24/2019 Dtmc Basic

117/185

P (Xn=j|X0=i) =P( )ij

the probability of being in jat thenth transition having

started ini

Once again collected in a matrix P(n) ={P(n)ij }

The rows ofP(n) can be interpreted like the rows ofP

We can define a new Markov chain on a larger time scale

(Pn)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

118/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

119/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

120/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

121/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

122/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

7/24/2019 Dtmc Basic

123/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 0 0

7/24/2019 Dtmc Basic

124/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 0 0

7/24/2019 Dtmc Basic

125/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 0 0

7/24/2019 Dtmc Basic

126/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p) 2qp

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 p p 0 0

7/24/2019 Dtmc Basic

127/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p) 2qp 0

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 p p 0 0

7/24/2019 Dtmc Basic

128/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p) 2qp 0 p2

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 p p 0 0

7/24/2019 Dtmc Basic

129/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example

1 p p 0 0

7/24/2019 Dtmc Basic

130/185

P=

1 p p 0 0q 0 p 0

0 q 0 p

0 0 q 1 q

P(2) =

(1 p)2

+pq (1 p)p p2

0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)0 q2 (1 q)q (1 q)2 +qp

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

7/24/2019 Dtmc Basic

131/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

7/24/2019 Dtmc Basic

132/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

133/185

n+m

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

134/185

n+m

At some intermediate timenwe must be in some state k

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

135/185

n+m

At some intermediate timenwe must be in some state k

We apply the law of total probability

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

136/185

n+m

At some intermediate timenwe must be in some state k

We apply the law of total probability

P (B) =

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

137/185

n+m At some intermediate timenwe must be in some state k

We apply the law of total probability

P (B) = kP (B|Ak)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n + m

7/24/2019 Dtmc Basic

138/185

n+m At some intermediate timenwe must be in some state k

We apply the law of total probability

P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n+m

7/24/2019 Dtmc Basic

139/185

+ At some intermediate timenwe must be in some state k

We apply the law of total probability

P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)

= k

P (Xn+m=j|X0 =i, Xn=k)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Chapmann Kolmogorov equations

There is a generalisation of the example above

Suppose we start in iat time 0 and wants to get to jat time

n+m

7/24/2019 Dtmc Basic

140/185

+ At some intermediate timenwe must be in some state k

We apply the law of total probability

P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)

= k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)

7/24/2019 Dtmc Basic

141/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

7/24/2019 Dtmc Basic

142/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we get

7/24/2019 Dtmc Basic

143/185

k

P (Xn+m=j|Xn=k)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we get(X j|X k ) (X k |X i)

7/24/2019 Dtmc Basic

144/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we getP (X j|X k )P (X k |X i)

7/24/2019 Dtmc Basic

145/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

= k

P(m)

kj

P(n)

ik

=

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we getP (X j|X k )P (X k |X i)

7/24/2019 Dtmc Basic

146/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

= k

P(m)

kj

P(n)

ik

= k

P(n)

ik

P(m)

kj

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we getP (X j|X k )P (X k |X i)

7/24/2019 Dtmc Basic

147/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

= k

P(m)

kj

P(n)

ik

= k

P(n)

ik

P(m)

kj

which in matrix formulation is

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we getP (X j|X k )P (X k |X i)

7/24/2019 Dtmc Basic

148/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

= k

P(m)

kj

P(n)

ik

= k

P(n)

ik

P(m)

kj

which in matrix formulation is

P(n+m) =P(n)P(m)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

k

P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)

by the Markov property we getP (X j|X k )P (X k |X i)

7/24/2019 Dtmc Basic

149/185

k

P (Xn+m=j|Xn=k)P (Xn=k|X0=i)

= k

P(m)

kj

P(n)

ik

= k

P(n)

ik

P(m)

kj

which in matrix formulation is

P(n+m) =P(n)P(m) =Pn+m

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

7/24/2019 Dtmc Basic

150/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

7/24/2019 Dtmc Basic

151/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

152/185

j

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

153/185

j

Define P (Xn=j)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

154/185

j

Define P (Xn=j) =p(n)

j

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

155/185

Define P (Xn=j) =p(n)

j

withp(n) ={p(n)

j }

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

156/185

Define P (Xn=j) =p(n)

j

withp(n) ={p(n)

j }we find

p(n)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

157/185

Define P (Xn=j) =p(n)

j

withp(n) ={p(n)

j }we find

p(n) =p

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

The probability ofXn

The behaviour of the process itself - Xn

The behaviour conditional onX0=iis known (P(n)ij )

7/24/2019 Dtmc Basic

158/185

Define P (Xn=j) =p(n)

j

withp(n) ={p(n)

j }we find

p(n) =pP(n) =pPn

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

159/185

q q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

160/185

withp=

13 , 0, 0,

23

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

161/185

withp=

13 , 0, 0,

23

we get

p(1)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

162/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

163/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

1 p p 0 0q 0 p 00 q 0 p

0 0 q 1 q

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

164/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

1 p p 0 0q 0 p 00 q 0 p

0 0 q 1 q

=

1 p

3 ,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

165/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

1 p p 0 0q 0 p 00 q 0 p

0 0 q 1 q

=

1 p

3 ,

p

3,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

166/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

1 p p 0 0q 0 p 00 q 0 p

0 0 q 1 q

=

1 p

3 ,

p

3,2q

3 ,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Small example - revisited

P=

1 p p 0 0q 0 p 0

0 q 0 p0 0 q 1 q

7/24/2019 Dtmc Basic

167/185

withp=

13 , 0, 0,

23

we get

p(1) =

1

3, 0, 0,

2

3

1 p p 0 0q 0 p 00 q 0 p

0 0 q 1 q

=

1 p

3 ,

p

3,2q

3 ,

2(1 q)

3

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p= 1

3

, 0, 0,2

3 , (1 p)2 +pq (1 p)p p2 0

2

7/24/2019 Dtmc Basic

168/185

P2 =

q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)0 q2 (1 q)q (1 q)2 +qp

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2)

7/24/2019 Dtmc Basic

169/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

7/24/2019 Dtmc Basic

170/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)

7/24/2019 Dtmc Basic

171/185

q qp p( q)0 q2 (1 q)q (1 q)2 +qp

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)

7/24/2019 Dtmc Basic

172/185

( )0 q2 (1 q)q (1 q)2 +qp

=

(1 p)2 +pq

3 ,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)

7/24/2019 Dtmc Basic

173/185

0 q2 (1 q)q (1 q)2 +qp

=

(1 p)2 +pq

3 ,

(1 p)p

3 ,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)

7/24/2019 Dtmc Basic

174/185

0 q2 (1 q)q (1 q)2 +qp

=

(1 p)2 +pq

3 ,

(1 p)p

3 ,

4qp

3 ,

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

p(2) =

1

3, 0, 0,

2

3

(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2

q2 0 2qp p(1 q)2 2

7/24/2019 Dtmc Basic

175/185

0 q2 (1 q)q (1 q)2 +qp

=

(1 p)2 +pq

3 ,

(1 p)p

3 ,

4qp

3 ,

2p(1 q)

3

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

First step analysis - setup

Consider the transition probability matrix

P=

1 0 0

0 0 1

7/24/2019 Dtmc Basic

176/185

0 0 1 Define

T =min {n0:Xn=0 orXn=2}

and

u= P(XT =0|X0 =1) v= E(T|X0=1)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

First step analysis - absorption probability

u = P(XT =0|X0=1)

=2

k=0

P(X1=k|X0 =1)P(XT =0|X0 =1, X1=k)

7/24/2019 Dtmc Basic

177/185

=2

k=0

P(X1=k|X0 =1)P(XT =0|X1 =k)

= P1,0 1 +P1,1 u+P1,2 0.

And we find

u= P1,0

1 P1,1=

+

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

First step analysis - time to absorption

v = E(T|X0 =1)

=2

k=0

P(X1=k|X0=1)E(T|X0 =1, X1 =k)

7/24/2019 Dtmc Basic

178/185

= 1 +2

k=0

P(X1 =k|X0=1)E(T|X0 =k)(NB!

= 1 +P1,0 0 +P1,1 v+P1,2 0.

And we find

v= 1

1 P1,1=

1

1

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

More than one transient state

P=

1 0 0 0

P1,0 P1,1 P1,2 P1,3P2,0 P2,1 P2,2 P2,3

7/24/2019 Dtmc Basic

179/185

0 0 0 1

Here we will need conditional probabilitiesui= P(XT =0|X0=i)

and conditional expectationsvi= E(T|X0 =i)

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Leading to

u1 = P1,0+P1,1u1+P1,2u2

u2 = P2,0+P2,1u1+P2,2u2

7/24/2019 Dtmc Basic

180/185

and

v1 = 1 +P1,1v1+P1,2v2

v2 = 1 +P2,1v1+P2,2v2

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

General finite state Markov chain

P

Q R

7/24/2019 Dtmc Basic

181/185

P=

0 I

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

General absorbing Markov chain

T =min {n0, Xnr}

In statejwe accumulate rewardg(j),wiis expected totalreward conditioned on start in state i

7/24/2019 Dtmc Basic

182/185

wi= ET1

n=0

g(Xn)|X0 =ileading to

wi=g(i) +

j

Pi,jwj

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

7/24/2019 Dtmc Basic

183/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

g(i) =1 expected time to absorption (vi)

7/24/2019 Dtmc Basic

184/185

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification

Special cases of general absorbing Markov chain

g(i) =1 expected time to absorption (vi)

7/24/2019 Dtmc Basic

185/185

g(i) =ikexpected visits to state kbefore absorption

Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification