markov models markov 26/02/2018 markov models 85 markov models •advantage: it provides a more...

MARKOV MODELS MARKOV CHAINS

Lecture 4.

26/02/2018 Markov Models 82

Markov Models

• Markov Model: is a stochastic model to describe a randomly changing system,

• by Andrej Andrejevics Markov – russan mathematician, 1856 – 1922

26/02/2018 Markov Models 83

Markov Models

• Example: Lupus

26/02/2018 Markov Models 84

Markov Models

• Markov Model: is a stochastic model to describe a randomly changing system,

• properties:

• it is a stochastic process without after-effects = the future states depend only on the current

state,

• the behaviour of the system depends directly on the preceding event, and is independent

from the past,

• the system states and the transition between the states are to be considered as a random event.

26/02/2018 Markov Models 85

Markov Models

• advantage: it provides a more powerful way of modelling systems, that are repairable, allowing variables such as the time taken to repair a system to be incorporated,

• described by state diagrams (chaines), eg: the reliablity graph of a two-state system:

• where:

• λ = failure rate [1/hour] – transition rate,

• µ = repair rate [1/hour] – transition rate, MTTR= 1/µ,

• 0: represents the working conditions,

• 1: represents the not working conditions.

26/02/2018 Markov Models 86

Markov Models

• Markov – chain is a stochastic model, in which the probability of an event depends

directly on the preceding event.

• independent of any previous event,

• the model detrmines the future behaviour of the given system,

• state of the system:

• several possible states (e.g. working/not working for just one element),

• state transitions:

• random events/stochastic event.

26/02/2018 Markov Models 87

Markov Models

• state transition described by: • state transition frequencies, rates [1/h] (λ, µ)

• constatnt rates: homogenous Markov-chain - described by two values:

• the probability of the initial state, Pi(0),

• the probability of the state transition Pji(∆t), where i menas the initial state and j means the following state,

• example:

• if j=i+1, Pji(∆t)= λji∆t – fauliure,

• if j=i-1, Pji(∆t)= µji∆t – repairation, • time-dependent rates: semi Markov-chain,

• representation: • state graph,

• mathematical model: • differential equation system.

26/02/2018 Markov Models 88

Markov Models

• application in the reliability theory:

• modelling systems with multiple failure states:

• eg. redundant systems,

• modelling systems with multidirectional state transitions:

• eg. repairable systems,

• application in the modelling:

• to determine the status of the sytem,

• knowing the architecture and the properties of a system real engineering task,

• determination of the transition rates:

• based on the architecture and the properties of the system/components.

26/02/2018 Markov Models 89

Markov Models

• application in the modelling:

• types of the graphs of a non-repairable system:

• transient (operable) states,

• absorbent (inoperable) states,

• a systems with multiple failure states may have several absorbent states (eg. passive and dangerous).

0

1

2

λ10

λ21

0

1 2

λ10 λ20

26/02/2018 Markov Models 90

Markov Models

• application in the modelling:

• types of the graphs of a repairable system:

• the states of a repairable system are achievable

periodically, that’s why those forme a closed conditions

set

0

1

2

λ10

λ21

μ01

μ12

26/02/2018 Markov Models 91

Mathematical Model

• the state model can be described mathematically by state equations

• steps to determine the state equations:

1. determination of the difference equations,

2. conversion to differential equation,

3. conversion to matrix form.

26/02/2018 Markov Models 92

Mathematical Model

0

1

2

λ10

λ21

         

     

       

      ttPtPttP

ttPttPtPttP

ttPtPttP

ttptptPtPttP









21122

21110011

10000

101010000









     

 

   

   

     

   tPtP

tPtPtP

tPtP

tP dt

tdP

tP t

tPttP

t

121

'

2

121010

'

1

010

'

0

'

0 0

0

010 00

lim

















 





 

 

 

 

 

   





  





  





  













  





  





tP

tP

tP

tP

tP

tP

2

1

0

21

2110

10

'

2

'

1

'

0

00

0

00







   tPAtP '

example: a 3 state system –

passive redundancy with n=2

26/02/2018 Markov Models 93

Mathematical Model

• remarks:

• the transition matrix is deduceable directly from the state graph,

• in the transition matrix, the sum of column is zero,

• in the main diagonal, the negative values in a given column are equals with the sum of the other

values.

0

1

2

λ10

λ21

 

 

 

 

 

   





  





  





  













  





  





tP

tP

tP

tP

tP

tP

2

1

0

21

2110

10

'

2

'

1

'

0

00

0

00







   tPAtP '

26/02/2018 Markov Models 94

Examples

0

1 2

λ10 λ20

0

1

2

λ10

λ21

μ01

μ12

 

 

 

 

 

   





  





  





  













  





  





tP

tP

tP

tP

tP

tP

2

1

0

20

10

2010

'

2

'

1

'

0

00

00

00







 

 

 

 

 

   





  





  





  













  





  





tP

tP

tP

tP

tP

tP

2

1

0

1221

12012110

0110

'

2

'

1

'

0

0

0







• models of previously learned systems:

• single system/element/component without repair:

26/02/2018 Markov Models 95

Markov Models

0 1

λ

  tetP 11

  tetP 0 

1 T

 

  







0

0





connection?

26/02/2018 Markov Models 96

Markov Models

• models of previously learned systems:

• active redundancy (parallel), 1 out of n system (here n=2):

0 1 2

2λ λ

   tt eetP  21 2  