eager markov chains parosh aziz abdulla noomene ben henda richard mayr sven sandberg texpoint fonts...

Eager Markov Chains

Parosh Aziz Abdulla

Noomene Ben Henda

Richard Mayr

Sven Sandberg

Info

rmat

ions

tekn

olog

i

Institutionen för informationsteknologi | www.it.uu.se

Outline

Introduction Expectation Problem Algorithm Scheme Termination Conditions Subclasses of Markov Chains

Examples Conclusion

Info

rmat

ions

tekn

olog

i


Introduction

Model: Infinite-state Markov chains Used to model programs with unreliable

channels, randomized algorithms…

Interest: Conditional expectations Expected execution time of a program Expected resource usage of a program

Info

rmat

ions

tekn

olog

i


Introduction Infinite-state Markov

chain

Infinite set of states Target set Probability

distributions

s0

s1 s2

s3

s4

s5

Example

0.3

0.20.5

1

0.5

0.5

1

0.1

0.9

0.7 0.3

Info

rmat

ions

tekn

olog

i


Introduction

Reward function

Defined over paths reaching the target set

s0

s1 s2

s3

s4

s5

0.3

0.20.5

1

0.5

0.5

1

0.1

0.9

0.7 0.3

Example

2

22

0

-3

-1

Info

rmat

ions

tekn

olog

i


Expectation Problem

Instance A Markov chain A reward function

Task Compute/approximate the conditional

expectation of the reward function

Info

rmat

ions

tekn

olog

i


Expectation Problem Example:

The weighted sum

The reachability probability

The conditional expectation

10.8

0.1

0.1

1

1

1

s0

2 2

0

-3

-5

0.8*4+0.1*(-5)=2.7

0.8+0.1=0.9

2.7/0.9=3

s1 s2

s3

s4

Info

rmat

ions

tekn

olog

i


Expectation Problem

Remark Problem in general studied for finite-state

Markov chains

Contribution Algorithm scheme to compute it for infinite-

state Markov chains Sufficient conditions for termination

Info

rmat

ions

tekn

olog

i


Algorithm Scheme At each iteration n

Compute paths up to depth n

Consider only those ending in the target set

Update the expectation accordingly

Path Exploration

Info

rmat

ions

tekn

olog

i


Algorithm Scheme

Correctness The algorithm computes/approximates the

correct value

Termination Not guaranteed: lower-bounds but no upper-

bounds

Info

rmat

ions

tekn

olog

i


Termination Conditions

Exponentially bounded reward function

The intuition: limit on the growth of the reward functions

Remark: The limit is reasonable: for example polynomial functions are exponentially bounded

Info

rmat

ions

tekn

olog

i



n0

· k®nThe abs of the reward

k

Bound on the reward

Info

rmat

ions

tekn

olog

i



Eager Markov chain

The intuition: Long paths contribute less in the expectation value

Remark: Reasonable: for example PLCS, PVASS, NTM induce all eager Markov chains

Info

rmat

ions

tekn

olog

i



n0

1

k· k®n

Prob. of reaching the target in more

than n steps

Bound on the probability

Info

rmat

ions

tekn

olog

i



Pf

Ws

Ce

E f

P

Wf

n0 nc

"(nc)

Info

rmat

ions

tekn

olog

i


Subclasses of Markov Chains Eager Markov chains

Markov chains with finite eager attractor

Markov chains with the bounded coarseness property

NTM

PVASS

PLCS

Info

rmat

ions

tekn

olog

i


Finite Eager Attractor Attractor:

Almost surely reached from every state

Finite eager attractor: Almost surely reached Unlikely to stay ”too

long” outside of it

A EA

Info

rmat

ions

tekn

olog

i


Finite Eager Attractor

EA

0

1

n

b

Prob. to return in More than n steps

· b̄ n

Info

rmat

ions

tekn

olog

i



Finite eager attractor implies eager Markov chain??

Reminder: Eager Markov chain:

· k®nProb. of reaching the target in more

than n steps

Info

rmat

ions

tekn

olog

i



FEA

Paths of length n that visit the attractor t times

Info

rmat

ions

tekn

olog

i



Proof idea: identify 2 sets of paths Paths that visit the attractor often without

going to the target set:

Paths that visit the attractor rarely without going the target set:

t · n=c

t > n=c

Info

rmat

ions

tekn

olog

i


Finite Eager AttractorPaths visiting the attractor rarely: t less than n/c

FEA

·P n=c

t=1 C t¡ 1n¡ 1b

t¯n¡ tPr_n

· (( cc¡ 1)(2c)t=c(1

c + b¯ )1=c¯)n

Info

rmat

ions

tekn

olog

i


Finite Eager AttractorPaths visiting the attractor often: t greater than n/c

FEA

Pt ̧ ¹Pl · (1¡ ¹ )

Po_n · !(1¡ ¹ )(1¡ (1¡ ¹ )1=! ) ((1¡ ¹ )

1c! )n

Info

rmat

ions

tekn

olog

i


Probabilistic Lossy Channel Systems (PLCS)

Motivation:

Finite-state processes communicating through unbounded and unreliable channels

Widely used to model systems with unreliable channels (link protocol)

Info

rmat

ions

tekn

olog

i


PLCS

ab

b

Send c!a

ab

Receive c?b

a

c?b

q0

q3 q2

q1

nop

c!a

c!b

aba

Channel c

nop1 21

1

1

Info

rmat

ions

tekn

olog

i


PLCS

c?b

q0

q3 q2

q1

nop

c!a

c!b

aba

Channel c

nop1 21

1

1

ab

b

Loss

b ba a

Info

rmat

ions

tekn

olog

i


PLCS Configuration

Control location Content of the

channel

Example [q3,”aba”]

c?b

q0

q3 q2

q1

nop

c!a

c!b

aba

Channel c

nop1 21

1

1

Info

rmat

ions

tekn

olog

i


PLCS

A PLCS induces a Markov chain:

States: Configurations

Transitions: Loss steps combined with discrete steps

Info

rmat

ions

tekn

olog

i


PLCS Example:

[q1,”abb”] [q2,”a”] By losing one of the

messages ”b” and firing the marked step.

Probability: P=Ploss*2/3

c?b

q0

q3 q2

q1

nop

c!a

c!b

aba

Channel c

nop1 21

1

1

Info

rmat

ions

tekn

olog

i


PLCS

Result: Each PLCS induces a Markov chain with finite eager attractor.

Proof hint: When the size of the channels is big enough, it is more likely (with a probability greater than ½) to lose a message.

Info

rmat

ions

tekn

olog

i


Bounded Coarseness

The probability of reaching the target within K steps is bounded from below by a constant b.

Info

rmat

ions

tekn

olog

i


Bounded Coarseness

Boundedly coarse Markov chain implies eager Markov chain??

Reminder: Eager Markov chain:

· k®nProb. of reaching the target in more

than n steps

Info

rmat

ions

tekn

olog

i


Bounded CoarsenessProb. Reach. ¸ b

Wit

hin

K s

teps

K nK steps2K

Pn · (1¡ b)nP2 · (1¡ b)2

Pn:Prob. of avoidingthe target in nK steps

P1 · (1¡ b)

Info

rmat

ions

tekn

olog

i


Probabilistic Vector Addition Systems with states (PVASS)

Motivation:

PVASS are generalizations of Petri-nets.

Widely used to model parallel processes, mutual exclusion program…

Info

rmat

ions

tekn

olog

i


PVASS Configuration

Control location Values of the

variables x and y

Example:

[q1,x=2,y=0]

q0

q3 q2

q1

nop

--x --y

++x

++y1 2

++x

1

4

1

1

Info

rmat

ions

tekn

olog

i


PVASS

A PVASS induces a Markov chain:

States: Configurations

Transitions: discrete steps

Info

rmat

ions

tekn

olog

i


PVASS Example:

[q1,1,1] [q2,1,0] By taking the marked

step.

Probability: P=2/3

q0

q3 q2

q1

nop

--x --y

++x

++y1 2

++x

1

4

1

1

Info

rmat

ions

tekn

olog

i


PVASS

Result: Each PVASS induces a Markov chain which has the bounded coarseness property.

Info

rmat

ions

tekn

olog

i


Noisy Turing Machines (NTM)

Motivation:

They are Turing Machines augmented with a noise parameter.

Used to model systems operating in ”hostile” environment

Info

rmat

ions

tekn

olog

i


NTM

Fully described by a Turing Machine and a noise parameter.

q1

q3q2

q4

a/b b

a/b

b #a/b

#

RRR

RR

S

S

ab# b #aa b

Info

rmat

ions

tekn

olog

i


NTMq1

q3q2

q4

a/b b

a/b

b #a/b

#

RRR

RR

S

S

ab# b #aa b

Discret Step

ab# b #aa b

bb# b #aa b

Info

rmat

ions

tekn

olog

i


NTMq1

q3q2

q4

a/b b

a/b

b #a/b

#

RRR

RR

S

S

ab# b #aa b

Noise Step

ab# b #aa b

#b# b #aa b

Info

rmat

ions

tekn

olog

i


NTM

Result: Each NTM induces a Markov chain which has the bounded coarseness property.

Info

rmat

ions

tekn

olog

i


Conclusion

Summary: Algorithm scheme for approximating

expectations of reward functions

Sufficient conditions to guarantee termination: Exponentially bounded reward function Eager Markov chains

Info

rmat

ions

tekn

olog

i


Conclusion

Direction for future work

Extending the result to Markov decision processes and stochastic games

Find more concrete applications

Thank you

Info

rmat

ions

tekn

olog

i


PVASS Order on

configurations: <= Same control

locations Ordered values of the

variables

Example: [q0,3,4] <= [q0,3,5]

q0

q3 q2

q1

nop

--x --y

++x

++y1 2

++x

1

4

1

1

Info

rmat

ions

tekn

olog

i


PVASS

Probability of each step > 1/10

Boundedly coarse: parameters K and 1/10^K

q0

q3 q2

q1

nop

--x --y

++x

++y1 2

++x

1

4

1

1

Targetset

K iterations

eager markov chains parosh aziz abdulla noomene ben henda richard mayr sven sandberg texpoint fonts...

Documents

eager markov chains

termination slide

reward slide

bounded slide

probability slide

finite eager attractor

program slide

ea slide