runtime enforcement of timed properties -...

Introduction Enforcement of timed properties Enforcement-safety property Enforcement-co-safety property Related work and Conclusion

Runtime Enforcement of Timed Properties

Y. Falcone, Th. Jeron, H. MarchandS. Pinisetty, A. Rollet, O. Nguena-Timo

Vacsim project meeting, INRIA Rennes

1 / 33

Outline

1 Introduction

2 Enforcement of timed properties

3 Enforcement-safety property

4 Enforcement-co-safety property

5 Related work and Conclusion

2 / 33

Context

Verification

Verification of a system/program : P |= Spec.

Spec : Set of requirements/properties.

Runtime Verification

Check if a run of a system satisfies the requirements.

Monitoring an executing system.

No system model.

Online/offline.

Runtime Enforcement

Runtime enforcement : Extension of runtime verification.

A run of a system should satisfy the requirements.

3 / 33

Context

Verification

No system model.

Online/offline.

Runtime Enforcement

3 / 33

Context

Verification

No system model.

Online/offline.

Runtime Enforcement

3 / 33

Verification and enforcement monitors

Verification Monitor

verdictsevents Monitor

Verification

σ ∈ Σ∞

Dω ∈ D∞σ |= ϕ?

Does the run satisfy theproperty ?

Input : stream of events.

Output : stream of verdicts.

Enforcement Monitor

events

memory

eventsEMϕ

o |= ϕσ |= ϕ?

(o � σ)

The run should satisfy theproperty.

Input : stream of events(may violate the property).

Output : stream of events(should satisfy the property).

4 / 33

Enforcement Monitor (Untimed case)

Dedicated to a property ϕ

Possibly augmented with a memorizationmechanism

events

memory

eventsEMϕ

o |= ϕσ |= ϕ?

(o � σ)

Enforcement mechanism

An EM modifies the current execution sequence (sometimes like a“filter”)

reads an input sequence σ ∈ Σ∗

outputs a new sequence o ∈ Σ∗

endowed with a set P of enforcement primitives

operates on the memorization mechanism Mdelete or insert events using the memory content and the currentinput

An EM behaves as a function

[[ϕ]]P(·) : Σ∗ → Σ∗

σ 7→ o = [[ϕ]]P(σ)

5 / 33

Motivation for runtime enforcement

Formal verification

More difficult as the size/complexity of the system grows (statespace explosion).Expensive (time and other costs).Applying to an existing system (may require re-design/developingentire system).

Runtime enforcement

Monitoring an executing system to assure correct execution.Only the requirements/properties should be formally expressed.Can be easily applied to an existing system (without any/minormodifications).

6 / 33

Formal verification

Runtime enforcement

6 / 33

Formal verification

Runtime enforcement

6 / 33

Motivation for timed enforcement

Specifying the timing behavior

Allow specifying desired behavior of a system more precisely (timeconstraints between events).

After an action ”a”, action ”b” should occur.

After an action ”a”, action ”b” should occur, with a delay of atleast 5 time units between them.

The system should allow consecutive requests.

The system should allow consecutive requests with a delay of atleast 10 time units between any two requests.

Many application domains

Real-time embedded systems, monitor hardware failures,communication protocols, web services and many more.

Example : monitor a firewall to prevent DOS attack ensuringminimal delay between input events.

7 / 33

Outline

1 Introduction

8 / 33

Enforcement of timed properties

From untimed to timed properties enforcement

New elements that have to be taken into account

Input/output sequences are timed words :σ = (δ1, a1) · (δ2, a2) · · · (δn, an), δi ∈ R≥, ai ∈ Σ

Property ϕ described by a timed automaton or a timed logic

Synthesis of the corresponding enforcer ?

Class of enforceable properties ?→ Focus on safety and co-safety properties modeled by TA.

Model of the enforcer ?→ Similar operations (Store, Dump, Halt, Off) + memory→ No finite structure

New notions of transparency1 The enforcer can only increase the delay between two actions

Real-time constraints (e.g. computation time)

9 / 33

Enforcement of timed properties

From untimed to timed properties enforcement

New elements that have to be taken into account

Input/output sequences are timed words :σ = (δ1, a1) · (δ2, a2) · · · (δn, an), δi ∈ R≥, ai ∈ Σ

Property ϕ described by a timed automaton or a timed logic

Synthesis of the corresponding enforcer ?

Class of enforceable properties ?→ Focus on safety and co-safety properties modeled by TA.

Model of the enforcer ?→ Similar operations (Store, Dump, Halt, Off) + memory→ No finite structure

New notions of transparency1 The enforcer can only increase the delay between two actions

Real-time constraints (e.g. computation time)

9 / 33

Timed word and Property

Definition (Timed word over Σ)

A finite or infinite sequence of pairs (δ0, a0) · (δ1, a1) · (δ2, a2) · · ·s.t. ∀ i ∈ N, ai ∈ Σ, and δi ∈ R+ ≥ 0.

Definition (Property)

Defined by a timed language ϕ ⊆ (R≥0 × Σ)∗.A timed word σ satisfies ϕ (noted σ |= ϕ) if σ ∈ ϕ.Focus on properties specified by a TA Aϕ.

Safety : nothing bad should ever happen(prefix closed).

Co-safety : something good will eventually happen within a finiteamount of time (extension closed).

Safety and co-safety properties specified by TA

l0 l1 l2

Σ1 \ {r}r ,

x := 0

Σ1 \ {r}

r , x ≥ 5,x := 0

r , x<5

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

10 / 33

l0 l1 l2

Σ1 \ {r}r ,

x := 0

Σ1 \ {r}

r , x ≥ 5,x := 0

r , x<5

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

10 / 33

l0 l1 l2

Σ1 \ {r}r ,

x := 0

Σ1 \ {r}

r , x ≥ 5,x := 0

r , x<5

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

10 / 33

Other Definitions

Observation- obs

observation of σ ∈ (R≥0 × Σ)∗ at time t ∈ R≥0 :

obs(σ, t)∆= max{σ′ | σ′ 4 σ ∧ time(σ′) ≤ t}

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Observation- obs

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Observation- obs

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Observation- obs

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Observation- obs

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Observation- obs

Example

(1, a) · (3, r) · (1, r)t = 1

(1, a) · (3, r) · (1, r)t = 2

(1, a) · (3, r) · (1, r)t = 4

(1, a) · (3, r) · (1, r)t = 10

11 / 33

Other Definitions

Delays- 4d

σ′ delays σ (noted σ′ 4d σ)if

ΠΣ(σ′) 4 ΠΣ(σ) and ∀i ≤ |σ′|,del(σ(i)) ≤ del(σ′(i)).

Examples

(1, a) · (3, r) · (1, r) 4d (1, a) · (3, r) X

(1, a) · (2, r) · (1, r) 4d (1, a) · (3, r) · (1, r) X

(1, a) · (3, r) 4d (1, a) · (3, r) · (1, r) X(1, a) · (3, r) · (5, r) 4d (1, a) · (3, r) · (1, r) X

12 / 33

Other Definitions

Delays- 4d

Examples

(1, a) · (3, r) · (1, r) 4d (1, a) · (3, r) X

(1, a) · (2, r) · (1, r) 4d (1, a) · (3, r) · (1, r) X

12 / 33

Other Definitions

Delays- 4d

Examples

(1, a) · (3, r) · (1, r) 4d (1, a) · (3, r) X

(1, a) · (2, r) · (1, r) 4d (1, a) · (3, r) · (1, r) X

12 / 33

Other Definitions

Delays- 4d

Examples

(1, a) · (3, r) · (1, r) 4d (1, a) · (3, r) X

(1, a) · (2, r) · (1, r) 4d (1, a) · (3, r) · (1, r) X

12 / 33

Other Definitions

Delays- 4d

Examples

(1, a) · (3, r) · (1, r) 4d (1, a) · (3, r) X

(1, a) · (2, r) · (1, r) 4d (1, a) · (3, r) · (1, r) X

12 / 33

Enforcement Monitoring in a timed context

Enforcement Function

Enforcementfunction

σ, t E(σ, t) |= ϕ

E : (R≥0 × Σ)∗ × R≥0 → (R≥0 × Σ)∗.

ϕ : Property which should be enforced (specified by a TA).

σ : Input timed word.

Output at time t : E (σ, t) should satisfy some additional constraints[Soundness,Transparency ,Optimality ].

E realized as a Enforcement Monitor (EM).

13 / 33

Enforcement Monitoring in a timed context

Enforcement Function

Enforcementfunction

σ, t E(σ, t) |= ϕ

E : (R≥0 × Σ)∗ × R≥0 → (R≥0 × Σ)∗.

ϕ : Property which should be enforced (specified by a TA).

σ : Input timed word.

Output at time t : E (σ, t) should satisfy some additional constraints[Soundness,Transparency ,Optimality ].

E realized as a Enforcement Monitor (EM).

13 / 33

Enforcement Monitor- EM

events

memory

eventsEMϕ

o |= ϕσ |= ϕ?

(o � σ)

Memory

Set of pairs (delay, action) : Timed word.

Operations

Store : stores the received event and a delay in the memory.

Dump : removes the event from memory and releases it as output.

Halt : (optimization) stop input if property cannot be satisfied.

Off : (optimization) stop the monitor if property is satisfied (for anyinput in future).

14 / 33

Enforcement Monitor- EM

events

memory

eventsEMϕ

o |= ϕσ |= ϕ?

(o � σ)

Memory

Set of pairs (delay, action) : Timed word.

Operations

Store : stores the received event and a delay in the memory.

Dump : removes the event from memory and releases it as output.

Halt : (optimization) stop input if property cannot be satisfied.

Off : (optimization) stop the monitor if property is satisfied (for anyinput in future).

14 / 33

Outline

1 Introduction

15 / 33

Enforcement of a safety property

Soundness

∀σ ∈ (R≥0 × Σ)∗,∀t ∈ R≥0, E (σ, t) |= ϕ.

Transparency

At any time instant t, the output E (σ, t) delays the input obs(σ, t) :∀σ ∈ (R≥0 × Σ)∗,∀t ∈ R≥0, E (σ, t) 4d obs(σ, t) ∧ time(E (σ, t)) ≤ t.

Optimality

If E is sound and transparent, it is optimal for any σ, t if

(Op1) E (σ, t) is among the longest correct timed words delayingobs(σ, t).

(Op2) Every prefix of E (σ, t) has the shortest possible last delay.

Example

l0 l1 l2

Σ \ {r}r ,

x := 0

Σ \ {r}

r , x ≥ 5,x := 0

r , x<5

σ = (3, r) · (1, r) · (1, a)

E (σ, 9) = (3, r) · (5, r) X

E (σ, 9) = (3, r) · (5, r) · (1, a) XE (σ, 9) = (3, r) · (6, r) · (1, a) X

16 / 33

Soundness

∀σ ∈ (R≥0 × Σ)∗,∀t ∈ R≥0, E (σ, t) |= ϕ.

Transparency

Optimality

Example

l0 l1 l2

Σ \ {r}r ,

x := 0

Σ \ {r}

r , x ≥ 5,x := 0

r , x<5

σ = (3, r) · (1, r) · (1, a)

E (σ, 9) = (3, r) · (5, r) X

E (σ, 9) = (3, r) · (5, r) · (1, a) XE (σ, 9) = (3, r) · (6, r) · (1, a) X

16 / 33

Soundness

∀σ ∈ (R≥0 × Σ)∗,∀t ∈ R≥0, E (σ, t) |= ϕ.

Transparency

Optimality

Example

l0 l1 l2

Σ \ {r}r ,

x := 0

Σ \ {r}

r , x ≥ 5,x := 0

r , x<5

σ = (3, r) · (1, r) · (1, a)

E (σ, 9) = (3, r) · (5, r) X

E (σ, 9) = (3, r) · (5, r) · (1, a) XE (σ, 9) = (3, r) · (6, r) · (1, a) X

16 / 33

Soundness

∀σ ∈ (R≥0 × Σ)∗,∀t ∈ R≥0, E (σ, t) |= ϕ.

Transparency

Optimality

Example

l0 l1 l2

Σ \ {r}r ,

x := 0

Σ \ {r}

r , x ≥ 5,x := 0

r , x<5

σ = (3, r) · (1, r) · (1, a)

E (σ, 9) = (3, r) · (5, r) X

E (σ, 9) = (3, r) · (5, r) · (1, a) XE (σ, 9) = (3, r) · (6, r) · (1, a) X

16 / 33

EM for a safety property

EM = 〈C ,C0, ΓEM , ↪→〉C = (R≥0 × Σ)∗ × R≥0 × R≥0 × B× Q is the set of configurations

Initial configuration is C0 = 〈ε, 0, 0, tt, q0〉 ∈ C ;

ΓEM =((R≥0 × Σ) ∪ {ε}

)× Op ×

((R≥0 × Σ) ∪ {ε}

)is the

input-operation-output alphabet, whereOp = {store(·),dump(·),del(·)} ;

↪→⊆ C × ΓEM × C

EM should fulfill the soundness, transparency and optimality conditions.

17 / 33

ΓEM =((R≥0 × Σ) ∪ {ε}

)× Op ×

((R≥0 × Σ) ∪ {ε}

)is the

↪→⊆ C × ΓEM × C

17 / 33

ΓEM =((R≥0 × Σ) ∪ {ε}

)× Op ×

((R≥0 × Σ) ∪ {ε}

)is the

↪→⊆ C × ΓEM × C

17 / 33

ΓEM =((R≥0 × Σ) ∪ {ε}

)× Op ×

((R≥0 × Σ) ∪ {ε}

)is the

↪→⊆ C × ΓEM × C

17 / 33

updates

Q × Σ× R≥0 → R≥0

(q, a, δ) 7→

min{δ′ ∈ R≥0 | ∃q1 ∈ FG , q(δ′,a)→ q1 ∧ δ′ ≥ δ}

∞ if ∀δ′ ∈ R≥0,∀q1 ∈ Q, (δ′ ≥ δ ∧ q(δ′,a)→ q1)⇒ q1 6∈ FG

where q = (l,X)

Can be realized using UPPAAL.

18 / 33

Operations

1 store : 〈σs , δ, d , tt, q〉(δ,a)/store(δ′,a)/ε

↪→ 〈σs · (δ′, a), 0, d , (δ′ 6=∞), q′〉with :

δ′ = updates(q, a, δ), =

minimal delay δ′ > δ s.t. (q(δ′,a)→ q′, q′ ∈ G)

q′ is defined as q(δ′,a)→ q′ if δ′ <∞ and q′ = q otherwise ;

2 dump :⟨(δ, a) · σs , s, δ, b, q

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, b, q〉 if δ 6=∞ ;

3 delay : 〈σs , s, d , b, q〉ε/del(δ)/ε↪→ 〈σs , s + δ, d + δ, b, q〉.

19 / 33

Operations

↪→ 〈σs · (δ′, a), 0, d , (δ′ 6=∞), q′〉with :

2 dump :⟨(δ, a) · σs , s, δ, b, q

19 / 33

Operations

↪→ 〈σs · (δ′, a), 0, d , (δ′ 6=∞), q′〉with :

2 dump :⟨(δ, a) · σs , s, δ, b, q

19 / 33

Example

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

Input σ = (1, a) · (3, r) · (1, r)ε/(ε, 0, 0, tt, < l0, 0 >)/(1, a) · (3, r) · (1, r)t = 0

ε/(ε, 1, 1, tt, < l0, 1 >)/(1, a) · (3, r) · (1, r)t = 1

del(1)

ε/((1, a), 0, 1, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 0, 0, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 3, 3, tt, < l0, 4 >)/(3, r) · (1, r)

del(3)

(1, a)/((3, r), 0, 3, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 0, 0, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 1, 1, tt, < l1, 1 >)/(1, r)

del(1)

(1, a) · (3, r)/((5, r), 0, 1, tt, < l1, 0 >)/ε

(1, a) · (3, r)/((5, r), 4, 5, tt, < l1, 4 >)/ε

del(4)

(1, a) · (3, r) · (5, r)/(ε, 4, 0, tt, < l1, 4 >)/ε

20 / 33

Example

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

ε/(ε, 1, 1, tt, < l0, 1 >)/(1, a) · (3, r) · (1, r)t = 1

del(1)

ε/((1, a), 0, 1, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 0, 0, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 3, 3, tt, < l0, 4 >)/(3, r) · (1, r)

del(3)

(1, a)/((3, r), 0, 3, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 0, 0, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 1, 1, tt, < l1, 1 >)/(1, r)

del(1)

(1, a) · (3, r)/((5, r), 0, 1, tt, < l1, 0 >)/ε

(1, a) · (3, r)/((5, r), 4, 5, tt, < l1, 4 >)/ε

del(4)

(1, a) · (3, r) · (5, r)/(ε, 4, 0, tt, < l1, 4 >)/ε

20 / 33

Example

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

ε/(ε, 1, 1, tt, < l0, 1 >)/(1, a) · (3, r) · (1, r)t = 1

del(1)

ε/((1, a), 0, 1, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 0, 0, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 3, 3, tt, < l0, 4 >)/(3, r) · (1, r)

del(3)

(1, a)/((3, r), 0, 3, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 0, 0, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 1, 1, tt, < l1, 1 >)/(1, r)

del(1)

(1, a) · (3, r)/((5, r), 0, 1, tt, < l1, 0 >)/ε

(1, a) · (3, r)/((5, r), 4, 5, tt, < l1, 4 >)/ε

del(4)

(1, a) · (3, r) · (5, r)/(ε, 4, 0, tt, < l1, 4 >)/ε

20 / 33

Example

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

ε/(ε, 1, 1, tt, < l0, 1 >)/(1, a) · (3, r) · (1, r)t = 1

del(1)

ε/((1, a), 0, 1, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 0, 0, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 3, 3, tt, < l0, 4 >)/(3, r) · (1, r)

del(3)

(1, a)/((3, r), 0, 3, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 0, 0, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 1, 1, tt, < l1, 1 >)/(1, r)

del(1)

(1, a) · (3, r)/((5, r), 0, 1, tt, < l1, 0 >)/ε

(1, a) · (3, r)/((5, r), 4, 5, tt, < l1, 4 >)/ε

del(4)

(1, a) · (3, r) · (5, r)/(ε, 4, 0, tt, < l1, 4 >)/ε

20 / 33

Example

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

ε/(ε, 1, 1, tt, < l0, 1 >)/(1, a) · (3, r) · (1, r)t = 1

del(1)

ε/((1, a), 0, 1, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 0, 0, tt, < l0, 1 >)/(3, r) · (1, r)t = 1

(1, a)/(ε, 3, 3, tt, < l0, 4 >)/(3, r) · (1, r)

del(3)

(1, a)/((3, r), 0, 3, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 0, 0, tt, < l1, 0 >)/(1, r)

(1, a) · (3, r)/(ε, 1, 1, tt, < l1, 1 >)/(1, r)

del(1)

(1, a) · (3, r)/((5, r), 0, 1, tt, < l1, 0 >)/ε

(1, a) · (3, r)/((5, r), 4, 5, tt, < l1, 4 >)/ε

del(4)

(1, a) · (3, r) · (5, r)/(ε, 4, 0, tt, < l1, 4 >)/ε

20 / 33

Enforcement function E (σ, t) associated to EM

E(σ, t) Op obs(σ, t)

ε |del(1)| ε

ε |store| (1, a)

(1, a) |dump| ε

ε |del(3)| ε

ε |store| (3, r)

(3, r) |dump| ε

ε |del(1)| ε

ε |store| (1, r)

ε |del(4)| ε

(5, r) |dump| ε

l0 l1 l2

Σ \ {r}r,

x := 0

Σ \ {r}

r, x ≥ 5,x := 0

r, x<5

E (σ, t) is sound, transparent and optimal.

21 / 33

Implementation

Enforcer

StoreProcess

DumpProcessMemory

σ, t E(σ, t)

Algorithm: StoreProcess(l,X)← (linit, [X ← 0])while tt do(δ, a)← await eventif (post(l,X, a, δ) /∈ G) thenδ′ ← update(l,X, a, δ)if δ′ =∞ thenterminate StoreProcess

end ifelseδ′ ← δ

end if(l,X) ← post(l,X, a, δ′)enqueue (δ′, a)

end while

Algorithm: DumpProcessd← 0while tt doawait (|σs| ≥ 1)(δ, a) ← dequeue (σs)wait (δ − d)dump (a)d← 0

end while

22 / 33

Outline

1 Introduction

23 / 33

Enforcement of a co-safety property

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Delays in the context of co-safety

Let σ, σ′ ∈ (R≥0 × Σ)∗ be two timed sequences. We note σ′ 4c σ for(ΠΣ(σ′) = ΠΣ(σ) ∧ ∀i ≤ |σ′|,del(σ(i)) ≤ del(σ′(i)))

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

Set of prefixes, set of sum of delays

γ(σ)∆= {σ′ 4c σ | σ′ |= ϕ} (sequences delaying σ and satisfying ϕ)

γt(σ)∆= {time(σ′) | σ′ ∈ γ(σ)} (set of sums of delays of sequences in γ(σ))

24 / 33

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

24 / 33

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

24 / 33

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

24 / 33

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

24 / 33

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

Enforcementfunction

σ, t E(σ, t) |= ϕ

Examples

(1, r) · (2, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

(1, r) · (10, g) · (5, a) 4c (1, r) · (8, g) · (5, a) X

24 / 33

Soundness

∀σ ∈ (R≥0 × Σ)∗, ∀t ∈ R≥0,E(σ, t) 6= ε⇒ (∃t′ ≥ t,E(σ, t′) |= ϕ).

Transparency

∀σ ∈ (R≥0 × Σ)∗, ∀t ∈ R≥0,E(σ, t) 6= ε⇒ (∃t′ ≥ t,E(σ, t′) 4c obs(σ, t))

Optimality

(Op1) |E(σ, t′)| = |Obs(σ, t)| where t′ = t + time((E(σ, t′)))

t is the first time instance at which a good location is reachable.(Op2) If the output is not ε, then

(1) The sum of delays is minimal for the minimal prefix which satisfies the property.

(2) For the rest of the sequence, the output is equal to the input.

Example

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

σ = (1, r) · (8, g)·(5, a) · (2, a)

E(σ, 2) = ε X

E(σ, 2) = (1, r) X

E(σ, 27) = (1, r) · (12, g)·(5, a) · (2, a) X

E(σ, 27) = (1, r) · (10, g)·(5, a) · (2, a) X

25 / 33

Soundness

Transparency

Optimality

Example

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

σ = (1, r) · (8, g)·(5, a) · (2, a)

E(σ, 2) = ε X

E(σ, 2) = (1, r) X

E(σ, 27) = (1, r) · (12, g)·(5, a) · (2, a) X

E(σ, 27) = (1, r) · (10, g)·(5, a) · (2, a) X

25 / 33

Soundness

Transparency

Optimality

Example

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

σ = (1, r) · (8, g)·(5, a) · (2, a)

E(σ, 2) = ε X

E(σ, 2) = (1, r) X

E(σ, 27) = (1, r) · (12, g)·(5, a) · (2, a) X

E(σ, 27) = (1, r) · (10, g)·(5, a) · (2, a) X

25 / 33

Soundness

Transparency

Optimality

Example

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

σ = (1, r) · (8, g)·(5, a) · (2, a)

E(σ, 2) = ε X

E(σ, 2) = (1, r) X

E(σ, 27) = (1, r) · (12, g)·(5, a) · (2, a) X

E(σ, 27) = (1, r) · (10, g)·(5, a) · (2, a) X

25 / 33

Soundness

Transparency

Optimality

Example

l3r , x := 0

Σ2 \ {r}

Σ2 \ {g},x < 10 ∨ x > 15

g ,10≤x ≤15

σ = (1, r) · (8, g)·(5, a) · (2, a)

E(σ, 2) = ε X

E(σ, 2) = (1, r) X

E(σ, 27) = (1, r) · (12, g)·(5, a) · (2, a) X

E(σ, 27) = (1, r) · (10, g)·(5, a) · (2, a) X

25 / 33

EM for a co-safety property

EM = 〈C ,C0, ΓEM , ↪→〉C = (R≥0 × Σ)∗ × R≥0 × R≥0 × B is the set of configurations, andthe initial configuration is C0 = 〈ε, 0, 0, tt〉 ∈ C ;

ΓEM =((R≥0 × Σ) ∪ {ε}

)× Op ×

((R≥0 × Σ) ∪ {ε}

)is the

input-operation-output alphabet, whereOp = {store-ϕ(·), store-ϕinit(·), store-ϕ(·),dump(·),delay(·)} ;

↪→⊆ C × ΓEM × C

26 / 33

updatec

updatec :(R≥0 × Σ)+ → R+≥0 × B such that for σ ∈ (R≥0 × Σ)+

updatec(σ)∆=

((δ1, . . . , δ|σ|), tt

|σ|∑i=1

δi = min{γt(σ)}, if γ(σ) 6= ∅((del(σ(1)), . . . ,del(σ(|σ|))), ff

), otherwise

27 / 33

Operations

1 store-ϕ : 〈σs , δ, d , ff〉(δ,a)/store−ϕ(δ,a)/ε

↪→ 〈σs · (δ, a), 0, d , ff〉 ifΠ2(updatec(σs · (δ, a))) = ff

2 store-ϕinit : 〈σs , δ, d , ff〉(δ,a)/store−ϕinit(δ′,a)/ε

↪→ 〈σ′s , 0, 0, tt〉 ifΠ2(updatec(σs · (δ, a))) = tt with

δ′ = Π1

(updatec(σs · (δ, a))

)σ′s =

⊙i∈[1,|σs |](Πi (δ

′), (σs(i))) · (δ′|σs |+1, a)

3 store-ϕ : 〈σs , δ, d , tt〉(δ,a)/store−ϕ(δ,a)/ε

↪→ 〈σs · (δ, a), 0, d , tt〉

4 dump :⟨(δ, a) · σs , s, δ, tt

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, tt〉

5 delay : 〈σs , s, d , b〉ε/delay(δ)/ε

↪→ 〈σs , s + δ, d + δ, b〉.

28 / 33

Operations

δ′ = Π1

)σ′s =

⊙i∈[1,|σs |](Πi (δ

′), (σs(i))) · (δ′|σs |+1, a)

↪→ 〈σs · (δ, a), 0, d , tt〉

4 dump :⟨(δ, a) · σs , s, δ, tt

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, tt〉

↪→ 〈σs , s + δ, d + δ, b〉.

28 / 33

Operations

δ′ = Π1

)σ′s =

⊙i∈[1,|σs |](Πi (δ

′), (σs(i))) · (δ′|σs |+1, a)

↪→ 〈σs · (δ, a), 0, d , tt〉

4 dump :⟨(δ, a) · σs , s, δ, tt

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, tt〉

↪→ 〈σs , s + δ, d + δ, b〉.

28 / 33

Operations

δ′ = Π1

)σ′s =

⊙i∈[1,|σs |](Πi (δ

′), (σs(i))) · (δ′|σs |+1, a)

↪→ 〈σs · (δ, a), 0, d , tt〉

4 dump :⟨(δ, a) · σs , s, δ, tt

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, tt〉

↪→ 〈σs , s + δ, d + δ, b〉.

28 / 33

Operations

δ′ = Π1

)σ′s =

⊙i∈[1,|σs |](Πi (δ

′), (σs(i))) · (δ′|σs |+1, a)

↪→ 〈σs · (δ, a), 0, d , tt〉

4 dump :⟨(δ, a) · σs , s, δ, tt

⟩ ε/dump(δ,a)/(δ,a)↪→ 〈σs , s, 0, tt〉

↪→ 〈σs , s + δ, d + δ, b〉.

28 / 33

Example

l3r, x := 0

Σ \ {r}

Σ \ {g};g, x < 10 | x > 15

g,10≤x ≤15

Input σ = (1, r) · (8, g) · (5, a)

ε/(ε, 0, 0, ff)/(1, r) · (8, g) · (5, a)t = 0

ε/(ε, 1, 1, ff)/(1, r) · (8, g) · (5, a)t = 1

delay(1)

ε/((1, r), 0, 1, ff)/(8, g) · (5, a)t = 1

store-¬ϕ

ε/((1, r), 8, 9, ff)/(8, g) · (5, a)t = 9

delay(8)

ε/((1, r) · (10, g), 0, 0, tt)/(5, a)

store-ϕinit

ε/((1, r) · (10, g), 1, 1, tt)/(5, a)

delay(1)

(1, r)/((10, g), 1, 0, tt)/(5, a)

t = 10

(1, r)/((10, g), 5, 4, tt)/(5, a)

(1, r)/((10, g) · (5, a), 0, 4, tt)/ε

store-ϕ

(1, r)/((10, g) · (5, a), 6, 10, tt)/ε

delay(6)

(1, r) · (10, g)/((5, a), 6, 0, tt)/ε

(1, r) · (10, g)/((5, a), 11, 5, tt)/ε

delay(5)

(1, r) · (10, g) · (5, a)/(ε, 11, 0, tt)/ε

delay(4)

t = 14

t = 20

t = 25

29 / 33

Example

l3r, x := 0

Σ \ {r}

Σ \ {g};g, x < 10 | x > 15

g,10≤x ≤15

Input σ = (1, r) · (8, g) · (5, a)

ε/(ε, 0, 0, ff)/(1, r) · (8, g) · (5, a)t = 0

ε/(ε, 1, 1, ff)/(1, r) · (8, g) · (5, a)t = 1

delay(1)

ε/((1, r), 0, 1, ff)/(8, g) · (5, a)t = 1

store-¬ϕ

ε/((1, r), 8, 9, ff)/(8, g) · (5, a)t = 9

delay(8)

ε/((1, r) · (10, g), 0, 0, tt)/(5, a)

store-ϕinit

ε/((1, r) · (10, g), 1, 1, tt)/(5, a)

delay(1)

(1, r)/((10, g), 1, 0, tt)/(5, a)

t = 10

(1, r)/((10, g), 5, 4, tt)/(5, a)

(1, r)/((10, g) · (5, a), 0, 4, tt)/ε

store-ϕ

(1, r)/((10, g) · (5, a), 6, 10, tt)/ε

delay(6)

(1, r) · (10, g)/((5, a), 6, 0, tt)/ε

(1, r) · (10, g)/((5, a), 11, 5, tt)/ε

delay(5)

(1, r) · (10, g) · (5, a)/(ε, 11, 0, tt)/ε

delay(4)

t = 14

t = 20

t = 25

29 / 33

Example

l3r, x := 0

Σ \ {r}

Σ \ {g};g, x < 10 | x > 15

g,10≤x ≤15

Input σ = (1, r) · (8, g) · (5, a)

ε/(ε, 0, 0, ff)/(1, r) · (8, g) · (5, a)t = 0

ε/(ε, 1, 1, ff)/(1, r) · (8, g) · (5, a)t = 1

delay(1)

ε/((1, r), 0, 1, ff)/(8, g) · (5, a)t = 1

store-¬ϕ

ε/((1, r), 8, 9, ff)/(8, g) · (5, a)t = 9

delay(8)

ε/((1, r) · (10, g), 0, 0, tt)/(5, a)

store-ϕinit

ε/((1, r) · (10, g), 1, 1, tt)/(5, a)

delay(1)

(1, r)/((10, g), 1, 0, tt)/(5, a)

t = 10

(1, r)/((10, g), 5, 4, tt)/(5, a)

(1, r)/((10, g) · (5, a), 0, 4, tt)/ε

store-ϕ

(1, r)/((10, g) · (5, a), 6, 10, tt)/ε

delay(6)

(1, r) · (10, g)/((5, a), 6, 0, tt)/ε

(1, r) · (10, g)/((5, a), 11, 5, tt)/ε

delay(5)

(1, r) · (10, g) · (5, a)/(ε, 11, 0, tt)/ε

delay(4)

t = 14

t = 20

t = 25

29 / 33

Example

l3r, x := 0

Σ \ {r}

Σ \ {g};g, x < 10 | x > 15

g,10≤x ≤15

Input σ = (1, r) · (8, g) · (5, a)

ε/(ε, 0, 0, ff)/(1, r) · (8, g) · (5, a)t = 0

ε/(ε, 1, 1, ff)/(1, r) · (8, g) · (5, a)t = 1

delay(1)

ε/((1, r), 0, 1, ff)/(8, g) · (5, a)t = 1

store-¬ϕ

ε/((1, r), 8, 9, ff)/(8, g) · (5, a)t = 9

delay(8)

ε/((1, r) · (10, g), 0, 0, tt)/(5, a)

store-ϕinit

ε/((1, r) · (10, g), 1, 1, tt)/(5, a)

delay(1)

(1, r)/((10, g), 1, 0, tt)/(5, a)

t = 10

(1, r)/((10, g), 5, 4, tt)/(5, a)

(1, r)/((10, g) · (5, a), 0, 4, tt)/ε

store-ϕ

(1, r)/((10, g) · (5, a), 6, 10, tt)/ε

delay(6)

(1, r) · (10, g)/((5, a), 6, 0, tt)/ε

(1, r) · (10, g)/((5, a), 11, 5, tt)/ε

delay(5)

(1, r) · (10, g) · (5, a)/(ε, 11, 0, tt)/ε

delay(4)

t = 14

t = 20

t = 25

29 / 33

Implementation

Enforcer

StoreProcess

DumpProcessMemory

σ, t E(σ, t)

Algorithm: StoreProcessgoalReached ← ff

while tt do(δ, a)← await eventenqueue(δ, a)if goalReached 6= tt then

(newDelays, R)← updatec(σs · (δ, a))if R == tt then

update delaysgoalReached ← tt

notify startDumpend if

end ifend while

Algorithm: DumpProcessawait startDumpd← 0while tt doawait (|σs| ≥ 1)(δ, a) ← dequeue (σs)wait (δ − d)dump (a)d← 0

end while

30 / 33

Outline

1 Introduction

31 / 33

Related Work

Enforce safety and information flow properties using process algebraMatteucci.

Targets discrete-time properties.Description is very abstract.No details about implementation proposed.

Other efforts mainly to verify timed properties at runtime.

Runtime verification of LTL and TLTL Bauer et al.The Analog Monitoring Tool.(monitoring specifications overcontinuous signals) Nickovic et al.LARVA (Safe monitoring of real-time Java programs)Colombo et al.

Automatic error handling, when violation of property detected by themonitor (changing system state).

32 / 33

Related Work

Enforce safety and information flow properties using process algebraMatteucci.

Targets discrete-time properties.Description is very abstract.No details about implementation proposed.

Other efforts mainly to verify timed properties at runtime.

Runtime verification of LTL and TLTL Bauer et al.The Analog Monitoring Tool.(monitoring specifications overcontinuous signals) Nickovic et al.LARVA (Safe monitoring of real-time Java programs)Colombo et al.

Automatic error handling, when violation of property detected by themonitor (changing system state).

32 / 33

Conclusion

Runtime enforcement for timed properties.

Enforcer adds additional delay between input actions in order tosatisfy the property.Additional constraints to ensure choosing ”best” delay betweenactions.

Focused on safety/co-safety properties.

Algorithms to implement the enforcers.Prototypes developed using Python and UPPAAL.Simulation, performance analysis demonstrating feasibility andscalability.

Ongoing/ future work

Enforcing more expressive properties.Improve implementation.Test on case studies (analysis, different architectures).New transparency conditions.

33 / 33

Conclusion

33 / 33

Conclusion

33 / 33

runtime enforcement of timed properties -...

Documents