efficient decentralized monitoring of safety in distributed systems

Efficient Decentralized Monitoring of Safety in Distributed Systems

Koushik Sen

Abhay Vardhan

Gul Agha

Grigore Rosu

University of Illinois at Urbana-Champaign, USA

04/19/232/20

Koushik Sen, Abhay Vardhan, Gul Agha, Grigore Rosu, U. of Illinois at Urbana-Champaign

Software Reliability

Software Validation Rigorous and Complete Methods

Model Checking Theorem Proving

– Infeasible for large-scale open distributed systems– Non-determinism and Asynchrony

Testing Widely used Ad-Hoc Good Test Coverage Required

Runtime Monitoring Adds rigor to Testing

04/19/233/20


Centralized Monitoring Approach

Monitoring – Use Formal Methods in Testing Synthesize light-weight Monitors from Specification

Automata, Rewriting-based Monitors, State machines Instrument code to insert monitors Execute instrumented code

Distributed System Monitoring Global state is distributed For every state update send state to a central monitor Central monitor assembles them to form consistent

execution traces (Vector Clocks) Sequence of global states

Monitor execution traces

04/19/234/20


An Example

Mobile node a requests certain value from node b

b computes the value and sends it to a

Property: no node receives a value from another node to which it had not sent a request

04/19/235/20


valRcv → (valComputed valReq)

Centralized Monitoring Example

valRcv → (valComputed valReq)

“If a receives a value from b then b calculated the value after receiving request from a”

valReq

valComputed

valRcva

b

valReqvalReq valComputed valReq(valComputed valReq)Monitor

04/19/236/20


Decentralized Monitoring Approach

“If a receives a value from b then b calculated the value after receiving request from a”

valRcv → @b((valComputed @a(valReq)))

valReq

valComputed

valRcva

b

valReqvalRcv → @b((valComputed @a(valReq)))

(valComputed @a(valReq))@a(valReq)valComputed @a(valReq)

04/19/237/20


Past time Distributed Temporal Logic (pt-DTL)

Past Time Linear Temporal Logic [Pnueli]

Extended with a Operator from epistemic logic (@) [Aumann76][Meenakshi et al. 00]

Properties with respect to a process, say p

Interpreted over sequence of knowledge that p has about global state

04/19/238/20


Remote Formulas in pt-DTL

@a F at process b

@ makes remote formula F at process a local to process b

“Alarm at process b implies that there was a fire at a”

alarm → @afire

a formula with respect to process b

04/19/239/20


Remote Expressions in pt-DTL

Remote expressions – arbitrary expressions related to the state of a remote process

Propositions constructed from remote and local expressions

“If my alarm is set then eventually in past difference between my temperature and temperature at process b exceeded the allowed value”

alarm → ((myTemp - @btemp) > allowed)

04/19/2310/20


Safety in Airplane Landing

“ If my airplane is landing then the runway that the airport has allocated matches the one that I am planning to use”

landing → (runway = @airportallocRunway)

04/19/2311/20


Leader Election Example

“If a leader is elected then if the current process is a leader then, at its knowledge, none of the other processes is a leader”

elected → (state=leader → /\i≠j(@j(state ≠ leader)))

04/19/2313/20


Interpretation of @jEj at process i

p3

p1

p2

m4

m3

m2

m1

x=7 x=9

@ 1(x=9)

04/19/2314/20


Monitoring Algorithm

Requirements Should be fast so that online monitoring is

possible

Little memory overhead

Additional messages sent should be minimal; ideally zero

04/19/2315/20


KnowledgeVector

Let KV be a vector

one entry for each process appearing in formula

KV[j] denotes entry for process j KV[j].seq is the sequence number of last

event seen at process j

KV[j].values stores values of j-expressions and j-formulae

04/19/2316/20


Monitoring using KnowledgeVector

Maintain KnowledgeVector about global state at each process

Attach KnowledgeVector with outgoing messages

Update KnowledgeVector with incoming messages

At each process monitor local KnowledgeVector

04/19/2317/20


KnowledgeVector Algorithm

[internal event]: (at process i)

store eval(Ei,si) and eval(Fi,si) for each @iEi and @iFi in KVi[i].values

[send m]:

KVi[i].seq Ã KVi[i].seq + 1. Send KVi with m as KVm

[receive m]:

for all j, if KVm[j].seq > KVi[j].seq then

KVi[j].seq Ã KVm[j].seq

KVi[j].values Ã KVm[j].value

store eval(Ei,si) and eval(Fi,si) for each @iEi and @iFi in KVi[i].values

04/19/2318/20


Example

p3

p2

p1X=5 X=9 X=6

Y=7 Y=3

0

5

0

5

0

5

0

9

1

9

1

6

2

6

2

6

2

6

2

6

2

6

2

6

violation

¡(Y ¸ @1X) at p2

KV[1].seq

KV[1].values

04/19/2319/20


DIANA Architecture

pt-DTL

Monitor

04/19/2320/20


Conclusion

pt-DTL can express interesting and useful safety properties of distributed systems

Decentralized Technique to effectively verify Distributed Systems at runtime

No extra message over-head for monitoring

KnowledgeVector as monitors

efficient decentralized monitoring of safety in distributed systems

Documents

process bkoushik sen

testingkoushik sen

avalreqkoushik sen

usakoushik sen

requestkoushik sen

b alarm

execution traceskoushik

global statekoushik