![Page 1: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/1.jpg)
Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems
Armin Wellig2 David Lawrence1 Didier Buchs1
1 University of Geneva
2 Honeywell International Sarl
6th International Workshop on Software
Engineering for Resilient Systems
16/10/2014
1
![Page 2: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/2.jpg)
System
… t += 1 print()
Introduction
2
• Correct internal state OR false positive?
• What about resilient system correctness?
• Ideas/Solutions => Improve observability
• System correctness
![Page 3: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/3.jpg)
System observation
• Automaton specification
o T(spec) are words ∈ (Σin x Σout)*
• Satisfaction relation
o For any program p based on a given specification spec
3
![Page 4: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/4.jpg)
Limited observability
• What if the developer made a fault with the
counter modulo?
4
![Page 5: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/5.jpg)
Improve observability
• Adding new observation points obs to a program p
• For any program p, obs are correct observers iff
5
![Page 6: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/6.jpg)
Observability: example
• Obs1: each increment of cnt
• Obs2: when “cnt mod 5 = 0”
6
![Page 7: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/7.jpg)
Observers expressiveness
7
• For a given program p, the observers obs
expressiveness can be defined as follow
![Page 8: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/8.jpg)
Metric: distance
8
• A distance between instrumented program traces
and specification traces can be stated
• What about the observers relevance?
![Page 9: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/9.jpg)
Metric: quality/relevance
9
H0
…
Hi
Hj
H
…
T0
…
…
T
Tj
Ti
H0
…
Hi
Hj
H
…
T0’
…
…
T’
Tj’
Ti’
P P + obs
T >> T’
![Page 10: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/10.jpg)
Instrumentation process
10
Spec. Model Develop
Compare
Program Traces
Impl.
Embed
Obs.
Observe
Instr. Impl.
Extract
Specification Traces
![Page 11: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/11.jpg)
Instrumentation: example void Thermostat (int* t) { if (t > 21) { *t += -0.1 * t; // Increase temperature } else if (t < 19) { t += 5 - 0.1 * t; // Decrease temperature } }
11
void Thermostat (int t) {
if (t > 21) {
t += -0.1 * t; // Increase temperature
} else if (t < 19) {
t += 5 – 0.1 * t; // Decrease temperature
}
return t;
}
t = 21 t=18.9 t=21.29 t=19.161
![Page 12: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/12.jpg)
Instrumentation: example
12
t = 21 t=18.9 t < 19 t=21.29 t=19.161
void Thermostat (int t) {
if (t > 21) {
t += -0.1 * t; // Increase temperature
} else if_instr (t < 19) {
t += 5 – 0.1 * t; // Decrease temperature
}
return t;
}
![Page 13: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/13.jpg)
Instrumentation: example
13
void Thermostat (int t) {
if_instr (t > 21) {
t += -0.1 * t; // Increase temperature
} else if_instr (t < 19) {
t += 5 – 0.1 * t; // Decrease temperature
}
return t;
}
t = 21 t=18.9 t < 19 t=21.29 t > 21 t=19.161
![Page 14: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/14.jpg)
Resilient system
14
Compensation
Normal behavior
Degradation
![Page 15: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/15.jpg)
Instrumenting resilient systems
• Resilient system designed with defined FSMs
• System crashes considered as degradation
• Instrumentation requirements
• LTL used to discuss properties
15
![Page 16: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/16.jpg)
Instrumenting resilient systems
• Extended system
o Fsys’’ = Fsys + feedback + recovery + obssys
• Compensation
o Fcomp’ = Fcomp + obscomp
• Degradation
o Fdeg’ = Fdeg + obsdeg
16
![Page 17: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/17.jpg)
System with
feedback and
recovery
System with
feedback
High level picture
17
Degradation Compensation
Environment
System
feedback
recovery
crash
![Page 18: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/18.jpg)
Preparing the system
• Preliminary preparation
o Add atomic propositions to distinguish states
18
s1
s2
ν(s2) = 2
ν(s1) = 1
![Page 19: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/19.jpg)
Extending the system
• 1st extension Fsys’ (Feedback)
1) Add crash state
2) Add feedback state
3) Add arcs from every state to crash state
• crash as arc inscription
• Unique state ID stored in crashed_state variable
4) Add arc between crash and feedback
• feedback as arc inscription
5) Add atomic propositions
19
![Page 20: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/20.jpg)
Extending the system
20
crashed_state := 1
crashed_state := 2
feedback
s1
ν(s1) = 1
s2
ν(s2) = 2
crash
ν(crash) = syscrash
feedback
ν(feedback) = feedback
![Page 21: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/21.jpg)
Extending the system
• 2nd extension Fsys’’ (Recovery handling)
1. Add arcs from feedback state to every system state
• recovery as arc inscription
2. Guards
• Avoid non determinism
• Recovery to last consistent state using crashed_state
21
![Page 22: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/22.jpg)
Extending the system
22
[crashed_state = 2]
[crashed_state = 1]
recovery
recovery
…
…
…
s1
ν(s1) = 1
s2
ν(s2) = 2
crash
ν(crash) = syscrash
feedback
ν(feedback) = feedback
![Page 23: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/23.jpg)
Compensation and degradation
23
crash feedback
recovery recovery
Compensation Degradation
idle detection normal inject
ν(idle) = …
ν(detection) = …
ν(normal) = …
ν(inject) = …
![Page 24: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/24.jpg)
System composition
• Complete resilient system
o Fres = Fsys’’ || Fcomp’ || Fdeg’
• Composition
1. Add arc looping on each state with εas inscription
2. Synchronized product of the two FSM
24
![Page 25: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/25.jpg)
System composition
• FSMcompo = FSM1 || FSM2
o Union of the alphabets
o Cartesian product of states
o Cartesian product of initial states
o Union of variables
o Union of atomic propositions
25
![Page 26: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/26.jpg)
System composition
26
s1 s2
s1’ s2’
a
a
s1, s1’
s2, s2’
a
[x=1] ∧ [y=1]
[y=1]
[x=1]
z := 1
w := 1
z := 1, w := 1
![Page 27: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/27.jpg)
System composition
27
s1 s2
s1’ s2’
s1, s1’
s1, s2’
a
s2, s1’
b
[x=1]
[y=1]
z := 1
w := 1
[y=1]
[x=1]
a
b
z := 1
w := 1
![Page 28: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/28.jpg)
System composition
28
s1, normal, idle
s2, normal, idle
crash, inject, idle
ν(s1, normal, idle) = 1
…
ν(s2, normal, idle) = 2
crash
crash
crashed_state := 1
crashed_state := 2
ν(crash, inject, idle) = ν(crash) ∪ ν(inject) ∪ ν(idle)
![Page 29: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/29.jpg)
Resiliency properties • System’s resiliency
• Improvements
29
![Page 30: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/30.jpg)
Model checking
30
• StrataGEM [López et al. 2014]
• Symbolic model-checker
• Using concepts of Term Rewriting
• Using Decision Diagrams for data representation
![Page 31: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/31.jpg)
Model checking
31
• Usage
• Resilient system translated as a transition system
• Strategies/rewriting rules defined independently
for each components
• State space computed
![Page 32: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/32.jpg)
Conclusion
• Theoretical basis on instrumentation
• Insights on resilient systems instrumentation
• Methodology to extend a system with resilience
o Even though the model is simple
• Temporal properties enunciated
o Mechanisms and overall resiliency
32
![Page 33: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/33.jpg)
Future works
• Level resiliency
o More complete/complex model [Trivedi et al. 2009]
• Model checking
o LTL with StrataGEM when available
o Other model checkers
• Tests generation
o Model based tests generation [Fraser et al. 2009]
o Timing insights [Braberman et al. 1997]
33
![Page 34: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/34.jpg)
Questions ?
34
Thank you
![Page 35: SERENE 2014 Workshop: Paper "Using Instrumentation for Quality Assessment of Resilient Software in Embedded Systems"](https://reader033.vdocuments.site/reader033/viewer/2022042700/5599c31d1a28abf3168b46e6/html5/thumbnails/35.jpg)
References
o Edmundo López Bóbeda, Maximilien Colange, and Didier Buchs. Stratagem: A generic petri net verification framework. In Gianfranco Ciardo and Ekkart Kindler, editors, Petri Nets, volume 8489 of Lecture Notes in Computer Science, pages 364– 373. Springer, 2014.
o Kishor S. Trivedi, Dong Seong Kim, and Rahul Ghosh. 2009. Resilience in computer systems and networks. In Proceedings of the 2009 International Conference on Computer-Aided Design (ICCAD '09). ACM, New York, NY, USA, 74-77.
o Fraser, G., Wotawa, F., & Ammann, P. E. (2009). Testing with model checkers: a survey. Software Testing, Verification and Reliability, 19(3), 215-261.
o Victor Braberman, Miguel Felder, and Martina Marr. Testing timing behavior of real-time software. In International Software Quality Week, 1997.
35