the state of hybrid model-based testing
DESCRIPTION
The State of Hybrid Model-Based Testing. Michiel van Osch 21-06-2005. Index. Introduction Testing a Water Tank Controller Hybrid Model-Based Testing Conformance Relation Test-case generation Work in Progress Questions. ?“stop traffic”. ?“stop traffic”. !“red”. !“red”. pass. - PowerPoint PPT PresentationTRANSCRIPT
The State of Hybrid Model-Based Testing
Michiel van Osch
21-06-2005
2 2
Index
• Introduction• Testing a Water Tank Controller• Hybrid Model-Based Testing
– Conformance Relation– Test-case generation
• Work in Progress• Questions
3 3
Model-Based Testing
DiscreteModel
Test-tool DiscreteSystem
pass
!“red”!“red”
?“stop traffic” ?“stop traffic”
4 4
Hybrid Systems
“brake”
Hydrolicpressure
“1”, “2”, “1”
Coffee withMilk and Sugar
Room Temperature
Desired Temperature: 25o
5 5
Hybrid Model-based Testing
Hybrid-ModelTest-tool
Hybrid System
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
pass
Lenstemperature
Lenstemperature
!”openCoolingValve”
!”openCoolingValve”
6 6
Challenges
• Theory: – Develop Conformance Concepts– Develop Test-case Generation Algorithms– Prove Correctness and Applicability of Theory
• Practice: – Implement Tooling– Exactly Model Relevant Aspects– Connect Models, Tools and Systems
7 7
Index
• Introduction• Testing a Water Tank Controller• Hybrid Model-Based Testing
– Conformance Relation– Test-case generation
• Work in Progress• Questions
8 8
The water Tank Controller
Tank Controller
• Discrete input: Set point
• Discrete output: communication
• Continuous input: Temperature
(from sensor)
• Continuous output: Heat
(from Heater)
WaterTank
Tank Controller
TemperatureSensor
Heater
set point
messages
9 9
Specification (1)
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Hybrid I/O automaton
Tank Controller
• Discrete input: Set_setpoint(sp)
• Discrete output: Print(“warning”)
• Continuous input: ST
• Continuous output: H
10 10
Specification (2)
Heater OFF Heater ON
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
H
H
ST
ST
input output input output
11 11
Specification Test-tool Implementation
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Test Setup
Tank Controller
TemperatureSensor Heater
Set_Setpoint(sp) Set-point
ST Temp. Sens.
Print(..) Message
HeatH
12 12
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
Specification
Test-tool
Implementation
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Testing...
Tank Controller
TemperatureSensor Heater
ST
H Heat
Temp. Sens.
Specification Implementation
13 13
Testing…
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
Test-tool
Set_setpoint(SP) Set-point
Specification
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Implementation
Tank Controller
TemperatureSensor Heater
ST
H Heat
Temp. Sens.
Specification Implementation
14 14
ST
H Heat
Temp. Sens.
Testing…
Test-tool
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
Specification
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Implementation
Tank Controller
TemperatureSensor Heater
Specification Implementation
15 15
Testing
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
Test-toolPrint(“warning”) “error”
Specification
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Implementation
Tank Controller
TemperatureSensor Heater
ST
H Heat
Temp. Sens.
Specification Implementation
fail
16 16
Testing…
Test-tool
Specification
S
Set_setpoint(sp)
T
Tank Controller
H
Print(“Warning”)
Implementation
Tank Controller
TemperatureSensor Heater
fail
Heater OFF Heater ON
SP £ S
SP > ST
T
?Set_setpoint(sp)
!Print(“warning”)
?Set_setpoint(sp)
ST
H Heat
Temp. Sens.
Specification Implementation
17 17
Index
• Introduction• Testing a Water Tank Controller• Hybrid Model-Based Testing
– Conformance Relation– Test-case generation
• Work in Progress• Questions
18 18
Conformance
A hybrid implementation is input-output conform a hybrid specification if for all observable traces of the specification:
The set of possible output actions and trajectories,
after the execution of a trace on the implementation,
is a subset of
the set of possible output actions or trajectories,
after the execution of a trace on the specification.
19 19
Some Decisions
• Defined on Hybrid I/O Automata– Input Action Enabled– Input Trajectory Enabled– Non-zeno
• State Based: possible output actions and trajectories in a certain state
• Input and output trajectories are treated together• There is no quiescence observation (as of yet)
20 20
Conformance relation
We Formally Define:
• A hybrid trace = 0 a1 1 a2 … as a sequence of discrete actions and trajectories of continuous variables (alternating)
• A after : the set of states in which A can be after executing trace
• out(q): the set of output actions or trajectories possible in state q• Out(Q): the set of output actions or trajectories possible in the set
of states Q
Then implementation HIOA i is hioco specification HIOA s if:
i hioco s if traces(s): out(i after ) out(s after )
21 21
Examples
!a
!b
!a
!b
!cCorrect!
implementationspecification
implementation
specification
Correct!
22 22
Examples
!a
!b
!a
!b
Incorrect!
implementationspecification
!c
implementation
q
specification
Incorrect! q
23 23
Test-Case generation & Execution
• Input actions: Select and stimulate• Output actions: Observe and validate• Trajectories:
1. Select a time up to which to apply input 2. Select an input trajectory to use for stimulation3. Stimulate and observe output variables :
– Until the end of the trajectory, without observing a mismatch between observed and specified output trajectories, in this case continue
– Until an output action is observed, in this case validate the output action and if correct, then continue
– Until a mismatch between observed and specified output variables occurs
• As long as verdict “fail” is given, the verdict “pass” can be given
24 24
Test-case generation
fail
failpass
pass fail
?Set_setpoint(SP)
!Print(“warning”) !Print(“error”)
{SP > ST}
25 25
Index
• Introduction• Testing a Water Tank Controller• Hybrid Model-Based Testing
– Conformance Relation– Test-case generation
• Work in Progress• Questions
26 26
Work in Progress (1)
• Does a continuous variable always have a value?
• State Changes During a Trajectory
“1”,”2”
Coffee
temperature
When no coffee comes
out of the machine, temperature
Is not continuously 0!
While applying an input trajectory
and observing the output,
States change
q q’
27 27
Work in Progress (2)
• Quiescence– Is quiescence a property on the discrete actions of the system
or also on continuous behavior of the system?
• Notation
What does this mean?
28 28
Current & Future Work
• Further Develop these Concepts (Formalization, Theorems, Proofs)
• Make Concepts Workable in Practice– Sampling of continuous behavior– Selecting Realistic Input Trajectories– Deal with Uncountable Many Possible Input and Output
Trajectories and States– Deal with Inaccuracy of Observed Output
• Implement Proof of Concept
29 29
Questions