system testing of timing requirements based on use cases and timed automata

.lusoftware verification & validationVVS

System Testing of Timing Requirements based on Use Cases and Timed Automata

Chunhui Wang, Fabrizio Pastore, Lionel Briand

ICST’17, Tokyo, March 14th, 2017

University of Luxembourg, Luxembourg

{chunhui.wang, fabrizio.pastore, lionel.briand}@uni.lu

Timing Requirements are crucial forsafety-critical systems

2

Timing Requirements: Automotive Domain

Airbag should be fully deployed within 50ms whena crash event is detected.

3

Timing Requirements: Automotive Domain

Car control system should signal overheating to driver whentemperature exceeds a safety threshold for a given time.

Most of safety criticalcomponents are not reliablewhen car is overheated(e.g. Airbag Control).

4

Timing Requirements: overheating

•A Temperature Error can be qualified (signaled)if overheating is detected for at least 3100ms(avoid signal toggling)

•A Temperature Error should be qualified nolater than 4800ms after overheating isdetected

5

Objectives

• Automatically generate test cases to stress timingrequirements with minimal modelling overhead

• Identification of input sequences that increase the likelihood that the system will break timing constraints

6

Context• Functional Requirements are captured through an

analyzable form of use case specifications

• Timing Requirements are modeled by UML statecharts or Timed Automata

• Functional test cases automatically generated from use case specifications with UMTG

Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolset to automatically generate system test cases from use case specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 942-945). ACM.

BASIC FLOW1. The system REQUESTS temperature FROM temperature sensor.2. The system VALIDATE THAT the temperature is valid.3. The system SEND occupant status TO airbag control unit.……

ALTERNATIVE FLOWRFS 21. The system set temperature error as detected.

7

THE ACTOR SENDTHE SYSTEM VALITHE SYSTEM DISTHE ACTOR SEND



Use Cases

Domain Model

Errors.size()== 0Status != null

t > 0 && t < 50

OCL constraints

Test Cases

Test Scenarios

UMTG [ISSTA’15]

8

Context• Functional Requirements are captured through an

analyzable form of use cases specifications

• Timing Requirements are modeled by UML statecharts or Timed Automata

• Functional test cases automatically generated from use case specifications with UMTG

Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolset to automatically generate system test cases from use case specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 942-945). ACM. 9

Strategy• Use Timed Automata to model Timing Constraints

• Automata are high-level, missing information about concrete test inputs that may lead to state transitions

• Rely upon Use Case Scenarios to identify the test inputs that trigger state transitions

• Rely upon edge coverage criteria to identify test sequences

• Integrate an ad-hoc metaheuristic search approach to build a testsuite that effectively stresses timing requirements by maximizingtest case diversity

10

Test Generation combining Timed Automata and Use Case

Specifications (TAUC)

11

Msg1?

Environment Automata

scenario1?[Error == true]

[Error == false]scenario2?

TimingReq Automata

ANALYSIS AND DESIGN 1

12

Modelling• Automata that capture timing properties of the functionality

under test (timing requirement automata)

• E.g. qualification of temperature error (overheating)

• Automata that model the environment:

• Capture frequency of arrival of interrupts / messages / inputs from sensors

13

Example: TimingReqAutomata• Automata that capture timing properties of the functionality

under test, e.g. qualification of of temperature error

14

x >= 3100

unDetected?isDetected = false, x = 0

detected?

isDetected = true, x = 0

isDetected = false, x = 0


x >= 6100

NotDetectedNotQualified DetectedNotQualified

DetectedQualifiedNotDetectedQualified

x <= 4800

x <= 8100

Invariant

Guard

Updates

unDetected?

detected?

isQualified = falseisQualified = trueA Temperature Error can be qualified if overheating is detected for at

least 3100ms.A Temperature Error should be qualified no later than 4800ms afteroverheating is detected

• Automata that model the environment:

• Capture frequency of arrival of interrupts / messages / inputs from sensors

15x = 0

x > 41DMAInterrupt!

x = 0

x >= 50TimerInterrupt!

x <= 50

x = 0

x > 40OccupancyInfo!

x > 40CarInfo!

x > 40AirbagInfo!

Example: EnvironmentAutomata

Msg1?




TimingReq Automata

scenario1 scenario2 scenario3


IDENTIFY FUNCTIONAL SCENARIOS 2

IDENTIFYDEPENDENCIES

3

Automated by UMTG

• Transitions are triggered by scenario outputs

• Scenarios can be executed only in specific states

• Useful to translate abstract test activities into concrete test inputs

16

17

Coverage-based Test Generation

x >= 3100


detected?




x >= 6100



x <= 4800

x <= 8100

unDetected?

detected?

isQualified = falseisQualified = true

18

Coverage-based Test Generation

x >= 3100


detected?




x >= 6100



x <= 4800

x <= 8100

unDetected?

detected?


Test CaseDetected

Wait 4800ms

unDetected

Wait 8100ms

How to translate“error detected”

into a concrete test input?

By Reusing InformationAvailable in

Use Case Specifications

19

How to translate“error detected”

to a concrete test input?

20

TemperatureSensor.allInstances()->forAll(t | t.temperature > 0 and t.temperature < 45)

UMTGgenerates the input

(e.g. temperature = 50)that leads to the detection of a

temperature error

21

A scenario brings the system into a specific state(i.e. assigns values to variables)

22

Errors.allInstances()->forAll(e | e.isQualified = false)

Precondition to execute the scenario: At least one error is in the states with the state variable isQualified = true

23

x >= 3100

unDetected?

detected?

x >= 6100

NotDetectedNotQualified DetectedNotQualifiedx <= 4800

x <= 8100

unDetected?

detected?


NotDetectedQualified DetectedQualified

Scenario

isQualified = true

24

• A scenario brings the system into a specific state (i.e. assign values to variables)

• A scenario can be executed only if the system has reached a specific state (i.e. requires that state variables have specific values)

Msg1?




TimingReq Automata





3

25



Msg1?

Timeliness Test Model


C

scenario1!

Error := true

Scenario Automata

TimingReq Automata

4800ms

Scenario1

ScenarioX

8100ms

Message1…

GENERATE TIMELINESS TEST MODEL

4 GENERATE EXECUTION SCENARIOS

5

TAUC

26

4800ms

Scenario1

ScenarioX

8100ms

Message1…

4800ms

Scenario1

ScenarioY

8100ms

Message2…

ScenarioX

ScenarioZ

8100ms

Message1…

Message3

Generated Test Suites

…4800ms

Scenario1

ScenarioX

8100ms

Message1…

Maximize test case diversity:• Execute more, diverse paths, including the same relevant edges• Execute paths with a diversity of input, interrupts, and messages• Increase the chances of timeliness failure detection

TAUC

27

Case Study• BodySense, embedded system for detecting occupancy

status in a car

• Evaluation:

• Effectiveness in terms of fault detection ratecompared to current practice at IEE

28

Effectiveness Evaluation• Derived 323 faulty version of BodySense by means of model

mutation• Each faulty version of BodySense models was generated by

executing a single mutation operatoron the original model

• Compare the Fault Detection Rate with the test suitesgenerated by• Random testing• Expertise-based manual testing

29

30

Fault Detection Rate

25 50 75 100 122TAUC 85% 88% 91% 91% 91%

Random 7% 12% 22% 30% 40%

Manual 60%

31

Evaluation of TAUC Test Suite



Original model

Mutate





…



TAUC



Msg1?


C

scenario1!

Error := true

Scenario Automata

TimingReq Automata

4800ms

Scenario1

ScenarioX

8100ms

Message1…

4800ms

Scenario1

ScenarioY

8100ms

Message2…

ScenarioX

ScenarioZ

8100ms

Message1…

Message3

Test Suites

…

GENERATE TIMELINESS TEST MODEL

4

GENERATE EXECUTION SCENARIOS

5


3



Msg1?




TimingReq Automata


Timing Requirements are crucial forsafety-critical systems

Fault Detection Rate

25 50 75 100 122

TAUC 85% 88% 91% 91% 91%

Random 7% 12% 22% 30% 40%

Manual 60%