testing the untestable: model testing of complex software-intensive systems

Testing the Untestable: Model Testing

of Complex Software-Intensive Systems"

Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli

Untestable Systems

2

“systems for which testing is highly expensive "or impossible to fully automate”

Untestable Systems"Test Generation

3

large, complex input spaces

Untestable Systems"Test Execution

4

hardware-in-the-loop

Untestable Systems"Identification of failures

5

dynamic behavior

•  Inputs are signals over time

•  Properties to be checked consider change over time, for independent or related output signals

•  Common when dealing with physical entities

Initi

al D

esire

d(ID

)

Desired ValueI (input)Actual Value (output)

Fina

l Des

ired

(FD

)

timeT/2 T

Smoothness

Responsiveness

Stability

Untestable Systems

6

limited testing

only a small number of test scenarios "can be executed and analyzed "

within time and budget constraints

Cyber-physical Systems as Untestable Systems

7

Testing Cyber-physical Systems: Challenges

8

complex systems

hybrid behavior

complex"environment

combinatorial"state

explosion complex"

requirements uncertainty

Example: Night Vision System

9

camera-based assistance system"providing improved vision at night

Large, complex input space: •  traffic situations • shape, speed, "

and distance •  lightning, weather

Example: Controllers

10

•  how to select test cases from the vast space of possible input signals? •  how to automatically distinguish faulty from non-faulty output signals?

Software V&V: Common Model-based Approaches

11

Model-based"Testing

Model"Checking

Model-based Testing

• Relies on models to generate test scenarios and oracles for the implemented system

• Has challenges with:

• hardware-in-the-loop

• uncertainty

• hybrid, discrete-dynamic behavior 12

Model Checking

• Verification of temporal properties over concurrent, finite state models

• Has challenges with:

• modeling complex properties of physical devices with continuous dynamics

• modeling complex, concurrent interactions with the environment

• Not easy to parallelize 13

Software V&V: Common Model-based Approaches

14

Model-based"Testing

Model"Checking

Scalability and practicality challenges in the context of untestable systems!!!

Model Testing

Model Testing

16

Raise the level of abstraction"at which (most of the) testing is performed

Model Testing

17

Shift the bulk of testing"from implemented systems "

to models of such systems and their environments

Model Testing: Strategy

• Make the best out of available time and resources for testing the implemented system

• Focus on high-risk test scenarios within budget and time constraints

18

• Capture relevant aspects of system behavior, structure, environment, and properties, as well as their uncertainty

• Are used as a basis for execution and risk analysis

• Use the results to focus the testing of the implemented system

The Role of Models

19

Model Testing vs. "Model-based Testing

20

The focus of model testing is on"testing the model,

not the implementation

Model Testing vs."Model Checking

• Not exhaustive, but risk-driven

• Oracles can be dynamic properties over time intervals and handle uncertainty

• Supports modeling the physical world, heterogeneous formalisms and integration languages

21

Model Testing: Objectives and Challenges (1/5)

22

Executable and testable models of the system and

its environment

Objective Definition and execution of testable models, including

dynamic and uncertain behavior

Challenge

Model Testing: Objectives and Challenges (2/5)

23

Finding, executing, "and analyzing "

high-risk test scenarios

Objective Automated, effective test selection in large input

spaces, partly defined by input signals

Challenge

Model Testing: Objectives and Challenges (3/5)

24

Modeling"uncertainty and risk

Objective

Automated detection of failures under uncertainty;

risk assessment

Challenge

Model Testing: Objectives and Challenges (4/5)

25

Change impact analysis "and regression testing

Objective

Regression tests prioritization or selection

when models change

Challenge

Model Testing: Objectives and Challenges (5/5)

26

Guiding testing on the implemented system

(with HiL)

Objective

Selecting test cases to run so as to reduce overall risk

Challenge

Other Questions

27

•  Cost of modeling

•  Fidelity of modeling and its impact on cost-effectiveness

•  Impact of uncertainty on finding high-risk scenarios and fault detection

Model Testing

28

Shift the bulk of testing"from implemented systems "

to models of such systems and their environments

Accounts for uncertainty, dynamic behavior, a diversity of modeling languages from different

domains, risk modeling Better focus testing of implemented systems"

on high-risk test scenarios within limited resources

Testing the Untestable: Model Testing

of Complex Software-Intensive Systems"

Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli

We’re hiring!!!