29-Jul-2013

Increasing Efficiency of ISO 26262

Verification and Validation by Combining

Fault Injection and Mutation Testing with

Model Based Development

Rakesh Rana1, Miroslaw Staron1, Christian Berger1, Jörgen Hansson1,

Martin Nilsson2, Fredrik Törner2

1Computer Science & Engineering, Chalmers/ University of Gothenburg, 2Volvo Car Corporation, Gothenburg Sweden

This Car Runs on Code

Avionics & onboard systems:

F-22 Raptor: 1.7 mLOC; F-35 Joint Strike Fighter: 5.7 mLOC

Boeing’s 787 Dreamliner: 6.5 mLOC

Ref: http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code

S-class Mercedes-Benz: 20 mLOC (only for Radio & navigation systems)

“It takes dozens of mircroprocessors running 100 million lines of code to get a premium car

out of the driveway, and this software is only going to get more complex”

• Software is today at the heart of automotive development.

• A typical premium car has up to 70 ECUs, connected by several

system buses to realize over 2000 functions (Broy, 2006).

• 90% of all innovations are driven by electronics and software

(Grimm, 2003).

• Many functions within automotive development are safety critical

Software in Automotive Domain

Images: Volov Cars; http://img.uphaa.com/uploads/777/uphaa-safety-funny_(12).jpg

Automotive Software Development

ISO 26262 - Road vehicles -- Functional safety

Automotive Software Development

Mellegard N, Staron M, Torner F. “A light-weight defect classification scheme for embedded automotive software

and its initial evaluation”. 23rd International Symposium on Software Reliability Engineering (ISSRE), 2012,

IEEE, 2012; 261–270.

Automotive Software Development

Model Based Development

Our Position for early defect detection

We contend that fault injection can be effectively used at the

model level to verify and validate the attainment or violation of

safety goals.

We also propose that it should be complemented with mutation

testing approach at the model level.

FI combined with Mutation testing can provide enough

statistical evidence for argumenting of fulfilment of safety goals

as per the ISO-26262 safety standard requirements.

ISO 26262 - Road vehicles -- Functional safety

ISO 26262 Chapter Reference to recommendation

4

Hardware-software integration

and testing

•Table 5 — Correct implementation of technical safety requirements at the hardware-software level.

•Table 8 — Effectiveness of a safety mechanism’s diagnostic coverage at the hardware-software level.

System integration and testing •Table 10a — Correct implementation of functional safety and technical safety requirements at the system

level

•Table 13b — Effectiveness of a safety mechanism's failure coverage at the system level

Vehicle integration and testing •Table 15 — Correct implementation of the functional safety requirements at the vehicle level

•Table 18 — Effectiveness of a safety mechanism's failure coverage at the vehicle level

5 Hardware integration and

testing

•Table 11 — Hardware integration tests to verify the completeness and correctness of the safety mechanisms

implementation with respect to the hardware safety requirements

6 Software unit testing •Table 10 — Methods for software unit testing

Software integration and

testing

•Table 13 — Methods for software integration testing

Rana, R., Staron, M., Berger, C., Hansson, J., Nilsson, M., Törner, F., 2013. Improving Fault Injection in Automotive Model Based Development using Fault Bypass Modeling. Accepted: 2nd

Workshop on Software-Based Methods for Robust Embedded Systems, Informatik 2013, Koblenz, Germany

Fault Injection

Images: www.sp.se; www.generalcomics.com

Mutation Testing

http://muclipse.sourceforge.net/about.php

Road map for early defect detection

a) Assign technical safety requirements (TSRs) corresponding to the

functional safety requirements (FSRs) to function’s z outputs.

b) Use fault injection techniques to inject faults (similar to commonly

occurring defect)s and other possible fault conditions at the x inputs.

c) Fault scenarios leading to violation of TSRs/FSRs are identified; statistics

are built on faults leading to failures; fault propagation properties are

studied; fault tolerance system is strengthened.

d) Repeat steps (b) & (c) to test, correct and validate the given

system/function for its dependencies on other functions/components.

e) Cause mutations to the “n” basic blocks of given functional model and

asses the detection effectiveness of test suite/cases for possible

implementation bugs.

f) Examine mutants not killed by given set of test cases for their effect on

FSRs. If a given mutation violates the FSRs then a suitable test case is

created to detect/kill such mutants, i.e. detect such bugs in actual code.

Road map for early defect detection

Best Practices for early defect detection

a) Build and maintain models corresponding to each abstraction layer of

software architecture.

b) Specify and test these models for FSRs and TSR at the appropriate

abstraction level.

c) Identify different types of defects/faults and at what stage they could be

modelled/injected in the behavioural models.

Testing models for common faults at the earliest would lead to models/software

being build robust right from the start, instead of adding fault tolerance

properties in the later stages of development.

Conclusions

1. Software today play a critical role in the automotive product development.

2. Software development in automotive domain has widely adopted the paradigm of model based development (MBD).

3. Many of software function development are safety critical.

4. There are stringent quality requirements and need to adherence to functional safety standards such as ISO 26262.

5. There exist some problems with late defect discovery.

6. Development of behavioural models in MBD offers significant opportunity to do functional testing early in the development process.

7. FI and M.Testing in combination can be used effectively verify and validate functional properties of a software functions EARLY @model level.

Thank You