presentation slides: "how to get 100% code coverage"

Intelligent Testing 2015

Bristol, 18th June 2015

Dr Guillem Bernat

[email protected]

How to get to 100% code

coverage

Introduction

Code coverage is important

Sometimes 100% is difficult

What do we do if we can't get 100%

Code coverage defined

What does it mean to get 100% code coverage?

“Testing shows the presence of bugs, not their absence”

Edsger Dijkstra (1969)

100% code coverage does not mean:

your code is correct


your software requirements are correct


the compiler translated your code correctly

const volatile int x;

volatile int y;

void foo(void){

for(y=0; y>10; y++)

{

int z = x;

}

}

foo:

mov1 $0, y

mov1 x, %eax

jmp .L2

.L3:

mov1 y, %eax

incl %eax

mov1 %eax, y

.L2:

mov1 y, %eax

cmp1 $10, %eax

jg .L3

ret

At the -0s optimization level, GCC 4.3.0 for IA32

compiles the C code on the left to the assembly on

the right.

The compiler output is wrong: the access to volatile-

qualified variable x should not be hoisted out of the

loop.


you have covered 100% of your object code

Source code Object code Compiled to

Does it matter which code coverage criteria I'm

using?

Coverage type DO-178B/C ISO 26262

(Software architecture)

ISO 26262

(unit test)

Function Used with MC/DC

ASIL C, D Highly

Recommended

ASIL A, B Recommended

-

Call Coverage Not required

ASIL C, D Highly

Recommended

ASIL A, B Recommended

-

Statement Level A, B, C

Required -

ASIL A, B Highly

Recommended

ASIL C, D Recommended

Decision Level A, B

Required - -

Branch Not required -

ASIL B, C, D Highly

Recommended

ASIL A Recommended

MC/DC Level A Required - ASIL D Highly Recommended

ASIL A, B, C Recommended

What are the barriers to 100% code coverage?

Missing requirements.

Missing or incorrect tests.

Dead code.

Deactivated code.

Defensive programming.

Impossible combinations of events.

Compiler-introduced errors.

Barrier #1: Missing Requirements

Add new requirements

Develop tests

Run the tests

What to do about it

Barrier #2: Missing/incorrect tests

What to do about it Implement additional tests

Ensure that they trace to

the correct requirement(s)

Run the new tests

Implement additional tests

Ensure that they trace to

the correct requirement(s)

Run the new tests

What to do about it

Barrier #3: Dead code

foo(a,c){

int b;

mode = MOVING;

if(mode == STATIONARY){

b = 1 + a;

}

c = a;

return c;

}

Remove the dead code if

possible or

Justify why the code cannot

be run

What to do about it

DO-178C recommends "The [dead] code should be removed and

an analysis performed to assess the effect and the need for

reverification." [6.4.4.3c]

For ISO 26262, the general guidance for "insufficient" coverage

applies, namely to provide a rationale for why the dead code is

acceptable [6:9.4.5].

What to do about it…

Barrier #4: Deactivated code

settings = read_input(CONFIG_PINS);

if ((settings & CONFIG_MASK_A) == CONFIG_A) {

/* code to execute in configuration A */

...

} else if ((settings & CONFIG_MASK_B) == CONFIG_B) {

/* code to execute in configuration B */

...

}

Justify why code won’t be

executed

Never executed during normal operation

Will not affect the execution of the system

What to do about it

Barrier #5: Defensive Programming

switch(a % 4){

case 0:

..........

break;

case 1:

..........

break;

case 2:

..........

break;

case 3:

..........

break;

default:

..........

break;

}

Justify why defensive

programming cannot be

executed

Provide evidence that if

triggered, the defensive

programming works

correctly

What to do about it

Barrier #6: Impossible Combinations of Events

-- “enabled” might already be set at this point

if speed < 1700 then

enabled := true;

end if;

if speed < 1500 and enabled then

...

end if;

Simplify the condition?

or

Justify why 100% could not

be achieved

What to do about it

Barrier #7: Compiler Introduced Errors

int a,x;

if a < 10{ /*Compiler generates “a <=10”*/

result = x / 5;

} else {

result = x / 10;

}

a x

Test case 1 3 4

Test case 2 10 4

Review existing code base

Add to coding guidelines /

code review guidelines

What to do about it

Justifying code: in-line justifications

void fatal_error (void)

{

#pragma RVS justification( "COV_FUNCTIONS", "The fatal_error

function cannot be executed during normal execution")

/* some code */

}

...

void check_status (int status)

{

if ( (check_enabled) && (FATAL_ERROR == status) ) {

#pragma RVS justification( "COV_DECISIONS", "Variable 'status'

cannot take value FATAL_ERROR during normal execution")

fatal_error();

}

}

...

Justifying code: justifications in a separate file

#pragma RVS [selftest.c:34] justification("COV_MCDC", "2: (RAM_ERROR ==

status)", "RAM_ERROR cannot be triggered during normal test");

#pragma RVS [selftest.c:439] justification("COV_FUNCTIONS", "Cannot trigger

emergency_shutdown() during normal test");

Justifications editor

Review – establish missing coverage

Identify cause Identify action to take

Dealing with less than 100% code coverage




Missing requirements.

Missing or incorrect tests.

Dead code.

Deactivated code.

Defensive programming.

Impossible combinations of events.

Compiler-introduced errors.




Remove code Add (or fix)

requirements Justify why code

won't be executed Justify why it isn't

possible to test

reduce testing timescales by running fewer on-target tests

reduce risk through greater tool flexibility

reduced effort for certification activities

Structural coverage analysis to DO-178B/C Level A and ISO 26262

Logos and images can be positioned within this area.

Change layout to blank (without position box) before use