qa&test 2007 – 17—19 october – bilbao, spain jan peleska, cornelia zahlten integrated...

QA&TEST 2007 – 17—19 October – Bilbao, Spain

Jan Peleska, Cornelia Zahlten

Integrated Automated Test Case Generation and Static AnalysisProf. Dr. Jan PeleskaCentre for Computing Technologies, University of Bremen, Germany

Dr. Ing. Cornelia Zahlten Verified Systems International GmbH, Bremen, Germany

Jan Peleska, Cornelia Zahlten2007-10-18

Motivation ...... for an integrated approach to automated module

testing and static analysis:Verification specialist’s perspective:

static analysis gives insight with respect to useful test cases and expected module behaviour

Module testing can help to verify or falsify fault hypotheses found in static analysis

Tool builder’s perspective: Algorithms needed for automated test case generation are

also useful for automated static analysis and vice versa

Tool capabilities required are illustrated by means of RT-Tester, developed by Verified

Systems


Overview1. The objectives of module testing + static analysis

2. The tool capabilities required

3. Fundamental techniques Structural test case generation enables functional

test case generation Tool architecture Static analysis by abstract interpretation Test data generation

4. Optional integration of techniques into model-driven development cycle


Objectives of Module Testing Given a functional specification of the required

module behaviour, find test cases such that every aspect of functional requirements is checked

at least once (functional testing) the module’s code is fully covered according to the

applicable coverage requirements (structural testing), while again checking compliance with functional requirements

the module is robust against illegal inputs, fulfils performance requirements, ... (non-functional tests)


Objectives of Static Analysis Verify the module’s conformance with non-

functional requirements: Absence of run-time errors, e.g.:

Correctness of memory access (array boundaries, pointer utilisation, string handling, memory copies)

Well-definedness of arithmetic operations Absence of unintended endless loops

Absence of unreachable code Module complexity is acceptable (SW metrics ...) Module code conforms to coding standards ...


The Tool Capabilities Required

Capability 1 – Module specification support Capability 2 – Stub specification support Capability 3 – Specification support for Module-

internal assertions Capability 4 – Automated test data generation for

structural coverage Capability 5 – Automated test data generation for

functional coverage Capability 6 – Run-time error detection Capability 7 – Advanced debugging support


Side Remark: The Objectives of Automation

The tool capabilities listed above allow verification specialists to focus on their most important tasks: Specify expected module behaviour Specify meaningful conditions about the

environment behaviour – for module testing, this is expressed by preconditions and stub specifications

Specify module-internal assertions Verify / falsify potential failures Identify error locations in module code


Capability 1 – Module specification support (1)

The RT-Tester tool allows module specification by means of Preconditions: Logical conditions about the legal

input parameters, global variable / object attribute values (pre-states) to be met when calling the function / method

Postconditions: Logical conditions about the return values, output parameters and resulting global variable / object attribute values (post-states) on function / method return



Example of Unit Under Test (UUT) specification

with RT-Tester:double globx;

@uut double f(double x, double y, int i){

@pre: 0 < x and x < 100 and

-10 < y and exp(y) < x and

0 <= i and i <= 10;

@post: @rttAssert( globx == globx@pre );

if (-10 < y and exp(y) < x )

{ @rttAssert( f == 1/(x-exp(y)) );}

else { @rttAssert( f < 0 ); }

};



double globx;

@uut double f(double x, double y, int i){

@pre: … and exp(y) < x … ;

@post: @rttAssert( globx == globx@pre );

if (-10 < y and exp(y) < x )

{ @rttAssert( f == 1/(x-exp(y)) );}

else { @rttAssert( f < 0 ); }

};

Keywords for module specification, pre- / post conditions

Conditions may contain arbitrary

parameter/variable relations

References to variable pre-state

Checked only

if condition

-10 < y and …

holds

Checked only if not(-10 < y and exp(y) < x)


Capability 2 – Stub specification support (1) Suppose UUT f() calls sub-function double g(double *w):

@stub double g(@inout double *w){

@assert: w and *w > 0;

@constraint: g > (*w)@pre and

(*w)@pre < *w;

};

Keyword for stub declaration

Condition to be checked

whenever UUT calls g() –

signal UUT failure if violated

Test data generator only

generates data satisfying

constraint

Keyword for input/output

reference parameter


Capability 2 – Stub specification support (2)

Further stub specification capabilities: Number and sequence of stub calls performed by

UUT can be referenced in postconditions Pre- / poststates of stub calls can be stored and

referenced in postconditions Stub body may also be explicitly programmed –

test data generator takes explicit stub code into account


Fundamental Techniques: Tool Architecture

Internal encoding of programs and specifications: Hierarchic Hybrid Transition Systems

Memory model allows evaluation of array / pointer handling

Selection of paths through the program / specification model

Solution of path constraints results in concrete test data

(1) C/C++ interpretation semantics

(2) Abstract interpretation semantics


Fundamental Techniques: Structural Test Case Generation Enables Functional Test Case Generation

Consider UUT f(…) Assume f(…) is to be tested against precondition

P(v) Assume expected results are specified by

postcondition

Q(v,v@pre) ≡

(C_1(v,v@pre) → Q_1(v,v@pre))

and … and (C_k(v,v@pre) → Q_k(v,v@pre))


Structural coverage of augmented function f_aug()

results in functional coverage of f(): 1. void f_aug(…) {

2. if ( P(v) ) { // Test data meets precondition

3. f(…); // UUT is invoked

4. if ( C_1(v,v@pre) ) {

5. // First functional feature has been tested

6. assert(Q_1(v,v@pre));}

7. if ( C_2(v,v@pre) ) …

8. }

9. else {

10. f(…); } // Robustness test

11.}


Static Analysis by Abstract Interpretation (1)

Static analysis derives program properties from program abstractions:

Suppose you wish to analyse C/C++-function/method f(…) and prove property P about f().

Instead of proving P directly, using C/C++ operational semantics and analysing all possible execution states of f(), we analyse an abstracted function A(f) of f() and an abstracted property A(P) such that (A(f) satisfies A(P)) → (f() satisfies P)



Static analysis derives program properties from program abstractions:

If property (A(f) satisfies A(P)) → (f() satisfies P)

holds then A(f) is called an abstract interpretation of f().

For static analysis of functions/methods, A(f) often keeps the same control structures as f() but operates on abstracted program variables



Using abstract interpretations speeds up the analysis process, but false alarms may occur, that is (A(f) satisfies A(P)) does NOT hold, but (f() satisfies P) holds

Therefore, if abstract input data A(x) can be found such that

A(f)(A(x)) runs into an abstract state satisfying not(A(P))

we look for concrete input data x such that f(x) runs into a state satisfying not(P)


Abstract Interpretation Example: Interval Analysis

Interval analysis uses interval ranges for variable valuations instead of concrete values:

Instead of investigating concrete input values x and calculating concrete results, say, y = f(x), we use input interval ranges A(x) = [x0 ,x1 ] and calculate

output interval ranges A(y) = [y0, y1 ] such that

For all x in [x0 ,x1 ] : f(x) in [y0, y1 ]


Abstract interpretation example: Interval Analysis (continued)

All operations x ω y on program variables x,y can be abstracted to corresponding interval analysis operations I [ω] J on intervals I, J by setting I [ω] J = [ a0 , a1 ] with

a0 = Infimum({ a ω b | a in I and b in J })

a1 = Supremum({ a ω b | a in I and b in J })


Abstract interpretation example: Interval Analysis (continued)

For many basic operations this leads to simple and easy-to-implement interval abstractions, e.g. [x1 , x2 ] [+] [y1 , y2 ] = [x1 + y1 , x2 + y2 ]

Abstract interpretation by interval analysis gives rise to 3-valued logic: For example, [x1 , x2 ] [<] [y1 , y2 ]

= true for x2 < y1 = false for y2 ≤ x1 = undecided otherwise: x1 < y2

and y1 ≤ x2


C-Sample-Function f() 1. double globx, globy;

2. double f(double x, double y, int i) {

3. double z;

4. int j, k, error0 = 0;

5. if ( i < 0 )

6. k = 0;

7. else

8. k = i;

9. if ( x < 5 and y < exp(x) )

10. x = x - y - globy;

11. else

12. x = y - y - globx;


C-Sample-Function (continued) 13. for ( k = k + 1;

14. k < x and error0 == 0;

15. k = k*2)

16. { if ( k < 0 )

17. error0 = 1; } // reachable ?

18. if ( error0 == 0 )

19. z = log((double)k-x);

20. else

21. z = 0;

22. return z;}


Example 1: Unreachable code in line 17 ?

Assume precondition x,y,globx,globy,i in [-10,10]

Perform abstract interpretation by interval analysis with input value abstraction A(x) = A(y) = ... = A(i) = [-10,10] :

double f(double x, double y, int i) {

// A(x) = A(y) = ... = A(i) = [-10,10]

double z;

// A(z) = [-∞,+∞], since stack values are undefined

int j, k, error0 = 0;

// A(j) = A(k) = [-∞,+∞], A(error0) = [0,0]

if ( i < 0 ) k = 0; else k = i;

// A(k) = [-10,10], because A(i)[<]0 = undecided

// Result can be improved to A(k) = [0,10],

// because k = i; is only assigned for 0 <= i


Example 1: Unreachable code in line 17 ?

For loops, calculate interval interpretation fixpoint

// A(k) = [0,10], A(error0) = [0,0], A(x) = [-30,30]

for ( k = k + 1;

k < x and error0 == 0;

k = k*2)

{ if ( k < 0 )

error0 = 1; } // line 17

// Loop interpretation fixpoint

// A(k) = [1,58], A(error0) = [0,0], A(x) = [-30,30]

As a consequence, line 17 is unreachable Static analysis provides information for structural coverage test data generation process


Example 2: Structural coverage test data generation

Suppose we wish to cover if-branches line 6 and line 10, and skip the for-loop in lines 13 –17 in the C-sample function f() introduced above – with precondition as given above. This results in the following constraints to be solved:

x,y,globx,globy,i in [-10,10] // Precondition

and (i < 0) // if-condition line 5 true

and (x < 5.0) // if-condition line 9 true, 1st conjunct

and (y < exp(x)) // if-condition line 9 true, 2nd conjunct

and (1 >= x – y – globy) // loop condition line 14, k < x, false

which evaluates to true if

A(i) = [-10,-1] and (A(x) subset [-10,5[) and

(A(y), A(globx), A(globy) subset [-10,10]) and

(A(y) [<] [exp](A(x))) and ([1,1] [≥] A(x) [-] A(y) [-] A(globy)) is true



Solution technique for interval constraints above:

1. Start with initial interval valuations: A(i) = [-10,-1] and (A(x) = [-10,5[ ) and (A(y), A(globx), A(globy) = [-10,10])

2. If one conjunct evaluates to false, no solution exists, and the path is infeasible, i.e. cannot be covered

3. If all conjuncts evaluate to true, interval solution has been found, select any x in A(x), y in A(y), ..., i in A(i): These inputs will lead to coverage of the desired path

4. If one conjunct evaluates to undecided, perform bi-partitioning on interval with largest diameter



Bi-partitioning example: Suppose (A(y) [<] [exp](A(x))) evaluates to undecided, and that

diameter(A(y)) < diameter(A(x))

Define A1(x) = [Inf(A(x)),(Inf(A(x))+Sup(A(x)))/2] A2(x) = [(Inf(A(x))+Sup(A(x)))/2,Sup(A(x))]

Re-perform steps 2 – 4 above with two possible solution candidates A1(x),A(y),A(globx),A(globy),A(i)

A2(x),A(y),A(globx),A(globy),A(i)

Use forward-backward constraint propagation to avoid too many bi-partitioning steps


Example 2: Structural coverage test data generation Test data generator uses the same abstract interpretation by

interval analysis technique as static analysis Static analysis applies over approximation: Possible concrete

executions are a subset of possible executions identified by static analysis fast, but may lead to false alarms

Test data generation applies under approximation: Every tuple (x,y,...) selected from solution intervals A(x), A(y),... covers the desired path slower, but guaranteed reachability

As a consequence, the test data generator can be used to verify potential errors indentified by static analysis, by means of constructing concrete inputs (x,y,...) leading to the error situation


Example 3: Test Case Generator Supports Static Analysis

Consider modified for-loop in example above:1. // A(k) = [0,10], A(error0) = [0,0], A(x) = [-30,30]

2. int kOld = k; // A(kOld) = [0,10]

3. for ( k = k + 1;

4. k < x and error0 == 0;

5. k = k*2) {

6. if ( kOld > k ) // (A(kOld) [>] A(k)) = [0,1]

7. error0 = 1; // = undecided

8. else

9. kOld = k;

10. }

11. // A(k) = [1,58], A(kOld) = [1,29]12. // A(error0) = [0,1], A(x) = [-30,30]


Example 3: Test Case Generator Supports Static Analysis Conventional static analysis tool might stop with false alarm “error0

= 1 (line 7) potentially reachable” Path selector, constraint generator and solver can falsify this result

as follows: Path selector unwinds loop for n = 0,1,2, ... cycles Constraint generator collects conditions to be fulfilled in order to

stimulate n loop cycles Solver establishes that at most n = 5 cycles are possible for initial

valuations A(k) = [0,10], A(x) = [-30,30]at loop entry

Solver establishes that if-condition (kOld > k) cannot be fulfilled for n ≤ 5.

Test case generation can confirm that “error0 = 1 (line 7) is reachable” if A(i) = [230, 230], A(x) = [231,∞] at program start


Integration of Techniques Into Model-Based Development Cycle (1)

In model-driven development code is generated from structural and functional models, for example in UML: Composite structure diagrams, class diagrams for

SW architecture Statecharts for reactive behaviour Method specifications for transformational

behaviour Transformational behaviour is typically coded

directly in the target programming language


Model-Based Development ... (2) The techniques described in this talk are applied

to the test+analysis of these hand-coded methods RT-Tester also performs automated test case

generation from UML-Statecharts: Test cases are paths through the Statechart RT-Tester calculates the necessary input data to

be passed to the system under test whenever an input method is invoked

Intermediate model representation allows to use the same techniques as sketched for module testing


Conclusion (1)

1. We have described the main objectives of modules testing and static analysis

2. The utilisation of combined testing+analysis has been illustrated, using the RT-Tester tool as an example

3. The fundamental techniques for implementing the capabilities described have been sketched


Conclusion (2)

4. The integration of the methods and techniques described in this presentation into the model-driven development process have been sketched

5. We expect that a variety of tools supporting testing+analysis will become available in the near future

6. All features are currently applied and evaluated in verification and test projects for railway control and avionic systems


Conclusion (3)7. Road map for RT-Tester:

Available NOW: Module testing support Test case generation from UML Statecharts – only in

combination with case tool Borland Together

January 2008: Static analysis and light-weight functional verification available

March 2008: Full model-based testing support available – support for various case tools

Acknowledgements: This work has been supported by BIG Bremer Investitions-Gesellschaft under research grant 2INNO1015B

qa&test 2007 – 17—19 october – bilbao, spain jan peleska, cornelia zahlten integrated...

Documents