technology of test case generation levi lúcio university of geneva marko samer vienna university of...

Technology ofTechnology ofTest Case GenerationTest Case Generation

Levi LúcioUniversity of Geneva

Marko SamerVienna University of Technology

H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

OverviewOverview

Introduction

Model Checking

Symbolic Execution

Theorem Proving

Conclusions


IntroductionIntroduction

• In this presentation three technologies for test case generation are introduced;

– Symbolic Execution;– Theorem Proving;– Model Checking (presentation not available);

• All the models used by these methods can be reduced to state machines;

• Test case generation can be seen as state space exploration for traces obeying a criteria.


OverviewOverview

Introduction

Model Checking

Symbolic Execution

Theorem Proving

Conclusions


Symbolic ExecutionSymbolic ExecutionTalk OverviewTalk Overview

• The technique– How symbolic execution works;

– Issues related to the approach;

– Symbolic execution of abstract models;

• Test case generation– A classification of test case generation

frameworks using symbolic execution;

– Abstract model based test case generation;

– Code based test case generation.


Symbolic ExecutionSymbolic Execution

• Technique invented in the 1970s for verifying the consistency of code;

• The main principle consists of executing the code with symbolic inputs…

• … in which a symbolic input consists of replacing a « real input » (number, string, structure, object) with a symbol;

• For example, if the program asks a number to the user, « » would be passed to the program as a representation of all possible numbers.


The techniqueThe technique

• Allows exploring the possible control paths in an program. Calculates the path condition for each control path, composed of symbolic equations.

Int foo (int a, int b) {1 a++;2 if (a>b)3 a=a-b;4 else5 a=b-a;6 if (a<=-1)7 a=-a;8 return a }

Control paths for foo

1,2,3,6,7,8

1,2,3,6,8

1,2,5,6,7,8

1,2,5,6,8


The technique (cont)The technique (cont)a: b:PC: True

a: +1b:PC: True

(1)

a: b:PC:

(3)

(8)

a: b:PC:

a: b:PC:

a: b:PC:

(2 -true) (2 - false)

…

a: b:PC:

a: b:PC:

(6 - true) (6 - false)

No solution for PC equations:Infeasible Control Path

(search backtracks)

Int foo (int a, int b) {1 a++;2 if (a>b)3 a=a-b;4 else5 a=b-a;6 if (a<=-1)7 a=-a;8 return a }


The technique (cont)The technique (cont)

• From the symbolic execution of foo, one can retrieve the following information:

Control Path Path Condition

1,2,3,6,7,8 Not feasible

1,2,3,6,8

1,2,5,6,7,8 Not feasible

1,2,5,6,8

• Each path condition fully documents the decisions taken during the execution of the code.


Test case generationTest case generation

• It is possible to generate test cases by finding values that satisfy the path condition equations:

Control Path Path Condition Example test case

1,2,5,6,8

• Test case (a=1, b=3) would force the execution of foo to follow the control path 1,2,5,6,8;

• Test cases are thus created as by-products of the activity of symbolic execution;

• Symbolic execution uncovered that in the case of foo instruction 7 is never reached...


Issues related to the approachIssues related to the approach

• Dealing with loops

– Loops generate infinite control paths (unless the number of iterations is known). Solution?

• Prompt the user at each loop iteration;• Establish an upper bound for the number of iterations;• Try to automatically find a fixed point to the loop (costly).

• Solving path condition equations

– Necessary for: • understanding when a path is infeasible;• generating test cases.

– Algorithms heavy on computing resources.


Symbolic execution of abstract Symbolic execution of abstract modelsmodels

• Up until now we have considered symbolic execution only as a “white box” verification technique…

• However, nothing prevents from applying the same technique to an abstract model (state machine)

CodeAbstract Model(state machine)

Symbolic Execution

Test Cases

• State space of a model can be symbolically searched for interesting control paths;

• Symbolic execution reduces state space explosion by associating classes of inputs.


Symbolic execution for test case Symbolic execution for test case generationgeneration

• In the text several frameworks for test case generation from different models of the application are studied;

• We classify the frameworks according to two axis:

Application model

Synergies with otherverification techniques

Model Checking

Theorem proving

Abstract (B, AUTOFOCUS, CO-OPN), Code (Java)


Abstract model based test case Abstract model based test case generationgeneration

• In the text three frameworks are described that have their starting point on different abstract models:– B (framework from Legeard, Peureux et al)– AUTOFOCUS (framework from Pretschner et al)– CO-OPN (framework from Buchs et al)

• The models can be considered as state machines and the technique for test case generation is similar:

Abstract Model(state machine)

LogicProgramming

ModelTest Cases

Translationstep

Symbolicstate space search


Abstract model based test case Abstract model based test case generation (cont)generation (cont)

• The step of translation into a logic programming language (theorem prover) is necessary to:

– Animate the (static) abstract specification;

– Search the state space;

– Perform symbolic execution.

• Two kinds of logic programming languages may be used:

– Pure Prolog;

– Constraint Logic Programming (CLP) Languages: enabled to deal with numeric constraints at the unification level.


Code based test case generationCode based test case generation

• Why take a detour from model based test case generation (title of the section)?

• In the text we present a framework from Khurshid et al that generates test cases from Java code. It uses:

– A model checker (Java PathFinder) to overcome some difficulties in symbolic execution;

– The symbolic execution technique to overcome some difficulties in model checking.

• We think an insight on the framework is enriching for the discussion…


Test generation from codeTest generation from code

ExampleExample

• Symbolic execution helps dealing with state space explosion by dealing with inputs as classes of values;

• Model checking deals automatically with loops, recursion or method invocation…

ModelChecker

LogicFormulas

StateSpace

WitnessTrace

Counter-exampleTrace

Formulas hold Formulas don’t hold


Test generation from codeTest generation from code

Example (cont)Example (cont)

• The code is instrumented to deal with symbolic values and accumulate path conditions (via Java libraries):

Java CodeInstrumented

Java codeTest case

traces

Java SymbolicExecution libraries

Modelchecking

• The algorithm that performs symbolic execution knows how to deal with complex dynamic structures (objects)!

• A phase of test case instantiation is required…


OverviewOverview

Introduction

Model Checking

Symbolic Execution

Theorem Proving

Conclusions

technology of test case generation levi lúcio university of geneva marko samer vienna university of...

Documents

symbolic execution theorem

symbolic equations

symbolic inputs

pc equations

int foo int

model checking presentation

technique cont

infeasible control path