Software Testing

Sudipto GhoshCS 406 Fall 99

November 16, 1999

11/16/99 CS 406 Testing 2

Learning Objectives

• Data flow-based testing• Test adequacy assessment

11/16/99 CS 406 Testing 3

Problems with path testing

• With each predicate, there is a combinatorial explosion in the number of paths.

• Potentially infinite number of paths because of loops.

• Not all paths a feasible, i.e., there may not be inputs for which the path is executed.

• Suppose that somehow, we satisfied this criterion. Would we find all the faults?

11/16/99 CS 406 Testing 4

More problems

• A path is tested only if it is present!!• Consider the example:

double logBase19 (double x) {

return ln(x)/ln(19);

}

• Missing condition is:if(x > 0)

11/16/99 CS 406 Testing 5

More problems

• It is possible to test every path without detecting the fault in the product.

• Function to test the equality of 3 integers:• (Fallacious) assumption:

• If the average of the 3 numbers is equal to the first, then they are equal.

boolean areEqual(int x,int y,int z){if ((x+y+z)/3 == x) return TRUE;else return FALSE;

}

• Exercise: Think of test cases.

11/16/99 CS 406 Testing 6

Reducing the number of paths to be covered• Find something between branch and path

coverage.• Select a set of paths that ensure branch

coverage and try some other paths that help reveal errors.• Consider two paths same if they differ only in

their sub-paths that are caused due to loops.• Restrict test cases to linear code sequences

• Identify set of points L, from which control flow may jump; includes entry, exit, branch statements

• Paths begin and end at elements in L.

11/16/99 CS 406 Testing 7

Data Flow-Based Testing

• Select paths to be executed based on data flow analysis

• Information about where • variables are defined• variables are used

• Idea is to make sure that the definitions of variables and their subsequent use is tested

11/16/99 CS 406 Testing 8

Variables in statements

• Variables occur in statements as• Definition – def

• Variables on the left side of a statement ( c )• c := 10;

• Variable is used in a computation – c-use• Variables on the right-hand side of a statement ( c )• temp := c + 5;• Variables used in a write statement

• Variable is used in a predicate – p-use• if( c!= MAXLENGTH ) {

11/16/99 CS 406 Testing 9

An example (xy)

1. scanf(x, y); if(y < 0)

2. pow = 0 – y;

3. else pow = y;

4. z = 1.0;

5. while(pow != 0)

6. { z = z * x; pow = pow – 1; }

7. if ( y < 0 )

8. z = 1.0/z;

9. printf(z);

11/16/99 CS 406 Testing 10

Def/use graph of example

1

2 3

6

4

5

7

8 9

def = {x, y}: c-use = {}

def = {pow}:c-use = {y}

def = {pow}:c-use = {y}

def = {z}: c-use = {}

def = {}: c-use = {}

def = {}: c-use = {}def = {z, pow}:c-use = {x, z, pow}

def = {z}:c-use = {z}

def = {}:c-use = {z}

y

y

y

pow

y

pow

11/16/99 CS 406 Testing 11

More on def/use graphs

• With each node, we associate the c-uses of a variable

• With each edge, we associate the p-uses of a variable

11/16/99 CS 406 Testing 12

Def-clear paths

• Def-clear path with respect to a variable x:• Path from node i to node j, such that there is

no def of x in the nodes in the path from i to j. Nodes i and j may have a def.

• Find out examples from the previous graph.• Path from node i to edge (j, k), such that there

is no def of x in the nodes in the path.• Find out examples from the previous graph.

11/16/99 CS 406 Testing 13

Global c-use

• A c-use of a variable is considered global if:• there is not def of x within the block preceding

the c-use• Example?

11/16/99 CS 406 Testing 14

Global def

• A def is global if it can be used outside the block in which it is defined.

• A def of a variable x, in a node i, is aglobal def, if:• it is the last def of x in the block (node) i• a def-clear path exists from i to another node

which has a global c-use of x• Example?

11/16/99 CS 406 Testing 15

Set def(i)

• Set of variables for which there is a global def in the node i.

• Examples:

11/16/99 CS 406 Testing 16

Set c-use(i)

• Set of variables for which there is a global def in the node i.

• Examples:

11/16/99 CS 406 Testing 17

Set p-use(i, j)

• For an edge (i, j), the set p-use(i, j), is the set of variables for which there is a p-use for the edge (i, j).

• Examples:

11/16/99 CS 406 Testing 18

Set dcu(x, i)

• dcu(x, i) represents all those nodes in which the global c-use of x uses the value assigned by the def of x in i.

• Suppose a variable x is in def(i) of node i.• Set of nodes, such that :

• Each node has x in its c-use• There is a def-clear path from i to j

• Examples:

11/16/99 CS 406 Testing 19

Set dpu(x, i)

• dpu(x, i) represents all those edges in which the p-use of x uses the value assigned by the def of x in i.

• Suppose a variable x is in def(i) of node i.• Set of edges, such that :

• each edge has x in its p-use• There is a def-clear path from i to (j, k)

11/16/99 CS 406 Testing 20

Test case selection criteria

• Let G be the def-use graph for a program• Let P be a set of all the complete paths of

G that were executed during the test.• Examples:

11/16/99 CS 406 Testing 21

All-defs criterion

• Make sure that the definitions of all variables are tested.

• For every def of every variable, one of its uses (either p-use or c-use) must be included in a path.

• For every node i in G and every x in def(i), P includes a def-clear path w.r.t. x to some member of dcu(x, i)

11/16/99 CS 406 Testing 22

All-p-uses criterion

• All p-uses of all definitions should be tested.

• For every x def (i), P includes a def-clear path w.r.t. x from i to all members of dpu(x, i).

• Note that a c-use may not be tested• Can require

• all p-uses, some c-uses criterion

11/16/99 CS 406 Testing 23

All c-uses criterion

• All c-uses of all definitions should be tested.

• For every x def (i), P includes a def-clear path w.r.t. x from i to all members of dcu(x, i).

• Note that a p-use may not be tested• Can require

• all c-uses, some p-uses criterion

11/16/99 CS 406 Testing 24

All-uses criterion

• All p-uses and all c-uses of a definition must be exercised.

• The set P must include,• for every node i and every x def(i)• a def-clear path w.r.t. x from i to all elements

of dcu(x, i) and to all elements of dpu(x, i).

11/16/99 CS 406 Testing 25

Number of test cases

• Can be shown theoretically that the limit of the size of test set is up to quadratic in the number of two-way decision statements in the program.

• Empirical studies have shown that the actual number of test cases that satisfy a criterion is quite small and increases linearly with the number of two-way decisions.

11/16/99 CS 406 Testing 26

Inclusion relationship between criteria

all-paths (path coverage)

all-uses

all-defs

all-c-uses/some-p-uses

all-p-uses/some-c-uses

all-p-uses

all-edges (branch coverage)

all-nodes (statement coverage)

11/16/99 CS 406 Testing 27

Implication of inclusion

• Suppose C1 includes C2 (C1 C2), I.e test cases satisfying C1 also satisfy C2

• Does it mean that C1 is always better than C2?

• Statistically, the set of test cases satisfying C1 will be better than those satisfying C2.

• Testing done using these criteria performs better than randomly selected test cases.

11/16/99 CS 406 Testing 28

Test assessment

• Develop• a test set T• a collection of test inputs

• Question:• How good is T?• Test assessment is the measurement of the

goodness of T.• Test assessment is carried out based on one

or more criteria.

11/16/99 CS 406 Testing 29

Test adequacy assessment

• These criteria are known as test adequacy criteria.

• Test assessment is also known as test adequacy assessment.

11/16/99 CS 406 Testing 30

Test adequacy assessment (contd)• Test adequacy assessment provides the

following information:• A metric, also known as adequacy score, or

coverage, usually between 0 and 1.• A list of all the weaknesses found in T, which

when removed, will raise the score to 1.• The weaknesses depend on the criteria used for

assessment.

11/16/99 CS 406 Testing 31

Test adequacy assessment (contd)• Once the coverage has been computed,

and the weaknesses identified, one can improve T.

• Improvement of T is done by examining one or more weaknesses and constructing new test requirements designed to overcome the weaknesses.

11/16/99 CS 406 Testing 32

Test adequacy assessment (contd)• This is continued until all weaknesses are

overcome, i.e., the adequacy criterion is satisfied (coverage = 1)

• In some instances, it may not be possible to satisfy the adequacy criteria for one or more of the following reasons:• Lack of sufficient resources (people, time)• Weaknesses that cannot be removed because

they are infeasible• The cost of removing the weakness is not

justified

11/16/99 CS 406 Testing 33

Test adequacy assessment (contd)• While improving T by removing its

weaknesses, one usually tests the program more thoroughly than it has been tested so far.

• This additional testing is likely to result in the discovery of remaining errors.

• Hence, we say that test assessment and improvement helps in the improvement of software reliability.

11/16/99 CS 406 Testing 34

Test adequacy assessment - summary procedure0. Develop T

1. Select an adequacy criterion C

6. DONE!!

2. Measure adequacy criterion of T w.r.t C

5. More testing is warranted?

3. Is T adequate?

4. Improve T

Yes

Yes

No

No