Software Testing

Mistake in coding is called error ,Error found by tester is called defect, Defect accepted by development team is called bug ,Product does not meet the requirements then it Is failure.”

The Importance of Test There are hundreds of stories about failures of

computer systems that have been attributed to errors in software

There are many reasons why systems fail but the issue that stands out the most is the lack of adequate testing.

Example Pepsi - $42 Billion Error Philippines -1992Due to a software error, 800,000 bottle caps were produced with number 349 instead of one. It was equivalent to $32 billion in prize money instead of $40,000

Testing Levels

Testing Levels Unit

Unit

Unit

…..

Unit

Unit

Unit

…..

…..

Module

Module

Module

Applicationon test bed

Applicationon finalplatform

Build

Build

…..

…..

integration testing

Interface testingsystem testing

usability testing installation testing

acceptance testing

Regression testing throughout4

Each of testing activities is performed during the process of building an application

Testing Goals Based on Test ProcessTesting Levels (Bezier)

Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness (software works).

Level 2 The purpose of testing is to show that the software doesn’t work.(the purpose is to show failures)

Level 3 Testing shows the presence , not absence or failure. If we use software, we expose some risk. The risk may be small and unimportant, or the risk may be great and the consequences catastrophic, but risk is always there(The purpose of testing is to reduce the risk of software ; both testers and developers work together to reduce risk.

Level 4 The purpose of testing is to improve the ability of the developers to produce high quality software.(Level 4 testing shows that testers and developers are on the same team)

Debugging and Testing

Debugging is done in the development phase by the developerIn debugging phase identified bugs are fixed

Testing is done in the tester phase by the tester.In testing, locating and identifying of the bugs are

included.

Software Bug

A software bug is an error, flaw, mistake, failure, or fault in a computer program or system

Bug is terminology of TesterSoftware bug produces an incorrect or

unexpected result, or causes it to behave in unintended ways

Software Fault

A static (physical)defect*, imperfection, flaw in the software.

short between wiresbreak in transistorinfinite program loop

A fault means that there is a problem within the code of a program which causes it to behave incorrectly.

*defect: Mismatch between the requirements

Software Error

Error is an incorrect internal state that is the demonstration of some fault.

In other words; Error is a deviation from correctness or accuracy.

Example Suppose a line is physically shortened to 0. (there is a

fault). As long as the value on line is supposed to be 0, there is no error.

Errors are usually associated with incorrect values inthe system state.

Software FailureA failure means that the program has not performed

as expected, and the actual performance has not met the minimum standards for success.

External, incorrect behavior with respect to the requirements or other description of the expected behavior

ExampleSuppose a circuit controls a lamp (0 = turn off, 1 = turn

on) and the output is physically shortened to 0 (there is a fault).

As long as the user wants the lamp off, there is no failure.

Example: Failure & Fault & Error

Consider a medical doctor making a diagnosis for a patient. The patient enters the doctor’s office with a list

of failures (that is, symptoms). The doctor then must discover the fault, or root

cause of the symptom. To aid in the diagnosis, a doctor may order tests that look for anomalous internal conditions. In our terminology, these anomalous internal

conditions correspond to errors.

Cause-and-Effect Relationship

Faults can result in errors. Errors can lead to system failuresErrors are the effect of faults. Failures are the effect of errorsBug in a program is a fault.Possible incorrect values caused by this bug is

an error. Possible crush of the operating system is a

failure

Program Exampleclass numZero { public static int numZero (int[] x) {// if x == null throw NullPointerException// else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) { count++; } }return count; } }

The Fault ,Error and Failure in the Example

The fault is that the program starts looking for zeroes at index 1 instead of index 0

For example, numZero ([2, 7, 0]) correctly results with 1, while numZero ([0, 7, 2]) incorrectly results with 0. In both of these cases the fault is executed. Both of the cases result in an error (because of next slide)

But the first case results with correct value.

Only the second case results in failure.

Error States in the Example To understand the error states, we need to identify the state for the program.The state for numZero consists of values for the

variables x, count , i , program counter (PC). For the first case: The state at the if statement on the

first iteration ( x = [2, 7, 0], count = 0, i = 1, PC = if)This state is in error because the value of i should be zero

on the first iteration. But the value of count is correct, the error state does not

propagate to the output, and the software does not fail.

In other words, a state is in error if it is not the expected state, even if all of the values in the state are acceptable.

Error States in the Example

In the second case the corresponding (error) state is (x = [0, 7, 2], count = 0, i = 1, PC = if).

In this case, the error propagates to the variable count and is present in the return value of the method. Hence a failure results.

Distinguish Testing from Debugging

The definitions of fault and failure allow us to distinguish testing from debugging.

The difference is that debugging is conducted by a programmer and the programmer fix the errors during debugging phase.

Tester never fixes the errors, but rather find them and return to programmer.

Testing versus Debugging

Testing activity is carried down by a team of testers, in order to find the defect* in the software.

Testers run their tests on the piece of software and if they encounter any defect,they report it to the development team.

Testers also have to report at what point the defect occurred and what happened due the occurrence of that defect.

All this information is used by development team to DEBUG the defect.

After finding out the bug, he tries to modify that portion of code and then he rechecks if the defect has been removed

After fixing the bug, developers send the software back to testers.

*actual results don't match expected results

What is Software Quality ?

According to the IEEE Software Quality is:The degree to which a system, component, or

process meets specified requirements.The degree to which a system, component, or

process meets customer or user needs or expectations.

Software Quality Factors

Software Quality Assurance

Verification– are we building the product right ?– performed at the end of a phase to ensure that requirements established during previous phase have been metValidation– are we building the right product ?– performed at the end of the developmentprocess to ensure compliance with productrequirements

Verification & Validation

Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase.

Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage.

Difference Between Verification &Validation- I

Verification is preventing mechanism to detect possible failures before the testing begin. It involves reviews, meetings, evaluating

documents, plans, code, inspections, specifications etc.

Validation occurs after verification and it's the actual testing to find defects against the functionality or the specifications

The Difference between Verification &Validation- II

Verification is usually a more technical activity that uses knowledge about the individual software artifacts, requirements, and specifications.

Validation usually depends on domain knowledge; that is, knowledge of the application for which the software is written. For example, validation of software for an airplane

requires knowledge from aerospace engineers and pilots.

Software Testing Types

Black box testing : You don't need to know the internal design in detail or have a good knowledge about the code for this test.

It's mainly based on functionality and specifications, requirements. White box testing : This test is based on detailed

knowledge of the internal design and code. Tests are performed for specific code statements and coding styles.

26

Traditional Testing Levels

Class A

method mA1()

method mA2()

Class B

method mB1()

method mB2()

main Class P Acceptance testing : Is

the software acceptable to the user?

Integration testing : Test how modules interact with each other

System testing : Test the overall functionality of the system

Module testing (developer testing) : Test each class, file, module or component

Unit testing (developer testing) : Test each unit (method) individually

27

Object-Oriented Testing Levels

Class A

method mA1()

method mA2()

Class B

method mB1()

method mB2()

Intra-class testing : Test an entire class as sequences of calls

Inter-class testing : Test multiple classes together

Inter-method testing : Test pairs of methods in the same class

Intra-method testing : Test each method individually

Coverage Criteria• Even small programs have too many inputs to fully test them all

– private static computeAverage (int A, int B, int C)– On a 32-bit machine, each variable has over 4 billion possible values– Over 80 octillion possible tests!!– Input space might as well be infinite

• Testers search a huge input space– Trying to find the fewest inputs that will find the most problems

• Coverage criteria give structured, practical ways to search the input space– Search the input space thoroughly– Not much overlap in the tests

28

Test Requirements and CriteriaTest Criterion : A collection of rules and a

process that define test requirements̶� Cover every statement̶� Cover every functional requirement

Test Requirements : Specific things that must be satisfied or covered during testing– Each statement is a test requirement

– Each functional requirement is a test requirement

29

Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures …

1. Graphs

2. Logic expressions

3. Input domains

4. Syntax descriptions

Coverage Criterion

A coverage criterion is simply a recipe for generating test requirements in a systematicway:Coverage Criterion: A coverage criterion is a rule or collection of rules that impose test requirements on a test set.

30

31

1

2

0

Node Coverage (Formal Definition): For each node n ∈ reachG(N0), TR contains the predicate “visit n.”

Node Coverage (NC): TR contains each reachable node in G.

Structural Coverage Criteria -I

Node Coverage (NC) (Standard Definition) Test set T satisfies node coverage on graph G if and only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n.TR = { 0, 1, 2 }path(t1) = [0, 1 , 2] T1={t1} T1 satisfies node coverage on the graph

Structural Coverage Criteria - II Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable

path of length up to 1, inclusive, in G.The “length up to 1” allows for graphs with one node and no

edges

Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }path (t1 ) = [ 0,1,2]

path (t2 ) = [ 0, 2 ]

T2={t1, t2} Test Paths = [ 0, 1, 2 ] [ 0, 2 ]

T2 satisfies edge coverage on the graph

32

1

2

0

Edge-Pair Coverage

33

6

0

2

1

3 4

5

Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 5, 4, 6 ]

TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6], [2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }

path (t1 )= [ 0, 1, 2, 3, 6 ] path (t2 )= [ 0, 1, 2, 4, 6 ]path (t3 )=[ 0, 2, 3, 6 ]path (t4 )= [ 0, 2, 4, 5, 4, 5, 4, 6 ]

T={t1 , t2 , t3 , t4 }

Graph Coverage for Source Code

Graph : Usually the control flow graph (CFG)Node coverage : Execute every statementEdge coverage : Execute every branchLoops : Looping structures such as for loops, while loops, etc.Data flow coverage : Augment the (CFG) control flow graph

– Defs are statements that assign values to variables– Uses are statements that use variables

34

Control Flow Graphs : The if Statement

35

if (x < y){ y = 0; x = x + 1;}else{ x = y;}

4

1

2 3

x >= yx < y

x = yy = 0

x = x + 1

if (x < y){ y = 0; x = x + 1;}

3

1

2x >= y

x < y

y = 0x = x + 1

Control Flow Graphs : while and for Loops

36

x = 0;while (x < y){ y = f (x, y); x = x + 1;}

1x = 0

x >= yx < y

x >= yx < y 43

y =f(x,y)x = x + 1

2for (x = 0; x < y; x++){ y = f (x, y);}

2

3 5

x >= yx < y

y = f (x, y)

4

1

x = x + 1

Control Flow Graphs : do Loop, break and continue

37

x = 0;do{ y = f (x, y); x = x + 1;} while (x < y);println (y)

1x = 0

2x >= y

x < y

y = f (x, y)x = x+1

3

x = 0;while (x < y){ y = f (x, y); if (y == 0) { break; } else if y < 0) { y = y*2; continue; } x = x + 1;}print (y);

1 x = 0

8

3

x = x + 1

break

y < 0

2

4

5

6

7

y =f(x,y)

y == 0

y = y*2continue

The Application of Graph Criteria:Source Code

It is usually defined with the control flow graph (CFG)

Node coverage is used to execute every statement

Edge coverage is used to execute every branch Loops are used to execute looping structures

such as for loops, while loops, etc. Data flow coverage is the extended form of

control flow graph (CFG) with defs and uses defs are statements that assign values to variables uses are statements that use variables

public static void computeStats (int [ ] numbers){ int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2]; mean = sum / (double) length;

varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println ("length: " + length); System.out.println ("mean: " + mean); System.out.println ("median: " + med); System.out.println ("variance: " + var); System.out.println ("standard deviation: " + sd); } 39

i = 0

i = 0

1

2

3

54

6

7 8

i >= lengthi < length

i++

Post-Unit TestingUnit

Unit

Unit

…..

Unit

Unit

Unit

…..

…..

Module

Module

Module

Applicationon test bed

Applicationon finalplatform

Build

Build

…..

…..

integration testing

Interface testingsystem testing

usability testing installation testing

acceptance testing

Regression testing throughout40

Each of post-unit testing activities is performed duringThe process of building an application

41

- Random values- For each input parameter we randomly select the

values- Tester Experience

- For each input we use our experience to select relevant values to test

- Domain knowledge- We use requirements information or domain

knowledge information to identify relevant values for inputs

How to select input values? Scenario I

42

• Equivalence classes (for black-box testing)- We subdivide the input domain into a small number of sub-domains- The equivalence classes are created assuming that they exhibits the same

behavior on all elements- Few values for each classes can be used for our testing

- Boundary values (for black-box testing) – Is a test selection technique that targets faults in applications at the

“boundaries” of equivalence classes– Experience indicates that programmers make mistakes in processing values

at and near the boundaries of equivalence classes

How to select input values? Scenario II

Black-Box Testing Example: Person a rents « the Matrix» on May5Person b rents «Gone with the Wind» on May 6Person a returns «the Matrix» on May 10 It does not take the manner in which the application

was designed or implemented This is analogous to building an automobile and then

testing it by driving it under various conditions.BUT If the car is built with a new design for its automatic transmission , we would be wise to use this knowledge White box testing is required

White box tests are based on design and implementation

Parameter Space of computeFine()

0 5 10 15-5

old releases

one-of-a-kind

new releases

Days latee.g., (6 days late, “Casablanca”)

Each element represents a pair: (Days late, Movie name)

44

The penalty also depends on the whether the movieis a «new» or «old» release or«one of a kind.»