chapter 2 basics of software testing 02/06/12 © aditya p. mathur 2005

Chapter 2

BASICS OF SOFTWARE TESTING

02/06/12

© Aditya P. Mathur 2005

Software and Hardware Testing [Contd.]

Test Coverage: Practically impossible to completely test a large piece of software (OS) or any complex integrated circuit (32 or 64 bit microprocessor). This leads to notion of Acceptable Test Coverage.

A

O

B

Correct

A S-a-1

O

B

Incorrect


Software and Hardware Testing [Contd.]

Correct NAND Gate

NAND gate with fault input in A abbreviated s-a-1

Correct NAND Gate Faulty NAND Gate

A B O A B O

0 0 1 0(1) 0 1

0 1 1 0(1) 1 0

1 0 1 1(1) 0 1

1 1 0 1(1) 1 0


Testing and verification

Program verification aims at proving the correctness of programs by showing that it contains no errors. This is very different from testing that aims at uncovering errors in a program.

Program verification and testing are best considered as complementary techniques. In practice, one can shed program verification, but not testing. © Aditya P. Mathur

2005

Testing and verification (contd.)

Testing is not a perfect technique in that a program might contain errors despite the success of a set of tests.

Verification might appear to be perfect technique as it promises to verify that a program is free from errors.

Disadvantages of Verification :

The person who verified a program might have made mistake in the verification process.

There might be an incorrect assumption on the input conditions; incorrect assumptions might be made regarding the components that interface with the program, and so on.

Verified and published programs have been shown to be incorrect.


Defect Management

Defect management is an integral part of development and test process. It is a subprocess of the development process.

Defect Prevention

Defect Discovery

Recording and Reporting

Classification

Resolution

Prediction© Aditya P. Mathur 2005

Defect Management [contd.]

Defect prevention is achieved by using good coding techniques, unit test plan, code inspection.

Defect discovery is achieved by identifying defects, found in static and dynamic testing.

Defects found are classified and recorded in database

[ High Severity, Low Severity ]

Defect prediction can be achieved by SOURCE CODE ANALYSIS before entering the testing phase, as how many defects might be in the program or the application.

Some of the defect Management tools are Bugzilla, an opensource and Fogbugz.



EXECUTION HISTORY

• Execution History of a program, also known as Execution trace, is an organized collection of information about various elements of a program during a given execution.

•An execution slice is an executable subsequence of execution history.

• E.g. t1: <x=2, y=3> t2: <x=1, y=0>

9

Test generation

Any form of test generation uses a source document. In the most informal of test methods, the source document resides in the mind of the tester who generates tests based on a knowledge of the requirements.

In most commercial environments, the process is a bit more formal. The tests are generated using a mix of formal and informal methods directly from the requirements document serving as the source. In more advanced test processes, requirements serve as a source for the development of formal models.


10

Test generation strategies

Model based: require that a subset of the requirements be modeled using a formal notation (usually graphical). Models: Finite State Machines, Timed automata, Petri nets, etc.

Specification based: require that a subset of the requirements be modeled using a formal mathematical notation. Examples: B, Z, and Larch.

Code based: generate tests directly from the code.


11

Test generation strategies (Summary)


Static Testing


Static Testing

Static testing is carried out without executing the application under test.

Static testing helps discover faults, ambiguities and errors in the requirements and other application relation documents.

Static testing is complementary to dynamic testing.

Static testing is done by team of individuals who did not write the code.

Elements of static testingStatic testing accesses the following

Requirement documents

Design documents

User manuals

Static testing tools

Application code and documentation

Requirements

Control flow, Data flow and other data

List of error recommendations

Walkthroughs

Walkthrough is an informal process to review any application related document.

Requirement walkthrough

Code walkthrough ( peer code review )

Review plan

User needs

Free from ambiguities and inconsistencies

Functional and nonfunctional requirements review

Detailed report generation by understanding desired application.

Inspections

Inspection is a formal process usually associated with code

Formal code inspection is done to improve code quality at lowest cost

The team works with the help of an inspection plan which consists of•Statement of purpose•Work product to be inspected ( Code & Document inspection)•Team formation ( Roles & Tasks to perform )•Rate at which inspection task is to be completed•Data collection forms ( Consist defects discovered, Code standard violation and Time spent in each task )

Inspections [Contd.]

Inspection team members roles•Moderator ( Takes charge of process and leads the review )•Reader ( Reads the code by using code browser and large monitor )•Recorder ( Records any error discovered or issue to be looked into )•Author ( Actual developer who helps other to understand the code )

1.Static code analysis tools – CFG, Data flow graph

2.Software complexity and static testing

Model based testing and model checking

Model based testing refers to the acts of modeling and generation of test from a formal model of application behavior

Model checking refers to a class of techniques that allow the validation of one or more properties from a given model of an application

Source:

Requirements, Experience, Program

Model

Property

Model checker

Property satisfied ?

Yes No

Update modelOrSource

19

Program representation: Control flow graphs

What is CFG?

• CFG captures the flow of control within a program.

• CFG assists testers in the analysis of a program to understand its behaviour in terms of flow of control.

• CFG can be constructed manually for small programs.

• CFG is also known as flow graph or program graph.

21

Program representation: Basic blocks

• A basic block in program P is a sequence of consecutive statements with a single entry and a single exit point.

• A block has unique entry and exit points. • Control always enters a basic block at its entry point and exits from its exit point. There is no possibility of exit or a halt at any point inside the basic block except at its exit point. • Entry point is the first statement and exit point is the last statement in a block.• The entry and exit points of a basic block coincide when the block contains only one statement.

22

Basic blocks: Example

Example: Computing x raised to y


Basic blocks: Example (contd.)

Basic blocks

24

Control Flow Graph (CFG)

• A control flow graph (or flow graph) G is defined as a finite set N of nodes and a finite set E of edges.

• An edge (i, j) in E connects two nodes ni and nj in N. We often write G= (N, E) to denote a flow graph G with nodes given by N and edges by E.

• Start and End are two special nodes in N, known as “Distinguished Nodes”

• Every node is reachable from Start and every node in the path terminates at End.

• Node start has no incoming edge and end has no outgoing edge.

25

Control Flow Graph (CFG)

• In a flow graph of a program, each basic block becomes a node and edges are used to indicate the flow of control between blocks. • A pictorial representation of flow graph is often used in the analysis of control behaviour of a program.• Each node is represented by symbol usually an oval shape or rectangular box. These boxes are labelled using their corresponding block nos.• The blocks are connected by lines representing edges. Arrows are used to indicate the direction of flow.• The blocks that ends in a decision has two edges.

CFG(contd)

• Blocks and nodes are labeled such that block bi corresponds to node ni. An edge (i, j) connecting basic blocks bi and bj implies that control can go from block bi to block bj.

27

CFG Example

N={Start, 1, 2, 3, 4, 5, 6, 7, 8, 9, End}

E={(Start,1), (1, 2), (1, 3), (2,4), (3, 4), (4, 5), (5, 6), (6, 5), (5, 7),

(7, 8), (7, 9), (9, End)}

28

CFG Example

N={Start, 1, 2, 3, 4, 5, 6, 7, 8, 9, End}

E={(Start,1), (1, 2), (1, 3), (2,4), (3, 4), (4, 5), (5, 6), (6, 5),

(5, 7), (7, 8), (7, 9), (9, End)}

Same CFG with statements removed.

29

Paths

Consider a flow graph G= (N, E). A sequence of k edges, k>0, (e_1, e_2, … e_k) , denotes a path of length k through the flow graph if the following sequence condition holds.

Given that np, nq, nr, and ns are nodes belonging to N, and 0< i<k, if ei = (np, nq) and ei+1 = (nr, ns) then nq = nr.

Path(contd)

• For nodes n,m of node set N, m is said to be descendant of n if there is path from n to m. ‘n’ is said to be ancestor of m.

• The set of all successor and predecessor nodes of n will be denoted by succ(n) and pred(n) respectively.

• ‘Start’ node has no ancestor and ‘End’ has no descendant.

• Node can be its own ancestor or descendant.

Complete Path

• A path through the flowgraph is considered to be complete if the first node along the path is start node and terminating node is end node.

32

Paths: feasible paths

p1= ( Start, 1, 3, 4, 5, 6, 5, 7, 8, 9, End)p2= (Start, 1, 1, 2, 4, 5, 7, 9, End)

A path p through a flow graph for program P is considered feasible if there exists at least one test case which when input to P causes p to be traversed.

33

Paths: sample paths through the exponentiation flow graph

p1= ( Start, 1, 2, 4, 5, 6, 5, 7, 9, End)p2= (Start, 1, 3, 4, 5, 6, 5, 7, 9, End)

Two feasible and complete paths:

Bold edges: complete path.

Dashed edges: subpath.

p1= ( (Start, 1), (1, 2), (2, 4), (4, 5), (5, 6), (6, 5), (5, 7), (7, 9), (9, End))

Specified unambiguously using edges:

34

Number of paths

There can be many distinct paths through a program. A program with no condition contains exactly one path that begins at node Start and terminates at node End.

Each additional condition in the program can increase the number of distinct paths by at least one.

Number of paths

• Presence of loops also can increase the no of paths. Each traversal of the loop body adds a condition thereby increasing no. of paths.

36

Types of software testing

37

Types of testing

C1: Source of test generation.

C2: Lifecycle phase in which testing takes place

C3: Goal of a specific testing activity

C4: Characteristics of the artifact under test

C5: Test Process

One possible classification is based on the following five classifiers for dynamic testing:

Classifier C1: Source of Test Generation

Black Box Testing :

• Tests generated from informally to formally specified requirements and without the aid of the code that is under test is known as Black box testing.

• When requirements are informally specified, one could use adhoc techniques such as equivalence partitioning and boundary value analysis.

• It is functional testing.


Model Based or specification based testing –

• Occurs when requirements are formally specified.

• Example statecharts and flowcharts are generated using formal requirements.

• This is also a form of black box testing


White Box testing :

• White box testing refers to the testing activity wherein code is used in the generation of the assessment of test case.

• A test case consists of inputs and expected outputs,one must use requirements to generate testcases, the code is used as additional artifact.

• There are techniques for generating testcases exclusively from code and expected output from requirement.

• This from of testing is structural testing.


White Box testing

Code could be used directly or indirectly to generate the testcases.

o In the direct case tool or human examines code and focuses on a given path to be covered. A test is generated to cover this path.

o In the indirect case, tests generated using black box technique are assessed against some code coverage criteria.

o Control flow and data flow can be used to generate testcases.


Interface Testing

• Tests are often generated using component’s interface.

• The interface is a part of component’s requirement and hence this is form of black box testing.

• Examples – Pairwise testing and interface mutation.

• Pairwise testing – The set of values for each input is obtained from the component’s requirement. It is black box technique.

• Interface mutation – Interface itself such as function coded in C serves to extract information needed to perform Interface mutation. It is white box technique.


• Adhoc testing – A tester generates tests from requirements but without the use of any systematic method.

• Random testing – tester uses systematic method to generate tests.

44

C1: Source of test generation


• Testing activtities takes place throughout the software lifecycle.

• Testing is categorized based on the phase it occurs

1. Unit testing –

- The testing of code written in early phase and before it is integrated with system components.

- Programmer focuses on unit or small component that has been developed.

- The goal is to ensure that unit function correctly in isolation.


2) Integration testing –

- When the units are integrated and a large component or subsystem forms, integration testing is carried out.

- Goal of integration testing is to ensure that collection of components function as desired.

- Most of Integration errors are discovered at this stage.


3) Sytem testing –

- When the entire system is built system testing takes place.

- Goal of system testing is to ensure that all desirwed functionality is in the system and works as per its requirements.


4) Beta Testing –

- Often a Carefully selected set of customers is asked to test a system before commercialization.

- In the case of contract software customer performs acceptability testing prior to making final decision as to whether to purchase the application for deployment.


5) Regression Test –

- Goal of regression testing is to ensure that modified system functions as per the specification.

- Regression testing may be performed using subset of the entire set of testcases used for system testing.

- Testcases selected for regression testing include those designed to test the modified code and any other code that might be affected by modification.

50


C3: Goal of specific testing activity

• Goal oriented testing looks for specific types of failures.

• Example – Vulnerability testing detects if there is any way by which the system under test can be penetrated by unauthorized users.

1) Robustness Testing

- Refers to the task of testing an application for robustness against unintended inputs.

- It differs from functional testing in that the tests for robustness are derived from outside of the valid input space whereas functional tests are derived from valid input space.


2) Stress Testing –

- One checks for the behavior of an application under stress.

- examples

1) Handling of overflow data storage.

2) Web application can be tested by stressing them with large no. and variety of requests.

- Conformance to functional as well as performance requirements under stress.


3) Performance testing

- Refers to that phase of testing where an application is tested specifically with performance requirements in view.

- ex. Compiler performance in terms of lines of code compiled per sec.

- Performance requirements are stated with respect to hardware and software cofiguration.


4) Load Testing –

- Refers to the phase of testing in which an application is loaded with respect to one or more operations.

- The goal is to determine if the application continues to perform as required under various load condition.

- Load testing can determine whether the application is handling exception in an adequate manner.

- Ex. Handling out of memory exception.

- Load testing is form of robustness testing i.e. testing for unspecified requirements.

- It allows to find threshold limit of an application.

C4: Artifact under test

• Testing techniques are named after the artifact being used.

• Batch processing applications pose a special challenge in testing.

• Ex: Payroll and student record processing application

• Sometimes an application might not be batch processing application but no. of tests may need to be run in batches. Ex. Embedded application.

• IBM’s Websphere is one of the several tool available for batch processing application.

57

C4: Artifact under test

C5: Test process models Testing in waterfall

model

Testing in

V-model

Spiral testing

Agile testing

Test-driven development (TDD)

Usually done toward the end of development cycle

Explicitly specifies testing activities in each phase of development cycle

Applied to software increments, proposed for evolutionary software development

Used in agile development methods, such as eXtreme programming (XP)

Requirements specified as tests

Testing in waterfall model

• The waterfall model is one of the earliest and least used software life cycle model.

• While verification and validation of documents produced in each phase is an essential activity, static as well as dynamic testing occurs toward the end of process.

• As the waterfall model adheres to sequential process, defects introduced in the early phase and discovered in the later phase could be costlier to correct.

• There is very little iterative or incremental development when using waterfall model.

Testing in waterfall model

Training and delivery

Requirement specification

Coding and unit testing

System testing

Design

Acceptance testing

Maintenance

Testing in V-model

• Testing activities begin from the start and continue until end of lifecycle.

• Test design begins soon after requirements are available.

Testing in V-model

Develop unit tests unit testing

System testing

Coding

Design

Integration testing

MaintenanceRequirem

ent analysis Accepta

nce testing

Validate requirementsDevelop system acceptance test

Validate designDevelop integration test

Spiral testing

• Spiral model and spiral testing are not same.

• While testing is a key activity in the spiral model, spiral testing refers to the test strategy that can be applied to any incremental software development process especially where a prototype evolves into an application.

• The sophistication of test activities increases with stages of an evolving prototype.

• Early stage in Spiral testing focus on test planning.

• Subsequent iterations refine requirements.

Spiral testing(Contd)• Once unit and integration test is performed, in final stage

when requirements are well defined, testers focus on system and acceptance testing.

• The cost of subsequent activities increases with subsequent iterations.

Spiral testing

Final test activities, application ready for acceptance test

Intermediate test activities

Initial test activities, early stages of development

Time

Cost

(prototype , evolves over time)

(cost of test activities increases with iterations)

The saturation effect

The saturation effect is an abstraction of a phenomenon observed during the testing of complex software system.

The horizontal axis in the figure refers to the test effort that increases over time.

The vertical axis refers to the true reliability and the confidence in the correct behavior of the application under test.

The vertical axis can also be labeled as the cumulative count of failures that are observed over time, that is, as the test effort increases.

The error correction process usually removes the cause of one or more failures.

Confidence and true reliability

Confidence in the figure refers to the confidence of test manager in the true reliability of the application under test.

An estimate of reliability , obtained using a suitable statistical method, can be used as a measure of confidence.

0 indicates lowest possible confidence and a 1 the highest.

Saturation region

Suppose that application A is in the system test phase. The test team needs to generate tests, encapsulate them into appropriate scripts, set up the test environment, and run A against the test.

The true reliability of A, with respect to the operational profile used in testing, increases as error are removed. Certainly the true reliability decreases in case that where fixing an error introduces additional errors-a case that we ignore.

False sense of confidence

The discovery and fixing of previously undiscovered faults might increase our confidence in the reliability of A.

It also increases the true reliability of A. In the saturation region when the expenditure of test does not reveal new faults, the reliability of A remains unchanged though our confidence is likely to increase due to a lack of observed failures.

∆1 in Figure is a measure of the deviation from the true reliability of A and a test manager’s confidence in the correctness of A.

Reducing delta ∆

Empirical studies reveal that every single test generation method has its limitations in the resulting test set is unlikely to detect all faults in an application.

The more complex an application, the more unlikely it is that test generated using any given method will detect all faults.

Now suppose that after having completed the test process using T1 we check how much code in A has been exercised.

Reducing delta ∆

Enhancement of tests based on feedback received from adequacy assessment leads to tests that are guaranteed to move A to at least a few states that were never covered when testing using T1.

Impact on test process A knowledge and appreciation of the saturation effect

are likely to be of value to any test while designing and implementing a test process.

Given that any method for the construction of functional tests is likely to lead to a test set that is inadequate with respect to code-based coverage criteria, it is important that test be assessed for their goodness.

73

Summary

We have dealt with some of the most basic concepts in software testing.

chapter 2 basics of software testing 02/06/12 © aditya p. mathur 2005

Documents