Testing and Inspecting to Ensure High Quality

Part 1:

Basic DefinitionsEffective and Efficient Testing

Lots of Personal Notes

2

OverviewTesting and Inspections – two very important concepts / activities in software development.

We often have ‘roles’ of testers; individual developers; for different ‘kinds’ of testing; independent test groups; quality assurance groups, etc.Quality Control; Editors; Various sizes and shapes….Very unique to the environment within which the software is developed.

We have many different types of tests too.Exist at many different levels for many different stakeholders.Some are automated; many are hands-on…Some tests test specific parts of system; others whole;Some at different stages of development; ….Many Many names!!!!

Heuristic: a successful test is one that identifies a flaw. We will present a lot of very important material in these lectures.

3

Tests – just a few types of tests….White box testing; Black Box Testing

Verification and validation (V&V)

Coverage Testing:

Branch testing, Path testing, Statement testing

Alpha testing, Beta testing

Acceptance Testing; Validation Testing

Unit test, Subsystem test, System test, Integrated tests

Environmental System Testing; Operational Field Testing

Running in parallel; Running live; Implement in pieces.

4

Organizational Considerations

Deficiency Report (DIREP) Burroughs Hardware / UNISYS – years ago….

Field Assistance Branch; Customer ServiceTracked, prioritized, resources allocated...Reported to senior levels of management

Incident Report – Honeywell:

Problem Ticket – Fix Tickets (LPS)

Office of Primary Responsibility (OPR); Office of Collateral Responsibility… (OCR)Product Manager; Account Rep…

5

Basic Definitions A failure is an unacceptable behavior exhibited by a system

—The frequency of failures measures the reliability

—Important design objective: achieve a

- very low failure rate and hence

- a high reliability.

Basic Definitions: Failure Causes:

A failure can result from a violation of an explicit or implicit requirement

Specific —functional requirement; —non-functional requirements…

6

Basic definitions: MTBF MTTR Metrics

We clearly want to increase the Mean Time Between Failures (MTBF) and reduce the Mean Time to Repair (MTTR):

and damage caused by failures.

MTBF and MTTR (old engineering metrics): are still considered measures of reliability.

7

© Lethbridge/Laganière 2001 8

Basic Definitions - continued

A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures

—Might take several defects to cause a particular failure —Defects can occur anywhere – requirements, design,

implementation, testing, etc.—They can also occur at any time!

True ‘War’ Story: “Mr. Smith said this could never happen!”

An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect

9

Effective and Efficient Testing

To test effectively, you must use a strategy that uncovers as many defects as possible.

Testing really attempts to find defects!

Pure & simple!

Effective and Efficient Testing

To test efficiently, find the

largest possible number of defects, using the

fewest possible tests.

Testing costs time, money and other resources!

Testing is not cheap! It is quite expensive!

This is where the problems come in. What is the expected return on investment?

There are many kinds of tests designed to uncover different kinds of defects.

10

11

Effective and Efficient Testing – more… Testing must be efficient. So what do we test??

We cannot test everything.

(Exhaustive Testing is not possible – proven mathematically)

Rather, we design different kinds of tests for ‘sufficient coverage.’

All kinds of tests: Many Categories

Some are lumped into Coverage Tests.

statement coverage

branch coverage

path coverage…..

Others tests focus on creating appropriate test values (Equivalence Testing)

Verification and Validation (V&V)

Developers usually do their own testing first (verification)—Usually unit testing; testing within development team. —May involve subsystem testing, integrated testing. … —But, in general, the developers are testing their products.—Generally their testing is white box testing, but can be

black box testing.

End users test system against requirements (validation)—This is essentially called black box testing, although end

users may choose to call it validation testing.

12

Black-Box Testing

Testers provide the system with inputs; observe outputs

They can see none of: — Source code— Internal data (no algorithms or data structures, …)— Design documentation describing the system’s internals

They simply do inputting based on requirements and observe outputs.

Testing Charge: Often depends on the kind and level of testing and what the ‘charge’ of the testing group (organization) really is.

13

Black-Box Testing - more Kind of testing end-users most frequently undertake – for obvious reasons.

Sometimes called

validation testing / acceptance testing - IF done when system is declared ‘finished.’

Black box testing may sometimes be called other things (alpha testing; beta testing…) but these tests (alpha and beta) usually have other connotations associated with them…

Black Box Testing is designed to show functionality – and often (sometimes) satisfaction of non-functional requirements. – (such as load considerations; simultaneously users, distribution, more)

15

Glass-Box Testing – White-Box Testing

Also called ‘structural’ testing;

Developers and Testers have access to the system design

They can —Examine the Design and these Documents —View the code for Standardization / conformance!—Observe at run time the steps taken by algorithms and

their internal data

Individual programmers often informally employ glass-box testing to verify their own code and functionality

Developers (using tools) can tell what parts of their programs are executed the most (most time spent in them) and then these may be candidates for optimization….

16

Glass-Box Testing - White Box Testing Unit Testing

Developers are always responsible for their own ‘modules’.

modules (units) = programs, classes, subsystems, etc.

Developers ‘verify’ their products.

(may undertake both black box and white box testing…)

Developer programs implement the design

Programs satisfy functional requirements, etc…

A lot of work to undertake ‘unit testing.’

Hazardous though, since they may be required to

simulate inputs, outputs, files passed,

parameter values passed, etc….

Many kinds of unit testing…

17

Glass-Box - White Box Testing - More

Path Testing – exhaustive; impossible; there is an infinite number of paths in a non-trivial program.

Branch testing – Edge testing – the most feasibleDesign a test so that all of the edges of a node are executed.Often shown via flowgraphs. (ahead)

What do we mean by ‘edge?’ (all ‘else’ branches taken; all ‘then’ branches executed…)

Consider a flowgraph of an algorithm (two slides up) …. Emphasis will be on control and coverage.

18

White Box: Branch Testing - Flow Graphs

Want minimum number of tests assuring high degree of reliability.

Know we cannot test all paths in a non-trivial program. Want a ‘representative set.’

Given a flow graph (next slides), there are a couple of formulas that will provide the minimum number of tests . They are:

(number of nodes – number of edges + 1) Number of Regions + 1 ….(cyclomatic complexity)

Number of Regions is considered to be a measure of complexity.

19

http://www.scism.sbu.ac.uk/law/Section5/chap3/s5c3p22.html

Consider: The number of possible paths through a subprogram is equal to the number of regions in the subprogram's flowgraph.

The flowgraph below is for a SquareRoot function and has regions numbered, as shown.

Can see there are four regions, as numbered, plus the outer region, or 5 in total.

20

http://www.scism.sbu.ac.uk/law/Section5/chap3/s5c3p22.html

This indicates that there are five possible paths through the flowgraph which can be described by listing the sequence in which the nodes must are visited. The five paths in the SquareRoot flowgraph are as follows.

etc…

21

White Box: Equivalence Class Testing

This is a Bigee!

It is inappropriate to test by brute force, using every possible input value

—Discuss; Takes a huge amount of time —Is impractical and is pointless!

Far better: divide the possible inputs into groups which you believe will be treated similarly by all algorithms.

—Such groups are called equivalence classes.

—Remember Computational Structures?? (COT 3100??)

—What does the equivalence relation, ‘equivalence mod 4’ do for you? (Be nice!!) What does imposing this relation on the set of positive integers do?

22

Equivalence Class Testing

Recall ramdomizing in hashing for tables and file structures

If we used an ‘Equivalence mod 4’, we essentially divided the set of all non-negative integers by four and only take their ‘remainders’ 0, 1, 2, and 3.

If a numeric key was used as a primary or relative key, this approach was often used to arrive at a disk address or an address in a hash table.

Thus ‘Equivalence mod 4’ effectively partitions the set of all integers into one of five disjoint equivalent classes.

ALL division operations result in one of these remainders – and those having the same remainder (members of the same equivalence class) need only require one of its members to be tested to represent ALL values that fall into this equivalence class..

We applied this equivalence relation to the set of integers and obtained a set of equivalence classes the Union of which constitutes the original set of Integers.

—EC1 U EC2 U … ECn = set of I+ (non-negative)

23

Example of Equivalence Classes

—A tester need only to test one member of an equivalence class rather than ALL numbers of that class.

—The tester has to - understand the required input, - appreciate how the software may have been designed

Example:—Valid integer input is a month number (1-12)—Equivalence classes are: [-∞..0], [1..12], [13.. ∞]

These are the three equivalence classes.

In Equivalence Class Testing, we select an input from each of the equivalence classes as inputs to testing.

This may not be sufficient testing, but it is a good start….

24

More on Equivalence Class Testing

To test for valid integers ranging from 1 through 12, we have three equivalence classes as indicated:

—Integers less than 1;—Integers between 1 and 12 inclusive—Integers greater than 12.

Is this enough?

Equivalence Class Testing cannot handle all kinds of testing:

What if there is more than a single value we are testing?

25

Combinations of Equivalence Classes

Combinatorial explosion means that you may not be able to realistically test every possible system-wide equivalence class.

Let’s say there are 4 inputs each of which have 5 possible values. This means that are 54 (i.e.625) possible system-wide equivalence classes.

While you should first make sure that at least one test is run with every equivalence class of every individual input, you should also test all combinations where one input is likely to affect the interpretation of another.

Let’s look at another type of test….

26

Compatibility Tests; Associativity Tests

• Compatibility tests; Associativity tests. —Example of Associativity Test: —Example from a Manpower Project in the USAF.

- Field/attribute: Rated Performance Indicator (rpi); text field; permissible values: 0 or other integer such as 1

- Rule: If the rpi input value for an individual had a value of non-zero (individual had to be on an aircrew), then the ‘rank’ attribute for this individual (s/he must be an officer), must be in the range between O1-O10 (and could NOT be in the range E1-E9).

- (O officer rank; E Enlisted rank)

—Example of Compatibility Test: - Range tests: If rank = ‘O4’,

» salary must fall between $nnnn and $nnnnnn.

• Many others and many different types of tests.

27

Testing at Boundaries of Equivalence Classes Boundary Value Testing

More errors in software occur at the boundaries of equivalence classes than at any other place!! AMEN!!!

The idea of equivalence class testing should be expanded to specifically test values at the extremes of each equivalence class

—E.g. The number 0 often causes problems

E.g.: If the valid input is a month number (1-12)—Test equivalence classes as before

- Test -24; 6, +24, (representative values) —Test boundaries: 0, 1, 12 and 13 as well as very large

positive and negative values—Are only integers allowed? (perhaps another test)