defects as assets: how to make your defects work to your ...€¦ · defect management opinion...

ROQ IT Company Confidential 2014 of 13

Defect Management Opinion piece

Defects as Assets: How to make your defects work to

your advantage

1. Introduction

The importance of rigorously managing the defects found during formal testing of your

system/software has been recognised for many years. Furthermore, the necessity of having a defined

workflow to ensure each defect is processed correctly in order to maintain quality is widely accepted.

Many organisations therefore have adopted a process and a supporting tool, whereby a defect is

allocated by priority/severity and then tracked accordingly through to a successful conclusion.

Nevertheless, these same defects are often regarded as a barrier and even an irritation, perhaps

understandably, by those working on projects with tight timescales or those trying to deliver essential

updates to mission critical applications.

We suggest however that a change of mind-set is required to regard your defects more as assets,

analyse them carefully and ultimately as a result derive maximum benefit in terms of process

optimisation, both in the testing arena and the wider Software Development Life Cycle (SDLC).

2. Having a stand-alone strategy for defect management

2.1. Why is it important to have a separate defect management strategy?

In order to establish a consistent and rigorous approach to the management of your defects, it is

important to have an overall defect management strategy that describes how your organisation will

manage defects. This should be applicable to all testing, regardless of whether it forms part of major

projects introducing new systems, upgrades to existing systems or on a smaller scale delivery of

fixes/enhancements. For this reason, the strategy should sit outside Master Test Plans and project

/programme specific documentation.

This strategy is particularly important for organisations who want to ensure consistency whilst running

many programmes/projects that might be in parallel, potentially working with a number of different

developing and testing partners.

2.1.1. What should the content include?

The content for this overarching defect management strategy should include the following:

Introduction – broadly setting out scope and purpose of the document

What is a defect? – setting out the organisations philosophy around definition of a defect and

therefore what should be logged. Avoids a situation where only strict software failures are

logged when we need to log requirements, design and other.

Defect Management process and workflow description – the lifecycle of a defect covering

all possible paths and roles at each step.

Guidelines for those logging and re-testing defects – ensuring the quality of the defect

records– see section 1.3 below.



Defect Priority and Severity – definitions of the various classifications to guide the tester

and help ensure consistency both within and between projects. Without these guidelines,

individual judgement may lead to serious variations in how the defects are classified. Bringing

consistency should in turn support decision making by senior stakeholders when outstanding

defects are under consideration.

Defect Management key roles and responsibilities – whilst every project/programme is

slightly different in terms of organisation chart/personnel, the key roles to be played by the

test manager, defect manager, testers, development lead and team, solutions architect,

release manager and environment specialists, and business leads should be spelt out. NB.

For a given project an individual may play more than one role.

Defect meetings – these are commonly referred to as Defect Triage meetings. The strategy

should spell out the purpose, who should attend and a proposed example agenda.

Defect reporting and metrics – the strategy should indicate the reporting that might be

expected as part of the Daily Execution report. Typically the defect are summarised by priority

and severity as well as the current status to given an overall picture alongside execution

progress metrics. An example of the defect content of the daily report might be included in the

Appendix section of the strategy for reference.

Examples are given below:

Priority

Defect Status Critical High Medium Low

Closed

Deferred

Failed re-test

Fixed

New

Open

Ready to re-test

Rejected

Re-open

Total



Severity

Defect Status 1 2 3 4

Closed

Deferred

Failed re-test

Fixed

New

Open

Ready to re-test

Rejected

Re-open

Total

Aged defect report

Number of Defects by time since first raised

Defect Status <1 day 1-5 days 5-10 days >10 days

Failed re-test

Fixed

New

Open

Ready to re-test

Rejected

Re-open

Total

Please also refer to section 1.6 for cumulative defect chart (S curve) which may be included in your

daily report.

3. Best practice when logging defects

3.1. Guidelines for all those logging defects

Defects can be raised by any team member involved in software/systems testing but most often by

the testing team(s), operational accepting testers (technical owners conducting Operational

Acceptance Testing (OAT)) and business users (during User Acceptance Testing (UAT)).

When raising a defect, it is important that as much relevant information as possible is captured as

clearly and concisely as possible and communicated in a way in which all involved parties can

understand. This enables faster resolution through the defect management process. Wherever

possible, raise the defect in your chosen defect management tool at the point of failing a test step.

By doing this, key information will be included in a timely fashion in the defect log including details

about the test script, step, expected result and actual result.



The following guidelines can be issued to your testing team(s) in order to ensure the quality of your

defect records and we would suggest including something similar in your defect management

strategy.

Reproducible If the defect is reproducible, detail the steps required to do this. If

not reproducible, you should try at least 3 times.

The defect should be recorded regardless of whether or not it can

be reproduced. If the defect is intermittent, please include this

information.

Isolated Does the defect only occur in specific situations? For example,

when a particular user is logged in, only on a certain environment

or at a particular time of day.

Generalise Is the same problem likely to occur in other areas of the system?

For example, other screens reports or other data entry.

Compare Does the defect occur on other versions of software, operating

systems, previous releases or in the live environment?

Neutralise Ensure just the facts are reported as they stand. Do not use

phrases, words or comments that could be considered to be

emotive or sarcastic.

Impact Provide an assessment of the impact of the defect both to test

execution progress and if the defect occurred in the production

system. This assessment will be further qualified by the mandatory

Priority and Status values assigned to the defect.

Reference: The Testing Practitioner - Part 5 Incident Management, Eric

VanVeenendaal



4. Global UAT and associated defect management

considerations

On one recent programme ROQ IT have been engaged to provide management and support for UAT

where the users span 220 countries, many different cultures and time zones which provides additional

challenges.

Key considerations in this situation can be grouped as follows:

Process – It is more important than ever to have a robust defect management process in

place and clear role definitions during UAT. Consider whether the User/testers will log their

own defects or do so through a test analyst. The mechanics of this need to be spelt out and

communicated effectively during UAT briefings/training.

Scope – Needs to be clearly understood including answering the following:

o Who are key contacts/business leads that should attend defect triage? Who may be

required on a more ad hoc basis?

o Who represents which area and give a view on the business impact of defects?

Defect Management day-to-day challenges

o Ensuring access to your defect management tool for all, when the testers are spread

across the globe (including any licencing considerations).

o Dealing with duplicates becomes more of a challenge when those logging the defects

have very different backgrounds.

o Standards/guidelines for bug reports become even more important given cultural

differences. There could potentially be great variation in the classification, rigour and

detail that are completed without the strong guidelines.

Please refer to our webinar material “Global Testing Programmes UAT – Not for the Faint

Hearted” for more detail.

4.1.1. Defects open when you move from Unit Testing to System Testing

We would strongly recommend that if the developers/technical resources are aware of any defects

that have not been resolved at the point when the solution moves from unit test (or similar) into

system testing that these are logged appropriately. This is particularly important as they may have

become apparent during “white-box testing” requiring detailed knowledge of the code/database and

not be immediately apparent to the testers.



5. Common pitfalls

5.1. Defect “Ping-Pong”

All too often the defects are re-assigned several times between the builder/developer and the

individual tester that has raised the defect before any attempt to resolve the defect is made. Part of

the solution to this problem lies in the recommendations and guidelines for those logging defects in

section 1.3.1 above. If your defect report provides concise, accurate information to the resolving

individual it is much less likely that it will be rejected and returned to you as a tester. The rest of the

answer to avoid this is most likely in the building of that important relationship between the teams,

effective communication (perhaps taking the time to actually call the developer and discuss the defect

once in a while rather than just impersonal e-mails).

5.2. Status – Resolved

A typical “out of the box” defect management workflow for your chosen tool might resemble the

following:

A key problem that you may encounter with the process illustrated above surrounds the status

“Resolved”. In this status, has the defect actually been deployed to the test environment and become

available for re-testing? Alternatively does it just indicate that the defect has been addressed in a

development environment (or even the developer’s workstation)?

For the test manager and the project manager, understanding where the potential bottleneck is

developing could be important. Put simply, should they be chasing the testing team to improve the re-

testing/turnaround of fixes or does the problem lie with the speed of deployment into the test

environment of the fixes?

We would recommend that for all reasonably complex projects, an additional status “Ready for re-

testing”, “Deployed to test environment” or similar is introduced to improve the transparency of the

overall defect situation and as a result improve the decision making.



6. What standards exist to help us in managing defects?

6.1. IEEE 1044

This standard has been around for some years but was last updated in 2010. The purpose of the

standard is to establish a common language for those involved in addressing software defects and

failures and also establish a classification system by defining the essential attributes.

In the body of the document, in section 3.2 and 3.3, the standard describes in detail the attributes that

should be recorded for a system/software defect and then a system/software failure respectively.

6.2. ISO/IEC 29119

This is a new standard that was released in 2013 after work commenced back in 2007. The intention

of this standard was to replace the previous plethora of disparate standards (BS 7925-1-2, IEEE 829

and IEEE 1044) with a single comprehensive standard that should be applicable regardless of the test

model. Despite the time devoted the important area of static testing seems to have been omitted as

no consensus could be reached.

The standard is split into five parts; with part two focussing on test processes. A section on Dynamic

test processes is where we can find the test incident reporting process described.

For more information on these standards refer to:

http://www.testingstandards.co.uk/

http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=5399058

http://www.softwaretestingstandard.org/

http://www.bcs.org/upload/pdf/sreid-120913.pdf

7. Defects as Quality Gate Exit Criteria

7.1. Formal Exit criteria

When determining whether a given test phase can formally exit and pass a quality gate, some of the

key criteria are likely to be around the status of the defects found. In the Agile world this might

translate into a “definition of done” that all the team have accepted up-front rather than a Quality Gate.

Examples of system testing exit criteria related to your system defects might be as follows:

“All priority one and two defects that have been detected have been closed to the satisfaction of

the key stakeholders.”

It is more than likely that given the definition of priority one and two defects, the projects should not

proceed to subsequent testing phases (UAT/OAT and similar) with these type of defects open.

http://www.testingstandards.co.uk/

http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=5399058

http://www.softwaretestingstandard.org/

http://www.bcs.org/upload/pdf/sreid-120913.pdf



“Not more than 15% or 20% of the total number of Priority 3 and Priority 4 defects are not closed,

whichever is the greater.”

We would recommend including a similar criteria in your test plan and subsequent quality gate

documents to the one above in order to reflect the fact that whilst individually each of these priority

three and four defects may not showstoppers, if you have a large number of such issues they can

become a serious quality issue and render the system very difficult to use.

7.2. Defect S-curves

When plotting the cumulative numbers of defects over time this curve would typically follow a

traditional “S” shape. As the testing starts to ramp-up, the frequency of defect detection makes a

relatively slow start and then starts to pick up speed as testing gets into full speed before hopefully

starting to tail-off as the system becomes stable and we have driven out the vast majority of defects.

Plotting the frequency of defects in this manner and observing the curve can be a useful guide to

decision making. Effectively when the curve starts to flatten out, providing testing is still occurring at

the same level of intensity, it could be an indication that we are approaching a position where we

should move onto the next phase of testing or even consider the system for pilot or production

release. However, whilst the curve still has a strong upward inclination, the likelihood is that we are

still likely to find more defects as we continue to test (sometimes referred to as the discovery stage).

This sort of visual guide may be a useful aid to a test manager making a case for the additional

regression testing that is required for example. The S curve could also be used to trigger early

intervention if for some reason the testing is not working or indeed if testing should be suspended as

the number of defects is overwhelming and not permitting progress.

0

20

40

60

80

100

120

07

/03

/20

14

11

/03

/20

14

13

/03

/20

14

17

/03

/20

14

19

/03

/20

14

21

/03

/20

14

25

/03

/20

14

27

/03

/20

14

01

/04

/20

14

03

/04

/20

14

07

/04

/20

14

09

/04

/20

14

11

/04

/20

14

15

/04

/20

14

Cumulative defects raised over time - Example curve

No of Defects raised

Cumulative



8. Using your defects to drive process Optimisation

8.1. Improving the test process

Many organisations have process improvement programmes and use models such as the Testing

Maturity Model (TMMi) or Test Process Improvement Next (TPI Next) to decide where to devote their

resource/efforts to improve. However, how do we know if all this effort is bearing fruit? Analysis of

your defects can play a major role in answering the question about how effective your testing has

been. The “proof of the pudding” could be determined by these defects and what they tell you about

the improvement experienced.

8.1.1. Defect Leakage

One measure of testing effectiveness for given phases is defect leakage. Essentially analysing your

defects and determining if these should, in reality have been found in an earlier phase.

Examples of leaked defects would include the following:

Any defects found in Production

Functional defects found in UAT

Defects found in System Testing that should easily have been found by the Developer (Unit

testing)

Design gaps found during Component Testing (should have been found in static testing – reviews

of relevant functional/technical specifications)

Typically this would be expressed as a percentage as follows

Defect Leakage = (Defects leaked / Total number of defects found by this phase) * 100

Example Defect Leakage matrix

Document Review Component

and

Component

Integration

Testing

System

Testing

System

Integration

Testing

User

Acceptance

Testing

Performance

Testing

Operational

Acceptance

Testing

Production

Acceptance

Testing

Document Review 2 2 0 0

Component and

Component Integration

Testing 1 19 20 1 5

System Testing 1 5 128 134 6 4

System Integration

Testing 0 3 2 107 112 5 4

User Acceptance

Testing 3 1 7 4 72 87 15 17

Performance Testing 0 0 1 2 0 81 84 3 4

Operational Acceptance

Testing 0 0 0 0 0 0 29 29 0 0

Production Acceptance

Testing 0 0 1 1 2 1 0 9 14 5 36

Production 0 0 0 1 1 0 1 0 3 3 100

Total 7 28 139 115 75 82 30 9 485 38 8

Defect Leakage Matrix

Leakage

Percentage

Test Phase detected in Test Phase Leaked From Total

Number of

defects

Total

Number of

Leaked

Defects



8.1.2. Defect Detection Percentage (DDP)

A further measure of how effective you’re testing has been in reality is the Defect Detection

Percentage (DDP). Whilst in some respects this might be regarded as quite a crude measure it can

reveal important trends and reveal some truths about the testing process

Calculation as follows:

DDP = (Defects found by this testing phase / Defects available to find) * 100

8.1.3. Defects found by testing phase – Example

Priority

Testing Phase Critical (P1) High (P2) Normal (P3) Low (P4) Total DDP

System testing 211 385 200 419 1215 83.45%

Systems Integration testing 26 56 31 9 122 52.14%

UAT Phase 48 11 32 21 112 94.12%

Production Running 2 4 1 0 7

In this example, the defects have been broken down by priority and you may wish to perform the DDP

calculation just for those defects that have been allocated particular priorities. It may also be sensible

to exclude some types of defect when performing your analysis. For example, you may wish to

exclude environment specific defects (having acknowledged the impact that these have had on your

testing effectiveness) to focus on functional defects.

This measure can also be used to look for trends as you look to improve your testing process and

want to understand what might be working or otherwise.



8.1.4. Example – Defects Detection by phase over three releases

.

8.2. Analysis of production defects

The initial defects found during the first month or so after production release (commonly referred to as

“hypercare”) can also be very strong assets in terms of identifying weakness in our testing process

and where we might need to improve.

1. Analyse the Support calls logged with the Service desk

2. Perform Root cause analysis (”What caused this issue?”)

3. Categorise the issues

a. Which should have been found in testing?

b. Which were known issues that were actually detected in testing? (To some extent we

might regard these as “self-inflicted wounds”)

c. Which defects could not have been realistically found in the testing?

4. Then ask the question: “What does this tell us about our testing process?”

a. Which phase of testing is supposed to be designed to detect each of these in turn?

b. Why did that phase not detect this?

0

10

20

30

40

50

60

70

80

90

100

Release 1 Release 2 Release 3

Production Running(after onemonth)

Production Acceptance Test

UAT Phase

Systems Integration testing

System Testing

Number of defectsfound by phase



8.3. Root Cause Analysis to drive process optimisation

As each defect is closed, the root cause should be allocated to the defect to help perform the

analysis. Whilst every organisation and solution is different a possible classification is shown below:

Root Cause Defect numbers by Root Cause

Incorrect configuration 8

Data Related 6

Code Related 15

Not a defect 4

Core Solution problem 6

Test Specification error 3

Requirements Specification 12

Screen Navigation 1

Environment Issue 4

Solution/Process Design Related 10

59

Incorrect configuration

12%

Data Related9%

Code Related22%

Not a defect6%

Core Solution problem

9%

Test Specification

error4%

Requirements Specification

17%

Screen Navigation

1%

Environment Issue6% Solution/Proces

s Design Related

14%

Defect numbers by Root Cause



This form of analysis is particularly important as certain organisations have “patterns of failure” and

effective blind spots to the weakness in their process. This sort of analysis should indicate if, for

example, the requirements process and sign-off mechanism. This is another instance where your

defect might truly be regarded as an asset in guiding process improvement.

9. Summary

Regarding your defects as assets and using them to guide your efforts should be an important part of

your process improvement strategy. We can learn a great deal about how we need to improve from

analysis of our defects. Both in the short term (intervention when testing may not be not going so well)

and long term (strategy and process improvement).

The Kolb learning cycle, widely used in the ITIL/Service management context, can also be adapted

and used in this context as illustrated in 9.1 below to help gain maximum benefit from our defects:

9.1. Kolb learning cycle applied to defect analysis and process improvement

defects as assets: how to make your defects work to your ...€¦ · defect management opinion...

Documents