john fodeh - spend wisely, test well

© 2009 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

SPEND WISELY, TEST WELLMaking a Financial Case for Testing

HP - Global Testing Practice

John Fodeh

[email protected]

Introductions

About You (and Me)

Briefly introduce yourself by telling us:

• Your name

• Your testing role

• How long you’ve been in the testing arena

• One thing you’d like to get out of this Workshop

About This Session

After you’ve completed this workshop, you should be able to:

• Address and even turn around the prevailing view of testing among business stakeholders

• Through the effective use of industry data:

− Establish and/or reinforce your credibility

− Substantiate your testing recommendations and strategies

• Demonstrate to business stakeholders – in terms they can understand and accept – the relative cost of testing –and not testing – for a specific project

• Credits: Ian Howles & Susan Herrick!

Administrivia

• The duration of this workshop is 1.5 hours

• There will be no breaks

• Ask questions whenever they occur to you

It’s All About Communication

• Remember, testing is a service organization.

• Our mission is to provide information about

quality and risk to decision makers within our

organization.

Testing

The Remainder of the Organization

Forgetting the Organizational Context

From The Nine Forgettings by Lee Copeland

• Testing does not create quality. All we do is look for it.

Forgetting the Organizational Context

• Testing should not be the Ship / Don’t Ship decision maker. That is not our stewardship.

The Business Spectrum

• Mass-market packaged software

• Market-dominating software

• Niche software

• High consumer tolerance

• Custom software

• Market-entry software

• Competitive software

• Low consumer tolerance

• Regulatory compliance

• Critical safety requirements

Defect Tolerance

High Low

Perfection is the enemy of the good

- Voltaire

Cost of Quality

Quality Perspective

C Quality = C Conformance + C Non-Conformance

WHERE

C Quality = Cost of Quality

C Conformance = Cost of Conformance

• Cost of prevention

• Cost of detection

• Cost of internal failure

C Non-Conformance = Cost of Non-Conformance

• Cost of external failure

• Other related costs

Cost of Quality

Business Perspective

PV – C Quality = NV

WHERE

PV = Potential Value, e.g., ALL sources of revenue associated with a software product, annualized cost reduction that results from productivity improvements

NV = Net Value, e.g., actual revenue for the software product, actual annualized cost reduction

AND

PV = PM or ROI (% + or -)

NV

WHERE

PM = Profit Margin

ROI = Return on Investment

It’s All Greek to Me!

The language of Testing focuses on:

• Quality

• Defects

• Test coverage

• Risk

The language of Business focuses on:

• Scope

• Time

• Money

• Risk

• (and, oh, yes, Quality)

Bridging the Communication Gap

• The decision-makers will tell you they care about quality and “customer satisfaction” but not if they believe that achieving either will adversely affect their bottom line

• Talking about defect discovery alone isn’t enough

• Testing has to be able to:

− Demonstrate the financial consequences of testing involvement − or lack thereof − in a project

− Substantiate the financial consequences with reliable data

Show Me the Money

Testing requires financial investment in:

− Resources, tools, equipment

− Time to develop, test, manage

− “Escaped” defects

− Rework and retesting

Business stakeholders legitimately expect a healthy return on that investment

Defining the Communication Challenge

Lets quickly discuss some views of testing that:

• You hear from decision-makers in your organization and/or on your projects

• You would like to change

Prevailing View vs. Desired View

• Testing is a back-end project phase

• Testing is easy and anyone cando it

• Significant rework is inevitable

• Testing provides advice that can be selectively ignored

• Testing creates bottlenecks and increases the risk of schedule overruns

• Testing costs too much

• Testing is an integral aspect of all phases in the product life cycle

• Effective and efficient testing requires skilled testing professionals

• Focusing on early defect discovery and prevention can significantly reduce rework

• Testing provides information that quantitatively specifies outstanding risk, “application readiness” and potential cost — ignore it at your peril!

• Testing early and testing often can actually reduce the risk of schedule overruns and accelerate time-to-market

• Not testing costs way too much — in rework (pre- and post-release), failures, and production support

Exercise:

Turning a Problem into a Solution

• The following exercise is derived from a tutorial developed by Steve Bender of The Quality Connection (www.qualityconn.com):

Effective Methods to Develop Creative Approaches to Today’s Challenges

• The exercise requires you to:

− Define a problem

− Define a solution that is:

• Stated in positive terms

• Under your control

• Integrated

• Testable

Exercise

Step 1 - Defining the Problem

Take 5 minutes to answer these questions.

1. What’s the problem?

2. How long has this problem been a problem?

3. Why does this problem exist?

4. How are you and/or your team limited by this problem?

5. What is the worst thing this problem has caused?

Exercise

Step 2 - Defining the Solution

Take 5 minutes to answer these questions.

1. What do you want? Be as specific as possible.

2. How will you know when you get it?

3. Where, when and with whom do you want it?

4. When you get it, what else will improve?

5. What resources do you already have that will help you get it?

6. What is the first step to take to get it?

Defining the Problem

An Example

1. What’s the problem?Perception that testing costs more than it’s worth

2. How long has this problem been a problem?For as long as I can remember (15+ years)

3. Why does this problem exist?No real proof to the contraryTesting world does not speak the language of business

4. How are you and/or your team limited by this problem?Inadequate allocation of time, money, resourcesMarginal influence over allocation decisionsLack of senior management support

5. What is the worst thing this problem has caused?5-10% of project effort allocated to testingUnnecessary schedule and cost over-runsDefective software in production

Defining the Solution

An Example

1. What do you want? Be as specific as possible. “Real”, tangible proof that the decision-makers “get it”

2. How will you know when you get it?Commitment through actionProjects - Upfront testing involvement in planning, scheduling, impact analysisSenior management – Testing Policy and Strategy

3. Where, when and with whom do you want it?Senior management and project “purse string holders”

4. When you get it, what else will improve?Testing involvement and influenceProduct quality, cost, speed to market

5. What resources do you already have that will help you get it?Industry dataAccess to audience

6. What is the first step to take to get it?Pull together the dataFilter and home in on compelling “proof”

Communication Principle

The meaning and merit of the communication is in the response it elicits

The Proof is in the Data

I’m from Missouri … show me

In God we trust – all others bring data

- W. Edward Deming

A Common Modus Operandi

If you don’t know where you are going, any road will get you there

- Lewis Carroll

An Alternative Perspective

• If you don’t know where you’re going, how willyou know when you’ve arrived?

OR

• If you don’t know what testing – or not testing – is really going to cost you, how can you know if you’ve spent wisely?

Show Me the Value

Two axioms of management are:

− What gets measured gets done

− What gets valued gets funded

So, it's important to remember that testing delivers valuethat we can ― and should ― measure

- Rex Black

Addressing the Challenge

• Focus on the financial impact of defect resolution through testing activities throughout the product life cycle

• Use the following industry data to substantiate claims about the financial impact of:

− Defect Insertion Rate

− Amplification Factor

− Defect Removal Effectiveness

− Time to Resolve a Defect

− Cost of Defect Resolution

− Rework Rate

• Address the assumption that “automation solves the problem”

• Demonstrate the cost of schedule compression and inadequate testing staff (numbers and experience)

• Quantify the cost and cost avoidance associated with testing – or not testing

Some Definitions

• Defect insertion rate defines the defects introduced in a project phase as a % of the defects discovered throughout the product lifecycle

• Amplification factor is based on the premise that every defect introduced in a project phase that remains undiscovered migrates to the next project phase AND introduces additional defects

• Defect removal effectiveness is a quantitative measure of the effectiveness with which each discrete testing activity finds defects and supports their resolution

Some More Definitions

• Time to resolve a defect is based on the time to discover and remove one (1) defect by project phase in the product life cycle

• Cost of defect resolution is based on the cost of discovering and removing one (1) defect by project phase in the product lifecycle

• Rework rate is the effort associated with fixing and retesting defects introduced in a project phase as a % of overall effort associated with fixing the defects discovered throughout the product lifecycle

Defect Insertion Rate

Requirements56%

Design27%

Other10%Code

7%

“56% of defects are introduced in the requirements phase of a project.“

From James Martin

Amplification Factor

Requirements

1 defect 1.5 defects in Design

Design

1 defect 1.25 defects in Code

From What IT Managers Should Know About Testing ROI by Rex Black

Defect Removal Effectiveness

Each discrete testing activity can detect and support resolution of 30% of latent defects:

Defects that escaped from the previous testing activity

AND

Defects that were introduced during the project phase associated with the current testing activity

A latent defect is a defect that is present and capable of becoming visible, obvious, active, or symptomatic

From Capers Jones

50

100

200

500

Requirements Design Code System Testing

Acceptance

Testing

Operation &Maintenance

1

Cost of Defect ResolutionIn

cre

menta

l Costs

Product Life Cycle Stage

From Boehm & Baziuk, Standish Chaos Report

On large, complex projects, 50-70% of project effort is devoted to rework − an avoidable cost

Rework is the root cause of project bottlenecks, schedule delays and rising costs

Time to Resolve a Defect

Rework Rate

Requirements82%

Design13%

Other4%Code

1%

“82% of effort in fixing defects is caused by inadequate requirements.“

From James Martin

Time is Money

Design Review Code Inspection Testing

8.44 hours 1.38 hours 0.17 hours

Collofello and Woodfield (1989)

For every hour spent on…

Putting it all Together

Putting the Data to Work

• Each of these data sources provides thought-provoking insight, but not enough to make a compelling, quantitative case for testing ― or not testing

• Pulling the data together can help your business stakeholders understand:

− Whether testing will really cost more than not testing

− How much and what types of testing will reduce cost and maximize value through profit and/or return on investment (ROI)

Time, Cost and Value Calculator

• Considers both pre-release and post-release activities

• Classifies testing activities as:

− Static Testing

Testing of a component or system at the specification or implementation level without execution of that software, e.g. reviews or static code analysis

− Dynamic Testing

Testing that involves the execution of the software of a component or system.

From Standard Glossary of terms Used In Software Testing, v 2.0 - ISTQB


Static Testing

Testing Category

Potential # of Discrete Testing Activities

Typically Includes:

Requirements Verification

2 • Ambiguity and Risk Analysis

• Content Review

Design Verification

2 • High-Level Design Review

• Detailed Design / Prototype Review

Code Verification 2 • Code inspection (manual)

• Code complexity analysis (automated)


Dynamic Testing

Testing CategoryPotential # of Discrete Testing Activities

Typically Includes:

Component-Level Testing

2 • Unit Testing

• Component Integration Testing

System-Level Testing

5 • System Testing

• System Integration Testing

• Performance Testing

• Regression Testing

• Consolidated Integration Testing

Acceptance Testing 3 • User Acceptance Testing

• Operational Acceptance Testing

• Field Acceptance Testing


Inputs

Uses the following factors as inputs for the overall project:• Predicted number of latent defects

• Multiplier for the cost to resolve one (1) defect

• Projected value (e.g., revenue, annualized cost reduction from productivity improvement)

• Potential additional costs (e.g., contract penalties, training)

Uses the following factors as inputs for each testing category:

• Defect removal effectiveness based on # of discrete testing activities

• Experienced Team (yes, no, capable of process improvement)

• Number of personnel as % of required number of personnel (100%, 75%, 50%, 25%)

• Schedule compression (none, 25%, 50%, 75%)

• Test automation (used or not used)

• Amplification factor for defects introduced


Outputs

• Calculates the following outputs for each testing category:

− Number of escaped defects

− Escaped defects as % of predicted total

− Number of defects introduced

− Number of defects removed

− Time to resolve one (1) defect

− Time to resolve all defects

− Cost to resolve one (1) defect

− Actual cost of defect resolution

• Calculates the following outputs for the overall project:

− Time to resolve all defects

− Total defect resolution cost

− Total cost

− Net value (i.e., profit or loss)

− Profit margin / ROI (%)

Collaborative Exercise

Step 1 – Calculate Time, Cost and Value

Use the following information:• Custom application for a single customer

− New development

− Enough developers and acceptance testers

− Inexperienced developers and acceptance testers

• Based on a “best guess”, the predicted number of discovered defects is 500

• Multiplier for cost to resolve one (1) defect is 10 (VERY conservative)


Step 2 – Calculate Time, Cost and Value

Use the following information about the contract:

• Application price $4,000,000

• All defects to be resolved by end of 90-day warranty period

• Potential contract penalties from defects at release

− 10-150 defects 2.5% of application price

− 151-250 defects 5% of application price



− 751-1000+ defects 50% of application price


Step 3 – Calculate Time, Cost and ValueUse the following information about planned testing activity:

• Organization has “decided” the following testing activities are sufficient:

− Component-Level Testing:Unit Testing (1 discrete testing activity)

− Acceptance Testing:User Acceptance Testing (1 discrete Testing Activity)

• Review the inserted Comments to determine the number of discrete testing activities you will perform in each category of Activities

• Organization is looking for a profit margin of at least 25%


Using the Time, Cost & Value Calculator

• We will now try and put it all together in the Time, Cost and Value Calculator

• Enter Projected Value

• Calculate the following results:

− Time (in effort hours) to resolve all defects

− Total Defect Resolution Cost

− Net Value

− Profit Margin (%)

• Let’s not forget to determine and enter the Additional Costs (based on penalty levels)


Step 1 – Initial Results

Time to Resolve All Defects 85,077 hours

Total Defect Resolution Cost $5,353,580

Additional Costs $1,000,000

Total Cost $6,353,580

Net Value -$2,353,380

Profit Margin / ROI -59%


Step 2 - Save Time and Make a Profit

Now, lets modify the strategy for this project to “bring it into the black” – we are going to aim for:

− Significant time reduction

− A Profit Margin/ROI of at least 25%

• We will:

− Consider Static Testing activities, where the cost of defect resolution is lower and the impact of the amplification factor is higher

− Experiment with experience level, schedule compression, and test automation

Reduce the number of post-release defects enough to significantly reduce or even eliminate the penalty (don’t forget the defect ranges!)


A “Good” Solution

• From the Format Menu, select Unhide T_C_V -Proposed

• Focuses on early defect detection and cross-functional collaboration by incorporating:

• Static Testing

− Review of requirements

− Review of Design

− Code Inspection

• Dynamic Testing

− System Testing

− Regression Testing

• Reduces time to resolve defects by >50%

• Achieves a profit margin of 36%

Refining the Data

• Industry data is sound, reliable and defensible

BUT

• YOUR data is equally, sound, reliable and defensible

… and also much more persuasive

A Final Message

If testing costs more than not testing, then don’t test

- Kent Beck

… but you’d better have the data to back up your decision

- John Fodeh

(Thanks to Susan Herrick &Ian Howles)

© 2009 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Contact:

John Fodeh

[email protected]