Good Enough Testing

James Bachjames@satisfice.com

http://www.satisfice.com

The author thanks ST Labs, Inc. for supporting the work that ledto some of the materials in this presentation.

Paradigms of Good Enough...

Not Too Terrible: “we’re still in business” Infallibility: “anything we do is good” Righteous Exhaustion: “perfection or bust” Customer Oracle: “customers seem to like it” Defined Process: “we follow a Good Process” Static Quality: “we satisfy The Requirements” Accountability: “we fulfill our promises” Advocacy: “we make every reasonable effort” Dynamic Quality: “we fulfill the mission”

Dynamic Quality Paradigm

Perfect

Awful

unnecessary quality

unacceptable quality

Item A

Item B

It’s more important to work on Item B.

Further improvement would not be a good use of resources.

Further improvement is necessary.

Good enoughquality bar

floating line

A Heuristic for Good Enough

1. X has sufficient benefits.

2. X has no critical problems.

3. Benefits of X sufficiently outweigh problems.

4. In the present situation, and all things considered, improving X would be more harmful than helpful.

BenefitsProblems

All conditionsmust apply.

Good Enough...

…with what level of confidence? …to meet ethical obligations? …in what time frame? …compared to what? …for what purpose? …or else what? …for whom?

Perspectiveis

Everything

You can use this to:

Argue persuasively for improvement, even in the midst of chaos.

Plan ahead for high quality. Understand what problems are keeping otherwise

smart, caring people from doing excellent work. Notice when “perfect” or “as good as humanly

possible” is not good enough. Structure a dialog about acceptable quality.

Why not Quantify Good Enough?

Law of Medium NumbersFor medium number systems, we can

expect that large fluctuations, irregularities,and discrepancies with any theory will occurmore or less regularly. -- Gerald Weinberg

Because quality is an emergentproperty of a complex system, any quantification

of quality is incomplete.

Heuristics are an alternative to quantitative models...

We can measure some things. We can observe some things. We can reason about those

observations and measurements.

A heuristic is useful method that sometimes doesn’t work.Heuristic means “serving to discover”

Test Project Dynamics:Star Model

T es t T eamExpert iseLoadingCohesionMot ivat ionLeadershipProj ect Integrat ion

T es t L abT est P lat formsT est T oolsT est L ibraryProblem T racking SystemOffi ce Faci l i t ies

Product MissionStakeholdersQual it y Criter iaReference Material

RequirementsProductProj ect L if ecycleProj ect ManagementConfi gurat ion ManagementDefect P revent ionDevelopment T eam

Development

M iss ionFind Imortant ProblemsAssess Qual it yCert if y to StandardFulfi l l P rocess MandatesSat isf y StakeholdersAssure Accountabil i t y

Advise about QAAdvise about T est ingAdvise about Qual it yMaxim ize Effi ciencyMinim ize CostMinim ize T ime

T es tP roc ess

St rategyLogist icsProducts

Test Project Dynamics:Star Model

T es t T eam T est L ab

RequirementsDevelopment

M iss ion

T estP roc ess

Test Project Dynamics:Star Model

GIV ENS GIV ENS

GIV ENSGIV ENS

M iss ion

CHOICES

CHOI

VENSMotivation

Capability

Test Project Dynamics:Givens vs. Choices

Motivation: What testing does the situation require? Capability: Can we perform that testing in this situation?

Analyzing a Test Process

Are GIVENS good enough?

Do CHOICES about process exploit the GIVENS and address the MISSION well enough?

Is MISSION is achieved well enough?

How do you know?

MISSION:The most important part

Find important problems Assess quality Certify to standard Fulfill process mandates Satisfy stakeholders Assure accountability

Advise about QA Advise about testing Advise about quality Maximize efficiency Minimize time Minimize cost

The quality of testing depends on which of thesepossible missions matter and how they relate.

Many debates about the goodness of testingare really debates over missions and givens.