do you know the potency of your test cases?

Do you know the “Potency” of your test cases?

Webinar: Jan 19, 2012, 1430-1530 IST

T [email protected]

in.linkedin.com/in/AshokSTAG ash_thiru

mailto:[email protected]

mailto:[email protected]

© 2012. STAG Software Private Limited. All rights reserved. 2

...associated with efficacy of an entity, the ability or capacity to achieve or bring about a particular result.

“the strength of the drug, as measured by the amount needed to produce a certain response”.

Dictionary definition.

Potency is ...


Affected area

The ‘Bug’

Drug(s)

Ability to get to target


Affected area

The ‘Bug’

Drug(s)

Is it strong enough?


Where is the target?

Who is the target?

Have I developed immunity?


Affected area

The ‘Bug’

Drug(s)

Is it strong enough?


Where is the target?

Who is the target?

Have I developed immunity?

Drugs ≈ Test casesHow “strong” arethe test cases?

Over time, some parts of the system “harden”. Implies that issues may not be there.

What types of defects are we looking for?

Requirements, Features, Modules, Components. Subsystems we looking in..

© 2012. STAG Software Private Limited. All rights reserved.

Potency is about :

6

“Area”

“Bug”

Test Cases

PDTPotential Defect Type

Entity

Potency

The least number of test cases with the ability to target specific types of defects in specific areas to ensure “clean software”.

Use caseFeatureSubsystemModuleScreen, API ...

Immunity

Resistant to bugsi.e. hardened entities

Where to

target

Who to target?

“Drug”


How “Potent”? The Typical Approach

7

“Drug”

“Area”

“Bug”

Test Cases


Entity

Potency

Where to

target

Who to target?

Requirements traceability“External area that I am covering”

Code coverage“Internal area that I am covering”

?

?

?

Immunity



How “Potent”? The Typical Approach

8

“Area”

“Bug”

Test Cases


Entity

Potency


Immunity


Where to

target

Who to target?



?

?

?Typically based on experience.

Trust me!

“Drug”


Assessing Potency

9

“Area”

“Bug”

Test Cases


Entity

Potency


Where to

target

Who to target?

“Drug”

Countability“Proving sufficiency of test cases”

Conformance:Robustness“Distribution of +Ve/-Ve test cases”

Levelization“Optimal targeting”

Fault Coverage“What PDTs uncovered by test cases”

Test case immunity“No defect yield from test cases”



Immunity



HBT : Hypothesis Based Testing A Quick Introduction

HypothesizePotential Defect Types

Nine Stage Defect Removal FilterCleanliness Assessment

SetupCleanliness Criteria

SUT

Personal, scientific test methodology.SIX stage methodology powered by EIGHT disciplines of thinking (STEMTM).

Click here to know more about HBT.http://stagsoftware.com/blog?p=570

10

http://stagsoftware.com/blog?p=570

http://stagsoftware.com/blog?p=570

http://slidesha.re/qBMNiy

http://slidesha.re/qBMNiy


Entity under test

Cleanliness Criteria

Potential Defect Types

Test CasesRequirementstraceability“what to test”

Faulttraceability“test for what”

should satisfy impeded by

It is necessary for test cases to be purposeful/goal-focused.Tracing test cases to requirement is a necessary condition for this, but not sufficient!Two way tracing i.e. Requirements + FAULT makes this sufficient.

Targeting...

11


Business Logic

Structure

Data

Environment

Usage

Error injection Fault proneness Failure

What kinds of erroneous data may be injected?

What kind of issues could data cause?

What kinds of bad data can be generated?

What conditions/values can be missed?

How can conditions be messed up?

What can be incorrect results when conditions are combined?

How can we setup incorrect “structure”?

How can structure mess up the behavior?

What kinds of structure can yield incorrect results?

What is incorrect environment setup?

How can resources in the environment cause problems?

How can environment be messed up?

In what ways can we use the entity interestingly?

What kinds of usage may be be inherently faulty?

What can be poor usage experience?

Snapshot of Defect HypothesisAsp

ects

Views


Fault Coverage - “What PDT’s uncovered by test cases”

PDT1

PDT2

PDT3

...

PDTn

R1

R2

R3

...

Rm

PDT1

PDT2

PDT3

...

PDTn

TC1

TC2

TC3

...

TCi

Requirements traceability

Faulttraceability

Faulttraceability

What to test?

Test for what?

13

Do the test cases for a given entity have the ability to detect specific types of defects?


Levelization- “Optimal targeting” Staged “Filtration”

Input cleanliness

Input interface cleanliness

Structural integrity

Behaviour correctness

Environment cleanliness

Attributes met

Flow correctness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

That inputs are handled wellInput data handling

That the functional behaviour is correctFunctionality

That the internal structure is robustInternal structural issues

That the user interface is cleanUI issues

That end-to-end flows work correctlyBusiness flow conditions, Linkages

That it does not mess up the environmentResource leaks, Compatibility...

That the stated attributes are metPerformance, security, volume, load...

That it deploys well in the real environmentCompatibility, migration

That user expectations are metUser flows, experience

Objective Issues

14


Countability - “Proving sufficiency of test cases”

C1

C2

C3

C4i1

i2

i3}

Outputs1

2

3

45

Inputs Conditions

Behaviour StimuliEntity under test

15

Conditions govern behaviourOutcome is Test Scenarios

Input (data) is StimuliOutcome is Test Cases

If #conditions & values foreach conditions are know, then #scenarioscan be proved to be sufficient.

For each scenario, knowing #values for each input can prove sufficiency of test cases.


Conformance:Robustness- “Distribution of +ve/-ve test cases”

16

L9

..................

L3

L2

L1

+ve test cases

-ve test cases

What do you think the ratio depends on?

How do the test cases distribute?

Note at higher levels test cases progressively become conformance oriented?


Test Case Immunity - “No defect yield from test cases”

17

Analysis of ‘defect type trends’ over ‘time’ by ‘entity under test’ helps us understand “what type of defects” are not present in “what entities”.

Example:1. As product matures, certain types of defects become harder to uncover. Say at Level-1 where the focus is on data validation issues, at later builds these kinds of issues do not surface. This kinda implies that “the system has developed immunity for this type of defect”.

2. Continuing from (1), it may be plausible that one of the entities may still continue to have data validation type of issues. Hence the phrase “entity under test” in ‘defect type trends’ over ‘time’ by ‘entity under test’.


Summarising...

18

“Area”

“Bug”

Test Cases


Entity

Potency


Where to

target

Who to target?

“Drug”

Countability“Proving sufficiency of test cases”

Conformance:Robustness“Distribution of +Ve/-Ve test cases”

Levelization“Optimal targeting”

Fault Coverage“What PDTs uncovered by test cases”

Test case immunity“No defect yield from test cases”



Immunity



Results

Assess the quality of test cases statically

Improved confidence, able to convince customer about effectiveness logically

19

Discovery of missing test cases


Thank you!

HBT is the intellectual property of STAG Software Private Limited.STEMTM is the trademark of STAG Software Private Limited.

www.stagsoftware.com

@stagsoft

blog.stagsoftware.com

Connect with us...

A division of STAG

Covered in detail in“Effective review of test cases” A HBT series workshopwww.cleansoft.in

http://www.stagsoftware.com

http://www.stagsoftware.com

http://www.cleansoft.in




do you know the potency of your test cases?

Education