test case potency assessment

Test Case Potency Assessment(TCPA)

© 2012. STAG Software Private Limited. All rights reserved. 2

Introduction

Test case potency assessment is primarily a diagnostic that that assess the quality of test cases. This is one of the applications of HBT (Hypothesis Based Testing).

Typically the quality of the test cases i.e. it ability to uncover defects that matter is understoodas being heavily dependent on experience. And therefore assessment of the same is seen as one that can be done by a person with deep domain knowledge.

No doubt domain knowledge is useful the act of assessment in TCPA is done using test case architecture of HBT as the basis and therefore done scientifically.

The key objectives of this 1. Are the right types of defects being targeted to be uncovered?2. Are the test cases covering the functional and non-functional aspects of the system?3. Are the designed test cases sufficient?4. Is the mix of test cases of +ve/-ve good enough?5. Is the baseline for test complete?

The assessment is done by examining the test case document(s) and test strategy statically after understanding the system.

© 2012. STAG Software Private Limited. All rights reserved.

Value Proposition

3

TCPA is intended to assess the quality of test scenarios/cases and help in improving the coverage of the same.

The key value proposition is to ensure that the test scenarios/cases can indeed cover the system under test well thereby ensuring that release quality of the system is indeed good.


Potency Assessment

“Area”

“Bug”

Test Cases

PDTPotential Defect Type

Entity

Potency

Where to

target

Who to target?

“Drug”

Countability“Proving sufficiency of test cases”

Conformance:Robustness“Distribution of +ve/-ve test cases”

Level-isation“Optimal targeting”

Fault Coverage“What PDTs are uncovered by the test cases”

Test case immunity“No defect yield from test cases”

Requirements traceability“External area that I am covering”

Code coverage“Internal area that I am covering”

Immunity Resistant to bugsi.e. hardened entities

... is done by‣ using a test case architecture that ‣ characterises properties of test cases and ‣ examine/assess potency related properties

Application

1

4


Organised by Cleanliness levels sub-ordered by items (features/modules..), segregated by type, ranked by importance/priority, sub-divided into conformance(+) and robustness(-), classified by early (smoke)/late-stage evaluation, tagged by evaluation frequency, linked by optimal execution order, classified by execution mode (manual/automated)

The external form and internal structure of test cases are seen as key to ensuring that the test cases are complete and effective.

Examining the organisation of test cases allows one to uncover “holes” in the test cases.

Level

Item

Type

Priority

Focus

Stage

Frequency

Order

Mode

HBT Test Architecture

5


Brief Flow of TCPA

6

1. Understand SUT2. Setup test baseline

i.e “what to test”3. Identify attributes that are

key

6. Check if the baseline used for strategy is compete

5. Hypothesise PDTs and types of test of interest

7. Check if test cases exist for these PDTs

4. Check if appropriate types of test have been

considered

8. Analyse test case count for completeness 9. Analyse +ve:-ve distribution

10. Document gaps in the test scenarios/cases 11. Produce TCPA report

12. Explain gaps using the TCPA report


Note

7

1. Since TCPA is assessment based on documented test cases, this cannot be done for system that is tested using no/very minimal test documentation.

2. Also any defect information is welcome as this would accelerate the PDT list creation. This can include customer defects and those found the vendor’s test team.

3. Note that in the case of a large system, this may be done by identifying similar components and assessing only sample of similar features/requirements to get a good indicative view of areas of improvement.


Inputs/Deliverables

8

TCPAPotency Assessment ReportThis highlights the gaps in the test scenarios/cases discovered by TCPA.

Test Strategy

Test Scenarios/Cases

Defects found by QA team

Defects reported by customer

test case potency assessment

Technology