copy of minimization and prioritization of test cases

Minimization and Prioritization of Test Cases

Development may take 2 to 4 years

Regression Testing

Same may have to be maintained for 10 to 15 years

Software maintenance is expensive

Reason for software change

1. Errors during actual use

2. Additional functionality

3. Changes in external world

4. Restructuring work

5. Change in technologies

6. Obsolete capabilities to be deleted.

Regression Testing

When we modify software, we typically retest it.

This retesting is called regression testing.

Development testing

Regression testing is the process of retesting the modified parts of the software and ensuring that no new errors have been introduced into previously tested code.

Regression Testing

Purposes

Increase confidence in the correctness of the modified program.

Locate errors in the modified program.

Preserve the quality & reliability of software.

Ensure the software’s continued operation.

We may perform regression testing during latter stage of software development.

Regression Testing

Regression TestingDevelopment TestingS.No.

1 We create test suites and test plan

We test all software components

Budget give time for testing

Once on software product

Performed under the pressure of release date

2

3

4

5

We can make use of existing test suites and test plans

We retest modified components or affected by modifications

No time

Many times

Performed in crisis situations, under greater time constraint.

Regression Testing

Reducing the Number of Test Cases

Select all those test cases that traverse the modified portion of the program and the portion that is affected by the modification .

These test case minimization techniques attempts to find redundant test cases

MINIMIZATION OF TEST CASES


Prioritization of Test Cases

A test case with highest rank has the highest priority and second highest rank test case has second highest priority and as so on.



Test case prioritization

General test case prioritization

Version specific test case prioritization



In general test case prioritization, for a given program with its test suite, we prioritize the test cases that will be usefulover a succession of subsequent modified version of original program without any knowledge of modification(s).

In version specific test case prioritization, we prioritize the test cases, when the original program is changed to the modified program, with the knowledge of the changes that have been made in the original program.


Prioritization GuidelinesAny guidelines should address two fundamental issues like:

What functions of the software must be tested?What are the consequences if some functions are not tested?

All prioritization guidelines should be designed on the basis ofrisk analysis. All risky functions should be tested on higher priority.Most important is the “impact of failure” which may range from “no impact” to “loss of human life” and must be studied very carefully.


Priority Category SchemePriority code 1 : Essential test casePriority code 2 : Important test casePriority code 3 : Execute, if time permitsPriority code 4 : Not important test casePriority code 5 : Redundant test case


Priority Category Scheme

Priority code 1 : Important for the customerPriority code 2 : Required to increase customer satisfactionPriority code 3 : Help to increase market share of the product

Risk Analysis

What is risk? Probability of occurrence of a problem (i.e. an event)Impact of that problem

Risk analysis is a process of identifying the potential problems and then assigning a ‘probability of occurrence of problem’ value and ‘impact of that problem value for each identified problem.Both of these values are assigned on a scale of 1 (low) to 10 (high). A factor ‘risk exposure’ is calculated for every problem which is the product of ‘probability of occurrence of problem’ value and ‘impact of that problem’ value.

Risks may be ranked on the basis of its risk exposure.

Risk Analysis

4321

Risk Exposure

Impact of that Problem

Probability of occurrence of problem

Potential ProblemS.No.

Risk Analysis Table

Risk Analysis

842School not available in the data base10

212Issued login-id is not in specified format

9623Lists not in proper format8818Ambiguous documentations71892Database corrupted610101Unauthorised access5.422Printing mistake in registration card4933Wrong entry in scheme detail form31226Wrong entry in students detail form2632Issued password not available1

Risk Exposure

Impact of that Problem

Probability of occurrence of problem

Potential ProblemsS. No.

Risk Analysis

• The potential problems ranked by risk exposure are 6, 2, 5, 3, 7, 10, 1, 8, 4 and 9.

Risk Analysis

Risk matrix is used to capture identified problems, estimate their probability of occurrence with impact and rank the risks based on this information.We may use the risk matrix to assign thresholds that group the potential problems into priority categories.

Risk Matrix

Risk Analysis

Threshold by quadrant

Impact of the problem

Probability of occurrence of the problem

PC-3 PC-1

PC-4

PC-2

0

10

9

8

7

6

5

4

3

1 2 3 4 5 6 7 8 9 10

2

1

1 3

24

5

6

78

9

10

Risk Analysis

The priority category in defined as:

Priority category 1 (PC-1) = High probability value and high impact value

Priority category 2 (PC-2) = High probability value and low impact value

Priority category 3 (PC-3) = Low probability value and high impact value

Priority category 4 (PC-4) = Low probability value and low impact value

We may decide to give more importance to high impact value over the high probability value. Hence, PC-2 and PC-3 will swap, but PC-1 and PC-4 will remain the same.

Risk AnalysisAlternate Threshold by quadrant



PC-2 PC-1

PC-4

PC-3

0

109

8

7

65

43

1 2 3 4 5 6 7 8 9 10

21

1 3

24

5

6

78

9

10

Risk AnalysisThreshold by diagonal quadrant



0

10987

65

4

3

2

1

1 2 3 4 5 6 7 8 9 10

PC-4

PC-3

PC-2

PC-1

Risk AnalysisThreshold based on high “Impact of Problem” value



10987

65

43

21

1 2 3 4 5 6 7 8 9 10

0

PC-1

PC-3 PC-2

PC-5 PC-4

Risk AnalysisThreshold based on high “probability of occurrence of problem”



10987

65

43

2

1

1 2 3 4 5 6 7 8 9 10

0

PC-4 PC-2

PC-5 PC-3PC-1

Risk Analysis

After the risks are ranked, the high priority risks are identified. These risks are required to be managed first and then other priority risks and so on.

These risks should be discussed in a team and proper action should be recommended to manage these risks.

A risk matrix has become a powerful tool for designing prioritization schemes.

All that matter when using risk matrix is the relative order of the probability estimates (which risks are more likely to occur) on the scale of 1 to 10.

The impact of the problem may be critical, serious, moderate, minor or negligible.

Execution history is given below

Test Case Array

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

1, 2, 20, 30, 40, 50

1, 3, 4, 21, 31, 41, 51

5, 6, 7, 8, 22, 32, ,42, 52

6, 9, 10, 23, 24, 33, 43, 54

5, 9, 11, 12, 13, 14, 15, 20, 29, 37, 38, 39

15, 16, 17, 18, 19, 23, 24, 25, 34, 35, 36

26, 27, 28, 40, 41, 44, 45, 46

46, 47, 48, 49, 50, 53, 55

55, 56, 57, 58, 59

3, 4, 60

Regression Testing

Suppose 1,2,5,15,35,45 and 55 are modified.

Simplest is to execute all test cases that have any of the modified lines of code.

Selected test cases : T1, T2, T3, T5, T6, T7, T8 and T9

Total : 8

Regression Testing

No. of matchesModified linesTest casesT1

T2

T3

T4

T5

T6

T7

T8

T9

T10

1, 2

1

5

--

5, 15

15, 35

45

55

55

--

2

1

1

0

2

2

1

1

1

0

Regression Testing

Number of matches found (NFOUND) is stored in an array and elements are stored in descending order.

The test case that has the maximum number of matches is selected by making its candidate value = 1. This test case will be executed first & is shown on next slide.

Regression Testing

CandidateMatches found

No. of matches (nfound)

Test cases

T1

T5

T6

T2

T3

T7

T8

T9

2

2

2

1

1

1

1

1

1, 2

5, 15

15, 35

1

5

45

55

55

1

0

0

0

0

0

0

0

Regression Testing

Test cases in descending order

T1 is selected for execution and it covers line nos. 1 and 2LOCs that are still to be executed:

[ 1, 2, 5, 15, 35, 45, 55 ] – [ 1, 2 ]

= [5, 15, 35, 45, 55]

Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the one with maximum number of matches.

Regression Testing



Test cases

T5

T6

T3

T7

T8

T9

2

2

1

1

1

1

5, 15

15, 35

5

45

55

55

1

0

0

0

0

0

Regression Testing

T5 is selected for execution and it covers line nos. 5 and 15.LOCs that are still to be executed:

[ 5, 15, 35, 45, 55 ] – [ 5, 15 ]

= [35, 45, 55]

Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the test case with maximum number of matches found.

Regression Testing



Test cases

T6

T7

T8

T9

1

1

1

1

35

45

55

55

1

0

0

0

Regression Testing

T6 is selected and covers line no. 35.

LOCs still to be executed : [ 45, 55 ]



Test cases

T7

T8

T9

1

1

1

45

55

55

1

0

0

Regression Testing

T7 is selected for execution covering line no. 45.LOCs still to be executed : [ 55 ]



Test cases

T8

T9

1

1

55

55

1

0

T8 is selected for execution

Order : T1, T5, T6, T7, T8

Regression Testing

Out of 10 test cases, we need to run only 5 test cases for 100% code coverage of modified lines of code.

Second Case

Lines of code have been modified and also deleted.

Example has 20 lines of code. 5 Test Cases as given on next slide.

50% saving of test case

Regression Testing

Execution HistoryTest Cases

T1

T2

T3

T4

T5

1, 5, 7, 15, 20

2, 3, 4, 5, 8, 16, 20

6, 8, 9,10, 11, 12, 13, 14, 17, 18

1, 2, 5, 8, 17, 19

1, 2, 6, 8, 9, 13

Modified lines : 6, 13, 17, 20

Deleted lines : 4, 7, 15

Regression Testing

From the test history information, remove the deleted lines say : 4,7,15.

Test case array will be:


T1

T2

T3

T4

T5

1, 5, 20

2, 3, 5, 8, 16, 20

6, 8, 9,10, 11, 12, 13, 14, 17, 18

1, 2, 5, 8, 17, 19

1, 2, 6, 8, 9, 13

Regression Testing

Now, we want to find redundant test case. This situation has become important due to deletion of few lines.,

If we see execution history carefully. We come to know that T1 & T5 are redundant test cases.

Regression Testing

Redundant Y/N

Found in test caseLine numbersTest cases

T1

T5

1

5

20

1

2

6

8

9

T4

T2

T2

T4

T2

T3

T3

T3

Y

Y

Y

Y

Y

Y

Y

Y

13 T3 Y

Regression Testing

Test case array is:


T2

T3

T4

2, 3, 5, 8, 16, 20

6, 8, 9, 10, 11, 12, 13, 14, 17, 18

1, 2, 5, 8, 17, 19

The remaining test cases are = T2, T3, T4

Regression Testing

Number of matches for modified lines are

Modified lines are : 6, 13, 17, 20

No. of matches found (nfound)

Matches foundTest cases

T2

T3

T4

20

6, 13, 17

17

1

3

1

Regression Testing

Test cases are sorted on the basis of matches found (descending order)

CandidateLine numbersNo. of matches foundTest cases

T3

T2

3

1

6, 13, 17

20

1

0

T4 1 17 0

Regression Testing

T3 is selected for execution. Remaining modified line is : 20

T1 & T5 should be deleted from the test suite.

Saving : 60%

Hence T3 and T2 are sufficient to execute modified lines.

Regression Testing

copy of minimization and prioritization of test cases

Documents