adopting the pairwise test design technique to optimize printer driver test coverage v1.adopting the...

Adopting the Pairwise Test Design Technique to

Optimize Printer Driver Test Coverage

© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.

Abstract

Abbreviations

Market Trends and Challenges

Solution

Case Study

Best Practices

Common IssuesCommon Issues

Conclusion

Reference

3

3

3

4

5

7

88

9

10

Table of Contents


Testing the interactions between printer driver variables has always been a challenge. Considering that testing all the possible combinations of driver variable interactions is quite impractical, there is always a demand for optimizing the test coverage. Pairwise testing, which is a proven technique in the software testing industry, can be adopted to provide optimized test coverage for printer driver testing.

AA case study has been conducted to explore the effectiveness of using the pairwise test design technique against the legacy approach that used randomization, using operational profiles. The input data contained around 61 variables resulting in approximately 2800 pairwise test cases to completely test all variable interac-tions. The test cases generated by the pairwise approach and the tests that were generated using the legacy approach were executed on the product under test and it was observed that the pairwise test scenarios were able to detect 19% more of defects than the ones that were detected by the legacy test cases with an execu-tion effort savings of 70%.

For any application in the current world, due to the distinct varieties of user groups, there is a need for accom-modating more input variables in a software system. With this increase, the complexity multiplies in testing the interactions between all these variables. With only 10 variables each with 2 states per se, the total test sce-narios to test all the possible combinations will be 210 = 1024 test cases. If just one more variable is added to the software system with 2 states, the total test scenarios become 2048!

In a printer driver, there are more than 60 input variables leading to an unimaginable number of test combi-nations; a complete coverage of which, is nearly impossible to provide. Considering the competitive market

Sl.No

1

2

3

4

5

6

77

DRE

TTM

NASA

OS

PDL

PICT

OATSOATS

Defect Removal Efficiency

Time To Market

Operating System

Page Description Language

Orthogonal Array Test Strategy

The National Aeronautics and Space Administration, USA

Pairwise Independent Combinatorial Testing tool from Microsoft

Full FormAcronyms

Abstract

Abbreviations

Market Trends and Challenges

Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 3


environment and the need for a reduced time to market (TTM), the test teams are challenged to come up with various strategies to optimize testing without compromising on the quality of the deliverables. This may include reducing the overall number of test cases to accelerate product time to market

Sometimes, test teams choose option (2) which can efficiently unveil a more number of defects but not most of them. With this option, providing a quantitative number in terms of test coverage is a challenge as well. Most often, option (1) is chosen by many test teams which brings ineffective test coverage, leading to poor Defect Removal Efficiency (DRE). A poor DRE means more field defects and less satisfied customers.

By randomly picking a subset of tests from the overall count without adopting a strategy (pure ran-domization or guessing)

By randomly picking a subset of tests from the overall count using operational profiles (more focus on customer usage like paper sizes / paper types, highest customer base like Windows 8.1, Mac 10.9, etc.) or

By choosing a statistically-based test design technique.

The solution to the combinatorial testing problem stated in the previous page is to adopt the Pairwise Test Design Technique for testing the printer driver’s variable interactions.

PairwisePairwise testing is an economical alternative to testing all the possible combinations of a set of variables. It is an effective test case generation technique based on the historical data that most faults are caused by interac-tions of two actions. In pairwise testing, test cases are designed to execute all the possible discrete combina-tions of each pair of input parameters. Pairwise-generated test suites cover all the combinations of two; and are therefore much smaller than exhaustive ones, yet still very effective in finding defects. Pairwise testing is also referred to as all-pairs testing or 2-way testing.

Graph 1 shows the cumulative percentage of software defects detected by the n-tuple interactions where it is apparent that for medical device software and NASA applications, pairwise interactions are able to detect more than 90% of software failures. Higher-order interactions (3-way, 4-way, etc.) can still be adopted, if pre-ferred, but the benefit-cost ratio will be considerably less.

Graph 1. Error detection rates for interaction strengths 1 to 6

Cumulative percent of

software failures

Interactions

Mediacal Devices

Browser

Sensor

NASA

Network Security

Solution



The process flow for the pairwise test generation goes as shown in Figure 1.

In terms of printer driver testing, the key suggestion is to combine the two forms of Combinatorial Testing – Configuration Testing (testing interactions of configuration related variables such as the OS, Applications, PDL, and Connectivity, etc.) and Input Parameter Testing (testing interactions of the printer driver features).

A case study was conducted where a launched printer driver version was chosen, and the effectiveness of the pairwise testing technique was studied against the legacy test approach that was adopted while launching the driver. The legacy approach used randomization using operational profiles. The detailed process flow of the case study is shown in Figure 2.

The input data contained around 61 variables with states ranging from 1 to more than 40 resulting in approxi-mately 2800 test cases to completely test all variable interactions. PICT (Pairwise Independent Combinatorial Testing tool) was used to generate the pairwise scenarios.

PICT is a free command line tool from Microsoft that takes a model file as input and generates a compact set of parameter value choices that represent the test cases to get comprehensive combinatorial coverage of the input parameters. PICT allows users to feed constraints while generating the pairwise scenarios. There were more than 75 constraints fed into PICT for this case study.

Figure 2. Detailed process flow of the case study

Figure 1. Process flow for the pairwisetest generation

Case Study



Table 1 represents a sample of the input data used for generating pairwise scenarios.

Table 2 represents a sample of the pairwise data generated for the case study.

Variables

States

Letter (8.5 x 11”)

Legal (8.5 x 14”)

Statement (5.5 x 8.5”)

Executive (7.25 x 10.5”)

8.5” x 13”

Automatically Select

Printer Default Type

Plain

Hole Punched

Transparency

Automatically Select

White

Blue

Yellow

Green

1 Sided Print

2 Sided Print

2 Sided print, Flip

on Short Edge

Media Size Media Type Media Color 2-Sided Printing

Variables

Total # of test cases

Resource(s) Utilized

Duration (In Man Days)

Effort Savings Through Pairwise Strategy

Unique defectd identified (excl. invalid & duplicatedefects)

% of defects detected additionally by Pairwisescenories

~ 5000

6

396

59 70

70%

19%

~ 2800

3

114

Legacy Approach Media Type

Test case #

No 6 3/4 Envelop

B5 (176 x 250mm)

New

A4 (210 x 297 mm)

Letter (8.5 x 11”)

B5 (176 x 250 mm)

1

2

3

4

5

6

Letterhead

Plain

Custom Type 2

Custom Type 1

Custom Type 5

Hole Punched

Clear (Transparency)

Gray

Goldenrod

Custom Color 3

Pink

Custom Color 1

1 Sided Print

2 Sided Print

1 Sided Print

2 Sided Print Flip on Short Edge

2 Sided Print

2 Sided Print Flip on Short Edge

Media Size Media Type Media Color 2-Sided Printing


Table 1. Sample of input data

Table 2. Sample of pairwise data

Table 3. Comparison data between the Pairwise approach and the Legacy approach


From Table 3, it is apparent that the pairwise approach found 19% more defects than the legacy approach with a savings in execution effort of 70%. It was also found that the pairwise scenarios were able to unveil all the defects that the legacy test approach discovered, leaving NO leakages.

Table 4 shows the key differences between the legacy approach and the pairwise approach.

Another approach in the industry that attempts to solve the problem defined in this paper is Orthogonal Array Test Strategy (OATS). Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array you receive an even distribution of all the pair-wise combinations of values in the array. OATS uses the concept of orthogonal arrays to design tests for software systems.

A sample orthogonal array [L9 (34)] has been shown in Table 5 for reference.

Best Practices

Legacy Approach

Randomized

Subjective

Coverage cannot be quantitatively measured as the

data is randomized

Cannot be automated easily

Optimized

Objective

Coverage can be quantitatively measured

Great candidate for automation!

Pairwise Approach


Table 4. Pairwise approach v/s Legacy approach


Unfortunately, OATS becomes ineffective when the system under test includes constraints because there is no way constraints can be fed into orthogonal arrays while generating test scenarios; this causes OATS to generate tests with invalid pairs.

Another disadvantage with OATS is that the Orthogonal Arrays contain all pairs an equal number of times. That means, using orthogonal arrays will result in more number of tests compared to the pairwise strategy because they include pairs equally. The pairwise strategy covers all the pairs in a lesser number of tests than OATS.

The challenge with the pairwise test strategy is that the requirements (including the constraints) should be accurate and complete; right the first time. The requirements should NOT change once the pairwise scenarios are generated; the entire pairwise matrix will have to be re-generated if any one requirement changes post the pairwise test generation. This is because each row item in the pairwise matrix is unique, attempting to cover as many unique pairs as possible with the least number of tests.

AA dedicated effort is also required for generating pairwise scenarios for every product under test, even for similar products; even a slight change in the requirements between two products will call for separate pairwise matrices.

Another challenge concerns the selection of the right tool to be used. There are many free and paid tools available in the market and one needs to clearly define the tool evaluation criteria for picking up the right tool. The evaluation criteria should consider, but not be limited to:

Common Issues


Control Factors

Expt. No

1

2

3

4

5

6

77

8

9

1

1

1

2

2

2

33

3

3

1

2

3

1

2

3

11

2

3

1

2

3

2

3

1

33

1

2

1

2

3

3

1

2

22

3

1

A B C D

Table 5. Sample Orthogonal Array


The budget allotted for the test activity, if a paid tool is to be used.

Tool support; most of the free tools will NOT have any tool support provided along with them.

The number of input variables and states.

The complexity of the constraints that come with the input data and the ability of the tool to accom-modate these constraints while generating pairwise scenarios.

Ease of use in terms of feeding the variables and constraints into the tool.

Support in importing the generated pairwise tests into a Test Management tool for test execution management.

The pairwise test design technique is very effective in providing optimized test coverage for testing the variable interactions of a printer driver. With just 2800 test scenarios derived using the pairwise test design technique as compared to approximately 5000 test cases using the legacy approach, the pairwise technique was able to detect 19% more of defects than the ones that were detected through the legacy approach with a test execution effort savings of 70%.

TheThe pairwise test design technique also enables the test teams to measure test coverage in a quantitative way as opposed to using other strategies like pure randomization, guessing and/or randomization using opera-tional profiles.

Adopting the pairwise test design technique for testing the variable interactions of the printer driver signifi-cantly reduces the execution effort, detects more defects with less number of tests, and provides us with a quantitative way of measuring test coverage. Thus, the pairwise technique proves itself to be the best combi-natorial test design technique with a reduced number of tests and an improved defect detection rate.

Conclusion


For more details contact: [email protected] us on twitter: http://twitter.com/hclers andOur blog http://www.hcltech.com/blogs/engineering-and-rd-servicesVisit our website: http://www.hcltech.com/engineering-rd-services

Hello, I’m from HCL’s Engineering and R&D Services. We enable technology led organizations to go to market with innovative products and solutions. We patner with our customers in building world class products and creating associated solution delivery ecosystems to help bring market leadership. We develop engineering products, solutions and platforms across Aerospace and Defense, Automotive, Consumer Electronics, Software, Online, Industrial Manufacturing, Medical Devices, Networking and Telecom, Office Automation, Semiconductor and Servers & Storage for our customers.

This whitepaper is published by HCL Engineering and R&D Services.

The views and opinions in this article are for informational purposes only and should not be considered as a substitute for professional business advice. The use herein of any trademarks is not an assertion of ownership of such trademarks by HCL nor intended to imply any association between HCL and lawful owners of such trademarks.

For more information about HCL Engineering and R&D Services,Please visit http://www.hcltech.com/engineering-rd-services

Copyright@ HCCopyright@ HCL TechnologiesAll rights reserved.

Swadish Aaryan HCL Engineering and R&D Services

Reference

Author Info

1) http://www.pairwise.org/

2) Pairwise Testing in the Real World:

Practical Extensions to Test-Case Scenarios. (http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic1)

3) James Bach and Patrick J. Schroeder , Pairwise Testing: A Best Practice That Isn’t

4) Jacek Czerwonka, PICT 3.3 User’s Guide

5) Practical Combinatorial Testing. SP 800-142. (Report). Natl. Inst. of Standards and Technology. 2010.
