Pairwise Testing

www.kanoah.com

ABOUT US

Kanoah is an innovative company providing ground-breaking solutions to software testing professionals on the

Atlassian JIRA platform

About Kanoah Tests

Kanoah Tests is a full featured test management, integrated seamless into JIRA with the same look-n-feel. No need to learn or switch between different applications

Coordinate all test management activities including test planning, authoring, execution, tracking and reporting from a central location

Kanoah Tests enables you to track testing progress and quality to foster collaboration and visibility across traditional and agile teams

Get real-time insights into your testing progress with out of the box reports

Easily integrate your automated tests and submit test results with Kanoah Tests’ powerful REST API or use the API to automate many areas of the application

After looking for several years at plugins for test management we finally found Kanoah Tests. The other solutions were either too complex, didn't integrate well with Jira, or were focused on a single project. Kanoah Tests proved to be an elegant solution that allowed linking between any project. Kanoah has been very responsive to feedback, requests, suggestions as well bugs. The customer service is awesome. I'd highly recommend Kanoah Tests to teams of any size looking to simplify test management and consolidate tools.

Don Pierce

Robert Murhamer

Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and especially with Agile. Test cases can be authored right from the story level, but has all other functionality a Test Case Management solution would need to have, ranging from creating test plans, executing test cases, importing test cases, API for automation, a.s.o. Additionally the team at Kanoah is amazing and responding to any question very quickly. It wasn't hard to sell to my management to purchase Kanoah. Will highly recommend Kanoah to anybody.

Zour Brosh

I just start working with Kanoah and I am impressed how it's simple to manage tests without unlimited non-used features like in most of the test management tools and still to get the needed functionality and results. The integration with Jira is a great working solution that enable to share testing and development in simple way on one system. I recommend to use Kanoah for testing management. It will help to do Kanoah as much as possible customizable like Jira to match each group methodology

For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews

Reviews

Key Features

Native seamless integration with JIRA

No need to learn or switch between different applications

Perfect for agile & traditional testing approaches

Manage, organize and track all your testing efforts in a central place

Reuse test cases across your projects

Powerful REST API

Establish clear traceability between requirements, test cases, and defects

Execute test cases and track results that matter

Get real-time insights into your testing progress with out of the box reports

Live statistics accessible to your entire team

Benefits for the testers

No need to lear or switch between different applications

Reuse test cases across projects for regression

Link test cases to requirements and defects

API support for automated efforts

Benefits for the teams

Informed decisions based on real-time insights

End-to-end traceability and impact analysis

Centralized Test Management

Save time and increase productivity

Why choose Kanoah Tests

Coordinate all test management activities right inside JIRA

1 Testing right inside JIRA

Take advantage of the built-in reports to track the results and measure progress

2Make informed decisions

Kanoah Tests users receive priority support, even during trials

3Stelar support

Pairwise Testing

What is Pairwise Testing?In computer science, all-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters.

Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by “parallelizing” the tests of parameter pairs.

The most common bugs in a program are generally triggered by either a single input parameter or an interactions between pairs of parameters.

Bugs involving interactions between three or more parameters are both progressively less common and also progressively more expensive to find, such testing has as its limit the testing of all possible inputs.

Thus, a combinatorial technique for picking test cases like all-pairs testing is a useful cost-benefit compromise that enables a significant reduction in the number of test cases without drastically compromising functional coverage. [1]

What is Pairwise Testing?Combinatorial test data generators generate data tables for testing. The most basic, commonly used combinatorial data generation strategy is what is known as pairwise testing, all-pairs testing, covering arrays or Taguchi designs.

Pairwise testing tries to alleviate the following practical problem: your system has ten configuration parameters, and every configuration parameter has ten different, interesting values. How can you test that your system behaves correctly with all the different configurations?

Because there are 10 billion different configurations available, you cannot test for all of them. The proposal of pairwise testing is that it is enough to test for all the possible value combinations for anytwo of the ten variables.

A rude calculation shows that there are 45 configuration variable pairs, and for every pair there are one hundred (ten times ten) different pairwise set values available, so 4,500 tests should suffice. This is obviously an improvement over 10 billion, but actually all the pairs can be covered in a much smaller test set, because every single test can cover multiple pairs at the same time. [2]

What is Pairwise Testing?

In computer systems, defects usually involve a single condition, independent of any other condition in the system. If there’s a problem with a device or a variable or a setting, the problem is usually in that device or variable or setting alone.

In testing, we want to be sure that we don’t miss problems based on conflicts between two or more conditions, variables, or configurations, so we often test in combinations in order to find defects most efficiently.

Suppose that a defect depends upon every variable in the system to be in one specific state. If only we could try all of the possible combinations by checking each aspect of the system in each of its states, combined with all of the other aspects in each of their possible states, we would thereby test every state and every combination of states in the system. [3]

All Pairs Technique

Suppose that there are five variables, each represented by a letter of the alphabet. And suppose that each variable can contain a value from one to five.

Let’s set variables A, B, C, and D all equal to 1. If those values are fixed, variable E can have a value from one to five. We’ll keep track of the total number of combinations in our leftmost column; our first five combinations will be numbered from 1 to 5. [3]

All Pairs Technique

Let’s set variable D to two instead of one, and keep A, B, and C at one. Five more combinations, numbered 6-10.

Then we’ll set D to 3, then to 4, and then to 5, setting E to values from 1 to 5 each time. We’ll have to go through 25 combinations before we exhaust all of the combinations for D and E, and only then can we change C to from 1 to 2. The columns on the right will roll over relatively quickly, but as we move leftwards, columns A, B, and C will change much less often. [3]

All Pairs TechniqueIn fact, if there are only five variables in the program, each with five possible states, the program would be absurdly simple, yet we’d have to test 3125 combinations.

If we could test one combination per minute, complete testing would take seven and a half days of continuous, seven-hour-a-day tests. There’s got to be a way to reduce the number of tests into something that we can handle. [3]

All Pairs Tables

Imagine that our defect depends upon checkbox A being cleared (that is, unchecked) and checkbox B being set (checked). If we try all of the possible settings in combination with one another, we’ll find the defect.

Combinations 1, 3, and 4 work all right, but Combination 2 exhibits the problem. We’d require four tests to make sure that we had covered all of the combinations in which A and B could be found. Two variables, and two settings for each; four combinations. [3]

All Pairs TablesNow consider something only a little more complex: a black-box test of a dialog with three sets of radio buttons on it, representing travel needs in North America.

The first set of radio buttons specifies a destination, by country; the second notes the choice of an aisle or window seat; and the third affords the choice of coach, business, or economy class. When the user makes her selections and presses “OK” to close the dialog, the application will take on a different path of execution for each combination of given options in given states.

Assume now that there is a defect that depends upon two conditions. This is known as a double-mode fault. [3]

All Pairs TablesThe most difficult kind of problem to find by black-box testing is one in which several variables are involved, and each must be in a specific, unique state to trigger the problem.

For example, when the country field is set to “USA”, the seat choice to “aisle”, and the service class to “First”, then and only then will the bug be triggered.

This is known as a triple-mode fault, or more generally as a multi-mode fault, which describes faults associated with three or more parameters.[3]

All Pairs Tables

This table makes sure that Canada is tested at least once with each of Coach, Business Class, and First Class, and with the checkbox in the Aisle state and the Window state. Similarly, every option is tested with every other option.

We capture the double-mode fault exposed in combination 3; we need not test combination 12, since its pairs are covered by combination 1, combination 11, and combination 3 (which tests Destination=USA and Class=Coach, and exposes the conflict between them). In this case we cut in half the number of test requirements, from 18 to nine. [3]

Orthogonal Array

Orthogonal array testing is a systematic way of testing all-pair combinations of variables using orthogonal arrays.

It significantly reduces the number of all combinations of variables to test all pair combinations.

By definition, it is a two-dimensional array constructed with special mathematical properties such that choosing any two columns in the array provides every pair combination of each number in the array. [3]

Orthogonal ArrayThere are three rules for selecting an orthogonal array:

There must be at least as many columns as factors (i.e. number of columns is equal to number of factors).

There must be at least enough numbers in the columns to hold the options for each factor. i.e. If you have two options for a factor, then test cases should be there for both the values and if there are three options for a factor, then test cases should be there for all the three values. If we have spare numbers that don’t map to any option for that factor can be represented by a tilde ( ~ ).

There must be at least as many rows or test cases as the product of the two largest numbers of options. For example, if one factor has 4 options and another has 3 options and yet another has 2 options, then 4 x 3 = 12 rows. [3]

Orthogonal Array1) Consider two factors X & Y. Now list down all possible combinations for the pair which will be 0 0, 0 1, 1 0 and 1 1.

2) Consider the factors X & Z and list down all possible combinations for the pair and here notice to shuffle the order while writing the pairs i.e. 0 0, 0 1, 1 1 and 1 0. [3]

Orthogonal Array

Note that here we changed the order of values of Z for the 3rd and 4th rows.

Write down all the values of X similar to how it is written for the pairs X & Y (2). While writing down the values of Z, initially all the values were written as same as Y in pairs X & Y. But then later on to cover all pairs of combinations for the factors Y & Z, the values of Z in 3rd and 4th row is interchanged as highlighted. [3]

Orthogonal Array3) Now consider the factors Y & Z, all the combinations for the pairs is now listed.

Here notice that if we had not changed the order for the pairs X & Z, all combinations would not have been covered for pairs Y & Z Or else in another way, write down the values of Y from X & Y factor and values of Z from X & Z factor.

If we combine this we will get the order of combinations for Y & Z. [3]

Pairwise Testing Techniques - Orthogonal Array

Now the table shown next are the test cases or test combinations developed using orthogonal array.

Note that there are only four test cases after applying this technique, half compared to conventional approaches. [3]

Pairwise Testing Tools

PICT – ‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp.

IBM FoCuS – ‘Functional Coverage Unified Solution’, provided by IBM.

ACTS – ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of the US Government.

Hexawise

Jenny

Pairwiser by Inductive AS.

VPTag free Pairwise Testing Tool. [4]

Conclusion

Pairwise testing technique can dramatically reduce the number of combinations to be covered but remains very effective in terms of fault detection.

It is indeed a smart test design technique that guarantees a win-win situation for both test effort and test effectiveness.

During the Test planning phase of software testing, Pairwise testing technique should always be taken into consideration. Either we are doing it manually or using any tool to generate test cases, it becomes a necessary component of the test plan because it in turn affects Test estimation. [4]

References

[1] https://en.wikipedia.org/wiki/All-pairs_testing

[2] https://www.conformiq.com/2012/01/understanding-pairwise-test-generation/

[3] http://www.developsense.com/pairwiseTesting.html

[4] http://www.softwaretestinghelp.com/what-is-pairwise-testing/

Next Steps

Getting Started Guide

GETTING STARTED RESOURCES FREE TRIAL

DocumentationSupport

TutorialsWhat we're reading this weekBlog

Start you 30-day free trial now