pairwise(tes*ng:(whatitis,(...

#PairOrNot #PairWiseTesting @XBOSoft @philiplew

Pairwise Tes*ng: What It Is, When to Use and Not to Use

#PairOrNot #PairWiseTes*ng

@XBOSo= @philiplew @QAIGlobal


Meet Your Instructor •  Philip Lew

– Software developer – Software product manager, DW product – CEO, software QA services company

•  Relevant specialties and education – Software Usability / UX design and

evaluation – Software Measurement and Metrics – Cornell University B.S. and M.Eng in

Operations Research 2 © 2016 XBOSo=, Inc.-‐ All Rights Reserved.


Today’s Discussion

3

1.  The nuts and bolts of pairwise tes*ng 2.  How to incorporate into your test design and

planning 3.  Introduc*on to tools for pairwise tes*ng

–  ACTS

4.  When and when not to use pairwise tes*ng – Advantages and limita*ons

5.  Going beyond pairwise tes*ng, taking it to the next level

© 2016 XBOSo=, Inc.-‐ All Rights Reserved.


WE LIVE INTO OUR EXPECTATIONS

© 2016 XBOSo=, Inc.-‐ All Rights Reserved. 4


Session Spirit and Expectations •  I can’t read… •  Slides for you as a take-away (many)

– You can write me and I’ll send these to you

•  Interactive – Lots of exercises and chances to learn

5 © 2016 XBOSo=, Inc.-‐ All Rights Reserved.


HOW MANY OF YOU USE A COMBINATORIAL TESTING TECHNIQUE? YES/NO

POLL



Today’s Discussion

7

1.  The nuts and bolts of pairwise tes*ng 2.  How to incorporate into your test design and

planning –  Some tools for pairwise tes*ng

3.  When and when not to use pairwise tes*ng –  Advantages and limita*ons

4.  Going beyond pairwise tes*ng, taking it to the next level

© 2016 XBOSo=, Inc.-‐ All Rights Reserved.


The Problem

•  Many defects only occur with one factor or variable.

•  Or when a combina*on of 2 or more inputs or events occur that interact with each other.

•  Tes*ng every combina*on is impossible. • What common situa*ons have you run into this problem?



Some Examples

•  Plaaorm stacks •  So=ware configura*on op*ons •  Mobile plaaorm •  Any others?



Typical Problem •  An applica*on with simple list box with 10 elements (Let's say 0,1,2,3,4,5,6,7,8,9) along with a checkbox, radio bugon, Text Box and OK Bugon. The Constraint for the Text box is it can accept integer values only between 1 and 100. – List Box -‐ 0,1,2,3,4,5,6,7,8,9 – Check Box -‐ Checked or Unchecked – Radio Bugon -‐ ON or OFF – Text Box – 1 to 100



How Many Test Cases?

•  List Box = 10 •  Check Box = 2 •  Radio Bugon = 2 •  Text Box = 100

•  Total Number of Test Cases using Cartesian Method : 10*2*2*100 = 4000


How can we reduce the # of test cases?


Using Equivalence Par**oning

•  We can consider the list box values as 0 and others •  Radio bugon and check box values cannot be reduced. Each one of them will have 2 combina*ons (ON or OFF).

•  The Text box values can be reduced into three inputs (Valid Integer, Invalid Integer, Alpha-‐Special Character).

•  Calcula*ng the number of cases : 2*2*2*3 = 24


That’s still a lot L


Problems with Equivalence Par**oning

•  For tes*ng printers instead of tes*ng with every single model of printer, we can set all Hewleg-‐Packard inkjet printers as roughly equivalent as a par**on, and all Canon inkjets as another. Or sepasrate by printer type, laser versus inkjet…

•  This can save us a *me and energy, BUT •  We must know how to create classes -‐ what is equivalent to what?

•  We must understand the technology and domain to be tested to create valid par**ons.



Pairwise Tes*ng – (All-‐Pairs Tes*ng)

•  The reasoning: the simplest bugs in a program are generally triggered by a single input parameter/variable.

•  The next simplest category of bugs consists of those dependent on interac*ons between pairs of parameters, which can be discovered with pairwise/all-‐pairs tes*ng.

•  Bugs involving interac*ons between three or more parameters are less common, and also more expensive-‐harder to find.



Let’s Use Pairwise 1.  Order the parameters such that one with most number of

values is the first and the least is placed as the last variable. 2.  Now, start filling the table column by column. Textbox (3). 3.  The next column under discussion would be check box.

Again, Check box can take 2 values. List box can take 2 values.

4.  Now, we need to ensure that we cover all combina*ons between list box and Check box.

5.  Now, we will use the same strategy for checking the Radio Bugon. It can take 2 values.

6.  Verify if all the pair values are covered.


Listbox (0, Others) Radio (On, Off) Checkbox (Check, Uncheck) Textbox(valid Int, Invalid int, Alpha)


Pairwise Results

•  Exhaus*ve Combina*on: > 4000 Test Cases. •  Equivalence par**oning: 24 Test Cases. •  Pair Wise: 6 Test Cases.


TEXT BOX LIST BOX CHECK BOX RADIO BUTTON

VALID INT 0 CHECK ON

VALID INT OTHERS UNCHECK OFF

INVALID INT 0 UNCHECK ON

INVALID INT OTHERS CHECK OFF

ALPHA 0 CHECK OFF

ALPHA OTHERS UNCHECK OFF


LET’S GO AT IT AGAIN!

Actually, enumera*ng all pairs is a mathema*cal challenge…



Pairwise TesDng

Problem: Test System S Inputs X = 1,2 Y = Q,R Z = 5,6

There are 23 = 8 possible combina*ons

System S

X Y Z


X Y Z

1 Q 5

1 R 5

1 Q 6

1 R 6

2 Q 5

2 R 5

2 Q 6

2 R 6


Pairwise TesDng

8 Test Cases You can get rid of Test Case 2 (T2) •  {1, -, 5} is covered in T1 •  {1, R, -} is covered in T4 •  { -, R, 5} is covered in T6

Test ID X Y Z

T1 1 Q 5

T2 1 R 5

T3 1 Q 6

T4 1 R 6

T5 2 Q 5

T6 2 R 5

T7 2 Q 6

T8 2 R 6

1, R, R, 5


Do This Now: •  How many more test cases can

you get rid of? •  List the Test cases you would

keep. •  Time yourself.


Pairwise TesDng Savings

Test ID X Y Z

T1 1 Q 5

T2 1 R 5

T3 1 Q 6

T4 1 R 6

T5 2 Q 5

T6 2 R 5

T7 2 Q 6

T8 2 R 6

Test ID X Y Z

T1 1 Q 5

T4 1 R 6

T6 2 R 5

T7 2 Q 6

8 Test Cases Pairwise testing 4 Test Cases


Darn, we only save ½ ½ is a lot but what does this tell us?


TOOLS MAKE THINGS EASIER Luckily…



No Lack of Tools •  ACTS -‐ AUTOMATED COMBINATORIAL TESTING FOR SOFTWARE (ACTS) •  Automa*c Efficient Test Generator or AETG™ •  Allpairs -‐ Free downloadable executable wrigen in python. •  All Pairs -‐ Free. Command-‐line executable based on a Perl script. By

Sa*sfice. •  Hexawise -‐ Comprehensive pairwise and combinatorial test genera*on

solu*on. So=ware as a service. Free demo accounts. •  Jenny -‐ Free. Executable (C code). •  pairwise -‐ Free, open source using Ruby. •  PICT (Pairwise Independent Combinatorial Tes*ng) tool •  rdExpert – commercial so=ware. Free trial available. •  SmartTest Commercial, executable so=ware. •  Testcover -‐ Web based. Free trial. •  VPTag -‐ Free downloadable executable.



ACTS – Results Output



Pairwise Tes*ng

1.  Pairwise tes*ng protects against pairwise (2 factor) bugs.

2.  Drama*cally reduces the number tests to perform. 3.  Pairwise bugs represent the majority of

combinatory bugs 4.  Available tools means you no longer need to create

these tests by hand.

28 © 2016 XBOSo=, Inc.-‐ All Rights Reserved. 28

Good Stuff, That’s All I need to know right?


WE’RE DONE FOR TODAY



LIMITATIONS OF PAIRWISE TESTING



Limita*ons of Pairwise Tes*ng

1.  Assumes random distribu*on of tests with no focus on important tests



Applying Simple Network Theory

•  How many nodes do you need to kill to take down the network?

•  Why did you choose the nodes you did?



Blind Implementa*on of Pairwise Tes*ng

•  Represents a random agack on the network •  Does not focus on hub nodes (important areas or defect-‐laden)

•  What are your ‘hub’ nodes in your so=ware? – Where is it vulnerable to failure?



Limita*ons of Pairwise Tes*ng

1.  Assumes random distribu*on of tests with no focus on important tests

2.  Some parameters are con*nuous variables and need to be discre*zed to dis*nct values.

3.  Some variables are dependent on others. 4.  How to determine the correct result that

should be expected for each of the combina*ons (000’s) of test inputs?



Pairwise Tes*ng Fails When

•  Not selec*ng the right input. •  You don’t have a good enough oracle (you don’t know what is the correct result).

•  High probability combina*ons not covered enough.

•  Not knowing how the variables interact (constraints and dependencies).



Where Were We?

________ ________ _________

36 © 2016 XBOSo=, Inc.-‐ All Rights Reserved.

Problem Pairwise Testing

Combinatorial Testing


Combinatorial Tes*ng •  Pairwise Tes*ng is a subset of Combinatorial tes*ng •  Pairwise (2-‐way combina*ons) tes*ng is some*mes used to

obtain reasonably good results at low cost, but pairwise tes*ng may miss 10% to 40% of the bugs.

•  Not sufficient for mission-‐cri*cal so=ware. •  Empirical data gathered by NIST and others suggest that

so=ware failures are triggered by only a few variables interac*ng (6 or fewer).

•  Combinatorial tes*ng beyond 2-‐way has been limited, due to a lack of good algorithms for higher interac*on levels (4-‐6)

•  New algorithms have made combinatorial tes*ng beyond pairwise prac*cal.



https://en.wikipedia.org/wiki/All-pairs_testing


What If? •  What if a failure is triggered by a combina*on of 3, 4, or more variables?

•  Pairwise tests may not detect these defects. •  Is tes*ng all 4-‐way combina*ons (T=4) enough to detect all errors?

•  What degree of interac*on (strength) , i.e. 2,3,4,5.. occurs in real failures?

•  What is my tolerance for risk?



T-‐Way Tes*ng # Interac*ng Parameters Leading to Failure


What is your risk tolerance? What is it dependent on?


“FLY WITH US, WE FOUND 90% OF THE DEFECTS”.


I’ll fly with you!


How much risk can we tolerate?

•  At first glance? •  210 = 1,024 possible combina*ons! •  To reduce that, we can use T-‐Way according to our goals


About 90% with 3-Way Interactions


APPLYING COMBINATORIAL TESTING TECHNIQUES

Input Parameters (Variables)



Word Processing Input Variables •  User can select 10 ways to modify some highlighted text: subscript, superscript, underline, bold, italic, strikethrough, emboss, shadow, small caps, or all caps.

•  System must process the input and modify the text on the screen correctly.

•  Most op*ons can be combined, such as bold and small caps, but some are incompa*ble, such as subscript and superscript.

•  How many combina*ons? © 2016 XBOSo=, Inc.-‐ All Rights Reserved. 44



Mobile Tes*ng Plaaorm prolifera*on AND OS op*ons



Android Op*on Prolifera*on Parameter Values # Values

HardKeyboardHidden No, Undefined, Yes 3

KeyboardHidden No, Undefined, Yes 3

Keyboard 12Key, NoKeys, Querty, Undefined 4

Naviga*onHidden No, Undefined, Yes 3

Naviga*on Dpad, NoNav, Trackball, Undefined, Wheel

5

Orienta*on Landscape, Portrait, Square, Undefined 4

ScreenLayout_Long Mask, No, Undefined, Yes 4

ScreenLayout_Size Large, Mask, Normal, Small, Undefined 5

TouchScreen Finger, Notouch, Stylus, Undefined 4


3x3x4x3x5x4x4x5x4=? 172,800

What Would You Reduce?


What Would You Reduce?


Parameter Values # Values

HardKeyboardHidden No, Undefined, Yes 3

KeyboardHidden No, Undefined, Yes 3

Keyboard 12Key, NoKeys, Querty, Undefined 4

Naviga*onHidden No, Undefined, Yes 3

Naviga*on Dpad, NoNav, Trackball, Undefined, Wheel

5

Orienta*on Landscape, Portrait, Square, Undefined 4

ScreenLayout_Long Mask, No, Undefined, Yes 4

ScreenLayout_Size Large, Mask, Normal, Small, Undefined 5

TouchScreen Finger, Notouch, Stylus, Undefined 4


How Can T-‐Way Help? t # Tests % of Total (172k) % Defects Found

2 29 .02 76%

3 137 .08 95%

4 625 0.4 97%

5 2532 1.5 99%

6 9168 5.3 100% **


•  The savings as a percentage of exhaus*ve tes*ng depends on the number of parameters and the # of values for the parameters.

•  With larger systems the savings can be enormous.


Key Factors When Using T-‐Way •  Number of parameters •  Number of values per parameter •  T (also called interac*ons, where T=2 for pairwise) •  Number of tests increases rapidly as the value of t is increased

•  Rate of increase depends on the number of values per parameter. –  Binary variables, result in far fewer tests than parameters with many values each.




Configura*on Tes*ng -‐ Let’s go back and look at this again!



Configura*on Tes*ng -‐ Plaaorms How many configura*ons are there? •  OS (Windows, Apple OSX, Linux) •  Browser (IE, Firefox) •  Protocol (IPv4, IPv6) •  Processor (Intel, AMD) •  Database (MySQL, Sybase, Oracle) •  What others are you used to? •  How many possible plaaorms? •  3x2x2x2x3=72



Configura*on Tes*ng – Plaaorms Pairwise Configura*ons (WOW!)


Test ID OS Browser Protocol CPU DBMS 1 Win

MySQL IE IPv4 Intel 10

2 Win Sybase FFX IPv6 AMD 10

3 OSX MySQL FFX IPv4 AMD 8

4 OSX Sybase IE IPv6 Intel 8

5 Linux Oracle IE IPv4 AMD 8

6 Linux Oracle FFX IPv6 Intel 7

7 Win MySQL ~IE IPv6 ~AMD 1

8 Win Sybase ~FFX IPv4 ~Intel 1

9 OSX Oracle ~IE ~IPv4 ~Intel 1

10 Linux MySQL ~FFX ~IPv6 ~Intel 1

11 Linux Sybase ~IE ~IPv4 ~AMD 1

12 Win Oracle ~FFX ~IPv6 ~AMD 1

1 Win MySQL IE IPv4 Intel 10





































































































































































































































































































































































































































































































































































































































































































































Variable Interac*ons


•  Some combina*ons don’t exist or invalid – OSX and IE

•  Can’t simply delete them, why? •  Need to delete and supplement •  What other interac*ons or interdependencies can you think of?


USING TOOLS Using ACTS Demonstra*on



Configura*ons Revisited

Parameter (variable)

Values

OS Windows, OSX, Linux Browser IE, Firefox Protocol IPv4, IPv6 CPU Intel, AMD DBMS MySQL, Sybase, Oracle



Using ACTS t # tests % of Total % defects found

2 10 14 76%

3 18 25 95%

4 36 50 97%

5 72 100 99%


We could test all these platforms. It’s a lot, but doable


ACTS with T=3



Invalid Combina*ons -‐ Interac*ons


Test ID OS Browser Protocol CPU DBMS

1 Win IE IPv4 Intel MySQL

2 Win FFX IPv6 AMD Sybase

3 Win IE IPv6 Intel Oracle

4 OSX FFX IPv4 AMD MySQL

5 OSX IE IPv4 Intel Sybase

6 OSX FFX IPv4 Intel Oracle

7 Linux IE IPv6 AMD MySQL

8 Linux FFX IPv4 Intel Sybase

9 Linux FFX IPv4 AMD Oracle

10 OSX FFX IPv6 AMD Oracle


ACTS with T=2 and Contraints



Using Pairwise and T-‐Way Together

•  Using Pairwise can help us find many … but not all and depending on our risk tolerance, maybe not enough

•  Using T-‐Way can help, but each T+1 adds a lot of effort…

•  Best to use T-‐Way with modified set of parameters and values according to our experience and exper*se based on the situa*on and goals



Understand Pairwise and T-‐Way in Perspec*ve

•  Where do you have many combina*ons?

•  Where do you have dis*nct inputs? Either naturally or ability to create equivalence classes.

•  Where do you have many parameter values?

•  Know the expected results


Test Strategy

Test Design

T-‐Way


Combining Combinatorial Applica*ons


System Under Test

Inputs (test cases)

Configura*ons


Numerous Ways/Places We can use Combinatorial Tes*ng

Telecomm How many configura*ons are there? •  Types of calls (local, LD, interna*onal) •  Types of payment (phone card, 800, 900, credit card, monthly bill)

•  Access (ISDN, T1, VOIP, PBX) •  Billing Server (Windows, Linux/MySQL, Oracle)



Reading Material 1.  All Pairs Tes*ng, Wikipedia, hgps://en.wikipedia.org/wiki/All-‐pairs_tes*ng 2.  AUTOMATED COMBINATORIAL TESTING FOR SOFTWARE (ACTS), NIST,

hgp://csrc.nist.gov/groups/SNS/acts/index.html 3.  Combinatorial Tes*ng: Ra*onale and Impact, Rick Kuhn,

hgp://csrc.nist.gov/groups/SNS/acts/documents/kuhn-‐icst-‐14.pdf 4.  Design of Experiments, Wikipedia, hgps://en.wikipedia.org/wiki/Design_of_experiments 5.  Efficient Tes*ng Using the Pairwise Approach, Bernie Berger,

hgp://www.s*ckyminds.com/ar*cle/efficient-‐tes*ng-‐using-‐pairwise-‐approach 6.  An Inves*ga*on of the Applicability of Design of Experiments to So=ware Tes*ng, D. Richard Kuhn, M. Reilly,

hgp://csrc.nist.gov/staff/Kuhn/kuhn-‐reilly-‐02.pdf 7.  Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday

Life, Albert-‐Laszlo Barabasi 8.  Pairwise Tes*ng, A Best Prac*ce That Isn’t, James Bach,

hgp://www.tes*ngeduca*on.org/wtst5/PairwisePNSQC2004.pdf 9.  Pairwise Tes*ng for Dummies, David Fries, hgp://blogs.msdn.com/b/davfries/archive/2004/06/17/158900.aspx 10.  Pairwise Tes*ng in the Real World, hgps://msdn.microso=.com/en-‐us/library/cc150619.aspx 11.  Pairwise Tes*ng, Michael Bolton, hgp://www.developsense.com/pairwiseTes*ng.html 12.  PRACTICAL COMBINATORIAL TESTING, D. Richard Kuhn, Raghu N. Kacker, Yu Lei, Na*onal Ins*tute of Standards

and Technology 13.  Tutorial on All Pairs Tes*ng, hgp://www.tutorialspoint.com/so=ware_tes*ng_dic*onary/all_pairs_tes*ng.htm



Thanks Ques*ons and Answers

Philip Lew @philiplew [email protected]

Some resources: http://www.xbosoft.com/knowledge_center/

www.xbosoft.com

pairwise(tes*ng:(whatitis,(...

Documents