orthogonal array testing
DESCRIPTION
TRANSCRIPT
ORTHOGONAL ARRAY TESTING
Prince Bhanwra [801031024]Prince Bhanwra [801031024]
Ravinder Dahiya [801031025]Ravinder Dahiya [801031025]
PRESENTED BYPRESENTED BY : :
Contents Software Testing Conventional Test Cases
Conventional Testing Issues Software Faults Why OATS..?? Terminology of OATS Examples OATS Advantages OATS Limitations Challenges Mixed Mode OA Available Test Automation Tools References
Thapar University, Patiala2
Testing
SOFTWARE TESTINGSOFTWARE TESTING
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test.
CONVENTIONAL / EXHAUSTIVE TESTINGCONVENTIONAL / EXHAUSTIVE TESTING
A test approach in which the test suite comprises all combinations of input values and preconditions.
Thapar University, Patiala3
Conventional Test Cases
Thapar University, Patiala4
Example:If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue).The possible combinations in conventional test cases would be 27 i.e. 33
Conventional Testing Issues
Conventional Test Cases: Variables:3
Input: 4
Possible cases: 81=34
Variables: 3
Input: 5
Possible Cases: 243 = 35
… Variables: 5
Input: 5
Possible Cases: 3125 = 55
Thapar University, Patiala5
Orthogonal Test Cases Variables:3
Input: 4
Possible cases: 9 Variables: 3
Input: 5
Possible Cases: 11 … Variables: 5
Input: 5
Possible Cases: 21
Software Faults
Thapar University, Patiala6
1. Region Faults.2. Isolated Faults.
Orthogonal Array based testing is highly effective for the detection of region faults with a relatively small number of tests.
Why Orthogonal Array Testing (OATS)..?? Systematic, statistical way to test pair-wise interactions. Interactions and integration points are a major source of
defects. Most defects arise from simple pair-wise interactions.
“When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.”
Exhaustive testing is impossible. Execute a well-defined, concise set of tests that are likely
to uncover most (not all) bugs. Orthogonal approach guarantees the pair-wise coverage
of all variables.
Thapar University, Patiala7
Terminology for working with OA’sOA’s are commonly represented as :LRuns(LevelsFactors) or OA(Runs, Factors, Levels, Strength) or
OAλ(Runs(N); Factors(k), Levels(v), Strength(t))
is an N × k array on v symbols such that every N × t sub-array contains all tuples of size t from v symbols exactly λ times.
Runs (N) – Number of rows in the array, which translates into the number of Test Cases that will be generated.Factors (k) – Number of columns in the array, which translates into the maximum number of variables that can be handled by the array.Levels (v) – Maximum number of values that can be taken on by any single factor.Strength (t) – The number of columns it takes to see all the possibilities equal number of times.
� λ -1 for software testing and is often omitted
Thapar University, Patiala8
Example 1
Thapar University, Patiala9
A Web Page has three distinct sections (Top, Middle, Bottom) that can be individually shown or hidden from user
No.of Factors=3 (Top,middle,Bottom)
No.of Levels =2 (Hidden or shown)
Array Type =L4(2 3) or OA(4,3,2,2)
If we go for exhaustive testing we need :2 x 2 x 2 = 8 Test Cases
OA(Runs, Factors, Levels, Strength)
Example 1 (contd..)
Thapar University, Patiala10
Fixed Level Array: L4(2 3)
F1 F2 F3
Run1 0 0 0
Run 2 0 1 1
Run 3 1 0 1
Run 4 1 1 0
L423 – OA with 4 Runs3 factors with 2 levels
Top Middle Bottom
Test 1 Hidden Hidden Hidden
Test 2 Hidden Visible Visible
Test 3 Visible Hidden Visible
Test 4 Visible Visible Hidden
The Four Test Scenarios (4 Vs. 8)
1 - Display the home page and hide all sections. 2 - Display the home page and show all but the Top section.3 - Display the home page and show all but the Middle section.4 - Display the home page and show all but the Bottom section.
Example 2
Thapar University, Patiala11
Example:If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue).The possible combinations in OATS test cases would be 9.
OATS advantage to select a test set:
Guarantees testing the pair-wise combinations of all the selected variables.
Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables.
Creates a test set that has an even distribution of all pair-wise combinations.
Exercises some of the complex combinations of all the variables.
Is simpler to generate and less error prone than test sets created by hand.
Thapar University, Patiala12
OATS basic fault model: Interactions and integrations are a major source of defects. Most of these defects are not a result of complex
interactions such as “When the background is blue and the font is Arial and
the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.”
Most of these defects arise from simple pair-wise interactions such as “When the font is Arial and the menus are on the right
the tables don’t line up properly.”
Thapar University, Patiala13
OATS basic fault model (contd..): With so many possible combinations of
components or settings, it is easy to miss one. Randomly selecting values to create all of the pair-
wise combinations is bound to create inefficient test sets and test sets with random, senseless distribution of values.
It covers 100% (9 of 9) of the pair-wise combinations, 33% (9 of 27) of the three-way combinations and 11% (9 of 81) of the four-way combinations.
Thapar University, Patiala14
The Challenges The number of tests may increase rapidly as the degree of
interactions increases ‰ Test generation algorithms must be more sensitive in
terms of both time and space requirements ‰ The need for test automation becomes even more
serious. Impractical to manually execute and inspect the results
of a large number of test runs However, on the Web there are pre-calculated
orthogonal array tables for certain numbers of variables and values.
Thapar University, Patiala15
Mixed Mode OA
Thapar University, Patiala16
Available Test Automation Tools
Thapar University, Patiala17
1.CATS (Constrained Array Test System) *)
[Sherwood] Bell Labs.
2.OATS (Orthogonal Array Test System) *) [Phadke] AT&T
3.AETG Telecordia
Web-based, commercial
4. IPO (PairTest) *) [Tai/Lei]
5. TConfig [Williams] Java-applet
6.TCG (Test Case Generator) *) NASA
7.AllPairs Satisfice
Perl script, free, GPL
8.Pro-Test SigmaZone
GUI, commercial
9.CTS (Combinatorial Test Services) IBM
Free for non-commercial use
References http://csrc.nist.gov/groups/SNS/acts/documents/ipo-nist.pdf http://csrc.nist.gov/groups/SNS/acts/documents/aberdeen.ppt http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-system
-design-summer-1998/lecture-notes/l3_matrix_experiments4.pdf http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-
system-design-summer-1998/lecture-notes/l8_orth_arrays.pdf http://delivery.acm.org/10.1145/1290000/1283384/p1-abam.pdf?
key1=1283384&key2=5753427921&coll=DL&dl=ACM&CFID=9321178&CFTOKEN=33991028
http://www.developsense.com/pairwiseTesting.html http://learnsoftwareprocesses.com/2007/08/21/orthogonal-array-testing-
strategy-oats/comment-page-1/#comment-7394 http://www.vietnamesetestingboard.org/zbxe/?document_srl=219676 http://www.pairwise.org/tools.asp
Thapar University, Patiala18
Queries..?Queries..?
Thapar University, Patiala19
Thank you !!Thank you !!
Thapar University, Patiala20