combinatorial testing using covering arrays: going beyond
Post on 28-Jun-2015
895 Views
Preview:
TRANSCRIPT
Combinatorial Testing Using Covering Arrays
- Going Beyond Pairwise Testing
Raghu Kacker, NIST
Yu Lei, UTA
5/15/06
Combinatorial Testing Using Covering Arrays 2
Outline
Introduction
The IPO Strategy
FireEye: An N-Way Testing Tool
Related Work
Conclusion
Combinatorial Testing Using Covering Arrays 3
Why Software Testing?
Modern society is increasingly dependent on the quality of software systems.
A NIST study reports that software failure costs the US economy billions of dollars every year.
Testing (or dynamic analysis) is the most widely used approach to ensuring software quality
Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale
Combinatorial Testing Using Covering Arrays 4
Testing Process
The testing process consists of three stages:
Test Generation: Generate test data For model-based testing, a model (or abstraction) of
the system has to be built.
Test Execution: Test setup and the actual test runs
Test Evaluation: Check if the output is in line with expectations
Combinatorial Testing Using Covering Arrays 5
All About Trade-Off
Testing is labor intensive and can be very costly
estimated to often consume more than 50% of the development cost
Exhaustive testing is often impractical due to resource constraints
From a certain perspective, testing is basically about making a good trade-off between test effort and quality assurance.
Combinatorial Testing Using Covering Arrays 6
Combinatorial Testing
Generates tests from an input parameter model by combining the values of the parameters.
Requires lightweight specification and no knowledge about the implementation structure
Can be virtually applied to any software and at different levels of abstraction
Can be implemented as a push-button feature
Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction
Motivation: Not every parameter contributes to every fault
Can dramatically reduce the number of tests while still preserving important fault detection capabilities.
Combinatorial Testing Using Covering Arrays 7
The ASTUCA Project
Develop new methods and tools for efficient t-way testing for up to 6-way testing
Create new algorithms for efficient test set construction
Provide adequate support for parameter relations and constraints
Explore integration with other software testing tools Conduct empirical evaluation of t-way testing in an
industrial setting
A collaborative effort among the following institutions:
The US National Institute of Standards and Technology George Mason University The University of Texas at Arlington
Combinatorial Testing Using Covering Arrays 8
State-of-the-Art
Existing work has mainly focused on pairwise testing
Many failures are caused by the interaction involving more than two parameters
For certain software, pairwise testing discovers a relatively low percentage of faults
• e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults.
Increased coverage leads to a higher level of assurance
Many applications, e.g., security protocols, have strict requirements on test coverage
Combinatorial Testing Using Covering Arrays 9
Technical Challenges
The computational complexity for t-way testing grows rapidly as the value of t increases
New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets
The number of tests also grows rapidly as the value of t increases
Impractical to manually execute and inspect the results of a large number of test runs
Test generation, test execution, and test evaluation must integrate together to enable test automation
Combinatorial Testing Using Covering Arrays 10
Terminology
N-Way Test set -> N-Way Covering array
Tests -> Rows
Parameters -> Factors or Columns
Values -> Levels
Combinatorial Testing Using Covering Arrays 11
Outline
Introduction
The IPO Strategy
FireEye: An N-Way Testing Tool
Related Work
Conclusion
Combinatorial Testing Using Covering Arrays 12
The Framework (1)
Builds a t-way test set in an incremental manner
A t-way test set is first constructed for the first t parameters,
Then, the test set is extended to generate a t-way test set for the first t + 1 parameters
The test set is repeatedly extended for each additional parameter.
Two steps involved in each extension for a new parameter:
Horizontal growth: extends each existing test by adding one value of the new parameter
Vertical growth: adds new tests, if necessary
Combinatorial Testing Using Covering Arrays 13
The Framework (2)
Strategy In-Parameter-Orderbegin /* for the first t parameters p1, p2 , …, pt*/ T := {(v1, v2, …, vt) | v1, v2, …, vt are values of p1, p2, …, Pk, respectively} if n = t then stop; /* for the remaining parameters */ for parameter pi, i = t + 1, …, n do begin /* horizontal growth */ for each test (v1, v2, …, vi-1) in T do replace it with (v1, v2, …, vi-1, vi), where vi is a value of pi
/* vertical growth */ while T does not cover all the interactions between pi and each of p1, p2, …, pi-1 do add a new test for p1, p2, …, pi to T; endend
Combinatorial Testing Using Covering Arrays 14
Example (1)
Consider a system with the following parameters and values:
parameter A has values A1 and A2
parameter B has values B1 and B2, and
parameter C has values C1, C2, and C3
Combinatorial Testing Using Covering Arrays 15
Example (2)
A BA1 B1A1 B2A2 B1A2 B2
A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1
A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1A2 B1 C2A1 B2 C3
Horizontal Growth Vertical Growth
Combinatorial Testing Using Covering Arrays 16
Comparison to AETG (1)
A commercial tool developed by Telcordia, and protected by a US patent
Starts with an empty set and adds one (complete) test at a time
Each test is locally optimized to cover the most number of missing pairs:
Generate a random order of the parameters Use a greedy algorithm to construct a test that
covers the most uncovered pairs Repeat the above two steps for a given number of
times (suggested 50), and select the best one
Combinatorial Testing Using Covering Arrays 17
Comparison to AETG (2)
A B CA1 B1 C1
A B CA1 B1 C1A1 B2 C2
A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1A2 B1 C2A1 B2 C3
Adds the 1st test Adds the 2nd test Adds the last test
A B C
Combinatorial Testing Using Covering Arrays 18
IPO vs AETG
IPO is deterministic, whereas AETG is inherently non-deterministic
IPO has a lower order of complexity, both in terms of time and space, than AETG
IPO constructs a test set one parameter at a time and in a more incremental nature.
The results generated by IPO are still competitive to those generated by AETG.
IPO is more flexible than AETG IPO can take a test set for a subsystem (i.e., for a
subset of parameters) and then extend it to a complete set for the entire system
Combinatorial Testing Using Covering Arrays 19
Outline
Introduction
The IPO Strategy
FireEye: A Prototype Tool
Related Work
Conclusion
Combinatorial Testing Using Covering Arrays 20
Major Features
Uses Java as the programming language Relatively easier to program, leading to reduced
development time and ease of maintenance Supports the concept of “write-once-run-
everywhere”
Data structures are carefully designed to optimize the runtime performance
A hierarchical structure is used to manage the possible interactions
Allows an incomplete test set to be extended to a complete one
Add new tests or parameters, if necessary, to achieve t-way coverage
Combinatorial Testing Using Covering Arrays 21
Initial Results (1)
[System]
Name: Test Configuration for TCAS
[Parameter]
-- only compare with MINSEP and MAXALTDIFF
Cur_Vertical_Sep : 299, 300, 601
High_Confidence : TRUE, FALSE
Two_of_Three_Reports_Valid : TRUE, FALSE
-- Low and High, only compare with Other_Tracked_Alt
Own_Tracked_Alt : 1, 2
Other_Tracked_Alt : 1, 2
-- only compare with OLEV
Own_Tracked_Alt_Rate : 600, 601
Alt_Layer_Value : 0, 1, 2, 3
-- compare with each other (also see NOZCROSS) and with ALIM
Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840
Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840
Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND
Other_Capability : TCAS_TA, OTHER
Climb_Inhibit : TRUE, FALSE
Combinatorial Testing Using Covering Arrays 22
Initial Results (2)
All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory.
Combinatorial Testing Using Covering Arrays 23
Outline
Introduction
The IPO Strategy
FireEye: An N-Way Testing Tool
Related Work
Conclusion
Combinatorial Testing Using Covering Arrays 24
Classification
Search-Based methods that are mainly developed by computer scientists
AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest
Algebraic methods that are mainly developed by mathematicians
Orthogonal Arrays Recursive Construction
Combinatorial Testing Using Covering Arrays 25
Orthogonal Arrays
Orthogonal arrays can be constructed very fast and are always optimal
Any extra test will cause a pair to be covered for more than once
However, there are several limitations: Orthogonal arrays do not always exist Every parameter must have the same number v of
values Existing methods often require v be a prime power. Every t-way interaction must be covered at the same
number of times
Combinatorial Testing Using Covering Arrays 26
Recursive Construction
Covering arrays are a more general structure, which requires every t-way interaction be covered at least once
Constructing a covering array from one or more covering arrays with smaller parameter sets
Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes
Combinatorial Testing Using Covering Arrays 27
Search-Based vs Algebraic Methods
Search-based methods: Advantages: no restrictions on the input model, and
very flexible, e.g., relatively easier to support parameter relations and constraints
Disadvantages: explicit search takes time, the resulting test sets are not optimal
Algebraic methods: Advantages: very fast, and often produces optimal
results Disadvantages: limited applicability, difficult to
support parameter relations and constraints
The advantages and disadvantages of the two types of methods seem to complement with each other
Combinatorial Testing Using Covering Arrays 28
Outline
Introduction
The IPO Strategy
FireEye: An N-Way Testing Tool
Related Work
Conclusion
Combinatorial Testing Using Covering Arrays 29
Conclusion
Combinatorial testing is a well-defined problem and has been used widely in practice.
The IPO strategy has a lower order of complexity than AETG, and still produces competitive results.
Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible.
Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing.
Combinatorial Testing Using Covering Arrays 30
References
1. Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25.
2. K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434.
3. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7.
4. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland.
5. R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6.
6. Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004)
7. Y. Lei and K. C. Tai , “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261.
8. K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1.
top related