generating minimal fault detecting test suites for boolean ... · (minimization) a test suite is...
TRANSCRIPT
![Page 1: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/1.jpg)
GENERATING MINIMAL FAULT DETECTING TEST SUITES FOR BOOLEAN EXPRESSIONS
ANGELO GARGANTINIUNIVERSITY OF BERGAMOITALY
GORDON FRASERUNIVERSITY OF SAARLANDESGERMANY
![Page 2: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/2.jpg)
OUTLINE1. Intoduction about testing of (DNF) logic expressions
• Boolean expressions: where to find them, how to test them• For boolean specification in DNF
• Fault classes• Classical testing criteria
2. A new way of generating fault detecting tests
• How to discover a fault• Using SAT solvers to generate tests• Optimizations
3. Experiments
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
2
![Page 3: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/3.jpg)
LOGIC PREDICATES AND CLAUSESA predicate is an expression that evaluates to a Boolean valuePredicates can contain
• boolean variables• non-boolean variables that contain >, <, ==, >=, <=, !=• boolean function calls
Internal structure is created by logical operators¬ the negation operator∧
the and operator
∨
the or operator→
the implication operator
⊕
the exclusive or operator↔
the equivalence operator
A clause is a predicate with no logical operators
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
3
![Page 4: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/4.jpg)
EXAMPLES(a < b) ∨
f (z) ∧
D ∧
(m >= n*o)
Four clauses:
• (a < b) – relational expression• f (z) – boolean-valued function• D – boolean variable• (m >= n*o) – relational expression
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
4
![Page 5: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/5.jpg)
DISJUNCTIVE NORMAL FORM
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
8
![Page 6: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/6.jpg)
FAULT CLASSES FOR BOOLEAN EXPRESSIONSThere exist typical errors done by programmers
Errors cause faults in the expression
• Faults grouped in fault classes• For DNF expressions there classical fault classes
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
10
![Page 7: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/7.jpg)
DNF FAULT CLASSES
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
11
![Page 8: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/8.jpg)
FAULT CLASS HIERACHY
Not all the faults are equal
• Among the fault classes it may exist a hierarchy
• A class F1 subsumes another F2 if a test suite that is able to detect all the faults in F1 then it will also detect all the faults in F2.
The hierarchy is useful when generating tests
A. GargantiniGenerating minimal fault detecting test suites for Boolean expressions 13
F1
F2
![Page 9: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/9.jpg)
FAULT DETECTION RELATIONSHIPS
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
14
Expression Negation Fault
ENF
Literal Insertion
FaultLIF
Term Omission Fault
TOF
Literal Reference
FaultLRF
Literal Negation Fault
LNF
Operator Reference
FaultORF+
Literal Omission
FaultLOF
Term Negation
FaultTNF
Operator Reference
FaultORF*
![Page 10: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/10.jpg)
TESTING CRITERIATo target these faults, several testing criteria have been (and are continusly) introduced
A testing criteria must define an algorithm to derive the tests
• It analyzes the structure of the expression• It find the right truth values for the clauses
simplest: implicant Coverage
• Make each implicant evaluate to “true”
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
15
![Page 11: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/11.jpg)
OTHER TESTING CRITERIAMAX-A and MAX-B
• Weyuker, Goradia, and SinghMultiple Unique True Points (MUTP)Multiple Near False Points (MNFP)Corresponding Unique True Point Near False Point (CUTPNFP)MUMCUT = MUTP + MNFP + CUTPNFP
• Chen, Lau, and YuIt has been proved that MUMCUT criteria detect all the faults in the hierarchy
• Very efficient (faults/number of tests)• Several variations to reduce number of tests• New criteria with different fault detection capability
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
16
![Page 12: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/12.jpg)
A NEW WAY TO GENERATE FAULT DETECTING TESTS
A. Gargantini
17Generating minimal fault detecting test suites for Boolean expressions
![Page 13: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/13.jpg)
BASIC PRINCIPLESInstead of introducing a new testing criteriona generation methods that targets explicitly the fault classes
• new fault classes can be added if neeeded • or removed
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
18
trendTesting and proving become complementary: tools, methods and techinques generally used for property verification can be effiently employed to solve testing problems
trendTesting and proving become complementary:tools, methods and techinques generally used for property verification can be effiently employed to solve testing problems
![Page 14: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/14.jpg)
DETECTION CONDITION
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
19
![Page 15: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/15.jpg)
DETECTION CONDITION EXAMPLE
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
20
![Page 16: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/16.jpg)
DETECTING ALL THE FAULTS IN A CLASS
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
21
![Page 17: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/17.jpg)
ADEQUACY OF A TEST SUITE
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
22
![Page 18: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/18.jpg)
SAT-BASED TEST GENERATION METHOD
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
23
Fault Classes
Bool Spec
Test Predicate Generator
Test Predicate
s
Test Suite Generator
Test Suite
Model = test
SAT
Test predicate
Very naive: a lot of tests and time
![Page 19: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/19.jpg)
UNFEASIBLE TEST PREDICATES
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
24
![Page 20: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/20.jpg)
MONITORIN G
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
25
Test Predicate
s
Test Suite Generator
Test Suite
Test + Cov Info
SAT
Test predicate
Coverage Evaluator Coverage Evaluator
Model = test A test covers other
test predicates?
![Page 21: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/21.jpg)
MONITORING COVERAGE
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
26
![Page 22: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/22.jpg)
ORDERING
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
27
Test Predicates
Test Suite Generator
Test Suite
Test + Cov Info
SAT
Test predicate
TP Ordering
TP Ordering
Coverage Evaluator
Model = test Ordering test
predicates may improve efficiency?
![Page 23: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/23.jpg)
ORDERING TEST PREDICATES
When monitoring is applied the order in which test predicates are selected may impact the size of the resulting test suite.
• Gordon Fraser, Angelo Gargantini, and Franz Wotawa. On the order of test goals in specification-based testing. Journal of Logic and Algebraic Programming, 78(6), 472-490, 2009.
Random order • randomly take the next tp
Subsuming order• If the subsuming relation between fault classes is known, or at least a
subsumption relationship is suspected to be in place due some empirical data, it can be used to order tps
• Start with the test predicates coming from top classes in the hierarchy• LIF, LRF, LOF, TOF, LNF, ORF+, ORF*, TNF, and ENF.
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
28
![Page 24: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/24.jpg)
COLLECTING
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
29
Test Predicate
s
Test Suite Generator
Test Suite
Test + Cov Info
SAT
Test predicate
TP Ordering
Coverage Evaluator
Model = test
Instead of ONE tp , take many TPs, so the test will cover them all
TP Collecting
TP Collecting
![Page 25: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/25.jpg)
COLLECTING TEST PREDICATES
Test suite builder
Model = test that covers all the test predicates collected
SATCollected test
predicatestp1 /\ tp2 /\ tp3
Instead of one test for every tp, collect the tps to build a conjoint
Note: When collecting, infeasible tps must be ignored, incompatible tps must be skipped
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
30
![Page 26: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/26.jpg)
REDUCTION
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
32
Test Predicate
s
Test Suite Generator
Test Suite
Test + Cov Info
SAT
Test predicate
TP Ordering
Coverage Evaluator
Model = test
Any unnecessary test in the test suite?
TP Collecting
ReductionReduction
![Page 27: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/27.jpg)
POST REDUCTION (MINIMIZATION)A test suite is minimal with regard to an objective if removing any test case from the test suite will lead to the objective no longer being satisfied.
• Some tests may be uselesssimple greedy heuristic to the minimum set covering problem for test suite minimizationNote: Monitoring and minimization can behave very differently:
• Minimization requires existing, full test suites • while monitoring checks test predicates on the fly
during test case generation
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
33
![Page 28: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/28.jpg)
EXPERIMENTS
A. Gargantini
34Generating minimal fault detecting test suites for Boolean expressions
![Page 29: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/29.jpg)
EXPERIMENTSBenchmark: 20 Boolean expressions in a traffic collision avoidance system (TCAS).
• Introduced for MAX-A and MAX-B (Weyuker et al.)• Used by MUMCUT (Chen, Lau, and Yu)• And minimal-MUMCUT (Kaminksy and Ammann)
GOAL: reduce the test suite size
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
35
![Page 30: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/30.jpg)
COMPARISON AMONG OUR STRATEGIES
Optimization Reduction of the test suite sizeAvg Var Max
Subsumption order instead of random order
5% 0.4% 19%
Reduction 6% 0.4% 31%Collection 24% 4% 71%
The smallest test suites are generated with monitoring, ordering by subsumption, collecting, and minimizing.
A. GargantiniGenerating minimal fault detecting test suites for Boolean expressions 36
![Page 31: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/31.jpg)
COLLECTION IS EXPENSIVE
328 times the time required by the strategy without collectionCollecting test predicates is effective at reducing the number of test cases, but computationally expensive.
NO COLL COLLRND SUB RND SUB
Time (sec) 190.0 44.2 45821.2 18380
A. GargantiniGenerating minimal fault detecting test suites for Boolean expressions 37
![Page 32: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/32.jpg)
COMPARISON WITH MUMCUTS
• Much better than the original MUMCUT strategy
• Always better than the new MUMCUT strategy
• Comparable w.r.t Minimal MUMCUT
A. GargantiniGenerating minimal fault detecting test suites for Boolean expressions 38
SA T
![Page 33: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/33.jpg)
COMPARISON WITH MIN- MUMCUT
Our method reduces the number of test cases necessary to cover all faults of these classes in comparison to MinimalMUMCUT.A. Gargantini
Generating minimal fault detecting test suites for Boolean expressions 39
![Page 34: Generating minimal fault detecting test suites for Boolean ... · (MINIMIZATION) A test suite is minimal with regard to an objective if removing any test case from the test suite](https://reader035.vdocuments.site/reader035/viewer/2022071004/5fc1be8d8808110e286c84d7/html5/thumbnails/34.jpg)
CONCLUSIONSIt is possible to generate tests explicitly targeting faults
• SAT solvers can be employedSeveral optimizations can be applied
• Monitoring, ordering, collecting, minimizationIn comparison to *MUMCUT, it reduces the number of test cases necessary to cover all faults of these classes
Future work:
• Not only DNF • Improve efficiency: reducing the number of runs of the SAT
A. Gargantini Generating minimal fault detecting test suites for Boolean expressions
40