mutation testing implements grammar-based testing · grammar-based integration mutation testing...
TRANSCRIPT
www.tugraz.at
Mutation Testing implementsGrammar-Based TestingManuel Jelinek,AK Softwaretechnologie
Graz, 20. Januar 2014
2
Agenda
Agenda
1. Grammar-Based Mutation Analysis
2. Grammar-Based Testing of Programs
3. Grammar-Based Integration Mutation Testing
4. Grammar-Based Testing Using Models
5. Grammar-Based Input Testing
6. Conclusion
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
3
Grammar-Based Mutation Analysis
Grammar-Based Mutation Analysis
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
4
Grammar-Based Mutation Analysis
Software engineers often use structures from automatatheory:
Programming languages defined in BNF (backus-naur-form)Program behavior described as finite state machinesAllowable inputs defined by grammars
BNF syntax example for bank transactions:bank ::= action*
action ::= dep | deb
dep ::= "deposit" account amount
deb ::= "debit" account amount
account ::= digit3
amount ::= "$" digit5 "." digit2
digit ::= "0" | "1" | "2" | "3" | "4" |
"5" | "6" | "7" | "8" | "9"
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
4
Grammar-Based Mutation Analysis
Software engineers often use structures from automatatheory:
Programming languages defined in BNF (backus-naur-form)Program behavior described as finite state machinesAllowable inputs defined by grammars
BNF syntax example for bank transactions:bank ::= action*
action ::= dep | deb
dep ::= "deposit" account amount
deb ::= "debit" account amount
account ::= digit3
amount ::= "$" digit5 "." digit2
digit ::= "0" | "1" | "2" | "3" | "4" |
"5" | "6" | "7" | "8" | "9"
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
5
Grammar-Based Mutation Analysis
bank -> action*
-> action action*
-> dep action*
-> deposit account amount action*
-> deposit digit3 amount action*
-> deposit digit digit2 amount action*
-> deposit 7 digit2 amount action*
-> deposit 7 digit digit amount action*
-> deposit 73 digit amount action*
-> deposit 739 amount action*
-> deposit 739 $ digit5 . digit2 action*
-> deposit 739 $digit2 .digit2 action*
-> deposit 739 $digit digit.digit2 action*
-> deposit 739 $1 digit.digit2 action*
-> deposit 739 $12. digit2 action*
-> deposit 739 $12. digit digit action*
-> deposit 739 $12.3 digit action*
-> deposit 739 $12.35 action*
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
6
Grammar-Based Mutation Analysis
Unmutated derivation test criteria
Syntax-based coverage criteria:
Terminal Symbol Coverage (TSC):The set of test requirements, TR, contains eachterminal symbol t in the grammar G(14 in bank example)Production Coverage (PC):TR contains each production p in the grammar G(17 in bank example)Derivation Coverage (DC):TR contains every possible string that can be derived from thegrammar G(200.000.000 possible strings in bank example)
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
7
Grammar-Based Mutation Analysis
Mutated derivation test criteria
Grammars describe both valid and invalid stringsBoth types can be produced as mutantsA mutant is a variation of a valid string
Mutants may be valid or invalid
Mutation is based on ”mutation operators”and ”ground strings”Never mutate more then one operator at the same time
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
8
Grammar-Based Mutation Analysis
Mutated derivation test criteria
Syntax-based coverage criteria:
Mutation Coverage (MC):For each m ε M, TR contains exactly one requirement, to kill mThe amount of mutants killed is called the mutation scoreMutation Operator Coverage (MOC):For each mutation operator, TR contains exactly onerequirement, to create a mutated string m that is derived usingthe mutation operatorMutation Production Coverage (MPC):For each mutation operator, TR contains several requirements,to create one mutated string m that includes every productionthat can be mutated by that operator
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
9
Grammar-Based Testing of Programs
Grammar-Based Testing of Programs
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
10
Grammar-Based Testing of Programs
Program-based mutation
The original and most widely known application of syntax-basedtesting is to modify programsOperators modify a ground string (program under test) to createmutant programsMutant programs must compile correctly (valid strings)Mutants are not tests, but used to find testsOnce mutants are defined, tests must be found to cause mutantsto fail when executedThis is called ”killing mutants”
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
11
Grammar-Based Integration Mutation Testing
Grammar-Based Integration MutationTesting
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
12
Grammar-Based Integration Mutation Testing
also called ”interface mutation testing”Testing connections among separate program unitsIn Java, testing the way classes, packages and components areconnectedThis tests features that are unique to object-orientedprogramming languagesFour types of mutation operators
Change a calling method by modifying values that are sentto a called methodChange a calling method by modifying the callChange a called method by modifying values that enter andleave a method
Includes parameters as well as variables from higher scopes (classlevel, package, public, etc.)
Change a called method by modifying return statementsfrom the method
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
13
Grammar-Based Testing Using Models
Grammar-Based Testing UsingModels
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
14
Grammar-Based Testing Using Models
Model-based grammars
Languages that describe software in abstract terms
Formal specification languages such as Z, SMV, OCL, etc.Informal specification languagesDesign notations such as statecharts, FSMs and other UMLdiagram notations
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
15
Grammar-Based Testing Using Models
Specification-based mutation
A FSM (finite state machine) is essentially a graph G
Nodes are statesEdges are transitions
A formalization of an FSM is:
States are implicitly defined by declaring variables withlimited rangeThe state space is then the Cartesian product of the rangesof the variablesInitial states are defined by limiting the ranges of some orall of the variablesTransitions are defined by rules that characterize the sourceand target of each transition
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
15
Grammar-Based Testing Using Models
Specification-based mutation
A FSM (finite state machine) is essentially a graph G
Nodes are statesEdges are transitions
A formalization of an FSM is:
States are implicitly defined by declaring variables withlimited rangeThe state space is then the Cartesian product of the rangesof the variablesInitial states are defined by limiting the ranges of some orall of the variablesTransitions are defined by rules that characterize the sourceand target of each transition
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
16
Grammar-Based Testing Using Models
Example SMV (symbolic model verifier)
MODULE main
#define false 0
#define true 1
VAR
x, y : boolean;
ASSIGN
init (x) := false;
init (y) := false;
next (x) := case next (y) := case
!x & y : true; x & !y : false;
!y : true; x & y : y;
x : false; !x & y : false;
true : x; true : true;
esac; esac;
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
17
Grammar-Based Testing Using Models
Mutations and Test CasesMutating FSMs requires mutation operatorsMost FSM mutation operators are similar to program languageoperators
Constant Replacement operator:changes a constant to each other constantin the next(y) case: !x & y: false
is mutated to !x & y: true
To kill this mutant, we need a sequence of states (a path) thatthe original machine allows but the mutated machine does notThis is what model checkers do
Model checkers find counterexamples – paths in themachine that violate some propertyProperties are written in ”temporal logic”– logicalstatements that are true for some period of time!x & y: false has different result from !x & y: true
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
18
Grammar-Based Testing Using Models
Counter-example SMV machinenext (y) := case
x & !y : false;
x & y : y;
!x & y : false;
true : true;
esac;
SPEC AG (!x & y) -> AX (y=true)
The model checkershould produce:
/*state 1*/ {x=0, y=0}
/*state 2*/ {x=1, y=1}
/*state 3*/ {x=0, y=1}
/*state 4*/ {x=1, y=0}
This represents a test case thatgoes from nodes FF to TT to FT toTF in the original FSMIf no sequence is produced, themutant is equivalent
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
19
Grammar-Based Input Testing
Grammar-Based Input Testing
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
20
Grammar-Based Input Testing
Input space grammar
The set of allowable inputs to a program, method or softwareCan be described in many waysMost input spaces can be described as grammarsGrammars are usually not provided, but creating them is avaluable service by the tester (errors will often be found simplyby creating the grammar)Software should reject or handle invalid data
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
21
Grammar-Based Input Testing
String mutation input-based testing
Mutating input grammars
Mutants are testsCreate valid and invalid stringsNo ground strings ⇒ no killing
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
22
Grammar-Based Input Testing
Input grammar mutation operators
Nonterminal replacementEvery nonterminal symbol in a production is replaced by othernonterminal symbols
Mutate ‘dep ::= "deposit" account amount’ to:dep ::= "deposit" amount amount
--> deposit $1500.00 $3789.88
dep ::= "deposit" account digit
--> deposit 4400 5
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
23
Grammar-Based Input Testing
Input grammar mutation operators
Terminal replacementEvery terminal symbol in a production is replaced by otherterminal symbols
Mutate ‘amount ::= "$" digit+ "." digit2’ to:amount ::= "." digit+ "." digit2
--> deposit 4400 .1500.00
amount ::= "$" digit+ "$" digit2
--> deposit 4400 $1500$00
amount ::= "$" digit+ "1" digit2
--> deposit 4400 $1500100
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
24
Grammar-Based Input Testing
Input grammar mutation operators
Terminal and nonterminal deletionEvery terminal and nonterminal symbol in a production is deleted
Mutate ‘dep ::= "deposit" account amount’ to:dep ::= account amount
--> 4400 $1500.00
dep ::= "deposit" amount
--> deposit $1500.00
dep ::= "deposit" account
--> deposit 4400
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
25
Grammar-Based Input Testing
Input grammar mutation operators
Terminal and nonterminal duplicationEvery terminal and nonterminal symbol in a production isduplicated
Mutate ‘dep ::= "deposit" account amount’ to:dep ::= "deposit" "deposit" account amount
--> deposit deposit 4400 $1500.00
dep ::= "deposit" account account amount
--> deposit 4400 4400 $1500.00
dep ::= "deposit" account amount amount
--> deposit 4400 $1500.00 $1500.00
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014
26
Conclusion
Conclusion
Grammars describe both valid and invalid stringsBoth types can be produced as mutantsA mutant is a variation of a valid stringMutation is based on ”mutation operators”and ”ground strings”Ground string: A string in the grammarMutation Operator: A rule that specifies syntactic variations ofstrings generated from a grammarMutant: The result of one application of a mutation operator
Manuel Jelinek, AK SoftwaretechnologieGraz, 20. Januar 2014