1

Automating the Generation of Mutation Tests

Mike Papadakis and Nicos Malevris

Department of Informatics Athens University of Economics and Business

Test Data Generation Approaches

Symbolic execution Select set of paths Produce a system of algebraic

constraints Solve and generate test cases

Challenges (Symbolic execution) Infeasible paths Complex-unhandled expressions Availability of source code

2

Test Data Generation Approaches

Search based approaches Definition of program input Dynamic program execution Fitness function

o (efficiency and effectiveness)

Challenges Handling of dynamic program inputs Require a high number of executions Handling of specific cases (e.g. flag

problem) Effective fitness function

3

Test Data Generation Approaches

Dynamic Symbolic Execution Simultaneously perform actual and

symbolic executiono Simplify the process based on concrete program

execution Produce a system of algebraic

constraints Solve and generate test cases

Challenges Infeasible paths Scalability issues

4

5

In order to kill a mutant, tests must

Stage 1: Reach the mutant

Stage 2: infect the program state

Stage 3: propagate the infected state

Killing Mutants

6

Reach the mutant (Reachability condition) Infect program state (necessity condition)

Program execution must cause a discrepancy in the program state

E.g. Orig: a > b, Mut = a ≥ b → a > b ≠ a ≥ b o a == b where Orig = false, Mut = true

Propagate (sufficiency condition) Heuristic approaches

o explore path space, measure of mutant’s impact

Joint satisfaction (Reach && Infect && Propagate)

Killing Mutants

7

Killing Mutants (path selection)

Symbolic Execution Reach

o Use paths from program input to mutant Infect

o Use necessity constraints to infect program stateo Original Statement ≠ Mutated Statement

Propagateo Explore the path space (from mutant to program output)

Path selection Strategy

8

Enhanced Test Model (Enhanced Control Flow Graph) Represent mutants in the model Guide the selection of paths and the produced

constraints based on the Shortest path strategy

Killing Mutants (path selection)

9

Killing Mutants (path selection)

Shortest Path Strategy, the k-value An efficient path selection strategy. (eliminates

the effects of infeasible paths) Select shortest paths from program input (S) to

the mutant statement (m)

Heuristic approach to weak mutation Strong mutation can be tackled incrementally

xmxSkkm ),()()(

),(),( )()()( FxmxxmxS trkm

10

Killing Mutants Dynamic approaches

Dynamic approaches Gain the required information from actual

program execution Require high number of program execution

Mutation Testing Need to unify the runtime information of the

original and mutant programs Need to include mutant neccesity and

sufficiency conditions into program structure Need for efficiency (high cost)

11

Mutant Schemata Mutant schemata-Parameterized mutants

Technique for efficiently producing mutants Embeds all mutants into one meta-mutant

program Use of global parameter to specify the

mutants Example original meta-mutant a > b + c → RelationalGT(a, ArithmeticPlus(M(b),

M(c)))

M(x) mutates variable x, e.g. abs(x)Arithmetic(x, y) mutates arithmetic operator, e.g. x

- yRelational(x, y) mutates relational operator, e.g. x ≥

y

12

Mutant Schemata (approach I) Weak mutation

Include the mutant evaluation inside schematic functionso Original expression ≠ Mutant expressiono Original statement ≠ Mutant statement

After evaluation continue with the original execution (schematic function returns the original expression)

Able to execute all mutants with one execution run

Embeds all mutant’s killable condition into program structure (mutant evaluation → branches)

Mutant is killedMutant is Alive

Mutant Schemata

NodeN-1

NodeN

NodeN+1

NodeN+2

NodeN-1

NodeN

NodeN+1

NodeN+2

NodeN_M[1]

NodeN_M[2]

NodeN_M[n-1]

NodeN_M[n]

...

Proposed approach (Mutants to branches) Mutants M[1]…M[n] on node N. Mutant

EvaluationBranches

13

14

Mutant Schemata (approach II) Strong mutation

In Same lines as Weak o Include the mutant evaluation inside schematic functions

After evaluation continue with the mutant execution (schematic function returns the mutant expression)

Incremental approach, from weak to strong Requires multiple execution runs

Record original and mutant’s execution paths Evaluate strongly killed mutants

15

Killing Mutants (DSE)

Use the shortest path heuristic Negate the condition that will lead closer to mutant

node (reach the mutant)

Weakly kill the mutant Fulfill the mutant necessity condition Make the mutant evaluation Branch true

Negate the produced conditions after the mutant Start from the mutant program Explores the mutant program path space

16

Target Mutant

DSE-Example Scenario

17

Target Mutant

DSE-Example Scenario

18

Target Mutant

DSE-Example Scenario

19

Target Mutant

DSE-Example Scenario

20

Target Mutant

Mutant is alive

Infect Mutant

DSE-Example Scenario

21

Target Mutant

Mutant is alive

Propagate

DSE-Example Scenario

22

Target Mutant

Mutant is alive

DSE-Example Scenario

23

Target Mutant

Mutant Killed

DSE-Example Scenario

24

DSE-Necessity condition

Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition

a + b > k ≠ a + abs(b) > k //Statement level condition

Test: a=15, b=2, k=0Conditions: b≥0 && a+b>k && …Negates: b≥0 → b < 0 Produces: a=15, b=-10, k=0

25

DSE-Necessity condition

Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition

a + b > k ≠ a + abs(b) > k //Statement level conditionTest: a=15, b=-10, k=0

Conditions: b<0 && a+b>k && a+abs(b)>k && …Negates: a+abs(b)>k → b<0 && a+b>k && a+abs(b)≤k →

InfeasibleNegates: a+b>k → b<0 && a+b≤kProduces: a=-15, b=-10, k=0

26

DSE-Necessity condition

Example: a + b > k → a + abs(b) > kRelationalGT(ArithmeticPlus(a, Abs(b)), k)Abs(x) → if (x<0) //mutant necessity condition

a + b > k ≠ a + abs(b) > k //Statement level conditionTest: a=-15, b=-10, k=0

Conditions: b<0 && a+b≤k && a+abs(b)≤k && …

Negates: a+abs(b)>k → b<0 && a+b≤k && a+abs(b)>k Produces: a=5, b=-10, k=0Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)

27

Killing Mutants (Search Based)

Weak-Strong mutation Measure the closeness of reaching a

mutant Measure branch distance of mutant

brancheso Closeness of weakly killing the targeted mutant (mutant

necessity condition) Use simplified necessity fitness for

improved performance Sufficiency condition can be

approximated by exploring the path space or based on mutants impact

28

Fitness function Approach level

Closeness of executing a targeted mutant Calculated by the number of control

dependent nodes missed Branch Distance

Closeness of flipping a specific branch

Mutation Distance Closeness of weakly killing the targeted

mutant𝑓𝑖𝑡𝑛𝑒𝑠𝑠= 2 ∗ 𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ 𝑙𝑒𝑣𝑒𝑙+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑ሺ𝐵𝑟𝑎𝑛𝑐ℎ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒ሻ + 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑ሺ𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒ሻ

29

Mutant Fitness (example)Operator Original expression Mutant Fitness

Relational a > b

a >= b: abs(a-b)a < b: ka <= b: 0

a != b: abs(a-b+k)a == b: abs(a-b)true: abs(a-b)false: abs(a-b+k)

Arithmetic a + ba - b:ka * b:ka / b:k

a % b:ka:kb:k

Absolute aabs(a):abs(a+k) -abs (a):abs(a)

0:abs(a)

Logical

a && b

a||b:min[Tfit(a)+Ffit(b), Ffit(a)+Tfit(b)]

a:Tfit(a)+Ffit(b)

b:Ffit(a)+Tfit(b)true:min [Ffit(a), Ffit(b)]false:Tfit(a)+Tfit(b)

a || b

a&&b:min[Tfit(a)+Ffit(b), Ffit(a)+Tfit(b)]

a:Ffit(a)+Tfit(b)

b:Tfit(a)+Ffit(b)true:Ffit(a)+Ffit(b)false:min[Tfit(a), Tfit(b)]

30

Path explosion problem

Original program 8 program paths

Mutant program 8 program paths per

mutant Schematic program

Mutant evaluation decisions

8 * 2number of mutants paths Introduction of fake

paths Same of them may be

useful for higher order mutation

Mutants

Mutants

Mutants

Mutants

31

Handling of path explosion Path selection strategy, (symbolic

execution) Use weights on the Enhanced Control Flow

Graph Shortest paths include only the original paths

Dynamic Symbolic Execution Use the mutant parameters as local values Dynamic production of constraints eliminates

the fake paths. Paths are produced dynamically Search based approaches

Approach level contains only the control depended nodeso Ignores the path space

32

Case Studies

Dynamic Symbolic Execution (Strong mutation) Program units: Tritype, Remainder Siemens Programs: Schedule, Tcas, Replace ABS, AOR and ROR operators

Search based (Weak mutation results) Hill climbing approach (AVM) Maximum 10 attempts per mutant Program units: Tritype, Triangle, Remainder,

Calendar ABS, AOR, ROR and LCR operators

DSE Study-Results

DSE Study-Results

DSE Study-Results

DSE Study-Results

DSE Study-Results

Program½ Iterations All Iterations

Produced TestsKilled

Mutants Solver Calls

No. Mut. Executions

Killed Mutants

Solver CallsNo. Mut.

Executions

Tritype 214 1012 500 216 1588 514 90

Remainder 234 2749 435 235 5498 741 268

Replace 514 14740 19400 520 30238 80471 8927

Tcas 136 1733 3729 137 3705 4720 422

Schedule 94 714 293 94 949 303 301

DSE Study-Results Strong mutation results

Replace 86.5%, Tcas 100%, Schedule 91% (compared with the accompanied test suite),

Remainder 97.5 and Tritype 96.8% (all killable mutants)

39

Search Based Study-Results

40

Search Based Study-Results

41

Search Based Study-Results

42

Search Based Study-Results

43

Search Based Study-Results

Test Objects

No. of Killed Mutants Mutation ScoreTime (Sec)

Random Metallaxis Random Metallaxis

Tritype 168 245 66.4% 96.8% 42

Triangle 246 275 89.5% 100% 70

Remainder 238 242 98.3% 100% 402

Cal 237 252 93.3% 99.2% 44

Weak mutation results

Conclusion

Mutation based test case generation Lack of attempts First steps using dynamic, state of the art

techniques

Dynamic approaches can be adopted to perform mutation Search based approaches

o Mutation distance leads to improved effectiveness Dynamic Symbolic Execution

o Quite effective 44

Conclusion

Test data generation approaches and tools can be effectively extended to mutation Practical Technique (schemata)

Future directions Efficient handling of infeasible paths Handling of the “flag” problem Efficient handling of equivalent

mutants Hybrid approaches

45

Thank you for your attention…

Questions ?

Contact

Mike Papadakis mpapad@aueb.gr

Nicos Malevris ngm@aueb.gr

46

47

References

Mike Papadakis and Nicos Malevris. "Automatic Mutation Test Case Generation Via Dynamic Symbolic Execution", in 21st International Symposium on Software Reliability Engineering (ISSRE'10), San Jose, California, USA, November 2010.

Mike Papadakis and Nicos Malevris. “Metallaxis an Automated Framework for Weak Mutation", Technical Report, http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf.

Mike Papadakis, Nicos Malevris and Maria Kallia. "Towards Automating the Generation of Mutation Tests", in Proceedings of the 5th International Workshop on Automation of Software Test (AST'10), Cape Town, South Africa, May 2010, pp. 111-118.

Mike Papadakis and Nicos Malevris. “An Effective Path Selection Strategy for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC'09), Penang, Malaysia, December 2009, pp. 422-429.