mutation testing tools - fbk · mutation testing: improving mutant score!analyze live...
TRANSCRIPT
![Page 1: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/1.jpg)
Mutation Testing ToolsSoftware Analysis and TestingCu Nguyen Duy (cunduy at fbk dot eu)Alessandro Marchetto (marchetto at fbk dot eu)Paolo Tonella (tonella at fbk dot eu)Mariano Ceccato (ceccato at fbk dot eu)
Academic Year 2011-2012
Friday, November 25, 11
![Page 2: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/2.jpg)
Introduction
• Mutation testing is expensive
• Compilation, execution, comparison cost
• Not many supported tools
• MuClipse/mujava, Jumblen
• Improving mutant score is time consuming
Friday, November 25, 11
![Page 3: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/3.jpg)
Mutation testing with MuClipse
Friday, November 25, 11
![Page 4: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/4.jpg)
MuClipse!An Eclipse plugin for MuJava (http://
cs.gmu.edu/~offutt/mujava/)!Highlight
! Traditional operators! OO operators! Java
!Functionality!Generate mutants!Run tests!Report mutant score
Friday, November 25, 11
![Page 5: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/5.jpg)
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 6: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/6.jpg)
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 7: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/7.jpg)
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 8: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/8.jpg)
MuClipse: generate mutants!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 9: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/9.jpg)
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 10: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/10.jpg)
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 11: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/11.jpg)
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 12: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/12.jpg)
MuClipse: run test!Eclipse: Run > Run Configurations
Friday, November 25, 11
![Page 13: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/13.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
Friday, November 25, 11
![Page 14: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/14.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
***********| Executing Mutants |*********
----------------------------------------------AOIS_11 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<6> ------------0 live, 1 killed.
Friday, November 25, 11
![Page 15: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/15.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_22 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<4> ------------0 live, 2 killed.
Friday, November 25, 11
![Page 16: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/16.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_33 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------1 live, 2 killed.
Friday, November 25, 11
![Page 17: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/17.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_44 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------2 live, 2 killed.
Friday, November 25, 11
![Page 18: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/18.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
----------------------------------------------AOIS_55 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<6> ------------2 live, 3 killed.
Friday, November 25, 11
![Page 19: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/19.jpg)
MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true
-------------------------------Results for class math.Add-------------------------------Live mutants: 12Killed mutants: 31Mutation Score: 72.0Writing to file...
Friday, November 25, 11
![Page 20: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/20.jpg)
Exercise 1
!Install MuClipse!http://selab.fbk.eu/swat/projects/muclipse.tgz
!Add java library to Exercise-3-MathOperators!http://selab.fbk.eu/swat/projects/mu-lib.tgz
!Generate mutants! Identify 10 changes in the generated mutants!Hints:
! Use eclipse comparison feature
Friday, November 25, 11
![Page 21: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/21.jpg)
Exercise 2
!Continue exercise 1!Report the mutant score of the test case:
TestCase1_add_Junit3_a of the project! use only traditional mutation operators
!Improve the mutant score of the test case
Friday, November 25, 11
![Page 22: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/22.jpg)
Exercise 3
!Compare the coverage of the test case: TestCase1_add_Junit3_a before and after being improved using mutant score
!Hints:!Enable Clover!Execute only the test case and report the
coverage
Friday, November 25, 11
![Page 23: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/23.jpg)
Mutation testing with Jumble
Friday, November 25, 11
![Page 24: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/24.jpg)
Installation
๏ Download Jumble from http://jumble.sourceforge.net
๏ Unzip the downloaded file to a folder, namely jumble_1_1_0
๏ Inside the folder eclipseplugin > plugin-export, Copy or Move the folder jumble to your Eclipse plugins folder: Eclipse > Plugins, or Eclipse > dropins > eclipse > plugins
๏ Restart Eclipse
2
Tuesday, November 9, 2010
![Page 25: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/25.jpg)
Run default mutation testing
๏ Condition: Class-To-Be-Mutated and test class have the same package, and test class name = <Class-To-Be-Mutated>Test.java
๏ Steps: 1. Right click on <YourClass>2. Choose Jumble > Jumble Class 3. Look at the Console view to see the mutant
score
3
Tuesday, November 9, 2010
![Page 26: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/26.jpg)
Run custom mutation testing
๏ Applied when <test class > # <class>Test.java, or in different packages
๏ Follow the step 1 and 2 of the default run (see previous section)
๏ Customize the execution arguments of Jumble to specify your test class. In Eclipse, go to Run > Run Configurations
4
Tuesday, November 9, 2010
![Page 27: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/27.jpg)
Run custom mutation testing Select Run Jumble Configuration In the Arguments tab, put your <TestClass> after
<YourClass> Run
5
Tuesday, November 9, 2010
![Page 28: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/28.jpg)
Mutation testing report Mutation points = number of mutants “M FAIL”: mutant was not killed by the current test
class, example: “M FAIL: math.Add:5: += -> -=” : line 5 of class math.Add,
“+=” is mutated to be “-=” Mutant score = (mutant points - “m-failed”) / mutant
points
6
Tuesday, November 9, 2010
![Page 29: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/29.jpg)
Exercise 1
Apply Mutation testing with Exercise-3-MathOperators
Report the mutant score of the test case: TestCase1_add_Junit3_a of the project
Improve the mutant score of the test case
7
Tuesday, November 9, 2010
![Page 30: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/30.jpg)
Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not
be killed, for example:
!Write more test cases to try to kill other mutants
public class Add { public static int sum (int a, int b){ return a++ +b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
Friday, November 25, 11
![Page 31: Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test](https://reader034.vdocuments.site/reader034/viewer/2022043022/5f3e70e3b2a851722176fc72/html5/thumbnails/31.jpg)
Mutation analysis!Use traditional mutation operators!Generate mutants of one of your classes
(your choice)!Execute unit test cases that test the chosen
class!Report the mutant score of the test set w.r.t
the chosen class!Improve the mutant score of the test set to
at least 90%
Friday, November 25, 11