i.t.a.k.e unconference - mutation testing to the rescue of your tests
TRANSCRIPT
![Page 1: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/1.jpg)
An event proudly designed by
Mutation Testing to the rescue of your
tests@nicolas_frankel
![Page 2: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/2.jpg)
Me, Myself and I• By day
• Hybris consultant
• By night
• Teacher/trainer
• Book Author: Vaadin, Integration Testing
• Blogger: https://blog.frankel.ch/
• Speaker
![Page 3: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/3.jpg)
Many kinds of testing• Unit Testing
• Integration Testing• End-to-end Testing
• Performance Testing• Penetration Testing• Exploratory Testing
• etc.
![Page 4: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/4.jpg)
Their only single goal
• Ensure the Quality of the production code
• How to check the Quality of the testing code?
![Page 5: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/5.jpg)
Code coverage
• “Code coverage is a measure used to describe the degree to which the source code of a program is tested”
• --Wikipediahttp://en.wikipedia.org/
wiki/Code_coverage
![Page 6: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/6.jpg)
Measuring Code Coverage
• Check whether a source code line is executed during a test• Or Branch Coverage
![Page 7: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/7.jpg)
Computing Code Coverage
• CC: Code Coverage(in percent)
• Lexecuted: Number of executed lines of code
• Ltotal: Number of total lines of code
![Page 8: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/8.jpg)
Java Tools for Code Coverage
• JaCoCo
• Clover
• Cobertura
• etc.
![Page 9: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/9.jpg)
100% Code Coverage?“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebo
bmartin/status/55966620509667328
![Page 10: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/10.jpg)
Assert-less testing@Testpublic void add_should_add() {new Math().add(1, 1);
}
But, where is the assert?
As long as the Code Coverage is OK…
![Page 11: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/11.jpg)
Code coverage as a measure of test quality
• Any metric can be gamed!
• Code coverage is a metric…
• ⇒ Code coverage can be gamed
• On purpose
• Or by accident
![Page 12: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/12.jpg)
Code coverage as a measure of test quality
• Code Coverage lulls you into a false sense of security…
![Page 13: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/13.jpg)
The problem still stands
• Code coverage cannot ensure test quality
• Is there another way?
![Page 14: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/14.jpg)
The CastOriginal Source Code
Modified Source Codea.k.a “The Mutant”
![Page 15: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/15.jpg)
Standard testing✔Execute Test
![Page 16: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/16.jpg)
Mutation testing
?Execute SAME Test
MUTATION
![Page 17: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/17.jpg)
Mutation testing✗
✔Execute SAME Test
Execute SAME Test
Mutant Killed
Mutant Survived
![Page 18: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/18.jpg)
Test the code
✔Execute Test
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1);}
![Page 19: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/19.jpg)
Surviving mutant
✔Execute SAME Test
@Testpublic void add_should_add() { new Math().add(1, 1);}
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
![Page 20: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/20.jpg)
Test the code
✔Execute Test
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1); Assert.assertEquals(sum, 2);}
![Page 21: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/21.jpg)
Killed mutant
✗Execute SAME Test
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1); Assert.assertEquals(sum, 2);}
![Page 22: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/22.jpg)
Mutation Testing in Java
• PIT is a tool for Mutation testing
• Available as
• Command-line tool
• Ant target
• Maven plugin
![Page 23: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/23.jpg)
Mutators
• Mutators are patterns applied to source code to produce mutations
![Page 24: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/24.jpg)
PIT mutators sampleName Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=Remove Conditionals foo == bar true
Math + -Increments foo++ foo--
Invert Negatives -foo fooInline local variable int foo= 42 int foo= 43
Return Values return true return falseVoid Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis()
long t = 0
Constructor Call Date d = new Date() Date d = null;
![Page 25: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/25.jpg)
Enough talk!
![Page 26: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/26.jpg)
Drawbacks• Slow
• Sluggish
• Crawling
• Sulky
• Lethargic
• etc.
![Page 27: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/27.jpg)
Metrics (kind of)• On joda-money
• mvn clean test-compile
• mvn surefire:test
• Total time: 2.181 s
• mvn pit-test...
• Total time: 48.634 s
![Page 28: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/28.jpg)
Why so slow?• Analyze test code
• For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
![Page 29: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/29.jpg)
Workarounds• Increase number of threads • Set a limited a set of mutators• Limit scope of target classes• Limit number of tests• Limit dependency distance• Don’t bind to the test phase • Use scmMutationCoverage
• Use incremental analysis
![Page 30: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/30.jpg)
Incremental analysis• Metadata stored between runs
• During each following run mutant will not be checked again, if the last time it:
• timed out, and class has not changed
• was killed, and neither class nor test have changed
• survived, and there are no new/changed tests for it
![Page 31: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/31.jpg)
False positives
• Mutation Testing is not 100% bulletproof
• Might return false positives
• Be cautious!
![Page 32: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/32.jpg)
Pit is imperfectif (p < 0)...// changed condition boundary// -> survived:if (p > 0)... return 0;
![Page 33: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/33.jpg)
Pit might be dangerous
void reboot() throws IOException { // removed method call: checkUserPermissions(); Runtime.getRuntime() .exec("reboot");}
![Page 34: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/34.jpg)
Testing is about ROI• Don’t test to achieve 100%
coverage• Test because it saves
money in the long run• Prioritize:
• Business-critical code• Complex code
![Page 35: I.T.A.K.E Unconference - Mutation testing to the rescue of your tests](https://reader036.vdocuments.site/reader036/viewer/2022070601/587e46621a28ab9f5d8b86af/html5/thumbnails/35.jpg)
Q&A• https://git.io/vznQK
• http://blog.frankel.ch/
• @nicolas_frankel
• https://leanpub.com/integrationtest/