mutants killer (revised) - pit: state of the art of mutation testing system
TRANSCRIPT
![Page 1: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/1.jpg)
PIT: state of the art of mutation testing systemRevised edition
MUTANTS KILLER
by Tàrin Gamberìni - CC BY-NC-SA 4.0
![Page 2: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/2.jpg)
Lavora da sempre con tecnologie Java-based free e Open Source presso la Regione Emilia-Romagna
Si interessa di Object Oriented Design, Design Patterns, Build Automation, Testing e Continuous Integration
TÀRIN GAMBERÌNI
Partecipa attivamenteal Java User GroupPadova
![Page 3: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/3.jpg)
Unit Testing
Test-Driven Development (TDD)
Continuous Integration
Code Coverage
Mutation Testing
DEVELOPMENT
![Page 4: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/4.jpg)
UNIT TESTING & TDD
The proof of working code
Refactor with confidence (regression)
Improve code design
Living documentation
TDD extremely valuable in some context
![Page 5: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/5.jpg)
CONTINUOUS INTEGRATION
Maintain a single source repository
Automate the build
Cheap statics for code quality
Automate unit & integration tests
Catch problems fast
![Page 6: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/6.jpg)
LINE COVERAGE
Is a measure of tested source code
line, statement, branch, … , coverage
![Page 7: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/7.jpg)
LINE COVERAGE 100%
Only able to identify not tested code
Doesn't check tests are able to detect faults
Measures only which code is executed by tests
![Page 8: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/8.jpg)
WEAK TESTS
public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL";}
@Testpublic void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false ));}
@Testpublic void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true ));}
The untested side effect
![Page 9: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/9.jpg)
WEAK TESTS
public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL";}
@Testpublic void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false ));}
@Testpublic void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true ));}
The untested side effect
![Page 10: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/10.jpg)
WEAK TESTS
public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; }}
@Testpublic void shouldReturnFooWhenGiven1() { assertEquals("foo", foo( 1 ));}
@Testpublic void shouldReturnBarWhenGivenMinus1() { assertEquals("bar", foo( -1 ));}
The missing boundary test
![Page 11: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/11.jpg)
WEAK TESTS
public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; }}
@Testpublic void shouldReturnBarWhenGiven1() { assertEquals("bar", foo( 1 ));}
@Testpublic void shouldReturnFooWhenGivenZero() { assertEquals("foo", foo( 0 ));}
@Testpublic void shouldReturnFooWhenGivenMinus1() { assertEquals("foo", foo( -1 ));}
The missing boundary test
![Page 12: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/12.jpg)
WEAK TESTS
public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO";} @Test
public void shouldPerformActionWhenGivenTrue() { foo(mockCollaborator,true); verify(mockCollaborator).performAction(); }
@Testpublic void shouldNotPerformActionWhenGivenFalse() { foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); }
The myopic mockist
![Page 13: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/13.jpg)
WEAK TESTS
public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO";} @Test
public void shouldPerformActionWhenGivenTrue() { String result = foo(mockCollaborator,true); verify(mockCollaborator).performAction(); assertEquals("MOCK", result);}
@Testpublic void shouldNotPerformActionWhenGivenFalse() { String result = foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); assertEquals("FOO", result);}
The myopic mockist
![Page 14: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/14.jpg)
WHO TESTS THE TESTS ?
![Page 15: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/15.jpg)
MUTATION TESTING
● originally proposed by Richard Lipton as a student in 1971
● 1st implementation tool was a PhD work in 1980
● recent availability of higher computing power has led to a resurgence
● expose weaknesses in tests (quality)
![Page 16: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/16.jpg)
MUTATION TESTING
● originally proposed by Richard Lipton as a student in 1971
● 1st implementation tool was a PhD work in 1980
● recent availability of higher computing power has led to a resurgence
● expose weaknesses in tests (quality)
![Page 17: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/17.jpg)
MUTATION
Is a small change in your program
original mutated
![Page 18: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/18.jpg)
MUTANT
A
mutated
version of
your
program
![Page 19: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/19.jpg)
MUTATION TESTING
generate lots of mutants
run all tests against all mutants
check mutants: killed or lived?
![Page 20: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/20.jpg)
MUTANT KILLED
Proof of detected mutated code Test is testing source code properly
If the unit test fails
![Page 21: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/21.jpg)
MUTANT LIVED
Proof of not detected mutated code
If the unit test succeed
Test isn't testing source code
![Page 22: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/22.jpg)
![Page 23: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/23.jpg)
PIT: BASIC CONCEPTS
Applies a configurable set of mutators
Generates reports with test results
Manipulates the byte code generated by compiling your code
![Page 24: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/24.jpg)
PIT: MUTATORSConditionals Boundary
< → <=<= → <> → >=>= → >
![Page 25: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/25.jpg)
PIT: MUTATORSNegate Conditionals
== → !=!= → ==<= → >>= → << → >=> → <=
![Page 26: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/26.jpg)
PIT: MUTATORSMath Mutator
+ → -- → +* → // → *% → *
& → || → &^ → &<< → >>>> → <<>>> → <<
![Page 27: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/27.jpg)
PIT: MUTATORSRemove Conditionals
if (a == b) { // do something}
if (true) { // do something}
→
![Page 28: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/28.jpg)
PIT: MUTATORSReturn Values Mutator Conditionals
public Object foo() { return new Object();}
public Object foo() { new Object(); return null;}
→
![Page 29: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/29.jpg)
PIT: MUTATORSVoid Method Call Mutator
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
→public int foo() { int i = 5;
return i;}
public void doSomething(int i) { // does something}
![Page 30: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/30.jpg)
PIT: MUTATORSMany others
Activated by defaultConditionals Boundary Mutator - Increments Mutator - Invert Negatives Mutator - Math Mutator - Negate Conditionals Mutator - Return Values Mutator - Void Method Calls Mutator
Deactivated by defaultConstructor Calls Mutator - Inline Constant Mutator - Non Void Method Calls Mutator - Remove Conditionals Mutator - Experimental Member Variable Mutator - Experimental Switch Mutator
![Page 31: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/31.jpg)
PIT: MUTATORSApplication field
Traditional and classPIT is suitable for "ordinary" java programs because mutators change "ordinary" java language instructions.
The right mutator for the right application fieldNot the best option if you are interesting testing:● Aspect Java programs, try AjMutator● Concurrent programs, try ExMan● Mutators for: reflection, java annotations, generics, … ?
![Page 32: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/32.jpg)
HOW PIT WORKSRunning the tests
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
Your program
![Page 33: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/33.jpg)
HOW PIT WORKSRunning the tests
PIT runs traditional line coverage analysis
![Page 34: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/34.jpg)
HOW PIT WORKSRunning the tests
Generates mutant by applaying mutators
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
mutant 2
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
mutant 1
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
mutant 3
![Page 35: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/35.jpg)
HOW PIT WORKSBrute force
x = 9
100 tests x 1.2ms = 0.12s
0.12s x 10000 mutants = 20min !!!1000 class x 10 mutants per class = 10000 mutants
![Page 36: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/36.jpg)
HOW PIT WORKSRunning the tests
PIT leverages line coverage to discard tests that do not exercise the mutated line of code
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
public int foo() { int i = 5; doSomething(i); return i;}
public void doSomething(int i) { // does something}
mutant 3
![Page 37: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/37.jpg)
HOW PIT WORKSExtremely fast
x = 1
x = 1
x = 2 4 < 9
![Page 38: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/38.jpg)
HOW PIT WORKSOutcomes
● Killed, Lived● No coverage: the same as Lived except there were no
tests that exercised the mutated line of code● Non viable: bytecode was in some way invalid● Timed Out: infinite loop● Memory error: mutation that increases the amount of
memory used● Run error: a large number of run errors probably
means something went wrong
![Page 39: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/39.jpg)
PIT ECOSYSTEMTools
Command line
Other IDEs as a Java application
( )
![Page 40: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/40.jpg)
EXAMPLEclass Ticket
![Page 41: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/41.jpg)
EXAMPLEtestGetPrice_withAge11
![Page 42: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/42.jpg)
EXAMPLEtestGetPrice_withAge71
![Page 43: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/43.jpg)
EXAMPLEtestGetPrice_withAge18
![Page 44: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/44.jpg)
EXAMPLEmvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 6 (75%)>> Ran 11 tests (1.38 tests per mutation)=================================================================- Mutators=================================================================> org.pitest...ConditionalsBoundaryMutator>> Generated 2 Killed 0 (0%)> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -----------------------------------------------------------------> org.pitest...ReturnValsMutator>> Generated 3 Killed 3 (100%)> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
![Page 45: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/45.jpg)
EXAMPLE
![Page 46: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/46.jpg)
EXAMPLE
![Page 47: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/47.jpg)
EXAMPLEtestGetPrice_withAge12
![Page 48: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/48.jpg)
EXAMPLEHelps discovering better test data
Well known techniques(1) for creating dataInput space partitioning, boundary values, error guessing.Cartesian product of test data leads to a combinatorial explosion.
PIT helps but YOU have to findPIT tells you when a mutant has survived, but you are in charge to find out the right test data which will kill it.
(1) Introduction to Software Testing, 2008 – by P. Ammann, J. Offutt
![Page 49: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/49.jpg)
EXAMPLEmvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 7 (88%)>> Ran 10 tests (1.25 tests per mutation)=================================================================- Mutators=================================================================> org.pitest...ConditionalsBoundaryMutator>> Generated 2 Killed 1 (50%)> KILLED 1 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -----------------------------------------------------------------> org.pitest...ReturnValsMutator>> Generated 3 Killed 3 (100%)> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
![Page 50: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/50.jpg)
EXAMPLE
![Page 51: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/51.jpg)
EXAMPLE
![Page 52: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/52.jpg)
EXAMPLEtestGetPrice_withAge70
![Page 53: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/53.jpg)
EXAMPLE
?!?!?!?!?
![Page 54: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/54.jpg)
EXAMPLEtestGetPrice_withAge70
![Page 55: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/55.jpg)
QUESTIONS ?PIT: state of the art of mutation testing system
byTàrin GamberìniCC BY-NC-SA 4.0
![Page 57: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/57.jpg)
SOURCE CODE
https://github.com/taringamberini/linuxday2015.git
![Page 58: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/58.jpg)
CREDITSMinion with gun - by jpartwork (no license available)https://nanookofthenerd.wordpress.com/2015/01/30/the-magnificent-maddening-marvelous-minion-episode-032/
Minion Hands on - by Ceb1031 (CC BY-SA 3.0)http://parody.wikia.com/wiki/File:Bob_minions_hands.jpg
Canon XTi components after disassembly - by particlem (CC BY 2.0)https://www.flickr.com/photos/particlem/3860278043/
Continuous Integration - by New Media Labs (no license available)http://newmedialabs.co.za/approach
Rally car - by mmphotography.it (CC BY-NC 2.0)https://www.flickr.com/photos/grantuking/9705677549/
Crash Test dummies for sale - by Jack French (CC BY-NC 2.0)https://www.flickr.com/photos/jackfrench/34523572/
Minion (no license available)http://thisandthatforkidsblog.com/2013/09/17/easy-to-make-diy-minion-halloween-pumpkin/
Images
![Page 59: MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system](https://reader031.vdocuments.site/reader031/viewer/2022022411/58edd2a01a28ab10298b463b/html5/thumbnails/59.jpg)
Mutant - by minions fans-d6txvph.jpg (no license available)http://despicableme.wikia.com/wiki/File:Evil_minions_by_minions_fans-d6txvph.jpg
Mutant killed - by Patricia Uribe (no license available)http://weheartit.com/entry/70334753
Mutant lived - by Carlos Hernández (no license available)https://gifcept.com/VUQrTOr.gif
PIT logo - by Ling Yeung (CC BY-NC-SA 3.0)http://pitest.org/about/
Circled Minions - by HARDEEP BHOGAL - (no license available)https://plus.google.com/109649247879792393053/posts/ghH6tiuUZni?pid=6004685782480998482&oid=109649247879792393053
Finger puppets – by Gary Leeming (CC BY-NC-SA 2.0)https://www.flickr.com/photos/grazulis/4754781005/
Questions - by Shinichi Izumi (CC BY-SA)http://despicableme.wikia.com/wiki/File:Purple_Minion.png
CREDITSImages