Test Your tests !

How do we test our tests ? How would I know my low level tests are well updated given in a fast pace agile environment ? What is the problem with code coverage ? How would I know the tests which I inherit are concrete ? One solution to deal with all these is Mutation testing.

image source https://www.google.com/

Mutation testing ?

Mutation testing is a process of analysing whether test kills the mutants which are injected to the piece of code. If the test fail when the mutants are injected we say that the mutants are killed ,else it’s still alive . The more mutants the test kills the better it is. So in short, mutation testing allows us to check the accuracy and efficiency of written tests against future changes or possible errors in our code. Apparently if your tests find mutants it probably will find out real bugs. Mutation testing results can be used to measure the effectiveness of test cases.

What are mutants ?

image source https://www.google.com/

Mutant operator’s are small changes done in the program (for ex: Changing > to >= ) to make the code slightly different from the original one ,Each transformation results in a new program, called mutant. The process of creating a mutant from the original program is called mutagenesis.

// original Code

if (i < 10) {
        return "fizz";
        }

// mutant version
        if (i > 10) {
        return "fizz";
        }

Change from < to > is mutant operator. Mutant operators can be anything as changing > to = or > to != etc or even the method names which are being called in the code.

Code coverage sucks ?

Code Coverage can be defined as the measure of percentage of code lines executed. Unfortunately lines of code executed is not tested code.Getting a bug in production is the most expensive thing in software development process and for this reason the lowest layer of test cases should be concrete.Code reviews at regular intervals is good but it wont solve the problem well, as couple of disadvantages in code reviews are : Feedback comes really slow , inconsistent and Labour work is high. Mutation testing thus allows to test the lower layer tests effectively. Survived mutants opens us the area of code which require high attention and has to be looked up on ASAP.

Why is mutation testing important ?

• Gives the team high confidence on the test cases and it effectively test your tests
• Improve code coverage
• Amplifies the use of TDD mechanisms to a greater extend
• It can highlight redundant code

Mutation Testing for Java

To get started with mutation Testing for java , PIT is highly recommended. You can read more about it here.

References

Couple of good papers I have come across related to mutation testing :

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46584.pdf

An intro to Mutation Testing - or why coverage sucks

What do you think about mutation testing ? Write me :)