Can Mutants be House-trained?


I was pointed to a lightning presentation on mutation testing the other day. I like the idea of mutation testing, which basically is to mutate code randomly – though the operations are well-defined to keep things well-formed –  creating lots and lots of mutants. A mutant is a variant of your code, with one or more changes applied. Mutation testing tells you how many of these variants are actually caught by your tests. If a test fails on a variant, that mutant has been detected is killed. Obviously you want your tests to kill as many of them as possible, for the highest possible death rate. I piped up on the subject in the forum at smidig, and this is sort of a rehash in English.

In the presentation, a piece of code and its tests are gradually hardened to withstand the assault of the mutants. More mutants are killed, and we prune the mutation space to achieve smaller numbers of mutants. The final move, so to speak, is to declare the method arguments to be, yes, final. This prevents a lot of mutation, since mutants that try to change the parameter simply won’t compile. The mutation space is smaller. This is important, since the number of mutants grows fast, and quickly overwhelms even a small code base.

Now, assigning to parameters is a bad practice in some quarters. (In my own quarter, it’s a howler.) And forbidding the practice of assigning to parameters does seem to help with the mutation rate, but it’s not really tempting to go over the code and harden it piecemeal, littering it with explicit finals all over the place. Also, those modifiers are a double-edged sword. What purpose do they really serve? First, they’re not really needed (unless you do the inner class thing), so one could argue that the only purpose they serve is to prevent mutation. I hate it when some external factor, such as a tool, affects my code so it looks weird. Second, a human programmer could just as easily drop the modifier anyway, if it got in her way. Some programmers remove redundant modifiers on general principle, I do it on morning grumpiness as well.

So, it doesn’t really help. Not really. Worse, it’s quite intrusive code-wise. After all, the mutation testing is just one of many things that the developer needs to know about when writing code, and it gets tiresome after a while to remember that, oh, this weird little thing must be done to keep that automated thing happy, and this one because the so-and-so framework will use reflection to access only, say, protected fields … look out for comments scattered all over saying // make muclipse happy and so on. Stuff like this adds to the time needed to get new people into the project, and generally to the load on programmers’ brains. That’s a scarce resource, so keeping the code as idiomatic and non-quirky as possible is a good thing. Really, try it some time.

Could we lower the mutation rate on a more sweeping scale, and without affecting the code so much? Sure, don’t we have something that we often use to forbid a practice? We do, and it’s called a code standard. A code standard could say “don’t assign to parameters, fool”, and some other tool could enforce that part.

Your IDE is great for it – IDEA points out my code standard misdemeanors with yellow lights.

If the code standard forbids the practice, it seems natural that the mutation tester should know about it. Why should it bother with exploring variants that would be caught by checking against the code standard? In this particular case, it would imply that the mutation tester should behave as if all parameters were final. Presumably, that should lower the number of mutants all over the place, not just in the places we have littered with (all those increasingly mysterious) finals.

This is a lot of philosophizing for a fellow who hasn’t actually tried mutation testing. The idea of marrying mutation testers with code standards could be a silly one, and if so I would love to hear why. If anyone know any reason why these two, and so on and so forth; then comment and talk geeky to me.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: