Do you think your tests are as DRY and effective as they can be?
Ortask has completed research that shows how redundant tests affect the SUT. Without further ado, here is a summary of the results (Warning: you might find them surprising!)
The average test suite has 25% duplicate tests
That’s right. On average, 1/4 of a suite’s tests are useless.
This might be indirectly related to the pesticide paradox, which is an informal argument that draws an analogy between the hardening of biological organisms to pesticides and software bugs becoming immune to the same tests (and testing approach) over time.
Interestingly, perhaps the fact that an average of 25% of tests are duplicates actually supports that analogy; or perhaps redundant tests even exacerbate the effect…
The higher the amount of duplicate tests, the higher the amount of bugs in the SUT
Related to the first finding, the research found a strong correlation between test redundancy and bugs in software.
Note that correlation does not imply that test duplication causes bugs, just like high code coverage does not cause their absence. However, test redundancy is symptomatic of poor test quality, which leads to bugs in software.
A Test Redundancy metric is useful for measuring test quality
In the research, a linear combination was found for the projects in the study to estimate the amount of reported bugs. That equation, which is quite close in its approximation of reported bugs, uses the following test quality criteria:
- Code coverage
- Mutation score
- Test redundancy ratio
By itself, code coverage is not a good indicator of overall test quality, and it is often milseading. For instance, see this informal explanation.
Mutation score is a way to measure the regression detection capability of tests. See the ongoing series of posts here for how to do mutation testing. However, by itself, it also exposes only part of the reality.
The results show that test redundancy is one of the missing pieces of the puzzle – a puzzle that still has many, many pieces left to uncover and unify.
Want to find the amount of redundancy in your own test suites?
The Ortask Comparison Engine, the program utilized in the research to find redundant tests, is available here for you to try it out.
You can access the complete results of the research here or view the short video now!