How many of these sound familiar to you?
- “this test is already covered by that other one”
- “this test is a superset of that one”
- “these tests seem to cover the same thing”
- “these two tests are very similar”
- “this test covers related aspects/functionality as that other test”
- “these two tests are equivalent”
I’ve heard these plenty of times. But when are two test cases really equivalent or duplicate?
I recently read a very interesting blog post in the test eye where Martin Jansson explains his thoughts about duplicate testing. In the post, Martin shares with us how he thinks test duplication might arise and touches on several aspects of testing such as testing methodologies, approaches, levels, concepts, and even bug recreation.
But there’s one thing missing from all the discussions out there about test duplicates: what exactly is a duplicate?!
Unfortunately, it’s all still opinions and beliefs. There’s nothing concrete that would help us determine, with absolute certainty and objectivity, that two test cases are duplicates.
Many of you already know that I’m currently writing a book about the mathematics of testing. Martin’s post is very timely to my situation because, coincidentally, I’ve been developing the chapter for the concepts of test equivalence.
In this post, I’ll explore some initial thoughts about test duplication and equivalence. Depending on how involved or long the content gets, I’ll divide this into a series of smaller posts.
So let’s get started.
I’m sure throughout your testing career, you have found that some test cases emerge that seem to “do almost the same thing” or “test exactly the same thing.” The degree of what the “same thing” is currently depends on human intuition and the areas or features in a SUT that appear to be exercised by the tests.
As in the opening sentences above comparing one test case to another, our intuition might tell us that, say, executing test a will allow us to forego the execution of test b because a does (or covers) what b does (or covers).
For instance, consider the two test cases presented side by side in the following figure.
For the sake of brevity and illustration, we do not specify all their steps. However, you can assume that all the steps from both tests are exactly the same. Also, it is clear that their post-conditions are the same.
Looking at the two test cases above, then, what is the difference between them? Apparently, there is no difference. One might consider doing away with one of the test cases in order to remove duplication since they both appear to have the same intent and cover the same thing.
But look at the tests in the figure below, which presents the same test cases with a bit more information.
It is now clear that the two tests are not duplicates. This is because their intent is now objectively different: they each focus on distinct operating systems.
Removing either one of the tests would clearly ignore an important combination and would increase the probability of risk that would otherwise be avoided by allowing the existence of the two tests.
Let’s play with another scenario.
Suppose two test cases have exactly the same intent (goal or purpose). In particular, suppose their intent is to verify that a certain report can be generated from within the solution. Would you consider those tests the same (i.e. duplicates)?
Suppose that you examine them further and find that one test generates the report from the menu bar (for instance, by navigating to View -> Reports -> Generate) while the other test leverages the context and opens the report by right-clicking (or double-tapping) on the window panel of the app.
Now those tests don’t seem to be duplicates anymore, do they? Even though they have the same intent, they go about their business very differently. As with the example above, removing one test increases the probability that we expose the user/customer to defects that would have been caught by keeping the (apparently duplicate) test case.
The two examples provide hypothetical situations where test cases at first seem equivalent, but upon further examination, differ in their pre-conditions (as in the first example above), or in their steps (as in the second example).
To understand test case equivalence and whether one test is duplicate of another one, we need to first understand the change of state induced by the tests on the SUT.
I’ll leave it here for the time being. For now, I’d like to encourage you to think of examples where you thought you had duplicate tests but turned out to be valid.
Until next time!