Test Case Equivalence Part 3


In the previous post, we explored the model of a test case and showed how we could start seeing when two tests were actually duplicates. But a model is not the entire story.

In this post, we will explore a new concept that will guide us closer into the realm of understanding test case duplication. This is done via the mathematical concept of a core.

You’ll recall that the model of a test case is the finite sequence of abstract states induced by that test case.

 

 An example test case model.

 M(t) = < q1, q2, q3, …, q|s|, q|s|+1 >

 

Now, those abstract states represent the different states that an abstract SUT goes through as the test executes. That is, as each step runs, it changes the state of the abstract system. Those changes of state, along with the order of their change, are captured by the test’s model.

Remember: every test case represents a simplification of the SUT, a virtual SUT, if you will. The model of a test case is, in fact, this simplification.

Are we ready to perform comparisons on test cases? Well, not yet.

Relying on the model to help us determine whether a test is a duplicate of another is certainly better than speculating endlessly, as many contemporary testers and consultants end up doing.

However, this endless speculation is simply due to the lack of good definitions in our industry. For instance, the following post is an example of such such speculation.

Note that my intent is not to bash the author of that post, Martin Jansson. I think he did a great job at summarizing current industry insights. But, those insights merely end up in the all-too-common “it depends” conclusion, which is no conclusion at all!

That is a sad state of affairs for our industry.

It was exactly that — the fact that we still get nowhere even with the long list of accumulated insights over at least two decades — the reason I’m now putting a part of my research in the open in the form of this series of posts.

It’s not because I want you to think that I’m a good person (although I certainly strive to be, but that’s a personal goal). Rather, I think all of us in the testing industry deserve better. Period.

By the way, here’s a shameless plug: this research will be included in my upcoming book. I know I’ve said it before, but just making sure you’re as excited as I am about this… But back to my point.

So my message to you, dear fellow tester, is: Yes! We can do better!

How much better? Well, let me tell you that there is a way to precisely and universally define what a test duplicate is (I’ve been gently guiding you to it via this series of posts). However, without that definition, we step into the pitfall of the long, opinionated discussions that have the same useless ending of: “so… it really depends.

Wouldn’t it be great if test management tools helped to determine when we have duplicate tests in our suites? At least, it would be great if they alleviated the problem somehow.

Alas, the reality is that existing test management tools provide no test management whatsoever. They are simply dumb bundles containing a database and charts that don’t solve our real test management problems. Rather, they solve the reporting problem. That’s all they do.

Think about it. They simply grab your work and show it on pretty charts. Oh, and your company is most probably having to pay a monthly fee for that “privilege.”

Let’s change that.

My ideal test management tool has exactly the kind of fantastically useful functionality that tells me: “Watch out, the test you’re about to save is a duplicate of this other one” and brings up that other test for me to look at.

It can also tell me when a test covers another test, so that even if they’re not duplicates, I can still decide which test to keep (the covered test or the covering one).

You excited yet? Good, because I have ranted enough to get me excited too. Let’s continue our post.

 

Settize my model, please

Now that we have the model at hand, let’s extract all those abstract states and keep only one instance of each distinct abstract state. That is, let’s discount all abstract states that are the same with the exception of their single, representative, abstract state. We do so by settizing the model (turning the sequence into a set). The notation M(t){} denotes this operation.

 

 Definition: the settized model.

 Let M(t){} be the set that results by collecting all elements in the model M(t).

 Formally,

M(t){} = {e : e  M(t) }

 

A settized model, for instance might look like this:

M(t){} = { q }

if and only if all abstract states in the model turned out to be the same. In practice, this can happen if the test is empty (i.e. has no steps), in which case q = Ø.

 

The essence of a test case

As you’ll recall from the previous post, abstract states contain only the relevant details of the result of applying each step to the virtual SUT. That is, they are excellent at casting away all the unnecessary information about the actual SUT that is of no relevance to a test.

So far, we’ve done some pretty good simplification. However, we need a further level of simplification. The reason for this is because we need to be able to compare any two abstract states in such a way that allows us to say, “these two abstract states are the same” with 100% certainty.

Additionally, we need to do that not only within a model, but also, and most importantly, across models. For that, we need to get to the essence of a test case. We do this via an object called the core.

 

Definition: the core of a test case.

 Let t ∈ TESTS be a test case. Let M(t){}/~ denote the quotient set of M(t){} and define it as the set of all possible equivalence classes of M(t){} induced by the equivalence relation ~. Then we define the core of test case t, written core(t), as

core(t) = M(t){}/~ = { [e]~ | e ∈ M(t){} }

 

Remember that our goal here is to be able to compare any two tests and determine if they are equivalent (i.e. duplicates). The core gives us what we need in terms of comparison of abstract states.

Yet, there is one last tiny wrinkle that needs to be fleshed out: the order of those abstract states.

 

The intent of a test case

Suppose you have the following tests, shown side by side in the figure below. Don’t worry about the missing steps; they are irrelevant.

 

Tests with abstract states in different order

Tests with abstract states in different order

 

My question to you is: are the tests duplicates?

Now, before you open up your lecture notes from Cem Kaner, fire the question to Michael Bolton via twitter, or even attempt to use the wrong tools like “test heuristics” or the “checking vs testing” nonsense, let me tell you that you should be able to use plain logic here to arrive at the answer.

No matter what happens in the steps of those tests, their pre-conditions and post-conditions indicate that the tests really are different (remember that pre-conditions and post-conditions are abstract states too.) Namely, one test goes from “FS full” to “FS empty,” while the other test goes the opposite way. Their purpose — their intent — is clearly different.

By the simple fact that one test must begin with the SUT in a state that has a full filesystem (as evidenced by its pre-condition), while the other requires that the SUT have its filesystem empty at the beginning, means that…? That’s right: it means that the tests are doing something different and cannot be considered duplicates.

This is why the order of abstract states is important. The core gives us the essence of a test case, while the order of abstract states gives us its purpose, or intent.

Therefore, another way to explain why the test cases in the above example are not duplicates is because, even though they might have the same essence (i.e. their cores might be equal), their intent is clearly different.

So how do we deal with abstract state order?

That’s a topic for the next post. Stay tuned!

 

 

Part 2

Part 1

 

 Permalink.

 

Leave a comment

Your email address will not be published. Required fields are marked *