A better way to hire developers and testers

The typical (inconsistent) hiring process

During my professional career, I have had the chance to interview with many companies big and small, distributed all over the map. Some have been international and well-known, while others have been small or even startups.

During all those interviews, the company culture has been palpable and has come across clearly.

World-wide or well-known companies tend to emit the culture of red-tape and are mostly process-driven. This is evident in the formal and dry nature of the conversations that tend to occur during interviews there.

Smaller companies, on the other hand, tend to be more flexible in both their attitude and their expectations from interviews. For example, the interviewers tend to be dressed down, in jeans or even shorts, and their mannerisms are more relaxed than in the regiment of bigger companies.

Of course, this is simply the average, and your mileage may vary.

But despite all these differences, there is one common aspect that permeates all these interview experiences: the interview process is generally the same.

Here is what I mean:

1. There is at least one (but typically several) interviewer(s).
2. There are typically several rounds of interview questions.
3. Some interviewers asks their own set of informal/non-technical questions such as:
   1. “What is your biggest accomplishment?“
   2. “Why should we hire you?“
4. Some interviewers might ask for coding exercises, either using pseudo-code, forcing the interviewee to recall aspects of the programming language specification in use by the company, or somewhere in between.
5. The interviewers get together to assess the responses.
6. The interviewers choose the candidate that they liked the most based on the candidate’s responses and behavior.

These steps are repeated for each candidate without much change.

This process is followed to the letter by bigger companies like IBM, Google, Microsoft. Smaller companies like Cerner, ESRI, Aonix and many others are more flexible and less formal than their bigger counterparts, but the process is essentially the same.

Some change is happening

Despite the apparent gauntlet of questions and exercises that is supposed to filter out “bad” candidates and leave the good ones, some people have noticed that such interview processes are very, very, very subjective and inconsistent.

What I mean is that in the end, such an interview process does not really help filter anything, and might actually repel good candidates that were turned off by the process, or reject worthy candidates simply because they are not able to answer some obscure, irrelevant puzzle posed by the interviewer.

For instance, Google conducted research on its own interview process and found that having interview candidates jump through the “solve me this puzzle here” process did not improve the ability or intellect of their workforce anymore than asking the traditional questions that are typical of interviews everywhere, such as:

1. “Where do you want to be in 5 years?“
2. “If you saw a bus filled with golf balls, what is the first question you would ask?“

For this and many other reasons, forward-looking companies have started doing pair programming and/or pair testing with interview candidates during the interview process.

Some companies have reported great success, while the majority has reported mixed results.

The good side of this approach is that it allows people to more accurately see how a candidate would behave and react to problems encountered in the day-to-day setting of the company and, of course, the job.

The down-side (and the reason several companies have reported mixed results) is that some candidates are given projects that have taken weeks or even months to brew in-house. This causes the candidates to utilize all the available time of the interview simply to catch up to the level of knowledge of the interviewer (who has had weeks or even months already to process the project and the tasks involved). 

In this regard, some candidates are set to fail even before they start, perhaps without the interviewers fully realizing.

Having noticed this problem with pair programming and testing during interviews, some companies have started to provide small, discrete, re-usable tasks that candidates can work on with the little available time they have.

For example, a discrete task would be either developing a sorting algorithm (for a development interview) or testing such an algorithm (for a testing interview).

Companies using this amended approach report much better results in applying the pair programming/testing technique during interviews.

And where is the consistency?

I have always wanted to create and lead my own company ever since I was young, so I knew that one day I would be hiring top-notch people, which implies having a top-notch interview process. For that, I have been collecting plenty of notes from all the interviews I have taken part of, as well as from the new trends on pair programming/testing currently taking place.

All this note-taking has been in the effort to find that best, more consistent, more objective process that can more reliably find people that are stars.

Applying pair programming/testing in an interview is certainly an improvement over previous methods. I also still think that some traditional interview questions hold tremendous value. But there is one important aspect missing from all these filtered processes and techniques: objectivity.

It is true that pairing with the potential developer or tester allows one to see what they do and what their thought process is with certain tasks, but even with pairing, it is still up to the person that paired with the candidate to somehow produce a rating. That rating, in turn, is solely based on the perception of the interviewer (i.e. the feelings and thoughts they had about the candidate while pairing).

The effect of such a (subjective) rating ends up being the same as before: if that interviewer did not get along with the candidate, but the candidate is really a star, then the rating will still be skewed and the candidate will get rejected.

The inverse scenario is also highly likely with this technique: the candidate might not be worthy but might get hired simply because they got along great with the interviewer.

Both scenarios are just examples, and many more possibilities exist. But all of these possibilities lead to the problem of inconsistency in the hiring process, which is the very problem we want to solve. 

In other words, the results of such new techniques continues to be inconsistent precisely because they continue to be subjective.

A better hiring approach

I like the pairing technique because it affords an in-view into the behavior and thought process of the candidate that other techniques and traditional questions simply do not provide.

But pairing only goes so far into assessing a candidate. What such interview techniques need is a normalization layer — a way to bring everything into perspective with a common, objective scale — that strengthen the subjective ratings into something that is usable across candidates, interviewers and interviews.

In particular, such a scale could be used to compare candidates more reliably than before, thus providing the desired level of consistency we want (or at least a bit more consistency in a scientific way).

So here is how I am thinking of applying the normalization layer to candidates, using the pairing technique as the launchpad process for development or testing interviews.

Suppose I get a candidate that is interested in, say, a development job. I follow best pairing-for-interview practice and do not immerse the candidate in a current or on-going project. Instead, I offer an independent, re-usable, discrete, manageable task for pairing (as we learned above).

I then pair program with them to see their abilities, their thought process and their behavior throughout the task. This part is unchanged.

What changes is how I use the artifacts produced by the candidates. In particular, instead of throwing away the candidate’s unit tests and code, I keep them for analysis.

In my opinion, it is highly desirable for candidates to already know and follow best development practices (particularly TDD). So I not only pay attention to how they design, develop and unit-test during the interview, but I rate those unit tests to determine how useful they really are and what quality they have.

And this is where the normalization layer comes in, because rating their unit-tests gives me an independent and objective perspective I need in order to really see how candidates approach the following important aspects:

1. The interview task.
2. The quality of their code and tests for the interview.
3. The quality of their work in general.

This is very important because the whole point of interviews is not only to get a sense of how candidates would tackle a development task if they got hired, but also to know with what quality.

The normalization layer would give a much better sense of the latter aspect, thus answering the question: “with what quality can I expect the candidate to tackle tasks if I hire them?”

So how would I rate the candidate’s unit-tests? With two very important tools: Mutator and TestLess.

By showing me the regression-detection quality of the candidates’ unit tests, Mutator would allow me to infer the overall quality that each candidate imbues in their work. Remember that 55% regression detection is the industry average, so candidates that achieve higher mutation scores would be placed into the “good candidates” bucket and considered further, while the rest would be objectively rejected.

That’s right: objective is the key word here.

Next, of the remaining candidates, I would run TestLess on their unit tests to see how smelly their tests are. I would do this because highly redundant (i.e. smelly) tests lead to technical debt and higher likelihood of bugs.

Since the industry average for test redundancy is 25%, good candidates would, of course, produce tests that are less smelly than the average. This translates to a lower likelihood of bugs existing in the candidate’s code and higher quality work from the get-go.

This is great, because now the ideal candidate can therefore be more closely represented by adding these two independent, objective and consistent metrics to the interview process.

Having these two scores, therefore, would allow me to

1. consistently and reliably compare candidates on the basis of their work with full objectivity, and
2. hire the closest-to-ideal candidate by enhancing any subjective ratings (say, how social the candidate is; or how well they fit into the company culture) with a purely objective perspective afforded by the scores.

The scores would also allow me to identify areas where the candidate needs mentoring and/or training after being hired.

Now, what if the candidate is interested in a testing job? Same thing. You all know that I am a very strong believer that good testers must be developers first. Testers that have no idea how things actually work will end up making bad/false assumptions about the testing tasks and slow down the team.

Moreover, candidates that seek testing positions provide all the more reason to rate their tests since that is what they say they are good at (i.e. testing), don’t you think?

So this is the process I plan to start applying when I interview candidates and I am very excited about it. Of course, this is just the beginning, so some course-correction might be required.

Any thoughts are welcome.