Interviewing people is hard. At Atlassian I'm involved in the recruiting process after the resumé has been selected. For roles in the engineering team, the first stage is usually an interview to discuss the candidate's experience and ask them some technical questions. The interview is done by two developers, where at least one is a technical lead or senior developer on our products.
Even for my teammates armed with a wealth of software experience and the knowledge of dozens of interviews, the interview process is still time-consuming and difficult. Often neither them nor I have a simple answer when we walk out after speaking with the candidate for over an hour. I think part of the difficulty is what we're looking for is so hard to quantify. The primary concern is simple to express: does the candidate have the skills and ability to perform the job well? Measuring how well the candidate performs against this criteria is the main challenge of the interview. But the other characteristics we're looking for in candidates are even more difficult to measure. Will the candidate be good to work with? Will she communicate well? What does she offer in experience or expertise that we could use in our team?
To cater for a wide variety of candidates, the interview process isn't prescriptive. We have a list of questions we tend to follow, but it's common to change questions based on how the candidate responds. If she gives a detailed and clear response to a question about the Java API, we won't bother asking more simple questions in this area. Instead, we will try to ask more detailed questions to find out the depth of her knowledge, or try to ask more questions in a different area (say, concurrency or debugging) to find out the breadth of her knowledge.
Knowledge is important for an engineering role, but the interview process involves more than that. The way the candidate thinks for a moment, answers the questions, and even asks questions back can tell a lot about her personality. However, if you asked me to describe the way a good candidate does this, I'm certain I could not say. Sometimes there's an element of excitement when describing their latest challenge. But then, some of the best developers I've met are completely deadpan when discussing technical problems. Perhaps it's when they describe a well-known system, they hint that they have much more knowledge than they're deigning to share. Then again, some of the best developers fly by the seat of their pants, discarding detailed knowledge in favour of assumptions about the underlying system (they just happen to always be the right assumptions). In short, the non-technical skills of candidates are so hard to quantify, but for a small company who tries to be selective in its hiring, they often make the difference between hiring and not.
I just finished reading Blink by Malcolm Gladwell, and it's been an interesting and inspiring read. The premise is that people make snap judgements based on intuition within just a few seconds, and these decisions are often the best possible outcome. In trained individuals, analysis of snap judgements appear impressive. Tennis professionals can tell whether another player will double-fault before the ball hits the raquet. Doctors can use four simple signs to diagnose a heart attack in patients with chest pain with 90% accuracy.
However, there are two major limitations of decisions made this way: they're susceptible to cultural biases, and the conscious mind often isn't aware of the unconcious mind affecting the decision. Leaders of Fortune 500 companies are much taller than the average American. Gladwell's explanation is that a cultural stereotype of tall leaders affects panels interviewing for these positions. He cites psychological studies that support this conclusion through a process called implicit association tests. Similar tests show that most North Americans associate crime more strongly with African Americans than with European Americans, even among those who words and actions show them as firm believers in race equality. The ways to overcome such limitations, he explains, is to ensure decision makers are trained and to institute appropriate limitations around the process. This is why the emergency department doctors don't just listen to the patient's description of the problem—they follow a simple process of four quick tests to enable their decision-making. When encountering an armed suspect, police are trained to take cover and give themselves time to think. This process ensures the conscious mind takes control of the decision-making so the bad side of instinctive decisions can be avoided.
To those involved in important activities like health, business and the military, intuition seems the opposite of the right way to make a decision. Typically in these areas, decisions are made by detailed analysis of all possible options. It might even involve a large group of people. Gladwell argues, however, that the evidence shows over-analysis results in worse decisions than the intuition of a trained expert. He cites several examples in different areas where over-analysis led to statistically worse decisions being made.
As with judging tennis balls and chest pain, I think the outcome of any interview is influenced by snap judgements made by the interviewers. I think overall this is a good thing. For experienced developers, especially those with many years working in a software team, their intuition is a useful asset in deciding whether the candidate will be suitable. But the drawback of intuitive decisions made in interviews is the same as in other situations: they can be unconsciously influenced by cultural biases without the knowledge of the interviewer. An interview process is needed therefore to overcome these biases. At Atlassian, we have the standard list of questions, against which all candidates can be judged. We also ensure we take the time to spend time with every interview candidate, so that intuitive decisions have time to be validated.
The opposite approach—enforcing a strict interview process that excludes the interviewer's intuition—seems attractive, but I doubt it could ever work. Although I haven't seen it used for interviews, this is the type of strategy used for career progression in many large companies. Each stage in your career has a number of criteria that must be met, as you work your way from plebeian to manager. This process doesn't tend to reward people for what a company needs to thrive: communication, innovation, and working outside the limits of one's job.
In summary, I'd say that our interview process is a good one, using the strength of our people to evaluate candidates. The questions cover an appropriate spread of ability, and provide a framework for ensuring candidates are evaluated on an even basis. Putting more process or rigid criteria around interviews is something I'd argue against.
The next stage in our recruitment process for engineers is usually a programming test. I'll cover how the Blink theory of snap decision-making affects this in a future post.
