A bad software eng interview is costly

The opinions here are my own and not that of any current or past employers.

This episode of HBO’s Silicon Valley aired in Spring 2018. Coincidentally, that same time I started my first software engineering job. Seven years later, having gone through dozens of software engineering interviews and having worked extensively on the engineering hiring at Square/Block, I look back on this clip and cringe.

I’m not cringing at the inaccuracy. Yes, there are probably many more than 63 distributed systems engineers looking for a job in the Bay Area at any given time; and no, it’s unlikely a big company would hire all available talent to spite your promising startup (although maybe that’s somewhat accurate). What I’m cringing at is the truth of this line:

It’s played off as a joke, Gilfoyle and Dinesh trying to pass the buck to Richard. But in this case, they are correct.

Engineering hiring is critical to a software company

Any technical founder of a startup company knows the importance of convincing great talent to join their company. This is difficult because the best candidates can get higher offers at stable companies like Hooli.

Even as a startup gains funding, reputation, and stability, the importance of finding great talent doesn’t decrease. “I can’t think of anything more important to the success of a software company than hiring excellent software engineers.” This rough quote is from one of the leaders of the eng hiring committee at Block. I heard this in 2023, years after the company had gone public.

And yet, as companies grow to series B, C, and onward, the candidate experience and the interview process often becomes an afterthought. The most important work of a company is to build a product people want. But a bad interview process is costly in many ways and can hinder the company’s ultimate goal.

What did it cost?

Let’s crunch the hypothetical numbers of interviewing 63 engineers.

Let’s assume an employee at Pied Piper’s time is worth $150 an hour. Let’s assume that Dinesh, Gilfoyle, and Richard each spent a 30 minutes talking to each candidate and another 30 minutes reviewing their resume and writing interview feedback. Three people * $150/hour * one hour per candidate * 63 candidates comes to $28,000. Richard also mentions screening other candidates. Let’s assume there were 30 more, adding another $4,500 of cost. And there was likely someone reviewing resumes and reaching out to candidates before they spoke to Richard. Let’s assume 5 minutes per candidate and 300 candidates for $9,000.

In total, this is $41,500. But this estimate is probably conservative. Interviews are disruptive to an engineer’s day. A 30 minute interview and 30 minutes of feedback also likely adds another hour lost in context switching between tasks. So the time taken away from productive focused work is likely double what is estimated. So it’s likely the company spent $83,000 to interview 63 qualified candidates and hire only three of them.

That’s just the monetary cost however. If you have watched the show, you’ll know that Dinesh and Gilfoyle are assholes. And Gilfoyle admits his interview process sucks. It’s likely that many of the candidates were put off by this experience and they will tell their friends to avoid interviewing at Pied Piper¹.

In the show, Pied Piper ends up hiring three candidates. But if the interview process is so poorly designed, who knows if these candidates will actually succeed? A phenomena I have seen too many times is companies putting frontend engineers through the same generic data structures and algorithm interview as backend engineers. You hire engineers with strong fundamentals, but you don’t actually know if they are skilled at the work they were hired for².

I remember talking to an engineering manager at a FAANG+ company about a bad hire. This engineer made the choice to create custom keys for a critical system still in use today. The better choice would have been to use a standardized format already in use at the company. This continues to be a headache for his engineering team well after that engineer was fired.

So there is monetary time loss of interviewing and rejecting good candidates, reputational loss of having a bad process that annoys candidates, and the potentially catastrophic damage that a bad hire can do. If this is such a critical problem . . .

Why does investing in a good interview process become an afterthought?

This was a question I thought a lot about in my last year working on engineering hiring at Block. If a candidate received a badly written coding interview question for their technical screen, they may be incorrectly rejected. That meant two hours of a recruiter’s time and two hours of an engineer’s time. But there isn’t a -$600 line item listed on the balance sheet. It just means the engineer is distracted from their regular work and the recruiter has to work longer to fill the role.

These are small costs but they add up across a company. And though they are invisible, their impact is not. Engineering velocity decreases; features are delivered later than the competition; engineers, recruiters, and engineering managers burn out sooner.

Where the buck stops

Let’s reread the quote from Gilfoyle:

Ultimately, it is the responsibility of engineering leadership to invest in a good interview process. Your recruiting team is keenly aware of the costs I discussed above and it is the responsibility of recruiting leadership to convince the engineering team of this importance.

The sooner an investment is made in a well designed software engineering interview, the sooner the impact will start to pay dividends in a company's success.

Are you looking for help designing or improving your software engineering interview process? I spent five years at Block working on improving engineering hiring. I wrote more interview questions than anyone else at the company and have thought deeply about what makes a good interview. I am available to hire as a consultant. Fill out my contact form or send me an email: contact <at> amoghk <dot> com.

If you want to hear more about engineering interviewing, subscribe to this blog! Some future topics: how write a quality coding question, the impact a good interviewing process can make, how to set up a candidate for success, and more!

Appendix

I didn’t cover the bias that a badly designed process can introduce. I’m not knowledgeable enough on this topic to write anything meaningful, but a rule of thumb is: the more relevant to the job and consistent an interview process is, the more fair it is too.

While this blog post discusses software engineer interviews—which are often considered being tedious to prepare for and often irrelevant to the job—the points made in this post can probably be applied to most professions.

I love this comment for how analogous this is to dating. I have a blog post coming up summarizing the best dating advice I have read. Don’t reject your dates or your candidates too soon.

¹ I had a lengthy list of poor interview experiences and how this ruined my impression of those companies forever. But it engulfed this blog post with my ranting. I’m going to clean that up and share it as a separate post in the future.

² Square was guilty of this too before I joined. I led the efforts to redefine the standard of a frontend coding question. These questions now provide relevant hiring signal and make the interview more enjoyable for candidates.