Software engineering coding interviews: hard != good

The opinions here are my own and not that of any current or former employers.

Early in my career, I contracted at a FAANG company. I referred a friend for a contracting role on a different team. One of the tenured members of that team—let’s call him Bob—was in charge of designing the interview and gave my friend an impossibly difficult coding question to solve. Later, I heard Bob’s team members balking at the idea of having to pass this question. Even after they had passed the notoriously difficult FAANG software engineer interview, they didn’t dare to try Bob’s question for contractors.

During the 2020 pandemic, I volunteered to give mock interviews to job seekers in my network. A few of these were top notch LeetCoders; they had hundreds of questions solved across all levels of difficulty.

I gave them a graph design question. Many people solving this question would overthink it. A common initial approach was to try a topological sort. My response to that, was why? Nothing in the question description required knowing which node comes first.

The most prolific LeetCoders completely bombed my question. They fixated on searching for some difficult and esoteric solution. But to solve the problem, there were two simple steps. First recognize that the input data is best represented by a graph. Second, write logic to turn the input data into a graph.

You know what they say when you assume

The assumption of the LeetCoders was that being able to solve harder questions would improve their chances of getting a job. This is generally true, but they were missing the forest for the trees. By imagining my question must be complicated and difficult, they missed the simplicity of the data structure entirely. The real challenge of the question was how elegantly could you organize your code to construct a graph.

The assumption of Bob that designing a hard question would be better was completely wrong. The goal of his team was to get contractors hired and productive as fast as possible. Filtering out good talent by asking a too difficult question will not accomplish that.

Why companies don’t actually ask LeetCode Hards

In my last round of interviewing 5 years ago, I only encountered one question I would consider “LeetCode Hard.” This is a good thing. I suspect companies quickly learn that a too-difficult question will simply filter out too many candidates. Even for the hard question I solved, I later learned that the most difficult part was a bonus. The actual passing point of the question was more at a “LeetCode Medium” difficulty.

Companies know that engineers who can solve their too-hard questions likely got lucky and happened to study the right combination of topics. Solving “LeetCode Hard” problems is not at an indicator of a 10x engineer¹ .

I learned the futility of hard questions firsthand as a question author. Testing questions on myself, I quickly ran out of time to code even when I knew the solution. And when it came to interviewing candidates live; many would get caught up on the earlier and easier parts of the problem. This isn’t even a bad thing! There was a lot to learn about a candidate’s skill when it came to debugging and writing maintainable code. Making a question difficult does not provide more signal on a candidate. If anything, it will provide less signal; the difficulty of a question takes up time that a candidate can use to demonstrate other positive aspects of their coding skills.

Origins of the hard interview

It’s safe to say Google popularized the algorithm & data structure coding interview format. Google and most FAANG+ companies ensure their questions are difficult.

I think these companies copy each other for simplicity. They are looking for a similar caliber of talent and are willing to pay similar amounts. Even the structure of their career engineering ladders are the same. For example, many start at level 3.

While many lament on LinkedIn about FAANG+ companies’ difficult interviews, others will grind LeetCode for the reward of a L3 software engineering job that pays $200K. And I concede that the juice is worth the squeeze. I estimate I probably spent 160 hours doing the algorithm grind in 2019-2020. It resulted in L4 offers from Google and Block (Square) with a total compensation more than twice as much as I was previously making.

But the problem with the “difficult algorithm and data structure question” is that it filters out many good engineers. Regardless of a company’s bank balance, having a poorly designed and difficult coding interview is costly. Using this math, we can roughly say that each candidate that makes it to the technical screen costs $1000 to interview; moving to the onsite costs an additional $10,000.

Hard questions incur more than monetary costs

FAANG+ companies can afford to conduct endless interviews and reject good candidates. I suspect this is somewhat of a deliberate strategy; you can afford to filter out good candidates if the alternative is making a bad hire (which is also costly).

I do agree that if you can solve a difficult algorithm question, you’re probably decent at coding. It doesn’t necessarily mean you’ll be a great software engineer, but coding is a big part of that job. Odds are, these engineers will be successful in a big tech environment.

Sticking to this system may work for FAANG+ companies; but it’s still not a great strategy for a company of any size.

Senior candidates you want to hire won’t tolerate this

Many candidates—especially senior ones—don’t have the patience or time to grind LeetCode. Unlike 21 year old college students, they have kids to look after or may be swamped with high-level work at their existing company.

It’s these candidates that are always in high demand. I remember conducting an interview for a senior staff engineer. The candidate was fairly annoyed to even be asked a coding question and didn’t see the point of it² . Not a great attitude to bring into an interview, but this engineer was in a place in their career that they didn’t have to conform to every company’s standard in order to find a good role.

You shouldn’t annoy candidates

I have an upcoming post on how your interview is a reflection of your hiring brand, but the thesis statement of that is you should not annoy your candidates! The one “LeetCode Hard” interview I got as a tech screen? That was Block. I joined Block, worked there for five years, and helped transform the interview process. But I’m still annoyed about that interview question! So much so that I’m still complaining about it to you!

Aside from my personal grumpiness, candidates will talk. If you have an annoying process, word will get out.

The landscape is shifting; hard algorithm questions no longer cut it

You may have heard the story of Chungin “Roy” Lee, a student who built a tool to cheat at algorithm interviews. My hot take on the situation? I’m glad this happened. Far too many interview questions are lazily written. And the fact that most can be solved by ChatGPT is a clear indication of that. Companies will need to evolve their coding questions to combat this.

There are ways to write challenging questions that can’t be so easily solved by ChatGPT. I’ll talk about that in a future post, so be sure to subscribe.

What is a candidate to do?

Sadly, my blog is not so influential that I can change the whole industry—not yet anyway. If you are a candidate, you still probably need to grind LeetCode.

But remember that most companies know a difficulty of question is not everything. They value things like your ability to communicate and collaborate with the interviewer. They don’t just care if your code runs, but the quality of how you have written it. Approach the coding interview as a collaborative process; not simply a standardized test.

And after you’ve solved enough LeetCode questions in each category, please stop! Continue solving questions for practice, but focus more of your attention towards mock interviews and soft skills.

Are you looking for help designing or improving your software engineer interview process? I spent five years at Block working on improving engineer hiring. I wrote more interview questions than any other engineer at the company. I am available for hire as a consultant. Fill out my contact form or send me an email: contact <at> amoghk <dot> com.

¹ I once lived in a hacker house in San Francisco. One of the software engineers there had just started a job at a promising startup. I later learned that they were a savant when it came to algorithm problems.

As for that promising startup? They got fired quickly. I heard through the grapevine that others in the hacker house found this engineer to be stubborn and difficult to work with. Surprise, surprise! Solving difficult problems doesn’t translate to being a good teammate and productive engineer.

² The point of asking L7 and L8 engineers coding questions was less about seeing if they could solve these problems, but more to understand how they approached the problem and if they could work collaboratively with the interviewer. This candidate refused to take my suggestions and didn’t get very far. This attitude is a big red flag and is the main reason they didn’t pass the interview.