How to write a ChatGPT-proof coding interview question - Part 1

The opinions here are my own and not that of any current or former employers.

Ok, sorry for the clickbait title. No question can be fully LLM-proof forever. But there are characteristics questions can have that will make it difficult for a candidate to cheat without you noticing. These characteristics also evaluate the candidate on their actual engineering skills and make the question more fun to solve.

In part 1, I’ll discuss some rules to follow when writing questions. In part 2 & 3, we’ll apply these rules in action and write a generalist / backend question and a frontend question.

Ideation

Don’t start with an overused question

For example, Stock Market Prices or LRU Cache are two famously common questions. I’ve gotten variations of these questions at FAANG+ company interviews. Many of your candidates will have solved these for practice (which is a separate problem) and LLMs will have already ingested these in their training data. It will be trivial for an LLM to solve any variation of these common problems.

Use the real world for inspiration

Some of my favorite backend / generalist questions were often based off real problems or concepts in software engineering. One question utilized sets / hash tables to store and retrieve data and asked candidates to think of efficient ways to “query” data. Another question had the candidate simulate some of the functionality of a relational database.

For frontend questions, I’ve been inspired by real apps I’ve used. Let’s take Google Maps as an example.

• There is the map UI that shows the routes. The data structure under-the-hood is a graph. Designing or working with the graph may be a good basis for a question.

• There are the parts of the app where users can select a route on the side. These are sorted and recommended to users in some way. How that is done can be a part of the problem that candidates have to solve.

In Part 3, we’re going to see how we can take these concepts and turn them into a frontend question.

Incorporate data structure design into your question

I don’t recommend asking a variation of LRU Cache only because it is overused, but it is a good question to take inspiration from. Solving it involves creating a linked-list and then writing functions to manipulate that linked-list. It’s a great exercise in design thinking. If a candidate had never seen this question before, they would initially think of a simple array as their structure. This works, but moving cached data in the array becomes an O(n) operation.

If you have a discussion with your candidate about their data structure design, this provides tremendous signal on problem analysis and communication. If they are using an LLM to cheat, they will have a harder time explaining and engaging with the interviewer.

The biggest issue with LRU cache as a question however is that there is one “best” answer.

Ensure there is more than one solution to the problem

This may not always be possible, but the best questions will have several solutions. This enables you to have a discussion with the candidate around tradeoffs. Evolving your question beyond only having an expectation of “best time and space complexity” allows you to get much more signal about the candidate; not just a list of topics they happened to study.

Expectations

Both a junior and senior candidate should be able to pass

In my early days of writing interview questions, I once interviewed a frontend engineer early in their career. They worked primarily in React and that showed during the interview.

All three of the coding questions they solved were vanilla JavaScript oriented. Due to their unfamiliarity, they failed the other two the interview questions. For my evaluation, I was on the fence. In my feedback, I wrote:

This one interview¹ helped solidify this part of my question writing philosophy.

I always find the interview more fun when a candidate is new to the concept we are working with. Experienced candidates should understand new concepts quickly having seen similar patterns in their work. For the candidates who know these concepts very well, they can show off their mastery.

All levels of candidates should be able to succeed and demonstrate their skills. If the goal of the interview is to determine someone’s experience level, you should just spend an extra two minutes reading their resume. A question should not be a pass-fail pop quiz—which is also easy for an LLM tool to solve.

Make it open-note and collaborative

At the start of every coding interview, I always tell candidates the following:

• Ask me any questions you have about the problem or the approach you are taking

• Run the code, debug, and log as much as you want

• Look up any documentation that you need

Open note questions are the most analogous to real work. I know some companies are even allowing candidates to actively use LLM GenAI coding tools, as this is how they will work day-to-day.

If you ask a candidate to share their screen during the call, you’ll see how they actively debug and use resources available to them. For candidates that don’t take advantage of this, that’s also valuable signal and an indication of how they might struggle to problem solve independently on the job.

Format

Make it incremental

I worked with product teams at both CodeSignal and CoderPad to add support for creating incremental step questions. These types of questions are beneficial for both the candidate and the interviewer.

Candidates can start simple. Large questions designed to take up the whole hour are overwhelming. Breaking up the problem will ease candidates into a problem solving mindset. Even if they spend the entire interview on the first part of the question they will have a less-stressful experience. Though they will not have passed, they will leave with a positive impression of your company.

For interviewers, incremental steps allow them to move questions in different directions. It’s easy enough to plug in LRU Cache into ChatGPT for the full solution code. But if a question starts simple and builds in an unexpected direction, it’s harder for a candidate to use ChatGPT and have a coherent conversation about the solution. You want to see the gears turning in their head as they think deeply about the problem.

Make it collaborative

Interviewers should be actively helpful and engaged during the interview. Ask the candidate to explain their approach. Give them suggestions if you think it could be helpful. See how they take this feedback and incorporate it into their work. In my years of platform engineering at Block, I spent much more time communicating and coordinating with other engineers at the company than I did writing code.

Staying engaged with a candidate throughout the interview makes it harder for them to quietly slip away and read back a response from ChatGPT.

Next post, we’ll put this all into practice and write a backend / generalist coding question. Be sure to subscribe!

Are you looking for help designing or improving your software engineer interview process? I spent five years at Block working on improving engineer hiring. I wrote more interview questions than any other engineer at the company. I am available for hire as a consultant. Fill out my contact form or send me an email: contact <at> amoghk <dot> com.

¹ The candidate did well in other non-coding parts of the interview, but I like to think my feedback above was a deciding factor in choosing to hire them. I would later go on to work directly this person and they proved to be an excellent teammate.