In this post, I take the concepts from part 1 and document my process of writing a question. This question is specifically for backend / generalist engineers. In part 3 we’ll use this same process to write a frontend question.
Coming up with an idea
Start timer - 00:00:00
This is the hardest part. When I wrote interview questions at Block, I typically let inspiration come to me rather than brainstorm.
A common pitfall is to write a graph question. This isn’t inherently bad, utilizing graph data structures is commonplace in software engineering. But most engineers don’t model data with graphs every day. If our goal in an interview question is to model real work, this is a better place to start thinking of ideas.
So let me think back on some of the work I did at Block:
Pub/sub systems - I spent my first few years at Block working on event streaming. I wrote middleware to send marketing events to third-party data vendors for attribution. Some engineering problems from that project:
Schema validation. I wrote recursive logic to validate a json object to match a given type. This is an ok idea for a question, but it’s graph oriented. I don’t consider recursion to be too difficult but schema validation code is better written in a relaxed 3-6 hour chunk of time. The recursion is complicated enough that it would not fit well into a 60 minute interview.
Authentication and error handling - this is an interesting idea! The middleware I wrote had well documented error handling. But what made this engineering work interesting was designing two parts of a pipeline to fit together. I’m not sure this makes much sense for a 60 minute interview.
Eventing libraries - I did a ton of work writing and improving marketing eventing libraries at Block:
Event queues - There could be something interesting about sending event data from a queue and then handling errors and re-entering events into the queue. For subsequent parts of the question, we could think about sorting the data back into the queue.
Timer - 00:20:22
Ok! Let’s run with this idea and see if we can flesh it out into a multi-step interview question.
Outlining the basics of the question and writing Part 1
Write a class that ingests events into a queue. When you reach 10 events, flush the queue. For now, flushing the queue will just mean printing out events and emptying the queue.
Let’s give this a shot by writing solution code.
class EventQueue {
constructor(limit) {
this.q = [];
this.limit = limit;
}
add(event) {
this.q.push(event);
if (this.q.length === this.limit) {
this.flush();
}
}
flush() {
this.q.forEach(console.log)
this.q = [];
}
}This took roughly 4 minutes. I like part 1 of this question so far. It’s purposely ambiguous and a good candidate will ask clarifying questions. I am passing the queue limit as a property of the class constructor meaning this is a dynamic value; as an interviewer, I like this design choice from the candidate. This is good opportunity to demonstrate abstraction to separate the flushing logic from the adding logic.
Part 1 feels too simple as is, but let’s leave it. The expectation is for a candidate to finish this part in 10 minutes max. We can add complexity as we go.
Timer - 00:31:40
Expanding complexity, writing Part 2
Let’s incorporate a network component into this. Write logic to send events over the internet.
class EventQueue {
constructor(limit, postEndpoint) {
this.q = [];
this.url = postEndpoint;
this.limit = limit;
}
add(event) {
this.q.push(event);
if (this.q.length === this.limit) {
this.post();
}
}
flush() {
this.q = [];
}
post() {
fetch(this.url, {
method: 'POST',
body: JSON.stringify(this.q),
headers: {
'Content-Type': 'application/json',
},
}).then(res => {
if (res.ok) {
this.flush();
}
});
}
}
That only took me 4 minutes. Oof, this question seems too easy. At this stage of question writing the solution may seem easy and obvious to you, the author. But an average candidate will need time to understand the question requirements and formulate a plan.
The real meat of the question - a hidden challenge in Part 2
A good interview candidate will at this point ask the question, “what happens if the request fails?” This provides good signal that the candidate is thinking about error handling and edge cases - you should hire them!
Just kidding, but it is positive hiring signal. If a candidate doesn’t bring up the failure case, ask them some questions about what to do if things go wrong. A great candidate will continue thinking down this path: “ok, so we retry sending the event, then what happens if another event is added to the queue while the network event is in flight? Should we let the queue grow past our limit of 10?”
I didn’t even think about this edge case during ideation, but this is proving to be a good question concept. I would tell the candidate that ideally, we don’t let our queue grow past 10, and that they should design a way to handle this edge case. This open-endedness will allow the candidate to show their creativity and code maintainability skills.
class EventQueue {
constructor(limit, postEndpoint) {
this.q = [];
this.url = postEndpoint;
this.limit = limit;
}
add(event) {
this.q.push(event);
if (this.q.length === this.limit) {
this.post();
}
}
flush() {
this.q = [];
}
post() {
this.inFlightQ = this.q;
this.flush();
this.postMethod(this.inFlightQ).then(res => {
if (res.ok) {
this.inFlightQ = []
} else {
this.postMethod(this.inFlightQ)
}
});
}
postMethod(dataArr) {
return fetch(this.url, {
method: 'POST',
body: JSON.stringify(dataArr),
headers: {
'Content-Type': 'application/json',
},
});
}
}
Adjusting the code took 5 minutes. It’s far from perfect though. There are two issues I see with this. Let’s fix these.
This doesn’t actually recurse. Upon closer inspection, my fetch logic would only run twice!
We run the risk of overwriting data. It is possible to fill up the queue while the previous array of data is still sending.
class EventQueue {
constructor(limit, postEndpoint, retryLimit) {
this.q = [];
this.url = postEndpoint;
this.limit = limit;
this.retryLimit = retryLimit;
}
add(event) {
this.q.push(event);
if (this.q.length === this.limit) {
// clones the queue so we're not mutating the same
// array of data
this.post([...this.q]);
// moves flushing logic outside of the post method
this.flush();
}
}
flush() {
this.q = [];
}
async post(queueOfEvents, attempt = 1) {
const response = await this.postMethod(queueOfEvents);
if (!response.ok) {
if (attempt >= this.retryLimit) {
console.error(`Unable to send data:\n${JSON.stringify(queueOfData)}`);
return;
}
this.post(queueOfEvents, attempt + 1);
}
}
postMethod(queueOfEvents) {
return fetch(this.url, {
method: 'POST',
body: JSON.stringify(queueOfEvents),
headers: {
'Content-Type': 'application/json',
},
});
}
}
Much better! That took me 10 minutes. The total time I’ve spent working on this question as a candidate is now ~25 minutes. That doesn’t even include time to test the code which would add another 5-10 minutes.
Timer - 01:08:34
This is looking good so far! Though part 2 starts simple, it gets more complex once you factor in handling failures; the strongest candidates will identify and address this themselves.
A skilled and experienced engineer can write asynchronous recursive logic and handle all possible failures. A less experienced but still good engineer will probably need more guidance on strategy and some hints, but should still be able to solve the problem in ~40 minutes.
For most companies, this is probably a satisfactory question. It can be reasonably solved in 45 minutes and it is difficult enough to weed out candidates you don’t want to hire. But let’s take the extra step and add a challenge.
Make it harder and with algorithmic complexity, writing Part 3
During ideation, I thought about having the queue flush every 5 seconds, but that’s boring and the logic is easy to code. For part 3, we want to give the candidate a real challenge!
Another idea I had early on was handling partial failures. The idea is that some events will be received by the server and others will be rejected; this borrows from my earlier question idea “Authentication and error handling.” The server will now send the following responses:
Status code | Action |
|---|---|
200 | Events have all been received. |
206 | Events are mixed, some success and some failures. An array of booleans will be returned with successes and failures. |
500 | All data has not been received. |
If the server responds with 206, only the failed events should be retried. Events that completely fail should be added back into the queue.
. . . Hmm, now that I think about it, this does not raise the question difficulty. The queueing logic is just slightly more complicated. If anything, this requirement could be added to part 2.
Time - 01:23:10
Back to brainstorming!
What if we preserve and sort the failed data into long term storage? In a real world engineering design, this would be similar to saving the data in cookies / local storage on the frontend or Redis / a database on the backend. We’ll simplify that concept in the question by simply saving to an array.
For events that have failed to send after all retries, save this to an in-memory array. Make sure events are sorted by their timestamp. Assume the long term array of events and your failed queue of events both start sorted from earliest event to latest event.
A good candidate will recognize this is basically a simplified merge sort. Candidates are likely to simply call a built in sort method which is O(n log n). But since our starting arrays are already sorted, we can do this in O(n), where n represents the combined length of the arrays.
Timer - 01:48:13
This code block only includes a new method I added. There are a few other small code changes to the codebase which you can infer.
addToFailed(newQueue) {
// relabeling arrays for simplicity
const aQueue = newQueue;
// copying array to prevent any pointer errors
const bQueue = [...this.failedQ];
// reversing arrays for time efficiency
// not needed, but nice to run each .pop in O(1) rather than each .shift in O(n)
aQueue.reverse();
bQueue.reverse();
const mergedQueue = [];
// assume events will be an object with an integer property of "time"
while (true) {
const aEvent = aQueue.pop();
const bEvent = bQueue.pop();
// if both arrays are empty
if (!aEvent && !bEvent) {
this.failedQ = mergedQueue;
return;
}
if (aEvent && !bEvent) {
newQueue.push(aEvent);
// reverse it back to soonest event first
aQueue.reverse();
this.failedQ = newQueue.concat(aQueue);
return;
}
if (bEvent && !aEvent) {
newQueue.push(bEvent);
// reverse it back to soonest event first
bQueue.reverse();
this.failedQ = newQueue.concat(bQueue);
return;
}
// unlikely edge case of both events having the same time
if (aEvent.time === bEvent.time) {
newQueue.push(aEvent);
newQueue.push(bEvent);
}
if (aEvent.time < bEvent.time) {
newQueue.push(aEvent);
// put event back for next comparison
bQueue.push(bEvent);
continue;
}
if (bEvent.time < aEvent.time) {
newQueue.push(bEvent);
// put event back for next comparison
aQueue.push(aEvent);
continue;
}
}
}This part 3 solution took 15 minutes. Part 3 isn’t complicated, but it is difficult enough for a skilled candidates to demonstrate their skills. An excellent candidate could solve the whole problem in 40 minutes, which is a good benchmark for an interview that is supposed to last 45 minutes to an hour. I’m satisfied with the question.
Total time to write question - 02:08:27
This only took me 2 hours, but I am an experienced question author; oftentimes, it takes much more iteration to reach this point. I got lucky this time and the first viable idea I came up with worked out. But I estimate I’ve scrapped 80% of questions I started writing (or at least rewritten significant parts of the question).
If we wanted to make this more difficult, we could instead have candidates sort these failed events back into the original queue and consider the queue size limit and flushing complexities that come with that.
To prepare this question for actual candidates, this is the work still ahead of me and my estimate on a time commitment:
Task | Time | Notes |
|---|---|---|
Write server code to handle network requests | 1 hour 30 minutes | The server code needs to be robust enough to handle unexpected failures from candidates sending invalid data. The server should have an adjustable failure rate, i.e. a certain number of network requests will randomly fail based on a variable that the interviewer can change. |
Write out question instructions. | 30 minutes | Make instructions as clear and concise as possible. Non-native english speakers should be able to understand the question. |
Prepare test data | 30 minutes | For each step of the question, generate some test data. The input needed to test all parts of the question (especially part 3) is complicated enough that the candidate should not be expected to generate this themselves. |
Format all this into a code interviewing platform | 1 hour | It always takes some time to format and prepare this in the interviewing platform of choice. |
Document this question | 1 hour | Interviewers using this question need to know some background data about the question. This starts pretty simple because we haven’t tested this on real candidates. |
Mock interview | 3-6 hours | Engineers at the company (or outside of it) need to help test the question. No matter how well the question is written, things will need to be adjusted after mock testing. |
Revise server code, instructions, test data, and documentation | 4 hours | This will naturally happen during testing. |
At this point (12-16 hours later), the question is ready for use on real candidates. But real candidates will vary much more significantly in their performance. We’ll need to continue adjusting the question for the first 10-20 interviews.
How ChatGPT proof is this?
Amogh from the future here. I used this question after starting a talent agency for software engineers. The question accomplished its goal of collecting relevant signal around engineering skill and being ChatGPT-proof.
I had a candidate solve the problem and their solution was too perfect. This cheating candidate didn’t ask me many questions. Their solution didn’t change much as the problem progressed. They stayed silent during most of the interview.
In contrast, the candidates who passed were regularly communicating with me. They were asking clarifying questions as they went. They considered different approaches and sometimes changed their ideas and refactored. They demonstrated a deep understanding of the real engineering context behind the problem.
The only reason the cheating candidate fooled me initially is because it was only my fifth time administering the question; I wasn’t familiar with the typical pitfalls and thought patterns candidates run through.
This rest of this post used to be me prompting ChatGPT through solving my question. Instead, I’m now going to share the cheating candidate’s solution so you can see the contrast for yourself.
Giveaways that this solution is LLM generated:
Logs are overly detailed and some of them are odd: no one else ever thought to count the
num_eventsin the flushing function.get_queue_sizeis a function that was declared but never used.merge_sorted_eventsis called in the wrong place. It should be called after all retries fail.The merge sort implementation looks straight out of the textbook.
As I had more practice administering the problem, I later realized making the request async is key to writing a good solution. A candidate must clone and clear their queue before sending the data to the client. Good Python candidates brought up this idea. This cheating candidate never thought of it; the LLM probably thought it was fine to clear the queue after the request.
class EventQueue:
def __init__(self, flush_threshold=10, api_url: str = None, max_retries=1):
self.events = []
self.flush_threshold = flush_threshold
self.api_url = api_url
self.max_retries = max_retries
self.failed_events = []
def add_event(self, event):
self.events.append(event)
if len(self.events) >= self.flush_threshold:
self.flush()
def flush(self):
if not self.events:
print("Queue is empty, nothing to flush")
return
num_events = len(self.events)
events_to_send = self.events.copy()
for attempt in range(self.max_retries):
if attempt == 0:
pass
else:
time.sleep(0.5)
try:
response = requests.post(
self.api_url,
json=events_to_send,
headers={"content-type":"application/json"}
)
if response.status_code == 200:
print(f"Successfully sent {num_events} events to API")
print(response.text)
self.events.clear()
return
else:
print(response.text)
if 400 <= response.status_code < 500:
print("Server Error")
return
if attempt < self.max_retries - 1:
print(f"Retrying.....{self.max_retries - attempt - 1} attempts remaining")
except Exception as e:
print(f"Error {str(e)}")
self.merge_sorted_events(events_to_send)
self.events.clear()
# print(f"flushing {len(self.events)} events")
# print(json.dumps(self.events, indent=2))
def get_queue_size(self):
return len(self.events)
def merge_sorted_events(self, new_failed_events):
if not new_failed_events:
return
merged = []
i, j = 0, 0
while i < len(self.failed_events) and j < len(new_failed_events):
failed_time = self.failed_events[i].get('time', 0)
new_time = self.failed_events[j].get('time', 0)
if failed_time <= new_time:
merged.append(self.failed_events[i])
i += 1
else:
merged.append(new_failed_events[j])
j += 1
merged.extend(self.failed_events[i:])
merged.extend(self.failed_events[j:])
self.failed_events = merged