# What applied-AI engineer interviews test in 2026

Source: https://www.techinterview.org/post/3233476824/ai-engineer-interview-rag-agents-evals/
Updated: 2026-07-23 · techinterview.org

The first screening question at a lot of applied-AI teams right now is some version of this: you have a 200-page PDF and a user question, walk me through everything that happens between the user pressing enter and the answer coming back. No trick to it. Within about ninety seconds the interviewer knows whether you've shipped a retrieval system or only read about one.

That question also maps how these interviews changed. Two years ago an ML engineer loop meant gradient descent on a whiteboard, a bias-variance question, maybe a model you'd train from scratch. Teams hiring for RAG, agents, and LLM products mostly dropped that. You don't train the model. You build a system around a model you don't control, and the questions follow the work.

## The shape of the loop

A typical 2026 loop for an applied-AI or LLM engineer role runs four or five rounds, and only one of them looks like a classic algorithms interview. Anthropic's Applied AI Engineer process is a useful reference point: a recruiter screen, a technical phone screen, a take-home or live coding round, a customer-conversation simulation, and an onsite that leans on system design. The customer-conversation round is the one candidates underestimate, and it quietly rejects a large share of people who cleared the coding stages. It puts you in front of a made-up customer with a vague problem and watches whether you can turn it into something you'd actually build.

The table below is the version I'd hand someone prepping this week.

| Round | What it actually tests | A question phrased the way it's asked |
| --- | --- | --- |
| Technical phone screen | Working knowledge of retrieval and prompting, not trivia | "You have a 200-page PDF and a question. What happens between enter and the answer?" |
| Take-home | Whether you can build and measure an LLM system end to end | "Here are 50 markdown docs. Build a Q&A system, an eval set of 20 questions, and report recall@5 and faithfulness." |
| Live coding / debugging | Fixing the integration by hand, no framework hand-waving | "This agent loops forever on some inputs. Find out why and fix it." |
| System design | Designing an eval and serving path around a model you don't own | "Design a support assistant. How do you know it's helping and not making things worse?" |
| Customer / product sense | Turning a fuzzy user problem into a scoped, measurable system | "A customer says the bot 'feels dumb.' What do you change first?" |

## The retrieval round

RAG questions start easy and get pointed fast. Chunk the docs, embed them, store the vectors, retrieve top-k, put them in the prompt: everyone can recite that. The interviewer is waiting for the second layer. How big are your chunks, and why? Fixed 512-token windows are the lazy default and they cut sentences in half; structure-aware chunking that splits on headings and keeps tables intact usually beats them, at the cost of more code. What's your k, and what breaks when you raise it? Bigger k buys recall and burns context budget and latency, and past a point it feeds the model more distractors than signal.

The strongest signal in this round is knowing when retrieval is the wrong tool. If the task is "summarize the one document the user just uploaded," you don't need a vector store, you need the whole doc in the context window. If answers depend on a number that changes every minute, retrieval over a stale index will hand back confident, wrong answers. Good candidates say that out loud instead of drawing more boxes.

Expect to be pushed on measurement. Retrieval quality gets scored with recall@k and precision@k, and ranking quality with MRR or NDCG when the order of results matters. If you can't name what "good retrieval" means as a number, the interviewer assumes you tuned yours by feel.

## Evals are the round people flunk

This is the widest skill gap in the field, and interviewers know it. Plenty of candidates can wire up a RAG pipeline. Far fewer can tell you whether it works, or catch the moment a prompt change quietly made it worse. Teams building this for real spend more time on evaluation than on the pipeline, and the interview reflects that.

A common take-home hands you a prompt and thirty sample input/output pairs and asks you to build an automated eval that scores faithfulness and relevance, then calibrate it against the human labels they provide. Faithfulness asks whether the answer stuck to the retrieved context or invented facts. Relevance asks whether it actually addressed the question. Deterministic checks catch what they can (exact match, a regex, does the cited source even exist), and an LLM-as-judge grades the rest.

The second you say "LLM-as-judge," a good interviewer asks how you know the judge is any good. That is the real question. Judge models have measurable biases: they favor longer answers, they favor whichever response comes first when comparing two, and they lean toward text from their own model family. The answer they want back is calibration. You score a set the judge has graded, compare against human labels, measure agreement, and tighten the rubric until they line up. A rough version of the judge is short:


```
def faithful(answer, context, judge):
    prompt = (
        "Context:\n" + context + "\n\n"
        "Answer:\n" + answer + "\n\n"
        "Does every claim in the answer follow from the context? "
        "Reply YES or NO, then list any unsupported claims."
    )
    verdict = judge(prompt)
    return verdict.startswith("YES"), verdict
```


A dozen lines. What the interviewer is checking is whether you'd trust its output before testing it against humans. The right answer is no.

## Agents, and the ways they fail

Agent questions moved from "what is an agent" to "your agent is misbehaving in production, debug it." The failures are specific to these systems and a normal test suite never catches them. An agent gets stuck calling the same tool in a loop until it burns the token budget. Retrieval drifts as the index grows and the answers slowly rot. A prompt tweak that helped one case regresses ten others you weren't watching. None of it shows up as a stack trace.

So the round turns into an observability conversation. How do you trace a multi-step run so you can see which tool call went sideways? How do you cap tool calls and spend per request so one runaway session can't cost fifty dollars? What happens when the model asks to run an action it shouldn't? Candidates who have operated one of these talk about tracing every step, setting hard limits on iterations and cost, and replaying failed runs against a fixed set of cases before shipping a prompt change. Candidates who haven't talk about the model being smart.

Cost and latency sit under all of it. If every step calls a frontier model, a five-step agent is five round trips, and a target like 800ms end to end forces real choices: a smaller model for routing, semantic caching on repeated queries, independent tool calls fired in parallel instead of in sequence. Interviewers want the numbers because production runs on them.

If you're prepping, build one of these before you read another question list. Take a folder of docs you know cold, stand up retrieval, write twenty questions with answers you can check, and measure the thing. The gap between people who have done that and people who have only studied it is visible in the first five minutes, and no amount of memorized terminology closes it.