A candidate walks into the final round for an AI product role having memorized every chain-of-thought and few-shot trick on the internet. The interviewer skips all of it and asks one thing: “Your summarization feature was at 92% on your test set. It shipped. Now support tickets say it’s inventing dates. Walk me through how you’d catch that before the next release.” The people who get offers start sketching an eval harness within thirty seconds. The people who don’t start suggesting better prompts.
That split is most of the game at OpenAI, Anthropic, Scale, Sierra, and the Series B AI companies hiring in 2026. Prompt craft is table stakes now. What separates a hire from a polite rejection is whether you treat a language model like something you can measure.
The prompt round is a trap if you only talk about prompts
You’ll still get a prompt-engineering round. It usually opens with a broken prompt and a vague complaint: the model won’t hold the output format, or it’s too verbose, or it hedges when it should commit. A candidate who jumps straight to rewriting the system message is doing the visible half of the job. Interviewers want the other half.
Say the model keeps returning prose when you asked for JSON. The obvious fixes are real: put a JSON schema in the system prompt, give one or two worked examples, drop temperature to 0, and switch to a structured-output or function-calling mode if the provider has one. But the follow-up is always some version of “how do you know it’s fixed?” If your answer is “I ran it a few times and it looked right,” you’ve lost the round. Three manual runs prove nothing about a model that’s stochastic by design.
Strong candidates turn a prompt fix into a measurement problem out loud. They’ll say something like: I’d pull fifty real inputs that triggered the bug, write down the expected shape for each, and run both the old and new prompt against all fifty. Now there’s a number, and the change is defensible. That instinct, more than any wording trick, is what the round is scoring.
Building an eval from a blank page
The question that decides most loops sounds almost too simple: “You’re launching a feature that answers questions from a company’s internal docs. How do you know it’s good enough to ship?” Nobody hands you a golden set. You have to invent the measurement.
Good answers move through it in a sensible order. They pin down what “good” means for this feature before anything else: for doc Q&A that’s usually faithfulness (did the answer stick to the source) and relevance (did it answer the question that was actually asked), with completeness close behind. They find labeled data by bootstrapping a golden set, a few dozen to a few hundred real questions with known-good answers, written by hand or pulled from support logs and reviewed. Then they hit the scaling wall, because a human can’t read every output, and that’s where LLM-as-judge comes in and where the interview gets sharp.
Here’s the shape of a judge harness, which you can put on a whiteboard in a few lines:
def judge(question, answer, source):
prompt = f"""Rate faithfulness 1-5. A 5 means every claim
is supported by the source. Penalize invented facts hard.
Question: {question}
Source: {source}
Answer: {answer}
Return JSON only: {{"score": int, "reason": str}}"""
return call_model(prompt, temperature=0)
regressions = []
for case in golden_set:
output = rag_pipeline(case.question)
verdict = judge(case.question, output, case.source)
if verdict["score"] < case.threshold:
regressions.append((case.id, verdict["reason"]))
What matters is showing you know this judge is neither free nor trustworthy by default, which is exactly the question coming at you next.
Why "just use GPT as the judge" gets you grilled
The moment you propose a model as the judge, a good interviewer pushes on whether it's actually reliable. Most candidates haven't thought this through, and it's where senior signal shows up.
Model judges have known failure modes. They favor the first option in a pairwise comparison, so you swap the order and average the two runs. They reward longer answers even when a shorter one is correct. They tend to rate their own model family above competitors. And a rubric that reads as obvious to you can drift badly on edge cases. The fix isn't to drop the judge. You treat the judge as one more thing that has to be validated.
The real version of that validation: hand-label a couple hundred outputs, run the judge over the same set, and measure agreement. Cohen's kappa or a plain agreement rate tells you whether the judge tracks human judgment or just produces confident noise. If it agrees with your labelers 65% of the time, it isn't ready, and you either rewrite the rubric, add few-shot examples of correct scoring, or route the hard slice to human review. Naming that loop, judge-against-humans before you trust judge-against-model, is the single answer that most cleanly marks an experienced candidate.
The table interviewers wish you'd draw
When someone asks how you'd score outputs, there's no single right method. The move is picking the one that fits the failure you care about, and being able to lay the options side by side with their costs reads as someone who has shipped evals rather than read about them.
| Eval method | What it measures well | Rough cost per 1,000 outputs | Best fit |
|---|---|---|---|
| Exact match or regex | Deterministic or structured outputs: JSON shape, classification labels | Near zero | Format checks, closed-set classification |
| Embedding similarity to a reference | Rough semantic closeness to a known-good answer | Low | Cheap first-pass filter, catching obvious misses |
| LLM-as-judge | Open-ended quality: faithfulness, tone, helpfulness | Medium (one model call per output) | Most generative features, run automatically in CI |
| Pairwise preference (human or model) | Which of two versions users would prefer | Medium to high | Comparing prompt or model versions before rollout |
| Human review | Ground truth on ambiguous or high-stakes cases | High | Building the golden set, auditing the judge |
Evals belong in CI, on every change
The last move that lands is treating evals like tests that run on every change. A prompt edit, a model version bump, a new retrieval index: each one reruns the golden set and blocks the deploy if the aggregate score drops past a threshold or any high-severity case regresses. Teams shipping LLM features this way talk about it the way they'd talk about a test suite, with a pass bar and a flaky-case list that grows as new production failures get folded back into the set.
Bring real numbers if you have them. A golden set of 200 cases, a judge model at a cent or two per case, a full run in a couple of minutes: cheap enough to gate every pull request, specific enough to sound like you've done it. When a provider ships a new model version, you rerun the same set and see the delta before your users feel it. That last sentence is the whole reason any of this exists.
Questions you should expect, phrased the way they're asked
- "You have a chatbot that's technically correct but users hate it. How would you measure and fix the tone?"
- "Design an eval for a code-generation feature. What counts as a pass?"
- "Your LLM judge and your human labelers disagree 30% of the time. What now?"
- "You can only afford to human-review 5% of outputs. How do you pick which 5%?"
- "A prompt change raised your eval score but users complained more. What happened?"
That last one is the trap worth rehearsing. It usually means the eval set doesn't match real traffic, or the metric you optimized (say, conciseness) traded against something nobody was measuring (helpfulness). The candidates who spot it fastest are the ones who've watched an offline score and an online metric point in opposite directions and had to work out why.
You don't need a secret prompt template for any of this. You need the belief that a language model is a component you can hold to a standard, plus the ability to describe that standard on a whiteboard while someone pokes holes in it. Design the measurement, defend the judge, say plainly where it breaks, and the prompt tricks stop being the thing anyone remembers.
