# What LLM interviewers actually ask about transformers

Source: https://www.techinterview.org/post/3233476821/what-llm-interviewers-ask-about-transformers/
Updated: 2026-07-23 · techinterview.org

Ask a candidate to explain attention and you learn more in ninety seconds than a take-home ever tells you. The engineers who land AI roles can derive scaled dot-product attention on a whiteboard, they know why a 4,000-token prompt costs more than four separate 1,000-token prompts, and they can say out loud when they would reach for retrieval instead of fine-tuning. What follows is the material that keeps coming up in LLM and machine learning interviews in 2026, with the answers the interviewer is actually listening for.

## Attention, and the follow-up that separates people

Every LLM interview eventually lands on self-attention. The clean way to explain it: each token is projected into three vectors, a query, a key, and a value. To decide how much token *i* should pay attention to token *j*, you take the dot product of query *i* with key *j*. Those scores pass through a softmax to become weights, and the output for token *i* is the weighted sum of all the value vectors.


```
scores  = Q @ K.T / sqrt(d_k)
weights = softmax(scores)     # a causal mask sets future positions to -inf first
output  = weights @ V
```


The follow-up that trips people: why divide by the square root of d_k? Because dot products grow with dimension. With keys and queries of dimension 64 or 128, unscaled scores get large, softmax saturates, and the gradients flowing back through it go almost to zero. Dividing by sqrt(d_k) keeps the variance of the scores near one so the model can still learn. If a candidate recites the formula but stalls on that question, the interviewer now knows they memorized rather than understood.

Multi-head attention runs several of these in parallel on different learned projections, so one head can track subject-verb agreement while another follows a long-range reference. In a decoder you apply a causal mask so position *i* can only see positions up to *i*, which is what makes generation left-to-right. Expect at least one of these:

- Why is attention O(n²) in sequence length, and what have people done about it?

- Where do positional encodings enter, and what changed with rotary embeddings (RoPE)?

- What is the difference between encoder-only (BERT), decoder-only (GPT), and encoder-decoder (T5) models?

## Tokenization: the trap questions hide in the token count

Models do not see characters, they see token IDs. Modern LLMs use subword tokenizers, usually byte-level BPE (what tiktoken implements for GPT models) or SentencePiece. BPE starts from single bytes and repeatedly merges the most frequent adjacent pair until it hits a target vocabulary size, so common words stay whole and rare ones split into pieces.

This is why the "how many r's in strawberry" failure happens: the model sees a couple of tokens, not nine letters, so character-level counting is genuinely hard for it. It also explains billing and context math. A rough rule for English is about 0.75 words per token, so 1,000 tokens is roughly 750 words. Code, JSON, and non-English text pack fewer words per token, which is why the same prompt in another language can cost noticeably more. When someone asks how you would cut API spend, "measure tokens, not characters, and trim the system prompt" beats any answer about the model itself.

## KV cache and why long context gets expensive

During generation the model produces one token at a time, and each new token attends to every previous one. Recomputing the keys and values for the whole sequence at every step would be quadratic waste, so the model caches them. That is the KV cache: the per-layer key and value tensors for every token seen so far.

Interviewers like this because it forces you to split inference into two phases. Prefill processes the whole prompt at once and fills the cache, which is compute-bound. Decode then generates tokens one at a time while reusing the cache, which is memory-bandwidth-bound. The cache grows linearly with sequence length and batch size, and for long contexts it, not the math, is usually what runs you out of GPU memory. A back-of-envelope size:


```
kv_bytes = 2 * n_layers * seq_len * d_model * batch * bytes_per_elem
# the 2 covers both K and V; d_model here is summed across heads
```


Plug in a mid-sized model and you see why 128k-token contexts need help. The usual answers are grouped-query attention (GQA) and multi-query attention (MQA), which share key and value projections across heads to shrink the cache, plus quantizing the cache to int8. If you can name GQA and note that Llama-class models adopted it to cut KV memory, you sound like you have shipped inference, not read about it.

## RAG versus fine-tuning, the answer that fails people

The most common way to bomb an AI system-design round is to answer "how would you make the model know our internal docs?" with "fine-tune it on the docs." Fine-tuning is poor at injecting fresh factual knowledge reliably, it cannot cite a source, and it goes stale the moment the docs change. Retrieval-augmented generation fetches relevant chunks at query time and puts them in the prompt, so the model grounds its answer in current text and can point at where the claim came from.

Fine-tuning earns its place for a different job: teaching consistent format, tone, or a domain reasoning style that prompting alone cannot hold. In practice most teams reach for retrieval first, prompt engineering second, and fine-tuning only when they need behavior the other two cannot produce. Parameter-efficient methods like LoRA train a small set of adapter weights instead of the whole model, which is why fine-tuning a 7B model no longer needs a cluster. Preference tuning (RLHF, or the simpler DPO) is a separate axis again, used to align outputs with what humans rate as good rather than to add knowledge.

| Approach | What it changes | Best for | Can cite sources? | Cost to run |
| --- | --- | --- | --- | --- |
| Prompt engineering / in-context | Only the input context, no weights | Quick behavior changes, few-shot examples, format nudges | No, unless you paste sources into the prompt | Lowest, just tokens |
| Retrieval-augmented generation (RAG) | Adds retrieved text to the prompt at query time | Fresh or frequently changing facts, citations, large knowledge bases | Yes | Low to moderate, retrieval plus longer prompts |
| LoRA / parameter-efficient fine-tuning | A small set of added adapter weights | Consistent format, tone, or domain style on a budget | No | Moderate to train, cheap to serve |
| Full fine-tuning (SFT) | All model weights | Deep behavior change when adapters are not enough | No | Highest to train and host |
| Preference tuning (RLHF / DPO) | Weights, shifted toward human-preferred outputs | Aligning helpfulness, safety, and refusal behavior | No | High, needs preference data |

## Evaluation, or how you avoid shipping a confident liar

"How would you know if this thing is any good?" is where strong candidates pull ahead, because most people wave at "we'll test it." A real answer has layers. Start with an offline set of graded examples that have known-good answers, run it on every change, and track a metric you trust. For open-ended output where exact match does not work, use an LLM as a judge against a written rubric, and check that judge against human labels so you are not trusting a grader nobody validated.

For retrieval systems, name the RAGAS-style metrics: faithfulness (does every claim follow from the retrieved context), answer relevance, and context precision and recall. Hallucination detection falls out of the same idea, since you score each claim in the response against the retrieved passages and flag the unsupported ones. Then say the part interviewers wait for: offline evals are necessary but not sufficient, so you also sample real production traffic every week and grade a hundred responses by hand to catch drift the test set never sees.

## Inference tricks worth naming out loud

Once the fundamentals are solid, a few optimizations show you have run these models under load. Quantization stores weights in int8 or int4 to cut memory and speed up decode with a small quality hit. Speculative decoding runs a small draft model to propose several tokens that the big model verifies in one pass, which can roughly double throughput when the draft is usually right. Continuous (in-flight) batching keeps the GPU busy by swapping finished sequences out and new ones in mid-batch instead of waiting for a whole batch to complete. FlashAttention reorders the attention computation to avoid writing the full score matrix to memory. On the sampling side, be ready to explain temperature and top-p (nucleus) sampling, and why temperature 0 gives you greedy, near-deterministic output for tasks like extraction.

If you only have an evening to prep, spend it on two things: deriving attention until the sqrt(d_k) question feels obvious, and being able to defend a RAG-versus-fine-tune decision for a concrete product. The first proves you understand the machine. The second proves you can build with it without lighting a budget on fire.