The fastest way to fail a transformer question is to recite the block diagram. “There’s attention, then a feed-forward layer, and you stack them.” Everyone says that. What gets you the offer is explaining why scaled dot-product attention divides by sqrt(d_k), and what breaks if you drop it.
The attention question that has a real answer
Every candidate can write the formula: attention is softmax(QK^T / sqrt(d_k)) V. The follow-up is where it gets interesting. Why the scaling term?
Q and K are vectors of dimension d_k, and their dot product sums d_k component-wise products. If those components are roughly unit-variance and independent, the sum has variance near d_k, so raw scores grow with head size. Push large-magnitude scores through softmax and it saturates: one value approaches 1, the rest approach 0, and the gradient through the softmax collapses toward zero. Dividing by sqrt(d_k) holds the score variance near 1 regardless of dimension, which keeps training stable. A candidate who says “otherwise softmax saturates and gradients vanish” has answered the question. “For normalization” has not.
The reason for multiple heads
Split the model dimension across several heads and each attends within a lower-dimensional subspace. A common question: with a 4096-dim model and 32 heads, each head works in 128 dims, so why not a single 4096-dim head? Because one softmax gives you one attention distribution per position. Several heads let the model attend to different relationships at once, one head tracking a subject-verb dependency while another follows the token next door. Same total compute as one big head, more expressive use of it.
Positional encoding is where people fumble
A question that trips up engineers who’ve used transformers but never studied them: attention is permutation-invariant. Shuffle the input tokens and, with no position signal, the output set is identical. Attention computes weighted sums over a set, and a set has no order. Language has order, so position has to be injected somewhere.
The original paper added fixed sinusoidal vectors to the token embeddings. Learned absolute embeddings came next, a lookup table indexed by position. Both carry a weakness the interviewer will push on: they encode absolute position and don’t extend past the lengths seen in training. Ask a model trained on 2K tokens to handle 8K and absolute schemes break down.
That’s why the 2026 default is rotary embeddings, or RoPE. Rather than adding a position vector, RoPE rotates the query and key vectors by an angle proportional to their position. Take the dot product of a query at position m and a key at position n and the rotation leaves a term that depends only on m - n, the relative distance. That relative property is what lets position interpolation and NTK-aware scaling stretch a model’s context window after training. ALiBi takes a simpler road, biasing attention scores by a penalty linear in distance, and it extrapolates too, which is why “how would you extend the context length” usually wants one of these named.
Encoder, decoder, and the masking that separates them
A frequent phrasing: “BERT and GPT are both transformers, so what’s the architectural difference?” The short answer is the mask.
An encoder like BERT uses bidirectional attention. Every token sees every other token, past and future, which is what you want for classification or retrieval where the whole input is present at once. A decoder, GPT and nearly every chat model, uses causal masking: position i attends only to positions ≤ i. You enforce it by setting the masked scores to negative infinity before the softmax so they contribute zero weight. That mask is the whole reason a decoder can be trained to predict the next token without peeking ahead.
Encoder-decoder models like T5 and the original translation transformer keep both halves and add cross-attention, where the decoder’s queries attend to the encoder’s keys and values. Asked when you’d still reach for that design, sequence-to-sequence tasks with a clean input/output split, translation or summarization, are still the natural fit.
The KV cache, and why inference engineers care so much
This question tells a senior interviewer whether you’ve run a model in production. During generation a decoder emits one token at a time, and token 500 attends to all 499 before it. Recompute the keys and values for every previous token at every step and you’ve built quadratic wasted work. The KV cache stores the key and value tensors for tokens already processed, so each step computes Q, K, and V for only the new token and reads the rest from cache.
The follow-up is always memory. Cache size is roughly 2 × layers × kv_heads × head_dim × seq_len × batch × bytes, the leading 2 for keys and values. Put a mid-size model at long context into that formula and you land in the tens of gigabytes for the cache alone, frequently more than the weights themselves. That number is why the attention variants below exist, and why “how large is your KV cache at 128K context” is a working question, not trivia.
Attention variants and what they trade
| Attention variant | Key/value heads relative to query heads | KV cache size | Quality vs standard multi-head | Models that use it |
|---|---|---|---|---|
| Multi-head (MHA) | one KV head per query head | baseline, largest | reference point | original Transformer, GPT-2 |
| Multi-query (MQA) | one KV head shared by all query heads | smallest, about 1/N | small quality drop, can destabilize | PaLM, early Falcon |
| Grouped-query (GQA) | KV heads shared across small groups | between MQA and MHA | near multi-head | Llama 2/3 70B, Mistral |
| Multi-head latent (MLA) | compressed low-rank KV, expanded per head | small | near multi-head, strong | DeepSeek-V2 and V3 |
GQA became the default because of a straightforward trade. MQA collapses every key-value head into one, shrinking the cache the most but risking quality and training stability. MHA keeps full quality at full memory cost. GQA sits between them, sharing keys and values across small groups of query heads, and in practice it holds almost all of multi-head quality while cutting the cache several-fold. When an interviewer asks “why did Llama 3 pick GQA over MQA,” they want that trade named, not a definition.
Why long context is expensive, and what flash attention does not fix
Attention is O(n^2) in sequence length: every token attends to every other, so the score matrix is n × n. Double the context and you quadruple both the compute and the memory for that matrix. This is the constraint sitting behind every “how do you handle long documents” question.
A common trap: candidates claim “flash attention makes it linear.” It does not. FlashAttention is an IO-aware kernel that avoids writing the full n × n matrix to high-bandwidth memory, computing attention in tiles that stay in fast on-chip SRAM. It cuts memory from quadratic to linear and runs several times faster on the clock, but the compute stays O(n^2). For genuinely sub-quadratic scaling you’re looking at sparse or sliding-window attention, the latter being Mistral’s approach, or state-space models like Mamba, each with its own quality trade-offs. Telling “less memory traffic” apart from “lower asymptotic complexity” is exactly what a sharp interviewer is checking.
What a 2026 decoder block actually looks like
If you learned transformers from the 2017 paper, your mental model is a generation behind, and lab interviewers notice. A current decoder layer has drifted in three specific ways. Normalization moved ahead of the sublayers instead of after them (pre-norm), because deep post-norm stacks are hard to train without careful warmup. LayerNorm gave way to RMSNorm, which drops the mean-centering and scales by the root mean square, a little cheaper and empirically as good. And the feed-forward layer traded ReLU for a gated activation, usually SwiGLU, spending about a third more parameters there for a steady quality gain.
None of this rewrites the attention story above, but rattling off “pre-norm, RMSNorm, RoPE, SwiGLU, grouped-query attention” when asked what a modern block contains signals you’ve read a recent model’s code and not only the textbook.
Questions worth rehearsing out loud
- Walk me through what happens to one token as it moves through a single decoder layer.
- Why divide the attention scores by the square root of the key dimension?
- Your model trained at 4K context and you need 32K at inference. What are your options?
- Estimate the KV cache memory for a 70B model at 100K context, batch size 8.
- Why did most open models move from multi-head to grouped-query attention?
The pattern under all of them is the same. The definition is the easy 20 percent, and the interviewer is fishing for the other 80: the reason a design choice exists and what it costs. Trace a single token through the block and explain each piece as a decision someone made under a real constraint, and you’re already past most of the field.
