An interviewer slides a constraint across the table: fine-tune a 13-billion-parameter model on a single A100, 40 gigabytes of memory, and be ready to explain every number you pick. Full fine-tuning is off the table before you start. The weights alone overflow the card once you add optimizer state and gradients. This is the moment LoRA exists for, and the real question underneath the exercise is whether you understand why.
The decomposition they want on the whiteboard
LoRA freezes the pretrained weight matrix and learns a low-rank update beside it. For a weight W0 of shape d×k, instead of training a full update the same shape, you train two small matrices: B (d×r) and A (r×k), with rank r far smaller than d or k. The forward pass becomes:
h = W0 x + (alpha / r) * B (A x)W0 never changes. Only A and B get gradients. Take a 4096×4096 attention projection: a full update is about 16.7 million parameters. With r=16 you train 4096×16 twice, roughly 131 thousand, a bit over a hundredth of the original. Multiply that saving across every targeted layer and the trainable footprint drops from billions to tens of millions, which is what makes the single-GPU exercise possible in the first place.
Then the follow-ups start. Why the alpha/r factor? Alpha is a scaling constant that sets how much the adapter’s output counts relative to the frozen base. Dividing by r keeps that scale roughly stable as you change rank, so you can raise r without silently cranking the effective learning rate. A common setting is alpha at twice r, though plenty of teams fix alpha and sweep r. If you answer “I set alpha to 32 because the tutorial did,” you have told the interviewer you do not know what it does.
Rank is the next probe. Small r, 8 or 16, works when the task sits close to what the model already does, like adjusting tone or a fixed output format. Push r to 64 or higher when the domain shift is larger, say teaching a code model a proprietary language. There is a real ceiling: past some rank the extra capacity stops helping and you pay memory for nothing. Which modules you attach to matters as much as the number. The original paper adapted only the query and value projections; current practice usually targets every linear layer, attention and MLP both, because the MLP blocks hold a large share of a transformer’s parameters and leaving them frozen caps how far the adapter can move the model.
How QLoRA squeezes a huge model onto one card
QLoRA is the answer when even LoRA’s frozen base will not fit, because those base weights in half precision are still the bulk of the memory. It quantizes the frozen model to 4 bits and trains the LoRA adapters on top in higher precision. The original work fine-tuned a 65-billion-parameter model on a single 48 GB GPU, which had not been possible before it.
Three pieces make that work, and interviewers ask about each. The 4-bit NormalFloat (NF4) format is built for weights that follow a roughly normal distribution, so its quantization levels land where the values actually cluster instead of spreading evenly. Double quantization compresses the quantization constants themselves, saving about another half a bit per parameter. Paged optimizers spill optimizer state to CPU memory when a long sequence causes a spike, so one awkward batch does not trigger an out-of-memory kill. The base stays 4-bit and frozen the whole time; gradients only ever flow into the small adapters, which live in bf16. That is the answer to “how does it train in 4-bit” — it does not. The precision that matters for the gradient is preserved exactly where the learning happens.
One consequence people miss: a plain LoRA adapter can be merged back into the base weights (W0 + BA), so inference carries zero extra latency. A QLoRA adapter cannot be folded cleanly into a 4-bit base without dequantizing first, so you either serve the adapter alongside the model or dequantize and merge into a 16-bit copy. If a candidate claims QLoRA hands you free inference, that is a good place to press.
| Approach | Trainable parameters (13B model) | Frozen base precision | Rough memory to train the 13B model | Adapter merges into base? | Reach for it when |
|---|---|---|---|---|---|
| Full fine-tuning | All ~13B | 16-bit | 100+ GB, multi-GPU (weights, gradients, and Adam optimizer state) | Not applicable, you update the weights directly | Large behavior change, lots of data, and the hardware budget to match |
| LoRA | Tens of millions (well under 1%) | 16-bit, frozen | ~30 GB, fits a single 40 GB card | Yes, fold BA into W0 for zero added inference latency | One or a few GPUs, adapting tone, format, or a task; you want a mergeable adapter |
| QLoRA | Tens of millions (same adapters as LoRA) | 4-bit NF4, frozen | ~10 GB, fits a consumer or single small card | Not directly, the 4-bit base must be dequantized first | The largest model that otherwise will not fit, or the tightest memory budget |
What breaks: catastrophic forgetting
You fine-tune a model on legal question answering and it sharpens on contracts while getting worse at writing Python. That is catastrophic forgetting: the update overwrites weights that supported capabilities you were not training. Because LoRA touches only a small low-rank slice of the weight space, it forgets less than full fine-tuning, but less is not none, especially at high rank or many epochs on a narrow dataset.
The interview version is usually a debugging story. Your task metric went up, a general benchmark went down, what do you do. Strong answers name the cause and the guardrail: keep a held-out set of general prompts and track it alongside the task metric, mix a slice of general instruction data into the training set so the model keeps rehearsing its old behavior, lower the learning rate, cut epochs, or drop the rank. A candidate who only reports the task metric climbing has not measured the thing that gets people paged after a deploy.
When the right answer is not to fine-tune
This is where interviews separate people who have shipped from people who have read the docs. Fine-tuning teaches a model form and behavior: a house style, a strict output schema, a domain’s vocabulary, a task it should perform without a page of instructions. It is a poor way to inject facts. If the product team wants the model to know this week’s documentation, fine-tuning bakes those facts into weights that go stale the moment the docs change, and it tends to make the model confidently wrong at the edges. Retrieval is the better tool there: put the current documents in a vector store and pull them into context at query time.
So the decision an interviewer wants to hear runs roughly like this. Try a better prompt before anything else. Reach for retrieval when the problem is missing or changing knowledge. Fine-tune when you have a stable task, a few hundred to a few thousand solid labeled examples, and a real constraint (latency, cost, or a behavior long prompts cannot reliably produce) that rules out the cheaper options. Often the production answer is retrieval and a small fine-tune together: the adapter fixes format and tone, retrieval supplies the facts.
Expect at least one question that sounds like a trap. “We have 50 labeled examples, should we fine-tune?” Fifty is few-shot territory; those go in the prompt. “The model needs to cite our internal policies word for word.” That is retrieval, and maybe structured decoding, not weights. Recognizing that the cheapest tool already clears the bar is the signal they are after.
Questions worth rehearsing out loud
- You have 2,000 labeled examples and one 40 GB GPU. Walk through fine-tuning a 13B model, and defend your rank and alpha.
- The fine-tuned model improved on our task but regressed on general reasoning. What happened, and how would you have caught it before shipping?
- QLoRA trains with a 4-bit base. Where in the pipeline does precision actually matter, and why does it still converge?
- Product wants the model to answer from our latest internal docs. Do you fine-tune? Make the case either way.
The pattern across all of it: the mechanics of LoRA are learnable in an afternoon, and interviewers know that. What they are testing is judgment about the parameters you cannot read off a config file, and about the cases where the impressive technique is the wrong one. Get the rank-and-alpha reasoning right, then be the candidate who can say “I would not fine-tune this at all,” and back it up.
