“How do you decide between RAG, fine-tuning, and prompt engineering?” is one of the most-asked AI/ML interview questions of 2026. Strong candidates have a decision framework. Weak candidates default to “well, it depends” and stall.
The three approaches
Prompt engineering
Craft the input to the model to get the desired output. No data plumbing, no training, just clever instructions and few-shot examples.
Retrieval-Augmented Generation (RAG)
Store documents in a vector database. At query time, retrieve relevant chunks and include them in the prompt. The model uses the retrieved context to answer.
Fine-tuning
Train the model further on your specific data, adjusting weights. Result: a model that specializes in your domain or style.
The decision tree
- Can you solve the problem with prompt engineering alone? Try this first.
- Does the model need access to specific information not in its training data? RAG.
- Does the model need to learn a specific style, format, or domain reasoning that prompts cannot teach? Fine-tuning.
- Combination of #2 and #3? RAG + fine-tuning.
When to choose prompting
- The model already has the knowledge needed. If a strong base model can answer from what it learned in pretraining, adding retrieval or training only buys you cost and new failure modes. Run the plain prompt first and see how far it gets before reaching for anything heavier.
- You need fast iteration with no infrastructure investment. You change behavior by editing text and re-running in seconds, with no pipeline to rebuild or index to warm up. That speed matters most early, when you are still figuring out what good output even looks like.
- The task is well-served by the base model’s capabilities. Generic work — summarizing, rewriting, pulling fields out of text — sits squarely in what the model already does well. Interviewers want to hear you reach for the cheapest tool that clears the bar, not the most impressive one.
- Examples: writing assistant, code review helper, summarization, translation. Each of these leans on general language ability rather than private data, so a well-structured prompt with a couple of examples usually gets you most of the way there.
When to choose RAG
- You have a corpus of documents (docs, support tickets, knowledge base) that the model needs. The model was never trained on your internal wiki or ticket history, so you retrieve the relevant pieces at query time and hand them to the model as context. Interviewers often probe how you chunk, embed, and rank those documents.
- The information changes over time. Retraining a model every time a price or policy shifts is impractical; with RAG you re-index the changed documents and answers update immediately. This is the single strongest argument for RAG over fine-tuning.
- You need source attribution. Because you know which chunks you retrieved, you can cite them — which matters for legal, medical, and support answers where “where did this come from?” is a required question, not a nice-to-have.
- Examples: enterprise knowledge bases, customer support agents, legal research, codebase Q&A. Each answers from a body of documents that is large, private, and changing — exactly the case retrieval handles better than stuffing everything into one prompt.
When to choose fine-tuning
- You need consistent style or format across many outputs. When every output must match a rigid schema or a specific tone, examples in the prompt drift; training the behavior into the weights makes it stick. Think JSON that never breaks its shape, or a support voice that stays on-brand across thousands of replies.
- You have task-specific reasoning that no amount of prompting can elicit. Some domains — specialized code, niche classification — need patterns the base model never saw, and fine-tuning teaches them where instructions cannot. Be ready to explain how you would collect and label that training data.
- You want to compress few-shot examples into model weights. If your prompt carries ten examples on every call, you pay for those tokens on every call; fine-tuning bakes the behavior in so the runtime prompt shrinks.
- You need lower latency than RAG can provide. Skipping the retrieval step and sending a shorter prompt means fewer tokens to process per request, which cuts response time — worth it on high-volume, latency-sensitive paths.
- Examples: domain-specific code generation, structured-output tasks, brand-voice writing. Each needs a learned behavior applied consistently, not fresh facts — that is the line separating fine-tuning from RAG.
The hybrid approach
Most production systems combine all three:
- Fine-tuned model for the specific style/format — the training handles how the system talks and structures answers, the part that should stay constant across every request.
- RAG for fresh information — retrieval supplies the facts that change, so you update the data without retraining the model.
- Prompt templates that orchestrate the two — the template decides what to retrieve, how to slot it into context, and how to instruct the fine-tuned model per request.
Cost and complexity
| Approach | Setup cost | Per-query cost | Maintenance |
|---|---|---|---|
| Prompt engineering | Low | Low | Low |
| RAG | Medium (vector DB, indexing) | Medium (retrieval + generation) | Medium (re-index on data changes) |
| Fine-tuning | High (training pipeline) | Lower (smaller prompts) | High (retrain as needs change) |
Interview red flags
- “I would always fine-tune” — over-engineering
- “I would never fine-tune; just use a bigger context” — under-engineering
- “RAG and fine-tuning are the same thing” — confused
- “Prompt engineering is hacky” — dismissive
The right answer to “which would you use for X?”
Walk through the decision tree out loud. Make the tradeoff explicit. Pick one. Justify. Acknowledge a backup approach.
Frequently Asked Questions
What about parameter-efficient fine-tuning (LoRA)?
LoRA and QLoRA are mainstream in 2026 — they make fine-tuning much cheaper. Mention them when fine-tuning comes up.
Is RAG getting replaced by long-context models?
Mostly no. Long context (1M+ tokens) helps but is still expensive per query. RAG remains more scalable for large knowledge bases.
How do I measure if my approach is working?
Build evals before you build the system. Without evals, you cannot tell if RAG is helping or hurting.
