“How do you decide between RAG, fine-tuning, and prompt engineering?” is one of the most-asked AI/ML interview questions of 2026. Strong candidates have a decision framework. Weak candidates default to “well, it depends” and stall.
The three approaches
Prompt engineering
Craft the input to the model to get the desired output. No data plumbing, no training, just clever instructions and few-shot examples.
Retrieval-Augmented Generation (RAG)
Store documents in a vector database. At query time, retrieve relevant chunks and include them in the prompt. The model uses the retrieved context to answer.
Fine-tuning
Train the model further on your specific data, adjusting weights. Result: a model that specializes in your domain or style.
The decision tree
- Can you solve the problem with prompt engineering alone? Try this first.
- Does the model need access to specific information not in its training data? RAG.
- Does the model need to learn a specific style, format, or domain reasoning that prompts cannot teach? Fine-tuning.
- Combination of #2 and #3? RAG + fine-tuning.
When to choose prompting
- The model already has the knowledge needed
- You need fast iteration with no infrastructure investment
- The task is well-served by the base model’s capabilities
- Examples: writing assistant, code review helper, summarization, translation
When to choose RAG
- You have a corpus of documents (docs, support tickets, knowledge base) that the model needs
- The information changes over time
- You need source attribution
- Examples: enterprise knowledge bases, customer support agents, legal research, codebase Q&A
When to choose fine-tuning
- You need consistent style or format across many outputs
- You have task-specific reasoning that no amount of prompting can elicit
- You want to compress few-shot examples into model weights
- You need lower latency than RAG can provide
- Examples: domain-specific code generation, structured-output tasks, brand-voice writing
The hybrid approach
Most production systems combine all three:
- Fine-tuned model for the specific style/format
- RAG for fresh information
- Prompt templates that orchestrate the two
Cost and complexity
| Approach | Setup cost | Per-query cost | Maintenance |
|---|---|---|---|
| Prompt engineering | Low | Low | Low |
| RAG | Medium (vector DB, indexing) | Medium (retrieval + generation) | Medium (re-index on data changes) |
| Fine-tuning | High (training pipeline) | Lower (smaller prompts) | High (retrain as needs change) |
Interview red flags
- “I would always fine-tune” — over-engineering
- “I would never fine-tune; just use a bigger context” — under-engineering
- “RAG and fine-tuning are the same thing” — confused
- “Prompt engineering is hacky” — dismissive
The right answer to “which would you use for X?”
Walk through the decision tree out loud. Make the tradeoff explicit. Pick one. Justify. Acknowledge a backup approach.
Frequently Asked Questions
What about parameter-efficient fine-tuning (LoRA)?
LoRA and QLoRA are mainstream in 2026 — they make fine-tuning much cheaper. Mention them when fine-tuning comes up.
Is RAG getting replaced by long-context models?
Mostly no. Long context (1M+ tokens) helps but is still expensive per query. RAG remains more scalable for large knowledge bases.
How do I measure if my approach is working?
Build evals before you build the system. Without evals, you cannot tell if RAG is helping or hurting.