# What is RAG? Retrieval-Augmented Generation vs Fine-tuning

Source: https://www.techinterview.org/post/3233460067/what-is-rag-retrieval-augmented-generation-vs-fine-tuning/
Updated: 2026-04-16 · techinterview.org

Retrieval-Augmented Generation (RAG) is one of the most widely deployed LLM patterns in production. Understanding when to use RAG versus fine-tuning — and what each one actually does well — is a core expectation in AI/ML system design interviews in 2026.

## What the Interviewer Is Testing

Can you reason about the trade-offs between RAG and fine-tuning given a specific use case? Do you know the components of a RAG pipeline well enough to design one? Can you identify when both are needed, when neither is enough, and when plain prompt engineering suffices?

## The Problem RAG Solves

LLMs have two fundamental limitations for enterprise use:

- **Knowledge cutoff:** The model only knows what was in its training data. Events after the cutoff, proprietary company knowledge, internal documentation — none of it is accessible.

- **Context window:** Even a 128K token context can't hold an entire knowledge base of millions of documents.

Fine-tuning addresses neither cleanly. You can fine-tune on internal documents, but the model will "memorize" that knowledge imperfectly, may hallucinate by blending learned facts incorrectly, and requires retraining every time documents change. RAG is different: it retrieves relevant documents at inference time and provides them to the model as context — fresh, accurate, and auditable.

## RAG Architecture


```
Offline (indexing):
  Documents → Chunker → Embedding Model → Vector DB

Online (inference):
  User Query
      → Embedding Model (same model as indexing)
      → Vector DB (ANN search: find top-K relevant chunks)
      → Retrieved chunks + original query → LLM
      → Response (with optional citations)
```


## Chunking Strategies

Documents must be split into chunks before embedding. The chunk is the retrieval unit — the model receives the top-K chunks as context. Chunk size is a fundamental trade-off:

- **Too small (50 tokens):** Each chunk lacks context. "The limit is 100" — 100 what? Retrieved chunks are confusing without surrounding text.

- **Too large (2,000 tokens):** The embedding averages over too many concepts. Retrieval recall drops because the embedding isn't specific enough.

- **Sweet spot:** 200–500 tokens with overlap (50–100 token overlap between adjacent chunks to avoid cutting mid-sentence).

Chunking strategies:


```
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=80,
    separators=["nn", "n", ". ", " ", ""],  # prefer paragraph breaks
)
chunks = splitter.split_documents(documents)
```


**Semantic chunking:** Split at natural topic boundaries rather than fixed token counts — detect when sentence embeddings shift significantly. Higher quality, more complex to implement. Worth it for long, heterogeneous documents.

## Embedding Models

The embedding model converts text to a dense vector. The same model must be used for both indexing and query embedding — otherwise the vector spaces won't align.

| Model | Dimensions | Cost | Best for |
| --- | --- | --- | --- |
| OpenAI text-embedding-3-large | 3072 (truncatable) | $0.13/M tokens (API) | Quality-first, already on OpenAI stack |
| OpenAI text-embedding-3-small | 1536 | $0.02/M tokens | Cost-sensitive, near text-ada quality |
| BGE-M3 (BAAI) | 1024 | Free (self-hosted) | Multilingual, best open-source option |
| E5-large-v2 | 1024 | Free (self-hosted) | Strong English performance, lightweight |
| Cohere Embed v3 | 1024 | $0.10/M tokens | Strong on domain-specific retrieval |

## Vector Databases

The vector DB stores embeddings and supports approximate nearest neighbor (ANN) search — finding the K vectors most similar to the query vector by cosine or dot product similarity.

| Database | Deployment | Best for |
| --- | --- | --- |
| Pinecone | Managed cloud | Production with minimal ops overhead |
| Weaviate | Self-hosted / cloud | Hybrid search (vector + BM25 keyword) |
| Qdrant | Self-hosted / cloud | High-performance, Rust-based, filtering |
| pgvector | PostgreSQL extension | Already on Postgres, smaller scale (10K documents) | RAG — can't fit in context or fine-tune on everything |
| Need consistent output format or tone | Fine-tuning — RAG doesn't change model behavior |
| Proprietary domain language / jargon | Fine-tuning or continued pretraining |
| Latency-critical (