RAG
Also called Retrieval-augmented generation
Retrieval-augmented generation: searching a corpus for the passages most relevant to a question and putting them in the prompt, so the model answers from real documents instead of memory. It is how a tool answers questions about a codebase or wiki far larger than any context window.
At a glance
- Stands for
- Retrieval-augmented generation
- Named in
- Lewis et al., 2020
- Pipeline
- Chunk, embed, retrieve, rerank, generate
- Changes model weights
- No
- Common alternative
- Agentic search with grep and file reads
How it works
Ahead of time, documents are split into chunks, each chunk becomes an embedding (a vector that captures its meaning), and the vectors go into an index. At question time the question is embedded the same way, the nearest chunks are fetched, often combined with keyword search and reranked, and the top few are placed in the prompt with an instruction to answer from them and cite them.
Every stage can lose the answer. Chunks cut mid-function lose their meaning, an embedding model tuned for prose may rank code poorly, too few chunks miss the fact, and too many bury it.
Why it matters
RAG is the standard way to ground a model in material it was not trained on: internal docs, a support knowledge base, last week's design decisions. If you are building a feature that answers questions over user data, you are probably building RAG, and retrieval quality will decide the feature's quality more than the choice of model will. It is also a cost control, since three relevant pages are cheaper than a whole manual on every request.
It does not eliminate hallucination. A model can still blend retrieved text with memory, or answer confidently when retrieval returned nothing useful, so ask for citations and check them.
Before tuning the model or the prompt, build a small set of real questions with known answers and measure whether the right passage appears in the retrieved set at all. If it does not, no amount of prompt work downstream will fix the answer.
RAG versus agentic search
Coding agents mostly do not use a precomputed vector index of your repository. They search on demand with grep, file listing and reading, following imports the way a developer would. That is always current, needs no indexing, and handles exact identifiers well. Embedding-based RAG wins on large prose corpora where the question's wording does not appear in the answer. Many systems combine both.
Common mistakes
- Treating a vector database as a finished RAG system without evaluating what it retrieves for real questions.
- Chunking source code by fixed character counts so functions are split across chunks.
- Assuming answers are grounded because retrieval ran, without checking the cited passages.
Related terms
Frequently asked questions
Is RAG the same as fine-tuning?
No. Fine-tuning changes the model's weights through more training, while RAG leaves the model unchanged and supplies relevant text in the prompt at request time. RAG is easier to update, since you only change the documents.
Do large context windows make RAG obsolete?
Not for large or changing corpora. Long contexts are slower and costlier, and models attend less reliably to facts buried in very long inputs, so selecting the right passages still helps.