What Is RAG? Retrieval-Augmented Generation Explained

Originally published on DevToolHub.

What is RAG, in one sentence? A way to make an LLM answer using documents it was never trained on — search those documents for relevant passages, hand the model the results as context, then generate. No retraining required.

What Is RAG?

A pattern, not a specific tool: retrieve relevant info for a query, generate using that info as context. The original 2020 paper from Facebook AI Research paired a pre-trained model with "a dense vector index" accessed by "a pre-trained neural retriever."

The pipeline

Chunk documents → embed each chunk → store the vectors → embed the incoming question → retrieve the closest chunks → generate an answer using them as context. Any LLM handles generation; retrieval is what makes RAG RAG.

Vector databases and local embeddings

A vector database indexes by geometric closeness, not exact values. Ollama's /api/embed endpoint runs this locally: {"model": "...", "input": "..."} in, an array of floats out. nomic-embed-text beats OpenAI's ada-002 and text-embedding-3-small on Ollama's own benchmarks, at 274MB.

More chunks isn't always better

A Stanford/Berkeley/Samaya AI study found LLM performance drops when relevant info sits in the middle of a long context — "lost in the middle." A tight set of well-ranked chunks beats a sloppy, larger one.

RAG vs fine-tuning

RAG adds knowledge without retraining — best for data that changes. Fine-tuning changes behavior and format by retraining weights. Different problems; many systems use both.

Mistakes to avoid

Chunking without testing retrieval. Assuming the retriever always finds something relevant. Skipping re-indexing after data changes. Reaching for RAG when the model's training data already covers it — see MCP-based tool access for when an agent needs live data instead of a static index.

Full article with FAQ: devtoolhub.com/what-is-rag