Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) is an architecture pattern that connects a language model to external knowledge at inference time. Instead of relying solely on what the model learned during training, you retrieve relevant documents from your own data and include them in the prompt. The model generates its answer grounded in that retrieved context.

The pattern exists because language models have a fixed knowledge cutoff, hallucinate when they lack information, and know nothing about your proprietary data. RAG addresses all three by giving the model access to current, authoritative, domain-specific content exactly when it needs it.

The core idea

A RAG system has two stages that run on every request:

Retrieve. Given the user's question, search a knowledge base for the most relevant documents or passages.
Generate. Pass the retrieved content into the prompt alongside the question. The model generates an answer grounded in that context.

The model never sees your entire corpus at once — only the pieces the retrieval system selects as relevant. This keeps the response grounded in specific sources you control, and it fits within the model's finite context window.

Why RAG instead of alternatives

There are several ways to give a model access to custom knowledge:

Prompt stuffing — paste documents into the prompt manually. Works for small, static content. Does not scale.
Fine-tuning — train the model on your data. Expensive, slow to update, and the model may still hallucinate rather than cite.
RAG — retrieve relevant content at query time and include it in the prompt. Scales to large corpora, reflects current data, and the model can cite its sources.

RAG wins when you need current information, your corpus changes frequently, you want traceable answers with citations, or the knowledge base is too large to fit in a single prompt.

Fine-tuning wins when you need the model to adopt a persistent style, learn a task it cannot perform from instructions alone, or handle domains where retrieval latency is unacceptable.

The retrieval pipeline

Retrieval is the hard part. The quality of the generated answer depends entirely on whether the right documents reach the model. A typical pipeline:

Indexing (offline). Prepare your documents for search:

Chunk large documents into passages sized for the context window (typically 256–1024 tokens per chunk).
Generate vector embeddings for each chunk using an embedding model.
Store chunks and their vectors in a search index (vector database or hybrid search engine).

Query processing (online). When a user asks a question:

Convert the question into a vector embedding using the same embedding model.
Run a similarity search against the index to find the closest chunks.
Optionally combine with keyword search (hybrid search) for better recall.
Re-rank results by semantic relevance.
Return the top-k most relevant chunks.

Prompt assembly. Place the retrieved chunks into the prompt as context, followed by the user's question and instructions for how to use the context.

Chunking and embeddings

Chunking strategy directly affects retrieval quality. Chunks too large dilute relevance with unrelated content. Chunks too small lose necessary context. Common strategies: fixed token windows with overlap, splitting on section boundaries, or recursive splitting that respects document structure.

Embeddings are dense vector representations of text. Two passages about the same concept land near each other in vector space, even when they use different words. This is why vector search finds conceptually relevant content that keyword search would miss.

Hybrid search and re-ranking

Neither keyword search nor vector search alone is optimal:

Keyword search excels at exact-term matching (product names, error codes, identifiers) but misses paraphrases.
Vector search captures semantic similarity but can miss exact terms and sometimes returns tangentially related content.

Hybrid search runs both in parallel and merges results. A re-ranker (often a cross-encoder model) then scores each result against the original query for final relevance ordering. This combination consistently outperforms either approach alone.

Challenges

Relevance. If retrieval returns the wrong chunks, the model generates wrong answers confidently — garbage in, garbage out.
Token budget. The context window is finite. You cannot retrieve everything. Selecting the right k (number of chunks) is a tuning problem.
Latency. Retrieval adds time. Vector search is fast, but re-ranking and embedding generation add milliseconds that compound.
Security. Opening private documents to a model requires access control. Users must only retrieve content they are authorized to see.
Freshness. Your index must stay current. Stale embeddings return stale answers.
Evaluation. Measuring RAG quality requires testing both retrieval precision (did we find the right chunks?) and generation accuracy (did the model use them correctly?).

Where RAG fits

RAG is not a product you install. It is an architecture pattern you build from components: an embedding model, a search index, a chunking strategy, a language model, and orchestration logic that wires them together. Every major cloud provider offers managed services for pieces of this pipeline, but the design decisions — how to chunk, what to index, how many results to retrieve, how to prompt — remain yours.

The pattern scales from a simple prototype (embed a few PDFs, search with cosine similarity, stuff results into a prompt) to production systems with hybrid search, re-ranking, access control, caching, and evaluation pipelines.