Design a RAG system

The single most important thing to get right about a RAG system is that it is a search engine with a language model at the end, and it fails like a search engine, not like a model. When an answer is wrong, the cause is usually that the right passage never reached the prompt, and no amount of generator quality can cite what retrieval never fetched. The name comes from Lewis et al. (2020), who paired a seq2seq generator with a dense vector index of Wikipedia so the model could read retrieved passages instead of relying on its weights, and the retriever that made the dense half practical is DPR, a dual-encoder that beat BM25 by 9 to 19 points of top-20 retrieval accuracy the same year. What production systems keep from those papers is the architecture, knowledge in an index you can update and inspect rather than in parameters you cannot.

Framed that way, the design has two jobs. The first is getting the right eight chunks out of ten million into the prompt inside a retrieval budget of tens of milliseconds, which is a ranking-funnel problem, cheap wide recall first and expensive precision last. The second is proving the answer came from those chunks, which is a grounding and measurement problem that runs from citation ids in the prompt to an evaluation gate that blocks regressing deploys. Chunkers, embedders, indexes, fusion, and rerankers all serve the first job, and the evaluation layer exists for the second.

The numbers in this article are measured, not guessed. The index and embedder figures come from my ann-bench project, which embedded 1.2 million MS MARCO passages, built five FAISS index types and scored them against exact ground truth, and swept six embedding models on BEIR datasets with human relevance judgments. Where a number comes from a paper or a vendor measurement, it is linked where it appears.

Scope and requirements

The concrete product to hold in mind is an internal knowledge assistant over a company's documents, the wikis, PDFs, design docs, runbooks, and support tickets that today get answered by interrupting a teammate. Call it two million source documents that chunk into roughly ten million retrievable passages, a few thousand questions a day, and a corpus that never sits still because people edit those documents all day. A user asks a question in plain language and gets a direct answer with citations pointing at the exact source passages, and when the corpus does not contain an answer the system says so instead of improvising one. That refusal behavior is a feature to design for, not an error state.

Four non-functional constraints shape everything downstream. Groundedness outranks fluency, meaning every claim in an answer must trace to a retrieved passage, because a confident wrong answer about an internal procedure costs more than no answer at all. Latency is a few seconds end to end, and since the generator will spend nearly all of it writing tokens, the entire retrieval stack has to fit in tens of milliseconds. Freshness is minutes, not days, so a page edited this morning must be retrievable before the next standup. And access control rides through retrieval, because a chunk a user cannot read at the source must never surface in their context, which makes permissions metadata on every chunk and a filter on every search. Multi-turn agentic research, fine-tuning the generator, and image or table understanding all sit below the line for this design.

The two pipelines

The system is two pipelines that meet at a pair of indexes. Ingestion is throughput-shaped batch work, it pulls sources, parses them, cuts chunks, embeds them, and writes indexes, and its unit of progress is documents per second. The query path is latency-shaped, it embeds one question, searches both indexes, fuses and reranks the candidates, and assembles a prompt, and its unit of progress is milliseconds. Keeping them separate is what lets a full overnight re-embed run without touching serving, and lets query replicas scale for a traffic spike without waking a single GPU embedding worker.

On the ingestion side, workers pull and parse sources, cut them into chunks with stable ids of the form {source}:{doc_id}:{chunk}, and write the text with its title, ACL, and updated timestamp into the chunk store. Changed chunks go onto an index queue, embedding workers drain it in GPU-sized batches and upsert vectors into the ANN index, and the same chunks upsert their terms into the BM25 inverted index. On the query side, the retrieval service runs the dense and lexical searches in parallel, fuses the two rankings, sends the survivors through the reranker, and the generator writes an answer from the assembled context and nothing else. The dashed return edge is the product, an answer whose every claim carries a citation back into the chunk store.

Two pipelines, one meeting point. Ingestion turns changed documents into vectors and terms, and the query path runs both retrievals in parallel, fuses by rank, reranks with a cross-encoder, and generates from the assembled context only.

Ingestion and chunking

Chunking exists because two budgets meet at the chunk. The embedding model must compress each chunk into one vector, and the prompt must later carry the retrieved chunks inside a token budget. Cut too large and one vector has to average several topics, so the chunk sits near everything and close to nothing, retrieval precision drops, and every hit drags thousands of tokens into the prompt. Cut too small and the chunk loses the context that made it meaningful, a lone table row or a sentence like "this flag is off by default" retrieves fine and answers nothing. Fixed-size windows of about 512 tokens with 10 to 15 percent overlap are the unexciting default that works, the size matches the 512-token cap of the common small embedders, and the overlap keeps a sentence that straddles a boundary alive in both neighbors.

Structure-aware splitting cuts on the boundaries authors already drew, headings, paragraphs, list items, and code fences, and produces variable-sized chunks that read as complete thoughts. Parent-child retrieval, sometimes called small-to-big, embeds and matches on small chunks for precision but hands the generator the enclosing section, which exploits the fact that the retrieval unit and the generation unit do not have to be the same text. Semantic chunking breaks where the embedding similarity between adjacent windows dips, which costs an extra embedding pass at ingestion and is worth benchmarking rather than assuming. The non-starter is one chunk per document, because a single vector for a forty-page runbook matches nothing specific, and the prompt cannot afford the payload even when it does match.

The best measured upgrade to any of these is contextual retrieval, which Anthropic documented in 2024. An LLM writes a sentence or two situating each chunk in its source document, the context is prepended before embedding and indexing, and in their measurements the rate at which the top 20 retrieved chunks missed the needed evidence fell from 5.7 percent to 3.7 percent with contextual embeddings alone, a 35 percent cut, to 2.9 percent when contextual BM25 joined, and to 1.9 percent with a reranker on top, a 67 percent cut overall. The price is one LLM call per chunk, paid at ingestion, and ingestion is the right place to pay it, because ingestion runs once per document edit while the query path runs on every question.

The embedding model sets the ceiling

No index can return a passage the embedding model failed to place near the query, so the model is chosen first and everything downstream inherits its ceiling. Three families cover the practical menu. The E5 family comes from weakly supervised contrastive pretraining on a curated large-scale corpus of text pairs, was the first model to beat BM25 zero-shot on BEIR without labeled data, and expects its query: and passage: prefixes. The BGE family from BAAI spans bge-small, bge-base, and bge-large in English, and bge-m3 emits dense vectors, sparse lexical weights, and multi-vector representations from one model, covers more than 100 languages, and reads 8,192 tokens. Qwen3-Embedding ships at 0.6B, 4B, and 8B parameters with 1,024, 2,560, and 4,096 dimensions, reads 32K tokens, takes task instructions in the prompt, and its 8B model led the MTEB multilingual leaderboard at a 70.58 mean score as of June 2025.

Leaderboards do not settle the choice, which is why I benchmarked six embedders on two BEIR datasets with exact, fully embedded evaluation in ann-bench. The winner on both SciFact and NFCorpus was bge-base-en-v1.5, a 110M-parameter, 768-dimension model, at 0.740 and 0.374 nDCG@10. The largest model in the sweep, Qwen3-Embedding-0.6B at 1,024 dimensions, finished fourth of six on SciFact and second on NFCorpus, ran dead last on speed at 114 documents per second, and on SciFact it lost to gte-small, a 384-dimension model that embeds more than twelve times faster. Two lessons travel. Bigger is not better in embedding space, and each model must be scored with its own prompt convention, because E5 without its prefixes or Qwen3 without its instruction quietly bleeds points and the harness never errors.

Dimension is a tax you pay forever. Ten million chunks at 768 dimensions in float32 is about 31 GB of raw vectors before any index overhead, and the same corpus at 384 dimensions is about 15 GB, which flows straight into index memory, search cost, and the size of every future rebuild. The best 384-dimension model in my sweep, gte-small, trailed bge-base by 1.4 nDCG points on SciFact and 2.5 on NFCorpus, and there are corpora where that trade is the right one. Matryoshka models like Qwen3-Embedding make the dial explicit by letting you truncate dimensions at a measured quality cost.

The vector index, recall against latency

Exact search states the problem plainly. In my benchmark over 1.2 million MS MARCO passages embedded at 1,024 dimensions, a flat exact scan answered in 241 ms at p50, which is 4 queries per second, correct and unusable inside a tens-of-milliseconds budget. HNSW builds a layered graph where each vector links to near neighbors and a query greedily descends toward its target, with a candidate-list width called efSearch trading recall for speed. On the same corpus, scored against exact ground truth, HNSW at efSearch 256 returned 99.1 percent of the exact top ten in 1.9 ms at 496 queries per second, a 126-fold speedup bought with 0.9 points of recall, and dropping efSearch to 16 pushed the same index to 0.23 ms and 4,257 queries per second at 95.6 percent recall, the full three orders of magnitude for 4.4 points. The whole sweep, with build times and memory, is in the ann-bench write-up.

The rest of the index menu trades memory. IVF-SQ8 reached 0.979 recall in 1.2 GB where HNSW used 5.2 GB, though it needed 32 ms at the nprobe setting that got it there, and IVF-PQ compressed the same 1.2 million vectors into 0.10 GB with recall capped at 0.648, which is the shape of the deal you accept when vectors reach the billions. One reading discipline keeps these numbers honest. ANN recall measures agreement with exact search under the same embedder, a statement about the index. It is not the retrieval-quality recall scored against human judgments, which is a statement about the model, and a benchmark that blends the two cannot tell you which one broke.

The index has several workable homes. pgvector puts an HNSW index inside Postgres, which at a few million chunks means one fewer system to run and ACL filtering in plain SQL next to the vectors. A managed vector database buys the operations you do not want to own. An in-process FAISS or hnswlib index gives the sub-millisecond path with no network hop, at the price of owning shard and rebuild logic yourself. The non-starters are running the flat exact scan in production, which the 4 QPS number closes on its own, and shipping index parameters you never validated against your own exact ground truth, because the recall curve moves with dimension, metric, and data.

Hybrid retrieval, BM25 plus dense

Dense retrieval matches meaning, so a question phrased nothing like the doc still lands, and that is the capability DPR demonstrated by beating BM25 by 9 to 19 points of top-20 accuracy on in-domain question answering. The catch arrived with BEIR, which evaluated retrievers zero-shot across 18 datasets and found BM25 a robust baseline that dense models often underperform once they leave their training distribution. An internal corpus full of error codes, config keys, service names, and version strings is exactly that out-of-distribution case, and those exact identifiers are what embeddings blur and an inverted index matches literally. So the design runs both legs, BM25 top-100 and dense top-100 in parallel, and Anthropic's contextual-retrieval measurements found the combination beating embeddings alone even before any reranker.

The two result lists cannot be merged by score, because BM25 scores are unbounded and cosine similarities live in a narrow band, and any weighted sum is a fragile calibration exercise. Reciprocal rank fusion merges by rank instead. Each list contributes 1 / (k + rank) per document, with k defaulting to 60 in Elasticsearch's implementation, following the 2009 SIGIR paper that introduced the method. The large k flattens the difference between rank 1 and rank 5, so a chunk both legs liked beats a chunk one leg loved and the other never saw, which is the behavior you want from evidence.

A worthwhile variant collapses the two legs into one model, since bge-m3 emits a dense vector and sparse lexical weights from the same forward pass, one encoder feeding two index types. The non-starter is normalizing and adding raw scores across legs, which reintroduces the calibration problem RRF exists to avoid and silently lets one leg drown the other whenever its score distribution shifts.

from collections import defaultdict

def rrf(rankings, k=60):
    """Fuse ranked lists of chunk ids. Rank-based, so BM25 and
    cosine scores never need to share a scale."""
    scores = defaultdict(float)
    for ranked in rankings:
        for rank, chunk_id in enumerate(ranked, start=1):
            scores[chunk_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

# each leg returns its top 100, the reranker sees the fused top 50
candidates = rrf([bm25_top100, dense_top100])[:50]

The reranker, a cross-encoder that earns its cost

Everything to this point used bi-encoders, which embed the query and the chunk independently and meet only at a dot product. That independence is the entire reason ANN search works, the same argument that makes two-tower retrieval possible, because every chunk vector can be computed before any query exists. It is also a cap on accuracy, since all interaction between question and passage is squeezed through two fixed vectors. A cross-encoder removes the cap by reading the query and the chunk together, attending across the pair, and outputting a relevance score directly. bge-reranker-v2-m3 is the working example here, a multilingual cross-encoder built on bge-m3 that scores a query-passage pair, maps it through a sigmoid to a value in [0, 1], and is intended for reranking the top of a retrieved list.

Nothing about a cross-encoder can be precomputed, every score depends on the specific pair, so the cost is one model forward pass per candidate per query. Run it over the ten-million-chunk corpus and that is ten million forward passes per question, which is the non-starter stated with its own arithmetic. Run it after hybrid retrieval and it sees 50 candidates, and its job is precision at the very top, so context assembly can keep 8 chunks with confidence instead of hedging with 30. The measured value is real, in Anthropic's numbers reranking took contextual hybrid retrieval from a 2.9 percent top-20 failure rate to 1.9 percent.

Context assembly and citation grounding

The chunks that survive reranking become the prompt, and assembly is where retrieval quality is either preserved or squandered. Overlapping windows from the same passage arrive as near-duplicates, so near-identical chunks are collapsed, chunks from one document are grouped under one source header, and each source carries a stable bracketed id, its title, and its updated date. The instruction block pins the contract, answer only from the sources, cite every claim by id, and say plainly when the sources do not contain the answer. Refusal is the promise the product made in the requirements, grounded or silent, and it has to be written into the prompt as an allowed outcome or the model will fill silence with fluency.

Citations are not presentation polish, they are the verification hook for the whole system. Because every claim names its chunk, a sampled checker can test each cited sentence against the chunk it cites, which is exactly the faithfulness measurement the evaluation section formalizes, and the UI can link every answer back to the living document, which is where user trust actually comes from. An uncited answer in this design is not a style violation, it is unverifiable output, and the evaluation layer treats it that way.

def build_prompt(question, chunks):
    # chunks arrive reranked, deduplicated, grouped by source
    sources = "\n\n".join(
        f"[{i}] {c.doc_title} (updated {c.updated})\n{c.text}"
        for i, c in enumerate(chunks, start=1)
    )
    return (
        "Answer using ONLY the numbered sources below.\n"
        "Cite every claim with its source id, like [2].\n"
        "If the sources do not contain the answer, say so\n"
        "instead of guessing.\n\n"
        f"Sources:\n{sources}\n\nQuestion: {question}"
    )

Evaluation, in two layers

Evaluation splits the same way the system does, retrieval scored against judged chunks and generation scored against the retrieved context, never one blended number, because the two halves fail for different reasons and a blended score cannot tell you which half broke. The retrieval layer runs on a golden set, a few hundred real user questions with hand-judged relevant chunks, replayed on every change. Recall@k asks whether the evidence entered the candidate set at all, and it is the first number to look at when answers go wrong. nDCG@10 rewards putting the most relevant chunks at the top under graded judgments, the metric BEIR and MTEB rank on. MRR tracks where the first relevant chunk lands, the right lens when one passage answers the question. Public anchors help calibrate a new harness, on the MS MARCO passage dev set BM25 sits near 0.187 MRR@10 while small dense embedders of the E5 and BGE class land around 0.33 to 0.42.

The generation layer is where RAGAS earns its citation, a reference-free framework whose metrics are scored by an LLM judge rather than against ground-truth answers. Faithfulness measures whether the claims in the answer are supported by the retrieved context, response relevancy measures whether the answer addresses the question, and context precision and context recall grade the retrieved set itself. Reference-free is the practical part, since nobody hand-labels answers for a private corpus, and the caution is the same as for any LLM judge, the scores are noisy and drift with the judge model, so they serve as trend and regression signals backed by a periodically audited human sample, not as absolute truth.

Both layers wire into one release gate. Any change, a new embedder, a chunk-size change, a prompt edit, a reranker swap, replays the golden set and a sampled slice of production traffic, and the gate blocks the deploy when retrieval or faithfulness regresses. The failure this catches is the quiet one, a prompt tweak that lifts fluency while faithfulness slides, which no eyeballed demo will ever notice.

Retrieval metrics replay a judged golden set, generation metrics run RAGAS over sampled production answers, and both feed one gate that blocks a regressing deploy before users meet it.

Freshness and incremental indexing

The argument for RAG over fine-tuning is that knowledge lives in an index you can update in minutes, so freshness is not a nice-to-have, it is the point of the architecture. Fine-tuning as a freshness mechanism is a non-starter twice over, a training run per documentation edit at GPU prices, and no citation trail at the end of it. The machinery is a manifest, the same pattern that gives ann-bench its resumability. Every chunk carries a content hash, ingestion recomputes hashes on every pull, and only chunks whose hash changed are re-embedded and upserted, so a one-line edit to one page costs a handful of embeddings rather than a corpus pass. Deletions write tombstones, and the updated timestamp on every chunk lets retrieval prefer the fresh version whenever an old copy is still in flight.

The two indexes age differently. The BM25 inverted index absorbs incremental updates as ordinary segment writes. HNSW accepts inserts happily but degrades under deletion, because removing nodes tears holes in the graph its search paths rely on, so deletes are tombstoned and filtered at query time, and when tombstones pass a threshold a scheduled job rebuilds the index cleanly and swaps it behind an alias, so serving never reads a half-built structure. The steady state is two cadences, intra-day upserts measured in minutes for edits, and a periodic compacting rebuild that also picks up embedder upgrades, since a new embedding model changes the geometry and mixing vectors from two models in one index returns plausible nonsense.

The latency and cost budget

Walk one question with the clock running. Embedding the query with a small model costs a few milliseconds. The ANN lookup measured 0.23 ms at the fast HNSW setting and 1.9 ms at the high-recall one, and BM25 runs in the same parallel window, so the whole candidate stage is single-digit milliseconds. The reranker spends one forward pass per candidate over 50 pairs, the only retrieval-side cost worth watching, and then generation takes over and dominates everything, hundreds of milliseconds to seconds of token streaming. The asymmetry is the design lesson. Almost any retrieval-quality investment, a second retrieval leg, rank fusion, a reranker, is close to free next to the generator, so the answer to whether you can afford hybrid plus reranking is almost always yes.

Money divides by pipeline. Ingestion pays embedding once per corpus and once per edit, and the measured throughputs price it directly, ten million chunks at bge-base's 320 documents per second is about 8.7 GPU-hours, and at gte-small's 1,675 documents per second about 1.7. Contextual retrieval adds an LLM call per chunk, again priced at ingestion. The query path pays in prompt tokens on every single question, eight chunks of roughly 512 tokens is about 4,000 prompt tokens, while hedging with 30 chunks is over 15,000, nearly four times the per-question cost, paid forever. That is the second thing the reranker buys, because the precision that makes an 8-chunk context safe is also the discipline that keeps the token bill flat.

Questions and answers

The core ideas as questions with the answers given outright. Each wrong multiple-choice option is marked with why it is wrong, and the ordering ones show the correct sequence.

References

1. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), The paper that named RAG, pairing a seq2seq generator with a dense Wikipedia index so the model reads retrieved passages instead of relying on its weights.
2. Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering (2020), The dual-encoder dense retriever, 9 to 19 points over BM25 on in-domain top-20 retrieval accuracy.
3. Thakur et al., BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (NeurIPS 2021), 18-dataset zero-shot retrieval benchmark where BM25 holds up as a robust baseline that dense models often underperform out of domain.
4. Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023), Reference-free, LLM-judged evaluation of RAG pipelines, the source of the faithfulness framing used here.
5. RAGAS documentation, available metrics, The metric catalog, faithfulness, response relevancy, context precision, context recall, and noise sensitivity.
6. Elasticsearch, Reciprocal rank fusion, The 1/(k + rank) formula with a default k of 60, following the 2009 SIGIR paper it cites.
7. Anthropic, Introducing Contextual Retrieval, Measured top-20 retrieval failure cuts of 35, 49, and 67 percent from contextual embeddings, contextual BM25, and reranking.
8. Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (2016), The graph index behind the 0.991 recall at 1.9 ms measurement in ann-bench.