Benchmarking vector search, from the embedding model to the index

Every retrieval-augmented system rests on one operation that runs billions of times. You turn a piece of text into a vector, and then you find the handful of stored vectors that sit nearest to a query vector. The quality of an answer is capped by whether that search found the right passage, and the cost of the product is set by how fast and how cheaply the search runs at scale. So the two numbers that matter most in a retrieval system live at opposite ends of the same step, and they usually get measured by two different teams, when they get measured at all.

The purpose of this project is to measure both ends honestly, with one codebase, on real data. It embeds a text corpus, builds five different quantized approximate-nearest-neighbour indexes over it, and reports recall, latency, and memory for each of them. Next to that, it runs a head-to-head of six embedding models scored against human relevance judgments. A second purpose shaped the design even more than the first, which was to keep the two questions strictly apart. Retrieval quality asks whether the embedding model put the right document near the top, and it is graded against official human judgments. ANN fidelity asks whether the approximate index returned what an exact search would have, and it is graded against a brute-force ground truth the project computes for itself. These are different questions. They fail for different reasons, and a benchmark that blends them into a single number is worse than no benchmark at all.

The same code runs in three tiers, and a config file picks which one. A CPU tier runs over a small synthetic corpus and finishes in seventy-five seconds, so the whole pipeline can run anywhere. A GPU tier runs on my desktop's RTX 4090 and embeds more than a million real MS MARCO passages through vLLM. A cloud tier is built for a Lambda instance and hundreds of millions to a billion chunks. The reusable half is a library, and the benchmark is an application on top of it, so the next retrieval project can reuse the platform without changing it. The code is at github.com/gradientsj/rag-platform. This page walks through the decisions first and the measured results after, because the decisions are what make the numbers mean something.

The pipeline at a glance

Six stages make up the pipeline, and each one is a command that takes only the config and the output of the stage before it. Any single stage can be re-run on its own, and the whole thing resumes after a crash instead of starting over.

ingest ──▶ embed ──▶ groundtruth ──▶ build-index ──▶ bench ──▶ report
  │          │            │               │            │          │
stream a   Ray Data     exact brute-    Flat, HNSW,   recall +   markdown +
corpus,    over a GPU   force top-k     SQ8, IVF-PQ,  latency    matplotlib
chunk it,  actor pool   with faiss,     RaBitQ, each  sweep      from the
write      into FP16    saved as        with a build  into a     run
parquet    vectors      qrels           record        SQLite DB  database

The manifest is what makes that resumability work. It records every shard, its content hash, and whether its vectors have been written, so a run that gets preempted re-reads the manifest and skips the work it already finished. Ray gives fault tolerance inside a run, and the manifest gives it across runs. At the scale this is built for, both of those matter.

Step 1: three tiers, one config, no forked code

The easy mistake in a project like this is to write a quick local script for yourself and a separate real pipeline for the cloud, and then watch the two drift apart until only one of them still works. Instead the whole thing is a single typed configuration that covers the corpus sources, the chunking, the embedder, the index specifications, and the sweep grids. Three profiles differ only in their values. The CPU profile points at a synthetic corpus and a 384-dimension MiniLM model. The GPU profile points at MS MARCO and a 1024-dimension Qwen3 embedder served by vLLM. The cloud profile stacks Wikipedia and web text and writes its output to S3. Nothing in the code hardcodes a path, a model, a dimension, or a GPU count, which is what lets the seventy-five-second CPU run and the multi-hour GPU run exercise the same lines of code.

One small part of this earns its keep. Before the GPU tier loads a model, it reads the card's free memory and stops with a clear message and a suggested smaller model if the embedder will not fit, instead of dying halfway through a run with an out-of-memory error. The check reads the GPU without importing torch, so the config layer stays cheap to load and works on a machine with no GPU at all.

Step 2: streaming ingestion, and a manifest you can resume from

Corpora at this scale do not fit in memory, and once billion-scale vectors enter the picture the disk is not much better, so the ingestion streams. Each source yields documents one at a time, a chunker cuts them into overlapping token windows, and fixed-row parquet shards are written as they fill. Every source sits behind one interface, whether it is a synthetic generator, MS MARCO through ir_datasets, or Wikipedia and FineWeb streamed from the Hugging Face hub, so the profiles combine them without the ingestion code knowing which is which. Document ids are made globally unique in the form {source}:{native_id}:{chunk}, and the per-source counts and licenses go into the manifest, because at a billion chunks the question of which documents you are even allowed to ship is a real one with a real answer.

The resumability is unit-tested by killing a run partway through and checking that the next run embeds exactly the shards the first one missed. That test caught more than one ordering bug, and it is the difference between a preemption on a rented GPU costing a few minutes and costing the whole run.

Step 3: the embedding model is the ceiling, so measure it first

No index can retrieve a document that the embedding model failed to place near the query. The model sets the ceiling, and the index only decides how cheaply you get close to it. So before comparing any indexes, I ran six embedding models head-to-head on two small benchmarks from BEIR, SciFact and NFCorpus. They are small enough that every document gets embedded and the retrieval-quality numbers are exact, with no sampling. Each model was given its own documented retrieval prompt, because scoring them all with a shared prompt would quietly hurt the models that expect their own. BGE uses "Represent this sentence...", E5 uses "query:" and "passage:", and Qwen3 uses an instruction prefix.

SciFact, with three hundred scientific-claim queries over five thousand abstracts:

NFCorpus, with three hundred medical queries that have many relevant documents each. It is a harder set, and every model scores lower on it:

Three findings in those tables were worth the whole exercise. The first is that bigger is not better. Qwen3-Embedding has 0.6 billion parameters and 1024 dimensions, which makes it the largest model in the sweep and by far the slowest at 114 documents per second, and it lands in the middle of the pack on both datasets. On SciFact it loses to gte-small, a 384-dimension model that runs fifteen times faster. The second finding is that the sweet spot is small. bge-base has 110 million parameters, it wins both benchmarks, and it embeds fast enough to make a billion-chunk corpus tractable, while the 384-dimension models trail it by only two to four points at a quarter of the vector storage. The third finding is a check on the harness itself. These scores land within a point or two of the published BEIR numbers for the same models, which is the evidence that the measurement can be trusted before any of it gets used to make a decision.

One caveat belongs next to those tables. These are two small English benchmarks, chosen because they can be fully embedded and they carry trustworthy judgments, not because they settle the question for every domain. The ranking is a good starting point and a reusable harness, not a universal verdict.

Step 4: exact ground truth, and why recall needs two definitions

An approximate index is only worth measuring against the truth, so the project computes that truth for itself. For a held-out set of queries it runs an exact brute-force top-100 search over the full corpus, sharded across the GPU with faiss and merged back on the driver with a heap, and it saves the result as qrels. Every recall number for an ANN index is scored against this exact result, and never against another approximate index, because that would only measure how two approximations agree with each other.

This is where keeping the two eval tracks separate earns its place, because the word "recall" means two different things depending on the track. In the retrieval-quality track, recall@100 is the fraction of the documents a human judged relevant that show up in the top hundred, which is a statement about the model. In the ANN-fidelity track, recall@10 is the fraction of the exact top ten that the approximate index also returned, which is a statement about the index. Same word, different denominator, different question. The run database keeps them in separate tables and the report keeps them in separate sections, and that discipline is the difference between a benchmark you can reason about and a pile of numbers you cannot.

Step 5: five FAISS indexes, and what each one trades away

Every index type is a different point on the triangle of recall, speed, and memory, and you only ever get to pick two of the three. The project builds five of them, from exact to heavily compressed, so the trade-off is measured instead of assumed.

• Flat stores every vector uncompressed and scans all of them for each query. It is exact, because it is the ground truth, but at a million vectors of 1024 dimensions that scan takes a quarter of a second per query. It is the honest baseline the others are measured against.
• SQ8 is still a full scan, but each float is compressed from four bytes to one by mapping every dimension onto 256 levels. That makes it four times smaller for a fraction of a point of recall.
• HNSW builds a navigable graph where each vector links to its nearest neighbours, and a query hops toward its target while keeping a candidate list whose width, efSearch, trades recall for speed. It is the fastest index here by three orders of magnitude, and it pays for that by storing the full vectors plus the graph.
• IVF-PQ splits the space into cells with k-means, scans only the nprobe nearest cells, and inside them replaces each vector with a product-quantized code. The 1024-dimension vector becomes 64 sub-vectors, each stored as a single byte, which is a sixty-four-fold compression. It fit a million vectors into a hundred megabytes, and it is where a real bug lived, described further down.
• RaBitQ is a newer one-bit-per-dimension quantizer with a provable error bound. It produced the smallest index of all, at a recall the write-up reports instead of hiding.

Every build path trains its coarse quantizer and codebooks on the GPU when a GPU is present, then moves the finished index to the CPU for serving, which is the realistic deployment for these quantization schemes at this scale. Each build also writes a record of its parameters, wall time, on-disk size, in-memory size, and peak memory, so the memory column in the results is measured rather than estimated.

The index benchmark, over 1.2 million real vectors

On the GPU tier, 1.2 million MS MARCO passages were embedded with Qwen3 into 1024-dimension vectors, and all five index types were built and searched against the exact ground truth. Recall is the best value over the search-parameter sweep, and latency is single-query and closed-loop:

IVF-SQ8 gives up a single point of recall to fit in a quarter of the memory, and IVF-PQ gives up a lot more to fit the whole index into a hundred megabytes, which is the trade you take at a billion vectors when a GPU-resident index is the only affordable option. There is no best index in that table, only a curve, and the point of the project is to draw the curve on your own data instead of trusting someone else's.

How to read these numbers

Every metric here scores a ranked list against a set of relevance judgments and then averages over queries. They differ only in what they reward. Take one query whose relevant documents are D3, which is highly relevant with grade 3, and D7, which is relevant with grade 1, and a system that returned [D5, D3, D9, D7, D1].

Recall@k asks whether the relevant documents were found at all. It is the fraction of the relevant set that appears in the top k, and it ignores order. Here recall@5 is 2 out of 2, or 1.0, because both relevant documents made the top five. It is a coverage measure, which is why recall@100 is the generous "did the model surface the answer anywhere near the top" number in the model sweep.

MRR@k, mean reciprocal rank, asks how high the first relevant document sits. It is one over its rank, or zero if none is in the top k. Here the first relevant document is D3 at rank two, so the reciprocal rank is 1 over 2, or 0.5. It only looks at the first hit and ignores everything after it, which makes it the right metric when there is essentially one correct answer and it needs to be at the top, as in question answering.

nDCG@k, normalized discounted cumulative gain, is the complete one, and the one worth being able to derive on a whiteboard. It uses graded relevance and rewards putting more-relevant documents higher. You walk down the ranking and add up each document's grade discounted by its position, which is grade / log2(rank + 1), so a great document at rank one counts for more than the same document at rank ten. For this list the discounted gain is 3/log2(3) + 1/log2(5) = 1.893 + 0.431 = 2.324. Then you compute the same sum for the ideal ranking, which is D3 and then D7, and that comes to 3/log2(2) + 1/log2(3) = 3.0 + 0.631 = 3.631. Dividing the two gives nDCG@5 of 2.324 over 3.631, or 0.640. Normalizing by the ideal puts every query on a 0 to 1 scale no matter how many relevant documents it has, so the average across queries is meaningful. nDCG rewards finding relevant documents, ranking the most relevant ones first, and putting them near the top, all at once, which is why the BEIR and MTEB leaderboards report it and why the model sweep is ranked on it.

The bugs only real hardware and real data will show you

The unit tests were green the whole way through and the type checker was clean, and none of that prevented six real bugs that showed up only when the code met a GPU and a real corpus. Writing them down is the point, because the fixes are the part of the project a reader learns the most from.

• Ray workers that could not import Ray. Under uv, Ray 2.56 noticed the launcher and started its workers with a copy of the project that had no dependencies installed, so the pipeline hung with the workers unable to run import ray. The fix was to turn off that auto-detection and let the workers use the installed environment.
• A CUDA library load-order conflict. Importing faiss before torch loaded faiss's older bundled cuBLAS, which was missing a symbol. Importing torch first, so its newer CUDA libraries win, fixed it, and a one-line helper now guarantees that order.
• IVF-PQ recall that fell as you searched harder. Product quantization encodes the residual from a cell's centroid, which is only correct under an L2 metric. Built under inner product, its recall dropped as nprobe rose, which is exactly backwards. Because the embeddings are unit-normalized, nearest by inner product is the same as nearest by L2, so building IVF-PQ under L2 restored recall that rises with search effort.
• A passage vLLM refused to embed. A real MS MARCO passage tokenized to 513 tokens against a 512 limit, and the engine rejected it instead of truncating it. Passing a truncation flag fixed it. Real corpora contain the inputs your tests never think to generate.
• A single-GPU deadlock in the distributed layer. Ray's actor pool restarted the vLLM worker after its engine stalled, but the dead engine never released its twenty gigabytes of GPU memory, so the restart waited forever. On one GPU the distributed machinery was pure overhead and this failure mode, so the fix routes the single-GPU case through a plain sequential loop and keeps Ray for the multi-GPU case it is actually meant for.
• A latent bug an adversarial review caught before any run. A panel of review agents reading the code flagged that the GPU ground-truth path under an L2 metric negated its distances and would have selected the farthest neighbours. It was unreachable by the inner-product profiles but real, and it is fixed.

Limitations

The billion-scale tier is designed, documented, and costed against Lambda pricing, but it has not been run. The largest run here is 1.2 million vectors on a single desktop GPU. The model sweep is two small English benchmarks, chosen because they can be evaluated exactly, not a broad multilingual or long-document study. The retrieval-quality MRR on MS MARCO itself is not reported, because computing it means embedding the full corpus through vLLM, and a stability issue in that engine under sustained embedding, the same one behind the deadlock above, made the reliable path the sentence-transformers backend, which is what the model sweep runs on. Every GPU code path is exercised on real data, but the full-corpus single-engine embed is gated on that fix. And everything is one run per configuration. The numbers come with their configs in the repository, so anyone can rebuild the identical benchmark and shake them.

Links

• Source on GitHub, the platform, the three config profiles, and the tests.
• Build log, the engineering narrative, phase by phase, with every bug and fix.
• Production runbook, the Lambda provisioning and the billion-scale cost projection.