Data pipelines and evaluation: better data, LLM judges, and honest measurement

Why this subject matters now

For most of the last decade the public conversation about language models was a conversation about architectures and parameter counts. The teams actually shipping frontier models learned a different lesson. With the transformer recipe now standardized, the durable advantages live in what the model is trained on and in how honestly its behavior is measured. The clearest public demonstrations are recent. The RefinedWeb work (Penedo et al., 2023) showed that carefully filtered and deduplicated web data alone could match corpora assembled from curated books and technical sources, overturning the assumption that web text is a filler ingredient. FineWeb (Penedo et al., 2024) released both a 15-trillion-token corpus and, more valuably, the full ablation methodology behind every filtering decision. DataComp-LM (Li et al., 2024) turned corpus construction itself into a benchmark, holding the model and training recipe fixed while participants compete only on data curation, and the headline result was that a simple classifier-based filter beat far more elaborate pipelines. Data work, long treated as janitorial, is now the published, ablated, competitive core of the field.

Evaluation went through the same maturation under more pressure. Once models began producing open-ended text that no exact-match metric could score, the field adopted strong models as judges of weaker ones, and Zheng et al. (2023) showed with MT-Bench and Chatbot Arena data that a strong judge agrees with human raters about as often as humans agree with each other. That result made LLM-as-judge the default instrument for iteration, and it also started the clock on a literature of judge biases, position preference, verbosity preference, self-preference, that any practitioner is now expected to know how to mitigate. Meanwhile the statistics of evaluation, long ignored, became load bearing. Public leaderboards rank models by tenths of a point on eval sets of a few hundred items, differences that a confidence interval dissolves instantly. A practitioner today is expected to know how corpora are built, how judges fail, and how to tell a real improvement from sampling noise, which is exactly the sequence this page walks through.

Core theory

The page runs in three arcs. The first follows a document from a raw crawl archive to a tokenized training shard and derives the deduplication math that dominates pipeline design. The second builds the LLM-as-judge instrument, derives the agreement statistic used to calibrate it, and catalogs its failure modes. The third derives the statistics of sampling and measurement, when a majority vote beats a single sample, what pass@k actually estimates, and how wide the error bars on an eval really are.

The anatomy of a pretraining data pipeline

Nearly every open web-scale corpus starts from Common Crawl, a nonprofit that has been archiving the web since 2008 and releases crawl snapshots several times a year, each containing billions of pages. A snapshot ships in three formats. WARC files hold raw HTTP responses including full HTML, WAT files hold metadata, and WET files hold text pre-extracted by a simple built-in tool. The distinction matters more than it looks, and the choice of format is the first quality decision a pipeline makes, for reasons the next section quantifies. On top of the crawl, serious corpora add code, typically permissively licensed repositories gathered the way The Stack (Kocetkov et al., 2022) did with license detection and an opt-out process, and curated sources such as encyclopedias, public-domain books, academic preprint archives, and forum archives with usable licensing. The Pile (Gao et al., 2020) established the pattern of mixing many named sources with explicit weights, and MassiveText (Rae et al., 2021) did the same privately for Gopher.

The pipeline that turns a crawl into training data is remarkably consistent across FineWeb, RefinedWeb, Dolma (Soldaini et al., 2024), and CCNet (Wenzek et al., 2020), even though the teams built them independently. The stages differ in detail but not in order, and the order is itself a design decision, cheap filters run first so expensive ones see fewer bytes.

WARC archives (petabytes)
   |
   v
URL filtering ......... blocklists, adult/spam domains
   |
   v
HTML text extraction .. trafilatura / resiliparse, boilerplate removal
   |
   v
language ID ........... fastText lid.176, keep target language above threshold
   |
   v
heuristic filters ..... Gopher rules, C4 rules, repetition stats
   |
   v
model-based quality ... fastText or small classifier score, keep top mass
   |
   v
deduplication ......... exact hash -> MinHash-LSH fuzzy -> substring dedup
   |
   v
PII scrubbing ......... emails, IPs, phone numbers rewritten or dropped
   |
   v
decontamination ....... n-gram overlap against eval benchmarks
   |
   v
tokenize + shard ...... fixed tokenizer, sequence packing, shuffled shards
   |
   v
mixing weights ........ per-source sampling rates, tuned by proxy ablations

Two properties make this a systems problem rather than a scripting problem. The input is petabytes, so every stage must stream, shard, and checkpoint, and a rerun of the full pipeline is expensive enough that stage boundaries double as recovery points. And every stage throws data away, typically leaving low single-digit percentages of the raw crawl by the end, so small changes in filter thresholds move terabytes. The engineering half of this page returns to how datatrove, Dolma's toolkit, and NeMo-Curator organize that work.

Extraction, where most of the quality is won or lost

The single most consequential early finding in the public pipeline literature is that text extraction quality dominates nearly everything downstream. The WET files that Common Crawl ships are extracted with a generic tool that keeps navigation menus, cookie banners, footers, and boilerplate, and C4 (Raffel et al., 2020) was built from WET files with aggressive line-level cleanup as compensation. RefinedWeb and FineWeb instead went back to the raw WARC HTML and ran trafilatura (Barbaresi, 2021), a purpose-built extractor that uses document structure to isolate main content, and both report in their ablations that models trained on WARC-extracted text clearly beat models trained on WET text even when every later filtering stage is identical. DataComp-LM reached the same conclusion with resiliparse, a faster extractor from the ChatNoir search group. The reason is mechanical rather than mysterious. Boilerplate is highly repetitive across pages from the same site, so it wastes model capacity on memorizing menus, it pollutes the n-gram statistics that later filters rely on, and it creates artificial near-duplicates that confuse deduplication. Fixing text at the source is worth more than any amount of downstream patching, which is a lesson that generalizes well beyond crawl processing.

Language identification and perplexity filtering

Language identification is nearly always fastText (Joulin et al., 2017), a linear classifier over character n-gram features whose public 176-language model classifies millions of documents per second per core. The design fits the job, at this stage the pipeline still holds billions of documents and anything heavier than a linear model is unaffordable. Each document gets a language label and a confidence, and the pipeline keeps documents whose target-language confidence clears a threshold. FineWeb reports keeping English documents scoring at least 0.65, a deliberately permissive setting, because aggressive thresholds disproportionately discard informal text, dialogue, and code-mixed writing that the model still benefits from seeing.

CCNet (Wenzek et al., 2020) added the second classic filter, perplexity under a small KenLM n-gram model (Heafield, 2011) trained on Wikipedia in each language. Score every document, bucket the corpus into head, middle, and tail by perplexity percentile, and treat low perplexity as evidence of well-formed text. The technique is cheap and effective, and its bias is exactly its mechanism, a Wikipedia-trained scorer prefers text that looks like Wikipedia. Poetry, transcripts, forum slang, and technical logs all score badly without being low quality, which is why later pipelines either keep the middle buckets too or replace perplexity filtering with learned quality classifiers whose training labels encode a broader notion of quality. Whatever scores a filter produces, the honest move is the one CCNet made, keep the score as metadata and let the training run decide the threshold, rather than discarding data irreversibly at pipeline time.

Heuristic filters and model-based quality scoring

Heuristic quality filters are rule lists tuned by inspection, and the two canonical sets are worth knowing precisely because nearly every modern pipeline starts from them. The C4 rules (Raffel et al., 2020) work line by line, keep lines that end in terminal punctuation, drop pages with fewer than a handful of sentences, drop lines containing boilerplate markers such as JavaScript warnings or lorem ipsum, drop pages containing words from a blocklist, and drop pages containing braces on the theory that they are code fragments in the wrong corpus. The Gopher rules (Rae et al., 2021) work at the document level, keep documents between 50 and 100,000 words, mean word length between 3 and 10 characters, at most 10 percent symbol characters relative to words, fewer than 90 percent of lines starting with a bullet, fewer than 30 percent ending with an ellipsis, at least 80 percent of words containing an alphabetic character, and at least two occurrences from a short list of English stop words. Each rule is transparent, auditable, and independently ablatable, which is the real argument for heuristics, and each encodes an opinion that will bite some legitimate text, which is the argument for measuring rather than trusting them. FineWeb's ablations found several C4 rules valuable but the terminal punctuation rule too destructive for what it bought, and added three custom filters derived by comparing statistics of kept and discarded text.

Model-based quality scoring replaces rules with a trained classifier. The pattern has two live variants. FineWeb-Edu had a strong instruction model score half a million web pages for educational value on a rubric, trained a small embedding-based regressor on those annotations, ran it over all of FineWeb, and kept documents scoring at or above 3 of 5. The resulting 1.3-trillion-token subset lifted knowledge-heavy benchmarks like MMLU and ARC well beyond the parent corpus, a paper-reported result that made classifier filtering standard practice almost overnight. DataComp-LM's variant is even simpler and slightly uncomfortable for anyone expecting sophistication to win, a fastText bigram classifier trained to distinguish instruction-formatted and high-vote answer text from random crawl text outperformed every heavier quality model the competition tried, and the resulting DCLM-Baseline corpus trained a 7B model to 64 percent five-shot MMLU on 2.6 trillion tokens, figures reported in the paper. The two lessons compose neatly, what matters is the label definition far more than the scorer capacity, and a linear model over good labels beats a deep model over vague ones.

Every filter, heuristic or learned, is a distribution shift applied on purpose, and the only defensible way to set thresholds is the ablation loop described at the end of this arc, train small proxy models on filtered and unfiltered variants and let benchmark deltas decide. The FineWeb report is the most complete public record of that loop in action, dozens of controlled 1B-to-2B-parameter training runs, one per pipeline decision, each judged on a fixed benchmark basket.

Deduplication, exact, fuzzy, and substring

Web crawls are saturated with repetition, mirrors, boilerplate, syndicated articles, template pages, and the same license text on millions of pages. Duplication distorts training in three distinct ways. It silently reweights the data distribution toward whatever is most copied, it wastes compute on gradient steps that teach nothing new, and it drives memorization, since Carlini et al. (2023) measured that verbatim regurgitation grows log-linearly with the number of times a sequence appears in training. Lee et al. (2022), in the line of work titled Deduplicating Training Data Makes Language Models Better, showed that removing duplicates cuts emitted memorized text by an order of magnitude while leaving perplexity unharmed or improved, which settled the question of whether dedup is worth pipeline complexity.

Deduplication runs at three granularities. Exact document dedup hashes each normalized document, MD5 or a 64-bit fingerprint, and keeps one representative per hash, which is cheap and catches only literal copies. Fuzzy document dedup catches near-copies that differ by a date stamp or an edited sentence, and is where the interesting math lives. Substring dedup, the ExactSubstr method of Lee et al., builds a suffix array over the whole corpus and removes any character span of 50 or more tokens that appears more than once, catching the repeated quotation or license block embedded inside otherwise distinct documents, a case both document-level methods miss by construction.

Fuzzy dedup starts from a similarity definition. Represent each document as its set of word n-grams, called shingles, with \( n = 5 \) a common choice, and measure the similarity of two documents as the Jaccard overlap of their shingle sets,

Computing \( J \) for all pairs of billions of documents is out of the question, and MinHash (Broder, 1997) is the fix. Apply a random hash function to every shingle in a set and record the minimum hash value. The minimum is achieved by whichever element of \( A \cup B \) hashes lowest, every element is equally likely to be that minimizer, and the two documents share the same minimum exactly when the minimizer lands in \( A \cap B \). Therefore a single MinHash collision has probability exactly \( J(A, B) \). The full proof, written as a statement about random permutations, is on the mining massive datasets page, which owns the general theory of similarity sketches. What this page needs is the estimator built from it. Take \( n_h \) independent hash functions, giving each document a signature of \( n_h \) minima, and estimate \( J \) by the fraction of agreeing signature positions. Each position is a Bernoulli trial with success probability \( J \), so the estimate is unbiased with variance \( J(1-J)/n_h \). At the FineWeb signature size of \( n_h = 112 \) and a true similarity of 0.75, the standard error is \( \sqrt{0.75 \times 0.25 / 112} = 0.041 \), tight enough to rank candidate pairs reliably.

The signature compresses each document to a few hundred bytes, but comparing all signature pairs is still quadratic. Locality-sensitive hashing by banding removes the quadratic step. Split the signature of \( n_h = b \cdot r \) values into \( b \) bands of \( r \) consecutive values, hash each band to a bucket, and declare any two documents sharing a bucket in any band a candidate pair. The probability arithmetic takes four lines. Two documents with Jaccard similarity \( J \) agree on one signature position with probability \( J \), positions are independent because the hash functions are, so a full band of \( r \) positions agrees with probability \( J^r \). A band fails to match with probability \( 1 - J^r \), the \( b \) bands use disjoint hash functions and therefore fail independently, so all bands fail with probability \( (1 - J^r)^b \), and the detection probability is the complement,

This is an S-curve in \( J \), flat near 0, flat near 1, and steep in between, with the transition centered near the similarity where a single band match becomes likely, approximately \( J^\ast \approx (1/b)^{1/r} \). Raising \( r \) sharpens the curve and pushes the threshold up, raising \( b \) pulls the threshold down and raises the false-candidate rate, and the product \( b \cdot r \) is the signature size being paid for. FineWeb's production configuration is \( n_h = 112 \) split as \( b = 14 \) bands of \( r = 8 \), which places the threshold at \( (1/14)^{1/8} = 0.719 \), deliberately just under the 0.75 similarity level the pipeline targets. Problem 1 works the exact detection numbers, including how many true duplicates such a configuration misses, which is the number pipeline designers actually argue about.

Candidate pairs are then merged into duplicate clusters with a union-find pass, and the pipeline keeps one representative per cluster, usually the longest document or the one from the most recent crawl. One finding from the FineWeb report deserves its own sentence because it is counterintuitive and expensive to discover. Deduplicating all 96 crawl snapshots against each other, the obviously thorough choice, produced worse models than deduplicating each snapshot independently, because global deduplication preferentially deletes the good text that recurs across years of the web while leaving each snapshot's unique junk untouched, effectively upsampling low quality. Dedup aggressiveness is a tunable with a quality optimum, not a virtue to maximize, and only the ablation loop finds the optimum.

Memorization, PII, and decontamination

Three cleanup obligations sit at the end of the document-level pipeline. The first is personally identifiable information. Open pipelines run regular-expression scrubbers for the structured cases, email addresses, IP addresses, phone numbers, rewriting them to fixed placeholder tokens, which is what FineWeb and Dolma both ship. The honest caveat is that regex PII removal is a floor, not a solution, names and addresses in free text survive it, and the memorization results above mean anything that appears often enough can be regurgitated. Deduplication is itself a privacy mitigation for exactly that reason, it removes the repetition that drives extraction attacks of the kind Carlini and collaborators demonstrated.

The second is decontamination against evaluation sets. A benchmark item that leaked into training data converts an evaluation of generalization into an evaluation of recall, and every serious training effort scans for overlap. The standard mechanism is n-gram overlap, mark a training document contaminated if it shares any n-gram of a chosen length, commonly around 13 tokens as popularized by the GPT-3 analysis, with any test item, then drop or truncate it. The n-gram length is a real tradeoff, short n-grams flag common phrases and destroy legitimate data, long n-grams miss paraphrased leakage entirely, and paraphrase contamination defeats string matching altogether, which is why post-cutoff benchmarks and perturbation tests exist. The self-improving agents page covers the eval-integrity side of contamination, including how training loops can launder leaked benchmarks into apparent capability.

The third is the boundary between cleaning and tokenizing. Once the document set is final, the corpus is tokenized with the model's fixed tokenizer, documents are packed into fixed-length sequences separated by end-of-document tokens, and the token stream is written as shuffled binary shards with an index, so that training can seek to any step deterministically and resume after a failure. Tokenizer construction itself, byte-level BPE and its tradeoffs, is derived on the language models from scratch page, and everything there is upstream of here, the tokenizer must exist before the corpus can be sharded.

Data mixing and the ablation loop

A corpus assembled from multiple sources needs sampling weights, how often training draws from web text versus code versus reference material, and the weights matter as much as the filtering. The Pile fixed weights by editorial judgment. Modern practice treats mixing as an experimental variable. The direct method is the proxy ablation loop, train small models, in public reports typically around 1B parameters on a few tens of billions of tokens, on candidate mixtures, evaluate on a fixed benchmark basket, and iterate. The loop rests on the assumption that data-quality rankings transfer across scale even though absolute scores do not, an assumption both FineWeb and DataComp-LM tested by spot-checking rankings at a larger scale before trusting it. DoReMi (Xie et al., 2023) automates the search, train a small proxy with group distributionally robust optimization over domains, read the learned domain weights out, and reuse them at scale, with the paper reporting transfer from a 280M proxy to an 8B model. The loop closes the arc because it is where every earlier decision is adjudicated, an extraction change, a filter threshold, and a dedup policy are all just mixtures to ablate, and a pipeline without the loop is a pile of opinions.

Judging generations, pointwise and pairwise

The second arc begins where exact-match scoring ends. Once outputs are paragraphs rather than labels, the measurement instrument of record is a strong language model prompted to evaluate, and the design space starts with a binary choice. A pointwise judge scores one response on an absolute scale, one to five or one to ten, against a rubric. A pairwise judge sees two responses to the same prompt and picks a winner or declares a tie. Pointwise scoring is cheaper, \( O(n) \) judge calls for \( n \) responses, produces scores that can be tracked over time without a fixed opponent, and is the right shape for regression gates in CI. Its weakness is that absolute scales drift and compress, a judge's 7 means different things on different days, different prompts, and different response styles, and most of the scale's resolution goes unused. Pairwise comparison anchors the judgment to a concrete alternative, which is why it is more reliable in practice and why the reliability results in Zheng et al. (2023) are pairwise results. Humans show the same asymmetry, relative judgments are easier and more repeatable than absolute ones, which is the psychometric reason preference data collection is pairwise almost everywhere.

Pairwise verdicts aggregate into rankings through the Bradley-Terry model, which assigns each model a strength \( \theta_i \) and models the probability that \( i \) beats \( j \) as \( \sigma(\theta_i - \theta_j) \), fit by maximum likelihood over the win-loss record. Chatbot Arena fits exactly this to millions of human votes, and Arena-Hard (Li et al., 2024) reproduces the construction with an LLM judge against a fixed baseline model, reporting bootstrap confidence intervals over the fitted strengths, which connects this arc to the third one, even judge-based rankings need error bars.

Rubrics, structured output, and reference-guided grading

A judge prompt is a measurement instrument, and the difference between a casual one and a designed one shows up directly in agreement statistics. The design elements that recur across MT-Bench, G-Eval (Liu et al., 2023), and Prometheus (Kim et al., 2024) are worth listing as a checklist. State the evaluation criteria explicitly and separately, correctness, completeness, safety, style, rather than asking for overall goodness, because a single blended score lets style leak into correctness. Give the scale anchors concrete descriptions, what a 2 looks like versus a 4, because unanchored numeric scales collapse toward the top. Require the judge to reason before it scores, G-Eval's chain-of-thought form filling, because verdict-first outputs correlate worse with humans. Emit structured output, a JSON object with per-criterion scores and a final verdict field, so the harness parses reliably and refusals are detectable. And when the task has a known correct answer, put a reference solution in the judge prompt, the reference-guided mode that Zheng et al. found necessary for math and reasoning grading, where judges otherwise confidently bless wrong arithmetic. G-Eval adds one refinement worth knowing, instead of taking the single sampled score, weight each possible score by the probability the judge assigns its token, recovering a continuous expected score from a discrete scale and reducing variance at no extra cost when logprobs are available.

Prometheus represents the complementary line, an open evaluator model fine-tuned specifically to follow user-supplied rubrics, built at KAIST as an alternative to closed-model judging, with the second version trained for both pointwise and pairwise formats. Open evaluator models matter operationally, a judge that changes silently behind an API invalidates every historical score it produced, and pinning the judge is as important as pinning the benchmark.

The documented biases of LLM judges and their mitigations

Every known judge bias is a way the instrument's reading moves without the quality moving. Four are documented well enough to treat as standing assumptions. Position bias, a pairwise judge preference for one slot, typically the first, demonstrated in Zheng et al. by swapping response order and watching verdicts flip. Verbosity bias, a preference for longer answers at equal content, demonstrated with padded responses that repeat information without adding any. Self-enhancement bias, a tendency to favor outputs in the judge's own style or from its own family, harder to isolate cleanly but observed consistently enough to design around. And style over substance, the general form of the previous two, where fluency, formatting, and confidence pull scores upward independently of correctness, which is the bias that makes uncalibrated judges dangerous for factual tasks.

The mitigations are mechanical and cheap relative to what they buy. Swapping order costs a factor of two in judge calls and removes a bias that can exceed the effect being measured. Length control has the cleanest public treatment in the length-controlled AlpacaEval (Dubois et al., 2024), which fits a regression predicting the judge's preference from the length difference and reports the win rate with the length term zeroed out, and the paper reports that the corrected metric agrees substantially better with Chatbot Arena rankings than the raw one. Ensembling judges from different model families attacks self-preference the way ensembling annotators attacks any single rater's quirks. None of the mitigations remove the need for the calibration step in the next section, they only make the instrument worth calibrating.

Agreement with humans, Cohen's kappa derived

Before a judge's scores decide anything, the judge must be scored, and the standard instrument is agreement with human labels on a gold set, corrected for chance. Raw percent agreement is the wrong statistic, and the reason is worth internalizing. If both the judge and the human pass 90 percent of responses, they agree 82 percent of the time by blind guessing alone, \( 0.9 \times 0.9 + 0.1 \times 0.1 = 0.82 \), so an observed 85 percent agreement is barely better than chance despite sounding strong. Cohen's kappa (Cohen, 1960) is the correction. Let \( p_o \) be observed agreement, the fraction of items where the two raters give the same label, and let \( p_e \) be the agreement expected if the raters were independent with their observed marginal label rates,

summing over label categories \( c \). Kappa rescales observed agreement so that chance sits at zero and perfection at one. The derivation is one line of geometry, the agreement scale runs from \( p_e \) (independence) to 1 (perfect), the raters sit at \( p_o \), and their position as a fraction of the achievable range above chance is

Kappa is 1 only at perfect agreement, 0 when agreement is exactly what independence predicts, and negative when the raters disagree more than chance. The conventional reading, due to Landis and Koch (1977), calls 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and above 0.80 near perfect, with the caveat that these bands are conventions from biostatistics, not laws. The statistic has a documented sensitivity worth knowing before quoting it, when one label dominates, \( p_e \) is forced high and kappa is compressed even at high raw agreement, the so-called kappa paradox, so a judge evaluated on a gold set where 93 percent of items pass can post a mediocre kappa while looking excellent on raw agreement. Problem 2 works both cases by hand.

Calibration, judges versus reward models, and cheap checks first

The calibration workflow that falls out of the last three sections is short enough to state as a procedure. Assemble a gold set of one to three hundred items labeled by people who understand the task, stratified to include real failures. Run the candidate judge prompt over it, compute kappa or pairwise agreement, and iterate on the rubric, the anchors, and the mitigations until the number clears a bar chosen in advance, with human-human agreement on the same items as the ceiling to compare against. Freeze the judge, model version and prompt together, version them like code, and cache every verdict. Re-run the gold set whenever anything changes and on a schedule even when nothing has, because API models drift. This costs a few hundred labeled items once, and it converts the judge from an oracle into an instrument with a known error rate.

A judge is not a reward model, and the distinction is operational. A reward model is a trained scalar head over a base model, fit on preference pairs with a Bradley-Terry likelihood, cheap enough to score every sample in a best-of-n loop or every step of an RLHF run, and opaque, its score is a learned correlate of preference with no rationale attached. A prompted judge is expensive per call, flexible per task, and interpretable, it emits reasons that can be audited. Reward models sit inside training loops and samplers, judges sit inside evaluations and release gates, and the benchmark line for the former, RewardBench and its descendants from AI2, measures exactly the failure the latter catches by construction, reward models that prefer style over correctness. The self-improving agents page treats reward models in depth, including process reward models and the overoptimization curve that emerges when a sampler is allowed to exploit one.

The last rule of this arc costs nothing and is violated constantly. When a deterministic check exists, it beats any judge. Schema validity is a parser call, code correctness is a test suite, math answers match a reference, citations resolve or do not, lengths and formats are string operations. A judge call costs four to six orders of magnitude more than a regex and returns a noisier answer wherever the regex applies at all. The design pattern in production harnesses is a cascade, programmatic checks first as hard gates, the judge only on responses that pass them, scoring only the properties no program can check, and humans only on the slice where the judge is uncertain or the stakes are high. Structured-output techniques that make more of a task programmatically checkable, covered on the applied generative AI page, shrink the judge's jurisdiction and are usually the cheapest reliability win available.

One sample or many, the sampling recap

The third arc needs one paragraph of decoding background. A language model defines a distribution over next tokens, and temperature \( T \) reshapes it, \( p_T(y) \propto p(y)^{1/T} \), sharpening toward the argmax as \( T \to 0 \) and flattening toward uniform as \( T \) grows. Greedy decoding takes the argmax at every step, which is deterministic and is the mode of each conditional, not the most probable full sequence and not in any sense the model's best considered answer. Nucleus sampling (Holtzman et al., 2020) samples from the smallest token set whose cumulative probability exceeds a threshold, cutting the unreliable tail that makes pure sampling degenerate while keeping diversity that beam-style maximization destroys. For this arc, all that matters is that any nonzero temperature makes the output a random draw, so any quantity computed from one sample is a one-draw estimate, and the question of the arc is when one draw is enough.

Self-consistency, the arithmetic of majority voting

Self-consistency (Wang et al., 2023) samples \( n \) reasoning chains at nonzero temperature, extracts each chain's final answer, and returns the most common one. The paper reported gains of roughly 18 points on GSM8K for PaLM-540B over a single chain-of-thought sample, a paper-reported figure that made sample-and-vote the default for offline reasoning workloads. The mechanism is a Monte Carlo estimate of the answer marginal, voting recovers the answer the model assigns the most total probability across all reasoning paths, which a single sample estimates with one draw and greedy decoding does not estimate at all.

The binomial arithmetic answers the design question, when does majority-of-\( n \) beat one sample. Take the binary case, the model answers correctly with probability \( p \) per independent draw and incorrectly with a single wrong answer otherwise. With \( n \) odd, the vote is correct when more than half the draws are, a binomial tail for \( X \sim \mathrm{Bin}(n, p) \),

For \( n = 3 \) the sum has two terms and the comparison against one sample can be settled algebraically. The vote is correct with probability \( 3p^2(1-p) + p^3 = 3p^2 - 2p^3 \), and it beats a single draw exactly when \( 3p^2 - 2p^3 > p \). Dividing by \( p \) and rearranging gives \( 2p^2 - 3p + 1 < 0 \), which factors as \( (2p - 1)(p - 1) < 0 \), true precisely when \( \tfrac12 < p < 1 \). Voting helps if and only if the model is more often right than wrong, and the same threshold governs all odd \( n \), with error decaying exponentially at rate \( 2(p - \tfrac12)^2 \) per sample by Hoeffding when \( p > \tfrac12 \) and converging to the wrong answer with equal confidence when \( p < \tfrac12 \). Voting is an amplifier of the modal answer, not a source of correctness. The multi-answer case, where what matters is the gap between the correct answer's probability and the strongest wrong answer's, is derived in full on the self-improving agents page, which owns the deeper test-time-compute treatment, including best-of-n selection against a reward model and the KL cost of selection. Problem 3 works the numbers, including the diminishing-returns schedule that decides how many samples are worth paying for.

Best-of-n and verifiers, briefly

Majority voting requires answers that can be counted, short final answers that either match or do not. For open-ended outputs the aggregation step becomes selection, sample \( n \) candidates and keep the one a verifier scores highest, with the verifier ranging from a unit-test suite for code, where selection is exact, to a reward model, where selection inherits the reward model's biases and overoptimizes them as \( n \) grows. The compute tradeoff is linear cost in \( n \) for gains that saturate, and the quality ceiling is the verifier's, not the sampler's, a best-of-64 run against a flawed reward model converges on the flaw. The quantitative treatment, including why selection pressure on a learned reward bends quality back downward, lives on the self-improving agents page. What belongs in this arc is the measurement corollary, any system that samples and selects must be evaluated end to end, sampling the same way in eval as in production, because pass@1 of the underlying model and pass@1 of the sample-and-select system are different quantities separated by exactly the machinery this arc prices.

pass@k and its unbiased estimator

For tasks with a programmatic verifier, capability at a sampling budget is summarized by pass@k, the probability that at least one of \( k \) independent samples succeeds. If a problem's per-sample success probability is \( p \), then \( \text{pass@}k = 1 - (1-p)^k \), averaged over problems. The naive protocol, draw exactly \( k \) samples and record whether any passed, is unbiased but wastes information, one Bernoulli observation per problem. The estimator of Chen et al. (2021) draws \( n \ge k \) samples, counts \( c \) successes, and asks what fraction of the \( \binom{n}{k} \) equally likely \( k \)-subsets of the draws contain no success. All-failure subsets must choose all \( k \) elements from the \( n - c \) failures, so

where the product form is the numerically safe way to compute it, since the raw binomial coefficients overflow floating point once \( n \) reaches a few hundred. Because the \( n \) draws are exchangeable, any \( k \)-subset of them is distributed exactly as a fresh \( k \)-sample, so the subset-counting fraction has expectation \( (1-p)^k \) and the estimator is unbiased, with variance shrinking as \( n \) grows past \( k \) because one set of draws now contributes \( \binom{n}{k} \) correlated subsamples instead of one. The tempting shortcut of plugging \( \hat p = c/n \) into \( 1 - (1-\hat p)^k \) is biased, and the direction follows from Jensen's inequality since \( (1-p)^k \) is convex in \( p \), a bias the sibling page quantifies. Problem 5 works both estimators on the same draw and then computes their exact expectations, which is the cleanest way to see what unbiased does and does not promise. One usage rule keeps pass@k honest, it is a capability metric for systems that get \( k \) tries against a verifier, and quoting pass@100 for a product that ships one sample without verification is quoting a different system's score.

Error bars, Wilson intervals, bootstrap, and paired tests

An accuracy on an eval set is a binomial proportion, and it deserves an interval. The textbook Wald interval \( \hat p \pm z \sqrt{\hat p (1 - \hat p)/m} \) misbehaves exactly where evals live, at small \( m \) and extreme \( \hat p \), collapsing to zero width at \( \hat p \in \{0, 1\} \). The Wilson score interval (Wilson, 1927) fixes this by inverting the score test rather than plugging in \( \hat p \). Ask which true values \( p \) are consistent with the observation, keeping every \( p \) satisfying

with \( z = 1.96 \) for 95 percent coverage. Expanding gives a quadratic in \( p \), \( (1 + z^2/m)\, p^2 - (2\hat p + z^2/m)\, p + \hat p^2 \le 0 \), whose roots are the interval endpoints,

The center is \( \hat p \) shrunk toward one half, the width is finite even at \( \hat p = 1 \), and at \( m = 200 \) items the 95 percent interval is roughly thirteen points wide, a number worth memorizing before reading any leaderboard. For metrics more complicated than a proportion, judge scores, length-controlled win rates, Bradley-Terry strengths, the bootstrap (Efron, 1979, and the Efron and Tibshirani book) replaces algebra, resample the eval items with replacement, recompute the metric each time, and read the interval off the percentiles of the resampled distribution, which is exactly how Arena-Hard reports its intervals. Evan Miller's error-bars treatment (2024) from Anthropic is the practical reference, compute a standard error for every reported number, cluster it when eval items share a source document, and treat model comparisons as paired data.

Pairing is the single largest free lunch in eval statistics. Two models evaluated on the same items share the item difficulty noise, so the variance of the difference is far smaller than the variance of either score, and the correct test conditions on the items where the models disagree. McNemar's construction makes it exact. Let \( n_{01} \) count items model A gets right and B wrong, \( n_{10} \) the reverse. Concordant items carry no information about the difference, and under the null hypothesis that the models are equally good, each discordant item is A-favoring with probability one half, so \( n_{01} \sim \mathrm{Bin}(n_{01} + n_{10},\, \tfrac12) \), testable exactly by a binomial tail or approximately by the corrected statistic \( (|n_{01} - n_{10}| - 1)^2 / (n_{01} + n_{10}) \) against a chi-square with one degree of freedom, the test Dietterich (1998) recommends for exactly this situation. Problem 4 runs the full computation on a realistic leaderboard-sized comparison and shows how a four-point gap on 200 items evaporates.

A decision guide, one-shot or sample-and-aggregate

The arc compresses to a decision rule with three inputs, the stakes of an error, the latency and cost budget, and whether a verifier or countable answer exists. One greedy sample is correct when the task is deterministic extraction or classification where the model's modal answer is the product, when latency is user-facing, and when the cost of an occasional error is low or downstream validation catches it. Constrained decoding and schema checks make this regime larger than it looks, a structured output that validates is a verified one-shot. Sample-and-aggregate is correct when the task involves multi-step reasoning where answer marginals are sharper than any single path, when the workload is offline so latency is irrelevant, when a verifier exists to make best-of-n exact, and always in evaluation, where the quantity being estimated is a distribution property and a single draw is a sample size of one. The boundary case is a high-stakes online task, and the production pattern there is a cascade, one fast sample plus a programmatic check, escalating to multi-sample voting or a judge only on check failure or low confidence, which buys most of the aggregate accuracy at a fraction of the aggregate cost.

Worked problems

Problems 1 and 2 appear in the theory sections above, the LSH detection computation and the Cohen's kappa computation. The three below complete the set, and every number in all five was verified by running the arithmetic in Python before it was written here.

Implementation

Three implementations, each small enough to read in one sitting and each the seed of a real tool. The first is the deduplication core of this page, a MinHash signature builder and LSH banding index in plain Python with no dependencies beyond the standard library. The signature uses one strong 64-bit base hash and 112 universal-hash mixes of it, split into 14 bands of 8 for the S-curve derived above. Run as written, it flags the two doctored near-duplicates in the demo corpus and passes the unrelated document through, and the estimated Jaccard values it prints match the shingle overlap by direct count.

"""Minimal MinHash + LSH near-duplicate detector, standard library only.

Signature: 112 hashes = 14 bands x 8 rows, so the S-curve is
P(candidate) = 1 - (1 - J^8)^14, steepest near J ~ 0.72.
"""
import hashlib
import random
import re
from collections import defaultdict

N_HASH, BANDS, ROWS = 112, 14, 8          # N_HASH = BANDS * ROWS
MERSENNE = (1 << 61) - 1                  # modulus for the hash family

def base_hash(shingle: str) -> int:
    """One fixed 64-bit hash of a shingle."""
    return int.from_bytes(hashlib.blake2b(
        shingle.encode(), digest_size=8).digest(), "big")

rng = random.Random(0)
AB = [(rng.randrange(1, MERSENNE), rng.randrange(MERSENNE))
      for _ in range(N_HASH)]              # (a, b) per hash function

def shingles(text: str, n: int = 5):
    """Word 5-grams after light normalization."""
    words = re.findall(r"\w+", text.lower())
    return {" ".join(words[i:i + n]) for i in range(len(words) - n + 1)}

def signature(text: str):
    """sig[i] = min over shingles of ((a_i * h + b_i) mod p).  Shape: (112,)"""
    hs = [base_hash(s) for s in shingles(text)]
    if not hs:
        return None
    return tuple(min((a * h + b) % MERSENNE for h in hs) for a, b in AB)

def jaccard_estimate(sig1, sig2) -> float:
    """Fraction of agreeing positions; unbiased for the true Jaccard."""
    return sum(x == y for x, y in zip(sig1, sig2)) / N_HASH

def find_candidates(docs: dict):
    """LSH banding: identical 8-row band slice anywhere -> candidate pair."""
    buckets = defaultdict(list)            # (band index, band tuple) -> ids
    sigs = {}
    for doc_id, text in docs.items():
        sig = signature(text)
        if sig is None:
            continue
        sigs[doc_id] = sig
        for band in range(BANDS):
            key = (band, sig[band * ROWS:(band + 1) * ROWS])
            buckets[key].append(doc_id)
    pairs = set()
    for ids in buckets.values():
        for i in range(len(ids)):
            for j in range(i + 1, len(ids)):
                pairs.add((min(ids[i], ids[j]), max(ids[i], ids[j])))
    # verification pass: keep pairs whose estimated Jaccard clears the bar
    return {(a, b): jaccard_estimate(sigs[a], sigs[b])
            for a, b in pairs if jaccard_estimate(sigs[a], sigs[b]) >= 0.7}

if __name__ == "__main__":
    base = ("The quarterly report shows revenue grew twelve percent year "
            "over year while operating costs held flat, driven by strong "
            "demand in the enterprise segment and steady renewal rates "
            "across all regions according to the finance team.")
    docs = {
        "a": base,
        "b": base.replace("twelve", "thirteen"),      # near duplicate
        "c": base + " Management expects the trend to continue.",
        "d": "Completely unrelated text about hiking trails in the "
             "mountains, wildflowers in spring, and the best months "
             "to visit alpine lakes with a lightweight tent and stove.",
    }
    for pair, j in sorted(find_candidates(docs).items()):
        print(f"{pair}: estimated Jaccard {j:.3f}")
    # prints ('a','b'): 0.786 and ('a','c'): 0.857; 'd' is never flagged

The second implementation is the model-based quality filter from the first arc, a fastText-style classifier, hashed bag of unigrams and bigrams, mean-pooled embeddings, linear head, written side by side in PyTorch and JAX. This is the actual shape of the model behind DCLM's winning filter and close kin to the FineWeb-Edu regressor, a linear probe over cheap features, trained on a modest labeled set and then run over billions of documents where its throughput, not its capacity, is the binding constraint. Both versions train on a toy good-versus-spam corpus in a few seconds on CPU and were run before inclusion, the PyTorch version scores the held-out good and spam documents at 0.92 and 0.06, the JAX version at 0.93 and 0.07.

"""fastText-style quality classifier: hashed n-grams -> mean embedding
-> linear head. The shape of most model-based pretraining filters."""
import re
import torch
import torch.nn.functional as F

VOCAB_BITS = 18                       # 2^18 = 262144 hashed buckets
DIM = 64

def featurize(text: str, max_len: int = 256) -> torch.Tensor:
    """Hash unigrams and bigrams into bucket ids.  Shape: (L,) int64."""
    words = re.findall(r"\w+", text.lower())
    grams = words + [a + "_" + b for a, b in zip(words, words[1:])]
    ids = [hash(g) % (1 << VOCAB_BITS) for g in grams[:max_len]]
    return torch.tensor(ids or [0], dtype=torch.long)

class QualityScorer(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = torch.nn.EmbeddingBag(1 << VOCAB_BITS, DIM, mode="mean")
        self.head = torch.nn.Linear(DIM, 1)

    def forward(self, ids, offsets):
        # ids: (total_tokens,), offsets: (B,) -> pooled: (B, DIM)
        pooled = self.emb(ids, offsets)
        return self.head(pooled).squeeze(-1)        # (B,) logits

def train_step(model, opt, texts, labels):
    ids = torch.cat([featurize(t) for t in texts])           # (total,)
    offsets = torch.tensor(
        [0] + [len(featurize(t)) for t in texts[:-1]],
        dtype=torch.long).cumsum(0)                          # (B,)
    y = torch.tensor(labels, dtype=torch.float32)            # (B,)
    logits = model(ids, offsets)
    loss = F.binary_cross_entropy_with_logits(logits, y)
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

if __name__ == "__main__":
    torch.manual_seed(0)
    model = QualityScorer()
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    good = ["The derivative of a product follows from the limit "
            "definition by adding and subtracting a cross term, which "
            "yields two difference quotients that converge separately."] * 8
    bad = ["click here best deals click here buy now subscribe "
           "click here best deals buy now limited offer"] * 8
    texts, labels = good + bad, [1.0] * 8 + [0.0] * 8
    for step in range(60):
        train_step(model, opt, texts, labels)
    with torch.no_grad():
        ids = torch.cat([featurize(t) for t in [good[0], bad[0]]])
        offs = torch.tensor([0, len(featurize(good[0]))])
        print("scores:", torch.sigmoid(model(ids, offs)).tolist())
        # scores: [0.92, 0.06] after 60 steps (seed 0, verified run)

"""Same hashed bag-of-n-grams quality classifier in JAX. Documents are
padded to a fixed length so the train step jits to one XLA program; a
mask keeps padding out of the mean pool."""
import re
import jax
import jax.numpy as jnp
import optax

VOCAB_BITS = 18
DIM = 64
MAX_LEN = 256

def featurize(text: str):
    """Returns (ids, mask), both shape (MAX_LEN,)."""
    words = re.findall(r"\w+", text.lower())
    grams = words + [a + "_" + b for a, b in zip(words, words[1:])]
    ids = [hash(g) % (1 << VOCAB_BITS) for g in grams[:MAX_LEN]]
    mask = [1.0] * len(ids) + [0.0] * (MAX_LEN - len(ids))
    ids = ids + [0] * (MAX_LEN - len(ids))
    return jnp.array(ids), jnp.array(mask)

def init_params(key):
    k1, k2 = jax.random.split(key)
    return {
        "emb": jax.random.normal(k1, (1 << VOCAB_BITS, DIM)) * 0.02,
        "w": jax.random.normal(k2, (DIM,)) * 0.02,
        "b": jnp.zeros(()),
    }

def score(params, ids, mask):
    # ids: (B, MAX_LEN) -> vecs: (B, MAX_LEN, DIM)
    vecs = params["emb"][ids]
    denom = jnp.maximum(mask.sum(-1, keepdims=True), 1.0)    # (B, 1)
    pooled = (vecs * mask[..., None]).sum(1) / denom          # (B, DIM)
    return pooled @ params["w"] + params["b"]                 # (B,)

def loss_fn(params, ids, mask, y):
    logits = score(params, ids, mask)
    return optax.sigmoid_binary_cross_entropy(logits, y).mean()

@jax.jit
def train_step(params, opt_state, ids, mask, y):
    loss, grads = jax.value_and_grad(loss_fn)(params, ids, mask, y)
    updates, opt_state = tx.update(grads, opt_state)
    return optax.apply_updates(params, updates), opt_state, loss

if __name__ == "__main__":
    good = ["The derivative of a product follows from the limit "
            "definition by adding and subtracting a cross term, which "
            "yields two difference quotients that converge separately."] * 8
    bad = ["click here best deals click here buy now subscribe "
           "click here best deals buy now limited offer"] * 8
    feats = [featurize(t) for t in good + bad]
    ids = jnp.stack([f[0] for f in feats])       # (16, 256)
    mask = jnp.stack([f[1] for f in feats])      # (16, 256)
    y = jnp.array([1.0] * 8 + [0.0] * 8)         # (16,)
    params = init_params(jax.random.PRNGKey(0))
    tx = optax.adam(3e-3)
    opt_state = tx.init(params)
    for step in range(60):
        params, opt_state, loss = train_step(params, opt_state,
                                             ids, mask, y)
    print(jax.nn.sigmoid(score(params, ids[:1], mask[:1])),
          jax.nn.sigmoid(score(params, ids[8:9], mask[8:9])))
    # prints [0.93] [0.07] after 60 steps (seed 0, verified run)

The third implementation is the statistics kit from the last arc, a Wilson interval, a paired bootstrap for accuracy differences, and an exact McNemar test, in sixty lines of standard-library Python. The demo constructs two synthetic models whose true accuracies differ by four points with shared item difficulty, exactly the Problem 4 situation, and on a 200-item eval the run prints overlapping Wilson intervals, a bootstrap interval for the difference that straddles zero, and a McNemar p-value near 0.67, a compact demonstration that a visible gap on a small eval is routinely indistinguishable from noise.

"""Honest eval reporting: Wilson interval for one accuracy, paired
bootstrap for a difference, exact McNemar for paired outcomes."""
import math
import random
from math import comb

def wilson(correct: int, n: int, z: float = 1.96):
    """95% Wilson score interval for a binomial proportion."""
    p = correct / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return center - half, center + half

def paired_bootstrap(a: list, b: list, iters: int = 10_000, seed: int = 0):
    """a[i], b[i] in {0,1}: per-item correctness of two models on the
    same eval set. Percentile 95% CI for mean(a) - mean(b), resampling
    items so shared item difficulty stays paired."""
    rng = random.Random(seed)
    n = len(a)
    deltas = []
    for _ in range(iters):
        idx = [rng.randrange(n) for _ in range(n)]
        deltas.append(sum(a[i] - b[i] for i in idx) / n)
    deltas.sort()
    return deltas[int(0.025 * iters)], deltas[int(0.975 * iters)]

def mcnemar_exact(a: list, b: list):
    """Two-sided exact McNemar p-value from discordant pairs only."""
    n01 = sum(1 for x, y in zip(a, b) if x == 1 and y == 0)
    n10 = sum(1 for x, y in zip(a, b) if x == 0 and y == 1)
    nd = n01 + n10
    if nd == 0:
        return n01, n10, 1.0
    k = max(n01, n10)
    tail = sum(comb(nd, i) for i in range(k, nd + 1)) / 2 ** nd
    return n01, n10, min(1.0, 2 * tail)

if __name__ == "__main__":
    rng = random.Random(1)
    # two models 4 points apart on average, item difficulty shared
    n = 200
    a, b = [], []
    for _ in range(n):
        difficulty = rng.random()
        a.append(1 if rng.random() < 0.75 - 0.5 * difficulty else 0)
        b.append(1 if rng.random() < 0.71 - 0.5 * difficulty else 0)
    print("acc A", sum(a) / n, "Wilson", wilson(sum(a), n))
    print("acc B", sum(b) / n, "Wilson", wilson(sum(b), n))
    print("bootstrap 95% CI for A-B:", paired_bootstrap(a, b))
    n01, n10, p = mcnemar_exact(a, b)
    print(f"discordant {n01}/{n10}, exact McNemar p = {p:.3f}")
    # verified run (seed 1): acc 0.480 vs 0.455, CIs overlap heavily,
    # bootstrap CI (-0.065, 0.115), McNemar p = 0.665 -> pure noise

How it is done in practice

The production data stack has consolidated around a handful of purpose-built tools. datatrove, the Hugging Face library FineWeb was built with, structures a pipeline as composable readers, filters, and writers that run identically on a laptop and on a Slurm cluster of thousands of CPU cores, with per-shard checkpointing so a failed node reruns one shard rather than one snapshot. The Dolma toolkit from AI2 takes a tagger-first design, every filter writes tags as metadata attributes alongside the raw text, and a separate mixer applies keep-or-drop rules over tags, which means a threshold change replays only the cheap mixing stage, an architectural decision that follows directly from how expensive it is to recompute filters over terabytes. CCNet remains the reference implementation of the KenLM perplexity-bucket design. NeMo-Curator from NVIDIA moves the same stages onto GPUs with RAPIDS, computing MinHash signatures and connected components on accelerators, which turns fuzzy dedup of a multi-terabyte corpus from days of CPU time into hours. General-purpose engines still appear underneath, Apache Spark and Beam for organizations already carrying them, and DuckDB has become the standard local instrument for interrogating a few hundred gigabytes of parquet shards with SQL before and after each stage.

Around the core pipeline sits the workflow layer. Orchestrators, Airflow, Dagster, and Prefect are the common choices, own scheduling, retries, and backfills, so that a monthly crawl refresh is a DAG run rather than a person with a checklist, and data-quality gates run as first-class tasks, with Great Expectations or handwritten assertions checking schema, language mix, length distributions, and dedup rates against the previous run before a new corpus version is blessed. The operational metric that matters is bytes surviving each stage, tracked per stage per run, because a filter bug announces itself as a survival-rate anomaly long before any model trains on the damage. Corpus versions are immutable and named, training configs pin them by hash, and the tokenized shards carry provenance back to source URLs, which is what makes decontamination audits and takedown requests answerable at all.

Judge pipelines in production look like miniature data pipelines. Verdicts are cached keyed by judge version, prompt version, and response hash, so reruns are free and historical scores never silently change. The gold set lives in version control next to the rubric, kappa against it is recomputed on every judge or prompt change, and a small human audit slice runs continuously to catch drift. Cost structures the cascade, a programmatic check costs microseconds, a small-judge call milliseconds and fractions of a cent, a frontier-judge call seconds and cents, and human review dollars, so each layer sees only what the previous one could not decide. Evaluation itself runs in CI, every candidate checkpoint gets the full eval basket with pinned sampling parameters, paired statistics against the current production model, and intervals on every number, and a promotion needs a significant paired win, not a bigger point estimate. The teams that publish their practice, the Arena-Hard confidence-interval reporting and Anthropic's error-bars recommendations among them, converge on the same rule, no number ships without its uncertainty.

The current research frontier

On the data side, the open question is what to do when filtering stops being enough. The classifier-filtering line is being pushed by the FineWeb team at Hugging Face, whose multilingual FineWeb2 extends the recipe beyond English, and by NVIDIA, whose Nemotron curation work layers ensembles of quality classifiers and synthetic rephrasing over Common Crawl. Synthetic data is the sharpest frontier, the textbook-style generation line that Microsoft's Phi models popularized showed small models trained on curated synthetic text punching far above their token budgets, and Hugging Face's Cosmopedia reproduced the approach openly. The unresolved questions are distributional collapse under recursive training on model output and how to verify synthetic factuality at scale. Data mixing is becoming a modeling problem in its own right, DoReMi from Xie and collaborators, and successor work on predicting mixture performance from small runs, aim to replace grid-search ablations with fitted scaling behavior. And the measurement of data itself is maturing, DataComp-LM established the fixed-recipe competition format, and the training-data-attribution line asks which documents caused which capabilities, which would turn curation from batch filtering into targeted acquisition.

On the evaluation side, three directions are active. Open evaluator models, Prometheus from KAIST and the JudgeLM line among them, aim to make judging reproducible and auditable rather than an API dependency, with meta-benchmarks emerging to evaluate the evaluators themselves. Bias control is getting more rigorous, the length-controlled AlpacaEval regression and the style-control analyses from the Arena group at Berkeley separate what a model says from how it says it, and reward-model benchmarking from AI2 measures the same style-substance confound inside training loops. And evaluation statistics is becoming a stated discipline, Anthropic's error-bars work argues for clustered standard errors and paired tests as reporting defaults, Arena-style leaderboards now ship Bradley-Terry intervals, and the contamination literature keeps demonstrating that static public benchmarks decay, pushing toward continuously refreshed and post-cutoff eval sets. The common thread across both sides is the same, the field is converting its two softest activities, choosing data and declaring wins, into measured, ablated, versioned engineering.

Open source to read

Nine repositories, each the reference implementation of something this page derived, with the file to open first.

• huggingface/datatrove, the pipeline library behind FineWeb. Start with the FineWeb example pipeline in examples/fineweb.py, which chains every stage of the first arc in one readable file.
• allenai/dolma, AI2's corpus toolkit. Read the taggers directory to see the tag-then-mix architecture, and the Rust deduper for the Bloom-filter design.
• facebookresearch/cc_net, the CCNet pipeline. The per-language KenLM perplexity bucketing is in cc_net/perplexity.py.
• google-research/deduplicate-text-datasets, the Lee et al. suffix-array substring dedup in Rust with Python drivers. Start from scripts/make_suffix_array.py.
• ChenghaoMou/text-dedup, compact reference implementations of MinHash, SimHash, and suffix-array dedup. The MinHash module is the production version of this page's first code block.
• NVIDIA/NeMo-Curator, GPU-accelerated curation. The fuzzy dedup module shows MinHash and connected components on RAPIDS.
• EleutherAI/lm-evaluation-harness, the standard eval harness. Read one task YAML end to end to see how scoring choices are encoded and versioned.
• lm-sys/FastChat, home of the MT-Bench judge. The judge prompts and the position-swap logic live under fastchat/llm_judge/.
• tatsu-lab/alpaca_eval, the AlpacaEval judge and the length-controlled regression, the cleanest small codebase for seeing bias correction implemented.

Common misconceptions

More data always beats better data. False at every scale that has been measured publicly. RefinedWeb matched curated corpora with filtered web data, FineWeb-Edu beat its own fifteen-trillion-token parent with a one-and-a-third trillion token filtered subset on knowledge benchmarks, and DataComp-LM's whole format exists because filtering choices move benchmarks more than data quantity once past the compute-optimal token count. Quantity matters when data is scarce relative to compute, quality dominates when it is not.

Deduplication is just about saving compute. Compute is the smallest of its three effects. Duplication reweights the training distribution toward the most-copied content, and it drives verbatim memorization, which Carlini et al. measured growing log-linearly in duplication count, making dedup a privacy and eval-integrity control as much as an efficiency one. And more of it is not monotonically better, FineWeb's ablations found per-snapshot dedup beat global dedup because aggressive global removal upsampled each snapshot's unique junk.

Exact-hash dedup gets most of the value. It catches only literal copies. The mass of web duplication is near-duplicate, the same article with a different date stamp, the same page with rotated ads, which only similarity-based methods find, and repeated spans inside otherwise distinct documents, which only substring methods find. Pipelines run all three because each catches a class the others provably cannot.

A judge that agrees with humans 80 percent of the time is an unbiased measurement instrument. Aggregate agreement and systematic bias are independent properties. A judge can match human majority preference on most pairs while consistently favoring the longer answer, the first position, and its own family's style, and those biases dominate exactly in the close comparisons that decide leaderboards. Raw agreement is also inflated by chance, which is why the calibration statistic is kappa, not percent agreement.

Temperature zero gives the model's best answer. Greedy decoding takes the mode of each next-token conditional, which is neither the most probable complete sequence nor the answer the model assigns the most total probability across reasoning paths. On multi-step problems the answer marginal is sharper than any single path, which is exactly why sampling several chains and voting beats one greedy chain, and the binomial arithmetic above says when.

Majority voting fixes a weak model. The vote amplifies the modal answer in whichever direction it points. Voting improves accuracy only when the correct answer beats the strongest wrong answer per sample, and when the model has a systematic error mode the vote converges confidently on the error, as the \( p = 0.45 \) computation in Problem 3 shows. Voting buys variance reduction, never bias correction.

A two-point lead on a few-hundred-item benchmark means the model is better. The Wilson interval on 200 items is about thirteen points wide, and the paired McNemar test on a realistic discordance pattern needs roughly an eight-point gap for significance at that size. Sub-interval deltas are noise, and treating them as signal is how teams ship regressions with celebratory release notes. Pair the comparison, test the discordant items, and size the eval to the effect you need to detect.

Self-check

References

1. Leskovec, J., Rajaraman, A., and Ullman, J. D. Mining of Massive Datasets, 3rd edition. Cambridge University Press, 2020. mmds.org. The textbook treatment of MinHash, LSH banding, and similarity search.
2. Efron, B. and Tibshirani, R. J. An Introduction to the Bootstrap. Chapman & Hall, 1993. The standard reference for resampling-based confidence intervals.
3. Broder, A. Z. On the Resemblance and Containment of Documents. Proceedings of SEQUENCES, 1997. doi:10.1109/SEQUEN.1997.666900. The MinHash paper.
4. Wilson, E. B. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22, 1927. doi:10.1080/01621459.1927.10502953.
5. Cohen, J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1), 1960. doi:10.1177/001316446002000104.
6. Dietterich, T. G. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1998. doi:10.1162/089976698300017197. The case for McNemar's test in classifier comparison.
7. Raffel, C., et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5 and C4). JMLR, 2020. arXiv:1910.10683.
8. Gao, L., et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. 2020. arXiv:2101.00027.
9. Rae, J. W., et al. Scaling Language Models: Methods, Analysis and Insights from Training Gopher (MassiveText and the Gopher filter rules). 2021. arXiv:2112.11446.
10. Wenzek, G., et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. LREC, 2020. arXiv:1911.00359.
11. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of Tricks for Efficient Text Classification (fastText). EACL, 2017. arXiv:1607.01759.
12. Barbaresi, A. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. ACL System Demonstrations, 2021. aclanthology.org/2021.acl-demo.15.
13. Penedo, G., et al. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. 2023. arXiv:2306.01116.
14. Penedo, G., Kydlicek, H., et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. 2024. arXiv:2406.17557. The companion technical blog post is at huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1.
15. Soldaini, L., et al. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. ACL, 2024. arXiv:2402.00159.
16. Li, J., et al. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. 2024. arXiv:2406.11794.
17. Lee, K., et al. Deduplicating Training Data Makes Language Models Better. ACL, 2022. arXiv:2107.06499.
18. Carlini, N., et al. Quantifying Memorization Across Neural Language Models. ICLR, 2023. arXiv:2202.07646.
19. Xie, S. M., et al. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS, 2023. arXiv:2305.10429.
20. Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023. arXiv:2306.05685.
21. Liu, Y., et al. G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. EMNLP, 2023. arXiv:2303.16634.
22. Kim, S., et al. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. EMNLP, 2024. arXiv:2405.01535.
23. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. 2024. arXiv:2404.04475.
24. Li, T., et al. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. 2024. arXiv:2406.11939.
25. Wang, X., et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023. arXiv:2203.11171.
26. Chen, M., et al. Evaluating Large Language Models Trained on Code (HumanEval and the pass@k estimator). 2021. arXiv:2107.03374.
27. Miller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. 2024. arXiv:2411.00640.

Key takeaway