lm-evaluation-harness walkthrough

Part I: The mental model

lm_eval (CLI)              __main__.py: parse --model, --tasks, --num_fewshot
      |
      v
simple_evaluate()          evaluator.py: build the model, resolve the tasks
      |
      v
get_model("hf")            api/registry.py -> HFLM / VLLM / OpenAI / Anthropic
      |
      v
TaskManager                index every lm_eval/tasks/**/*.yaml
      |
      v
ConfigurableTask           built from one YAML: doc_to_text, doc_to_choice,
      |                     metric_list, num_fewshot, filters
      v
build_all_requests         per doc: few-shot context + Instance(request_type)
      |
      v
lm.loglikelihood(...)      the model fulfils batched requests
lm.generate_until(...)     (three primitives, nothing task-specific)
      |
      v
filters -> process_results per-doc metrics (acc, exact_match, ...)
      |
      v
aggregation + bootstrap    one number per task, plus a standard error

The one-sentence identity: lm-evaluation-harness is the reference evaluation platform that separates what a benchmark asks from how a model answers, by reducing every task to three request primitives that any backend can implement, so the same YAML-defined benchmark runs identically against a local Hugging Face model, a vLLM server, or a closed API behind a paywall. A task never calls a model directly and a model never knows what task it is serving. In between sits a small vocabulary of requests, and that seam is the whole design.

Three load-bearing ideas follow from that seam. First, the request/response abstraction. A model backend implements exactly three methods, loglikelihood, loglikelihood_rolling, and generate_until, and every one of the hundreds of tasks in the repository is expressed in terms of those three. Add a new model type and every benchmark works. Add a new benchmark and every model runs it. Second, tasks are data, not code. A benchmark is a YAML file that names a dataset, a prompt template, a set of answer choices, and a list of metrics, so contributing a task is usually writing a config rather than writing Python. Third, and least glamorous but most important, the actual product is reproducibility. Tasks are versioned, the prompt that reached the model is logged, and multiple-choice scoring is a deterministic function of the model weights, so a score reported in a paper can in principle be regenerated byte-for-byte. The rest of this chapter is how those three ideas play out, and where reproducibility quietly breaks.

Everything here is written against a recent state of the main branch. The harness moved to a YAML task system and a ConfigurableTask engine in its v0.4 rewrite, and the project keeps evolving, so where a detail is likely to have moved I describe the component by its role rather than pinning an exact file path.

Part II: Using it

The harness is a pure Python package. It runs anywhere PyTorch and the Hugging Face stack run, including a laptop for the small models. Install from source when you plan to add tasks, or from PyPI when you just want to run them:

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

# or, without the source tree
pip install lm-eval

# optional backends and helpers are extras
pip install -e ".[vllm]"     # fast local inference
pip install -e ".[api]"      # OpenAI / Anthropic / generic API models
pip install -e ".[math]"     # answer checkers for MATH-style tasks

A first real session should be a tiny model on a couple of quick tasks. The command-line entry point is lm_eval. You pick a model type with --model, pass its constructor arguments as a comma-separated string to --model_args, and name the benchmarks with --tasks:

lm_eval --model hf \
  --model_args pretrained=EleutherAI/pythia-160m \
  --tasks lambada_openai,hellaswag \
  --device cuda:0 \
  --batch_size 8

Expect a burst of setup logging, the datasets downloading, the tasks building their requests, a progress bar per request type, and finally a formatted results table with one row per task and columns for each metric and its standard error. hellaswag reports acc and acc_norm, lambada_openai reports perplexity and acc. On a 160M model the numbers will be modest, which is the point, the mechanics are identical at every scale. The knobs you reach for constantly are few-shot count, batch size, a sample limit for smoke tests, and sample logging to inspect the exact prompt:

lm_eval --model hf \
  --model_args pretrained=EleutherAI/pythia-1.4b,dtype=bfloat16 \
  --tasks arc_challenge \
  --num_fewshot 25 \
  --batch_size auto \
  --limit 100 \
  --output_path results/ \
  --log_samples

--batch_size auto probes for the largest batch that fits and is worth using on real hardware. --limit 100 evaluates only the first hundred documents, which is for iterating on a config and never for a reported number. --log_samples writes every prompt, the model's response, and the per-document score to disk, and it is the single most useful flag when a score looks wrong, because you can read the literal text the model saw.

The same thing from Python, which is how you drive the harness inside a training run or a notebook:

from lm_eval import evaluator

results = evaluator.simple_evaluate(
    model="hf",
    model_args="pretrained=EleutherAI/pythia-160m",
    tasks=["hellaswag", "arc_easy"],
    num_fewshot=0,
    batch_size=8,
)
print(results["results"]["hellaswag"])
# {'acc,none': 0.29..., 'acc_stderr,none': 0.004...,
#  'acc_norm,none': 0.31..., 'acc_norm_stderr,none': 0.004..., ...}

You can also hand simple_evaluate an already constructed model object instead of a string, which avoids reloading weights you already have in memory:

from lm_eval.models.huggingface import HFLM

lm_obj = HFLM(pretrained="EleutherAI/pythia-160m", batch_size=16)
results = evaluator.simple_evaluate(model=lm_obj, tasks=["hellaswag"])

Now the mistakes people make. First, comparing a chat model scored with --apply_chat_template against a base model scored without it, which changes the prompt entirely and makes the two numbers incomparable. Second, reporting a --limit run, whose subset is not the benchmark. Third, mixing up acc and acc_norm, which can differ by several points on the same task and are not interchangeable. Fourth, and most fundamental, assuming that two frameworks reporting the same benchmark name produce the same number. The benchmark name is not the benchmark. The prompt template, the few-shot examples, the normalization, and the answer-extraction rule together are the benchmark, and the harness's job is to pin all of them down so a score means something.

Part III: When it is the right tool

lm-evaluation-harness is the right tool when you want a standardized, reproducible score that other people can compare against, when you are evaluating many models against many academic benchmarks, or when you need one backend that speaks to Hugging Face, vLLM, and hosted APIs with the same task definitions. It is the lingua franca. If a paper says a model got some HellaSwag or MMLU number, there is a good chance it came from here, and matching the harness's methodology is often the fastest way to be believed. It is also the natural place to add a new academic benchmark, because the YAML task system means a new task is usually a config plus a small preprocessing function.

The honest cases for alternatives. Stanford's HELM aims at holistic, multi-metric evaluation across many scenarios with calibration, bias, and efficiency measured alongside accuracy, and is the better fit when you want breadth of measured properties rather than a single leaderboard number. Hugging Face's lighteval covers similar ground to the harness with a different internal design and is worth knowing if you live in that ecosystem. OpenAI Evals is oriented around their API and is convenient when that is your only backend. For agentic and safety-style evaluations with tool use and multi-step tasks, the UK AISI's Inspect framework is built for that shape and the harness is not. And for pure generation-quality judgments, an LLM-as-judge setup or a purpose-built arena is a different instrument entirely. The harness is deliberately about likelihood-based and constrained-generation benchmarks with exact metrics, not open-ended quality.

The shape-of-the-problem warning is about generation cost and determinism. Multiple-choice tasks scored by loglikelihood are cheap and fully deterministic, one forward pass per choice, no sampling, the same answer every time. Generative tasks scored by generate_until are the opposite, they cost a full decode per document, they depend on stop sequences and decoding parameters, and if you sample with a temperature they are not even deterministic run to run. The failure mode that eats a weekend is running a large generative suite with a slow backend and a badly chosen stop sequence, so the model rambles past the answer and the regex extracts nothing:

cheap and deterministic          expensive and fragile
  multiple_choice                  generate_until
  loglikelihood(ctx, choice)       decode until a stop string
  argmax over choices              regex-extract the answer
  1 fwd pass per choice            1 full generation per doc
  same score every run             depends on stop seq + sampling

For the generative half, pairing the harness with a fast serving backend like vLLM or SGLang turns an overnight run into a coffee break, which is exactly why those backends are first-class model types.

Part IV: The full life of one evaluation

The specimen: one document of HellaSwag, a four-way multiple-choice task, evaluated zero-shot on a Hugging Face model, launched with lm_eval --model hf --tasks hellaswag. HellaSwag gives a context and four possible endings, exactly one of which is the natural continuation. Follow one question all the way down and back up.

Stage 1: the CLI and simple_evaluate

lm_eval/__main__.py parses the arguments and calls cli_evaluate, which is a thin wrapper over evaluator.simple_evaluate in lm_eval/evaluator.py. That function is the front door for both the CLI and library use. It resolves the model, resolves the tasks, sets the random seeds (there are separate seeds for Python, NumPy, and Torch, plus a few-shot sampling seed, all logged for reproducibility), and then hands off to the lower-level evaluate. Everything downstream is SPMD-free ordinary Python, one process orchestrating batched calls to a model.

Stage 2: resolving the model through the registry

--model hf is looked up in the model registry in lm_eval/api/registry.py. Model classes register themselves with a @register_model("hf", ...) decorator, so the string hf maps to the HFLM class in lm_eval/models/huggingface.py. The --model_args string is parsed into keyword arguments and passed to its constructor, which loads the tokenizer and weights, figures out the device and dtype, and sets the batching strategy. HFLM is a subclass of TemplateLM, which is itself a subclass of the abstract LM base class in lm_eval/api/model.py. That inheritance is the point. The evaluator only ever sees the LM interface, so hf, vllm, openai-completions, and anthropic are interchangeable at the seam.

Stage 3: resolving the task through the TaskManager

--tasks hellaswag is resolved by the TaskManager in the lm_eval/tasks package, which at import time indexes every YAML file under lm_eval/tasks/** and builds a name-to-config map. The hellaswag YAML names its dataset on the Hugging Face Hub, its splits, its prompt template, its choice list, and its metrics. The manager instantiates a ConfigurableTask (lm_eval/api/task.py) from that config. The task's __init__ loads the dataset via the datasets library, runs any process_docs preprocessing declared in a sibling utils.py, and stores the evaluation split. Because output_type is multiple_choice, the task knows each document will emit several loglikelihood requests, one per candidate ending.

Stage 4: building the request for one document

build_all_requests iterates the evaluation split. For our document it first builds the context. Zero-shot, the context is just the prompt template applied to the doc, doc_to_text(doc), which for HellaSwag stitches the activity label and the sentence beginning into a single string. Then it reads the candidate endings with doc_to_choice(doc) and the gold index with doc_to_target(doc). Now construct_requests emits one Instance per choice:

# conceptually, for a 4-way multiple_choice doc
requests = [
    Instance(
        request_type="loglikelihood",
        doc=doc,
        arguments=(context, " " + ending),   # note the leading space
        idx=i,
    )
    for i, ending in enumerate(choices)
]

Each Instance (lm_eval/api/instance.py) is a small dataclass carrying its request type, the source document, the argument tuple the model will consume, an index, and empty slots for the raw response (resps) and the filtered response (filtered_resps). The argument tuple for a loglikelihood request is (context, continuation). The leading space on the continuation is not a detail, it is the difference between the model scoring " Paris" and "Paris", which tokenize differently, and getting it wrong silently shifts every score. This is the sort of thing the harness exists to standardize.

Stage 5: the model fulfils the requests

The evaluator collects every request of a given type across all documents and all tasks, then makes one batched call, lm.loglikelihood(all_ll_instances). Inside HFLM, each (context, continuation) pair is handled by tokenizing the two together, running a single forward pass, and reading off the log-probabilities. For each continuation token the model looks up the log-probability the network assigned to that exact token given everything before it, and sums those over the continuation. The return value per request is a tuple (loglikelihood, is_greedy), the summed log-probability of the continuation and a boolean saying whether every continuation token was also the argmax at its position. Crucially there is no generation and no sampling. This is a scoring operation, so it is deterministic given the weights, which is why loglikelihood multiple-choice is the reproducible core of the whole harness. The mathematics of turning logits into these per-token log-probabilities is the softmax story built up in the ML implementations section.

Two efficiency details live here. Requests are sorted by length before batching so that similar-length sequences pack together, then the responses are un-sorted back into the original order, a Collator in the model utilities. And with --use_cache a SQLite cache keyed on the request can return a previously computed loglikelihood, so re-running a task after a code change to an unrelated part costs nothing.

Stage 6: filters and process_results

The four loglikelihoods flow back into their Instances' resps. For a multiple-choice task the filter stage is trivial, the raw responses pass straight through. Then the task's process_results(doc, results) runs. It receives the four loglikelihoods, takes the argmax, and compares it to the gold index to produce acc, a 1.0 or 0.0. It also computes acc_norm, the argmax after dividing each loglikelihood by the byte length of its ending, which corrects the model's built-in bias toward shorter continuations. Both land in a per-document dict, {"acc": 1.0, "acc_norm": 0.0} for a document the model got right raw but wrong once length-normalized.

Stage 7: aggregation and the standard error

After every document is scored, the per-document values are aggregated. Each metric is paired with an aggregation function in the task config, and for acc and acc_norm that is the mean. The harness also computes a standard error for each metric, a closed-form standard error of the mean for accuracy and a bootstrap resample for metrics without a clean formula, controlled by bootstrap_iters. If hellaswag were part of a group like mmlu, a second aggregation would combine the subtask means into a group score, optionally weighted by subtask size. The result is a nested dictionary keyed by task, then metric, that becomes the printed table. That closes the loop of one evaluation, a YAML file in, one Instance per choice, four forward-pass log-probabilities, an argmax, and a number with an error bar out.

Part V: Internals deep dives

Deep dive: the request/response model abstraction

The entire harness pivots on a tiny interface. The abstract LM class in lm_eval/api/model.py declares three request-fulfilling methods, and every model backend implements them:

class LM(abc.ABC):
    @abc.abstractmethod
    def loglikelihood(self, requests) -> list[tuple[float, bool]]:
        """Score a continuation given a context.
        Each request arg is (context, continuation).
        Returns (sum log-prob of continuation, is_greedy)."""

    @abc.abstractmethod
    def loglikelihood_rolling(self, requests) -> list[float]:
        """Score a whole string with a sliding window.
        Each request arg is (string,). Used for perplexity."""

    @abc.abstractmethod
    def generate_until(self, requests) -> list[str]:
        """Generate text until a stop string.
        Each request arg is (context, gen_kwargs dict)."""

Those three cover every task shape. loglikelihood powers multiple-choice and any task where you score a fixed target string. loglikelihood_rolling powers perplexity over long documents, where the string is windowed to the model's context length and the per-token log-probs are summed across windows. generate_until powers everything open-ended, math word problems, code, free-form question answering, where the model decodes until it emits one of the stop strings and the answer is extracted afterward. There is a fourth task-level output type, multiple_choice, but it is not a fourth model method, it desugars into a batch of loglikelihood requests, one per choice, as we saw in Part IV.

Because a model is defined entirely by how it answers these three questions, adding a backend is a self-contained exercise and every existing benchmark immediately works against it, while adding a benchmark requires knowing nothing about any model. The intermediate TemplateLM class factors out the common plumbing for language models that expose token-level logits, so a new local backend usually only has to implement token encoding, the low-level _loglikelihood_tokens over token ids, and generate_until. This is why the lm_eval/models directory can hold Hugging Face transformers, vLLM, SGLang, GGUF via llama.cpp, NeMo, Mamba, OpenAI and Anthropic APIs, and more, each a few hundred lines, all interchangeable at the evaluator.

Deep dive: tasks as data, the YAML config system

A benchmark is a YAML file consumed by ConfigurableTask. The config is parsed into a TaskConfig dataclass and the fields are the entire contract between a benchmark and the engine. A trimmed multiple-choice task looks like this:

task: my_science_qa
dataset_path: some-org/science-qa     # a Hugging Face dataset
output_type: multiple_choice
training_split: train                 # pool for few-shot examples
validation_split: validation
test_split: test
doc_to_text: "Question: {{question}}\nAnswer:"
doc_to_choice: "{{choices}}"          # a list field on the doc
doc_to_target: "{{label}}"            # the index of the correct choice
num_fewshot: 0
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0

The three doc_to_* fields are the heart of it. Each can be a plain string, a Jinja2 template rendered against the document (the {{question}} above), or a reference to a Python function in a sibling file via a !function utils.my_fn tag when the formatting is too complex for a template. doc_to_text produces the prompt, doc_to_choice produces the list of candidate answers, and doc_to_target produces the gold answer, as an index into the choices for multiple-choice or as a target string otherwise. The version in metadata is not decoration, it is stamped into the results so a score is tied to the exact task definition that produced it, and it is bumped whenever the prompt or scoring changes in a way that would move numbers.

The YAML system has an inheritance mechanism that carries the real weight of the repository. A task can declare include: _default_template_yaml to pull in a shared base config and override a few fields. MMLU is the canonical example, fifty-seven subject subtasks that each include one common template and differ only in their dataset subset name. That is how the lm_eval/tasks tree holds well over a thousand task and subtask YAMLs without a thousand copies of the same prompt. Groups compose on top, a group YAML lists member tasks and an aggregation policy, so mmlu is a group whose members are the fifty-seven subject tasks and whose score is their mean. The harness also distinguishes loose tags, which let you run a bundle of related tasks without producing an aggregate number, from real groups, which do aggregate. Getting that distinction right matters, because averaging across tasks that measure different things with different baselines produces a number that looks meaningful and is not.

For a generative task the config swaps the choice fields for a generation and extraction spec. A math-word-problem task in the GSM8K style looks roughly like this:

task: gsm8k_like
dataset_path: gsm8k
output_type: generate_until
doc_to_text: "Question: {{question}}\nAnswer:"
doc_to_target: "{{answer}}"
generation_kwargs:
  until:
    - "\n\n"          # stop when the model starts a new problem
  do_sample: false    # greedy decoding for determinism
filter_list:
  - name: strict-match
    filter:
      - function: regex
        regex_pattern: "answer is (\\-?[0-9\\.,]+)"
      - function: take_first
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true

The until list is the set of stop strings, the filter_list is how a free-form generation is turned into a comparable answer, and exact_match compares the extracted answer against the target. That filter pipeline is its own subsystem, covered below.

Deep dive: few-shot construction and samplers

Few-shot prompting is where reproducibility either holds or quietly slips, so the harness makes the construction explicit. When num_fewshot is greater than zero, the task asks a Sampler (lm_eval/api/samplers.py) for that many example documents drawn from the few-shot pool, which is the fewshot_split if the config names one, otherwise the training split, otherwise the validation split. Each example is rendered exactly the way a real document would be, doc_to_text(example) then a target delimiter then doc_to_target(example), and the examples are joined with a few-shot delimiter, by default a blank line. Any task description is prepended, and then the actual question's context is appended. The whole thing is one string that becomes the context of the request.

context for a 2-shot multiple_choice question:

[description]
[doc_to_text(ex1)][target_delim][doc_to_target(ex1)]
[fewshot_delim]
[doc_to_text(ex2)][target_delim][doc_to_target(ex2)]
[fewshot_delim]
[doc_to_text(the actual question)]      <- continuation scored here

The sampler is where determinism is enforced. The default draws a fixed set of examples with a seeded choice, and the seed is logged, so the same few-shot examples appear every run. Alternative samplers are selectable in the config, a first-n sampler, a random sampler, a balanced sampler that spreads examples across labels, and a manual sampler that pins a hand-chosen set. The sampler also guarantees a document never appears as its own few-shot example. The subtle correctness point is that few-shot choice is part of the benchmark, not a free parameter, so two runs that differ only in which examples were sampled are measuring two different things even though they carry the same task name.

Chat models complicate this. With --apply_chat_template the context is rendered through the model's tokenizer chat template rather than concatenated as plain text, and with --fewshot_as_multiturn each few-shot example becomes a separate user and assistant turn instead of being folded into one block. Both change the literal tokens the model sees, which is why a base-model number and a chat-template number for the same task are not comparable, a trap worth repeating because it is the most common way people accidentally lie with harness numbers.

Deep dive: metrics, filters, and aggregation

The scoring pipeline has two stages, a per-document filter and extraction stage, then a metric and aggregation stage. Filters live in lm_eval/filters and are how a raw generation becomes a comparable answer. A filter_list entry names a pipeline of steps, and the common ones are a regex extractor that pulls the first capture group matching a pattern, a take_first selector, a majority-vote selector for self-consistency setups, and various whitespace and formatting transforms. GSM8K famously ships two filter pipelines, a strict one that requires the model to say the answer in a specific phrasing and a flexible one that grabs the last number in the output, and it reports both, which is a neat illustration that answer extraction is a modeling choice with its own score.

Metrics live in lm_eval/api/metrics.py and register themselves in the registry alongside an aggregation function and a higher_is_better flag. The vocabulary you will actually meet:

Metric	What it measures	Typical task shape
acc	argmax loglikelihood equals gold	multiple_choice
acc_norm	acc after length-normalizing each choice	multiple_choice
perplexity / word_perplexity	exp of mean negative log-prob per token or word	loglikelihood_rolling
bits_per_byte	compression view of the same log-probs	loglikelihood_rolling
exact_match	extracted answer equals target string	generate_until
f1	token-overlap F1 against a reference	generate_until

Aggregation is per metric, usually the mean, with weighted variants for perplexity that respect token counts. The standard error matters as much as the point estimate. For a mean of zero-one accuracies the harness uses the closed-form standard error of the mean, and for metrics without a clean formula it resamples the per-document scores with replacement, bootstrap_iters times by default a large number, to estimate the spread. Two models whose scores differ by less than the error bars have not been distinguished, and the harness printing the standard error next to every number is a quiet argument against over-reading a leaderboard's third decimal place. Group aggregation stacks on top, combining member-task means either by a simple average or weighted by the number of documents in each member, a choice the group config makes explicit because it changes what the group number means.

Deep dive: why reproducible eval is genuinely hard

This is the deep dive the whole chapter has been circling. The harness is elaborate not because scoring a model is intrinsically complex, a single argmax is not, but because every choice around that argmax moves the number, and pinning those choices down is the actual engineering. Here are the ones that bite.

Prompt formatting. The exact template, the Question: and Answer: labels, the newline placement, the target delimiter, whether the continuation carries a leading space, all shift how the model tokenizes and therefore what probability it assigns. The most cited case is MMLU, which Hugging Face investigated when the Open LLM Leaderboard first launched. The same model evaluated by three faithful implementations, the original Hendrycks code, HELM, and this harness, produced markedly different MMLU scores, differing by more than ten points for some models, purely from prompt format and from whether the score compared generated letters, the loglikelihood of a single letter, or the loglikelihood of the full answer text. None of the three was buggy. They were three different measurements wearing one name.

Normalization. Language models assign higher total probability to shorter strings, so a raw-loglikelihood argmax over answer choices is biased toward the short ones. acc_norm divides by byte length to counter this, and there is a mutual-information variant that also conditions out the answer's unconditional probability. Which normalization a leaderboard reports is a real decision, and comparing an acc from one source to an acc_norm from another is a category error that happens constantly.

Answer extraction. For generative tasks the score depends entirely on the regex or checker that turns free text into an answer. A model that computes the right answer but phrases it in a way the strict filter misses scores zero, so the extraction rule is part of the benchmark. This is why generative benchmarks report the filter they used and why two GSM8K numbers can disagree even with identical generations.

Chat templates and system prompts. Applying a chat template, or a system instruction, or rendering few-shot as turns, all change the literal input. A model can look better or worse by several points depending on whether it was evaluated the way it was trained to be prompted, and there is no single correct answer, only a choice that must be reported.

Tokenization edge cases. Leading whitespace, byte-pair merges across the context-continuation boundary, and special tokens all interact with how the continuation's log-probs are computed. The harness tokenizes context and continuation together and then splits, precisely to avoid a class of boundary bugs, and this is the kind of correctness detail a reference implementation exists to get right once for everyone.

Contamination. If the evaluation data leaked into the model's training corpus, the score measures memorization rather than capability. The harness provides hooks, should_decontaminate and a doc_to_decontamination_query in the task config, and an n-gram overlap check that compares evaluation documents against a supplied training corpus. But the hook only helps if you actually have the training data to check against, and for most released models you do not. Contamination is the one reproducibility problem the harness cannot solve on its own, because it is not a property of the evaluation code, it is a property of data you usually cannot see. The honest posture is to version tasks, log the exact prompts, report the normalization and the filter, and treat any single benchmark number as one noisy measurement among many.

Deep dive: the Open LLM Leaderboard

The clearest demonstration of the harness's value is that Hugging Face built the Open LLM Leaderboard on top of it. Rather than invent scoring, the leaderboard runs the harness on a fixed set of tasks with fixed few-shot counts and pins the harness version, so every submitted model is measured the exact same way. The first leaderboard used a suite of well-known academic benchmarks in the ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, and GSM8K family, and a later revision moved to harder and less saturated tasks as models caught up to the old ones, including math, graduate-level reasoning, and instruction-following benchmarks. The exact composition has changed over time and is best checked against the current leaderboard, but the architecture is the durable lesson. A leaderboard is only as reproducible as the harness under it, and the reason thousands of models can be ranked on one page is that a versioned task plus a versioned harness plus logged prompts makes each score a regenerable artifact rather than a claim.

Part VI: Reading the repository

The harness is larger than it looks because most of the size is the task library, hundreds of YAML files that you read on demand, not front to back. The engine itself is small and worth reading completely.

Stage 0, orientation. Read the top-level README.md and the docs on the task interface, then run one small evaluation with --log_samples and open the written sample file. Seeing the literal prompt a model received, next to its response and score, orients you faster than any amount of code reading. Questions to hold, what string did the model actually see, and where did each piece of it come from.

Stage 1, the evaluator. Read lm_eval/__main__.py then lm_eval/evaluator.py, following simple_evaluate into evaluate. Questions, where are the tasks turned into requests, where are requests of the same type collected across tasks, and where do responses get routed back to their Instances.

Stage 2, the abstractions. Read lm_eval/api/model.py for the three primitives and TemplateLM, lm_eval/api/instance.py for the request object, and lm_eval/api/task.py for ConfigurableTask and TaskConfig. This trio is the conceptual core. Questions, what exactly is in an Instance, how does construct_requests differ across output types, and how does a multiple-choice task become loglikelihood requests.

Stage 3, one model backend. Read lm_eval/models/huggingface.py end to end, in particular the loglikelihood path that tokenizes a pair, runs a forward pass, and reads per-token log-probs, and the batching and Collator logic in the model utilities. Then skim the vllm and an API model to see the same interface met three different ways. Questions, how is is_greedy computed, and how does the pair get tokenized to avoid boundary bugs.

Stage 4, one task family end to end. Pick a multiple-choice task like hellaswag and a generative task like gsm8k, and read their YAML plus any sibling utils.py. Then read the MMLU directory to see the include inheritance and the group config. Questions, what does doc_to_text render to, which filters run on the generation, and how do the subtasks aggregate into a group.

Stage 5, scoring and sampling. Read lm_eval/api/metrics.py, lm_eval/api/samplers.py, and the lm_eval/filters package. Questions, which metrics use a bootstrap standard error and which use a closed form, how does the default sampler stay deterministic, and what does a filter pipeline look like as objects.

Where not to start. Do not begin in the task library, the hundreds of YAMLs are a reference you consult once you know the engine, not a tutorial. The decontamination module is important conceptually and lightly used in practice, so meet it after the core. And the many model backends are variations on one interface, so read one deeply rather than all shallowly.

Part VII: Hands-on labs

Labs 1 through 4 run on a single small GPU or even CPU with a tiny model. Labs 5 and 6 are about reading behavior, not scale. Log formats and default numbers shift with the fast pace of main.

Lab 1: a first score with the samples logged. Concept: the evaluation lifecycle of Part IV.

lm_eval --model hf \
  --model_args pretrained=EleutherAI/pythia-160m \
  --tasks hellaswag \
  --limit 50 \
  --log_samples \
  --output_path out/

Open the sample file under out/ and find one document. Read the exact context string, the four continuations, the four loglikelihoods, and the resulting acc and acc_norm. Confirm by hand that acc is the argmax of the raw loglikelihoods and acc_norm is the argmax after dividing by the length of each ending. Find one document where the two disagree.

Lab 2: few-shot changes the prompt. Concept: few-shot construction and samplers.

lm_eval --model hf --model_args pretrained=EleutherAI/pythia-410m \
  --tasks arc_easy --num_fewshot 0 --limit 100 --log_samples --output_path out0/
lm_eval --model hf --model_args pretrained=EleutherAI/pythia-410m \
  --tasks arc_easy --num_fewshot 5 --limit 100 --log_samples --output_path out5/

Diff a single document's context between the two sample files and watch five formatted examples appear before the question. Note the few-shot delimiter and the target delimiter in the literal text. Then re-run the five-shot command a second time and confirm the few-shot examples are identical, which is the seeded sampler doing its job.

Lab 3: loglikelihood by hand. Concept: the scoring primitive with no task around it.

# ll_lab.py
from lm_eval.models.huggingface import HFLM
from lm_eval.api.instance import Instance

lm = HFLM(pretrained="EleutherAI/pythia-160m")
reqs = [
    Instance("loglikelihood", {}, ("The capital of France is", " Paris"), 0),
    Instance("loglikelihood", {}, ("The capital of France is", " London"), 1),
]
for r, (ll, greedy) in zip(reqs, lm.loglikelihood(reqs)):
    print(r.arguments[1], "->", round(ll, 3), "is_greedy:", greedy)

The correct continuation should have the higher (less negative) loglikelihood and likely is_greedy=True. This is exactly what a multiple-choice task does per choice, with the task machinery stripped away. Change the continuations and predict the ordering before running.

Lab 4: a generative task and its filter. Concept: generate_until plus answer extraction.

lm_eval --model hf --model_args pretrained=EleutherAI/pythia-1.4b \
  --tasks gsm8k --num_fewshot 5 --limit 20 --log_samples --output_path outg/

Read the generations. Find one where the model reached the right answer but the strict filter failed to extract it, and one where the flexible filter succeeded. This is the answer-extraction problem of Part V made concrete, and it explains why GSM8K reports two numbers.

Lab 5: acc versus acc_norm as a story. Concept: length normalization.

Take the HellaSwag sample file from Lab 1 and sort documents by the length spread of their four endings. Confirm that acc and acc_norm diverge most on documents where the endings differ a lot in length, and agree on documents where they are similar. You have just rediscovered why the normalization exists, the raw score rewards short endings.

Lab 6: the same task, two backends. Concept: the model abstraction is real.

# scored via Hugging Face transformers
lm_eval --model hf --model_args pretrained=EleutherAI/pythia-410m \
  --tasks hellaswag --limit 200 --output_path hf/
# scored via vLLM, same weights, same task (needs the vllm extra)
lm_eval --model vllm --model_args pretrained=EleutherAI/pythia-410m \
  --tasks hellaswag --limit 200 --output_path vllm/

The two acc numbers should match to within tiny numerical noise, because both backends implement the same loglikelihood primitive over the same weights. When they do not match, the difference is a real bug in one backend's scoring, and catching exactly that is one reason the abstraction is drawn where it is.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is lm-evaluation-harness, in one sentence?

The de facto standard evaluation platform that separates benchmarks from models by reducing every task to three request primitives, loglikelihood, loglikelihood_rolling, and generate_until, so YAML-defined tasks run identically against any backend, and it is the engine behind the Open LLM Leaderboard.

2. What are the three request primitives and what does each power?

loglikelihood scores a continuation given a context and powers multiple-choice and fixed-target tasks. loglikelihood_rolling scores a whole string with a sliding window and powers perplexity. generate_until decodes until a stop string and powers open-ended tasks whose answers are extracted afterward.

3. If multiple_choice is an output type, why is it not a fourth model method?

Because a multiple-choice document desugars into one loglikelihood request per candidate answer, and the task picks the argmax over the returned loglikelihoods. The model only ever answers the three primitives, so it never needs to know a task was multiple-choice.

4. Why is loglikelihood scoring deterministic while generate_until may not be?

Loglikelihood is a single forward pass that reads off the model's assigned probability for a fixed continuation, no sampling involved, so it returns the same value every run given the weights. generate_until decodes tokens, and if it samples with a temperature the output and therefore the score can vary run to run.

5. What is the difference between acc and acc_norm, and why does acc_norm exist?

acc is the argmax over raw continuation loglikelihoods. acc_norm divides each loglikelihood by the byte length of its choice before the argmax. It exists because models assign higher total probability to shorter strings, so raw loglikelihood is biased toward short answers and length normalization removes that bias.

6. A task is a YAML file. Name the fields that define its behavior.

The dataset path and splits, the output type, doc_to_text for the prompt, doc_to_choice for the candidates, doc_to_target for the gold answer, num_fewshot, a metric_list with aggregations, an optional filter_list for extraction, and a metadata version. Templates are Jinja2 or Python functions via a !function tag.

7. Why is few-shot example choice part of the benchmark rather than a free parameter?

Because different few-shot examples produce different prompts and therefore different scores, so a run is only comparable to another if the sampled examples match. The harness enforces this with a seeded sampler whose seed is logged, and by excluding a document from being its own example.

8. Why did three faithful MMLU implementations produce different scores for the same model?

Because they differed in prompt format and in what they scored, one generated a letter and compared it, one compared the loglikelihood of a single letter, and one compared the loglikelihood of the full answer text. None was buggy, they were three different measurements, which is exactly why the harness standardizes and versions the prompt and the scoring.

9. What does a filter pipeline do, and why does GSM8K report two numbers?

A filter pipeline turns a free-form generation into a comparable answer, typically a regex extraction followed by a selection like take-first. GSM8K ships a strict pipeline that requires a specific phrasing and a flexible one that grabs the last number, and because answer extraction is itself a modeling choice, the two pipelines can score the same generations differently, so both are reported.

10. Why does the harness print a standard error next to every score?

Because a benchmark score is an estimate from a finite sample, and two models whose scores differ by less than their error bars have not actually been distinguished. Accuracy uses a closed-form standard error of the mean, and metrics without a clean formula use a bootstrap over the per-document scores.

11. Why can the harness not fully solve the contamination problem?

Because contamination is a property of the model's training data, not of the evaluation code. The harness offers n-gram overlap hooks, but they only work if you can supply the training corpus to check against, and for most released models that data is not available.

12. Adding a new model backend versus adding a new benchmark, what does each require?

A new backend implements the three primitives (often just token encoding, low-level token loglikelihood, and generate_until via TemplateLM) and then every existing task works. A new benchmark is usually a YAML config plus a small preprocessing function and needs to know nothing about any model. The seam between them is the whole design.

13. When would you reach for HELM, lighteval, or Inspect instead?

HELM when you want holistic multi-metric evaluation across many scenarios rather than a single number. lighteval when you are in the Hugging Face ecosystem and prefer its design. Inspect when you need agentic, tool-using, multi-step safety-style evaluations, which are a different shape than the harness's likelihood and constrained-generation benchmarks.

14. Two backends give different acc on the same task and weights. What does that mean?

It means one backend's loglikelihood computation is wrong, because scoring a fixed continuation under fixed weights should be backend-independent up to tiny numerical noise. The shared interface is precisely what makes that discrepancy a detectable bug rather than an unexplained mystery.

Part IX: Design lessons

Put the seam between task and model in one small interface. Three request primitives is the entire contract, and everything else, hundreds of tasks and a dozen backends, hangs off it independently. This is dependency inversion at its cleanest, the same instinct as a database driver interface or a filesystem abstraction, define the narrow thing both sides agree on and let each side grow without the other.

Make the benchmark data, not code. A task is a YAML config, so contributing one is filling in fields rather than writing a program, and the config can be diffed, versioned, and inherited. The lesson generalizes to any place where the variety lives in configuration and the logic is shared, express the variety as data and write the engine once.

Version the thing that produces the number. A task carries a version stamped into every result, so a score is tied to the exact prompt and scoring that made it. Wherever a reported measurement can drift silently, benchmark suites, data pipelines, API contracts, versioning the producer turns an irreproducible claim into a regenerable artifact.

Standardize the details that quietly move the answer. Leading spaces, tokenization boundaries, length normalization, stop strings, the harness gets each right once so a thousand downstream users do not each get them wrong differently. A reference implementation earns its status by absorbing exactly the tedious correctness details nobody wants to rediscover.

Report the uncertainty, not just the point. A standard error next to every score is a structural argument against over-reading a leaderboard. Systems that surface their own confidence get trusted correctly, and the ones that print a bare number invite false precision.

Name the limits you cannot fix. Contamination is outside the code's reach, and the honest move is to provide the hooks, document the gap, and treat any single number as one noisy measurement. A tool that is candid about what it cannot guarantee is more useful than one that pretends completeness.

Part X: Memorization framework

The one-sentence summary: lm-evaluation-harness resolves a model to three request primitives and a benchmark to a YAML task, turns each document into loglikelihood or generate_until requests built on a seeded few-shot context, batches them through the model, filters and scores the responses per document, and aggregates into one versioned number with a standard error.

lm_eval CLI -> simple_evaluate -> get_model + TaskManager
  -> ConfigurableTask from YAML (doc_to_text / choice / target)
  -> build_all_requests: few-shot ctx + Instance(request_type)
  -> lm.loglikelihood / lm.generate_until  (batched, three primitives)
  -> filters -> process_results (acc / acc_norm / exact_match)
  -> aggregation + bootstrap stderr -> results table

The chain mapped to source:

entry            lm_eval/__main__.py, lm_eval/evaluator.py
model registry   lm_eval/api/registry.py -> lm_eval/models/*.py
model interface  lm_eval/api/model.py (LM, TemplateLM)
request object   lm_eval/api/instance.py (Instance)
task engine      lm_eval/api/task.py (ConfigurableTask, TaskConfig)
task library     lm_eval/tasks/**/*.yaml (+ sibling utils.py)
few-shot         lm_eval/api/samplers.py
scoring          lm_eval/api/metrics.py, lm_eval/filters/

Memorize these blocks:

Three primitives: loglikelihood (score a continuation, multiple-choice), loglikelihood_rolling (windowed perplexity), generate_until (decode to a stop string, extract the answer).
multiple_choice desugars: one loglikelihood request per choice, then argmax for acc and length-normalized argmax for acc_norm.
Task is YAML: dataset, splits, doc_to_text / doc_to_choice / doc_to_target, num_fewshot, metric_list, filter_list, version.
Few-shot is seeded: examples drawn deterministically from the few-shot pool, excluding the document itself, so runs are comparable.
Reproducibility breaks at: prompt format, normalization (acc vs acc_norm), answer extraction, chat templates, tokenization boundaries, and contamination, which is the one the code cannot fix.

Part XI: Papers and further reading

The benchmarks and the methodology in this walkthrough each trace back to a paper, and every one rewards a direct read. Where this site treats the same idea in depth, the companion link points there.

Gao et al., A framework for few-shot language model evaluation, Zenodo, 2023. The harness's own citable record, the reference hundreds of papers point at when they report a score.
Biderman et al., Lessons from the Trenches on Reproducible Evaluation of Language Models, 2024. The maintainers' account of where evaluation breaks and how the harness's design answers each failure, essentially Part V of this chapter in paper form. The statistics of comparing noisy scores are worked through in the data pipelines and evaluation class on this site.
Brown et al., Language Models are Few-Shot Learners, 2020. The GPT-3 paper that defined the few-shot prompting setting the harness's seeded sampler machinery standardizes.
Hendrycks et al., Measuring Massive Multitask Language Understanding, 2020. The MMLU benchmark, fifty-seven subjects that became the canonical example of both YAML inheritance and prompt-format divergence.
Zellers et al., HellaSwag, Can a Machine Really Finish Your Sentence?, 2019. The four-way completion task this chapter traces end to end in Part IV.
Clark et al., Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018. The grade-school science benchmark behind arc_easy and arc_challenge.
Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021. The GSM8K dataset whose strict and flexible filters made answer extraction visible as a modeling choice, and the same dataset drives the RL runs in the verl walkthrough.
Lin et al., TruthfulQA, Measuring How Models Mimic Human Falsehoods, 2021. The falsehood-mimicry benchmark from the first Open LLM Leaderboard suite.
Liang et al., Holistic Evaluation of Language Models, 2022. Stanford's HELM, the breadth-first alternative whose MMLU numbers famously disagreed with the harness's for the same models.
Sclar et al., Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, 2023. Measures accuracy swings of tens of points from formatting changes alone, the quantified case for pinning the prompt.
Oren et al., Proving Test Set Contamination in Black Box Language Models, 2023. A statistical test for contamination that needs no access to the training corpus, aimed at exactly the gap the harness's n-gram hooks cannot close.

Part XII: Final takeaway

If the single-model pieces this repository scores are the gap, the ML implementations section builds the softmax and log-probability machinery the loglikelihood path depends on, and the serving backends that make generative evaluation fast are the subject of the vLLM and SGLang chapters. Then come back and read evaluator.py once more. It will read like a plain loop that asks a model three kinds of question and counts the answers, which is the entire point.

Key takeaway: lm-evaluation-harness shows that a trustworthy benchmark number is not an argmax, it is everything around the argmax pinned down. Reduce every model to three request primitives and every benchmark to a versioned YAML task, log the exact prompt, standardize the tokenization and normalization, and report the standard error, and a score stops being a claim and becomes a regenerable artifact, which is why one harness can rank a thousand models on one page.