Transformers

Part I: The mental model

"Qwen/Qwen2.5-1.5B-Instruct"          a string, a name on the Hub
        │
        ▼
config.json ──► AutoConfig ──► model_type: "qwen2"     identity resolution
        │
        ▼
Auto registry ──► Qwen2ForCausalLM                     class resolution
        │
        ▼
model.safetensors ──► from_pretrained ──► weights      state resolution
        │
        ▼
tokenizer.json ──► token ids ──► generate() loop       the actual work
        │              ┌──────────────────────┐
        ▼              │ forward → logits →   │  one iteration
   decoded text ◄──────│ process → sample →   │  per new token
                       │ append → check stop  │
                       └──────────────────────┘

The one-sentence identity: Transformers is a registry of hundreds of deliberately self-contained PyTorch model files behind one loading interface. Everything the library does is a consequence of that sentence. The loading interface, from_pretrained, works because every checkpoint on the Hub carries its own identity in a config.json whose model_type field names its architecture, so a string is enough to resolve a config class, a tokenizer class, a model class, and a set of weight files. The registry works because each architecture lives in its own directory under src/transformers/models/ as plain, repetitive, readable PyTorch, and a small set of shared base classes (PreTrainedModel, PreTrainedConfig, GenerationMixin) supplies the machinery every model needs but no researcher wants to reread.

Three layers sit on top of the model zoo. Tokenizers turn text into ids and back, mostly by delegating to the Rust tokenizers backend through a tokenizer.json file shipped in the checkpoint. generate() is the autoregressive decoding loop shared by every causal model, with sampling strategies and stopping rules expressed as composable objects. Pipelines and Trainer are the convenience layers, gluing preprocessing, the model, and postprocessing behind a task name, or wrapping a full training loop around any model.

One version note before anything else. Transformers v5, current as of this writing, is PyTorch-only: the TensorFlow and Flax modeling code that older tutorials mention was removed, which you can confirm by noticing modeling_tf_utils.py no longer exists in the tree. If a blog post tells you about TFAutoModel, it is describing v4.

Part II: Using it, a practical tutorial

Install the library together with its backend. On Linux and macOS the command is the same; the [torch] extra pulls in PyTorch and accelerate:

python -m venv .venv && source .venv/bin/activate
pip install "transformers[torch]"

On macOS this gives you Metal (MPS) acceleration through the regular PyTorch wheel; on Linux, pick the PyTorch build matching your CUDA version first if you care which one you get. The first real session is one call. pipeline bundles preprocessing, the model, and postprocessing behind a task name:

from transformers import pipeline

generator = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B-Instruct")
out = generator("The secret to a good walkthrough is", max_new_tokens=40)
print(out[0]["generated_text"])

The first run downloads roughly 3 GB of files into ~/.cache/huggingface/hub with a progress bar per file; subsequent runs load from cache and print nothing but your text. The output is a list with one dict per input, and generated_text includes your prompt followed by the continuation. Exact continuations vary because sampling is stochastic.

One level down is the pairing you will actually use in real code, a tokenizer and a model loaded by name, with generation invoked explicitly so you control the knobs:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": "Explain attention in two sentences."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=128)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

The same from_pretrained call works for every architecture in the library and accepts a local path just as happily as a Hub name, which is the entire user-facing contract in one method. Now the mistakes, because they are predictable and every one of them teaches something about the machinery.

Mistake one: decoding the prompt back out

generate returns the full sequence, prompt included, so decoding the whole tensor gives you your own words back:

# wrong: prints the prompt followed by the answer
print(tok.decode(out[0]))

# right: slice off the prompt tokens first
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Mistake two: feeding a chat model raw text

Instruction-tuned checkpoints were trained on conversations wrapped in special tokens, and they degrade badly when prompted without them. The template lives with the tokenizer, so the fix is one call:

# wrong: the model may ramble, echo, or answer as a base model
inputs = tok("Explain attention in two sentences.", return_tensors="pt")

# right: let the checkpoint's own chat template do the wrapping
inputs = tok.apply_chat_template(
    [{"role": "user", "content": "Explain attention in two sentences."}],
    add_generation_prompt=True, return_tensors="pt",
)

Mistake three: tuning knobs that are switched off

Sampling parameters only apply when sampling is on. If do_sample=False (greedy decoding), then temperature and top_p do nothing, and the library tells you so in a warning that people reliably ignore:

# wrong: greedy decoding, temperature silently irrelevant
model.generate(inputs, do_sample=False, temperature=0.9)

# right: sampling on, knobs live
model.generate(inputs, do_sample=True, temperature=0.9, top_p=0.95)

A related surprise: many checkpoints ship a generation_config.json with their own defaults for do_sample, temperature, and friends (Qwen2.5 does), so what you did not specify is not necessarily off. Part V's generate() deep dive covers the precedence.

Mistake four: batched generation with right padding

To batch prompts of different lengths you must pad, and for decoder-only models the padding must go on the left, because the model continues from the last position and a run of pad tokens there poisons the continuation. The library warns about this too:

# wrong for decoder-only models: default right padding
batch = tok(prompts, padding=True, return_tensors="pt")

# right: pad on the left, and pass the attention mask through
tok.padding_side = "left"
batch = tok(prompts, padding=True, return_tensors="pt")
out = model.generate(**batch.to(model.device), max_new_tokens=64)

Mistake five: tensors on the wrong device

device_map="auto" places the model; it does not place your inputs. The resulting error names two devices and is the most common first crash in the library. The fix is the .to(model.device) you have seen in every snippet above.

Part III: When it is the right tool

Transformers is the right tool when the model itself is what you are working on or with: fine-tuning a checkpoint, running evaluations, prototyping a product feature, doing research that edits a modeling file, or writing anything that must load arbitrary Hub checkpoints by name. It is the reference implementation the rest of the ecosystem builds against, which means when you need to know what a model actually computes, this is the code to read, and when a serving engine misbehaves, this is the implementation you compare it to.

The alternatives win at the edges of that territory. For production serving under concurrent traffic, vLLM and SGLang exist precisely because model.generate is a straight Python loop with no continuous batching or paged KV memory; they load the same Hub checkpoints and configs while replacing the execution stack. For laptop and edge inference, llama.cpp and Ollama run aggressively quantized GGUF conversions with no Python at all. For pretraining at scale, purpose-built training stacks like torchtitan own the parallelism story; Trainer is a fine-tuning workhorse, not a thousand-GPU pretraining framework.

The architecture-shaped warning follows directly: do not put model.generate behind a per-request web endpoint and call it a serving stack. Generation holds the GPU for the entire duration of a request, the loop handles one batch at a time, and requests queue behind each other, so the second concurrent user doubles the first user's latency. This works in a demo and collapses in production, and the failure is architectural, not a bug you can fix with threads.

Dangerous: web handler wraps generate()
  users ──► Flask/FastAPI ──► model.generate() ──► GPU
                 │ requests serialize; latency stacks up

Safe: engine serves, Transformers defines
  users ──► vLLM server (continuous batching, paged KV) ──► GPU
                 ▲
  Transformers formats: config.json, tokenizer.json, safetensors

Part IV: The full life of one generate call

The canonical operation. We follow this line all the way down and back:

pipeline("text-generation", model="Qwen/Qwen2.5-1.5B-Instruct")("Hello, ")

Everything below names actual files under src/transformers/ in the v5 tree, and every stage is one you can put a breakpoint in.

Stage 1: task resolution (pipelines/__init__.py)

The pipeline() factory lives in pipelines/__init__.py, which holds a SUPPORTED_TASKS dict and a PIPELINE_REGISTRY built from it. The string "text-generation" is checked against the registry and resolves to TextGenerationPipeline, defined in pipelines/text_generation.py. If you pass a model name but no task, get_task asks the Hub API what the model's declared task is. Nothing has touched the network for weights yet; the factory now needs to construct the pipeline's three ingredients: config, tokenizer, model.

Stage 2: config resolution (models/auto/configuration_auto.py)

AutoConfig.from_pretrained downloads exactly one file, config.json, via cached_file in utils/hub.py, which wraps the huggingface_hub client and the content-addressed cache under ~/.cache/huggingface/hub. For our model the JSON says "model_type": "qwen2" and "architectures": ["Qwen2ForCausalLM"]. The model_type string is looked up in CONFIG_MAPPING, whose underlying table, CONFIG_MAPPING_NAMES, lives in models/auto/auto_mappings.py as an ordered dict of entries like ("qwen2", "Qwen2Config"). The result is an instantiated Qwen2Config carrying hidden size 1536, 28 layers, grouped-query attention with 2 KV heads, and a 151,936-token vocabulary. The config is the keystone: every later resolution keys off either its model_type string or its Python class.

Stage 3: tokenizer resolution (models/auto/tokenization_auto.py)

AutoTokenizer.from_pretrained first fetches tokenizer_config.json, which may name a tokenizer_class directly; otherwise the model type from Stage 2 is looked up in TOKENIZER_MAPPING. In v5 nearly every model resolves to a fast tokenizer backed by the Rust tokenizers library, which loads the checkpoint's tokenizer.json, a complete serialized description of the BPE vocabulary, merges, normalizers, and special tokens. The chat template, a Jinja string, rides along in the tokenizer config. The base classes live in tokenization_utils_base.py.

Stage 4: model class and weights (auto_factory.py, modeling_utils.py)

AutoModelForCausalLM.from_pretrained, via _BaseAutoModelClass.from_pretrained in models/auto/auto_factory.py, runs Stage 2's config resolution again if needed, then does the second dictionary lookup: _get_model_class indexes cls._model_mapping, a lazy mapping built from MODEL_FOR_CAUSAL_LM_MAPPING_NAMES in models/auto/modeling_auto.py, by the config's class, yielding Qwen2ForCausalLM from models/qwen2/modeling_qwen2.py. Control then passes to the concrete class's from_pretrained, inherited from PreTrainedModel in modeling_utils.py, which does the heavy lifting: _get_resolved_checkpoint_files looks for model.safetensors first, falls back to the sharded model.safetensors.index.json plus its shards for large models, and only then to the legacy pytorch_model.bin. The module skeleton is built without materializing weights, tensors stream in from safetensors, dtype is applied (our checkpoint declares bfloat16), and with device_map="auto" the accelerate library places layers across available devices. Finally tie_weights honors the config's tie_word_embeddings: true by pointing the output head at the input embedding matrix.

Stage 5: preprocess (pipelines/text_generation.py)

Now the call itself. Pipeline.__call__ in pipelines/base.py runs the three-step contract: preprocess, _forward, postprocess. For text generation, preprocess tokenizes the prompt, and if you passed a list of chat messages instead of a string it routes through apply_chat_template so the special tokens are right. Out comes input_ids and an attention mask, moved to the model's device.

Stage 6: generation setup (generation/utils.py, generation/configuration_utils.py)

_forward calls self.model.generate(...), which every causal model inherits from GenerationMixin in generation/utils.py, a file of over four thousand lines that is the single most load-bearing piece of shared code in the library. _prepare_generation_config merges three sources in priority order: your keyword arguments, then the checkpoint's generation_config.json, then defaults. From the merged config a GenerationMode is chosen; with do_sample=True that is SAMPLE, and a dispatch table maps it to the _sample method (greedy search maps to the same method with sampling off). _get_logits_processor assembles the ordered list of logit transforms and _get_stopping_criteria the stop conditions; both are covered in the deep dive. A KV cache, by default a DynamicCache from cache_utils.py, is prepared to store attention keys and values across iterations.

Stage 7: prefill, then the decode loop (models/qwen2/modeling_qwen2.py)

_sample first runs _prefill: one forward pass over the whole prompt that fills the KV cache and produces logits for the next token. Then the loop: while any sequence is unfinished, run the model forward on just the newest token (the cache supplies the past), pull outputs.logits[:, -1] as float32, apply the logits processors, softmax and torch.multinomial to sample (or argmax when greedy), append the token to input_ids, hand it to the streamer if one is attached, and evaluate the stopping criteria. Each forward pass walks the modeling file from Part I's philosophy: embedding lookup, 28 decoder layers of RMSNorm, grouped-query attention against the cache, and a gated MLP, then the final norm and the tied lm_head projecting to 151,936 logits. If you want the numerical story of the softmax at the heart of both attention and sampling, that is the softmax page; the tensor mechanics of autograd-free inference are on the PyTorch walkthrough.

Stage 8: stopping (generation/stopping_criteria.py)

The loop exits when every sequence has hit a criterion: EosTokenCriteria saw the end-of-sequence id (151645 for this checkpoint, per its config), MaxLengthCriteria hit the token budget, StopStringCriteria matched a user-supplied stop string against decoded text, or MaxTimeCriteria ran out the clock. Finished sequences in a batch are masked and padded while stragglers continue.

Stage 9: decode and postprocess

Back in TextGenerationPipeline.postprocess, the token ids run through tokenizer.decode, which is the Rust backend reassembling BPE pieces into text, special tokens are stripped, and the result is wrapped as [{"generated_text": ...}]. The round trip is complete: a string went in, hit a registry twice, streamed gigabytes of safetensors once, looped one forward pass per generated token, and came back as a string.

Part V: Internals deep dives

Deep dive 1: the Auto registry and config-driven resolution

The whole Auto system is two dictionary lookups. Lookup one: model_type string to config class, in CONFIG_MAPPING. Lookup two: config class to model class, one mapping per task head, in tables like MODEL_FOR_CAUSAL_LM_MAPPING_NAMES. Both tables are plain ordered dicts of strings in models/auto/auto_mappings.py and models/auto/modeling_auto.py, wrapped in lazy mappings so that naming three hundred architectures does not import three hundred modules at startup.

config.json                 auto_mappings.py            modeling_auto.py
"model_type": "qwen2" ──►  ("qwen2","Qwen2Config") ──► ("qwen2","Qwen2ForCausalLM")
                                    │                            │
                              Qwen2Config()  ──────────►  Qwen2ForCausalLM(config)

The Auto classes are a dictionary lookup wearing a trench coat, and that is precisely what makes them durable: adding a new architecture means adding registry entries, not changing any caller's code. The same two-lookup pattern repeats for tokenizers, image processors, and every other artifact type, and it is user-extensible: AutoConfig.register and AutoModelForCausalLM.register let your own architecture participate without forking the library.

The registry has an escape hatch worth understanding precisely: remote code. A checkpoint whose config carries an auto_map can name model classes defined by Python files inside the model repo itself, and auto_factory.py will download and import them, but only if you pass trust_remote_code=True, because this is arbitrary code execution by design. The famous misconception in this area is milder: people assume AutoModel and AutoModelForCausalLM are interchangeable. AutoModel resolves to the bare backbone (Qwen2Model), which has no language-modeling head and no generate; if your loaded model cannot generate, check which registry you asked.

Deep dive 2: one model, one file, and the machinery that makes it survivable

Open models/llama/modeling_llama.py and models/qwen2/modeling_qwen2.py side by side and you will find long stretches of near-identical code: an RMSNorm, a rotary embedding class, an attention class, an MLP, a decoder layer. In most codebases this duplication would be a defect; here it is the stated philosophy. A researcher should be able to understand and modify one model by reading one file top to bottom, because machine learning moves too fast for shared abstractions to survive: yesterday's elegant base class becomes today's obstacle the moment a new architecture breaks its assumptions. Repetition also contains the blast radius, since editing Llama cannot break the other several hundred architectures.

The maintenance cost is managed mechanically. Duplicated blocks carry comments in the exact machine-readable form # Copied from transformers.models.llama.modeling_llama.LlamaMLP, and utils/check_copies.py (repository root, not the package) parses those comments in CI and fails the build if a copy has drifted from its source; running it with --fix_and_overwrite re-stamps the copies from the original. The comment syntax even supports mechanical renames, like with Llama->Qwen2, so a copied class can be checked after identifier substitution.

Modular Transformers is the newer, stronger version of the same idea. A model can be authored as a small modular_*.py file that subclasses another model's components and states only the deltas (models/gemma/modular_gemma.py is a real example), and utils/modular_model_converter.py expands it into the full flat modeling_*.py that users actually read. Authoring is DRY; the artifact is self-contained. The practical trap: for modular models the flat modeling file is generated output, and hand-edits to it will be overwritten; the header of such files says so, and the modular file is the one to change. The transferable lesson is that this library chose reviewability over DRY at the scale of hundreds of architectures, then built linters instead of abstractions to keep it honest.

Deep dive 3: inside generate()

Three pieces of machinery deserve separate attention: the config precedence, the logits processor chain, and the cache.

Precedence first, because it explains most "generation ignored my settings" reports. The effective GenerationConfig (defined in generation/configuration_utils.py) is built from, in increasing priority: library defaults, the checkpoint's generation_config.json, and your keyword arguments to generate. The middle layer is the one people forget exists. If sampling seems on when you never asked for it, or an unexpected repetition penalty applies, read the checkpoint's generation_config.json on the Hub before reading your own code.

The processor chain is the sampling machinery made composable. Every entry in generation/logits_process.py (a file of over three thousand lines) is a callable taking (input_ids, scores) and returning modified scores, and _get_logits_processor assembles them in a defined order into a LogitsProcessorList:

logits ─► RepetitionPenalty ─► NoRepeatNGram ─► Temperature ─► TopK ─► TopP ─► MinP
                                                              (warpers: reshape the
                                                               distribution, then
                                                               softmax + multinomial)

Temperature divides the logits, top-k keeps the k largest and sets the rest to negative infinity, top-p keeps the smallest set of tokens whose probabilities sum past p. Because each is a small class (TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper), you can instantiate and compose them yourself, which Lab 4 does. The stopping side mirrors this: StoppingCriteriaList over small criterion classes in generation/stopping_criteria.py.

The cache is why generation is fast at all: without it, step t recomputes attention keys and values for all t previous tokens, making the loop quadratic in total work. DynamicCache grows as tokens append; a preallocated StaticCache exists for torch.compile-friendly fixed shapes. Two corrections to common beliefs. First, greedy decoding is not a separate implementation: GenerationMode.GREEDY_SEARCH dispatches to the same _sample method with the sampling branch replaced by argmax. Second, v5 slimmed the core: exotic strategies like DoLa, contrastive search, and group beam search were moved out of the library into Hub-hosted custom_generate repositories under transformers-community/, loaded on demand; the in-tree methods are now _sample, _beam_search, and _assisted_decoding (speculative decoding with a small draft model).

Deep dive 4: Trainer, at concept level

trainer.py is a single long file implementing the training loop almost everyone fine-tuning a model needs: train handles checkpoint resumption and hands off to _inner_training_loop, which iterates epochs and batches, calling training_step per batch, which calls compute_loss, backpropagates, and lets the optimizer and scheduler step. Mixed precision, gradient accumulation, gradient clipping, distributed execution via accelerate, evaluation, logging, and checkpointing are all switched on by fields of the TrainingArguments dataclass in training_args.py rather than by code you write. The intended extension point is subclassing: override compute_loss for a custom objective, add TrainerCallbacks for custom control flow, and leave the loop alone. The ecosystem division of labor matters more than any single method: datasets live in datasets, parameter-efficient fine-tuning in peft, preference training in trl (whose trainers subclass this one), and multi-device orchestration in accelerate. I keep this section conceptual deliberately; the file is large and moves often, and the durable knowledge is the call chain plus the override points. To see everything Trainer abstracts written out longhand in three hundred lines, read the nanoGPT chapter next.

Part VI: Reading the repository

Nearly everything that matters is under src/transformers/. The stages below are a syllabus: read the named files in order, and check yourself against the questions before moving on.

Stage 0, the outside view. Run the Part II snippets, then look at what landed in ~/.cache/huggingface/hub, and skim a checkpoint's files on the Hub: config.json, generation_config.json, tokenizer.json, tokenizer_config.json, model.safetensors. You should be able to answer: which file names the architecture? Which file sets default sampling parameters? Why does the cache store blobs and snapshots separately?

Stage 1, one modeling file. Read src/transformers/models/llama/configuration_llama.py and then modeling_llama.py top to bottom: LlamaRMSNorm, LlamaRotaryEmbedding, LlamaMLP, LlamaAttention, LlamaDecoderLayer, LlamaModel, LlamaForCausalLM. Reading one modern decoder-only model carefully means you have effectively read dozens. You should be able to answer: where does the KV cache enter the attention forward? What does GenerationMixin add to LlamaForCausalLM? Which classes would differ in modeling_qwen2.py?

Stage 2, resolution and loading. Read models/auto/auto_factory.py, skim the tables in models/auto/auto_mappings.py and models/auto/modeling_auto.py, then read the loading path in modeling_utils.py starting from from_pretrained and _get_resolved_checkpoint_files, with configuration_utils.py beside it. You should be able to answer: what are the two dictionary lookups? In what order are weight file formats tried? What does trust_remote_code actually permit?

Stage 3, generation. Read generation/utils.py selectively: generate itself, _prepare_generation_config, then _sample in full. Then skim generation/logits_process.py and generation/stopping_criteria.py for the class inventory. You should be able to answer: what happens in prefill versus decode? In what order do processors apply? How does a batch finish when sequences stop at different times?

Stage 4, the layers above. Read pipelines/base.py for the preprocess/_forward/postprocess contract, pipelines/text_generation.py as its concrete instance, and walk trainer.py only along the train to training_step to compute_loss spine. You should be able to answer: why can tools compose pipelines without knowing the architecture inside? Where would a custom loss go?

Where not to start: modeling_utils.py from line one (it is enormous and only makes sense once you know what loading must accomplish), the package's __init__.py lazy import machinery, any large multimodal model as your first modeling file, and the tests/ tree, which is excellent for confirming behavior but hopeless as a map.

Part VII: Hands-on labs

Each lab is runnable on a laptop; the ones marked CPU-friendly use gpt2 (124M parameters) so no GPU is required.

Lab 1: anatomy of the cache (hub resolution). Load a model, then inspect what the Hub client wrote:

python -c "from transformers import AutoModelForCausalLM; AutoModelForCausalLM.from_pretrained('gpt2')"
find ~/.cache/huggingface/hub/models--gpt2 -maxdepth 2

Observe the three-way structure: refs/ mapping branch names to commits, snapshots/<commit>/ holding a directory of symlinks per revision, and blobs/ holding content-addressed files the symlinks point to. This is why two revisions of a model share unchanged files on disk, and why revision= pinning is cheap.

Lab 2: the registry with your own hands (Auto resolution). CPU-friendly:

from transformers import AutoConfig
from transformers.models.auto.configuration_auto import CONFIG_MAPPING

cfg = AutoConfig.from_pretrained("gpt2")
print(type(cfg).__name__, cfg.model_type)   # GPT2Config gpt2
print(CONFIG_MAPPING["gpt2"])               # <class ...GPT2Config>

Then download a config.json, edit model_type to a nonsense string, and load the config from that local directory: the error you get is the registry lookup failing, and now you can read it as one.

Lab 3: reimplement the decode loop (the generate machinery). CPU-friendly. Write the loop that _sample runs, in miniature:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
ids = tok("The meaning of life is", return_tensors="pt").input_ids
past = None
for _ in range(30):
    out = model(ids if past is None else ids[:, -1:],
                past_key_values=past, use_cache=True)
    past = out.past_key_values
    logits = out.logits[:, -1, :] / 0.8            # temperature
    v, _ = torch.topk(logits, 50)                  # top-k
    logits[logits < v[:, [-1]]] = -float("inf")
    probs = torch.softmax(logits, dim=-1)
    ids = torch.cat([ids, torch.multinomial(probs, 1)], dim=1)
print(tok.decode(ids[0]))

Observe that this is functionally the core of generate(do_sample=True, temperature=0.8, top_k=50), and that everything else in generation/utils.py is batching, configuration, caching strategy, and stopping logic around these ten lines.

Lab 4: processors as objects (sampling machinery). CPU-friendly. Replace the manual temperature and top-k lines of Lab 3 with the library's own classes and confirm identical behavior under a fixed seed:

from transformers import LogitsProcessorList, TemperatureLogitsWarper, TopKLogitsWarper
procs = LogitsProcessorList([TemperatureLogitsWarper(0.8), TopKLogitsWarper(50)])
scores = procs(ids, out.logits[:, -1, :])

Observe that order matters: temperature before top-k and top-k before temperature keep different candidate sets.

Lab 5: what the KV cache buys (cache deep dive). Time generation with and without the cache:

import time
for use_cache in (True, False):
    t = time.time()
    model.generate(ids[:, :8], max_new_tokens=200, do_sample=False, use_cache=use_cache)
    print(use_cache, round(time.time() - t, 2), "s")

Expect the no-cache run to be several times slower and to get relatively worse as max_new_tokens grows; the exact ratio varies by machine. You are measuring the difference between linear and quadratic recomputation.

Lab 6: break a copy, watch CI catch it (one-file philosophy). Clone the repository, find any block with a # Copied from comment (grep for it under src/transformers/models/), change one line inside the copy, and run:

git clone --depth 1 https://github.com/huggingface/transformers
cd transformers
python utils/check_copies.py            # fails, names your file
python utils/check_copies.py --fix_and_overwrite   # restores the copy

(The script needs the transformers package importable, which your Part II install already provides.)

Observe that the enforcement of the no-abstraction philosophy is itself just code, and that fixing a bug in an original propagates mechanically.

Part VIII: Understanding checks

Model answers in two to six sentences each; use them to test yourself after reading, not instead of reading.

1. What is Transformers, in one sentence? A registry of hundreds of deliberately self-contained PyTorch model definitions behind one loading interface, from_pretrained, serving as the ecosystem's reference implementation and interchange format for pretrained checkpoints.

2. How does a string like "Qwen/Qwen2.5-1.5B-Instruct" become a Python object? The library downloads the repo's config.json, reads its model_type field, looks that up in CONFIG_MAPPING to build a config object, then looks the config's class up in the task-specific model mapping to pick a class like Qwen2ForCausalLM, instantiates it, and streams the safetensors weights into it. Two dictionary lookups plus a download.

3. Why does the library duplicate attention code across hundreds of files instead of sharing a base class? Because architectures diverge faster than abstractions can absorb, a shared base class would accumulate flags and branches until unreadable, and a researcher should understand one model from one file. The cost of duplication is handled mechanically by # Copied from checks and, more recently, by modular files that generate the flat code.

4. What is the difference between AutoModel and AutoModelForCausalLM? They are different registries over the same config resolution. AutoModel returns the bare backbone with no task head and no generate; AutoModelForCausalLM returns the class with the language-modeling head and the GenerationMixin.

5. Where do generation defaults come from when you pass nothing? From the checkpoint's generation_config.json if present, otherwise library defaults; explicit keyword arguments to generate override both. Debugging surprising sampling behavior starts with reading that file on the Hub.

6. Why is the KV cache essential rather than an optimization? Without it every decode step recomputes keys and values for the entire prefix, making total work quadratic in output length; with it each step does one token's worth of new computation. For any nontrivial generation length the difference is the difference between usable and not.

7. What actually happens differently between greedy decoding and sampling? Almost nothing structurally: both run _sample, both apply the logits processors, and they differ only in whether the next token comes from argmax or from softmax plus multinomial. This is why sampling knobs are inert when do_sample=False.

8. What does trust_remote_code=True actually do, and why is it off by default? It permits the library to download and execute Python modeling files stored inside a model repository, resolved through the config's auto_map. It is arbitrary code execution from a remote source, so it requires explicit opt-in, and you should read the repo's code or pin a revision before enabling it.

9. Why are weights shipped as safetensors instead of pickle? Pickle files can execute code on load, while safetensors is a pure data format with a JSON header and raw tensor bytes, safe to load from untrusted sources and memory-mappable. The loader in modeling_utils.py prefers model.safetensors and treats pytorch_model.bin as a legacy fallback.

10. How do sharded checkpoints load? Large models ship a model.safetensors.index.json mapping each parameter name to the shard file that contains it; the loader reads the index, fetches shards, and fills the skeleton module tensor by tensor, so no single file needs to hold the whole model.

11. When would you choose vLLM over Transformers, and what do they share? Choose vLLM (or another engine) for serving concurrent traffic, because it adds continuous batching and paged KV memory that generate lacks. They share the artifact formats: engines load Transformers-format configs, tokenizers, and safetensors, which is why Transformers remains the source of truth even where it is not the execution stack.

12. Why must decoder-only models pad on the left for batched generation? Generation continues from the last position of each row, so with right padding the model would be asked to continue from pad tokens. Left padding puts the real prompt at the end of every row, and the attention mask keeps the pads from influencing the result.

13. A user reports their fine-tuned model generates garbage through pipeline but fine in their training code. What do you check first? The chat template and special tokens: whether the pipeline input is being wrapped with apply_chat_template the same way training data was, and whether the tokenizer saved with the checkpoint carries the right template and eos token. Format mismatch between training and inference is the most common cause of this exact symptom.

14. What is modular Transformers and what problem does it solve? A model can be authored as a small modular_*.py stating deltas against existing models, which a converter expands into the flat self-contained modeling_*.py. It keeps authoring DRY while preserving the readable one-file artifact, resolving the tension the # Copied from system only patched.

15. Where does Trainer end and accelerate begin? Trainer owns the loop semantics: batching, loss, accumulation, evaluation, checkpointing, callbacks. Accelerate owns device placement and distributed execution underneath it, so the same Trainer script runs on one GPU or many. When debugging a hang in distributed training, the boundary tells you which project's issues to search.

16. Why did v5 move strategies like contrastive search out of the core? To keep the core loop small and maintained: rarely used strategies now live as Hub-hosted custom_generate repositories loaded on demand, while the library keeps _sample, _beam_search, and assisted decoding in-tree. It is the same extension mechanism as remote code, applied to decoding algorithms.

Part IX: Design lessons

Data carries its own identity. Putting model_type inside the checkpoint means the artifact, not the caller, decides how it is interpreted, so code written before an architecture existed can load it the day it lands. The same move appears in file magic numbers, container image manifests, and schema registries in event pipelines.

A registry is the cheapest extension point that works. Two ordered dicts of strings have absorbed hundreds of architectures without a caller changing. Compare database driver registries, serde format registries, and plugin systems everywhere; the trench-coat dictionary outlives clever dispatch hierarchies.

Optimize for reading, then enforce with linters rather than abstractions. The one-model-one-file policy trades DRY for reviewability and contains blast radius, and check_copies.py turns the resulting duplication from a rot risk into a checked invariant. Kernel code and high-reliability firmware make the same trade: explicit repeated code, mechanically audited.

Make behavior a composition of small objects. Logits processors and stopping criteria are lists of tiny callables with one shared signature, so new sampling research lands as a new class, not a new branch in a monolithic loop. This is middleware in web frameworks and passes in compilers.

Configuration layering needs explicit precedence. Kwargs over generation_config.json over defaults is simple, documented, and still the number one source of user confusion, which teaches the corollary: when you layer configs, invest in making the effective value inspectable.

Be the format, not just the tool. Transformers' durable moat is that its config, tokenizer, and weight formats are what everyone else loads; execution stacks come and go around a stable interchange layer. Protocols and formats outlive implementations, from SQL to ONNX to POSIX.

Part X: The memorization framework

The whole system in one sentence: a checkpoint's config names its architecture, registries turn that name into classes, safetensors fill the class with state, and a shared decode loop turns forward passes into text.

name → config → class → weights → ids → loop → text

The chain mapped to the actual source files:

name    →  utils/hub.py                       (cached_file, the Hub cache)
config  →  models/auto/configuration_auto.py  (+ auto_mappings.py registry)
class   →  models/auto/auto_factory.py        (+ modeling_auto.py registry)
weights →  modeling_utils.py                  (from_pretrained, safetensors)
ids     →  tokenization_auto.py + tokenizer.json (Rust backend)
loop    →  generation/utils.py                (_sample; logits_process.py,
                                               stopping_criteria.py)
text    →  pipelines/text_generation.py       (postprocess, decode)

Memorize these blocks:

The two lookups. model_type string to config class; config class to model class per task head. All Auto behavior reduces to these.

The weight file ladder. model.safetensors, then model.safetensors.index.json plus shards, then legacy pytorch_model.bin, in that order.

The generate anatomy. Merge config (kwargs over checkpoint json over defaults), pick mode, build processor list and stopping list, prefill once, then loop: forward, process, sample or argmax, append, check.

The philosophy pair. One model one file for readers; # Copied from plus modular conversion for maintainers.

The v5 deltas. PyTorch only; exotic decoding strategies moved to Hub custom_generate; config base class spelled PreTrainedConfig.

Part XI: Papers and further reading

The library is the meeting point of a decade of papers, and each of the works below maps to a piece of this chapter. Where this site develops the same idea in depth, the companion link points there.

1. Wolf et al., HuggingFace's Transformers, State-of-the-Art Natural Language Processing, EMNLP 2020 system demonstrations. The library's own paper, written when the registry held dozens of architectures rather than hundreds, and still the clearest statement of its goals. The fine-tuning ecosystem that grew around it is covered in the PEFT walkthrough.
2. Vaswani et al., Attention Is All You Need, 2017. The architecture every modeling file in the repository implements. The mechanism is derived on the attention page on this site.
3. Radford et al., Language Models are Unsupervised Multitask Learners, 2019. The GPT-2 report, and the checkpoint that powers the CPU-friendly labs in Part VII. A model of the same shape is built end to end in the language models from scratch class.
4. Touvron et al., LLaMA, Open and Efficient Foundation Language Models, 2023. The model behind modeling_llama.py, the file Part VI recommends as your first careful read.
5. Qwen et al., Qwen2.5 Technical Report, 2024. Documents the checkpoint family this chapter's running example loads, including the config, chat template, and generation defaults traced in Part IV.
6. Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, 2015. The subword algorithm behind the BPE vocabulary serialized in tokenizer.json. Tokenization gets a fuller treatment in the NLP with deep learning class.
7. Holtzman et al., The Curious Case of Neural Text Degeneration, 2019. Introduces nucleus sampling, the top-p warper that sits in the logits processor chain of the generate deep dive.
8. Su et al., RoFormer, Enhanced Transformer with Rotary Position Embedding, 2021. The rotary position embedding class that appears near the top of every modern modeling file in the tree.
9. Ainslie et al., GQA, Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023. The grouped-query attention that the Qwen2.5 config declares with its 2 KV heads.
10. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022. The idea behind _assisted_decoding, one of the three strategies v5 keeps in-tree.
11. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. Explains why serving engines replace the generate loop under concurrent traffic, covered in the vLLM walkthrough.