torchtune

Part I: The mental model

tune run [--nproc_per_node N] RECIPE --config CONFIG  key=value ...
      |
      v
torchtune/_cli/run.py     resolve recipe + config names, torchrun if distributed
      |
      v
recipe_main(cfg)          @config.parse turns YAML + CLI overrides into a DictConfig
      |
      v
Recipe.setup(cfg)         a plain class in recipes/, no hidden Trainer base
      |
      |  config.instantiate(cfg.model)   ->  lora_llama3_1_8b(...)
      |  checkpointer.load_checkpoint()  ->  base weights (HF safetensors)
      |  set_trainable_params(adapter)   ->  freeze base, train LoRA only
      v
Recipe.train()            for each batch: forward -> chunked CE -> backward -> step
      |
      v
save_checkpoint()         merge (or keep) adapters, write back in the SAME format

The one-sentence identity. torchtune is a library of fine-tuning recipes, not a fine-tuning framework, so the training loop is a flat script you read top to bottom and the config is a component graph that says which building blocks to plug in. A framework hides the loop behind a Trainer.fit() and asks you to configure its behavior through hundreds of flags and callbacks. torchtune inverts that. Each recipe, for example recipes/lora_finetune_single_device.py, is a self-contained Python file with an ordinary for loop over batches. There is no base Trainer class doing work you cannot see. When you want to change how training works you copy the recipe with tune cp and edit the loop, which is the whole point.

Two consequences follow. First, the config is deliberately small and declarative. A YAML file names components by their dotted Python path under a _component_ key, and torchtune.config.instantiate turns each block into the object it describes. The config does not encode control flow, it only chooses parts. The recipe holds the control flow in plain sight. Second, torchtune brings its own model, dataset, and module implementations in native PyTorch instead of wrapping Transformers, so what you read is the real computation rather than a thin adapter over someone else's abstractions. That is the difference this whole design exists to buy, hackability, and it is worth stating plainly before anything else.

One honest note before we go deep. torchtune tracks recent PyTorch closely and moves fast, and the PyTorch post-training effort has at times signaled that pieces of this work are being consolidated with other projects. Treat version-specific spellings of an API as true for a given commit rather than forever. Everything in this chapter is written at the level of the durable design, and where a name is likely to have shifted I say so and stay at the concept. torchtune is the fine-tuning counterpart to the pretraining platform in the torchtitan chapter, and the two share a philosophy, keep the model plain and apply everything else as composable PyTorch.

Part II: Using it

torchtune installs from PyPI and needs a recent PyTorch. The single-device LoRA and QLoRA recipes run on one consumer or data-center GPU. The distributed recipes want two or more:

pip install torch torchvision torchao
pip install torchtune

# or from source, to read and hack the recipes directly:
git clone https://github.com/pytorch/torchtune
cd torchtune
pip install -e .

torchao is not optional decoration. It provides the NF4 4-bit tensor type that QLoRA quantizes the base model into, and the quantization-aware training and post-training quantization that some recipes use, so install it alongside torch. Once installed you have a tune command, which is the single door into the whole library. Four subcommands cover almost everything:

tune ls                 # list every built-in recipe and its configs
tune download ...       # pull model weights + tokenizer from the HF Hub
tune cp RECIPE|CONFIG   # copy a recipe or config into your tree to edit
tune run RECIPE ...     # launch a run (single-device or, with torchrun, multi)
tune validate CONFIG    # type-check a config without training

Start with tune ls. It prints a table of recipes, for example lora_finetune_single_device, lora_finetune_distributed, full_finetune_single_device, full_finetune_distributed, lora_dpo_single_device, and qat_distributed, and under each the named configs it ships with, one per model and setting. The names are the currency of the CLI, so this is where you learn what exists. Now download a base model. You need a Hugging Face token and to have accepted the model license:

tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
  --output-dir /tmp/Meta-Llama-3.1-8B-Instruct \
  --ignore-patterns "original/consolidated.00.pth" \
  --hf-token <YOUR_TOKEN>

The --ignore-patterns flag skips the large duplicate original-format checkpoint when the safetensors are all you need, though for Llama tokenizers you often do want the original/tokenizer.model file that ships alongside the weights. Now the real thing, a LoRA fine-tune of Llama 3.1 8B on a single GPU:

tune run lora_finetune_single_device \
  --config llama3_1/8B_lora_single_device

That resolves the config name llama3_1/8B_lora_single_device to a YAML file shipped in recipes/configs/, hands it to the recipe, and starts training. Expect a config dump, a model build, the base checkpoint loading, then step lines with loss, tokens per second, learning rate, and peak memory, with the loss falling. QLoRA is the same recipe with a different config, because quantizing the base model is a property of the model builder rather than of the loop:

tune run lora_finetune_single_device \
  --config llama3_1/8B_qlora_single_device

Every value in a config is overridable on the command line using OmegaConf dotlist syntax, which is key=value appended after the config, not --key value. This is the second thing to internalize after the recipe/config split:

tune run lora_finetune_single_device \
  --config llama3_1/8B_qlora_single_device \
  batch_size=4 \
  gradient_accumulation_steps=4 \
  model.lora_rank=16 \
  model.lora_alpha=32 \
  dataset.packed=True \
  max_steps_per_epoch=200

Distributed is not a different mental model, it is the same recipe family launched under torchrun, which tune run does for you when you pass --nproc_per_node:

# full fine-tune of 8B across 4 GPUs with FSDP2
tune run --nproc_per_node 4 full_finetune_distributed \
  --config llama3_1/8B_full

# LoRA across 4 GPUs
tune run --nproc_per_node 4 lora_finetune_distributed \
  --config llama3_1/8B_lora

Now the mistakes beginners make. First, forgetting that overrides are key=value, not flags. Writing --batch_size 4 is a CLI error, the recipe wants batch_size=4. Second, pointing the checkpointer at the wrong directory. The config's checkpointer.checkpoint_dir and checkpoint_files must match exactly where tune download put the weights, and the model_type must match the model, because the checkpointer uses it to pick the right weight-conversion mapping. Third, confusing single-device and distributed recipes. A *_single_device recipe launched under torchrun is wrong, and a distributed recipe run without --nproc_per_node will not shard. Fourth, and most in the spirit of the tool, do not fight the config to make the loop do something new. Run tune cp lora_finetune_single_device my_recipe.py, edit the Python, and run your copy. The library is designed to be forked, and reaching for the fork early is using it correctly, not abusing it.

A word on fitting big models on small GPUs, because that is the whole reason single-device recipes exist. The configs expose a stack of memory levers you turn on as you scale down the hardware. QLoRA quantizes the frozen base to 4-bit NF4. enable_activation_checkpointing recomputes activations in backward instead of storing them. enable_activation_offloading moves saved activations to CPU. The optimizer_in_bwd path in the full single-device recipe fuses the optimizer step into the backward pass so full gradients never coexist with optimizer state. An 8-bit optimizer such as bitsandbytes.optim.PagedAdamW8bit shrinks optimizer state further. compile=True lowers the model through torch.compile. And dataset.packed=True packs samples to a fixed sequence length to remove padding waste. Each is one line in the config.

Part III: When it is the right tool

torchtune is the right tool when you want to fine-tune a supported open model and you value owning and understanding your loop. Researchers who need to change the training math, teams who want a PyTorch-native stack with no Hugging Face Trainer in the call graph, people learning how fine-tuning actually works, and anyone who needs first-class multi-GPU LoRA or full fine-tuning with FSDP2 in the open source. It is also the natural early home for PyTorch-native techniques, torchao quantization, QAT, torch.compile, activation offloading, and export paths toward ExecuTorch, because it is written by the same team.

The honest cases for alternatives, named plainly. axolotl is a YAML-configured fine-tuning framework built on top of the Hugging Face stack, Transformers, PEFT, TRL, and accelerate, with DeepSpeed and FSDP underneath. Its strength is breadth and convenience. It supports an enormous range of models and techniques out of the box and you rarely write code, you write a config. Its cost is that the actual training happens deep inside the HF Trainer, so when you need to change the loop you are patching someone else's framework rather than editing a script. unsloth takes the opposite tack. It hand-writes fused Triton kernels and manual backward passes for specific model architectures and monkey-patches them into Hugging Face models, winning large speedups and memory reductions on a single GPU. Its cost is coverage and openness, the deep optimizations apply to the architectures they have hand-tuned, and multi-GPU has historically been outside the open tier.

So the triangle is roughly this. Reach for unsloth when single-GPU speed and VRAM on a popular architecture dominate everything else and you are happy to run inside their patched models. Reach for axolotl when you want maximum breadth of models and knobs with minimal code and you are content for the HF stack to own the loop. Reach for torchtune when you want a readable PyTorch-native loop you can fork, correct reference implementations, and honest multi-GPU FSDP2 in the open, and you accept that it may be slower than unsloth on one GPU and cover fewer exotic models than axolotl. These are genuinely different bets, and torchtune's is hackability over both raw single-GPU speed and sheer breadth.

Two more boundaries. torchtune fine-tunes, it does not pretrain from scratch at the thousand-GPU scale, that is torchtitan's job, and the two are deliberately separate projects with different defaults. And torchtune trains, it does not serve. When the fine-tune is done you hand the resulting checkpoint, written back in standard Hugging Face format, to an inference engine like vLLM for production serving. The clean handoff at the checkpoint boundary is a feature, and the format compatibility that makes it work is a design point we return to in Part V.

Part IV: The full life of one fine-tune

The specimen. One run of tune run lora_finetune_single_device --config llama3_1/8B_lora_single_device on one GPU. Almost every stage below is the same in the distributed recipe, and where it forks for FSDP2 I follow both briefly. The value of tracing it is that the recipe is a real script, so the trace is just reading it in order.

Stage 1: the tune CLI resolves names

tune is a small argument parser in torchtune/_cli/. The run subcommand (_cli/run.py) takes a recipe name and a --config name and looks them up in the recipe registry (torchtune/_recipe_registry.py), which maps the short name lora_finetune_single_device to the file recipes/lora_finetune_single_device.py and the config name llama3_1/8B_lora_single_device to the YAML shipped under recipes/configs/. If the name is not a builtin it is treated as a path, which is exactly how your tune cp copies get run. If you passed --nproc_per_node the runner execs torchrun with that many processes, otherwise it runs the recipe module in-process. Either way it forwards the config path and every key=value override to the recipe.

Stage 2: config.parse builds a DictConfig

Every recipe ends with a recipe_main function decorated with @config.parse (torchtune/config/_parse.py). That decorator reads the --config YAML with OmegaConf, applies the key=value overrides on top as a dotlist, and hands the resulting DictConfig to the function. No component has been constructed yet. The config at this point is pure data, a tree of dotted paths and scalar values, with each intended object still described only by a _component_ string and its keyword arguments. This is the last moment the whole run is just a document, and it is a good moment to run tune validate against.

config as data (a DictConfig):

model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  lora_attn_modules: ['q_proj', 'v_proj', 'output_proj']
  apply_lora_to_mlp: true
  lora_rank: 8
  lora_alpha: 16

      |  config.instantiate(cfg.model)
      v
a live nn.Module: lora_llama3_1_8b(lora_attn_modules=[...], lora_rank=8, ...)

Stage 3: the recipe object and setup

recipe_main constructs the recipe class, for example LoRAFinetuneRecipeSingleDevice(cfg), then calls recipe.setup(cfg) and recipe.train(). The recipe implements FTRecipeInterface from recipes/interfaces.py, which is documentation of the lifecycle rather than a base class doing work, setup, train, save_checkpoint, cleanup. Inside setup the recipe does a fixed sequence you can read in one screen. It builds the checkpointer from cfg.checkpointer and calls load_checkpoint() to get the base model's state dict already converted into torchtune's internal parameter names. It builds the model with config.instantiate(cfg.model), which calls lora_llama3_1_8b(...) and returns a TransformerDecoder whose attention and MLP linears have been swapped for LoRALinear layers. It loads the base weights into that model. Then it calls set_trainable_params after get_adapter_params to freeze every base weight and leave only the LoRA A and B matrices requiring gradients.

The rest of setup builds the optimizer from cfg.optimizer over just the trainable params, the loss (typically CEWithChunkedOutputLoss), the tokenizer, and the dataset and dataloader. In the distributed recipe this stage has one extra, load-bearing step. Between building the model and loading weights it shards the model with FSDP2's fully_shard, wrapping each transformer layer and then the whole model, and materializes it from the meta device, which is the same discipline the torchtitan chapter traces in detail. Single-device skips all of that and just moves the model to cuda.

Stage 4: the data path and the label mask

The dataloader yields already-tokenized batches, because the tokenization happened inside the dataset. The chain is worth holding in mind. A dataset such as alpaca_cleaned_dataset is an SFTDataset configured with a message transform that turns each raw row into a list of Message objects, an optional prompt template that adds role-specific wrapping text, and the model's ModelTokenizer. When the dataloader asks for item i the dataset builds the messages, applies the template, and calls tokenizer.tokenize_messages, which returns token ids and a boolean mask. Messages marked as the prompt are masked out, so their label positions become CROSS_ENTROPY_IGNORE_IDX (-100) and contribute no loss. The model is trained only on the tokens of the response. A collate function, padded_collate_sft, then pads the variable-length sequences in a batch and stacks them into tensors of input ids and labels.

Stage 5: forward, chunked loss, backward

train() is an ordinary loop over epochs and batches. Read it directly, it is the heart of the recipe and it is short. For each batch it moves tensors to the GPU, runs logits = model(tokens, mask=..., input_pos=...), and computes the loss. The loss deserves a note. Full-vocabulary logits for a long sequence are enormous, so torchtune's default CEWithChunkedOutputLoss computes cross-entropy over slices of the sequence and sums, which keeps a large logits tensor from ever being materialized whole. This pairs with the model returning unreduced per-token outputs. Then loss.backward() flows gradients, and because only the LoRA adapters require grad, gradients exist only for the small A and B matrices while the frozen base contributes none. This is exactly why LoRA fits where full fine-tuning does not.

Gradient accumulation lives here as a plain modulo. The recipe scales the loss by 1 / gradient_accumulation_steps, calls backward every microbatch, and only steps the optimizer once the accumulation count is reached, which is how a single-GPU config reaches a large effective batch size. When optimizer_in_bwd is enabled in the full single-device recipe this stage changes shape, the optimizer step for each parameter fires from a gradient hook the moment that parameter's gradient is ready, so full gradients never all exist at once. In the FSDP2 distributed recipe the backward additionally reduce-scatters each layer's gradients across ranks, but the recipe code you read is unchanged, that communication is a property of the sharded parameters, not of the loop.

Stage 6: optimizer step, clip, and metrics

On an accumulation boundary the recipe optionally clips the gradient norm, calls optimizer.step() and optimizer.zero_grad(), and advances the learning-rate scheduler, commonly a cosine schedule with warmup from torchtune.training. It then logs to whichever metric logger the config selected, DiskLogger, StdoutLogger, WandBLogger, or TensorBoardLogger, reporting step, loss, learning rate, tokens per second, and peak memory from training.get_memory_stats. In the distributed recipe the logged loss is reduced across ranks so the number is a true global average rather than one rank's view. None of this is hidden behind callbacks, it is straight-line code in train().

Stage 7: the checkpoint written back in its own format

At the end of each epoch, or at a configured interval, save_checkpoint() runs. For LoRA this is where the two audiences are served. The adapter weights are collected via get_adapter_params and saved on their own as a small file you can keep and stack. If configured to save a merged model, get_merged_lora_ckpt folds the low-rank updates back into the base weights, and then, critically, the checkpointer converts torchtune's internal parameter names back into the original external format and writes safetensors that look exactly like the model you downloaded. A torchtune fine-tune comes out in the same format it went in, so the output loads straight into Transformers or vLLM with no bespoke conversion. For resumable training the recipe also writes a recipe-state file with the optimizer state, epoch, and seed, and the distributed recipes can use torch.distributed.checkpoint for the intermediate sharded state. That closes the loop of one run, base weights in, adapter gradients trained, a standard checkpoint out.

Part V: Internals deep dives

Deep dive: recipes and the component config

The recipe/config split is the whole architecture, so it is worth being precise about the seam between them. A recipe is a training program. A config is a document that selects and parameterizes the objects the program will use. The bridge is torchtune.config. Two functions carry it. instantiate(node) reads the _component_ key, imports that dotted path, and calls it with the remaining keys as keyword arguments, recursing into nested components, so a config node becomes a live object. parse is the decorator that loads the YAML, layers CLI overrides on with OmegaConf, and feeds the recipe. There is no plugin system and no registry of behaviors, only Python import paths and function calls. That is a deliberate constraint. Anything you can import you can name in a config, and anything you name in a config you can find by reading the code it points at.

config.instantiate resolution:

_component_: torch.optim.AdamW      import torch.optim.AdamW
lr: 3e-4                            call AdamW(params, lr=3e-4, weight_decay=0.01, ...)
weight_decay: 0.01
fused: true

nested components resolve first, bottom-up, then the parent is called

What this buys, and what it costs. It buys transparency, a config is a manifest of parts with no hidden control flow, and forkability, since changing behavior means editing a visible loop rather than configuring an invisible one. It costs the convenience of a framework that has a flag for everything, because if the recipe does not do something you must add it to the recipe. torchtune accepts that trade on purpose. The recipes are kept short and parallel to each other precisely so that copying one and changing ten lines is the expected way to do something new. A misconception to correct here, the config is not a weak scripting language. It cannot branch or loop, and trying to make it do so is a sign you should copy the recipe instead.

Deep dive: LoRA and QLoRA modules

LoRA lives in torchtune/modules/peft/. The core is LoRALinear, a drop-in replacement for nn.Linear that keeps the original weight frozen and adds a trainable low-rank update. Conceptually the forward is the base projection plus a bottleneck through two small matrices, and the base path never receives gradients:

# the essential shape of LoRALinear.forward, in words not verbatim source
out = frozen_base(x)                 # W0 @ x, W0.requires_grad_(False)
lora = dropout(x) @ A.T @ B.T        # rank-r bottleneck, A: (r, in), B: (out, r)
return out + (alpha / rank) * lora   # scaled low-rank update added on

Only A and B require gradients, so the trainable parameter count is a few million against the base model's billions. Which linears become LoRA is a config choice, that is what lora_attn_modules: ['q_proj', 'v_proj', 'output_proj'] and apply_lora_to_mlp control in the model builder. DoRALinear implements DoRA, a variant that additionally learns a magnitude, and it lives beside LoRALinear under the same package. A handful of helper functions make the recipe code clean. get_adapter_params(model) returns just the adapter tensors, set_trainable_params freezes everything else, validate_missing_and_unexpected_for_lora confirms a base checkpoint loaded into a LoRA model has exactly the expected missing keys, the adapters, and get_merged_lora_ckpt folds adapters back into the base for export.

QLoRA adds one idea on top, quantize the frozen base. When the model builder is called with quantize_base=True, for example through the qlora_llama3_1_8b convenience builder, each base linear's weight is stored as a 4-bit NF4Tensor from torchao rather than as bf16 or fp16. The forward dequantizes on the fly for the matmul, and because the base is frozen the 4-bit storage is only ever read, never updated, so its error never accumulates. The LoRA adapters stay in full precision and carry all the learning. QLoRA's whole trick is that the enormous frozen part can be lossy because it is never trained, while the tiny trainable part stays exact, which is why a 4-bit base plus full-precision adapters fine-tunes a large model on a single consumer GPU. The precision and dtype mechanics here connect to the broader story in the mixed precision write-up. A trap worth naming, NF4 is a storage-and-compute-in-4-bit format for the frozen weights, it is not the same as quantizing the fine-tune output for inference, which is a separate post-training step done by quantize.py or torchao's PTQ.

Deep dive: distributed recipes and FSDP2

The distributed recipes, full_finetune_distributed and lora_finetune_distributed, differ from their single-device siblings in exactly one region of setup, the sharding. torchtune uses FSDP2, the fully_shard API, which keeps every parameter an individual DTensor sharded across the data-parallel mesh rather than flattening a module's parameters into one opaque buffer the way the original FSDP did. The recipe builds the model on the meta device so no full copy is ever allocated, applies fully_shard per transformer layer and once over the whole model, then materializes and loads each rank's shard. The mechanics, sharding conditions, mixed-precision policy, and meta-device init, are provided by helpers in torchtune.training, and the deeper theory of device meshes and per-parameter sharding is the subject of the torchtitan chapter and the parallel computing class.

Two things to hold onto. First, torchtune's distributed story is deliberately data-parallel with sharding, FSDP2, and it does not try to be a full N-dimensional parallelism platform. Tensor, pipeline, and context parallelism are torchtitan's territory. That is the right division of labor, fine-tuning an 8B or 70B model fits comfortably in FSDP2 across a node or two, and adding pipeline parallelism would trade the readable single-program loop for complexity a fine-tune rarely needs. Second, the recipe you read is nearly identical to the single-device one, the collectives are a property of the sharded DTensor parameters, so forward, backward, and step look the same in the source while all-gathers and reduce-scatters happen underneath. That parallel structure between the single-device and distributed recipes is not an accident, it is what lets you learn one and read the other.

Deep dive: datasets, messages, and prompt templates

The data plumbing is where torchtune does the most quiet work to keep model-specific formatting out of the loop. The central abstraction is the Message (torchtune/data/), a small object with a role (system, user, assistant, or ipython for tool output), content, and a masked flag that decides whether its tokens count toward the loss. Every dataset, whatever its raw shape, is normalized into a list of messages by a message transform. InputOutputToMessages handles simple instruction/response columns, ShareGPTToMessages and OpenAIToMessages handle multi-turn conversation formats, and ChosenRejectedToMessages handles preference pairs for DPO. The dataset classes on top, SFTDataset for supervised fine-tuning, PreferenceDataset for DPO, TextCompletionDataset for raw text, and ConcatDataset for mixing, are thin wrappers that apply a transform, an optional template, and the tokenizer.

raw row  --message_transform-->  [Message(system), Message(user, masked=True),
                                  Message(assistant, masked=False)]
         --prompt_template-->    wrap each message's content with role text
         --tokenizer.tokenize_messages-->  (token_ids, mask)
         --padded_collate_sft-->  batched input_ids, labels (masked -> -100)

The prompt template is the second half, and it is separate from the message transform on purpose. A message transform decides what the conversation is, a template decides how a role's text is decorated, for example a summarization template that wraps the input in instruction text, or the ChatMLTemplate. The model's ModelTokenizer then adds the model-specific special tokens and turns messages into ids with tokenize_messages, applying the mask so that prompt tokens are ignored and only the response is learned. This layering is why swapping datasets is a config change. Point dataset._component_ at a different builder, or set the source and column_map of the generic instruct_dataset, and nothing in the recipe moves. The misconception to correct, the mask is not cosmetic. Getting it wrong trains the model to generate the prompt, and the whole reason the plumbing is this careful is to make the mask correct by construction.

Deep dive: checkpointing and format fidelity

Checkpointers live in torchtune/training/checkpointing/ and their defining job is round-tripping the external format. The three main ones are FullModelHFCheckpointer for Hugging Face safetensors, FullModelMetaCheckpointer for Meta's original consolidated .pth format, and FullModelTorchTuneCheckpointer for torchtune's own layout, and a DistributedCheckpointer wraps DCP for sharded intermediate state. On load, the checkpointer reads the external files and converts their parameter names and tensor layouts into torchtune's internal module names using a per-model-family weight mapping. On save it runs the inverse conversion. This conversion is the feature. Because torchtune restores the exact external naming and layout on the way out, a fine-tuned model is indistinguishable in format from the base you downloaded, which is what makes the clean handoff to serving possible. The trap is the same one every checkpoint system has, the mapping is keyed on module names, so a model implementation whose names drift from what the converter expects will fail to load or, worse, load the wrong tensor into the wrong place. The model_type field in the config is what selects the right converter, which is why it must match the model exactly.

Part VI: Reading the repository

The tree is small and unusually flat, which is half of what makes it teachable. Read it in this order. Paths are described by role where an exact spelling is likely to drift.

Stage 0, orientation. Read the top-level README.md, then run tune ls and skim one config end to end, for example recipes/configs/llama3_1/8B_lora_single_device.yaml. Questions to hold. What is a recipe versus a config, what does a _component_ key mean, and what does tune run actually do differently for single-device versus distributed?

Stage 1, one recipe top to bottom. Read recipes/lora_finetune_single_device.py as a whole. It is a single class implementing recipes/interfaces.py's FTRecipeInterface, with setup, train, save_checkpoint. Questions. Where are the adapters frozen, where does the label mask come from, how does gradient accumulation appear in the loop, and what exactly does save_checkpoint write?

Stage 2, the config machinery. Read torchtune/config/, specifically the instantiate and parse implementations. Questions. How does a dotted string become an object, how are nested components resolved, and how do key=value overrides merge onto the YAML?

Stage 3, the modeling building blocks. Read torchtune/modules/, the TransformerDecoder and its TransformerSelfAttentionLayer, MultiHeadAttention, RotaryPositionalEmbeddings, RMSNorm, FeedForward, and KVCache, then modules/peft/ for LoRALinear and its helpers. Then read one model family in torchtune/models/, for example llama3_1, where the _component_builders.py assembles the decoder and the _model_builders.py exposes llama3_1_8b, lora_llama3_1_8b, and qlora_llama3_1_8b. Questions. What makes the model definition plain PyTorch, and how does a builder decide which linears become LoRA?

Stage 4, the data path. Read torchtune/data/ for Message, the message transforms, and the prompt templates, then torchtune/datasets/ for SFTDataset and a concrete builder like alpaca_dataset, and finally the tokenizer for one model. Questions. Where does the loss mask get set, and what is the boundary between a message transform and a prompt template?

Stage 5, training utilities and the distributed recipe. Read torchtune/training/, the checkpointing package, the FSDP helpers, and the memory and profiling utilities, then read recipes/lora_finetune_distributed.py beside the single-device recipe you already know and diff them in your head. Questions. Exactly which lines differ, and why is that difference so small?

Stage 6, the frontier. The DPO, PPO, QAT, and knowledge-distillation recipes, and the evaluation (eleuther_eval.py), generation (generate.py), and quantization (quantize.py) recipes. These reuse everything above and show how the same building blocks compose into other post-training objectives.

Where not to start. The multimodal and vision model families add a cross-attention path and image transforms that are best met after the text story is solid, and the RL-style recipes (PPO) introduce a reward model and a rollout loop that are a lot to hold before the supervised recipe is second nature.

Part VII: Hands-on labs

Labs 1 through 4 need at most one GPU. Labs 5 and 6 want two or more. Log formats and exact config names vary with the pace of the repo, so match on role, not on a literal string.

Lab 1: list, copy, read. Concept, the recipe/config split.

tune ls
tune cp lora_finetune_single_device ./my_recipe.py
tune cp llama3_1/8B_lora_single_device ./my_config.yaml

Open my_recipe.py and find the train method's for loop, then open my_config.yaml and match each top-level key (model, optimizer, dataset, checkpointer) to where it is instantiated in the recipe. You now hold the entire architecture in two files.

Lab 2: a real QLoRA fine-tune. Concept, the life of one run from Part IV.

tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
  --output-dir /tmp/Meta-Llama-3.1-8B-Instruct --hf-token <TOKEN>

tune run lora_finetune_single_device \
  --config llama3_1/8B_qlora_single_device \
  max_steps_per_epoch=100 \
  metric_logger._component_=torchtune.training.metric_logging.StdoutLogger

Watch the setup logs in order, config dump, model build, base checkpoint load, then step lines with falling loss. Match each phase to a stage of Part IV. Note the peak-memory line, that is QLoRA's 4-bit base earning its keep.

Lab 3: override without editing. Concept, OmegaConf dotlist overrides.

tune run lora_finetune_single_device \
  --config llama3_1/8B_lora_single_device \
  batch_size=4 gradient_accumulation_steps=2 \
  model.lora_rank=16 model.lora_alpha=32 \
  model.apply_lora_to_mlp=True

Compare trainable-parameter count and step time against the default rank-8 attention-only run. Then set a nonsense override like model.lora_rank=hello and read how the config layer rejects it. Run tune validate my_config.yaml to type-check a config with no GPU at all.

Lab 4: swap the dataset. Concept, message transforms and templates.

# in a copied config, point dataset at your own instruction data
dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: json
  data_files: /path/to/my_data.json
  column_map:
    input: prompt
    output: response
  packed: False

Run the LoRA recipe against this config. The recipe does not change at all. Confirm by inspecting a single tokenized batch that the prompt positions carry label -100 and only the response tokens carry real labels, which is Stage 4 of Part IV made concrete.

Lab 5: LoRA across GPUs with FSDP2. Concept, the distributed recipe.

tune run --nproc_per_node 2 lora_finetune_distributed \
  --config llama3_1/8B_lora \
  batch_size=2

Open both lora_finetune_single_device.py and lora_finetune_distributed.py and diff the setup methods. The only substantive difference is the fully_shard region. Watch per-GPU peak memory fall relative to the single-device run at the same effective batch size, because parameters, gradients, and optimizer state are now sharded.

Lab 6: fine-tune, then serve. Concept, format fidelity at the checkpoint boundary.

# after a LoRA run with a merged save, the output dir holds standard safetensors
ls /tmp/Meta-Llama-3.1-8B-Instruct/  # + your output dir with epoch_N/

# hand the merged checkpoint straight to an inference engine
vllm serve /path/to/your/merged_output_dir

Confirm the merged output loads with no conversion step. That the fine-tune comes out in exactly the format the base came in is the payoff of the checkpointer deep dive, and the clean seam between training in torchtune and serving in vLLM.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is torchtune, in one sentence?

The PyTorch team's native fine-tuning library, where each fine-tune is a readable, forkable recipe script wired together by a small YAML config that names components by import path, favoring hackability over both a framework's convenience and a specialized kernel stack's raw speed.

2. What is the difference between a recipe and a config?

A recipe is a training program, a self-contained Python script with a visible loop. A config is a document that selects and parameterizes the objects that program uses, with no control flow of its own. torchtune.config.instantiate turns a _component_ node into a live object, and config.parse loads the YAML and applies CLI overrides.

3. How does LoRA reduce the trainable parameter count?

LoRALinear freezes the original weight and adds a trainable low-rank update through two small matrices A and B of rank r. Only A and B require gradients, so a few million parameters train against billions of frozen ones, and the update is scaled by alpha over rank before being added to the base projection.

4. What does QLoRA add on top of LoRA, and why is it safe?

It quantizes the frozen base weights to 4-bit NF4 from torchao while keeping the LoRA adapters in full precision. It is safe because the base is never trained, so its quantization error never accumulates through optimization, while all learning happens in the exact, full-precision adapters.

5. Where does the loss mask come from and why does it matter?

From the masked flag on each Message. Prompt messages are masked, so their label positions become CROSS_ENTROPY_IGNORE_IDX (-100) and contribute no loss, and the model is trained only on the response tokens. Getting the mask wrong trains the model to generate the prompt.

6. Trace the data path from a raw dataset row to a batched tensor.

A message transform turns the row into a list of Message objects, an optional prompt template decorates each role's content, the model tokenizer's tokenize_messages produces token ids and a mask, and padded_collate_sft pads and stacks a batch, setting masked label positions to -100.

7. How does the distributed recipe differ from the single-device one?

In one region of setup. It builds the model on the meta device and applies FSDP2's fully_shard per transformer layer and once over the whole model, so parameters, gradients, and optimizer state are sharded across ranks. The loop is unchanged, because the collectives are a property of the sharded DTensor parameters rather than of the code.

8. Why can a torchtune fine-tune load straight into vLLM or Transformers?

Because the checkpointer converts torchtune's internal parameter names and layouts back into the exact external format on save, so the output safetensors are indistinguishable in format from the base model that was downloaded. The model_type field selects the right conversion.

9. Name three ways a single-device config fits a large model on one GPU.

QLoRA's 4-bit NF4 base, activation checkpointing and activation offloading to trade compute or CPU bandwidth for activation memory, and either optimizer_in_bwd in the full recipe or an 8-bit optimizer to shrink optimizer state. Chunked cross-entropy and sample packing help further.

10. When would you choose unsloth or axolotl over torchtune?

unsloth when single-GPU speed and VRAM on a popular architecture dominate and you are content inside its patched Hugging Face models. axolotl when you want maximum breadth of models and knobs with minimal code and are content for the HF Trainer to own the loop. torchtune when you want a readable PyTorch-native loop you can fork and honest multi-GPU FSDP2 in the open.

11. Why does torchtune not do tensor or pipeline parallelism?

Because fine-tuning models in the 8B to 70B range fits comfortably in FSDP2 across a node or two, and adding tensor or pipeline parallelism would trade the readable single-program loop for complexity a fine-tune rarely needs. That N-dimensional territory is deliberately left to torchtitan.

12. What does tune cp exist for?

To copy a builtin recipe or config into your own tree so you can edit it and run it by path. It is the sanctioned way to change behavior the config cannot express, and reaching for it early is using the library as intended rather than fighting it.

13. What exactly is in a config's _component_ value?

A dotted Python import path to a callable, a model builder, an optimizer class, a dataset builder, a loss. The remaining keys in that node are the callable's keyword arguments. instantiate imports the path and calls it, recursing into nested components first.

14. Why are LoRA adapters saved separately as well as merged?

The separate adapter file is small, stackable, and swappable at inference, so you can keep many task adapters over one base. The merged file is a standard full checkpoint for direct serving. The recipe can do either or both, controlled by config.

Part IX: Design lessons

Ship a loop, not a Trainer. By making each recipe a flat, self-contained script instead of a subclass of a framework base, torchtune keeps the thing you most need to understand, the training loop, in plain sight. The cost is some duplication across recipes, and it pays that willingly because a duplicated line you can read beats a shared abstraction you cannot. This is the same instinct as the one-model-one-file philosophy in Transformers and the deliberate minimalism of nanoGPT.

Configuration selects parts, code holds behavior. A config that can only name and parameterize components, with no branching, forces every behavior to live in readable code. That is a constraint that pays, because the config stays a manifest you can audit and the loop stays the single source of truth. The same pattern, name the pieces declaratively and keep the wiring in real code, runs through dependency injection and build systems that eventually admit they are just Python.

Make the output indistinguishable from the input. Round-tripping the exact external checkpoint format means a fine-tune drops into any tool that accepts the base model, with no bespoke converter and no lock-in. Preserving the interface at the boundary, rather than inventing a proprietary artifact, is what lets torchtune hand off cleanly to serving stacks and evaluators it knows nothing about.

Correctness lives in the boring plumbing. The label mask, the global token count, the weight-name conversion. The parts most tempting to hand-wave are exactly where a fine-tune silently goes wrong, so torchtune spends its abstraction budget making the message and mask machinery correct by construction rather than on surface features. Reference implementations exist to get the boring parts right.

Let one design own each problem. torchtune does fine-tuning and hands pretraining to torchtitan and serving to vLLM, and it does data-parallel FSDP2 rather than reinventing every parallelism. Drawing the boundary honestly keeps each project small enough to read, which is the property the whole ecosystem is optimizing for.

Part X: Memorization framework

The one-sentence summary. torchtune expresses a fine-tune as a readable recipe script plus a component config, freezes the base model and trains small LoRA adapters (optionally over a 4-bit NF4 base for QLoRA), shards across GPUs with FSDP2 when distributed, builds its batches from masked messages, and writes the result back in the same checkpoint format it read.

tune run RECIPE --config CONFIG key=value
  -> config.parse (YAML + overrides -> DictConfig)
  -> Recipe.setup: instantiate model, load base ckpt, freeze base + LoRA on
  -> Recipe.train: data (masked messages) -> fwd -> chunked CE -> bwd -> step
  -> save_checkpoint: merge adapters, write back in the original format

The chain mapped to source:

cli            torchtune/_cli/ (tune ls/download/cp/run), _recipe_registry.py
config         torchtune/config/ (instantiate, parse), recipes/configs/*.yaml
recipe         recipes/lora_finetune_single_device.py (+ interfaces.py)
model          torchtune/models/<family>/ + torchtune/modules/ (TransformerDecoder)
peft           torchtune/modules/peft/ (LoRALinear, get_adapter_params, merge)
data           torchtune/data/ (Message, transforms, templates) + datasets/
distributed    torchtune/training/ (fully_shard helpers, FSDP2)
checkpoint     torchtune/training/checkpointing/ (FullModelHFCheckpointer)

Memorize these blocks:

• The split: recipe is code with the loop, config names components by import path under _component_, overrides are key=value.
• LoRA: freeze base weight, train low-rank A and B, scale by alpha/rank, add to the base projection. QLoRA is the same with a 4-bit NF4 frozen base.
• Distributed: the single-device and distributed recipes differ only in the fully_shard FSDP2 region of setup.
• Data: row -> messages -> template -> tokenize_messages -> mask, prompt tokens become label -100 and only responses train.
• Checkpoints: loaded and saved through a per-model converter, so the fine-tune output is standard HF format and serves directly.
• Positioning: hackable readable loops, vs unsloth's single-GPU kernel speed and axolotl's config-driven breadth over the HF stack.

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The low-rank adapter that LoRALinear implements, freeze the base weight and train two small matrices scaled by alpha over rank. The PEFT walkthrough on this site covers the Hugging Face implementation of the same idea.
2. Dettmers et al., QLoRA, Efficient Finetuning of Quantized LLMs, 2023. The 4-bit NF4 frozen base with full-precision adapters that the qlora builders reproduce, and the source of the paged-optimizer idea. The precision mechanics connect to the mixed precision write-up.
3. Liu et al., DoRA, Weight-Decomposed Low-Rank Adaptation, 2024. The magnitude-and-direction decomposition behind DoRALinear.
4. Grattafiori et al., The Llama 3 Herd of Models, 2024. The model family this chapter fine-tunes, and the reference for the architecture torchtune reimplements in plain PyTorch.
5. Zhao et al., PyTorch FSDP, Experiences on Scaling Fully Sharded Data Parallel, 2023. The sharded data parallelism underneath the distributed recipes, which FSDP2 reworks with per-parameter DTensors. The deeper treatment is in the torchtitan walkthrough and the parallel computing class.
6. Rafailov et al., Direct Preference Optimization, Your Language Model is Secretly a Reward Model, 2023. The preference objective behind the DPO recipes and the ChosenRejectedToMessages plumbing. Derived in the DPO note and run at scale in the TRL walkthrough.
7. Chen et al., Training Deep Nets with Sublinear Memory Cost, 2016. The recompute-in-backward idea behind enable_activation_checkpointing.
8. Su et al., RoFormer, Enhanced Transformer with Rotary Position Embedding, 2021. The rotary position encoding that RotaryPositionalEmbeddings implements. The attention variants write-up works through the geometry.
9. Dettmers et al., 8-bit Optimizers via Block-wise Quantization, 2021. The quantized optimizer state behind the PagedAdamW8bit option in memory-tight configs. The state being shrunk is the moment pair from the Adam note.
10. Ouyang et al., Training language models to follow instructions with human feedback, 2022. The instruction-tuning pipeline whose supervised stage is what the SFT recipes implement. The applied generative AI class places fine-tuning in that larger pipeline.

Part XII: Final takeaway

If the single-device pieces this repository assumes are the gap, the ML implementations section builds attention, optimizers, and precision from scratch, the sharding underneath the distributed recipes is traced in the torchtitan chapter and the parallel computing class, and the engine that serves the result is the vLLM chapter. Then come back and read lora_finetune_single_device.py once more. It will read like plain PyTorch with a config bolted on the front, which is exactly the point.