Axolotl

Part I: The mental model

axolotl train config.yml
      |
      v
axolotl/cli/main.py       Click command group, resolve + validate the YAML
      |
      v
load_cfg + Pydantic       utils/schemas/config.py -> one typed config object
      |
      v
accelerate launch         one process per GPU (DDP / FSDP / DeepSpeed)
      |
      v
axolotl/train.py          load_tokenizer, load_model (+ PEFT adapter / quant)
      |
      v
prompt strategies         datasets[].type -> input_ids with loss-masked labels
      |
      v
multipack sampler         pack many samples into each sequence_len window
      |
      v
trainer builder           core/builders -> AxolotlTrainer / DPO / GRPO / RM
      |
      v
transformers/TRL loop     forward (flash-attn, packed) -> loss -> backward -> step
      |
      v
save                      LoRA adapter or full weights -> output_dir

The one-sentence identity: Axolotl is the layer that makes a fine-tune a declarative document instead of a bespoke training script, because a single validated YAML config is expressive enough to describe SFT, DPO, GRPO, and reward modeling across many model families, and Axolotl compiles that config into the right calls against transformers, PEFT, TRL, and DeepSpeed. The usual way to fine-tune a model is to copy a training script, wire up a tokenizer, a dataset map, a PEFT config, a TrainingArguments, and a Trainer by hand, and edit Python every time you change models or objectives. Axolotl inverts that. The script is fixed and audited, and the thing you edit is data. Change base_model and you are training a different architecture. Change rl: dpo to rl: grpo and you are running a different learning algorithm. Change adapter: qlora to a full fine-tune by deleting one line.

Two consequences follow. First, the YAML surface is the product. Almost every hard-won detail of modern fine-tuning, quantized loading, adapter placement, packing without cross-contamination, chat-template masking, gradient checkpointing, FSDP or DeepSpeed sharding, has a field, a sensible default, and a validation rule. Getting a run right becomes a matter of setting fields correctly rather than remembering an incantation. Second, Axolotl is a broad integrator rather than a from-scratch trainer. It does not reimplement attention, PEFT, or the RL objectives. It orchestrates libraries you already trust and adds the connective tissue, prompt strategies, packing, monkey-patches, and a validated schema, that those libraries leave to you. Everything here is checked against the main branch of the axolotl-ai-cloud/axolotl repository in July 2026. The project moves quickly, so where a detail is likely to shift I say so and stay at concept level.

Part II: Using it

Axolotl is a Linux-and-NVIDIA-GPU project at heart, built on a recent PyTorch and CUDA. The recommended install path uses uv, installs a matching torch first, and then installs Axolotl with the extras you want without build isolation so the flash-attention and DeepSpeed builds can see your torch:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 && source .venv/bin/activate
uv pip install torch==2.7.1 torchvision
uv pip install --no-build-isolation axolotl[deepspeed]
# or run everything in the maintained image:
# docker run --gpus '"all"' --ipc=host --rm -it axolotlai/axolotl:main-latest

There are tagged releases on PyPI as well, each tested against a specific torch. Extras like [flash-attn] and [deepspeed] pull in the pieces that need compilation. On macOS the package installs and is pleasant to read and step through, but real training needs CUDA, so the honest Mac workflow is reading code locally and running on a Linux box.

The fastest way to a first run is to fetch the bundled examples and train one. axolotl fetch examples copies the example configs into your working directory, and every command takes a YAML path:

axolotl fetch examples                       # copy example configs locally
axolotl preprocess examples/llama-3/lora-1b.yml   # tokenize + cache the dataset
axolotl train examples/llama-3/lora-1b.yml        # run the fine-tune
axolotl inference examples/llama-3/lora-1b.yml --lora-model-dir="./outputs/lora-out"
axolotl merge-lora examples/llama-3/lora-1b.yml   # fold the adapter into the base

The lora-1b example fine-tunes Llama 3.2 1B with a LoRA adapter on an Alpaca-formatted dataset, and it finishes quickly on a single modern GPU, which also makes it the right thing to step through with a debugger on a first read. axolotl preprocess is optional but worth running first. It resolves every dataset, applies the prompt strategy, tokenizes, and caches the result to disk (the dataset_prepared_path), so train starts immediately and every rank in a multi-GPU run reads the same prepared data instead of tokenizing redundantly.

Now look at what a real config actually contains. This is close to the shipped lora-1b example, trimmed to the load-bearing fields:

base_model: NousResearch/Llama-3.2-1B

datasets:
  - path: teknium/GPT4-LLM-Cleaned
    type: alpaca                 # the prompt strategy that formats + masks each row
dataset_prepared_path: ./last_prepared
val_set_size: 0.1
output_dir: ./outputs/lora-out

adapter: lora                    # delete this line for a full fine-tune
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true         # attach LoRA to every linear layer

sequence_len: 2048
sample_packing: true             # pack many short rows into each 2048 window
pad_to_sequence_len: true
flash_attention: true

micro_batch_size: 2              # per-GPU batch
gradient_accumulation_steps: 2
num_epochs: 1
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0002
bf16: auto
gradient_checkpointing: true

special_tokens:
  pad_token: "<|end_of_text|>"

Every one of those fields is a validated attribute on a config schema, not a free-form string. The schema is why a typo in optimizer or an impossible combination of quantization and adapter fails at load time with a clear message rather than deep inside the training loop an hour later.

Scaling to multiple GPUs is a launch concern, not a model-code concern. The axolotl CLI wraps accelerate, and you point it at a DeepSpeed or FSDP config. Axolotl ships the common DeepSpeed JSON files, which you fetch and then reference from the YAML:

axolotl fetch deepspeed_configs        # writes deepspeed_configs/ locally
# then add to the YAML:  deepspeed: deepspeed_configs/zero3_bf16.json
axolotl train config.yml               # accelerate spawns one process per visible GPU

# the explicit form is still valid and useful for debugging:
accelerate launch -m axolotl.cli.train config.yml

Now the mistakes beginners make. First, forgetting that micro_batch_size is per GPU, so the effective global batch is micro_batch_size times gradient_accumulation_steps times the number of data parallel ranks, and a run that looked fine on one GPU quadruples its batch on four. Second, mismatching the dataset type to the data, which produces a technically valid run that trains on wrong or unmasked text, the quietest failure in the whole system, covered in the prompt-strategies deep dive. Third, enabling sample_packing without flash attention. Packing depends on an attention implementation that respects per-sample boundaries inside a packed sequence, so it pairs with flash_attention: true. Fourth, expecting axolotl train to produce a merged model when adapter is set. A LoRA run saves adapter weights, and axolotl merge-lora is the separate step that folds them back into the base for serving. Fifth, and most fundamental: do not fork the trainer to change behavior you can express in config. The moment you patch the Python to add an objective or a dataset shape that a field already covers, you have left the paved road the whole framework exists to keep you on.

Part III: When it is the right tool

Axolotl is the right tool when your job is adapting an existing checkpoint, when you want that job to be reproducible and reviewable as a config file, and when you value breadth, many model families and many objectives behind one consistent surface, over owning every line of the loop. It is a natural fit for teams that run many fine-tunes and want them to look the same in version control, for researchers comparing SFT against DPO against GRPO without rewriting a harness each time, and for anyone who wants QLoRA on a 70B model across a few GPUs without hand-assembling FSDP and bitsandbytes themselves.

The honest cases for alternatives. Reach for Hugging Face TRL directly when you want a small custom loop and are willing to write and maintain the glue Axolotl would have written for you, which is the better choice for genuinely novel training procedures. Reach for torchtune when you want a PyTorch-native, recipe-based stack with fewer external dependencies and are comfortable editing a recipe in Python. Reach for Unsloth when you are optimizing a single-GPU QLoRA run for maximum speed and minimum memory, since its custom kernels are hard to beat on one card, though its open multi-GPU story is thinner. LLaMA-Factory is the closest peer, another config-and-CLI driven fine-tuner with a web UI and comparable model coverage, and the choice between them is largely taste and ecosystem. For pretraining from scratch at large scale, Axolotl is the wrong layer entirely, and torchtitan or Megatron is the tool. For serving the model you produced, Axolotl hands off to an inference engine like vLLM, which it also uses internally to generate rollouts during GRPO.

The architecture-shaped warning is about what Axolotl is not. It is not a thin, minimal codebase you can hold entirely in your head, and it is not a fixed-version library. It tracks fast-moving upstreams, transformers, PEFT, TRL, DeepSpeed, and flash attention, and part of its value is being the place where a new model or a new objective is wired up correctly soon after it lands upstream. That currency is a feature and a cost. Pinning versions matters, the maintained Docker images exist precisely to freeze a known-good set, and a config that ran last quarter can need a field renamed this quarter. Treat the schema and the examples as the source of truth over any blog post, including this one.

Part IV: The full life of one fine-tune

The specimen: one LoRA supervised fine-tune of Llama 3.2 1B, launched with axolotl train config.yml using the config from Part II. Most of the machinery below runs identically for a full fine-tune or a 70B QLoRA across eight GPUs. Where the path forks for preference tuning or RL, I note it and return in the internals deep dives.

Stage 1: the CLI and config resolution

The axolotl command is a Click group defined in src/axolotl/cli/main.py, with one subcommand per verb (train, preprocess, inference, merge-lora, vllm-serve, and more). The train subcommand reads the YAML into a plain dict, then loads it into a typed config through the loader in src/axolotl/cli/ and the Pydantic schema in src/axolotl/utils/schemas/config.py. This is where validation and normalization happen. Field types are checked, defaults are filled, deprecated field names are migrated, and cross-field rules run, for example that quantized loading is compatible with the chosen adapter, or that packing has a compatible attention implementation. The output is one config object that the rest of the system treats as ground truth.

Stage 2: accelerate launch and the worker entry

For anything beyond a single process the CLI hands off to accelerate, which spawns one worker per GPU and sets the usual distributed environment. Each worker ultimately calls the train() function in src/axolotl/train.py, which is the real entry point and the function worth reading first. Everything from here runs once per rank in the standard single-program, multiple-data shape, with DeepSpeed or FSDP deciding how parameters, gradients, and optimizer state are sharded across those ranks.

Stage 3: tokenizer and model loading

train() first loads the tokenizer and then the model through the loaders in src/axolotl/loaders/ (tokenizer.py and model.py). Model loading is where a surprising amount of config becomes real. If load_in_4bit is set, the model is loaded through bitsandbytes with a quantization config. If adapter is lora or qlora, PEFT wraps the model and installs the low-rank adapters onto the target modules, with lora_target_linear: true resolving to every linear layer in the transformer blocks. The correct attention implementation is selected here too, flash attention when requested, which matters for the packing story two stages down. The result is a ready-to-train model where only the adapter parameters (or all parameters, for a full fine-tune) require gradients.

Stage 4: datasets, prompt strategies, and masking

Next the dataset pipeline in src/axolotl/utils/data/ (sft.py for supervised runs) walks the datasets list. For each entry it resolves the type field to a prompt strategy in src/axolotl/prompt_strategies/, loads the raw data from the Hub or local disk, and applies that strategy to every row. A strategy does two jobs. It formats the row into a single token sequence using a prompt template or a chat template, and it builds the labels tensor with the prompt positions masked out to the ignore index so loss is computed only on the tokens the model should learn to produce. This masking is the quiet heart of instruction tuning. Get it wrong and the model learns to parrot the prompt. The tokenized, masked, concatenated dataset is then cached to dataset_prepared_path, so this whole stage is skipped on subsequent runs and shared across ranks.

Stage 5: sample packing

With sample_packing: true, Axolotl does not train on one short example per sequence_len window and waste the padding. Instead a multipack sampler in src/axolotl/utils/samplers/multipack.py bin-packs many tokenized rows into each sequence up to the length budget, and records where each row starts and ends. The collator in src/axolotl/utils/collators/ assembles these packed sequences and builds the position_ids that reset at every row boundary. That boundary information is what keeps the packing correct, and it is picked up by the attention path in the next stage. Packing is the single biggest throughput lever for datasets of short examples, often several times fewer steps for the same data, which is why it is on by default in most examples.

Stage 6: the trainer builder and the training loop

Now the config chooses a trainer. The builders in src/axolotl/core/builders/ (causal.py for language modeling, rl.py for preference and RL objectives) construct a Hugging Face TrainingArguments from the config fields and wrap the model in the right trainer class from src/axolotl/core/trainers/. For plain SFT that is AxolotlTrainer, a subclass of the transformers Trainer that adds the packing-aware collator, custom samplers, callbacks, and metrics. For rl: dpo or rl: grpo or reward_model: true the builder selects the matching TRL-based trainer instead. Then trainer.train() runs the familiar loop, pull a batch, forward through the model, compute the loss, backward, and step the optimizer, with gradient accumulation, gradient checkpointing, mixed precision, and DeepSpeed or FSDP sharding all governed by the config. The model code contains no distributed logic. Sharding and communication are supplied by DeepSpeed or FSDP from outside, the same mechanism-from-outside idea that torchtitan takes further.

Stage 7: what the forward pass actually does with packing

Inside the forward pass, a packed sequence would be wrong under naive attention, because token 3 of example B must not attend to example A sitting earlier in the same row. Axolotl handles this by using flash attention in its variable-length mode, driven by the per-row boundaries computed during packing (expressed through position_ids and cumulative sequence lengths). Attention is computed independently within each packed example, so a packed batch produces exactly the loss an unpacked batch would, at a fraction of the padding waste. The monkey-patches in src/axolotl/monkeypatch/ (historically per-architecture attention hijacks, increasingly folded into upstream transformers) are what make this true for each model family. The memory-traffic reason flash attention can do this cheaply is the tiling and online-softmax story in the FlashAttention chapter.

Stage 8: saving and merging

At the configured interval and at the end, the trainer saves to output_dir. For a LoRA run that is a small adapter, the low-rank matrices plus the PEFT config, not a full copy of the base model, which is why fine-tuning a 70B model can produce a checkpoint measured in tens of megabytes. To serve the model as a single set of weights, axolotl merge-lora reloads the base, applies the adapter, and writes merged weights. A full fine-tune skips this and saves complete weights directly. That closes the loop of one fine-tune, config in, tokenized and masked and packed data through a wrapped trainer, adapter out.

Part V: Internals deep dives

Deep dive: the config schema, and why the YAML is the product

The center of Axolotl is not the trainer. It is the Pydantic schema in src/axolotl/utils/schemas/, split into focused modules (config.py for the top-level input config, plus datasets.py, peft.py, training.py, trl.py, fsdp.py, quantization.py, vllm.py, and more). The YAML you write is deserialized into these models, and that single act does an enormous amount of work. Fields get types, so a string where an int belongs fails immediately. Defaults get filled, so a short config is really a long config with most fields left at sensible values. Deprecated spellings get migrated, so old configs keep running. And validators encode the cross-field rules that are genuinely hard to remember, which quantization settings pair with which adapters, which objectives require which dataset shapes, when packing needs flash attention, whether an FSDP and DeepSpeed combination is coherent.

This is the load-bearing design choice. By making the config a validated schema rather than a bag of keys, Axolotl turns a large fraction of fine-tuning expertise into rules the machine enforces, and turns a run into a document you can diff, review, and version. The CLI even exposes the schema directly, so you can ask what a field means without leaving the terminal:

axolotl config-schema                 # print the full config schema
axolotl config-schema --field adapter # explain one field
axolotl agent-docs --list             # task-oriented docs (sft, grpo, dpo, ...)

The correction of a common misconception: the YAML is not a thin convenience wrapper over a Python API you are meant to graduate to. The config is the API. Reading utils/schemas/ tells you almost everything Axolotl can do, and every field there is a promise the trainer keeps.

Deep dive: prompt strategies and dataset formats

A prompt strategy is the function that turns a raw dataset row into tokens with a loss mask. It is the concept that most rewards careful reading, because a mismatched strategy is the most common silent failure in fine-tuning. The strategies live in src/axolotl/prompt_strategies/, and the type field on each dataset selects one. The families worth knowing:

The modern default is chat_template (chat_template.py). Rather than hardcode a prompt format, it renders a conversation through the tokenizer's own Jinja chat template, so the format matches exactly what the base model was trained to expect, and you set chat_template: llama3 or tokenizer_default to pick which one. It then masks every token that is not part of an assistant turn, so the model learns only to generate assistant responses. The older alpaca and ShareGPT-style strategies remain for datasets already in those shapes. The input_output strategy (input_output.py) is the escape hatch for full control, you provide pre-segmented text with a boolean per segment saying whether it contributes to the loss, which is how you express masking that no template captures. Custom strategies plug in through user_defined.py and the base classes in prompters.py and prompt_tokenizers.py.

The practical lesson: always verify the masking before a long run. axolotl preprocess config.yml --debug renders a few tokenized examples so you can see exactly which tokens are masked and which contribute to the loss. Two minutes there prevents a run that trains flawlessly on the wrong tokens.

Deep dive: sample packing and multipack

Padding is wasted compute. If sequence_len is 4096 and the average example is 300 tokens, more than nine tenths of every unpacked batch is padding the model computes attention over and then discards. Sample packing removes that waste by placing many examples end to end in each sequence, and the multipack sampler in src/axolotl/utils/samplers/multipack.py does the bin-packing, approximately, since optimal bin-packing is expensive, using a fast first-fit style approach that reaches high occupancy while balancing work across ranks.

The correctness problem packing introduces is cross-contamination. If examples A, B, and C share one sequence, tokens in C must not attend to A or B. Axolotl solves this at the attention level, not by hoping the model ignores the wrong tokens. The collator emits position_ids that restart at each example boundary, and flash attention runs in variable-length mode so each packed example is attended independently, as if it were its own sequence. Packing is only correct because attention is made boundary aware, which is why sample packing and flash attention travel together and why turning on one without the other is a configuration error rather than a mild inefficiency. The payoff is large. On short-example datasets, packing routinely cuts step counts several-fold with no change to the loss the model sees, and it is the first thing to enable when a run feels slow.

Deep dive: one surface, four objectives

The reason a single config can express supervised fine-tuning, preference optimization, RL, and reward modeling is that Axolotl maps each objective onto a trainer from the TRL family and lets the schema decide which one. The switch is small in the config and large in what it selects. See the RL section for the theory behind these objectives.

Preference tuning. With rl: dpo the dataset rows carry a prompt, a chosen response, and a rejected response, formatted by a strategy under prompt_strategies/dpo/, and the DPO trainer optimizes the model to prefer chosen over rejected relative to a frozen reference. KTO, ORPO, and IPO are sibling switches with their own data expectations and their own strategy directories (kto/, orpo/). The point is that moving from SFT to DPO is a config edit, not a new harness.

GRPO. Reinforcement learning is the most involved path. With rl: grpo, training needs to generate candidate completions and score them, so Axolotl runs a separate vLLM server for fast rollouts, started with axolotl vllm-serve config.yml, while the trainer samples groups of completions per prompt, scores each with your reward functions (importable Python callables referenced from the config), and updates the policy toward higher-reward completions using the group-relative baseline that gives GRPO its name. The GRPO trainers live in src/axolotl/core/trainers/grpo/, including asynchronous variants and a replay buffer that keep the GPUs busy by overlapping generation with optimization. This is the one objective where Axolotl orchestrates two engines at once, a trainer and an inference server, and the config wires them together.

Reward modeling. With reward_model: true, Axolotl trains a scalar reward head on top of a base model using paired chosen and rejected responses and a Bradley-Terry objective, with the pairing handled by the bradley_terry/ strategy. The stepwise variant (stepwise_supervised.py) supports process reward models that score intermediate reasoning steps rather than only final answers. A reward model trained here is exactly the kind of scorer a GRPO run consumes, so the two objectives compose into a full RLHF pipeline expressed as a small set of configs.

Deep dive: multi-GPU and sequence parallelism

Axolotl treats scale-out as configuration layered over the trainer, not as a rewrite of it, the same discipline explored in the parallel computing class notes. Three axes stack.

Data parallelism is the default. Under accelerate, each GPU holds a full copy of the model and processes different data, and gradients are all-reduced. Sharded data parallelism arrives through DeepSpeed ZeRO or PyTorch FSDP, both first-class in the schema (deepspeed: pointing at a JSON config, or fsdp: with fsdp_config:). ZeRO stages 1 through 3 progressively shard optimizer state, gradients, and parameters across ranks so a model far larger than one GPU fits, and this is what makes QLoRA on a 70B or 405B base practical on a handful of cards. Axolotl supports both FSDP1 and FSDP2.

Sequence parallelism is the newer axis and the answer to a different problem, a single sequence too long to fit on one GPU even at batch size one. Setting sequence_parallel_degree: N splits each sequence into N equal chunks placed on N GPUs, so a very long context is spread across devices along the token dimension. The GPUs must still compute attention over the whole sequence, so they exchange key and value chunks during the attention step, implemented in src/axolotl/monkeypatch/ring_attn/ on top of the ring-flash-attention project. Sequence parallelism is orthogonal to data parallelism and composes with it, and recent Axolotl extends this into N-dimensional parallelism that combines the axes. The through-line is that none of these axes are visible to the model definition. You choose degrees and backends in the YAML, and Axolotl arranges the sharding and the collectives, so scaling up is a config change rather than a code change. The attention-memory reasoning that makes long-context training tractable is the same tiling story in the FlashAttention chapter.

Deep dive: plugins and the integration surface

Axolotl keeps a plugin system in src/axolotl/integrations/ (with a BasePlugin in base.py) so optional, fast-moving optimizations can bolt on without bloating the core. You activate one by naming its class in the config:

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

The Liger integration swaps in fused Triton kernels for RMSNorm, RoPE, SwiGLU, and a fused linear-plus-cross-entropy that avoids materializing the full logits, cutting memory and speeding training. Other integrations in the same tree include cut-cross-entropy, Spectrum for selective layer training, knowledge distillation, expert parallelism for MoE models, and lm_eval hooks for evaluation. The pattern is deliberate. The core stays a stable orchestrator, and each integration is a self-contained module gated behind a config flag, so an experimental kernel can ship and be adopted without destabilizing everyone else's runs.

Part VI: Reading the repository

The tree is larger than a from-scratch trainer because it is an integrator, but it has a clear spine. All paths are under src/axolotl/ and verified on main in July 2026. Directory layouts move as the project refactors, so anchor on roles, not exact filenames.

Stage 0, orientation. Read the top-level README.md, then axolotl fetch examples and open a few configs under examples/, for instance examples/llama-3/lora-1b.yml and a DPO example. The config is the interface, so time spent reading real configs is the fastest way in. Question: for each field, can you name what component consumes it?

Stage 1, the entry points. cli/main.py (the command group) and train.py (the worker entry). Read train() top to bottom as the spine of the whole system. Questions: where does the YAML become a typed object, in what order are tokenizer, model, and datasets built, and where is the trainer chosen?

Stage 2, the schema. utils/schemas/config.py and its siblings (datasets.py, peft.py, training.py, trl.py). This is the map of everything Axolotl can do. Questions: which fields have validators, and what cross-field rules do those validators encode?

Stage 3, data and prompts. utils/data/sft.py for the loading pipeline, then prompt_strategies/ with chat_template.py, alpaca_chat.py, completion.py, and input_output.py as the destinations, plus prompters.py and prompt_tokenizers.py underneath. Questions: where exactly is the loss mask built, and how does a type string resolve to a strategy?

Stage 4, packing and collation. utils/samplers/multipack.py and utils/collators/, read against monkeypatch/multipack.py and the attention patches in monkeypatch/. Questions: how are packed-sequence boundaries represented, and how do they reach the attention kernel?

Stage 5, trainers and models. core/builders/ (causal.py, rl.py), core/trainers/ (base.py, dpo/, grpo/, trl.py), and the loaders in loaders/ (model.py, tokenizer.py, adapter.py). Questions: what does AxolotlTrainer add over the transformers Trainer, and how does the RL builder select a TRL trainer from the rl field?

Stage 6, the frontier. The integrations/ tree for optional kernels and objectives, monkeypatch/ring_attn/ for sequence parallelism, and the examples/distributed-parallel/ configs for N-D parallelism. These are the fast-moving edges, interesting after the spine is solid.

Where not to start: the per-architecture monkey-patches. There are many, they exist to bridge specific model quirks to features like packing, and reading them before you understand what they are bridging is confusing. Meet them only when a specific model needs one.

Part VII: Hands-on labs

Labs 1 through 4 need a single GPU (lab 2 needs none). Labs 5 and 6 want two or more. Log and field details vary with the fast pace of main, so read the schema when a field disagrees with the text.

Lab 1: a first LoRA in one command. Concept: the full life of Part IV.

axolotl fetch examples
axolotl preprocess examples/llama-3/lora-1b.yml
axolotl train examples/llama-3/lora-1b.yml

Watch the phases in order, config validation, model and tokenizer load, dataset preparation, then training steps with a falling loss, then a saved adapter in ./outputs/lora-out. Match each phase to a stage of Part IV. Then run axolotl inference examples/llama-3/lora-1b.yml --lora-model-dir=./outputs/lora-out and talk to your model.

Lab 2: read the masking before you trust it. Concept: prompt strategies. No GPU needed for the render.

axolotl preprocess examples/llama-3/lora-1b.yml --debug

Read the rendered examples and identify which tokens are masked (they will not contribute to the loss) and which are not. Then change the dataset type from alpaca to completion and rerun the debug render. Observe that masking disappears entirely, because completion is raw language modeling. This is the fastest way to internalize why the strategy must match the data.

Lab 3: turn packing off and measure it. Concept: sample packing throughput.

axolotl train examples/llama-3/lora-1b.yml --sample_packing=false --num_epochs=1
axolotl train examples/llama-3/lora-1b.yml --sample_packing=true  --num_epochs=1

Compare the number of steps per epoch and the wall-clock time between the two runs on the same dataset. The gap is padding you were computing attention over and discarding. Note that the final loss is comparable, because packing changes throughput, not the objective.

Lab 4: full fine-tune versus adapter. Concept: adapters and merging.

# delete the adapter line to switch from LoRA to a full fine-tune:
axolotl train examples/llama-3/lora-1b.yml --adapter=""
# with the adapter, inspect how small the saved artifact is, then merge:
axolotl train examples/llama-3/lora-1b.yml
ls -la ./outputs/lora-out           # adapter weights, not a full model
axolotl merge-lora examples/llama-3/lora-1b.yml

Compare the size of the saved artifact in the two cases, and note that only the merge step produces single-file weights ready for a serving engine.

Lab 5: shard a model too big for one card. Concept: DeepSpeed ZeRO.

axolotl fetch deepspeed_configs
# add to the config:  deepspeed: deepspeed_configs/zero3_bf16.json
axolotl train config.yml            # accelerate uses every visible GPU

Start from a QLoRA config for a model that would not fit on one GPU, add a ZeRO-3 config, and watch per-GPU memory drop as optimizer state, gradients, and parameters are sharded. Compare ZeRO-2 against ZeRO-3 and reason about the memory-versus-communication trade.

Lab 6: split a long sequence across GPUs. Concept: sequence parallelism.

# on 2+ GPUs, raise sequence_len and add:
#   sequence_parallel_degree: 2
axolotl train long-context-config.yml

Push sequence_len until a single-GPU run would OOM, then set sequence_parallel_degree to a divisor of your GPU count and observe the long sequence fitting because it is split across devices along the token dimension. This is the axis that data parallelism cannot help with.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is Axolotl, in one sentence?

A config-first fine-tuning framework whose real interface is a single validated YAML file, which it compiles into correct calls against transformers, PEFT, TRL, and DeepSpeed to run SFT, DPO, GRPO, and reward modeling across many model families.

2. Why is the YAML called the product?

Because the config is a Pydantic schema, not a bag of keys. Setting fields correctly, with types checked, defaults filled, and cross-field rules validated, is how you express a run, so a fine-tune becomes a diffable, reviewable document and a huge fraction of fine-tuning expertise becomes rules the machine enforces.

3. What does a prompt strategy do, and why does it matter so much?

It turns a raw dataset row into tokens and, critically, builds the label mask so loss is computed only on the tokens the model should learn to produce. A mismatched strategy trains on wrong or unmasked text, which is a silent failure, so verifying the masking with a debug render before a long run is essential.

4. Why do sample packing and flash attention travel together?

Packing places several examples in one sequence, and correctness requires that tokens never attend across example boundaries. Axolotl enforces this with flash attention in variable-length mode driven by per-example position ids, so a packed batch yields the same loss as an unpacked one. Without a boundary-aware attention path, packing would silently corrupt attention.

5. How does one config express four different objectives?

The schema routes the objective to a trainer in the TRL family. Default is SFT with AxolotlTrainer, rl: dpo and its siblings select preference trainers, rl: grpo selects the GRPO trainer with vLLM rollouts, and reward_model: true selects a Bradley-Terry reward trainer. Changing objectives is a field edit, not a new harness.

6. What is special about the GRPO path?

It is the one objective that runs two engines at once. A separate vLLM server, started with axolotl vllm-serve, generates candidate completions quickly, and the trainer scores groups of completions per prompt with your reward functions and updates the policy using a group-relative baseline, with async trainers and a replay buffer overlapping generation and optimization.

7. Where does distributed sharding come from, and where does it not?

From outside the model, via accelerate plus DeepSpeed ZeRO or PyTorch FSDP, all configured in the YAML. The model definition contains no distributed logic, which is why data parallelism, sharded data parallelism, and sequence parallelism are all config choices rather than model rewrites.

8. What problem does sequence parallelism solve that ZeRO does not?

A single sequence too long for one GPU even at batch size one. ZeRO shards parameters, gradients, and optimizer state across the batch and model, but each GPU still holds a full sequence. Setting sequence_parallel_degree splits each sequence into chunks across GPUs and exchanges key and value chunks during attention, so very long contexts fit.

9. Why run axolotl preprocess before train?

It resolves, formats, tokenizes, masks, and caches the dataset to the prepared path once, so training starts immediately and every rank reads the same prepared data instead of tokenizing redundantly. It also gives you the debug render to verify masking.

10. A LoRA run finished but serving expects one set of weights. What did you forget?

axolotl merge-lora. A LoRA run saves only the small adapter, and merging reloads the base, applies the adapter, and writes merged weights. A full fine-tune, with no adapter set, skips this because it already saves complete weights.

11. When would you pick TRL, torchtune, or Unsloth over Axolotl?

TRL directly when you want a small custom loop and will maintain the glue yourself, which suits genuinely novel procedures. torchtune when you want a PyTorch-native recipe stack with fewer dependencies. Unsloth when you are squeezing a single-GPU QLoRA run for maximum speed and minimum memory. Axolotl wins on breadth of models and objectives behind one reproducible config surface.

12. A run trains with a falling loss but the fine-tuned model behaves as if it never trained. Name a likely cause from this chapter.

A prompt-strategy or masking mismatch, so the loss fell on the wrong tokens (for example the model learned to reproduce prompts rather than answers), or a chat template that did not match the base model's expected format. The debug render is how you catch both.

13. Why is Axolotl not the right tool for pretraining from scratch?

It is built to adapt existing checkpoints, leaning on PEFT, TRL, and quantized loading, and its defaults and abstractions assume a base model. Large-scale pretraining wants a platform designed around N-dimensional parallelism over a plain model, which is torchtitan or Megatron.

14. What is the single most valuable file to read to understand Axolotl's capabilities?

The config schema under utils/schemas/. Because the YAML is the product, the schema enumerates everything the framework can do, and every field there is a promise the trainers keep.

Part IX: Design lessons

Make the config the product. Axolotl turns a fine-tune into a validated document rather than a script, so runs become diffable, reviewable, and reproducible, and expertise becomes rules the schema enforces. Terraform, Kubernetes manifests, and CI pipeline files win the same way, the artifact people edit is declarative data with a strict schema, not imperative code.

Integrate, do not reimplement. Axolotl does not rewrite attention, PEFT, or the RL objectives. It orchestrates transformers, PEFT, TRL, and DeepSpeed and supplies the connective tissue they omit. The lesson is that enormous value lives in the glue between good libraries, and a project that owns the glue well can stay small in what it invents while broad in what it enables.

Push correctness into the boundaries. Packing is only safe because attention is made boundary aware, and masking is handled where tokenization happens rather than hoped for later. Wherever a shortcut risks silent corruption, encode the invariant at the layer that can enforce it, so the fast path and the correct path are the same path.

Validate at the front door. A Pydantic schema rejects an incoherent run at load time with a clear message instead of failing deep in the loop an hour later. Cheap, early, legible validation is one of the highest-leverage investments a configurable system can make, and it pays every single run.

Keep scale-out orthogonal to the model. Data parallelism, ZeRO or FSDP sharding, and sequence parallelism are all config choices layered over an unchanged trainer, so scaling up is a field edit. Keeping distribution out of the model definition, the same instinct as torchtitan, is what lets one loop run from one GPU to many.

Gate the frontier behind flags. The plugin system lets fast-moving kernels and experimental objectives ship as self-contained modules activated by a config flag, so the stable core is never held hostage to the experimental edge. New models and new methods land soon after they appear upstream without destabilizing everyone else's runs.

Part X: Memorization framework

The one-sentence summary: Axolotl loads a validated YAML into a typed config, uses it to load a tokenizer and a possibly quantized and adapter-wrapped model, formats and masks datasets through a prompt strategy, packs them into full sequences with boundary-aware attention, and hands the result to a TRL-family trainer chosen by the config, with distribution supplied from outside by accelerate, DeepSpeed, or FSDP.

axolotl train config.yml -> Click CLI -> Pydantic schema (typed config)
  -> accelerate launch -> train(): load tokenizer, load model (+PEFT/quant)
  -> prompt strategy: tokenize + mask -> multipack: pack to sequence_len
  -> builder picks trainer: SFT / DPO / GRPO / RM (TRL)
  -> transformers/TRL loop: fwd (flash-attn, packed) -> loss -> bwd -> step
  -> save adapter or full weights (merge-lora for serving)

The chain mapped to source (all under src/axolotl/):

entry             cli/main.py, train.py
config            utils/schemas/config.py (+ datasets, peft, training, trl)
load              loaders/model.py, loaders/tokenizer.py, loaders/adapter.py
data + prompts    utils/data/sft.py, prompt_strategies/{chat_template,alpaca,...}
packing           utils/samplers/multipack.py, utils/collators/, monkeypatch/
trainers          core/builders/{causal,rl}.py, core/trainers/{base,dpo,grpo,trl}
scale-out         DeepSpeed/FSDP configs, monkeypatch/ring_attn/ (sequence parallel)
plugins           integrations/ (liger, cut_cross_entropy, spectrum, kd, ...)

Memorize these blocks:

• Identity: the validated YAML is the product, and Axolotl compiles it into transformers, PEFT, TRL, and DeepSpeed calls.
• Objectives: default SFT, rl: dpo/ipo/kto/orpo for preference, rl: grpo for RL with vLLM rollouts, reward_model: true for reward modeling.
• Prompt strategy = format + mask: the type field picks it, and a mismatch is a silent failure. Verify with preprocess --debug.
• Packing needs boundary-aware attention: sample_packing pairs with flash_attention, position ids reset per example, loss is unchanged.
• Three scale-out axes: data parallelism by default, ZeRO or FSDP for sharding, sequence_parallel_degree for sequences too long for one GPU.

Part XI: Papers and further reading

The ideas this framework assembles come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The adapter behind adapter: lora, trainable low-rank matrices on frozen base weights. The PEFT walkthrough covers the library that installs it.
2. Dettmers et al., QLoRA, Efficient Finetuning of Quantized LLMs, 2023. The 4-bit NormalFloat quantization plus LoRA recipe behind adapter: qlora, which is what makes a 70B fine-tune fit on a handful of GPUs.
3. Ouyang et al., Training language models to follow instructions with human feedback, 2022. The instruction-tuning and RLHF pipeline that made frameworks like this one matter.
4. Rafailov et al., Direct Preference Optimization, Your Language Model is Secretly a Reward Model, 2023. The objective behind rl: dpo, preference tuning with no separate reward model. Derived step by step in the DPO note on this site.
5. Ethayarajh et al., KTO, Model Alignment as Prospect Theoretic Optimization, 2024. The sibling objective behind rl: kto, which learns from unpaired desirable and undesirable examples rather than chosen-rejected pairs.
6. Shao et al., DeepSeekMath, Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. Introduces GRPO, the group-baseline objective behind rl: grpo. The GRPO note on this site works the math.
7. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The kernel whose variable-length mode makes sample packing correct, derived in the FlashAttention walkthrough.
8. Liu et al., Ring Attention with Blockwise Transformers for Near-Infinite Context, 2023. The idea behind sequence_parallel_degree, passing key and value blocks around a ring of GPUs so one long sequence spans many devices.
9. Rajbhandari et al., ZeRO, Memory Optimizations Toward Training Trillion Parameter Models, 2019. The sharding arithmetic behind the DeepSpeed configs, derived in the DeepSpeed walkthrough.
10. Zhao et al., PyTorch FSDP, Experiences on Scaling Fully Sharded Data Parallel, 2023. The PyTorch-native sharded backend the schema exposes alongside DeepSpeed.
11. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The engine behind GRPO rollouts, covered in the vLLM walkthrough.

Part XII: Final takeaway

If the single-model pieces underneath this framework are the gap, the Transformers chapter explains how a checkpoint becomes a runnable architecture, the FlashAttention chapter derives the attention kernel that makes packing and long context tractable, and the RL section covers the objectives Axolotl routes to. Then come back and read one example config end to end. It will read like a complete description of a training run, which is the entire point.