Unsloth

Part I: The mental model

your script            model, tok = FastLanguageModel.from_pretrained(...)
      |
      v
loader.py dispatch     model_type -> FastLlamaModel / FastMistralModel / FastQwen3Model ...
      |
      v
pre_patch()            LlamaAttention.forward = LlamaAttention_fast_forward
      |                RMSNorm, RoPE, gated MLP, and the loss are swapped for Triton-backed versions
      v
HF transformers model  ordinary nn.Module, loaded in 4-bit (bitsandbytes NF4) on ONE GPU
      |
      |  get_peft_model(...)   inject LoRA adapters, freeze the base, pick "unsloth" checkpointing
      v
patched + LoRA model   every hot op is now a custom autograd.Function + Triton kernel
      |
      v
TRL SFTTrainer         a completely ordinary Hugging Face training loop
      |
      v
forward / backward     RoPE, RMSNorm, SwiGLU-MLP, cross-entropy run fused, minimal saved tensors
      |                activations for each block are offloaded to CPU RAM, streamed back on backward
      v
save                   LoRA adapter, merged 16-bit weights, or GGUF for llama.cpp and Ollama

The one-sentence identity: Unsloth is a drop-in accelerator for LoRA and QLoRA fine-tuning that keeps your model a plain Hugging Face transformer and rewrites only the handful of hot operations as hand-derived autograd functions backed by Triton kernels, so the same fine-tune runs about twice as fast in a fraction of the VRAM with the exact same numerics. There is no new trainer to learn and no new checkpoint format. You call FastLanguageModel.from_pretrained instead of AutoModelForCausalLM.from_pretrained, wrap the result with get_peft_model, and hand it to the same Transformers or TRL trainer you were already using.

Two ideas carry the whole project. The first is that a transformer fine-tuning step spends almost all of its time and memory in a very short list of operations, and PyTorch's generic autograd pays for that generality by materializing and saving an intermediate tensor for nearly every elementwise op. Unsloth collapses each hot path into a single fused kernel and writes the backward by hand, so it saves only the few tensors the gradient actually needs and recomputes the rest. The second idea is that you should not have to fork the model to get this. Rather than ship a modified architecture, Unsloth patches the live classes in the transformers library the moment a model is loaded, so the speedup rides on top of the ecosystem instead of replacing it. Everything below is verified against the repository in July 2026. Unsloth tracks new model families closely and the file layout moves, so where a detail is likely to shift I say so and stay at the level of the idea.

Part II: Using it

Unsloth is a Linux-and-NVIDIA-GPU project first (recent AMD and Intel support exists and is improving). The open-source library targets a single GPU or a single node, which is exactly why it fits a free Colab T4 or one workstation card. It is not a multi-node distributed trainer, and if that is what you need the right tool is something like torchtitan. Install is an ordinary pip install:

pip install unsloth

# for the newest kernels, install from git, choosing the extra
# that matches your CUDA and torch (the README prints the exact string):
pip install --upgrade --no-cache-dir \
  "unsloth @ git+https://github.com/unslothai/unsloth.git"

The most reliable path is one of the pinned Colab or Kaggle notebooks the project maintains, because the exact combination of torch, triton, bitsandbytes, and xformers matters and the notebooks track a known good set. On your own machine, install into a fresh environment and let the library autodetect your GPU. A first real session is a QLoRA fine-tune of a small model, which loads in 4-bit and trains in well under a gigabyte of adapter state:

from unsloth import FastLanguageModel   # import BEFORE transformers/trl so the patches land
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name     = "unsloth/Meta-Llama-3.1-8B-bnb-4bit",  # a pre-quantized 4-bit repo
    max_seq_length = 2048,   # RoPE scaling is handled internally
    dtype          = None,   # None autodetects bf16 on Ampere+ or fp16 otherwise
    load_in_4bit   = True,   # QLoRA. set False for a 16-bit LoRA
)

model = FastLanguageModel.get_peft_model(
    model,
    r              = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha     = 16,
    lora_dropout   = 0,       # 0 lets Unsloth fuse the adapter path
    bias           = "none",  # "none" lets it fuse further
    use_gradient_checkpointing = "unsloth",  # offload activations to CPU RAM
    random_state   = 3407,
)

Two things above are load-bearing. First, the import unsloth must come before you import transformers or trl, because the import is what installs the monkey-patches. Import them in the other order and you silently get the stock, slow paths. Second, the argument defaults are the same ones the library ships, verified from get_peft_model in the model code, so this block is close to what you would write in practice rather than a toy. Setting lora_dropout = 0 and bias = "none" is not cosmetic, those are the values that let Unsloth take its fully fused LoRA MLP and attention paths instead of falling back.

The model that comes back is a plain Hugging Face model with LoRA adapters, so training is whatever trainer you like. The common choice is TRL's SFTTrainer:

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset("yahma/alpaca-cleaned", split = "train")

trainer = SFTTrainer(
    model         = model,
    tokenizer     = tokenizer,
    train_dataset = dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,   # effective batch 8
        warmup_steps    = 5,
        max_steps       = 60,
        learning_rate   = 2e-4,
        optim           = "adamw_8bit",    # 8-bit optimizer states, more VRAM saved
        logging_steps   = 1,
        output_dir      = "outputs",
    ),
)
trainer.train()

TRL has moved several fields (things like dataset_text_field and max_seq_length) between the trainer and SFTConfig across versions, so match the exact placement to your installed TRL. The training loop itself is entirely ordinary. Unsloth is invisible at this layer, which is the point. When training finishes you export in whatever shape you want to serve:

model.save_pretrained("lora_model")   # tiny LoRA adapters only
tokenizer.save_pretrained("lora_model")

# merge the adapters back into 16-bit weights for a standalone model:
model.save_pretrained_merged("merged_16bit", tokenizer, save_method = "merged_16bit")

# or export straight to GGUF for llama.cpp and Ollama:
model.save_pretrained_gguf("gguf_model", tokenizer, quantization_method = "q4_k_m")

The llama.cpp GGUF export is a first-class path, not an afterthought, which is a large part of why Unsloth is popular for the whole train-then-run-locally loop. For fast generation from the fine-tuned model in the same process, flip the model into inference mode:

FastLanguageModel.for_inference(model)   # enables the fused inference path
out = model.generate(**tokenizer("Hello", return_tensors = "pt").to("cuda"))

Now the mistakes beginners make. First, the import ordering above, which is the single most common cause of a fine-tune that runs but is mysteriously not faster. Second, expecting multi-GPU device_map = "auto" sharding to behave like a distributed trainer, the open-source path is built around one device and picks device_map = "sequential" by default. Third, passing a raw Hugging Face model to get_peft_model, it expects the object returned by FastLanguageModel.from_pretrained, because that object is already patched. Fourth, and most conceptually, Unsloth changes the implementation of the hot paths, never their result, so if a fine-tune diverges from a stock run the bug is almost always in your data or hyperparameters, not in the kernels.

Part III: When it is the right tool

Unsloth is the right tool when you are fine-tuning (LoRA, QLoRA, or full) a supported open model on a single GPU and you want the run to fit and finish. That covers most of the practical fine-tuning world, adapting Llama, Mistral, Gemma, Qwen, Phi, and their kin on one consumer or datacenter card, teaching in a notebook, and the train-quantize-serve loop that ends in a GGUF running under llama.cpp or Ollama. It supports the usual objectives (supervised fine-tuning, DPO, and GRPO-style RL through TRL) and has grown paths for vision models (FastVisionModel) and for reasoning-style RL with a vLLM-backed rollout for generation.

The honest cases for alternatives. Plain Hugging Face peft plus bitsandbytes when you want the most vanilla, most portable stack and do not care about the last 2x, since that is exactly what Unsloth patches on top of. Axolotl or LLaMA-Factory when you want a configuration-driven trainer with many recipes and multi-GPU orchestration handled for you, and are happy to let them own the run. torchtune when you want the PyTorch team's own native fine-tuning library with first-class distributed support. torchtitan when the task is pretraining across many nodes rather than adapting a checkpoint on one. And if all you want is the fused kernels as composable pieces to drop into your own model code, the Liger Kernel project offers a similar set of Triton kernels with a more a-la-carte philosophy. Unsloth's distinctive bet is the tight package, patched model plus kernels plus checkpointing plus export, tuned to run on the smallest hardware that could plausibly work.

The architecture-shaped warning is about where the speed does and does not come from. Unsloth accelerates the parts of a step it rewrote, and attention itself is not one of them. The attention math is delegated to FlashAttention or xformers, whose kernels are already excellent, so on a very attention-bound workload (extremely long sequences, tiny MLPs) the headline multiplier shrinks toward what the attention library alone gives you. The wins concentrate where Unsloth actually lives, the projections and their LoRA adapters, the norms, the rotary embeddings, the SwiGLU MLP, and the loss over a large vocabulary. Knowing that keeps expectations honest, a claimed 2x is a fine-tuning average, not a promise for every shape.

Part IV: The full life of one training step

The specimen: one optimizer step of a QLoRA fine-tune of Llama 3.1 8B, loaded in 4-bit with LoRA adapters on the seven projection matrices and use_gradient_checkpointing = "unsloth". The TRL trainer calls the model's forward, computes a loss, calls backward, and steps the optimizer, exactly as it would for a stock model. What follows is what happens inside that call once Unsloth has patched the classes.

Stage 1: the forward enters a patched model

At load time pre_patch() in the model module set LlamaAttention.forward = LlamaAttention_fast_forward, replaced the decoder layer and model forward methods, and wrapped the causal-LM head with CausalLM_fast_forward. So when the trainer calls the model, control immediately enters Unsloth's code rather than the stock transformers implementation. The hidden states flow into the first decoder block. Nothing about the module tree changed, the weights are still self.q_proj and friends, only the forward functions were swapped.

Stage 2: the input RMSNorm

The block's first operation is the input RMSNorm, which now routes through fast_rms_layernorm. That calls the Fast_RMS_Layernorm autograd function, whose forward launches a Triton kernel that reads the row, computes the mean square, and writes the normalized, scaled output in one pass. Crucially it saves only the input and the per-row reciprocal standard deviation for backward, not the several intermediate tensors a naive graph of square, mean, rsqrt, multiply would each stash. That single decision, repeated at every norm in every block, is a large slice of the memory savings.

Stage 3: QKV projection through the fused LoRA path

The normalized hidden state hits the query, key, and value projections. Because the base weights are 4-bit and each has a LoRA adapter, this is where LoRA_QKV earns its keep. Its forward dequantizes each 4-bit weight on the fly with fast_dequantize, computes the base projection, and adds the low-rank adapter contribution through matmul_lora, fusing the three projections and their adapters so the dequantized weights are used and discarded without being written back to memory. The manual backward recomputes what it needs rather than holding the full-precision projections resident across the whole backward pass.

Stage 4: rotary embeddings and attention

The query and key tensors are rotated by fast_rope_embedding, which calls Fast_RoPE_Embedding. Its forward reshapes the heads, groups them (an internal ROPE_GROUP_SIZE amortizes the cosine and sine loads across several heads), and applies the rotation in a Triton kernel. It saves only the small cosine and sine tables in the context, and the backward re-runs the same kernel with a backward flag rather than storing the rotated activations. Attention itself is then handed to FlashAttention or xformers. This is the honest seam of the design, Unsloth wraps and feeds the best available attention kernel rather than reimplementing the attention tiling story, which is told in the FlashAttention chapter.

Stage 5: output projection, residual, and the MLP

The attention output passes through the output projection (a LoRA_W single-matrix fused path), adds the residual, and hits the post-attention RMSNorm, another Fast_RMS_Layernorm. Then the gated MLP, which is the single most fused thing in the codebase. LoRA_MLP, via apply_lora_mlp_swiglu, takes the up and gate projections (base plus adapter), applies the SwiGLU activation with the swiglu_fg_kernel, and runs the down projection, all as one autograd function. The intermediate activation, which for an 8B model is large, is never handed to PyTorch's autograd to save. Instead the manual backward recomputes it with swiglu_DWf_DW_dfg_kernel and produces the parameter and input gradients directly. That fusion is where a big share of both the speed and the VRAM headroom comes from.

Stage 6: the loss over a large vocabulary

After the final norm, the language-model head projects the last hidden states to logits and CausalLM_fast_forward computes the loss with fast_cross_entropy_loss. Its forward, Fast_CrossEntropyLoss, launches a Triton kernel that computes a numerically stable log-sum-exp per row and the loss, saving only the per-row log-sum-exp (a float32 vector of length rows) rather than the full softmax matrix. For a small vocabulary a single kernel handles a row. For a large vocabulary like Gemma's 256K it splits each row into chunks, takes a log-sum-exp per chunk, and reduces them, using the identity that a log-sum-exp of per-chunk log-sum-exps equals the global one. The backward recomputes the gradient of the logits in place. Saving one scalar per token instead of a full rows by vocab softmax is the same online-softmax idea that powers streaming attention, applied to the loss, and it is why the classifier head stops being the memory bottleneck.

Stage 7: backward, offloaded activations, and the step

The trainer calls loss.backward(). Because each block was wrapped by the "unsloth" gradient checkpointer, the block activations were not kept on the GPU during forward, they were offloaded to pinned CPU RAM. As backward reaches each block the checkpointer streams that block's inputs back to the GPU asynchronously, overlapping the copy with the compute of the block after it, recomputes the forward, and lets the manual backward passes above run. Gradients accumulate only into the LoRA adapter tensors, since the 4-bit base weights are frozen, so the gradient and optimizer state footprint is tiny relative to the model. Finally the 8-bit AdamW optimizer steps the adapters. That closes one step, tokens in, a handful of small adapter gradients out, and a peak memory dominated by a few resident tensors rather than a full autograd tape.

Part V: Internals deep dives

Deep dive: manual autograd, and why it saves memory

PyTorch's autograd is general, so for a chain like rsqrt(mean(x*x)) * x * weight it records every intermediate as a node and saves whatever each node's backward will need. That generality is exactly the cost. Unsloth's answer is to write each hot path as a torch.autograd.Function with a hand-derived backward, which lets it make two moves the generic engine cannot. It fuses the whole chain into one Triton kernel launch, and it chooses to save the minimum and recompute the rest. The verified examples make this concrete. RMSNorm saves the input and the reciprocal standard deviation, nothing else. RoPE saves only the cosine and sine tables and re-applies the rotation backward by flipping a flag. Cross-entropy saves one log-sum-exp scalar per token and reconstructs the gradient in the backward kernel. The MLP saves the projection inputs and recomputes the SwiGLU intermediate. Custom autograd turns memory from a tax the framework collects automatically into a resource the author spends deliberately, and recomputation is nearly always cheaper than the memory traffic of keeping a large intermediate resident. This is the recompute side of the same coin as activation checkpointing, pushed down to the granularity of individual operators.

Deep dive: the hot kernels

The kernels live in unsloth/kernels/, and the list is short on purpose, because a fine-tuning step really is dominated by a few operations. Each pairs a Triton kernel with a hand-written backward:

A few details reward a close read. The cross-entropy kernel takes logit_softcapping and logit_scaling arguments, because Gemma-2 caps logits and some models scale them, and folding that into the loss kernel avoids a separate pass over the whole logits tensor. The chunked log-sum-exp path only turns on above roughly a 65K vocabulary, so Llama and Mistral use the single-kernel path and Gemma's 256K vocabulary uses the chunked one. The RoPE kernel groups heads so a single load of the cosine and sine tables serves several heads at once. None of these are approximations, which is the repeated theme, the kernels are engineered to produce the same numbers faster, and Unsloth leans hard on that exactness as a correctness guarantee. If you want to understand the Triton programming model these kernels are written in, blocks and program ids and masked loads, the parallel computing class notes build it up from GPU first principles.

Deep dive: model patching, the load-time swap

The mechanism that connects those kernels to a real model is monkey-patching. FastLanguageModel.from_pretrained in loader.py inspects the model type and dispatches to a per-family class, FastLlamaModel, FastMistralModel, FastGemma2Model, FastQwen3Model, and so on. Before the weights are loaded, that class runs a pre_patch() that reassigns methods on the live transformers classes, for example LlamaAttention.forward = LlamaAttention_fast_forward, the decoder layer forward, the model forward, and the loss function. The model that gets instantiated is a completely ordinary transformers module, it simply has faster methods. This is why Unsloth composes with the ecosystem instead of replacing it, the object you get back is still something TRL, PEFT, and generate all understand.

Two supporting pieces matter. mapper.py maps ordinary model names to Unsloth's pre-quantized 4-bit repositories (the ...-bnb-4bit ones), so asking for a base model can transparently pull a 4-bit copy that downloads faster and skips a local quantization step. And _utils.py holds the shared patching helpers and the version and dependency checks, including the gradient-checkpointing patchers imported from the companion unsloth_zoo package. The obvious fragility of this approach is that it binds to the internals of a fast-moving library, transformers changes an attention signature and a patch must follow, which is precisely why Unsloth pins and tests against specific transformers versions and warns loudly on mismatches. The layout of which family lives in which file (llama.py, mistral.py, gemma2.py, qwen3.py, and the newer additions) changes as models are added, so treat the dispatch table in loader.py as the source of truth rather than any fixed list.

Deep dive: LoRA fusion and 4-bit dequantization

QLoRA loads the base weights in 4-bit NF4 through bitsandbytes and trains small low-rank adapters on top. The naive way to run that is to dequantize a weight to 16-bit, matmul, then separately add the adapter, which touches memory several times. Unsloth's fused LoRA path collapses it. matmul_lora and fast_dequantize in kernels/utils.py let LoRA_MLP, LoRA_QKV, and LoRA_W dequantize a weight, use it, and drop it without writing the full-precision copy back, while computing the base and adapter contributions together. The autograd functions also arrange the backward so that the adapter gradients, the only trainable ones, are the primary output, and the frozen 4-bit base contributes to the input gradient without accumulating any gradient of its own. Freezing the base is what makes the optimizer state tiny, and fusing the dequantize-matmul-adapter chain is what keeps the transient 16-bit weights from ever becoming resident memory pressure. The requirement that lora_dropout = 0 and bias = "none" to unlock the fastest path comes straight from this, dropout or a bias term breaks the fused expression and forces a more general, slower route.

Deep dive: the offloaded gradient checkpointer

Setting use_gradient_checkpointing = "unsloth" selects Unsloth_Offloaded_Gradient_Checkpointer, which lives in the unsloth_zoo.gradient_checkpointing module and is wired in through _utils.py. Ordinary gradient checkpointing trades compute for memory by discarding a block's activations during forward and recomputing them during backward. Unsloth's variant adds a second trade, it moves the small set of tensors it does keep off the GPU entirely, into pinned CPU RAM, and streams them back asynchronously when backward reaches that block, so the copy overlaps with adjacent compute. The effect is that peak GPU memory stops scaling with sequence length the way a resident activation tape would, which is the concrete reason Unsloth advertises much longer context on the same card. The cost is PCIe bandwidth and some added complexity, paid back by the overlap. It is the same recompute-and-move instinct as the manual autograd, lifted from the operator level to the block level.

Part VI: Reading the repository

The tree rewards reading kernels-first, because the kernels are the thesis and the model files are the plumbing that installs them. Paths below are from the repository in July 2026 and the newer model files in particular change as families are added.

Stage 0, orientation. Read the top of unsloth/kernels/__init__.py to see the exported surface, the fast cross-entropy, RMSNorm, layernorm, RoPE, SwiGLU and GeGLU kernels, the fused LoRA entry points, and the dequant utilities. That one file is a map of everything Unsloth actually replaced. Question, which operations are here and, just as telling, which are not (attention is not).

Stage 1, one kernel end to end. kernels/rope_embedding.py is the friendliest, read Fast_RoPE_Embedding.forward and backward together and notice that the context saves only the cosine and sine tables and the backward re-runs the kernel with a backward flag. Then kernels/rms_layernorm.py for the same pattern with a saved reciprocal standard deviation. Question, what does each forward stash in ctx, and what does that imply for peak memory?

Stage 2, the loss. kernels/cross_entropy_loss.py is the best single demonstration of the whole philosophy. Read the single-row kernel and then the chunked one, and find the comment deriving that a log-sum-exp over per-chunk log-sum-exps equals the global log-sum-exp. Question, why does saving a per-row log-sum-exp instead of the softmax matrix change the memory class of the LM head, and how do logit_softcapping and logit_scaling fold in?

Stage 3, the fused LoRA MLP. kernels/swiglu.py for the forward-gate and combined backward kernels, then kernels/fast_lora.py for LoRA_MLP, LoRA_QKV, and LoRA_W, with kernels/utils.py for matmul_lora and fast_dequantize. Question, where is the SwiGLU intermediate recomputed rather than saved, and why do zero dropout and no bias unlock this path?

Stage 4, the patching machinery. models/loader.py for the dispatch from model type to a per-family class and the from_pretrained signature, then models/llama.py for pre_patch(), LlamaAttention_fast_forward, CausalLM_fast_forward, and the get_peft_model that injects adapters. Then models/mapper.py for the 4-bit name mapping and models/_utils.py for the shared helpers and the gradient checkpointer wiring. Question, at what exact moment does the transformers class get its forward replaced, and why must import unsloth precede importing transformers?

Stage 5, the frontier. models/rl.py and models/rl_replacements.py for the DPO and GRPO patches over TRL, models/vision.py and the FastVisionModel path, the kernels/moe/ subpackage and the MoE model files (qwen3_moe.py, glm4_moe.py) for expert layers, and kernels/fp8.py for the FP8 linear patches. These are the most active and least stable corners, read them last.

Where not to start, the per-family model files are long and full of version-specific special cases (softcapping, sliding windows, inference fast paths), and reading them before the kernels makes the kernels look like magic they call. Read the kernels first and the model files read like careful glue.

Part VII: Hands-on labs

Labs 1 through 4 need one GPU (a free Colab T4 is enough for a small model). Lab 5 needs only CPU plus llama.cpp. Log formats and exact numbers vary with hardware and version.

Lab 1: a real QLoRA fine-tune. Concept: the whole loop of Part II and IV.

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model, tok = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.2-1B-Instruct", max_seq_length = 2048, load_in_4bit = True)
model = FastLanguageModel.get_peft_model(
    model, r = 16, use_gradient_checkpointing = "unsloth", random_state = 3407)

ds = load_dataset("yahma/alpaca-cleaned", split = "train[:1000]")
SFTTrainer(model = model, tokenizer = tok, train_dataset = ds,
    args = SFTConfig(max_steps = 60, per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4, learning_rate = 2e-4,
        optim = "adamw_8bit", logging_steps = 1, output_dir = "out")).train()

Watch the loss fall over 60 steps and note the peak memory reported at startup. This is the concrete thing Unsloth exists to make cheap. Save the adapters with model.save_pretrained("lora") and confirm the directory is only a few megabytes, that is the whole point of freezing the 4-bit base.

Lab 2: prove the patches landed. Concept: the load-time monkey-patch.

import transformers.models.llama.modeling_llama as L
print(L.LlamaAttention.forward.__qualname__)   # stock name
import unsloth
from unsloth import FastLanguageModel
m, t = FastLanguageModel.from_pretrained("unsloth/Llama-3.2-1B-Instruct", load_in_4bit = True)
print(L.LlamaAttention.forward.__qualname__)   # now an Unsloth fast_forward

Observe that the same class attribute names a different function after the Unsloth import. Then do the experiment that teaches the ordering rule, in a fresh process import transformers and build a model before importing unsloth, and confirm the fast forward is not installed on that pre-existing instance.

Lab 3: measure the checkpointer. Concept: the offloaded gradient checkpointer of Part V.

# run the Lab 1 fine-tune twice, changing only:
#   get_peft_model(..., use_gradient_checkpointing = True)        # standard
#   get_peft_model(..., use_gradient_checkpointing = "unsloth")   # offloaded
import torch; print(torch.cuda.max_memory_allocated() / 1e9, "GB")

Compare peak memory and step time between the two. The "unsloth" mode should show lower peak memory, more so at longer max_seq_length, and let you push a sequence length or batch size that the standard mode cannot fit. That gap is the CPU offload in Stage 7 made visible.

Lab 4: call a kernel directly and check it. Concept: exactness of the manual autograd.

import torch
from unsloth.kernels import fast_cross_entropy_loss

logits = torch.randn(8, 32000, device = "cuda", dtype = torch.float32, requires_grad = True)
labels = torch.randint(0, 32000, (8,), device = "cuda")
ref = torch.nn.functional.cross_entropy(logits, labels)
fast = fast_cross_entropy_loss(logits, labels)
print(float(ref), float(fast), abs(float(ref) - float(fast)))

Confirm the two losses agree to floating-point tolerance, then backward through each and compare the logit gradients. This is the experiment that turns the exactness claim from marketing into something you have checked with your own hands. Repeat with fast_rope_embedding against a plain PyTorch rotary implementation.

Lab 5: export and run locally. Concept: the train-then-serve loop.

model.save_pretrained_gguf("gguf_model", tok, quantization_method = "q4_k_m")

# then load the produced .gguf in llama.cpp or Ollama
ollama create mymodel -f Modelfile      # Modelfile points at the exported gguf
ollama run mymodel "hello"

Confirm the fine-tuned behavior survives the round trip into llama.cpp's GGUF format and 4-bit quantization. This closing step is why Unsloth is popular for the complete loop rather than just the training minute.

Lab 6 (optional, needs more VRAM): GRPO with a vLLM rollout. Concept: RL fine-tuning reusing the same patched model.

model, tok = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.2-3B-Instruct", max_seq_length = 1024,
    load_in_4bit = True, fast_inference = True)   # fast_inference uses vLLM for generation

With fast_inference = True the same model generates rollouts through vLLM and trains through the Unsloth kernels, which is how GRPO-style RL fits on a single card. Follow one of the project's GRPO notebooks for the reward and trainer wiring, the point of the lab is to see that the acceleration and the RL loop share one model object.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is Unsloth, in one sentence?

A drop-in accelerator for LoRA and QLoRA fine-tuning that keeps the model a plain Hugging Face transformer and rewrites only the hot operations, RoPE, RMSNorm, the gated MLP, and cross-entropy, as hand-derived autograd functions backed by Triton kernels, patched in at load time.

2. Where does the speed and memory saving actually come from?

From fusing each hot path into one Triton kernel and writing its backward by hand so it saves the minimum and recomputes the rest, from fusing the 4-bit dequantize, base matmul, and LoRA adapter so transient full-precision weights never become resident, from freezing the base so optimizer state is tiny, and from offloading block activations to CPU RAM. Attention itself is delegated to FlashAttention or xformers and is not where the wins come from.

3. Why must you import unsloth before transformers and trl?

Because the import is what installs the monkey-patches that reassign forward methods on the transformers classes. Import the other libraries first and you build models that still point at the stock, slow methods, so the run works but is not accelerated.

4. What does a custom autograd.Function buy over stock autograd here?

Two things the generic engine cannot do, fusing a whole operation into one kernel launch, and deciding exactly which tensors to save versus recompute. RMSNorm saves the input and reciprocal std, RoPE saves only cos and sin, cross-entropy saves one log-sum-exp per token, the MLP recomputes its SwiGLU intermediate. Memory becomes something the author spends deliberately rather than a tax the framework collects on every intermediate.

5. Why can cross-entropy save just one scalar per token?

Because the loss and its gradient can be reconstructed in the backward kernel from the row's log-sum-exp and the stored logits, so the full rows by vocab softmax never needs to be kept. For a 256K vocabulary it computes the log-sum-exp in chunks and reduces them, using the identity that a log-sum-exp of per-chunk log-sum-exps is the global one. It is the online-softmax trick applied to the loss.

6. How does the model get patched without forking transformers?

from_pretrained dispatches on model type to a per-family class whose pre_patch() reassigns methods on the live transformers classes, for example LlamaAttention.forward = LlamaAttention_fast_forward, before the weights load. The instantiated model is an ordinary transformers module with faster methods, so PEFT, TRL, and generate all still understand it.

7. What is special about use_gradient_checkpointing = "unsloth"?

It selects the offloaded checkpointer, which not only discards and recomputes block activations but moves the tensors it keeps to pinned CPU RAM and streams them back asynchronously during backward, overlapping the copy with compute. That decouples peak GPU memory from sequence length and is why much longer context fits on the same card.

8. Why do lora_dropout = 0 and bias = "none" matter for speed?

Because the fully fused LoRA MLP and attention paths express the base plus adapter as one computation. A nonzero dropout or an added bias term breaks that fused expression and forces a more general, slower route, so the fastest path is unlocked only at those values.

9. Does Unsloth change training results relative to a stock run?

No, the kernels are engineered to be exact rather than approximate, so for the same seed, data, and hyperparameters the numerics match to floating-point tolerance. If a fine-tune diverges from a stock run the cause is almost always the data or configuration, not the kernels, which is a claim you can and should verify directly.

10. Why is attention not accelerated by an Unsloth kernel?

Because FlashAttention and xformers already provide near-optimal attention kernels, so Unsloth wraps and feeds them rather than reimplementing the tiling and online-softmax math. On a strongly attention-bound shape the headline multiplier therefore shrinks toward what the attention library alone delivers.

11. When would you reach for Axolotl, torchtune, or torchtitan instead?

Axolotl or LLaMA-Factory for a configuration-driven trainer with recipes and multi-GPU orchestration handled for you. torchtune for the PyTorch team's native fine-tuning library with first-class distributed support. torchtitan when the task is multi-node pretraining rather than single-GPU adaptation. Unsloth wins when you want the smallest hardware that could work to fit and finish a fine-tune fast.

12. A fine-tune runs but is no faster than a plain PEFT run. First suspect?

Import order. If transformers or a model was imported or built before import unsloth, the fast forwards were never installed. Confirm by printing the qualified name of a patched class's forward before and after the Unsloth import.

13. What is in a saved adapter, and why is it tiny?

Only the low-rank LoRA matrices for the targeted projections, a few megabytes, because the 4-bit base is frozen and untouched. To ship a standalone model you merge the adapters back into 16-bit weights with save_pretrained_merged, or export straight to GGUF for llama.cpp with save_pretrained_gguf.

14. Why does Unsloth pin specific transformers versions?

Because monkey-patching binds to transformers internals, an attention signature or a forward structure changing upstream can break a patch. Pinning and version checks let Unsloth guarantee the patched paths match the library it is patching, and it warns rather than silently misbehaving on a mismatch.

Part IX: Design lessons

Optimize the short list, not the whole program. A fine-tuning step spends its time and memory in a handful of operations, so Unsloth rewrites exactly those and leaves everything else alone. Finding the true hot set and going deep on it beats spreading effort evenly, which is the profile-first instinct behind every serious performance win.

Spend memory deliberately with manual backward. Generic autograd saves an intermediate for every op by default. Writing the backward by hand turns saved memory into an explicit choice, save the reciprocal std, recompute the SwiGLU, keep one log-sum-exp per token. Recomputation is usually cheaper than the memory traffic of keeping a large tensor resident.

Ride the ecosystem, do not replace it. By patching live transformers classes instead of forking the model, Unsloth stays compatible with PEFT, TRL, and generate for free. The cost is a tight coupling to upstream internals, paid for with version pins. Extending a system from the outside keeps its whole surrounding world usable.

Exactness is a feature, market it. Because the kernels reproduce the same numbers, Unsloth can promise no accuracy change and invite users to verify it. An optimization you can prove equivalent is far easier to adopt than one that asks for trust, and it makes debugging tractable, any divergence is your bug, not the kernel's.

Normalize by the global token count. Unsloth's team surfaced and fixed a subtle gradient-accumulation bug where a loss averaged per microbatch, rather than over the total valid tokens, silently mis-weights variable-length sequences across accumulation steps. This is the same correctness point that torchtitan makes by all-reducing the valid-token count before computing the loss, whenever you split a batch, normalize by the global count, not the per-shard one.

Make the smallest hardware the target. Designing so a real fine-tune fits on a free T4 forces every VRAM decision to be honest and puts the tool in far more hands than a design that assumes an eight-GPU box. Constraining to the cheapest plausible machine is a product decision as much as an engineering one.

Part X: Memorization framework

The one-sentence summary: Unsloth patches a stock Hugging Face model at load time so its hot paths, RoPE, RMSNorm, the fused LoRA gated MLP, and cross-entropy, become hand-written Triton kernels with manual backward passes that save the minimum and recompute the rest, and it offloads block activations to CPU RAM, so an ordinary fine-tune runs about twice as fast in a fraction of the VRAM with identical numerics.

import unsloth (installs the patches) -> FastLanguageModel.from_pretrained (4-bit, dispatch, pre_patch)
  -> get_peft_model (LoRA adapters, freeze base, "unsloth" checkpointing)
  -> SFTTrainer.train (ordinary loop) -> fused RoPE/RMSNorm/SwiGLU-MLP/cross-entropy
  -> backward with recompute + CPU-offloaded activations -> 8-bit AdamW on adapters
  -> save adapter / merged 16-bit / GGUF

The chain mapped to source:

entry            unsloth/models/loader.py (FastLanguageModel, from_pretrained, dispatch)
patching         models/llama.py (pre_patch, *_fast_forward, get_peft_model), models/mapper.py, models/_utils.py
kernels          kernels/rope_embedding.py, rms_layernorm.py, swiglu.py, cross_entropy_loss.py
lora fusion      kernels/fast_lora.py (LoRA_MLP, LoRA_QKV, LoRA_W), kernels/utils.py (matmul_lora, fast_dequantize)
checkpointing    use_gradient_checkpointing="unsloth" -> unsloth_zoo.gradient_checkpointing (offloaded)
export           save_pretrained_merged, save_pretrained_gguf -> llama.cpp / Ollama

Memorize these blocks:

• The hot set: RoPE, RMSNorm, SwiGLU/GeGLU MLP, cross-entropy, and the fused LoRA matmuls. Attention is delegated to FlashAttention or xformers, not rewritten.
• Manual autograd rule: fuse into one kernel, save the minimum (reciprocal std, cos/sin, one log-sum-exp per token), recompute the rest in backward.
• Patch, don't fork: import unsloth first, it reassigns forward methods on live transformers classes so PEFT, TRL, and generate still work.
• QLoRA math: 4-bit NF4 base frozen, tiny LoRA adapters trained, dequant-matmul-adapter fused, lora_dropout=0 and bias="none" unlock the fastest path.
• Offloaded checkpointing: "unsloth" mode streams block activations to pinned CPU RAM and back, decoupling peak memory from sequence length.
• Scope: single GPU or single node, exact numerics, roughly 2x faster and around 70% less VRAM on the open-source path.

Part XI: Papers and further reading

The ideas in this walkthrough come from a short list of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The adapter method every Unsloth fine-tune trains, a frozen base plus small low-rank matrices. The PEFT walkthrough covers the library that injects them.
2. Dettmers et al., QLoRA, Efficient Finetuning of Quantized LLMs, 2023. The 4-bit NF4 base plus LoRA recipe that Unsloth accelerates, and the source of the bitsandbytes quantization its fused paths dequantize on the fly.
3. Tillet et al., Triton, An Intermediate Language and Compiler for Tiled Neural Network Computations, MAPL 2019. The tile-based GPU language every Unsloth kernel is written in. The Triton walkthrough and the parallel computing class build the programming model from first principles.
4. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The attention kernel Unsloth delegates to rather than rewrites, covered in the FlashAttention walkthrough.
5. Su et al., RoFormer, Enhanced Transformer with Rotary Position Embedding, 2021. Introduces the rotary embedding that Fast_RoPE_Embedding applies in one fused kernel and re-applies in backward instead of saving activations.
6. Zhang and Sennrich, Root Mean Square Layer Normalization, 2019. The normalization Fast_RMS_Layernorm computes while saving only the input and the reciprocal standard deviation.
7. Shazeer, GLU Variants Improve Transformer, 2020. Where the SwiGLU and GeGLU gates in the fused MLP kernels come from.
8. Milakov and Gimelshein, Online normalizer calculation for softmax, 2018. The online log-sum-exp idea behind the chunked cross-entropy kernel, derived in the softmax note on this site.
9. Chen et al., Training Deep Nets with Sublinear Memory Cost, 2016. The recompute-instead-of-store argument that both gradient checkpointing and the offloaded checkpointer rest on.
10. Dettmers et al., 8-bit Optimizers via Block-wise Quantization, 2021. The adamw_8bit optimizer that shrinks the optimizer state, from the same bitsandbytes lineage as the NF4 base.

Part XII: Final takeaway

If the single-device pieces this project assumes are the gap, the ML implementations section builds LoRA, attention, and normalization from scratch, the softmax and cross-entropy notes derive the log-sum-exp trick the loss kernel uses, the mixed-precision notes cover the bf16 and 4-bit numerics, and the parallel computing class teaches the Triton model these kernels are written in. Then read kernels/rope_embedding.py once more, it will read like an ordinary rotary embedding that simply refused to save what it could recompute, which is the entire idea.