PEFT

Part I: The mental model

base = AutoModelForCausalLM.from_pretrained(...)   plain frozen model (transformers)
      |
      v
cfg  = LoraConfig(r=8, target_modules=["q_proj","v_proj"], ...)   the recipe
      |
      v
get_peft_model(base, cfg)          src/peft/mapping_func.py
      |
      |  PeftModel wraps base, freezes every weight
      |  BaseTuner.inject_adapter walks named_modules()
      |  matches target_modules, swaps nn.Linear -> lora.Linear (keeps base_layer)
      v
PeftModel                          only lora_A / lora_B require grad (~0.1% of params)
      |
      v
train loop                         Trainer / accelerate, gradients touch adapters only
      |
      v
model.save_pretrained("out/")      adapter_config.json + adapter_model.safetensors (MBs)
      |
      v
merge_and_unload()                 fold (B A) * scaling into W, return a plain model

The one-sentence identity. PEFT is a thin transformation layer over a frozen Transformers model, which reads a config object describing an adapter method and injects small trainable modules into named submodules so that fine-tuning updates a fraction of a percent of the weights and saves as megabytes rather than gigabytes. Full fine-tuning copies and updates every parameter, which for a 7B model means optimizer state and gradients that dwarf the model itself and a separate multi-gigabyte checkpoint per task. PEFT inverts the economics. The base weights stay frozen and shared, each task owns a tiny adapter, and the same base can carry dozens of adapters swapped at will.

Two design commitments make this work. First, the base model is never rewritten by hand. PEFT walks the module tree, matches layers by name against a pattern in the config, and replaces each matched module with a wrapper that holds the original as a base_layer and adds the adapter beside it. The model code you fine-tune is the ordinary Transformers model, untouched. Second, the method is data, not code. Whether you use LoRA, QLoRA, IA3, or a soft-prompt method is a choice of config class, and the dispatch from config to the machinery that edits the network is a lookup table. Everything in this chapter is written against a recent PEFT on the main line in 2026. The library moves steadily and adds methods often, so where a file path or a field name is likely to have shifted I describe the component by its role and stay at the level of the idea.

Part II: Using it

PEFT installs from PyPI and rides on top of Transformers, Accelerate, and PyTorch. For the 4-bit QLoRA path you also want bitsandbytes.

pip install peft transformers accelerate
# for QLoRA (4-bit base weights) on an NVIDIA GPU:
pip install bitsandbytes

A first real session is a LoRA fine-tune of a small causal model. The shape of the code is the same at every scale. Load a plain model, describe an adapter with a config, wrap the model with get_peft_model, then train it like any other nn.Module. The only PEFT-specific lines are the config and the wrap.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
tok  = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")

peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                       # rank of the low-rank update
    lora_alpha=16,            # scaling, applied as alpha / r
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],   # which linears get an adapter
    bias="none",
)

model = get_peft_model(base, peft_config)
model.print_trainable_parameters()
# trainable params: 851,968 || all params: 1,236,666,368 || trainable%: 0.0689

The print_trainable_parameters line is the whole pitch in one number. Under a tenth of a percent of the weights carry gradients. From here the model trains with the ordinary Transformers Trainer or a hand-written loop, and only the adapter tensors move. Saving writes just those tensors.

from transformers import Trainer, TrainingArguments

trainer = Trainer(model=model, args=TrainingArguments("out", num_train_epochs=1),
                  train_dataset=train_ds, data_collator=collator)
trainer.train()

model.save_pretrained("my-lora")   # writes adapter_config.json + adapter_model.safetensors

The output directory is tiny, a few megabytes for a rank-8 adapter, because it holds only lora_A and lora_B, not the base. Reloading needs the base plus the adapter.

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
model = PeftModel.from_pretrained(base, "my-lora")   # base + adapter, ready to run

Now the mistakes beginners make. First, target_modules has to name modules that exist. The names are suffixes of the module paths in the base model, so q_proj matches model.layers.0.self_attn.q_proj and every sibling. Name a module that is not there and PEFT raises rather than silently attaching nothing. When unsure, pass the special value target_modules="all-linear" to adapt every linear layer except the output head, or print the module names first.

for name, module in base.named_modules():
    if isinstance(module, torch.nn.Linear):
        print(name)      # q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, ...

Second, forgetting to save the head. For sequence classification you attach a fresh classifier that is not part of any adapter, so it must be listed in modules_to_save or it will not be trained or saved. Third, precision surprises. A LoRA adapter is created in a working dtype and, if the base is in fp16 or bf16, the adapter math runs in a compatible dtype, so mixing a half-precision base with an fp32 optimizer without care can waste memory or silently downcast. Fourth, and most common, expecting the base to change. It does not. Until you call merge_and_unload, the base weights are frozen and the adapter is a separate additive term, which is exactly what lets you disable it, swap it, or keep several around.

Part III: When it is the right tool

PEFT is the right tool when you are adapting an existing pretrained checkpoint to a task or a style and you want the result to be cheap to train, cheap to store, and easy to serve alongside other adaptations of the same base. That covers most of applied fine-tuning, instruction tuning a base model, giving a chat model a domain voice, task-specific classifiers over a shared encoder, and any setting where you will produce many adaptations of one base and cannot afford a full checkpoint for each. It integrates directly with the Transformers Trainer, with TRL for preference and RL fine-tuning, with Accelerate and DeepSpeed and FSDP for multi-GPU, and with Diffusers for image models, so the adapter you train is a first-class citizen across the ecosystem.

The honest cases for alternatives. Full fine-tuning when the base is small, the target task is far from the pretraining distribution, and you have the memory, because updating all weights is still the highest-ceiling option when you can pay for it. torchtitan and similar pretraining platforms when the job is training a model from scratch or heavy continued pretraining rather than adapting a finished one, which is a different problem with different defaults. Unsloth when you want a tuned, kernel-optimized LoRA and QLoRA path that squeezes more speed and memory out of a single GPU, it wraps the same ideas with custom Triton kernels. Axolotl or LLaMA-Factory when you want an opinionated YAML-driven training harness rather than a library you call from Python, both of which use PEFT underneath. torchtune for a PyTorch-native fine-tuning stack with its own recipes. The original AdapterHub line of work, now adapters, for a research-oriented take on bottleneck adapters and adapter composition. PEFT itself is deliberately a library of methods and a clean injection mechanism, not a training framework with a thousand options.

The method-shaped warning is that not every parameter-efficient method suits every task. LoRA and its relatives change what the weights compute and reach the widest range of tasks, including ones that need the model to learn genuinely new behavior. Soft-prompt methods, prompt tuning and prefix tuning and P-tuning, steer a frozen model by learning inputs rather than weights, which is cheaper still and elegant for steering a large model toward a task it already almost knows, but weaker when the task demands new capability and sensitive to how many virtual tokens you give it. IA3 is extremely light and strong in the few-shot regime it was designed for. Reaching for a soft prompt where the task really needs LoRA is the classic mismatch, and the symptom is a training curve that plateaus early well short of what a small LoRA would reach.

Part IV: The full life of one adapter

The specimen is a rank-8 LoRA on the query and value projections of a causal language model. We follow it from the config through get_peft_model, into a single linear layer, through one forward pass, out to disk, and finally through merge_and_unload for inference. The same path holds for QLoRA, where the only difference is that the frozen base linear is a 4-bit quantized layer rather than an ordinary one.

Stage 1: the config as a recipe

A LoraConfig is a dataclass that subclasses PeftConfig (in src/peft/config.py). It records peft_type=PeftType.LORA, the rank r, the scaling lora_alpha, the dropout, the target_modules pattern, the bias policy, modules_to_save, and a set of newer switches like use_rslora and use_dora. It is pure data. It knows the recipe but does nothing to a model. Its peft_type is the key that later selects which tuner edits the network.

Stage 2: get_peft_model and the dispatch

get_peft_model(base, peft_config) lives in the mapping module (src/peft/mapping_func.py on recent versions, historically mapping.py). It reads peft_config.task_type to choose a task-specific PeftModel subclass, for a causal model that is PeftModelForCausalLM, and constructs it around the base. PeftModel.__init__ (in src/peft/peft_model.py) then reads peft_config.peft_type, looks up the matching tuner class in the peft-type-to-model mapping, and builds it. For LoRA that class is LoraModel (in src/peft/tuners/lora/model.py), a subclass of the shared BaseTuner in src/peft/tuners/tuners_utils.py. Constructing the tuner is what triggers the actual surgery.

Stage 3: inject_adapter walks the tree

BaseTuner.inject_adapter collects every module name from base.named_modules() and, for each one, asks _check_target_module_exists(peft_config, key) whether it matches. The match is a suffix comparison against the strings in target_modules, or a full regex if you passed one, so "q_proj" matches every ...self_attn.q_proj. For each match the tuner calls _create_and_replace, which builds a new module with _create_new_module and swaps it into the parent with _replace_module. The dispatch inside _create_new_module is by the type of the target. An nn.Linear becomes a lora.Linear, an nn.Embedding becomes a lora.Embedding, a Conv2d becomes a lora.Conv2d, and a bitsandbytes Linear4bit becomes the 4-bit LoRA layer from src/peft/tuners/lora/bnb.py. This type dispatch is the single seam that makes QLoRA fall out of the same code path.

for name in base.named_modules():
    if _check_target_module_exists(cfg, name):        # suffix / regex match
        parent, target, target_name = _get_submodules(base, name)
        new = _create_new_module(cfg, adapter_name, target)   # type dispatch
        _replace_module(parent, target_name, new, target)     # keep target as base_layer

The replacement wrapper is the crux. A lora.Linear (in src/peft/tuners/lora/layer.py) stores the original linear as self.base_layer and adds, keyed by adapter name, an nn.ModuleDict of low-rank factors. For an adapter named "default" on a linear with in_features by out_features, it creates lora_A["default"] = nn.Linear(in_features, r, bias=False) whose weight is shaped (r, in_features), and lora_B["default"] = nn.Linear(r, out_features, bias=False) whose weight is shaped (out_features, r). It records the scaling lora_alpha / r, sets up the dropout, initializes lora_A with a Kaiming-style random init and lora_B to all zeros, so at step zero the adapter contributes nothing and training starts from the frozen base exactly. Finally _mark_only_adapters_as_trainable sets requires_grad=False on every base parameter and leaves only the adapter tensors trainable, honoring the bias policy.

Stage 4: one forward pass

When input x reaches a wrapped linear, the layer runs the frozen base and adds the low-rank correction on top. In its simplest form the forward is this.

# peft/tuners/lora/layer.py, Linear.forward, simplified
result = self.base_layer(x)                       # frozen W x  (+ bias)
for adapter in self.active_adapters:
    A = self.lora_A[adapter]
    B = self.lora_B[adapter]
    drop = self.lora_dropout[adapter]
    s = self.scaling[adapter]                     # = lora_alpha / r
    result = result + B(A(drop(x))) * s           # (out_features) low-rank add
return result

The base path is untouched, so the model still computes exactly what it did before, and the adapter adds a small residual. The cost of that residual is two skinny matmuls, x down to r dimensions through A and back up to out_features through B, which is why rank r is the whole knob. During backward only A and B accumulate gradients because everything else was frozen in stage 3, so the optimizer state is proportional to the adapter size, not the model size. That is the memory win that lets a 7B LoRA fine-tune fit where a full one never could.

Stage 5: saving only what changed

model.save_pretrained("out") calls get_peft_model_state_dict (in src/peft/utils/save_and_load.py), which filters the full state dict down to the keys the adapter owns, the lora_A and lora_B weights plus any modules_to_save, and writes them as adapter_model.safetensors. Alongside it writes adapter_config.json, the serialized LoraConfig including the base model name so a reload knows what to attach to. Nothing of the base is copied. This is why the same 7B base can host a folder of adapters where each is a few megabytes, and why you can share a task adapter as an artifact that assumes the reader already has the base.

Stage 6: merge and unload for inference

At inference the extra matmul per adapted layer is pure overhead you no longer need, because training is over and the adapter can be folded into the weight it corrects. merge_and_unload (on LoraModel) does exactly that. For every wrapped layer it computes the delta weight and adds it to the base weight in place, then replaces the wrapper with the now-updated base layer, so the returned object is a plain Transformers model with no PEFT code in the forward path and no latency penalty.

# the folded update, from get_delta_weight, conceptually
delta_W = (B.weight @ A.weight) * scaling          # shape (out_features, in_features)
base_layer.weight.data += delta_W                   # now W' = W + (alpha/r) B A

merged = model.merge_and_unload()                   # returns a plain nn.Module
merged.save_pretrained("merged-model")              # full model, no adapter needed

That closes the loop. A recipe became a wrapper, the wrapper learned a low-rank correction while the base sat frozen, the correction saved as megabytes, and at the end the correction dissolved back into the weights it was always a delta of. If you would rather keep the adapter separate at serving time, do not merge. Serving stacks like vLLM and SGLang can load LoRA adapters dynamically and even multiplex many adapters over one shared base in a single process, which is the deployment payoff of keeping the adapter as a detachable term.

Part V: Internals deep dives

Deep dive: deriving the LoRA parameter savings

LoRA rests on a hypothesis and a small piece of arithmetic. The hypothesis is that the change a task makes to a big weight matrix has low intrinsic rank, so instead of learning a dense update you can learn a product of two thin matrices. The arithmetic is what makes it cheap. Take a linear layer with weight W of shape (d_out, d_in). Full fine-tuning trains every entry, so it updates d_out * d_in parameters.

LoRA freezes W and learns ΔW = B A, where A has shape (r, d_in) and B has shape (d_out, r), with the rank r much smaller than either dimension. The effective weight becomes W + (α/r) B A. The trainable count is the size of the two factors.

full fine-tune of W (d_out x d_in):   d_out * d_in trainable

LoRA:   W frozen,  dW = B A,  applied as (alpha / r) * B A
        A: (r, d_in)     ->  r * d_in params
        B: (d_out, r)    ->  d_out * r params
        trainable = r * d_in + r * d_out = r * (d_in + d_out)

square case d_in = d_out = d:
        full = d^2        LoRA = 2 r d        ratio = 2 r / d

d = 4096, r = 8:   ratio = 16 / 4096 = 1 / 256 ~ 0.39% per matrix

The saving is the ratio 2r/d, so at rank 8 on a 4096-wide projection you train about one parameter for every 256 the full update would, and the win grows linearly as the model gets wider because d grows while r stays fixed. The scaling factor α/r multiplies the update but adds no parameters, it only rescales how strongly the learned correction is applied, which decouples the learning-rate-like magnitude from the rank. A rank-stabilized variant divides by sqrt(r) instead to keep the scale steady as rank changes, selectable with use_rslora.

Now a whole model, to see the number PEFT prints. Take a Llama-style 7B with hidden size 4096 and 32 layers, and put a rank-8 LoRA on just q_proj and v_proj, each a 4096 x 4096 matrix.

per matrix (4096 x 4096), r = 8:   2 * 4096 * 8 = 65,536
q_proj + v_proj per layer:         2 * 65,536 = 131,072
x 32 layers:                       4,194,304  ~ 4.19M trainable
as fraction of ~6.7B base:         ~0.062%

Four million trainable parameters against a nearly seven billion parameter base. That is the concrete meaning of "under a tenth of a percent", and it is why the optimizer state, the gradients, and the saved checkpoint all shrink by three orders of magnitude while the forward pass stays essentially the base model plus a rounding error of extra compute. Push the rank up to reach more expressive corrections, widen target_modules to include the MLP projections to reach more of the network, and the count scales predictably by the same formula.

Deep dive: the config system and the tuner registry

The config layer is small and worth copying. Every method ships a config dataclass that subclasses PeftConfig, and each config carries a peft_type from the PeftType enum (in src/peft/utils/peft_types.py). The get_peft_model entry point uses two lookups. The task_type selects a PeftModel subclass that knows how a causal, seq2seq, sequence-classification, or token-classification model wires inputs and outputs, and the peft_type selects the tuner that edits the modules. Both are plain dictionaries from enum value to class. Adding a new method to PEFT is, at the plumbing level, writing a config, a tuner model, a layer, and registering the pair, which is why the library can carry a long list of methods without the core growing.

That list is genuinely long. Alongside LoRA the tree under src/peft/tuners/ holds IA3, prefix tuning, prompt tuning, P-tuning, AdaLoRA (rank that adapts during training), LoHa and LoKr (Hadamard and Kronecker low-rank forms), OFT and BOFT (orthogonal fine-tuning), VeRA (shared random bases with tiny per-layer vectors), the adaption-prompt Llama-Adapter method, and more. They all reach the network through the same BaseTuner and BaseTunerLayer contract, so once you understand how LoRA injects, you understand the mechanism for all of them, and only the per-layer math differs. The exact set of methods grows release to release, so treat any list as a snapshot.

Deep dive: QLoRA and the 4-bit base

QLoRA is not a separate PEFT method. It is LoRA whose frozen base is quantized to 4 bits, and the reason it works cleanly is the type dispatch from stage 3. You load the base through Transformers with a bitsandbytes quantization config, so the linear layers arrive as Linear4bit modules holding 4-bit weights, and when the tuner walks the tree it dispatches those to the 4-bit LoRA layer in src/peft/tuners/lora/bnb.py instead of the ordinary one. The adapter factors A and B stay in a normal compute dtype, bf16 by convention, and the frozen base is dequantized on the fly inside the matmul.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NormalFloat4, from the QLoRA paper
    bnb_4bit_use_double_quant=True,         # quantize the quantization constants too
    bnb_4bit_compute_dtype=torch.bfloat16,  # dtype the dequantized matmul runs in
)

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B",
                                            quantization_config=bnb)
base = prepare_model_for_kbit_training(base)     # cast norms to fp32, enable grad ckpt, ...

cfg = LoraConfig(task_type="CAUSAL_LM", r=16, lora_alpha=32,
                 target_modules="all-linear", lora_dropout=0.05)
model = get_peft_model(base, cfg)

The three ideas from the QLoRA paper are all visible here. NF4 is a 4-bit data type shaped for the roughly normal distribution of neural network weights, so it packs more accuracy into four bits than a plain integer grid. Double quantization quantizes the per-block scaling constants themselves, shaving a further fraction of a bit per parameter. And paged optimizers, requested through the bitsandbytes optimizer rather than PEFT, absorb the transient memory spikes that would otherwise cause out-of-memory during long-sequence steps. The prepare_model_for_kbit_training helper (in src/peft/utils/other.py) does the housekeeping that keeps 4-bit training stable, casting layer norms and the language head to fp32, enabling gradient checkpointing, and making the input embeddings produce gradients so activation checkpointing has something to attach to. The result is the headline from the paper, that a 65B model can be fine-tuned on a single 48GB GPU, because the frozen base costs about four bits per parameter and only the tiny adapter carries optimizer state.

One caution specific to the 4-bit path. Merging is lossy. Folding a bf16 delta into a 4-bit weight means dequantizing, adding, and requantizing, which does not round-trip exactly, so for QLoRA the common practice is to keep the adapter separate at serving time, or to merge into a dequantized higher-precision copy of the base rather than back into 4 bits. The mechanism supports both, and the choice is about how much accuracy you are willing to trade for a single merged artifact. For quantized deployment more broadly the story continues in llama.cpp and TensorRT-LLM.

Deep dive: LoRA versus the soft-prompt methods and IA3

LoRA edits weights. The soft-prompt family edits inputs, and IA3 edits activations by a learned scale. They divide cleanly by where the new parameters live, and that division is mirrored in the code by whether they go through the tuner-injection path or the prompt-encoder path.

IA3, which stands for infused adapter by inhibiting and amplifying inner activations, is the lightest weight-space method here. Instead of adding a low-rank matrix it learns a vector per target module and multiplies the module elementwise, scaling the keys and values in attention and the intermediate activations in the feed-forward block. An IA3Config names both the target_modules it rescales and, separately, the feedforward_modules among them, because for a feed-forward module the learned vector scales the input to the down projection rather than the output. In src/peft/tuners/ia3/layer.py the forward is a multiply, not an add, and because a diagonal rescale commutes into the weight it merges by scaling rows or columns of the base matrix. IA3 trains even fewer parameters than a small LoRA and was designed for the few-shot regime where that frugality is a strength.

The soft-prompt methods live in a different branch of PeftModel. Their configs subclass the prompt-learning config rather than being tuner configs, and PEFT does not inject anything into the base modules. Instead PeftModel.forward calls a prompt encoder to produce virtual token representations and prepends them to the sequence. Prompt tuning (in src/peft/tuners/prompt_tuning/) learns a small table of input embeddings prepended once at the bottom of the network, the lightest possible intervention. P-tuning (in src/peft/tuners/p_tuning/) generates those virtual embeddings through a small LSTM or MLP encoder rather than storing them directly, which stabilizes optimization. Prefix tuning (in src/peft/tuners/prefix_tuning/) is the heaviest of the three because it injects trainable key and value vectors into the attention of every layer, delivered as the model's past_key_values, so it steers the whole depth of the network rather than only the input. During training the prefix is produced through a reparameterizing MLP for stability, which can be dropped afterward. Because these methods never touch the base weights there is nothing to merge, the learned prompt is simply supplied at inference.

Deep dive: the BaseTuner and BaseTunerLayer contract

The reason all the weight-space methods feel the same is a small contract in src/peft/tuners/tuners_utils.py. BaseTuner owns the tree walk, the match test, and the replacement, and asks each method to supply the pieces that differ, how to test a target, how to build the new module, and how to mark trainables. BaseTunerLayer is the mixin every adapted layer implements. It holds the base_layer, tracks which adapters are active, and provides merge, unmerge, and the accessor that returns the wrapped original. This is what makes several capabilities uniform across methods. You can hold multiple adapters on one model and switch with set_adapter, add another with add_adapter, temporarily turn all of them off with the disable_adapter context manager to recover the base model's behavior, and merge or unmerge without unloading when you want to toggle folded weights in place. None of that logic lives in LoRA specifically, it lives in the shared contract, so a new method inherits it for free.

model.add_adapter("french", LoraConfig(...))    # a second adapter on the same base
model.set_adapter("french")                      # route forward through it
with model.disable_adapter():
    base_output = model(**batch)                 # frozen base, no adapter
model.set_adapter("default")

Part VI: Reading the repository

The source lives under src/peft/ and is small enough to read in a sitting. Paths below are described by role, since the tree is reorganized from time to time.

Stage 0, orientation. Read the top-level README, then the conceptual pages in the docs on LoRA and on the adapter injection idea, then skim src/peft/__init__.py to see the public surface, the config classes, get_peft_model, PeftModel, and the method configs, all re-exported from one place. Question, which names are the front door and which are internal.

Stage 1, the dispatch. Read the mapping module that defines get_peft_model and the two lookup tables, from task type to PeftModel subclass and from peft_type to tuner. Then read config.py to see what a PeftConfig is. Questions, where does the branch between weight-space tuners and prompt-learning methods happen, and what exactly does get_peft_model return.

Stage 2, PeftModel. Read src/peft/peft_model.py, first the base PeftModel, then PeftModelForCausalLM. Watch how it builds the tuner, how save_pretrained and from_pretrained route through the state-dict helpers, and how the prompt-learning path prepends virtual tokens in forward. Question, what state does a PEFT model add beyond the base.

Stage 3, the tuner core. Read src/peft/tuners/tuners_utils.py, the BaseTuner and BaseTunerLayer contract, as the spine of the whole library. Follow inject_adapter, _check_target_module_exists, and the merge and unmerge methods. Questions, what is the minimal interface a new method must implement, and where is the frozen-versus-trainable decision made.

Stage 4, LoRA end to end. Read the LoRA package, tuners/lora/config.py, then tuners/lora/layer.py for the linear forward and get_delta_weight, then tuners/lora/model.py for _create_new_module and merge_and_unload, then tuners/lora/bnb.py for the 4-bit variant. Questions, how does the type dispatch pick a layer class, and how does merging fold B A into W.

Stage 5, the other methods and the utils. Skim tuners/ia3/ to see a multiply instead of an add, then one soft-prompt method under tuners/prompt_tuning/ or tuners/prefix_tuning/ to see the prompt-encoder path. Finish with utils/save_and_load.py, which decides what a checkpoint contains, and utils/other.py, home of prepare_model_for_kbit_training and the ModulesToSaveWrapper that carries a fully-trained head.

Where not to start. The long tail of research methods, AdaLoRA, OFT and BOFT, VeRA, LoHa and LoKr, is best met after LoRA is solid, since each is a variation on the same injection mechanism. The integration code for DeepSpeed, FSDP, and the various quantization backends beyond bitsandbytes is important in production but a distraction on a first read. And the mixed-adapter and adapter-composition paths are elegant but presume you already have the single-adapter model in your head.

Part VII: Hands-on labs

Labs 1 through 4 run on a single modest GPU, and several will run on CPU for a tiny model if you are patient. Lab 5 wants a GPU with bitsandbytes.

Lab 1: see the savings. Concept, the parameter arithmetic of Part V.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("gpt2")
for r in (4, 8, 16, 32):
    m = get_peft_model(base, LoraConfig(r=r, target_modules=["c_attn"]))
    m.print_trainable_parameters()
    base = m.unload()          # detach the adapter, restore the plain base

Watch the trainable count grow linearly with r, and check it against 2 * r * (d_in + d_out) summed over the matched modules. Then widen target_modules and watch the count grow with the number of adapted layers, which is the formula made real.

Lab 2: prove the base is frozen. Concept, the additive adapter of Part IV.

import torch
model = get_peft_model(base, LoraConfig(r=8, target_modules=["c_attn"]))
x = tokenizer("the cat sat on the", return_tensors="pt")
with model.disable_adapter():
    a = model(**x).logits
b = model(**x).logits          # adapter active, but B is zero-initialized at step 0
print(torch.allclose(a, b))    # True before any training step

Because lora_B starts at zero, the freshly wrapped model is numerically identical to the base. Take one optimizer step on any batch, rerun, and watch the two diverge, which is the adapter coming to life.

Lab 3: train, save, measure the folder. Concept, saving only the delta from Part IV.

python train_lora.py           # a short LoRA fine-tune on any small dataset
ls -lh out/                    # adapter_model.safetensors is megabytes
du -sh out/                    # a few MB, versus GBs for a full base checkpoint

The point is visceral. The saved adapter is a rounding error next to a full checkpoint of the base. Load it back with PeftModel.from_pretrained(base, "out") and confirm the fine-tuned behavior returns.

Lab 4: merge and time it. Concept, merge_and_unload from Part IV.

import time
peft_model = PeftModel.from_pretrained(base, "out")
def bench(m):
    t = time.time()
    for _ in range(50): m(**x)
    return time.time() - t
print("with adapter :", bench(peft_model))
merged = peft_model.merge_and_unload()
print("merged       :", bench(merged))       # no extra matmul per layer

The merged model should match the base model's latency, while the unmerged one pays the small low-rank overhead per adapted layer. Confirm the two produce the same logits to floating-point tolerance, since merging is exact for a non-quantized base.

Lab 5: QLoRA on a real base. Concept, the 4-bit path from Part V.

from transformers import BitsAndBytesConfig
import torch
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B",
                                            quantization_config=bnb)
base = prepare_model_for_kbit_training(base)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32,
                                        target_modules="all-linear"))
model.print_trainable_parameters()

Watch the memory footprint of the loaded base drop to roughly a quarter of its fp16 size, and confirm the adapter still trains. Then try to merge_and_unload and read the warning about merging into a quantized base, which is the lossy-merge caveat made concrete.

Lab 6: swap two adapters. Concept, the multi-adapter contract of Part V.

model = get_peft_model(base, LoraConfig(r=8, target_modules=["c_attn"]),
                       adapter_name="taskA")
model.add_adapter("taskB", LoraConfig(r=8, target_modules=["c_attn"]))
for name in ("taskA", "taskB"):
    model.set_adapter(name)
    print(name, "active adapters:", model.active_adapters)

One frozen base, two adapters, routed by name, none of it duplicating the base weights. This is the shape of multi-tenant serving, one model in memory answering for many fine-tunes.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is PEFT, in one sentence?

A library that fine-tunes a frozen pretrained model by injecting small trainable adapters into named submodules, driven by a config object and one call to get_peft_model, so that training touches a fraction of a percent of the parameters and saves as megabytes.

2. What does get_peft_model actually do?

It reads the config, uses task_type to pick a PeftModel subclass and peft_type to pick a tuner, builds the tuner, and the tuner walks the base model's modules, matches them against target_modules, and replaces each match with a wrapper that keeps the original as a frozen base_layer and adds the adapter beside it.

3. How does a LoRA adapter change a linear layer's forward?

The frozen base still computes W x, and the adapter adds a low-rank term, B(A(dropout(x))) * (alpha/r), where A projects the input down to rank r and B projects back up to the output width. Since B starts at zero the freshly wrapped model equals the base until training moves the factors.

4. Derive the LoRA parameter count for one square projection.

For a d x d weight, LoRA trains A of shape (r, d) and B of shape (d, r), totaling 2 r d against d^2 for full fine-tuning, a ratio of 2r/d. At r = 8 and d = 4096 that is 1/256, about 0.39% per matrix, and it shrinks further as the model widens.

5. What is the difference between QLoRA and LoRA?

Only the base. QLoRA loads the frozen base as 4-bit NF4 weights through bitsandbytes, dequantizing on the fly in the matmul, while the LoRA factors stay in bf16. The tuner's type dispatch routes the 4-bit linears to a 4-bit LoRA layer, so it is the same method over a cheaper frozen base, which is what lets very large models fit on one GPU.

6. Why does merge_and_unload eliminate the inference overhead?

It computes the delta weight (B A) * (alpha/r), adds it to the base weight in place, and swaps the wrapper back for the updated base layer, leaving a plain model whose forward has no extra matmul. The correction was always a delta of the weight, so folding it in is exact for a full-precision base.

7. When is merging a bad idea?

When the base is quantized, since folding a bf16 delta into 4-bit weights requires a lossy requantize, and when you want to serve many adapters over one base, since merging bakes one adapter in and defeats the point of keeping them detachable. In both cases you keep the adapter separate at inference.

8. How do soft-prompt methods differ from LoRA in the code?

They do not inject anything into the base modules. Their configs are prompt-learning configs, and PeftModel.forward runs a prompt encoder to produce virtual token representations and prepends them to the sequence. Prompt tuning acts only at the input, P-tuning generates the tokens with a small encoder, and prefix tuning injects key and value prefixes into every layer's attention.

9. What is IA3 and how is it lighter than LoRA?

IA3 learns a single scaling vector per target module and multiplies the module's activations elementwise, rescaling attention keys and values and the feed-forward intermediate. A vector costs on the order of a dimension rather than 2 r d, so it trains even fewer parameters, and because a diagonal scale commutes into the weight it also merges cleanly.

10. What exactly ends up in a saved adapter folder?

The filtered adapter tensors, the LoRA factors and any modules_to_save such as a classifier head, written as adapter_model.safetensors, plus adapter_config.json recording the method, its hyperparameters, and the base model name. None of the base weights, which is why the folder is megabytes and a reload needs the base.

11. Why must a new classification head go in modules_to_save?

A fresh head is not matched by any adapter, so by default it would be frozen with the rest of the base and would neither train nor save. Listing it in modules_to_save wraps it so it is trained in full precision and its weights are written into the adapter checkpoint.

12. When would you not use PEFT at all?

When you are pretraining or heavily continued-pretraining a model rather than adapting a finished one, where a platform like torchtitan fits, or when the base is small, the task is far from pretraining, and you can afford a full fine-tune whose higher ceiling is worth the cost.

13. How can several fine-tunes share one model in memory?

Because adapters are additive and detachable, one frozen base can hold many adapters through add_adapter, routed by set_adapter, and serving stacks that support dynamic LoRA load and switch adapters per request over a single shared base, which is impossible with fully merged per-task checkpoints.

14. What does prepare_model_for_kbit_training do and why?

It casts sensitive layers like the norms and the language head back to fp32, enables gradient checkpointing, and makes the input embeddings produce gradients, all of which keep 4-bit training numerically stable and compatible with activation checkpointing before the LoRA adapter is attached.

Part IX: Design lessons

Represent the method as data, dispatch by type. The choice of LoRA, IA3, or a soft prompt is a config with a peft_type, and the machinery that edits the model is a table lookup. New methods slot in without touching the core. This is the strategy pattern with a registry, the same instinct as codecs keyed by format or serializers keyed by content type.

Wrap, do not rewrite. The base module is preserved as base_layer and the adapter sits beside it, so the original computation is always available, the adapter can be disabled or removed, and nothing in the base model had to change. Keeping the wrapped original reachable is what buys disable, unmerge, and clean unload, and it is why decorators that hide the thing they wrap age worse than ones that expose it.

Make the invariant hold at step zero. Initializing B to zero means the wrapped model starts exactly equal to the base, so fine-tuning is a smooth departure from a known good point rather than a jolt. Starting a modification as the identity is a recurring safety property, present in residual connections, feature-flag rollouts, and any migration that begins as a no-op.

Persist the delta, not the whole. A saved adapter is the change plus the identity of what it changed, not a copy of the base, which turns a per-task gigabyte checkpoint into a shared base and a folder of megabyte diffs. Storing diffs against a stable reference is the same economy as version control, container layers, and copy-on-write snapshots.

Give one seam many uses. The type dispatch that picks a layer class for an nn.Linear is the exact seam that makes QLoRA free, because a 4-bit linear is just another type to dispatch on. A single well-placed extension point that several features flow through beats a special case per feature, which is why the 4-bit path reads as a variant rather than a fork.

Part X: Memorization framework

The one-sentence summary. PEFT freezes the base, reads a config whose peft_type selects a tuner, walks the module tree to replace matched layers with wrappers that keep the original frozen and add a tiny trainable adapter, trains only that adapter, saves it as a megabyte diff, and optionally folds it back into the weights for zero-overhead inference.

LoraConfig -> get_peft_model -> PeftModel(task_type) + tuner(peft_type)
  -> BaseTuner.inject_adapter: match target_modules, swap nn.Linear -> lora.Linear
  -> forward: W x + (alpha/r) B A x     (B zero-init, base frozen)
  -> save_pretrained: adapter_config.json + adapter_model.safetensors
  -> merge_and_unload: W += (alpha/r) B A, return plain model

The chain mapped to source.

entry        src/peft/mapping_func.py  (get_peft_model)
model        src/peft/peft_model.py    (PeftModel, PeftModelForCausalLM)
config       src/peft/config.py, utils/peft_types.py (PeftType, TaskType)
tuner core   src/peft/tuners/tuners_utils.py (BaseTuner, BaseTunerLayer)
lora         src/peft/tuners/lora/{config,layer,model,bnb}.py
ia3 / prompt src/peft/tuners/{ia3,prompt_tuning,p_tuning,prefix_tuning}/
save/load    src/peft/utils/save_and_load.py, utils/other.py

Memorize these blocks.

• The wrap: a LoRA layer keeps the frozen linear as base_layer and adds lora_A (r, d_in) and lora_B (d_out, r), with B zero-initialized so the model starts equal to the base.
• The forward: W x + (alpha/r) B A x, two skinny matmuls, scaling alpha/r adds no parameters.
• The savings: full is d^2, LoRA is 2 r d, ratio 2r/d, about 0.39% at r=8 on d=4096, roughly 4M trainable for a 7B with q and v adapted.
• QLoRA: same method, base loaded as 4-bit NF4 with double quant, dequantized in the matmul, adapters in bf16, merging is lossy.
• The family: LoRA adds to weights, IA3 scales activations, soft-prompt methods prepend learned inputs and never touch the base, so only LoRA-style methods and IA3 merge.

Part XI: Papers and further reading

Every method in this walkthrough traces back to a specific paper, and each one rewards a direct read. Where this site covers the surrounding ideas in depth, the companion link points there.

1. Houlsby et al., Parameter-Efficient Transfer Learning for NLP, 2019. The bottleneck-adapter paper that opened the field PEFT packages, and the ancestor of the adapters library mentioned in Part III.
2. Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020. The evidence that task adaptation lives in a low-dimensional subspace, which is the hypothesis LoRA turns into a method.
3. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The low-rank update this chapter follows through the code. The applied generative AI class puts it to work, and the torchtune and TRL walkthroughs train the same adapters in other stacks.
4. Li and Liang, Prefix-Tuning, Optimizing Continuous Prompts for Generation, 2021. The per-layer key and value prefixes behind prefix tuning. The attention note derives the tensors it prepends to.
5. Lester et al., The Power of Scale for Parameter-Efficient Prompt Tuning, 2021. The input-layer soft prompts that become PEFT's prompt tuning, with the finding that they close the gap on full fine-tuning as models grow.
6. Liu et al., GPT Understands, Too, 2021. The P-tuning paper, which generates the virtual tokens through a small encoder to stabilize optimization.
7. Liu et al., Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, 2022. Introduces IA3, the learned activation scales of Part V, and makes the case for adapters over prompting in the few-shot regime.
8. Dettmers et al., QLoRA, Efficient Finetuning of Quantized LLMs, 2023. NF4, double quantization, and paged optimizers, the 4-bit frozen base this chapter's QLoRA path rests on. The Unsloth walkthrough covers a kernel-tuned version of the same recipe.
9. Zhang et al., AdaLoRA, Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, 2023. Lets the rank vary across weight matrices during training, the AdaLoRA tuner in the method list.
10. Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, 2023. The sqrt(r) scaling behind the use_rslora switch in Part V's derivation.
11. Liu et al., DoRA, Weight-Decomposed Low-Rank Adaptation, 2024. Splits the update into a magnitude and a direction, the use_dora switch in the config.

Part XII: Final takeaway

If the single-model pieces PEFT assumes are the gap, the base models it adapts come from Transformers and the attention math behind prefix tuning's key and value prefixes is derived in the FlashAttention chapter. The pretraining side of the story, training a base worth adapting, lives in nanoGPT and torchtitan, and the mesh-and-sharding vocabulary that scales any of this across GPUs is worked out in parallel computing. Then read one LoRA forward once more, and it will look like a frozen matmul with a small correction added, which is the entire idea.