Accelerate

Part I: The mental model

accelerate launch train.py     reads default_config.yaml, picks a launcher
      |
      v
torch.distributed.run          spawns num_processes ranks (one per GPU / CPU / TPU core)
      |                         sets RANK, LOCAL_RANK, WORLD_SIZE in the environment
      v
Accelerator()                  reads the env, builds the AcceleratorState singleton
      |
      v
prepare(model, opt, dl, sched)   adapts each object to the current backend
      |     model     -> .to(device), then DDP / FSDP / DeepSpeed wrap + autocast
      |     dataloader -> shard across ranks, or dispatch from rank 0
      |     optimizer  -> AcceleratedOptimizer (owns the GradScaler, the sync gate)
      |     scheduler  -> AcceleratedScheduler (steps once per real optimizer step)
      v
your unchanged loop            for batch in dl: out = model(batch); accelerator.backward(loss)
      |                         opt.step(); sched.step(); opt.zero_grad()
      v
collectives (NCCL / gloo / xla)  DDP all-reduce of grads, gather/reduce for metrics

The one-sentence identity. Accelerate is a device-and-backend abstraction that makes the distribution strategy a property of the launch environment rather than a property of your training code, so a single loop written for one device runs unchanged across all of them. The classic way to add multi-GPU training is invasive. You sprinkle .cuda() and .to(device) through the model and the batch, wrap the model in DistributedDataParallel, build a DistributedSampler, guard every print and save behind a rank check, and add a GradScaler for mixed precision. Each of those is a place the code now assumes a particular world. Accelerate removes the assumptions. You ask the Accelerator for the device, you let prepare() wrap the objects, and the same file runs whether the launcher spun up one process or a thousand.

The load-bearing idea sits inside prepare(). It is a polymorphic adapter. You hand it your model, your optimizer, your dataloader, and your learning-rate scheduler, and it returns objects of the same shape that behave correctly in whatever environment the launch configured. On one CPU it does almost nothing. On eight GPUs it moves the model to the local device and wraps it in DDP, replaces the dataloader with one that shards batches across ranks, and wraps the optimizer so it cooperates with mixed precision and gradient accumulation. On a DeepSpeed or FSDP config it swaps the wrapping strategy without touching the surface. Because the adaptation lives in the returned objects rather than in your loop, the loop reads like ordinary single-device PyTorch and the distribution is somebody else's problem, which is the entire point.

A useful contrast to hold from the start. Accelerate is deliberately not a trainer. It has no fit(), no callbacks that own your epoch, no config schema for your model. It gives you the primitives to keep writing the loop yourself. That is the opposite bet from a framework like PyTorch Lightning, and it is a different job from a pretraining platform like torchtitan, which owns the model definition and composes four parallelisms over DTensor. Everything in this chapter is described against Accelerate as it stands in mid 2026. The public API here has been stable for years, but the plugin surfaces for FSDP and DeepSpeed track fast-moving upstreams, so where a detail is likely to have shifted I say so and stay at the level of the concept.

Part II: Using it

Accelerate is a pure-Python library and installs from PyPI with no build step. It runs anywhere PyTorch runs, including macOS and plain CPU, which is part of why it is pleasant to learn on.

pip install accelerate
# optional extras that pull in the integrations you plan to use:
#   pip install "accelerate[deepspeed]"     for the DeepSpeed backend
#   pip install "accelerate[test]"          for the test suite

The first thing to run is not training, it is the config wizard. accelerate config asks a short series of questions about your machine, this compute environment, how many processes, whether you want mixed precision, whether to use DeepSpeed or FSDP, and writes the answers to a YAML file. The default location is ~/.cache/huggingface/accelerate/default_config.yaml, and accelerate launch reads it unless you pass another with --config_file.

accelerate config          # interactive: writes default_config.yaml
accelerate env             # prints the resolved environment and the config it found
accelerate test            # runs a built-in script that exercises the full setup path

accelerate test is the equivalent of a smoke test. It launches the configured world, runs gather and reduce operations across the processes, and reports whether the distributed setup actually works, which is a far better first failure than discovering a broken NCCL config three hours into a run. Now the code. A minimal training script that already runs on every backend looks like this, and the comments mark the only four kinds of change from a single-device loop.

from accelerate import Accelerator
import torch

accelerator = Accelerator()                      # 1. one object, reads the launch env

model = MyModel()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
train_dl = torch.utils.data.DataLoader(train_ds, batch_size=16, shuffle=True)
scheduler = torch.optim.lr_scheduler.LinearLR(optimizer)

# 2. prepare everything together, in one call, and reassign the results
model, optimizer, train_dl, scheduler = accelerator.prepare(
    model, optimizer, train_dl, scheduler
)

for epoch in range(num_epochs):
    for batch in train_dl:                       # already sharded, already on device
        optimizer.zero_grad()
        inputs, targets = batch                  # 3. no .to(device) needed
        outputs = model(inputs)
        loss = loss_fn(outputs, targets)
        accelerator.backward(loss)               # 4. not loss.backward()
        optimizer.step()
        scheduler.step()

accelerator.print("done")                        # prints on the main process only

Read the four changes as a set. First, construct one Accelerator. Second, pass model, optimizer, dataloader, and scheduler through prepare() and reassign the returned handles, because the returned objects are the wrapped ones and the originals are now stale. Third, stop moving tensors to a device by hand, since the prepared dataloader yields batches already on accelerator.device. Fourth, call accelerator.backward(loss) instead of loss.backward(), because backward is where the library applies the mixed-precision gradient scaler and the gradient accumulation scaling. Everything else, the epoch loop, the loss function, the optimizer step, is untouched.

To run it, do not call python train.py. Use the launcher, which reads the config and spawns the processes.

accelerate launch train.py                       # uses default_config.yaml
accelerate launch --multi_gpu --num_processes 4 train.py    # override on the CLI
accelerate launch --num_processes 1 --cpu train.py          # force single CPU

The same script also runs under plain python train.py, where it degrades gracefully to one process on whatever device is available, and it runs under torchrun if you prefer that launcher, because Accelerate reads the standard distributed environment variables either way. That property, one script that runs three ways, is what makes it comfortable to develop on a CPU and deploy on a cluster.

Now the mistakes beginners make. First and most common, forgetting to reassign the results of prepare(). If you write accelerator.prepare(model, optimizer, ...) and discard the return value, your loop keeps using the unwrapped model and the unsharded dataloader, and the run either crashes or silently trains on duplicated data across ranks. Second, calling loss.backward() out of habit. With fp16 that skips the gradient scaler and your loss goes to NaN, and with gradient accumulation it skips the loss scaling, both silent. Third, mishandling metrics. Every process computes its own loss and its own accuracy, and if you log the local number you are logging one rank's view, so you gather first.

# wrong: this logs rank 0's local loss, not the global loss
accelerator.print(loss.item())

# right: gather across processes, then reduce, and only the main process logs
all_losses = accelerator.gather(loss.detach())
if accelerator.is_main_process:
    log(all_losses.mean().item())

Fourth, batch-size confusion, which is the same trap as everywhere in distributed training. By default each process gets a full batch_size, so the effective global batch is batch_size times num_processes. If you want a fixed global batch split across processes instead, you set split_batches=True. Fifth, saving. Do not torch.save(model.state_dict()) a DDP- or FSDP-wrapped model, because the keys carry the wrapper prefix and, under FSDP, each rank only holds a shard. Unwrap first, and let the library gather.

accelerator.wait_for_everyone()                  # barrier so all ranks reach the save
unwrapped = accelerator.unwrap_model(model)      # strip DDP / FSDP wrapper
accelerator.save_model(unwrapped, "out/")        # gathers shards, writes on main only

Part III: When it is the right tool

Accelerate is the right tool when you want to keep writing your own training loop and still run it across whatever hardware you happen to have. That describes a lot of real work, a researcher iterating on a custom loss who moves from a laptop to a single A100 to a node of eight, a team that wants explicit control over the loop but does not want to hand-roll DDP and samplers, and anyone adding mixed precision or gradient accumulation to an existing script with the smallest possible diff. It is also the layer that Transformers' own Trainer sits on, so understanding Accelerate is understanding what the Trainer does underneath, and the two share the same launch and config machinery.

The honest cases for alternatives. If you want the framework to own the loop, with callbacks, logging, checkpointing, early stopping, and multi-backend support handled for you, PyTorch Lightning is the mature choice and you write a LightningModule rather than a loop. If your job is large-scale pretraining of transformer models and you want a reference platform that composes FSDP2, tensor, pipeline, and context parallelism, torchtitan is built for exactly that and owns the model definition to get it. If you need the deepest DeepSpeed features and want to configure ZeRO stages, offload, and custom optimizers directly, using DeepSpeed on its own gives you the whole surface, though Accelerate can also drive it. If you want a training-specific library with speedup methods baked in, MosaicML Composer took that path, and if you want cluster orchestration and data pipelines around training, Ray Train operates a level up. Accelerate deliberately does less than all of these. It is the thin waist, not the platform.

The scope-shaped warning. Accelerate handles distribution and precision, and it does not make your model architecture fast. It will not fuse your attention, it will not pick your parallelism degrees for a 70B model, and it will not rescue a loop that is bound by a slow Python dataloader. It sits above the collectives and below the model, and it is excellent in that band. The moment your problem is really about how the model itself is sharded across a large cluster, you are in torchtitan or Megatron territory, and the moment it is really about serving tokens fast you are in vLLM territory. Reaching for Accelerate to solve those is using a good tool on the wrong layer.

layer map (where Accelerate sits)

  your training loop            you write this, Accelerate keeps it yours
  ---------------------------
  Accelerate                    device placement, DDP/FSDP/DeepSpeed wrap,
                                data sharding, mixed precision, accumulation
  ---------------------------
  torch.distributed / NCCL      the collectives themselves
  ---------------------------
  CUDA / XLA / CPU              the hardware

Part IV: The full life of one call to prepare(), and one step

The specimen. A script launched with accelerate launch --multi_gpu --num_processes 4 --mixed_precision bf16 train.py on one node with four GPUs, using the minimal loop from Part II with Accelerator(gradient_accumulation_steps=2). We follow the program from the launch command through the setup and one full training step, and note where the path forks for other backends.

Stage 1: accelerate launch and the launcher choice

accelerate launch is a CLI entry point implemented in src/accelerate/commands/launch.py. It parses your flags, loads the config YAML (unless the flags fully specify the world), and then chooses a launcher based on the resolved distributed_type. For a multi-GPU run it uses PyTorch's elastic launch machinery, effectively the same thing torchrun does, spawning num_processes worker processes and setting RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, and MASTER_PORT in each. For a TPU run it routes through the XLA multiprocessing spawn. For CPU or a single process it just runs the script. From here every worker is running the same train.py, which is the SPMD model, one program on every rank, and the launcher's whole job was to set the environment so each process knows who it is.

Stage 2: Accelerator() and the state singleton

The first line that matters is Accelerator(), defined in the large src/accelerate/accelerator.py. Its constructor does not talk to any GPU yet. It builds an AcceleratorState, which is defined in src/accelerate/state.py and is a singleton in the Borg sense, all instances share one class-level dictionary of state. AcceleratorState is layered on top of a lighter PartialState, which reads the environment, decides the DistributedType (here MULTI_GPU), initializes the process group through torch.distributed if more than one process exists, and records num_processes, process_index, local_process_index, and the device for this rank. The split between PartialState and AcceleratorState is deliberate. The distributed facts of the world are separated from the training-specific configuration like mixed precision and the DeepSpeed plugin, so code that only needs to know the rank and device can ask PartialState without pulling in a trainer's worth of setup. Because the state is a singleton, constructing a second Accelerator anywhere in the program reuses the same world rather than reinitializing it.

Two more things are constructed here. A GradientState, also a singleton, which tracks whether the current step is a gradient sync step and whether the dataloader has reached its end. And, because we asked for bf16, the mixed-precision mode is recorded. Note that bf16 needs no gradient scaler, so no GradScaler is created. Had we asked for fp16, the constructor would build self.scaler = torch.cuda.amp.GradScaler(), and that object would thread through backward and the optimizer step later.

Stage 3: prepare() dispatches on type

accelerator.prepare(model, optimizer, train_dl, scheduler) is the heart of the library. It walks its arguments, checks the type of each, and routes it to the right internal method, then returns the wrapped objects in the same order. The dispatch is roughly this shape.

# conceptual sketch of prepare's dispatch, not the verbatim source
def prepare(self, *args):
    result = []
    for obj in args:
        if isinstance(obj, torch.nn.Module):
            result.append(self.prepare_model(obj))
        elif isinstance(obj, torch.optim.Optimizer):
            result.append(self.prepare_optimizer(obj))
        elif isinstance(obj, torch.utils.data.DataLoader):
            result.append(self.prepare_data_loader(obj))
        elif isinstance(obj, LRScheduler):
            result.append(self.prepare_scheduler(obj))
        else:
            result.append(obj)                   # passed through untouched
    return tuple(result)

The reason you pass everything in one call, rather than preparing the model here and the optimizer there, is that the wrappers need to know about each other. The optimizer wrapper needs the gradient scaler and the gradient state, the scheduler wrapper needs to know how many processes there are so it can step the right number of times, and under DeepSpeed the model and optimizer are initialized together into one engine. Preparing them as a set lets the library wire the references correctly.

Stage 4: prepare_model wraps for the backend

prepare_model is where the distribution strategy becomes concrete. In order, it moves the model to accelerator.device (unless device placement is disabled), then applies the backend wrapper. For our MULTI_GPU run that is torch.nn.parallel.DistributedDataParallel with device_ids=[local_process_index], so gradients all-reduce across the four ranks during backward. For an FSDP config it instead applies FullyShardedDataParallel from torch.distributed.fsdp, configured by the FullyShardedDataParallelPlugin, which shards parameters, gradients, and optimizer state across ranks. For DeepSpeed the model is not wrapped here at all, it is held for the joint deepspeed.initialize() call. Finally, if mixed precision is on, prepare_model patches the model's forward so it runs under torch.autocast and casts its outputs back to fp32, which is why you never write an autocast block in the loop yourself. The single method prepare_model is the seam where CPU, DDP, FSDP, and DeepSpeed diverge, and everything above it in your loop is written as if that seam did not exist.

Stage 5: prepare_data_loader shards the data

prepare_data_loader, in src/accelerate/data_loader.py, replaces your DataLoader with one of two Accelerate classes. The default for a map-style dataset is DataLoaderShard, which arranges for each of the four processes to see a disjoint quarter of every batch stream and moves the batches to the device as they come out. The alternative, DataLoaderDispatcher, iterates the dataset only on the main process and scatters slices to the others, which is the right choice for iterable datasets or when the data pipeline is expensive and you do not want four copies of it running. Two options shape the behavior. split_batches decides whether the batch_size you set is per-process or split across processes. even_batches pads the last, uneven batch so every process runs the same number of iterations, which matters because a process that finishes early would hang the others at the next collective. That padding is a small correctness debt that Stage 8 pays back.

Stage 6: prepare_optimizer and prepare_scheduler

The optimizer is wrapped in AcceleratedOptimizer, from src/accelerate/optimizer.py. This wrapper holds a reference to the gradient scaler and the gradient state, and it changes the behavior of step() in two ways that matter later. It only actually steps when the gradient state says this is a sync step, which is how gradient accumulation is enforced, and when a scaler is present it routes the step through scaler.step() and scaler.update() so that fp16 inf-and-NaN handling works. The scheduler is wrapped in AcceleratedScheduler, from src/accelerate/scheduler.py, which by default steps once per real optimizer step and accounts for the number of processes, so your learning-rate schedule advances at the rate you intended rather than once per rank.

Stage 7: the training step, forward and backward

Now the loop runs. A batch comes off the prepared dataloader already on the GPU. model(inputs) runs the DDP-wrapped forward under bf16 autocast. The loss is computed in fp32. Then accelerator.backward(loss) does three jobs that a bare loss.backward() would not. First, because we set gradient_accumulation_steps=2, it divides the loss by two, so that summing gradients over the two microbatches yields the mean, not the sum. Second, if a scaler existed it would scale the loss before backward, though for bf16 it does not. Third, for a DeepSpeed engine it would call the engine's own backward instead. Underneath, DDP's backward hooks fire, and on a sync step they all-reduce the gradients across the four ranks so every process ends with the same averaged gradient. On a non-sync accumulation step, the accumulate context manager would have entered no_sync to skip that all-reduce, which is the whole efficiency argument for accumulation, though the plain loop here relies on the optimizer's sync gate rather than the context manager.

Stage 8: step, and gathering metrics honestly

optimizer.step() on the AcceleratedOptimizer checks the gradient state. On the first of the two accumulation microbatches it is a no-op, gradients simply keep accumulating. On the second it performs the real step, applying the averaged gradients, and if fp16 were active the scaler would step and update here and skip the step entirely on an inf. Then scheduler.step() advances the learning rate once, and zero_grad() clears for the next group. When you want to log the loss or compute accuracy, you gather across processes, and here the padding from Stage 5 comes due. A plain gather() would include the duplicated samples that even_batches added, which would bias your metric. The library provides gather_for_metrics(), which gathers and then drops exactly those padding duplicates using the dataloader's recorded length. Using gather_for_metrics rather than raw gather is the difference between a correct evaluation number and one that is slightly wrong in a way you will never notice until it matters. That closes one step. Data sharded in, gradients all-reduced across the mesh, parameters updated once per accumulation group, and a metric gathered without double counting.

Part V: Internals deep dives

Deep dive: the Accelerator, and the state singletons

The Accelerator object is a facade. It holds almost no state of its own and instead delegates to the singletons, which is why you can create it freely and why libraries built on Accelerate can each construct their own without fighting over the world. The two singletons do different jobs. PartialState is the answer to who am I, it knows the distributed type, the device, the process count, and the rank, and it exposes conveniences like state.is_main_process, the state.main_process_first() context manager for doing work on rank zero before the others proceed, and state.split_between_processes() for splitting a list of inputs across ranks during distributed inference. AcceleratorState adds the training configuration on top, the mixed-precision mode and the FSDP or DeepSpeed plugin. GradientState is the third, and it is the quiet coordinator behind gradient accumulation, tracking sync_gradients and whether the dataloader has reached its end so the final short accumulation group still syncs.

The useful trap to internalize is that these are process-global. There is exactly one AcceleratorState per process, so a stray second Accelerator with conflicting arguments does not create a second world, it either reuses the first or raises. This is what lets the Trainer in Transformers and your own code coexist, and it is why the constructor is cheap. The state is built once and shared.

Deep dive: how prepare wraps DDP, FSDP, and DeepSpeed

All three backends enter through the same prepare() call and diverge inside prepare_model and the DeepSpeed path. The table names what each one shards and where its wrapping decision lives.

DDP is the simplest and the default for multi-GPU. Every rank holds a full copy of the model, and DDP's backward hooks all-reduce gradients so the replicas stay in step. You tune it with a DistributedDataParallelKwargs handler passed to the Accelerator, which is how you set options like find_unused_parameters without Accelerate needing a flag for every DDP knob.

FSDP shards the model itself, so each rank stores only a slice of the parameters, gradients, and optimizer state, and all-gathers a layer's full parameters just in time for its forward and backward. Accelerate drives this through the FullyShardedDataParallelPlugin in src/accelerate/utils/dataclasses.py, which carries the sharding strategy, the auto-wrap policy that decides which submodules become their own FSDP units, the mixed-precision policy, and CPU offloading. Recent Accelerate also supports the newer FSDP2 design that keeps parameters as per-parameter DTensors, selected through a version field on the plugin, and the exact plugin fields have moved as upstream FSDP has evolved, so treat the plugin as the stable seam and read its current fields rather than memorizing them. The important idea is that FSDP trades communication for memory, and Accelerate exposes that trade as configuration rather than as code you write.

DeepSpeed is the different one. Here Accelerate does not wrap the model in prepare_model, it calls deepspeed.initialize() with the model, the optimizer, and the scheduler together, producing a single engine that owns all three. That is why, under DeepSpeed, accelerator.backward(loss) routes to the engine's backward and the optimizer step routes to the engine's step, because DeepSpeed implements ZeRO partitioning, its own loss scaling, and its own optimizer internally. The DeepSpeedPlugin lets you set the ZeRO stage and offloading in Python, and it resolves the many auto values in a DeepSpeed JSON config against your batch size and precision so the two config worlds agree. The lesson of the three backends is that Accelerate does not reimplement any of them, it normalizes their surfaces so your loop calls one backward and one step regardless of which one is running underneath.

Deep dive: the dataloader, sharding versus dispatch

data_loader.py is the most underappreciated file in the repository, because getting data distribution right is subtle and it hides all of that. The two strategies answer the same question, how do four processes consume one dataset without overlap, in opposite ways.

DataLoaderShard (default, map-style datasets)
   dataset [0 1 2 3 4 5 6 7 ...]
   rank0 sees [0 4 8 ...]   rank1 [1 5 9 ...]   rank2 [2 6 ...]   rank3 [3 7 ...]
   each rank reads independently, in parallel, and moves its batch to its device

DataLoaderDispatcher (iterable datasets, or expensive pipelines)
   only rank0 iterates the dataset
   rank0 --scatter batch slices--> rank1, rank2, rank3
   one data pipeline, results distributed by collective

DataLoaderShard is the parallel-read strategy. Each process runs the pipeline over its own share of indices, chosen so the shares are disjoint and cover the dataset. It is efficient because the four processes read in parallel, and it is the default for map-style datasets where random access is cheap. DataLoaderDispatcher is the single-reader strategy. Process zero owns the one dataloader, and each iteration it scatters the per-rank slices to the others over a collective. This is the correct choice for a streaming or iterable dataset that cannot be indexed, and for a pipeline so expensive that four copies would be wasteful. The even_batches option, mentioned in Part IV, pads the last batch so all processes iterate the same number of times, and the padding is later removed by gather_for_metrics. split_batches controls whether your configured batch_size is the per-process size or the global size that gets divided. In newer Accelerate these options are grouped into a DataLoaderConfiguration dataclass passed as dataloader_config to the Accelerator, replacing the older direct keyword arguments, so old examples that pass split_batches=True straight to Accelerator still describe the right behavior even if the argument has moved.

Deep dive: gradient accumulation and the sync gate

Gradient accumulation lets you simulate a large batch on small memory by running several microbatches before each optimizer step. The naive version has two bugs, and Accelerate fixes both. The first bug is communication waste, because DDP would all-reduce gradients on every microbatch when only the last one needs the synchronized result. The second is scaling, because summed gradients over N microbatches are N times too large unless the loss is divided by N. The intended API is the accumulate context manager.

accelerator = Accelerator(gradient_accumulation_steps=4)
model, optimizer, dl = accelerator.prepare(model, optimizer, dl)

for batch in dl:
    with accelerator.accumulate(model):
        outputs = model(batch)
        loss = loss_fn(outputs, batch.labels)
        accelerator.backward(loss)               # divides loss by 4 automatically
        optimizer.step()                         # a real step only every 4th time
        optimizer.zero_grad()

The context manager consults GradientState to decide whether this microbatch is a sync step, meaning the last in a group of four or the end of the dataloader. On the three non-sync microbatches it enters model.no_sync(), so DDP skips the gradient all-reduce and the gradients just accumulate locally. On the sync microbatch it lets the all-reduce happen. Meanwhile accelerator.backward divides the loss by four so the accumulated gradient is a mean, and the AcceleratedOptimizer's step checks the same sync flag and only performs the real update on the fourth microbatch, treating the other three as no-ops. The elegance is that the loop is written as if every iteration steps, and the sync gate inside the wrapped optimizer and the context manager quietly turns three out of every four into accumulation. The subtle correctness win is the end-of-loader handling. If the dataset does not divide evenly into groups of four, the final short group still triggers a sync and a step, because GradientState knows the dataloader has ended, so you do not silently drop the last few batches of gradients.

Deep dive: mixed precision and the scaler

Mixed precision is set once, with Accelerator(mixed_precision="bf16") or on the launch command, and it changes three things without a line in your loop. First, prepare_model wraps the model's forward in torch.autocast so the matmuls run in the low-precision type while accumulations stay in fp32, and it casts outputs back to fp32 so your loss code sees normal numbers. Second, for fp16 specifically the Accelerator builds a GradScaler, because fp16 gradients underflow to zero without loss scaling, and accelerator.backward scales the loss up before backward while the AcceleratedOptimizer unscales and steps through the scaler, skipping the step when it sees an inf or NaN. Third, bf16 needs no scaler at all because its exponent range matches fp32, so the scaler is simply never created. There is also an fp8 path, driven by an FP8RecipeKwargs handler over backends like Transformer Engine, for hardware that supports it.

The one operation you must route through Accelerate is gradient clipping. If you clip the raw gradients while an fp16 scaler is active, you clip the scaled values and get the wrong norm, so you call accelerator.clip_grad_norm_(model.parameters(), max_norm), which unscales first and, under FSDP, reduces the norm across shards correctly. For manual regions where you want autocast around code that is not the model forward, there is with accelerator.autocast():. Everything else about mixed precision is invisible, which is the intended experience, you ask for a precision at construction and the numerics are handled.

Deep dive: big model inference, the other half of the repo

Accelerate has a second pillar that the training story rarely mentions but which powers a feature everyone has used, the device_map="auto" that lets Transformers load a model too large for one GPU. This lives in src/accelerate/big_modeling.py and src/accelerate/utils/modeling.py. The entry points are init_empty_weights(), a context manager that builds the model on the meta device so it has shapes but no allocated storage, infer_auto_device_map(), which decides which layers land on which GPU, on CPU, or on disk given the available memory, and load_checkpoint_and_dispatch(), which streams the checkpoint shard by shard onto those devices. The runtime trick is in src/accelerate/hooks.py, where forward pre-hooks move each submodule's inputs to wherever that submodule's weights live and post-hooks move outputs back, so a forward pass can flow through layers scattered across a GPU, system RAM, and an NVMe disk as if they were one device. The same meta-device-then-dispatch idea that lets torchtitan initialize a 405B model without it fitting on one host is what lets Accelerate run inference on a model that does not fit either, and both rest on PyTorch's meta device. It is worth knowing this exists, because a large fraction of Accelerate's real-world usage is this inference path rather than the training loop.

Deep dive: accelerate launch, config, and notebook_launcher

The launch machinery is a small, readable command layer under src/accelerate/commands/. config/ implements the wizard that writes the YAML, launch.py implements the launcher that reads it and spawns processes, and there are utility commands like env, test, and estimate-memory for sizing a model against your hardware. The config YAML is the single source of truth for a machine's setup, and its core fields are worth recognizing.

# a typical multi-GPU default_config.yaml
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU        # or NO, FSDP, DEEPSPEED, XLA
mixed_precision: bf16              # no, fp16, bf16, fp8
num_processes: 4                   # total processes across all machines
num_machines: 1
machine_rank: 0
main_process_ip: null              # set for multi-node
main_process_port: null
gpu_ids: all

For multi-node training you run accelerate launch on each machine with the same config but a different machine_rank, and the launcher wires the ranks into one world through main_process_ip and main_process_port. Any field can be overridden on the command line, so accelerate launch --num_processes 8 --mixed_precision fp16 train.py beats the file without editing it, which is convenient in job scripts.

The notebook_launcher, in src/accelerate/launchers.py, solves a different launch problem. In a Jupyter or Colab notebook there is no shell to run accelerate launch, so you wrap your whole training function and hand it to the launcher, which spawns the processes for you.

from accelerate import notebook_launcher

def training_function(args):
    accelerator = Accelerator()
    # build model, data, optimizer, prepare, and run the loop entirely in here
    ...

notebook_launcher(training_function, args=(config,), num_processes=2)

Under the hood it spawns processes with PyTorch's multiprocessing start in fork mode for multi-GPU, and routes through the XLA spawn on a TPU, which is how a single Colab TPU notebook trains across eight cores. There is one rule that trips up everyone the first time. You must not have initialized CUDA in the notebook before calling notebook_launcher, which means no tensor may have touched a GPU in the parent process, which is why the entire model, data, and loop must be built inside the training function rather than in notebook cells above it. Fork a process that already holds a CUDA context and the children inherit a broken one, and the run fails in a confusing way. Build everything inside the function and it works.

Part VI: Reading the repository

The repository is compact for what it does, and it rewards reading in a particular order that follows the data rather than the file listing. All paths are under src/accelerate/ and reflect the layout in mid 2026, and a couple of the utility files are large, so read for the shape rather than every line.

Stage 0, orientation. Read the top-level README.md and the two-file example it shows, then open examples/ in the repo and read nlp_example.py or complete_nlp_example.py. These are the canonical minimal loops and they anchor everything else. Question to hold, which four lines change between a single-device loop and an Accelerate loop, and why is prepare called on all the objects at once.

Stage 1, the facade. Read accelerator.py with three methods as the destination, prepare, backward, and prepare_model. Do not try to read the whole file, it is long because it absorbs every backend's quirks. Follow just those three. Questions, how does prepare decide what each argument is, what are the three jobs backward does beyond calling loss.backward, and where in prepare_model does the DDP versus FSDP versus DeepSpeed decision happen.

Stage 2, the state. Read state.py and meet PartialState, AcceleratorState, and GradientState. Questions, why is the state a shared singleton, what does PartialState hold that AcceleratorState does not care about, and what two flags does GradientState track that make gradient accumulation correct.

Stage 3, the wrapped objects. Read optimizer.py and scheduler.py, which are short, then data_loader.py, which is not. Questions, when does AcceleratedOptimizer.step actually step, why does the scheduler need to know the process count, and what is the difference between DataLoaderShard and DataLoaderDispatcher in one sentence each.

Stage 4, the plugins. Read the plugin dataclasses in utils/dataclasses.py, at least FullyShardedDataParallelPlugin, DeepSpeedPlugin, and the kwargs handlers like DistributedDataParallelKwargs and GradScalerKwargs. Then skim utils/operations.py for gather, reduce, pad_across_processes, and send_to_device, which are the collective helpers the rest of the library calls. Questions, how does a plugin field become a wrapping decision in prepare_model, and where does the padding that gather_for_metrics removes get added.

Stage 5, launch and big models. Read commands/launch.py for how a config becomes spawned processes, launchers.py for notebook_launcher, and then, if you use it, the big-model path across big_modeling.py, utils/modeling.py, and hooks.py. Questions, how does the launcher choose between the elastic launch and the XLA spawn, and how does a forward pass flow through layers that live on different devices.

Where not to start. The test_utils/ tree and the enormous matrix of hardware-specific branches, XPU, MLU, NPU, MPS, are important for the library's portability but are noise on a first read, and the Megatron-LM integration is a niche path you should meet only after DDP, FSDP, and DeepSpeed are solid.

Part VII: Hands-on labs

Lab 1 needs nothing but a CPU. Labs 2 through 5 want two or more GPUs, or a single GPU to at least see the setup path. Log formats vary across versions.

Lab 1: the same script three ways. Concept: the environment carries the distribution, not the code.

# write the minimal loop from Part II as train.py, then run it three ways:
python train.py                                   # 1 process, whatever device exists
accelerate launch --cpu --num_processes 1 train.py
accelerate launch --num_processes 2 train.py      # needs 2 devices, or use --cpu on a big machine

Observe that the script is byte-for-byte identical across the three runs and only the launch command changes. Add accelerator.print(f"rank {accelerator.process_index} of {accelerator.num_processes} on {accelerator.device}") at the top of the loop and watch the numbers change with the launcher.

Lab 2: build a config and inspect it. Concept: the launch config as the single source of truth.

accelerate config              # answer the questions for your machine
accelerate env                 # see the resolved config and environment
accelerate test                # run the built-in setup smoke test
cat ~/.cache/huggingface/accelerate/default_config.yaml

Read the YAML field by field against the deep dive in Part V. Then rerun accelerate launch with --mixed_precision bf16 overriding the file and confirm from your logs that the override won.

Lab 3: gather, and gather_for_metrics. Concept: honest metrics under even_batches padding.

# inside a prepared eval loop over a dataset whose length is not divisible by world size
preds = model(batch).argmax(dim=-1)
# wrong for a final metric, includes padding duplicates:
all_preds_raw = accelerator.gather(preds)
# right, drops the padding the dataloader added:
all_preds = accelerator.gather_for_metrics(preds)
if accelerator.is_main_process:
    accelerator.print(len(all_preds_raw), len(all_preds))   # raw is larger

Run it on two processes with a dataset length that is odd relative to the world size and watch the two lengths differ by exactly the padding count. That gap is the double counting gather_for_metrics removes.

Lab 4: gradient accumulation, checked against a big batch. Concept: N microbatches equal one large batch.

acc = Accelerator(gradient_accumulation_steps=4)
# ... prepare, then:
for batch in dl:
    with acc.accumulate(model):
        loss = loss_fn(model(batch), batch.labels)
        acc.backward(loss)
        opt.step(); sched.step(); opt.zero_grad()

Train once with batch size 8 and accumulation 4, then once with batch size 32 and accumulation 1, with the same seed and learning rate. The loss curves should track closely, which demonstrates that accumulation is simulating the larger batch. Print acc.sync_gradients inside the context and watch it be true only every fourth step.

Lab 5: switch backends without touching the loop. Concept: prepare is the seam.

accelerate launch --multi_gpu train.py                       # DDP
# reconfigure with `accelerate config` choosing FSDP, or:
accelerate launch --use_fsdp --num_processes 2 train.py      # FSDP, same train.py
# with the deepspeed extra installed and a config:
accelerate launch --use_deepspeed --num_processes 2 train.py # DeepSpeed, same train.py

Confirm that train.py is unchanged across all three and that per-GPU peak memory drops as you move from DDP to FSDP or DeepSpeed ZeRO, because the model state is now sharded. This is Part IV Stage 4 made visible, the same loop, a different wrapper chosen at launch.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is Accelerate, in one sentence?

A thin PyTorch layer that makes the distribution strategy a property of the launch environment rather than the training code, so one loop runs unchanged on CPU, one GPU, many GPUs, or TPU by wrapping the model, optimizer, dataloader, and scheduler through prepare().

2. What are the four kinds of change to a single-device loop?

Construct one Accelerator, pass model, optimizer, dataloader, and scheduler through prepare() and reassign the results, drop the manual .to(device) calls because the prepared dataloader places batches for you, and replace loss.backward() with accelerator.backward(loss).

3. Why must you reassign the output of prepare()?

prepare() returns wrapped objects, the DDP or FSDP model, the sharded dataloader, the accelerated optimizer, and the originals are now stale. If you ignore the return value your loop uses the unwrapped model and an unsharded dataloader, so ranks train on duplicated data or the run crashes.

4. Why is everything prepared in one call rather than separately?

Because the wrappers reference each other. The optimizer wrapper needs the gradient scaler and gradient state, the scheduler wrapper needs the process count, and under DeepSpeed the model, optimizer, and scheduler are initialized together into one engine. Preparing them as a set lets the references be wired correctly.

5. Where do DDP, FSDP, and DeepSpeed diverge?

Inside prepare_model and the DeepSpeed init path. DDP wraps the model in DistributedDataParallel and replicates it, FSDP wraps it in FullyShardedDataParallel and shards parameters, gradients, and optimizer state, and DeepSpeed is initialized as an engine that owns all three and implements ZeRO. Above that seam your loop is identical.

6. What three jobs does accelerator.backward do?

It divides the loss by the gradient accumulation steps so accumulated gradients are a mean, it scales the loss through the fp16 GradScaler when one is active, and under DeepSpeed it routes to the engine's own backward. For bf16 with no accumulation it is close to a plain backward.

7. How does gradient accumulation avoid wasted communication?

The accumulate context manager reads GradientState, and on non-sync microbatches it enters model.no_sync() so DDP skips the gradient all-reduce, letting gradients accumulate locally. The all-reduce and the real optimizer step happen only on the last microbatch of each group or at the end of the dataloader.

8. What is the difference between gather and gather_for_metrics?

gather concatenates a tensor across all processes. gather_for_metrics does the same and then drops the duplicate samples that even_batches added to make every process run the same number of iterations, so a metric computed over the result is not biased by double counting.

9. What does mixed_precision change without a line in the loop?

prepare_model runs the forward under autocast and casts outputs back to fp32, for fp16 the Accelerator builds a GradScaler that backward and the accelerated optimizer use for loss scaling and inf handling, and for bf16 no scaler is created because its range matches fp32.

10. Why clip gradients through accelerator.clip_grad_norm_?

Because with an fp16 scaler active the raw gradients are scaled, so clipping them directly uses the wrong norm. The Accelerate method unscales first and, under FSDP, reduces the norm across shards, giving the correct clip.

11. Why is the state a shared singleton?

So that constructing an Accelerator is cheap and idempotent, and so that your code and a library like the Transformers Trainer can each create one without initializing the distributed world twice or disagreeing about the device and rank.

12. What is the one rule for notebook_launcher?

CUDA must not be initialized in the parent process before the call, so no tensor may have touched a GPU in a notebook cell above it. Build the model, data, optimizer, and loop entirely inside the training function you hand to the launcher, otherwise the forked children inherit a broken CUDA context.

13. When would you pick Lightning or torchtitan over Accelerate?

Lightning when you want the framework to own the loop with callbacks, logging, and checkpointing built in. torchtitan when you are pretraining large transformers and want a platform that composes FSDP2, tensor, pipeline, and context parallelism over DTensor. Accelerate when you want to keep your own loop and only abstract the device and backend.

14. Accelerate loads a model too large for one GPU with device_map=auto. How?

Through the big-model path. The model is built on the meta device with init_empty_weights, infer_auto_device_map assigns layers to GPUs, CPU, or disk by available memory, load_checkpoint_and_dispatch streams the weights onto them, and hooks move each layer's inputs and outputs to wherever its weights live so one forward flows across all of them.

Part IX: Design lessons

Abstract the environment, not the loop. The single best decision in Accelerate is refusing to own your training loop. It abstracts the thing that genuinely varies across machines, the device and the distribution backend, and leaves the thing you actually want to control, the loop, in your hands. Wherever a library is tempted to take over your control flow to hide a concern, ask whether it can hide the concern behind an object instead, the way prepare() does.

Make one function the seam. Every backend difference is funneled through prepare and prepare_model, so there is exactly one place where CPU, DDP, FSDP, and DeepSpeed diverge and the entire rest of the codebase is backend-agnostic. Centralizing the variation in one audited seam is the same instinct as a single factory or a single dependency-injection root, and it is why adding a new hardware backend does not ripple through the loop.

Put shared truth in a singleton, deliberately. The world has one rank, one device, one process count, and Accelerate models that as a shared state rather than passing it through every constructor. A singleton is usually a smell, but for genuinely process-global facts it is the honest representation, and separating PartialState from AcceleratorState keeps the who-am-I facts apart from the training configuration.

Hide correctness debts and pay them automatically. even_batches introduces duplicated samples to keep the collectives from deadlocking, which is a correctness debt, and gather_for_metrics pays it back by dropping exactly those duplicates. Good abstractions do not just hide a mechanism, they remember the debt the mechanism created and settle it so the user never sees a wrong number.

Configuration belongs to the machine, code belongs to the author. The distribution strategy lives in a YAML tied to the machine, and the training logic lives in a script tied to the problem, so the same script runs on a laptop and a cluster with a different config. Keeping environment configuration out of source code is the same separation that twelve-factor apps make for deployment, and it is why one file can serve every backend.

Reuse the platform, do not reimplement it. Accelerate wraps DDP, FSDP, and DeepSpeed rather than writing its own sharding, and it leans on PyTorch's meta device for big models rather than inventing a lazy loader. It adds a normalizing surface over powerful lower layers, which is a smaller and more durable contribution than a competing implementation would have been.

Part X: Memorization framework

The one-sentence summary. Accelerate reads the launch environment into a shared state singleton, adapts your model, optimizer, dataloader, and scheduler to the current backend through one polymorphic prepare(), and lets an ordinary training loop run unchanged on CPU, one GPU, many GPUs, or TPU, with device placement, data sharding, gradient accumulation, and mixed precision handled inside the wrapped objects.

accelerate launch -> torch.distributed.run -> num_processes ranks (env set)
  -> Accelerator() -> PartialState / AcceleratorState / GradientState (singletons)
  -> prepare(model, opt, dl, sched)
       model -> device + DDP / FSDP / DeepSpeed + autocast   (prepare_model)
       dl    -> DataLoaderShard or DataLoaderDispatcher       (data_loader.py)
       opt   -> AcceleratedOptimizer (scaler + sync gate)     (optimizer.py)
       sched -> AcceleratedScheduler                          (scheduler.py)
  -> loop: model(x) -> accelerator.backward(loss) -> opt.step() -> sched.step()
  -> gather_for_metrics, save_model(unwrap_model(model))

The chain mapped to source, under src/accelerate/:

launch            commands/launch.py, commands/config/
facade            accelerator.py (prepare, backward, prepare_model)
state             state.py (PartialState, AcceleratorState, GradientState)
wrapped objects   optimizer.py, scheduler.py, data_loader.py
plugins + ops     utils/dataclasses.py, utils/operations.py
notebook launch   launchers.py (notebook_launcher)
big models        big_modeling.py, utils/modeling.py, hooks.py

Memorize these blocks:

• Four changes: one Accelerator, prepare() everything and reassign, drop .to(device), use accelerator.backward(loss).
• prepare dispatch: model to prepare_model, optimizer to AcceleratedOptimizer, dataloader to a shard or dispatch loader, scheduler to AcceleratedScheduler, anything else passed through.
• Backends: DDP replicates, FSDP shards params, grads, and optimizer state, DeepSpeed is a ZeRO engine that owns all three. All chosen at launch, none in the loop.
• Accumulation: accumulate skips the DDP all-reduce on non-sync steps via no_sync, backward divides the loss by the steps, the wrapped optimizer steps only on the sync step.
• Mixed precision: autocast on the forward, a GradScaler for fp16 only, clip through accelerator.clip_grad_norm_, gather metrics with gather_for_metrics.
• notebook_launcher rule: no CUDA before the call, build everything inside the training function.

Part XI: Papers and further reading

Accelerate is a normalizing layer over systems that each have a paper behind them, and reading those papers is how the wrappers stop looking like magic. Where this site develops the same idea in depth, the companion link points there.

1. Li et al., PyTorch Distributed, Experiences on Accelerating Data Parallel Training, VLDB 2020. The DDP design that prepare_model applies for a plain multi-GPU run, including the bucketed all-reduce in backward and the no_sync path that gradient accumulation leans on.
2. Zhao et al., PyTorch FSDP, Experiences on Scaling Fully Sharded Data Parallel, VLDB 2023. The sharded backend behind the FullyShardedDataParallelPlugin, and it explains the communication-for-memory trade that Part V describes. The parallelism vocabulary is developed in the parallel computing class on this site.
3. Rajbhandari et al., ZeRO, Memory Optimizations Toward Training Trillion Parameter Models, 2019. The partitioning arithmetic behind the DeepSpeed engine's stages, derived in the DeepSpeed walkthrough on this site.
4. Ren et al., ZeRO-Offload, Democratizing Billion-Scale Model Training, USENIX ATC 2021. The CPU offload strategy that surfaces as offload fields on both the FSDP and DeepSpeed plugins.
5. Shoeybi et al., Megatron-LM, Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. The tensor-parallel world that Accelerate's niche Megatron integration connects to, covered in the Megatron-LM walkthrough.
6. Micikevicius et al., Mixed Precision Training, ICLR 2018. The fp32 master weights and loss scaling that the GradScaler implements, worked through in the mixed precision note on this site.
7. Micikevicius et al., FP8 Formats for Deep Learning, 2022. The E4M3 and E5M2 encodings behind the fp8 path that FP8RecipeKwargs drives through backends like Transformer Engine.
8. Goyal et al., Accurate, Large Minibatch SGD, Training ImageNet in 1 Hour, 2017. The linear scaling rule and warmup for large effective batches, which is the arithmetic to remember when num_processes and accumulation steps multiply your batch size.
9. Paszke et al., PyTorch, An Imperative Style, High-Performance Deep Learning Library, NeurIPS 2019. The library whose surface Accelerate adapts rather than replaces, covered in the PyTorch walkthrough.

Part XII: Final takeaway

If the single-device PyTorch that Accelerate assumes is the gap, the ML implementations section builds those loops from scratch, and the parallelism vocabulary that FSDP and DeepSpeed use is developed in the parallel computing class notes and shown composed at scale in the torchtitan chapter. The natural next step up is Transformers, whose Trainer is a full training framework built on exactly the Accelerate machinery in this chapter. Read accelerator.py once more after all of that, follow just prepare, backward, and prepare_model, and the library will read like a careful set of adapters over PyTorch, which is the whole point.