ResNet

What it is and when you reach for it

A residual network is a deep network assembled from blocks of the form y = F(x) + x: a couple of convolutions (or any small subnetwork) F, plus a shortcut that carries the block's input straight to its output, where the two are added elementwise. The problem it solves is the degradation problem, the empirical fact that plain very deep networks were harder to optimize than shallower ones, and the solution generalizes far past vision: whenever you want to stack many transformations and still be able to train the stack, you make each transformation a residual correction around an identity path. Within its original habitat, ResNet (He, Zhang, Ren, and Sun, 2015) sits as the successor to VGG-style plain stacks and the predecessor of everything since: DenseNet reuses the idea with concatenation, EfficientNet and ConvNeXt keep the residual skeleton with better blocks, and the transformer keeps the skeleton with attention and MLPs as the F. Reach for a ResNet directly when you need a strong, fast, thoroughly understood convolutional backbone; reach for the residual idea any time a deep stack refuses to train.

The math

The degradation problem

The motivating experiment in the original paper is worth stating precisely because it is so often misremembered as overfitting. On CIFAR-10, a plain 56-layer convolutional stack had higher training error than a plain 20-layer one, and the same pattern held on ImageNet. A deeper model's hypothesis space strictly contains the shallower model's: the 56-layer network could in principle imitate the 20-layer network exactly by making 36 of its layers compute the identity. It did not, because a stack of nonlinear layers has no easy way to represent the identity. Every layer is conv, batch norm, ReLU; pushing that composition to behave as f(x) = x requires the weights to land in a thin, precise region, and gradient descent from random initialization does not find it. Depth was adding layers the optimizer could not afford to make harmless.

Learn the residual, not the mapping

Reframe what a block must learn. If the desired mapping for a block is H(x), let the block's layers compute F(x) = H(x) − x and wire the output as:

y = F(x) + x

The representable functions are unchanged; only the parameterization moved. But the parameterization is everything here: if the best thing a block can do is nothing, it now needs F ≈ 0, and driving weights toward zero is the single easiest thing gradient descent does, helped further along by weight decay. The identity stopped being a needle in weight space and became the default. Layers now learn perturbations around "pass it through", and a network of 152 layers trains because most layers only need to nudge.

Why gradients flow through the skip

The optimization story is visible directly in the backward pass. Differentiate y = x + F(x):

∂y/∂x = I + ∂F/∂x

∂L/∂x = ∂L/∂y · (I + ∂F/∂x)  =  ∂L/∂y  +  ∂L/∂y · ∂F/∂x

The derivative of the identity is the identity, so the incoming gradient ∂L/∂y arrives at the block input intact, plus a correction through the layers. Compose blocks and the effect telescopes. In the fully identity-path form (the pre-activation variant below), x_l+1 = x_l + F(x_l) unrolls to:

x_L = x_l + Σ_(i=l..L-1) F(x_i)

∂L/∂x_l = ∂L/∂x_L · ( I + ∂/∂x_l Σ F(x_i) )

Any deep layer's gradient reaches any shallow layer through a term that is a pure sum, never a pure product. Contrast the plain stack, where ∂x_L/∂x_l is a product of L − l layer Jacobians. A small worked number shows the gap: suppose each layer's Jacobian scales gradient norm by a well-behaved 0.9. Through 50 plain layers the shallow layers receive at most 0.9⁵⁰ ≈ 0.005 of the signal, a two-hundredfold attenuation from depth alone; through 50 residual blocks the identity term delivers a factor of 1 regardless, and the products only affect the corrections riding on top. This is the same additive-highway argument as the LSTM's cell state, played out in depth instead of time; the two designs are the same idea discovered in two different dimensions. The skip is not the whole story, and the traps section returns to what else it changes, but it is the reason a 1000-layer network is trainable at all.

The original block, batch norm, and the pre-activation variant

The original block orders operations as conv, BN, ReLU, conv, BN, then adds the shortcut, then applies a final ReLU:

original (v1):        x ─ conv ─ BN ─ ReLU ─ conv ─ BN ─(+)─ ReLU ─ y
                      └──────────────── identity ─────────┘

pre-activation (v2):  x ─ BN ─ ReLU ─ conv ─ BN ─ ReLU ─ conv ─(+)─ y
                      └──────────────── identity ──────────────┘

Batch norm and the residual connection are partners, not rivals. BN keeps every block's pre-activations in a consistent range no matter the depth, which keeps the F branch's contribution well scaled against the identity branch; residual learning keeps the optimization landscape navigable. The original ResNet needed both, and ablating either one breaks ImageNet-scale training. But the v1 ordering has a wrinkle: the ReLU after the addition sits on the identity path itself, so the "identity" shortcut is not quite an identity across blocks, and the telescoped sum above holds only approximately. The pre-activation variant (He et al., 2016, "Identity Mappings in Deep Residual Networks") moves BN and ReLU before the convolutions, leaving the skip path completely clean from the first block to the last. For networks of ordinary depth the two perform similarly; past several hundred layers the clean path wins clearly, and a 1001-layer pre-activation ResNet trains where the v1 form degrades. Modern transformer blocks made the same migration for the same reason: pre-norm (normalize inside the branch, keep the residual stream untouched) is the pre-activation ResNet's argument, re-adopted.

The downsampling shortcut

One honest wrinkle remains: a network must reduce spatial resolution and grow channels as it deepens, and at those boundary blocks the input and output shapes differ, so elementwise addition is undefined. The paper's option B, used by every standard implementation, projects the shortcut with a 1×1 convolution at stride 2 followed by BN. It is the one place the shortcut carries weights, it exists purely to reconcile shapes, and forgetting its stride or its BN is the classic way hand-rolled ResNets silently underperform. Both implementations below flag it in comments.

Implementation, twice

ResNet-18 is the smallest of the standard family: a 7×7 stem, four stages of two BasicBlocks each with widths 64, 128, 256, 512, global average pooling, and a linear classifier. The PyTorch version deliberately mirrors torchvision's module names (conv1, bn1, layer1 through layer4, fc, and downsample inside blocks), which is what makes the verification at the end a one-liner: torchvision's pretrained state dict loads into it directly. The Flax version expresses the same architecture; note how batch norm's train/eval distinction, implicit in PyTorch's model.train(), becomes an explicit train argument and a mutable batch_stats collection in Flax.

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    """conv-BN-ReLU-conv-BN, plus identity; ReLU after the add."""

    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_ch)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        # Downsampling shortcut ("option B"): when the block halves
        # the spatial size or changes width, the raw input cannot be
        # added elementwise, so project it with a strided 1x1 conv
        # plus BN. Everywhere else the shortcut is a pure identity.
        # Classic silent bug: forgetting the stride here, which makes
        # shapes clash, or forgetting the BN, which unbalances the
        # two branches' scales.
        self.downsample = None
        if stride != 1 or in_ch != out_ch:
            self.downsample = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),
                nn.BatchNorm2d(out_ch),
            )

    def forward(self, x):
        identity = x if self.downsample is None else self.downsample(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + identity)   # add BEFORE the last ReLU

class ResNet18(nn.Module):
    """Module names mirror torchvision so its pretrained
    state_dict loads directly (see the verification section)."""

    def __init__(self, num_classes=1000):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 64, 7, 2, 3, bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(3, 2, 1)
        self.layer1 = self._stage(64, 64, stride=1)
        self.layer2 = self._stage(64, 128, stride=2)
        self.layer3 = self._stage(128, 256, stride=2)
        self.layer4 = self._stage(256, 512, stride=2)
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(512, num_classes)

    @staticmethod
    def _stage(in_ch, out_ch, stride):
        # First block of a stage downsamples; the second never does.
        return nn.Sequential(BasicBlock(in_ch, out_ch, stride),
                             BasicBlock(out_ch, out_ch, 1))

    def forward(self, x):                  # x: (B, 3, 224, 224)
        x = self.maxpool(self.relu(self.bn1(self.conv1(x))))
        x = self.layer4(self.layer3(self.layer2(self.layer1(x))))
        x = self.avgpool(x).flatten(1)     # (B, 512)
        return self.fc(x)                  # (B, num_classes)

import jax
import jax.numpy as jnp
import flax.linen as nn

class BasicBlock(nn.Module):
    """conv-BN-ReLU-conv-BN, plus identity; ReLU after the add.
    Flax convs are NHWC: x is (B, H, W, C)."""
    features: int
    stride: int = 1

    @nn.compact
    def __call__(self, x, train: bool):
        norm = lambda: nn.BatchNorm(use_running_average=not train)
        y = nn.Conv(self.features, (3, 3), strides=self.stride,
                    padding=1, use_bias=False)(x)
        y = nn.relu(norm()(y))
        y = nn.Conv(self.features, (3, 3), padding=1,
                    use_bias=False)(y)
        y = norm()(y)
        # Downsampling shortcut ("option B"): same subtlety as in
        # PyTorch. Shape change => project the identity with a
        # strided 1x1 conv + BN; otherwise leave it untouched.
        identity = x
        if self.stride != 1 or x.shape[-1] != self.features:
            identity = nn.Conv(self.features, (1, 1),
                               strides=self.stride,
                               use_bias=False)(x)
            identity = norm()(identity)
        return nn.relu(y + identity)

class ResNet18(nn.Module):
    num_classes: int = 1000

    @nn.compact
    def __call__(self, x, train: bool = False):
        x = nn.Conv(64, (7, 7), strides=2, padding=3,
                    use_bias=False)(x)
        x = nn.relu(nn.BatchNorm(use_running_average=not train)(x))
        x = nn.max_pool(x, (3, 3), strides=(2, 2),
                        padding=((1, 1), (1, 1)))
        for features, stride in ((64, 1), (128, 2),
                                 (256, 2), (512, 2)):
            x = BasicBlock(features, stride)(x, train)
            x = BasicBlock(features, 1)(x, train)
        x = jnp.mean(x, axis=(1, 2))        # global average pool
        return nn.Dense(self.num_classes)(x)

model = ResNet18()
# BatchNorm keeps running statistics in a separate mutable
# collection; init returns both params and batch_stats.
variables = model.init(jax.random.PRNGKey(0),
                       jnp.zeros((1, 224, 224, 3)), train=False)

Using it on a real shape of problem

A realistic first run is CIFAR-10 shaped: batches of (128, 3, 32, 32), ten classes. (For 32×32 inputs, practitioners usually swap the 7×7 stride-2 stem for a 3×3 stride-1 conv and drop the max pool, since the standard stem would crush a CIFAR image to 8×8 before the stages start; the code below keeps the ImageNet stem for weight compatibility and simply accepts the small handicap.) The training step is ordinary supervised classification; the point of running it is to watch residual behavior, so the experiment worth doing is training the model twice, once as written and once with the shortcuts deleted, and watching the plain version fall behind at equal parameter count.

import torch
import torch.nn.functional as F

model = ResNet18(num_classes=10).cuda()
opt = torch.optim.SGD(model.parameters(), lr=0.1,
                      momentum=0.9, weight_decay=5e-4)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=200)

def train_epoch(loader):
    model.train()                      # BN uses batch statistics
    for images, labels in loader:      # (128, 3, 32, 32), (128,)
        images, labels = images.cuda(), labels.cuda()
        loss = F.cross_entropy(model(images), labels)
        opt.zero_grad(set_to_none=True)
        loss.backward()
        opt.step()
    sched.step()

@torch.no_grad()
def evaluate(loader):
    model.eval()                       # BN uses running statistics
    hits = total = 0
    for images, labels in loader:
        pred = model(images.cuda()).argmax(dim=1).cpu()
        hits += (pred == labels).sum().item()
        total += labels.numel()
    return hits / total

import jax
import jax.numpy as jnp
import optax

model = ResNet18(num_classes=10)
variables = model.init(jax.random.PRNGKey(0),
                       jnp.zeros((1, 32, 32, 3)), train=False)
params, batch_stats = variables["params"], variables["batch_stats"]
tx = optax.sgd(0.1, momentum=0.9)
opt_state = tx.init(params)

@jax.jit
def train_step(params, batch_stats, opt_state, images, labels):
    def loss_fn(params):
        logits, updated = model.apply(
            {"params": params, "batch_stats": batch_stats},
            images, train=True, mutable=["batch_stats"])
        loss = optax.softmax_cross_entropy_with_integer_labels(
            logits, labels).mean()
        return loss, updated["batch_stats"]

    (loss, batch_stats), grads = jax.value_and_grad(
        loss_fn, has_aux=True)(params)
    updates, opt_state = tx.update(grads, opt_state, params)
    params = optax.apply_updates(params, updates)
    return params, batch_stats, opt_state, loss

for images, labels in loader:          # NHWC: (128, 32, 32, 3)
    params, batch_stats, opt_state, loss = train_step(
        params, batch_stats, opt_state, images, labels)

What to expect: loss starts near ln(10) ≈ 2.30, the entropy of a uniform guess over ten classes, and drops below 1.0 within the first few epochs. With standard augmentation (random crops and flips) and a cosine schedule over a couple hundred epochs, a ResNet-18-class model on CIFAR-10 lands in the mid-90s percent test accuracy on a single consumer GPU in a few hours; exact figures depend on augmentation, schedule, and the stem choice, so treat that as a band. The instructive curve is the comparison run: the no-shortcut twin trains fine for the first few epochs, then its loss plateaus visibly higher, the degradation problem reproduced on your own machine.

Applications

For roughly a decade after 2015, "the backbone" meant a ResNet. It won ImageNet classification, ImageNet detection, COCO detection, and COCO segmentation in the year it appeared, and ResNet-50 became the standard-issue feature extractor of computer vision: Faster R-CNN and Mask R-CNN detect and segment on top of ResNet features, feature pyramid networks tap its stage outputs (the layer1 through layer4 activations) at four scales, and years of detection, segmentation, pose, and video systems shipped with a ResNet under the hood. It also became the reference workload of an era of systems work: "ResNet-50 on ImageNet" was for years the benchmark on which training-throughput papers and hardware were compared.

The larger legacy is the residual connection itself, detached from convolutions. A transformer block is two residual blocks in sequence: x ← x + Attention(Norm(x)), then x ← x + MLP(Norm(x)). Every layer of every large language model is a residual block; the "residual stream" that mechanistic interpretability work reads and edits is exactly the identity highway this page derives. The pre-norm placement that modern LLMs use is the pre-activation lesson re-learned, and you can read the pattern in a minimal, concrete codebase in my notes on nanoGPT, where each block is literally two x = x + f(norm(x)) lines. Stable diffusion's U-Net, AlphaFold's Evoformer, speech encoders, graph networks: the load-bearing skeleton everywhere is additive identity paths with learned corrections.

As a pure classifier, ResNet has been passed by ConvNeXt and vision transformers at the accuracy frontier, but ResNet-18 and ResNet-50 remain the default when you need a fast, robust, deployment-friendly vision model with a decade of tooling behind it, and they remain the standard baseline every new vision architecture is still measured against.

Against the real libraries

torchvision ships the canonical implementations: torchvision.models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1) gives the pretrained network (about 69.8 percent ImageNet top-1), and resnet50 comes with both the original V1 weights (about 76.1) and V2 weights (about 80.9) retrained with a modern recipe, the same architecture trained better. Over the reference code above, torchvision adds the full family (18 to 152, plus wide and ResNeXt variants), pretrained weights with documented preprocessing transforms bundled per weight enum, the zero_init_residual trick (initializing each block's last BN gamma to zero so every block starts as an exact identity), and integration with the detection and segmentation models that consume backbones. timm goes further: dozens of ResNet variants (the ResNet-strikes-back training recipes, SE-ResNets, ResNeXt, stem and downsample variations), uniform APIs for feature extraction at every stage, and consistently stronger pretrained weights than the originals. On the JAX side, Flax maintains an official ImageNet ResNet example much like the model above, and the batch-norm handling it demonstrates (mutable batch_stats, explicit train flag) is the part worth copying.

The from-scratch version is genuinely enough when the network is the object of study rather than the tool: architecture ablations, experiments on normalization or initialization, or teaching. The moment you need competitive accuracy per GPU-hour, pretrained weights are the whole game, and those live in the libraries.

Verification is where mirroring torchvision's module names pays off. Because the reference model shares torchvision's parameter tree exactly, its pretrained state dict loads with strict=True, and if the loaded model then agrees with torchvision's own output on random inputs, the architecture is byte-for-byte right: every stride, every padding, every shortcut. This is a much stronger check than eyeballing layer printouts, and it is the first thing to run after writing any reimplementation. The JAX tab ports the same weights across frameworks, which additionally forces you to get the layout conversions right: PyTorch convs are OIHW and Flax convs are HWIO, PyTorch linears store (out, in) against Flax's (in, out), and BN running stats move into batch_stats.

import torch
from torchvision.models import resnet18, ResNet18_Weights

tv = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1).eval()
mine = ResNet18().eval()

# strict=True: every parameter name and shape must match, so a
# clean load already proves the architectures are identical.
mine.load_state_dict(tv.state_dict(), strict=True)

x = torch.randn(4, 3, 224, 224)
with torch.no_grad():
    torch.testing.assert_close(mine(x), tv(x),
                               rtol=1e-4, atol=1e-4)
print("matches torchvision resnet18 with pretrained weights")

import numpy as np
import torch
import jax.numpy as jnp
from torchvision.models import resnet18, ResNet18_Weights

tv = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1).eval()
sd = {k: v.numpy() for k, v in tv.state_dict().items()}

def conv(k):   # PyTorch OIHW -> Flax HWIO
    return {"kernel": jnp.asarray(sd[k].transpose(2, 3, 1, 0))}

def bn(k, params, stats):
    params.update(scale=jnp.asarray(sd[k + ".weight"]),
                  bias=jnp.asarray(sd[k + ".bias"]))
    stats.update(mean=jnp.asarray(sd[k + ".running_mean"]),
                 var=jnp.asarray(sd[k + ".running_var"]))

params, stats = {}, {}
params["Conv_0"] = conv("conv1.weight")
bn("bn1", params.setdefault("BatchNorm_0", {}),
   stats.setdefault("BatchNorm_0", {}))
# nn.compact numbers submodules in call order, so BasicBlock_0..7
# map to layer1.0 .. layer4.1; inside a block, Conv_0/1 are
# conv1/2 and Conv_2 + BatchNorm_2 are the downsample projection.
for i in range(8):
    t = f"layer{i // 2 + 1}.{i % 2}"
    p, s = {}, {}
    p["Conv_0"] = conv(t + ".conv1.weight")
    p["Conv_1"] = conv(t + ".conv2.weight")
    bn(t + ".bn1", p.setdefault("BatchNorm_0", {}),
       s.setdefault("BatchNorm_0", {}))
    bn(t + ".bn2", p.setdefault("BatchNorm_1", {}),
       s.setdefault("BatchNorm_1", {}))
    if t + ".downsample.0.weight" in sd:
        p["Conv_2"] = conv(t + ".downsample.0.weight")
        bn(t + ".downsample.1", p.setdefault("BatchNorm_2", {}),
           s.setdefault("BatchNorm_2", {}))
    params[f"BasicBlock_{i}"], stats[f"BasicBlock_{i}"] = p, s
params["Dense_0"] = {"kernel": jnp.asarray(sd["fc.weight"].T),
                     "bias": jnp.asarray(sd["fc.bias"])}

x = np.random.default_rng(0).standard_normal(
    (4, 224, 224, 3)).astype(np.float32)         # NHWC for Flax
y_jax = model.apply({"params": params, "batch_stats": stats},
                    jnp.asarray(x), train=False)
with torch.no_grad():                            # NCHW for torch
    y_tv = tv(torch.from_numpy(x.transpose(0, 3, 1, 2)))
np.testing.assert_allclose(np.asarray(y_jax), y_tv.numpy(),
                           rtol=1e-3, atol=1e-3)

Traps and misconceptions

"The degradation problem was overfitting." No: the deeper plain networks were worse on the training set. That single fact is what makes the phenomenon an optimization problem, and it is why the fix is a reparameterization rather than regularization. If it had been overfitting, dropout and weight decay would have been the answer, and ResNets would not exist.

"Skip connections only help by fixing vanishing gradients." Gradient flow is the cleanest part of the story, but it is not all of it: with batch norm in the stack, raw gradient magnitudes in plain networks are not necessarily tiny. Residual connections also reshape the loss landscape (visualization work shows plain deep networks developing chaotic loss surfaces that residual ones smooth out) and change the functions the network represents near initialization, since every block starts close to the identity. "Better-conditioned optimization through an identity default" is the honest summary; "vanishing gradients" is one lens on it.

"The shortcut is always an identity." At every stage boundary it is a strided 1×1 convolution plus BN, and this is where reimplementations quietly go wrong: use stride 1 in the projection and the tensors will not add; skip the BN and the two branches arrive at the addition on different scales. The strict-mode weight load in the verification section catches both mistakes immediately, which is a good reason to run it.

"Addition after the ReLU, or ReLU inside the shortcut." The v1 block adds first, then applies ReLU; applying ReLU to the residual branch's output before the addition, or worse to the shortcut itself, constrains the block to non-negative corrections and dirties the identity path. If you want the fully clean path, that is precisely the pre-activation variant, where nothing at all sits between one block's addition and the next.

"Evaluating with batch norm in training mode." Forgetting model.eval() in PyTorch, or passing train=True at inference in Flax, makes BN normalize by the statistics of the test batch itself. Accuracy then varies with batch size and collapses at batch size one, a bug that looks like a modeling problem and is actually a mode flag. The Flax version makes this failure hard to write, which is one quiet argument for explicit state.