Multi-task and meta-learning: shared structure, fast adaptation, and in-context learning

Why this subject matters now

Five years ago meta-learning was a distinct subfield with its own benchmarks, its own algorithms, and its own leaderboards, and a practitioner could ignore it unless they worked on few-shot classification. That world inverted. The dominant workflow in machine learning today, pretrain a large model on broad data, then adapt it to a downstream task with a small amount of task-specific signal, is structurally a meta-learning pipeline, in which an expensive outer process produces an initialization from which cheap inner adaptation succeeds. Prompting a language model with a handful of examples is few-shot learning executed in a forward pass. Brown et al. named the phenomenon in-context learning precisely because GPT-3 behaved like a meta-learner nobody had explicitly trained to be one. Understanding why a pretrained initialization adapts quickly, what the adaptation is doing to the representation, and when transfer turns negative is no longer niche knowledge, it is the daily physics of applied machine learning.

The multi-task side matured in parallel and in production. Recommendation systems predict clicks, watch time, likes, and purchases from one shared network. Driving stacks predict depth, segmentation, detection, and motion from one backbone, and speech models transcribe, translate, and detect language with one set of weights. All of them confront the three problems this page derives. They must weight losses whose scales differ by orders of magnitude, handle task gradients that point in opposing directions, and detect that a task is being hurt rather than helped by its neighbors. Meanwhile the questions meta-learning asked in 2017, what it means to learn an algorithm rather than a function, have become central scientific questions about large models. The induction-head and learned-optimizer literatures are meta-learning analysis applied to transformers. A practitioner is expected to know both the classical machinery and why most of it was absorbed, rather than defeated, by scale.

Core theory

When does sharing help? The bias-variance accounting

Every method on this page is a bet that tasks share structure, and the bet can lose. The cleanest way to see both sides is the smallest possible example. Suppose two tasks each require estimating a scalar mean. Task 1 has true mean \( \mu_1 \), task 2 has \( \mu_2 \), and each provides \( n \) samples with noise variance \( \sigma^2 \). Estimating separately, each task uses its own sample mean, which is unbiased with variance \( \sigma^2/n \), so the per-task mean squared error is \( \sigma^2/n \). Estimating jointly, both tasks use the pooled mean of all \( 2n \) samples. Pooling halves the variance to \( \sigma^2/2n \), but the pooled estimator converges to \( (\mu_1+\mu_2)/2 \), so for task 1 it carries squared bias \( (\mu_1 - \mu_2)^2/4 \). Writing \( \Delta = \mu_1 - \mu_2 \), the comparison is

and pooling wins exactly when \( \Delta^2 < 2\sigma^2/n \). Every qualitative fact about transfer is already in this inequality. Sharing helps when data is scarce (small \( n \)), when noise is large, and when tasks are similar (small \( \Delta \)). It hurts when any of those reverse, and the harm does not go away with more data, because the bias term is constant while the variance saving shrinks. Negative transfer is not a pathology of neural networks. It is this inequality running in the wrong direction. The practical methods later on the page, soft sharing, task grouping, gradient surgery, are all devices for interpolating between the two columns rather than committing to either.

The same accounting scales up to learned representations. Baxter formalized inductive bias learning in 2000. If \( T \) tasks share a representation drawn from a class \( \mathcal{F} \) and each task fits a head from a class \( \mathcal{G} \) on top, the per-task sample complexity behaves like \( O\!\big( \tfrac{C(\mathcal{F})}{T} + C(\mathcal{G}) \big) \) rather than \( O\!\big( C(\mathcal{F}) + C(\mathcal{G}) \big) \). The cost of learning the shared representation is amortized across tasks, while the cost of the per-task head is not. Maurer, Pontil, and Romera-Paredes sharpened this into excess-risk bounds of the form \( O\!\big( \sqrt{C(\mathcal{F})/(nT)} + \sqrt{C(\mathcal{G})/n} \big) \) in 2016, and Tripuraneni, Jordan, and Jin (Berkeley, 2020) showed the amortization argument requires a task-diversity condition. The training tasks must collectively span the directions of variation the representation needs, otherwise \( T \) grows and the bound does not shrink. That condition is the theoretical name for a failure mode every practitioner has seen, where twenty near-duplicate tasks teach the trunk no more than one.

The taxonomy, stated precisely

The settings on this page differ along three axes, what is available at training time, what changes between training and deployment, and what the deliverable is. Conflating them causes real confusion, so here is the taxonomy with each setting's defining assumption.

The distinctions that matter most are these. Transfer learning cares only about the target and treats the source as scaffolding. Domain adaptation is transfer where the shift is in \( p(x) \) rather than in the task, which is why it admits theory (the Ben-David bound below) that generic transfer does not. Meta-learning is the only setting whose output is not a model but a learning procedure, which is why it alone requires the bi-level objective. Continual learning adds the arrow of time, forbidding revisits and thereby creating the forgetting problem. In-context learning, covered near the end, is meta-learning where the adaptation procedure is a forward pass instead of an optimizer.

Multi-task learning: hard and soft sharing

The standard multi-task objective is a weighted sum over tasks,

where \( \theta_{\text{sh}} \) is shared and \( \theta_i \) is task-specific. Hard parameter sharing, dating to Caruana's 1997 treatment of multitask learning, makes \( \theta_{\text{sh}} \) literally the same tensor for all tasks, a shared trunk with per-task heads. It is the strongest regularizer, the cheapest at inference, and the most prone to negative transfer, because every task must live in one representation. Soft sharing gives each task its own column of parameters and couples the columns with a penalty, for instance \( \sum_{i < j} \|\theta_i - \theta_j\|^2 \) or a trace-norm penalty on the stacked parameter matrix. The tasks can then disagree wherever the data insists, at the price of \( T \) times the parameters. Cross-stitch and sluice networks, in the architecture section below, learn the coupling instead of fixing it.

The immediate practical problem is choosing \( w_i \). Losses for depth regression (meters squared), segmentation (cross-entropy in nats), and detection (a mixture) differ in scale by orders of magnitude, and uniform weights simply hand the optimization to whichever task has the largest gradients. Grid search over weights is exponential in \( T \). The field produced two families of answers, reweight by uncertainty or operate directly on gradients.

Uncertainty weighting, derived

Kendall, Gal, and Cipolla (Cambridge, 2018) derived weights from a probabilistic model rather than a heuristic. Give each regression task \( i \) a homoscedastic noise model. The label is the network output plus Gaussian noise whose variance \( \sigma_i^2 \) is a learned constant per task, not per input,

The negative log-likelihood of one observation is, keeping every term,

Summing over tasks and dropping the constant, the joint objective over \( \theta \) and the noise parameters is

The structure is exactly what was needed. Each task's loss is divided by twice its noise variance, so noisy or hard tasks are automatically down-weighted, and the \( \log \sigma_i \) term prevents the degenerate solution \( \sigma_i \to \infty \) that would zero out every loss. The weights are not hyperparameters. They are trained by the same gradient descent. Setting the derivative with respect to \( \sigma_i \) to zero shows what the optimum does. From \( \partial \L / \partial \sigma_i = -\L_i/\sigma_i^3 + 1/\sigma_i = 0 \) follows \( \sigma_i^2 = \L_i \), so at the optimum each task is weighted by the reciprocal of its own current loss, a self-normalizing scheme. For classification the same derivation goes through with a temperature-scaled softmax \( p(y \mid x) = \softmax(f_\theta(x)/\sigma_i^2) \), yielding approximately \( \L_i/\sigma_i^2 + \log \sigma_i \). In practice one parameterizes \( s_i = \log \sigma_i^2 \) for positivity and optimizes \( \sum_i \frac{1}{2} e^{-s_i} \L_i + \frac{1}{2} s_i \), which is numerically stable and two lines of code.

GradNorm, from Chen et al. (2018), attacks the same imbalance at the level of gradient norms instead of likelihoods. Define \( G_i = \|\nabla_{W} w_i \L_i\| \), the norm of task \( i \)'s weighted gradient at the last shared layer \( W \), and let \( \bar{G} \) be the mean across tasks. Define each task's relative inverse training rate \( r_i = \tilde{\L}_i / \E_j[\tilde{\L}_j] \), where \( \tilde{\L}_i = \L_i(t)/\L_i(0) \) measures how far task \( i \) has descended relative to its starting loss. GradNorm then treats the weights \( w_i \) as parameters of an auxiliary objective

optimized only with respect to the \( w_i \) (the target \( \bar{G} r_i^\alpha \) is treated as a constant), with the weights renormalized to sum to \( T \) after each step. Tasks training slowly (high \( r_i \)) get their gradient norms pushed above average, tasks racing ahead get pulled below, and \( \alpha \) sets how aggressive the rebalancing is. The honest summary from later benchmark work (Kurin et al., Oxford, 2022) is that a tuned fixed weighting is a much stronger baseline than the original papers assumed, a theme this page returns to.

Gradient conflict: PCGrad and CAGrad

Weighting cannot fix a geometric problem. Two task gradients can point in opposing directions, \( g_i \cdot g_j < 0 \), so any positive combination of them moves against at least one task. Yu et al. (2020, a Stanford, Berkeley, and Google collaboration) named this gradient conflict and proposed projecting it away. PCGrad processes tasks pairwise. Whenever \( g_i \cdot g_j < 0 \), replace \( g_i \) by its projection onto the plane normal to \( g_j \),

The algebra of the projection is worth doing once. Decompose \( g_i = g_i^{\parallel} + g_i^{\perp} \) with \( g_i^{\parallel} = \frac{g_i \cdot g_j}{\|g_j\|^2} g_j \) the component along \( g_j \). When the dot product is negative, \( g_i^{\parallel} \) points against \( g_j \), and subtracting it removes exactly the part of task \( i \)'s update that damages task \( j \), leaving \( g_i^{\perp} \cdot g_j = g_i \cdot g_j - \frac{(g_i \cdot g_j)}{\|g_j\|^2}\, (g_j \cdot g_j) = 0 \). The surgered gradient is exactly orthogonal to the conflicting task, neither helping nor hurting it. Each task's gradient is surgered against the others in random order and the results are summed. The cost is one extra gradient computation per task (you need per-task gradients, not the fused sum) plus \( O(T^2) \) dot products.

CAGrad, from Liu et al. (UT Austin, 2021), replaces the pairwise heuristic with an optimization. Let \( g_0 \) be the average gradient. CAGrad seeks the update \( d \) that maximizes the worst-case improvement across tasks while staying within a ball around the average,

The two limits explain the design. At \( c = 0 \) the update is exactly the average gradient, so CAGrad inherits ordinary multi-task SGD's convergence to a stationary point of the average loss. As \( c \to \infty \) it approaches the multiple-gradient descent direction that optimizes only the worst-off task, which can stall global progress. The constraint makes the worst-case objective a correction to average descent rather than a replacement, and the inner problem reduces by Lagrangian duality to a low-dimensional optimization over task weights, solvable cheaply since \( T \) is small. The same benchmark caveat applies. Kurin et al. showed that with equal tuning budgets and proper regularization, plain scalarization matches PCGrad and CAGrad on several of their original benchmarks. The geometric pathology is real, but it is rarer in well-conditioned training setups than the surgery papers implied, and per-task gradients cost a factor of \( T \) in backward passes.

Task grouping and negative transfer

The strongest lever is not how to combine tasks but which tasks to combine. Two lines of evidence made this concrete. Taskonomy (Zamir et al., Stanford, 2018) measured pairwise transfer among 26 vision tasks by actually fine-tuning between every pair, and found a structured, asymmetric transfer graph. Surface normals transfer well to depth, but several semantically close pairs transfer poorly, and transfer is not symmetric. Standley et al. (2020) then measured joint training directly and found that which tasks train well together is not predicted by which tasks transfer well, and that the best partition of five tasks into groups often beats both one big multi-task network and fully separate networks. Their framework trains networks on subsets and searches over partitions, effectively treating task grouping as a combinatorial hyperparameter. Follow-up work (Fifty et al., Google, 2021) approximates the search by measuring inter-task gradient affinity during a single training run, measuring how much a step on task \( i \)'s gradient reduces task \( j \)'s loss.

Negative transfer, the case where adding a task or a source domain hurts, is the empirical face of the bias term in the opening inequality. It shows up reliably in three situations, tasks with genuinely different optimal representations forced through a small shared trunk (capacity contention), tasks with very different data volumes where the large task dominates the trunk (the weighting problem in disguise), and source domains whose superficial similarity hides label-function disagreement, where the shared assumption \( p(y \mid x) \) itself is false. The remedies, in order of cost, are to reweight, then de-conflict, then regroup, then unshare. The reliable diagnostic is to train the single-task model anyway. If the multi-task model loses to it, that gap is the negative transfer, measured.

Multi-task architectures

The architectural spectrum runs from one extreme, everything shared except a linear head, to the other, nothing shared except a coupling penalty. The named points on the spectrum follow.

hard sharing            cross-stitch / sluice          mixture-of-experts
                                                          (MMoE)
 x                        x        x                        x
 │                        │        │                   ┌────┼────┐
 trunk                 trunk A  trunk B                E1   E2   E3
 │                        │  ╲  ╱  │                    └──┬─┴─┬──┘
 ├── head 1               │  stitch │              gate1──▶●   ●◀──gate2
 ├── head 2               │  ╱  ╲  │                       │   │
 └── head 3            head A    head B                head 1  head 2

 one representation     learned linear mixing        per-task soft routing
 for all tasks          between task columns         over shared experts

Cross-stitch networks (Misra et al., CMU, 2016) instantiate one network column per task and insert, after each layer, a learned \( 2 \times 2 \) (for two tasks) mixing matrix. The activation entering task A's next layer is \( \tilde{h}_A = \alpha_{AA} h_A + \alpha_{AB} h_B \), and likewise for B. The \( \alpha \) values are trained with everything else, so the network learns per-layer how much to share. Initialized near identity, the model starts as two independent columns and shares only where gradients push it to. Sluice networks (Ruder et al., 2019) generalize the idea with subspace-level sharing and learned skip connections. Both make sharing a continuous learned quantity, the right abstraction, and both pay \( T \) columns of compute, which is why production systems rarely use them directly.

Mixture-of-experts brings conditional computation to the problem. Multi-gate MoE (Ma et al., Google, 2018) shares a pool of expert subnetworks across tasks but gives each task its own gating network. Task \( k \)'s representation is \( \sum_e g^k_e(x) \, f_e(x) \) with \( g^k(x) = \softmax(W_k x) \). Tasks that want the same features learn similar gates and share experts. Tasks that conflict learn disjoint gates and partition the expert pool, so the architecture discovers the task grouping that Standley et al. searched for combinatorially. MMoE and its successor architectures run in production ranking systems at YouTube-scale precisely because the negative-transfer failure mode, one task polluting a monolithic trunk, becomes a soft routing decision instead of a fight over shared weights.

Adapters and modular approaches invert the question. Instead of deciding what to share, share everything and inject small task-specific modules. Houlsby et al. (Google, 2019) froze a pretrained transformer and inserted two-layer bottleneck MLPs (down-project, nonlinearity, up-project, residual) after each sublayer, training only those, roughly 3 percent of the parameters, to near-full fine-tuning quality. This line, through LoRA and the parameter-efficient fine-tuning family, is the multi-task architecture that actually won in the large-model era, with one frozen trunk and a library of cheap task modules to swap at inference. It works because the trunk is pretrained well enough to contain most of what every task needs.

Transfer and domain adaptation: what theory guarantees

Transfer learning's two default modes are feature extraction, freeze the pretrained representation and train a head, and fine-tuning, continue training everything at a small learning rate. The classical guidance, small target datasets favor freezing and large ones fine-tuning, holds, with one modern refinement. Fine-tuning can distort pretrained features early in training while the randomly initialized head produces large gradients, which motivates probe-then-fine-tune schedules (Kumar et al., 2022, who showed fine-tuning can underperform linear probing out-of-distribution for exactly this reason). Parameter-efficient methods now occupy the middle of the spectrum at almost no accuracy cost.

Domain adaptation is the one corner of transfer with a sharp theory. Ben-David et al. (2010) bound the target error of a hypothesis \( h \) trained on the source. Define \( \epsilon_S(h) \) and \( \epsilon_T(h) \) as the error of \( h \) on source and target distributions, and define the \( \mathcal{H}\Delta\mathcal{H} \)-divergence between the domains as

the largest disagreement-rate gap between domains that any pair of hypotheses in the class can exhibit. The bound is

The derivation is three applications of the triangle inequality for classification error, worth seeing because each term of the bound falls out of a specific step. Let \( h^{*} \) be the joint minimizer achieving \( \lambda^{*} \), and write \( \epsilon(h, h') \) for the probability that two hypotheses disagree. First, \( \epsilon_T(h) \le \epsilon_T(h^{*}) + \epsilon_T(h, h^{*}) \), which says target error is at most the best joint hypothesis's target error plus the disagreement with it on the target. Second, the disagreement on the target is at most the disagreement on the source plus the largest possible cross-domain disagreement gap, \( \epsilon_T(h, h^{*}) \le \epsilon_S(h, h^{*}) + \tfrac{1}{2} d_{\mathcal{H}\Delta\mathcal{H}} \), which is exactly what the divergence was defined to control (the pair \( (h, h^{*}) \) is one of the pairs the supremum ranges over). Third, \( \epsilon_S(h, h^{*}) \le \epsilon_S(h) + \epsilon_S(h^{*}) \) by the triangle inequality on the source. Summing, \( \epsilon_T(h) \le \epsilon_S(h) + \tfrac{1}{2} d_{\mathcal{H}\Delta\mathcal{H}} + \epsilon_S(h^{*}) + \epsilon_T(h^{*}) \), which is the bound. Read as instructions, the bound says to minimize source error (the first term), make the domains indistinguishable to the hypothesis class (the second), and hope the task is jointly learnable at all (the third, \( \lambda^{*} \), which no algorithm can reduce and which is the formal location of negative transfer, since if no single hypothesis does well on both domains, aligning them cannot save you).

DANN, domain-adversarial neural networks (Ganin et al., 2016, a collaboration spanning Skoltech and Montreal), turns the second term into a training signal. The key observation is that the \( \mathcal{H} \)-divergence is estimable without target labels, because it only asks whether a classifier can tell the domains apart. Train a domain discriminator \( d \) on features \( f_\theta(x) \) to predict source-vs-target. If the discriminator achieves error \( \epsilon_d \), then the empirical divergence is approximately \( 2(1 - 2\epsilon_d) \), so a discriminator at chance (\( \epsilon_d = 0.5 \)) certifies indistinguishable feature distributions. DANN then trains the feature extractor to maximize discriminator error while minimizing source label error, implemented with the gradient reversal layer, identity in the forward pass, multiply by \( -\lambda \) in the backward pass, so a single backward pass pushes the featurizer down the label loss and up the domain loss simultaneously. The honest caveat is baked into the bound. DANN minimizes the divergence term while assuming \( \lambda^{*} \) stays small, and adversarial alignment can violate that assumption by collapsing class-discriminative structure to fool the discriminator, which is why conditional and class-aware variants followed.

When the shift is pure covariate shift, \( p(x) \) changes but \( p(y \mid x) \) does not, importance weighting gives an unbiased correction without any representation change. The identity is one line,

so reweighting each source example by the density ratio \( w(x) = p_T(x)/p_S(x) \) makes source-domain training minimize target-domain risk in expectation. The ratio is estimated without density estimation by training a probabilistic domain classifier. If \( P(\text{target} \mid x) \) is its output and \( n_S, n_T \) are the sample counts, Bayes' rule gives \( w(x) = \frac{P(\text{target} \mid x)}{P(\text{source} \mid x)} \cdot \frac{n_S}{n_T} \). The failure mode is variance. Where the domains barely overlap, a few source points carry enormous weights and the effective sample size collapses, which is why practical versions clip or temper the weights.

Test-time adaptation pushes adaptation past deployment, with no source data, no target labels, just the unlabeled test stream. TENT (Wang et al., 2021) adapts only batch-norm affine parameters by minimizing the entropy of the model's own predictions on test batches, on the theory that confident predictions correlate with correct features under mild shift. Test-time training (Sun et al., Berkeley, 2020) attaches a self-supervised task and fine-tunes the shared encoder on each test input. Both work under corruption-style shifts and can diverge under label shift, cheap insurance rather than adaptation guarantees.

The meta-learning formalism

Meta-learning changes the object of study from a model to an adaptation procedure, and everything else follows from writing that down carefully. Assume a distribution over tasks \( \T_i \sim p(\T) \). Each task is itself a learning problem. It has a loss \( \L_{\T_i} \) and data, which is split into a support set \( \D^{\text{s}}_i \) (the few examples the learner may adapt on) and a query set \( \D^{\text{q}}_i \) (the examples that measure whether adaptation worked). The meta-learner is a pair, a parameterized adaptation procedure \( \text{Adapt}_\theta \) mapping a support set to task parameters, and meta-parameters \( \theta \) shared across tasks. The meta-training objective is the bi-level problem

The support/query split inside each task is not bookkeeping. It is the entire point. Evaluating the adapted parameters on the same data they adapted on would reward memorization of the support set, and the procedure that wins that game is a lookup table. Evaluating on held-out queries makes the outer objective measure generalization-after-adaptation, so the meta-learner is selected for producing procedures that generalize from small samples, which is the actual goal. The split mirrors, one level up, the train/validation split of ordinary learning, and meta-train tasks are to meta-test tasks as training data is to test data. At meta-test time the protocol is fixed. Draw an unseen task, hand the learner its support set, run the same adaptation procedure, and report query performance. Nothing about the meta-test task's query set ever influences \( \theta \).

The taxonomy of meta-learners is a taxonomy of what \( \text{Adapt}_\theta \) is, a few steps of gradient descent from a learned initialization (optimization-based, MAML and family), a nearest-class comparison in a learned embedding space (metric-based, prototypical and matching networks), or an arbitrary learned function that eats the support set and emits predictions (black-box). The three families trade inductive bias against flexibility in that order, and the black-box family is the one that scale ended up vindicating.

MAML, derived in full

Model-agnostic meta-learning (Finn, Abbeel, and Levine, Berkeley, 2017) chooses the most portable adaptation procedure available, gradient descent itself. The meta-parameters are nothing but the initialization. For task \( i \), the inner update with learning rate \( \alpha \) is one (or a few) SGD steps on the support loss,

and the outer objective is the query loss at the adapted parameters, summed over a meta-batch of tasks,

The meta-gradient requires differentiating through the inner update, because \( \theta \) appears both directly and inside the gradient being subtracted. Apply the chain rule with Jacobians. The inner update's Jacobian with respect to \( \theta \) is

where \( \nabla^2_\theta \L^{\text{s}}_i \) is the Hessian of the support loss at the initialization. The meta-gradient of one task's outer loss is therefore

Read it right to left. Compute the ordinary gradient of the query loss at the adapted point, then transport it back through the linearization of the update map. The identity term carries the query gradient straight through, which is what the first-order approximation keeps. The Hessian term is the correction that accounts for how moving the initialization changes where the inner step lands. With \( k \) inner steps \( \theta = \phi^{(0)} \to \phi^{(1)} \to \cdots \to \phi^{(k)} \), the chain rule stacks one such factor per step,

with the Hessians evaluated at each intermediate iterate, which is why naive MAML must store the whole inner trajectory for the backward pass, with memory linear in \( k \). No Hessian matrix is ever materialized in practice. Reverse-mode autodiff computes the Hessian-vector product \( \nabla^2 \L^{\text{s}} \cdot v \) at the cost of roughly one extra backward pass, so second-order MAML costs a small constant factor over first-order per step, plus the trajectory storage.

FOMAML, the first-order approximation, drops the Hessian term entirely. It applies the query gradient evaluated at \( \phi_i \) directly to \( \theta \), as if the Jacobian were the identity. Formally it approximates \( I - \alpha \nabla^2 \L^{\text{s}} \approx I \), which is accurate when \( \alpha \) is small or the support loss is nearly flat at \( \theta \), and empirically loses little on the standard benchmarks. Reptile (Nichol, Achiam, and Schulman, OpenAI, 2018) is even simpler. Run \( k \) inner SGD steps to get \( \phi_i \), then move the initialization toward the adapted parameters, \( \theta \leftarrow \theta + \epsilon\, (\phi_i - \theta) \), with no query set and no explicit meta-gradient at all. That this works demands an explanation, and the Taylor-expansion analysis in their paper provides one. Problem 3 below works it through. The summary is that in expectation over minibatch orderings, both FOMAML's and Reptile's updates contain, at second order in the step size, the term \( -\tfrac{\alpha^2}{2}\, \nabla_\theta \big( \bar{g}_1 \cdot \bar{g}_2 \big) \), the gradient that increases the inner product between gradients computed on different minibatches of the same task. Maximizing within-task gradient alignment is a sensible proxy for fast adaptation. From such an initialization, a step on any minibatch also reduces the loss on the others, so few steps suffice. MAML's Hessian term implements this alignment pressure exactly. FOMAML and Reptile inherit it on average through the curvature of the inner trajectory.

Two structural results reshaped how MAML is understood. ANIL (Raghu et al., 2020) froze the network body during the inner loop, adapting only the final linear head, and matched full MAML on the standard few-shot image benchmarks. Probing the representations showed they barely change during MAML's own inner loop anyway. The conclusion, feature reuse rather than rapid learning, is that MAML's outer loop mostly learns a strong shared representation, and the inner loop mostly re-fits a linear classifier on top of it. This finding quietly predicts the later empirical success of the pretrain-plus-linear-probe baselines in the metric-learning section, and it is the intellectual bridge to why plain pretraining ate the field. The second result is implicit MAML (Rajeswaran, Finn, Kakade, and Levine, 2019), which removes the trajectory-storage problem. Define the inner solution not as \( k \) explicit steps but as the solution of a regularized problem,

At the minimizer, the gradient vanishes, \( \nabla_\phi \L^{\text{s}}_i(\phi^{*}) + \lambda(\phi^{*} - \theta) = 0 \). This stationarity condition implicitly defines \( \phi^{*}(\theta) \), and the implicit function theorem differentiates it without unrolling anything. Differentiate the condition with respect to \( \theta \),

The meta-gradient is this inverse applied to the query gradient, computed approximately with a few conjugate-gradient iterations on Hessian-vector products. Everything depends only on the solution \( \phi^{*} \), not on the path to it, so the inner loop can run any optimizer for any number of steps in constant memory. The trade is that the inner problem must actually be solved near to optimality for the implicit gradient to be accurate, and the regularization strength \( \lambda \) becomes a real hyperparameter coupling the inner problem to the initialization. Meta-SGD (Li et al., 2017) extends MAML in the opposite, pragmatic direction. It learns a per-parameter inner learning rate vector \( \alpha \) (same shape as \( \theta \)) jointly with the initialization, so the meta-learner learns not just where to start but how fast to move each coordinate, at the cost of doubling the meta-parameters.

Metric-based meta-learning

The metric family replaces inner-loop optimization with a comparison in a learned embedding space, which makes adaptation a forward pass and sidesteps second-order gradients entirely. The lineage starts with siamese networks (Koch et al., 2015, building on verification networks from the Toronto line of work). These train a shared encoder so that a pairwise head can judge whether two examples belong to the same class, then classify a new example by its nearest verified neighbor in the support set. Matching networks (Vinyals et al., DeepMind, 2016) made the comparison end-to-end and episodic. The prediction is an attention-weighted vote over the support set,

with \( c \) cosine similarity and, importantly, training episodes constructed to match the test protocol. Sample N classes, K support examples each, train to classify queries from exactly that support set. The episodic principle, train the way you test, was the paper's most durable contribution.

Prototypical networks (Snell, Swersky, and Zemel, Toronto, 2017) simplify the vote to a centroid comparison and are worth deriving because the simplification is exactly what makes them strong. Embed every support example with \( f_\theta \), and represent each class by its prototype, the mean embedding

Classify a query by a softmax over negative squared Euclidean distances to the prototypes,

trained by cross-entropy on query points over episodes. Two identities explain why this works better than it has any right to. First, expand the squared distance with \( z = f_\theta(x) \),

The \( -\|z\|^2 \) term is identical across classes and cancels in the softmax, leaving logits that are affine in \( z \), \( \text{logit}_k = w_k \cdot z + b_k \) with \( w_k = 2 c_k \) and \( b_k = -\|c_k\|^2 \). A prototypical network is therefore a linear classifier whose weights are manufactured from the support set by averaging, no optimization required. The metric method and the fine-tune-a-linear-head method are the same family, differing only in whether the head is computed in closed form or by gradient steps. Second, the probabilistic reading. If each class is modeled as a spherical Gaussian \( \N(c_k, \tfrac{1}{2} I) \) with equal priors, Bayes' rule gives exactly the softmax over negative squared distances, so the prototype rule is the Bayes classifier of an equal-covariance mixture density whose parameters are estimated by the sample means, which is also why the class mean is the right summary. For any Bregman divergence (squared Euclidean included) the point minimizing total divergence to a cluster is its mean. Using cosine distance instead breaks the mixture interpretation, and the original paper reports squared Euclidean working measurably better. Relation networks (Sung et al., 2018) replace the fixed metric with a small learned comparison network over concatenated embeddings, buying flexibility at the price of the closed-form structure.

Then came the deflationary results. Chen et al. (2019, a Georgia Tech and National Taiwan University collaboration), in a controlled comparison with identical backbones, showed that a plain classifier trained on all meta-training classes, then given a new linear head on the support set at test time (Baseline, and Baseline++ with a cosine head), is competitive with MAML, prototypical, matching, and relation networks, and that the ranking of meta-learners reshuffles when the backbone changes. Much of the reported progress had been backbone and implementation differences. Tian et al. (2020) pushed further. Pretrain an embedding on the merged meta-training set with ordinary cross-entropy, freeze it, fit a regularized linear classifier on each test episode's support set, and this beats essentially every published meta-learning algorithm of the preceding three years on miniImageNet and its relatives. Combined with ANIL's feature-reuse finding, the picture is coherent. On the standard benchmarks the dominant factor is representation quality, the episodic outer loop is a weak and expensive way to train a representation, and the adaptation step is well served by the cheapest possible head. The open question, explored by Meta-Dataset below, is whether this survives real distribution shift between meta-train and meta-test, where the answer is partially no.

Black-box and memory-based meta-learners

The black-box family makes the adaptation procedure a learned sequence model. Feed the support set in as a sequence, let a network with memory produce predictions for queries, and train the whole thing end-to-end across tasks. No gradient steps, no metric assumption. The inductive bias is only whatever the architecture carries. Santoro et al. (DeepMind, 2016) used a Neural Turing Machine variant fed \( (x_t, y_{t-1}) \) pairs, the label offset by one step so the network must bind an input to its label when the label arrives one step later, store the binding in external memory, and retrieve it when a similar input recurs. SNAIL (Mishra et al., Berkeley, 2018) replaced the recurrent core with temporal convolutions interleaved with soft attention, arguing that attention is the right primitive for pinpoint retrieval from a long experience buffer while convolutions aggregate context. It was, in hindsight, most of the way to using a transformer as the meta-learner.

Neural processes (Garnelo et al., DeepMind, 2018, in conditional and latent-variable variants) frame the same idea as learning a map from datasets to predictive distributions. Encode the support (context) points into an aggregated representation \( r = \tfrac{1}{n}\sum_j h(x_j, y_j) \) (order-invariant by construction, respecting dataset exchangeability), optionally sample a latent \( z \) to capture global function uncertainty, and decode \( p(y \mid x, r, z) \) at query points. They behave like a cheap, amortized cousin of Gaussian processes. The reason to care about this family grew rather than shrank. A decoder-only transformer trained autoregressively on sequences of \( (x, y) \) pairs is exactly a black-box meta-learner in this sense, the formal bridge between 2016-era memory-augmented meta-learning and in-context learning in language models. The same idea reached tabular prediction as prior-fitted networks (TabPFN, Hollmann et al., 2023). Pretrain a transformer on millions of synthetic datasets sampled from a prior, and it fits a new small dataset in one forward pass, approximate Bayesian inference meta-learned offline.

Bayesian meta-learning

Hierarchical Bayes is the formal frame that was always underneath. Posit task parameters \( \phi_i \) drawn i.i.d. from a shared prior \( p(\phi \mid \theta) \), and data for each task drawn given its parameters. The marginal likelihood of everything is

and meta-learning is empirical Bayes. Fit the prior's parameters \( \theta \) by (approximately) maximizing this marginal likelihood, then solve a new task by posterior inference \( p(\phi \mid \D^{\text{s}}, \theta) \) under the learned prior. Every meta-learner on this page is an approximation to this picture with a particular inference scheme. Grant et al. (Berkeley, 2018) made the MAML case precise. One truncated inner gradient descent from initialization \( \theta \) is approximate MAP inference of \( \phi_i \) under a Gaussian prior centered at \( \theta \), exactly so in the linear-model case, where early stopping after gradient steps from \( \theta \) is equivalent to MAP with a particular quadratic prior whose covariance depends on the step size, step count, and the loss curvature, and approximately so for networks. MAML's outer loop is then empirical Bayes on the prior's mean. This reading explains a MAML behavior the optimization view does not. The inner learning rate and step count act as the prior's precision, and tightening them trades adaptation flexibility against protection from overfitting the few support points, which is visibly what tuning them does in practice.

The point estimate is the frame's main deficiency. With five support points, the task posterior is genuinely wide, and a single \( \phi_i \) ignores that. PLATIPUS (Finn, Xu, and Levine, 2018) makes the adaptation stochastic, learning a Gaussian over initializations and using variational inference so that sampling then adapting draws approximate samples from the task posterior. Ambiguous support sets yield visibly diverse adapted functions. Bayesian MAML (Yoon, Kim, et al., KAIST, 2018) instead maintains a small ensemble of particles updated with Stein variational gradient descent in the inner loop, capturing multimodal task posteriors that a Gaussian cannot. Both matter wherever a few-shot learner must know that it does not know, and both cost roughly an ensemble factor over MAML.

In-context learning as emergent meta-learning

The empirical phenomenon, reported for GPT-3 by Brown et al. (OpenAI, 2020), is that a language model trained on nothing but next-token prediction, when given a prompt containing a few input-output demonstrations of a task it was never explicitly trained on, completes new inputs correctly, with accuracy improving in the number of demonstrations and in model scale. No weights change. In the vocabulary of this page, the forward pass conditioned on a support set is an adaptation procedure, pretraining was the outer loop, and the task distribution was implicit in the diversity of text. The paper's own framing was explicitly meta-learning. What was new was that nobody wrote the bi-level objective. It emerged from scale plus data diversity, which is the single most consequential fact in this subject's recent history.

Mechanistically, the best-understood ingredient is the induction head (Olsson et al., Anthropic, 2022). An induction head is a two-head circuit implementing the rule. Find an earlier occurrence of the current token, look at what followed it, and predict that. The first attention head copies each token's predecessor into its representation (a previous-token head). The second attends from the current position to positions whose predecessor matches the current token, then copies the attended token to the output, completing the pattern \( A\,B \ldots A \to B \). The evidence that these circuits carry in-context learning is unusually strong for interpretability work. Induction heads appear abruptly during training in a phase change coinciding exactly with the drop in the model's in-context learning score (the loss gap between late and early tokens in a context), the timing responds causally to interventions that shift when the heads can form, and ablating the heads after training removes most of the score. Generalized to attend over semantically matching rather than identical patterns, induction-style copying is a plausible substrate for few-shot-prompt behavior, though the paper claims mechanism only for small models and correlation at scale.

The theory results make the meta-learning reading exact in a restricted setting. Garg et al. (Stanford, 2022) trained transformers from scratch on sequences \( (x_1, f(x_1), \ldots, x_k, f(x_k), x_{\text{q}}) \) with \( f \) drawn from a function class, and showed they learn to predict \( f(x_{\text{q}}) \) for unseen \( f \). For linear functions they match the optimal least-squares predictor in error and degrade gracefully under distribution shift, and sparse linear functions and shallow trees are also learnable in context. Von Oswald et al. (ETH Zurich and Google, 2023) supplied a constructive mechanism. A linear self-attention layer can implement one step of gradient descent on the in-context regression loss. The construction is short enough to derive. Take the in-context loss over the \( k \) demonstrations,

One gradient step from \( W_0 \) with rate \( \eta \) changes the prediction on the query point \( x_{\text{q}} \) to

Now read the correction term as attention. It is a sum over context tokens of a value, the residual \( y_j - W_0 x_j \), weighted by an unnormalized attention score, the inner product \( x_j\T x_{\text{q}} \) between the token's key and the query token's query. A linear attention head whose key and query projections extract the \( x \)-part of each token and whose value projection computes the current residual therefore adds exactly the gradient-descent correction to the query token's prediction slot. With \( W_0 = 0 \) the value projection only needs to extract \( y_j \). Stacking \( L \) such layers executes \( L \) steps of gradient descent in the forward pass, and their experiments show trained linear-attention transformers converge to weights implementing precisely this, while Akyürek et al. (2023) showed standard transformers on the same tasks behave like higher-order methods, closer to ridge regression or Newton steps, than like plain GD.

A complementary line locates the task representation itself. Hendel et al. (Tel Aviv, 2023) and Todd et al. (Northeastern, 2023) independently showed that the hidden state at a particular layer and position of a prompted model contains a compact task vector or function vector. Extract that activation from a few-shot prompt, patch it into a zero-shot prompt for a fresh input, and the model performs the demonstrated task with most of the few-shot accuracy, no demonstrations present. The forward pass evidently factorizes, approximately, into infer-the-task then apply-the-task, which is precisely the structure PEARL builds by hand in meta-RL below.

The honest limits. The gradient-descent constructions live in linear attention on linear regression with tokens arranged just so. Real models on real language have nonlinear MLPs between layers, and no one has exhibited GD-in-forward-pass in a production model. The Bayesian reading (Xie et al., 2022: in-context learning as implicit posterior inference over latent concepts) explains different phenomena, and neither story cleanly survives Min et al.'s 2022 observation that randomizing demonstration labels often barely hurts few-shot accuracy, which suggests much of the benefit is task location, activating something already learned, rather than task learning from the mapping. The current synthesis is that small transformers on algorithmic tasks demonstrably implement learning algorithms in-context, while large language models on natural tasks do some mixture of retrieval, task location, and genuine in-context estimation, in proportions that remain an open research question.

Continual learning and catastrophic forgetting

Train a network on task A, then train it on task B with A's data gone, and performance on A collapses. This is catastrophic forgetting, documented for connectionist networks by McCloskey and Cohen in 1989 and unchanged in character since. The cause is nothing exotic. SGD on task B's loss moves parameters wherever B's gradients point, and nothing in the objective mentions A. The stability-plasticity tradeoff is the design axis. A perfectly stable model cannot learn task B, a perfectly plastic one cannot retain task A, and every method chooses a point on the axis by deciding which parameters are allowed to move and how far.

Elastic weight consolidation (Kirkpatrick et al., DeepMind, 2017) derives its answer from a Laplace approximation, and the derivation is the important part. What we want after seeing both datasets is the posterior \( \log p(\theta \mid \D_A, \D_B) = \log p(\D_B \mid \theta) + \log p(\theta \mid \D_A) - \log p(\D_B \mid \D_A) \), which is task B's likelihood plus a prior that is exactly task A's posterior. A's data is gone, so approximate its posterior. Around the mode \( \theta^{*}_A \) found by training on A, second-order Taylor expansion of the log-posterior gives

with no linear term because the gradient vanishes at the mode, and \( H \) the negative Hessian of the log-posterior there. This is the Laplace approximation. The posterior is replaced by \( \N(\theta^{*}_A, H^{-1}) \). The Hessian of a neural network log-likelihood is intractable and not even guaranteed negative semidefinite away from the exact optimum, so EWC substitutes the Fisher information matrix,

justified because at a maximum-likelihood solution the expected Hessian of the negative log-likelihood equals the Fisher (the standard information-matrix identity), and the Fisher is positive semidefinite by construction and computable from squared first derivatives, with labels sampled from the model's own predictive distribution, not the dataset labels (using dataset labels gives the empirical Fisher, a different and often worse approximation). Keeping only the diagonal for tractability, training on task B becomes

a quadratic anchor per parameter, stiff where A's likelihood was sharply curved (the parameter mattered to A) and loose where it was flat. The known deficiencies follow directly from the derivation's approximations. The anchor is local, so large moves that would preserve A's function along a curved valley are wrongly penalized, the diagonal ignores parameter interactions, and with many sequential tasks the sum of quadratic penalties over-constrains the model. Synaptic intelligence (Zenke, Poole, and Ganguli, Stanford, 2017) reaches a similar penalty online, without a separate Fisher pass. Accumulate each parameter's path integral of gradient-times-displacement during task A's training, \( \omega_d = \sum_t g_d(t) \, \Delta\theta_d(t) \), which measures how much that parameter's motion contributed to reducing A's loss, then set the anchor stiffness to \( \Omega_d = \omega_d / \big( (\Delta\theta_d)^2 + \xi \big) \) where \( \Delta\theta_d \) is the parameter's total displacement over the task and \( \xi \) prevents division blowup.

The other families. Replay stores a small buffer of past examples and mixes them into every batch, directly optimizing the joint objective the regularizers only approximate. In fair comparisons a few hundred replayed examples routinely beat every regularization method, the field's slightly uncomfortable open secret. Generative replay (Shin et al., 2017) replaces the buffer with a generative model of past tasks when storage or privacy forbids raw data. GEM and A-GEM (Lopez-Paz and Ranzato, Meta AI, 2017-2019) use the buffer differently, constraining updates to not increase loss on stored examples, PCGrad's geometry applied across time rather than across tasks. Parameter isolation allocates capacity per task. Progressive networks freeze old columns and add new ones with lateral connections, and PackNet prunes and re-trains freed weights per task. Isolation eliminates forgetting by construction, gives up transfer from new tasks back to old ones, and usually needs the task identity at test time to select the right mask.

Which brings up the evaluation problem, because continual learning results are notoriously incomparable. Van de Ven and Tolias's 2019 taxonomy separates three scenarios by what is known at test time, task-incremental (task identity given, so a multi-head model just selects the right head), domain-incremental (same heads, shifted inputs), and class-incremental (must distinguish all classes seen so far with no identity hint). The same method can look excellent in the first scenario and near-chance in the third. EWC in particular performs respectably task-incrementally and collapses class-incrementally, where replay dominates. Papers that do not state their scenario, buffer size, and head structure produce incomparable numbers, and enough of the early literature did exactly that to make standardized protocols (the Avalanche library's, among others) necessary correctives.

Meta-reinforcement learning

Meta-RL instantiates the formalism with tasks as MDPs drawn from a distribution. Reward functions vary (goal positions, target velocities) or dynamics vary (masses, terrains), and the support set becomes experience collected in the new MDP. RL\(^2\) (Duan et al., 2016, with a simultaneous DeepMind paper, Wang et al.'s learning-to-reinforcement-learn) is the black-box instantiation. A recurrent policy receives \( (s_t, a_{t-1}, r_{t-1}, \text{done}) \), its hidden state persists across episode boundaries within a trial on one MDP, and it is trained with ordinary policy gradient to maximize return summed over the whole trial. The recurrent state is forced to become a task-inference and adaptation variable. Early episodes explore, later episodes exploit, and the "RL algorithm" executed within a trial is an emergent property of the weights, hence the name. MAML-RL is the optimization instantiation. The inner loop is a policy-gradient step on trajectories from the new task and the outer loop differentiates through it, with the complications that the inner gradient is itself a high-variance estimate, so the meta-gradient compounds variance, and that differentiating through the sampling distribution needs care the supervised case never meets.

PEARL (Rakelly et al., Berkeley, 2019) is the field's cleanest factorization and directly prefigures the task-vector finding in language models. It separates task inference from control. An inference network \( q(z \mid c) \) encodes context transitions \( c = \{(s, a, r, s')\} \) from the current task into a posterior over a latent task embedding, structured as a product of per-transition Gaussian factors (so it is permutation-invariant and sharpens with evidence), and the policy and critic are ordinary soft actor-critic networks conditioned on \( z \). Training maximizes SAC objectives with a variational information-bottleneck term \( \KL\big( q(z \mid c) \,\|\, \N(0, I) \big) \), and because adaptation is inference rather than gradient steps, PEARL runs off-policy and is one to two orders of magnitude more sample efficient at meta-training than the on-policy methods it replaced. Its exploration story is posterior sampling. Sample \( z \) from the prior, act as if that task hypothesis were true, collect evidence, update, resample, a learned analogue of Thompson sampling.

Exploration is the specifically meta-RL difficulty. The inner loop can only adapt to what the pre-adaptation policy managed to observe, so meta-training must produce behavior good at gathering task-identifying information, which the plain MAML-RL objective does not reward, since credit for post-update returns flowing back to pre-update exploration is exactly the second-order path that first-order approximations sever. E-MAML (Stadie et al., 2018) adds that credit explicitly. PEARL and RL\(^2\) get exploration implicitly from posterior uncertainty or recurrent state. The same tension reappears in modern in-context RL. Algorithm distillation (Laskin et al., DeepMind, 2022) trains a transformer on entire RL learning histories so the forward pass reproduces improvement over episodes, RL\(^2\) rebuilt at scale with the recurrent core swapped for attention.

Worked problems

Problem 1 (PCGrad) appears in the gradient-conflict section above. The four here cover the sharing tradeoff numerically, the MAML meta-gradient by hand, the Reptile expansion, and the implicit-gradient and Fisher machinery.

Implementation

Three implementations appear below, PyTorch and JAX side by side. They are MAML with explicit second-order gradients on the classic sinusoid few-shot regression problem, a prototypical-network episode, and EWC. The sinusoid experiment was actually run on this machine (NVIDIA H100 80GB, PyTorch 2.7.0 with CUDA 12.8, JAX 0.6.0), and the numbers quoted below are from those runs, not from a paper.

MAML on sinusoid regression, with the second-order term

The task family is \( y = A \sin(x + \varphi) \) with amplitude \( A \sim U(0.1, 5) \) and phase \( \varphi \sim U(0, \pi) \). A task shows the learner ten support points. This is the problem from the original MAML paper, and it is chosen adversarially against joint training. The average of many sinusoids with random phase is close to zero everywhere, so a jointly trained network converges to a nearly flat function and ten points cannot rescue it, while a meta-trained initialization encodes the family and snaps to any member in a step or two. The functional APIs make the higher-order differentiation explicit. In PyTorch, torch.func.grad produces the inner gradient as a differentiable expression, so when loss.backward() later reaches it, autograd differentiates the differentiation, which is exactly the Hessian-vector product in the meta-gradient formula. In JAX, jax.grad of a function that itself calls jax.grad composes the same way. FOMAML in this style is one detach (or stop_gradient) on the inner gradient, which severs precisely the second-order path and nothing else. vmap batches the whole inner loop over the meta-batch of tasks.

import math, torch
from torch.func import functional_call, grad, vmap

dev = "cuda"
torch.manual_seed(0)

# Task family: y = A sin(x + phi), A ~ U(0.1, 5), phi ~ U(0, pi).
def sample_tasks(n_tasks, k_support, k_query):
    A   = torch.empty(n_tasks, 1, 1, device=dev).uniform_(0.1, 5.0)
    phi = torch.empty(n_tasks, 1, 1, device=dev).uniform_(0.0, math.pi)
    xs  = torch.empty(n_tasks, k_support, 1, device=dev).uniform_(-5.0, 5.0)
    xq  = torch.empty(n_tasks, k_query,   1, device=dev).uniform_(-5.0, 5.0)
    return xs, A * torch.sin(xs + phi), xq, A * torch.sin(xq + phi)

model = torch.nn.Sequential(                   # the classic MAML regressor
    torch.nn.Linear(1, 40), torch.nn.ReLU(),
    torch.nn.Linear(40, 40), torch.nn.ReLU(),
    torch.nn.Linear(40, 1)).to(dev)

def mse(params, x, y):                          # x: (K, 1), y: (K, 1)
    return ((functional_call(model, params, (x,)) - y) ** 2).mean()

INNER_LR, FIRST_ORDER = 0.01, False

def adapt(params, xs, ys):
    # grad() builds the inner step INSIDE the autograd graph, so
    # backpropagating through the result differentiates through the
    # update itself: that is where the Hessian term comes from.
    g = grad(mse)(params, xs, ys)
    if FIRST_ORDER:                             # FOMAML: cut the graph, drop
        g = {k: v.detach() for k, v in g.items()}   # the second-order term
    return {k: params[k] - INNER_LR * g[k] for k in params}

def task_loss(params, xs, ys, xq, yq):          # one task's meta-objective
    return mse(adapt(params, xs, ys), xq, yq)   # query loss AFTER adapting

# vmap over the task axis: params broadcast, data mapped
batched = vmap(task_loss, in_dims=(None, 0, 0, 0, 0))

params = dict(model.named_parameters())
opt = torch.optim.Adam(params.values(), lr=1e-3)
for it in range(15_000):
    xs, ys, xq, yq = sample_tasks(25, 10, 10)   # meta-batch of 25 tasks
    loss = batched(params, xs, ys, xq, yq).mean()
    opt.zero_grad(); loss.backward(); opt.step()

# Evaluation: 1000 held-out tasks, 10 support points, plain SGD at test.
def eval_adaptation(n_tasks=1000, k=10, steps=10):
    xs, ys, xq, yq = sample_tasks(n_tasks, k, 200)
    def one_task(xs, ys, xq, yq):
        p = {k_: v.detach() for k_, v in params.items()}
        out = []
        for s in range(steps + 1):
            out.append(mse(p, xq, yq))
            g = grad(mse)(p, xs, ys)
            p = {k_: p[k_] - INNER_LR * g[k_] for k_ in p}
        return torch.stack(out)
    return vmap(one_task)(xs, ys, xq, yq).mean(0)   # (steps+1,)

curve = eval_adaptation()
print(f"query MSE: pre-adapt {curve[0]:.3f}, "
      f"1 step {curve[1]:.3f}, 10 steps {curve[10]:.3f}")
# One run of this exact script on the H100:
#   query MSE: pre-adapt 3.095, 1 step 0.395, 10 steps 0.112

import math, time
import jax, jax.numpy as jnp
import optax

def init_mlp(key, sizes=(1, 40, 40, 1)):        # fan-in uniform init
    params = []
    for m, n in zip(sizes[:-1], sizes[1:]):
        key, sub = jax.random.split(key)
        kw, kb = jax.random.split(sub)
        lim = 1.0 / jnp.sqrt(m)
        params.append({
            "w": jax.random.uniform(kw, (m, n), minval=-lim, maxval=lim),
            "b": jax.random.uniform(kb, (n,),  minval=-lim, maxval=lim)})
    return params

def forward(params, x):                          # x: (K, 1)
    h = x
    for layer in params[:-1]:
        h = jax.nn.relu(h @ layer["w"] + layer["b"])
    return h @ params[-1]["w"] + params[-1]["b"]

def mse(params, x, y):
    return jnp.mean((forward(params, x) - y) ** 2)

INNER_LR = 0.01

def adapt(params, xs, ys):
    # jax.grad returns a differentiable function of params, so grads of
    # task_loss w.r.t. params flow THROUGH this update: second-order MAML.
    # FOMAML: wrap g in jax.lax.stop_gradient to sever that path.
    g = jax.grad(mse)(params, xs, ys)
    return jax.tree_util.tree_map(lambda p, gi: p - INNER_LR * gi, params, g)

def task_loss(params, xs, ys, xq, yq):
    return mse(adapt(params, xs, ys), xq, yq)

def meta_loss(params, xs, ys, xq, yq):
    per_task = jax.vmap(task_loss, in_axes=(None, 0, 0, 0, 0))
    return jnp.mean(per_task(params, xs, ys, xq, yq))

def sample_tasks(key, n, k):
    ka, kp, kx, kq = jax.random.split(key, 4)
    A   = jax.random.uniform(ka, (n, 1, 1), minval=0.1, maxval=5.0)
    phi = jax.random.uniform(kp, (n, 1, 1), minval=0.0, maxval=math.pi)
    xs  = jax.random.uniform(kx, (n, k, 1), minval=-5.0, maxval=5.0)
    xq  = jax.random.uniform(kq, (n, k, 1), minval=-5.0, maxval=5.0)
    return xs, A * jnp.sin(xs + phi), xq, A * jnp.sin(xq + phi)

key = jax.random.PRNGKey(0)
params = init_mlp(key)
opt = optax.adam(1e-3)
opt_state = opt.init(params)

@jax.jit
def step(params, opt_state, key):
    xs, ys, xq, yq = sample_tasks(key, 25, 10)
    loss, g = jax.value_and_grad(meta_loss)(params, xs, ys, xq, yq)
    updates, opt_state = opt.update(g, opt_state)
    return optax.apply_updates(params, updates), opt_state, loss

for it in range(15_000):
    key, sub = jax.random.split(key)
    params, opt_state, loss = step(params, opt_state, sub)

def eval_task(params, xs, ys, xq, yq, steps=10):
    losses, p = [mse(params, xq, yq)], params
    for _ in range(steps):
        p = adapt(p, xs, ys)
        losses.append(mse(p, xq, yq))
    return jnp.stack(losses)

xs, ys, xq, yq = sample_tasks(jax.random.PRNGKey(1), 1000, 10)
curve = jax.vmap(eval_task, in_axes=(None, 0, 0, 0, 0))(
    params, xs, ys, xq, yq).mean(0)
print("query MSE at 0/1/10 steps:", curve[0], curve[1], curve[10])
# One run on the H100: 20.2 s jit-compiled, MSE 2.979 / 0.411 / 0.138

The measured results, one training run per method, evaluated on 1000 held-out tasks with 10 support points and 200 query points, adapting with plain gradient descent at test time, follow.

The table is the theory made visible. All methods start at roughly the same pre-adaptation loss near 3.0, which is about the variance of the task family itself. Before seeing support points, no method can know the amplitude or phase, and a near-zero prediction is the best constant answer. One gradient step on ten points then separates them completely. The meta-trained initializations drop by a factor of about 7.5 (2.949 to 0.399), while joint training barely moves (2.936 to 2.328) and after ten steps is still an order of magnitude worse (1.544 versus 0.130), because its weights sit at the task-average solution from Problem 3, not at a point from which the family is reachable. FOMAML tracks full MAML within noise on this problem, matching the Reptile analysis in Problem 4. One honest footnote is that at this tiny scale, wall clock differences between MAML and FOMAML reflect implementation overheads, not Hessian-vector products. The second-order cost becomes binding at larger models and step counts, which is historically when everyone switched to first-order variants. A rerun with different seeds moves the third decimal, not the story.

A prototypical-network episode

One episode of prototypical-network training embeds support and query points, averages support embeddings per class into prototypes, classifies queries by softmax over negative squared distances, and returns the cross-entropy. This function is the entire algorithm. Meta-training is a loop that samples an N-way K-shot episode from the class pool and takes an optimizer step on this loss. The distance-to-linear equivalence from the derivation means the returned logits could equally be computed as \( 2 c_k \cdot z - \|c_k\|^2 \). The distance form is used because it is numerically direct.

import torch, torch.nn.functional as F

def proto_episode_loss(embed, x_support, y_support, x_query, y_query, n_way):
    # x_support: (N*K, ...) support inputs, y_support: (N*K,) in [0, N)
    # x_query:   (N*Q, ...) query inputs,   y_query:   (N*Q,)
    z_s = embed(x_support)                        # (N*K, D)
    z_q = embed(x_query)                          # (N*Q, D)
    D = z_s.shape[-1]
    # class prototypes: mean support embedding per class      (N, D)
    protos = torch.zeros(n_way, D, device=z_s.device)
    protos = protos.index_add_(0, y_support, z_s)
    counts = torch.bincount(y_support, minlength=n_way).unsqueeze(1)
    protos = protos / counts
    # squared Euclidean distance, query to prototype          (N*Q, N)
    d2 = torch.cdist(z_q, protos).pow(2)
    logits = -d2                                  # softmax over -distance
    loss = F.cross_entropy(logits, y_query)
    acc = (logits.argmax(-1) == y_query).float().mean()
    return loss, acc

# meta-training loop (sketch): sample an N-way K-shot episode from the
# training classes, call proto_episode_loss, step the optimizer. At
# meta-test time the SAME function runs on novel classes; only the
# episode sampler changes. No inner-loop optimization exists anywhere.

import jax, jax.numpy as jnp

def proto_episode_loss(embed_fn, params, x_s, y_s, x_q, y_q, n_way):
    z_s = embed_fn(params, x_s)                   # (N*K, D)
    z_q = embed_fn(params, x_q)                   # (N*Q, D)
    # class prototypes via one-hot matmul: mean per class      (N, D)
    onehot = jax.nn.one_hot(y_s, n_way)           # (N*K, N)
    protos = (onehot.T @ z_s) / onehot.sum(0)[:, None]
    # squared Euclidean distance, query to prototype           (N*Q, N)
    d2 = ((z_q[:, None, :] - protos[None, :, :]) ** 2).sum(-1)
    logits = -d2
    logp = jax.nn.log_softmax(logits, axis=-1)
    loss = -logp[jnp.arange(len(y_q)), y_q].mean()
    acc = (logits.argmax(-1) == y_q).mean()
    return loss, acc

# jax.grad(lambda p: proto_episode_loss(embed_fn, p, ...)[0]) gives the
# episode gradient; wrap the whole episode in jax.jit for speed. The
# one-hot matmul form of the prototype computation is used instead of a
# scatter so the function stays vmap- and jit-friendly.

EWC: Fisher accumulation and the quadratic anchor

There are two functions. One estimates the diagonal Fisher after finishing a task, sampling labels from the model's own predictive distribution as the derivation requires, and one computes the quadratic penalty added to the next task's loss. Both were run as part of a small two-task sanity check (rotated Gaussian classification). The point of the listing is the structure, in particular where the model's own samples enter and where the anchor parameters are frozen.

import torch, torch.nn.functional as F

def fisher_diagonal(model, loader, n_batches=100):
    # Diagonal Fisher at the current parameters. Labels are sampled from
    # the model's own predictive distribution (true Fisher), NOT taken
    # from the dataset (that would be the empirical Fisher).
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
    seen = 0
    for i, (x, _) in enumerate(loader):
        if i >= n_batches:
            break
        logits = model(x)                              # (B, C)
        y_hat = torch.distributions.Categorical(logits=logits).sample()
        ll = F.log_softmax(logits, -1)[torch.arange(len(y_hat)), y_hat].sum()
        model.zero_grad()
        ll.backward()                                  # d log p / d theta
        for n, p in model.named_parameters():
            fisher[n] += p.grad.pow(2)                 # squared score
        seen += len(y_hat)
    return {n: f / seen for n, f in fisher.items()}

def ewc_penalty(model, fisher, star, lam):
    # star: dict of anchor parameters theta*_A (detached clones).
    loss = 0.0
    for n, p in model.named_parameters():
        loss = loss + (fisher[n] * (p - star[n]).pow(2)).sum()
    return 0.5 * lam * loss

# usage after finishing task A:
#   fisher = fisher_diagonal(model, loader_A)
#   star = {n: p.detach().clone() for n, p in model.named_parameters()}
# while training task B:
#   loss = F.cross_entropy(model(x), y) + ewc_penalty(model, fisher, star, lam)

import jax, jax.numpy as jnp

def fisher_diagonal(apply_fn, params, x, key):
    # Per-example score gradients via vmap, then mean of squares.
    logits = apply_fn(params, x)                        # (B, C)
    y_hat = jax.random.categorical(key, logits)         # model's own labels

    def example_ll(p, xi, yi):                          # scalar log-lik
        return jax.nn.log_softmax(apply_fn(p, xi[None]))[0, yi]

    per_ex = jax.vmap(jax.grad(example_ll), in_axes=(None, 0, 0))
    scores = per_ex(params, x, y_hat)                   # pytree, leading B
    return jax.tree_util.tree_map(lambda g: (g ** 2).mean(0), scores)

def ewc_penalty(params, star, fisher, lam):
    sq = jax.tree_util.tree_map(
        lambda p, s, f: (f * (p - s) ** 2).sum(), params, star, fisher)
    return 0.5 * lam * sum(jax.tree_util.tree_leaves(sq))

# task B objective:
#   def loss_fn(params, x, y):
#       return nll(params, x, y) + ewc_penalty(params, star, fisher, lam)
# jax.grad(loss_fn) then flows through both terms; star and fisher are
# constants (no gradient), which JAX guarantees since they are inputs
# that loss_fn never differentiates with respect to.

How it is done in practice

The uncomfortable headline is that explicit meta-learning, the episodic bi-level machinery this page spent its middle third deriving, lost the mainstream to large-scale pretraining plus prompting or light fine-tuning, and the reasons were foreshadowed by results within meta-learning itself. ANIL showed MAML's power was mostly representation quality, the baseline papers showed a well-trained embedding plus a trivial head beats sophisticated episodic training, and pretraining at web scale produces representations, and in the language case in-context adaptation procedures, that no episodic run on a few hundred classes can approach. The bi-level outer loop is also expensive per unit of data seen, with second-order gradients, small episodes, and task-sampling infrastructure, against the ruthless simplicity of next-token prediction on everything. The explicit version survives where scale's task distribution does not cover the deployment distribution.

Those niches are real and specific. The first is few-shot learning under genuine distribution shift. Meta-Dataset-style evaluation shows pretrained-then-probe degrades when test tasks (satellite imagery, medical slides, industrial defects) sit far from the pretraining distribution, and adaptive methods recover some of the gap. The second is robotics with expensive data. Adaptation to a payload change, motor wear, or a new terrain must happen from seconds of experience, and meta-trained dynamics models and PEARL-style task inference are among the few approaches with demonstrated real-robot wins (Nagabandi et al.'s online adaptation of legged-robot dynamics). The third is personalization. Adapting a shared model to each user from a small data stream, on-device keyboards and cold-start recommendation, is Reptile-shaped, and federated personalization methods are explicit about the connection. The fourth is hyperparameter and optimizer adaptation. Learned optimizers (Google's VeLO, trained across thousands of tasks) and gradient-based hyperparameter optimization are bi-level optimization deployed as engineering tools, with iMAML's implicit-gradient machinery doing the differentiation.

Multi-task learning, unlike meta-learning, never left production. It consolidated. Industrial ranking systems are the largest deployment, multi-objective heads (click, watch, like, purchase) over shared-plus-expert trunks in the MMoE style, with loss weighting tuned as a first-class product decision because the weights literally encode business tradeoffs. Perception stacks in driving and robotics run one backbone with a dozen heads because the on-vehicle compute budget forces hard sharing, and task-weighting plus occasional gradient surgery is the daily toolkit. Speech and translation went massively multi-task (Whisper's joint transcription-translation-language-ID training, and multilingual translation balancing hundreds of language pairs with temperature-scaled sampling, loss weighting under another name). And the pretrain-then-adapt paradigm inherited meta-learning's formalism wholesale. A foundation model is meta-parameters \( \theta \), LoRA and adapters are a parameterized \( \text{Adapt}_\theta \) with hand-chosen rather than meta-learned structure, instruction tuning on many tasks (FLAN, and MetaICL's explicit few-shot episodes built from task collections) is the outer loop, and few-shot prompting on held-out tasks is, protocol for protocol, meta-testing with support and query sets. The vocabulary changed, but the objective did not.

On evaluation, the field's self-corrections are worth internalizing because they generalize. Omniglot (Lake et al., 2015: 1623 characters, 20 drawings each) saturated years ago. miniImageNet results were shown by Chen et al. to be dominated by backbone choice rather than algorithm, and Meta-Dataset (Triantafillou et al., 2020), spanning ten image sources with variable way and shot per episode, reshuffled the rankings again, with centroid methods and fine-tuning-from-pretraining strong once given fair backbones. Meta-overfitting is the phenomenon behind the reshuffles. An episodic learner can overfit the meta-training task distribution itself, learning regularities of how episodes are constructed (class balance, way, shot, image statistics) that do not transfer to differently constructed episodes. The general lessons are to fix the backbone before comparing adaptation methods, evaluate across task distributions rather than within one, and treat any few-shot leaderboard whose baseline is not a tuned pretrained-plus-linear-probe with suspicion.

The current research frontier

The most active front is the science of in-context learning. The mechanistic line (Anthropic's induction-head program and its successors) is working upward from circuits toward claims about production-scale models, the theory line is broadening the class of algorithms transformers provably implement in context, from gradient descent (von Oswald et al.) through ridge regression and Bayesian model averaging (Akyürek et al. and others), and the function-vector line (Tel Aviv, Northeastern) is turning the infer-then-apply factorization into an editing tool, composing function vectors to steer behavior without prompts. The dispute between task learning and task retrieval remains unresolved for natural-language tasks at scale.

Elsewhere, in-context RL is being scaled (DeepMind's algorithm distillation and the AdA adaptive-agent work showing human-timescale adaptation in an open-ended 3D task space). Prior-fitted networks are exporting amortized inference to tabular data (TabPFN, now with widely reported strong small-data results). Learned optimizers continue at Google with mixed adoption because generalization across architectures remains fragile. Continual learning is being reframed around large models, where the questions become forgetting during fine-tuning, model merging as a continual mechanism, and replay via synthetic data from the model itself. Parameter-efficient adaptation is converging with modular multi-task learning, mixtures of LoRA experts with per-task routing (Microsoft Research, Cohere, and groups at Edinburgh and EPFL) rebuilding MMoE inside frozen foundation models. In multi-task optimization the benchmark-skeptical line has largely won the framing battle. New methods are now expected to beat tuned scalarization, and few do by much.

Open source to read

learnables/learn2learn is the most complete PyTorch meta-learning library, with MAML, ANIL, prototypical networks, Reptile, and meta-RL under one API. Open learn2learn/algorithms/maml.py first. The MAML.adapt method is the differentiable inner update, and the first_order flag is a one-line view of exactly what FOMAML discards.

facebookresearch/higher is Meta AI's library for making arbitrary PyTorch modules and optimizers differentiable, the pre-torch.func way to write second-order MAML. Open examples/maml-omniglot.py, where thirty lines around the higher.innerloop_ctx context manager show the unrolled inner loop and the meta-gradient flowing through a patched optimizer.

tristandeleu/pytorch-meta offers clean episodic data loading for Omniglot, miniImageNet, and friends, plus modular meta-network components. Open torchmeta/utils/gradient_based.py, where gradient_update_parameters is the MAML inner step in one readable function, with the create_graph flag deciding first- versus second-order.

google-research/meta-dataset is the benchmark that made few-shot evaluation honest. Open meta_dataset/data/sampling.py to see how realistic episodes are constructed, variable way, variable shot, class imbalance, which is half of what makes the benchmark hard.

google-deepmind/optax provides JAX optimizers as pure, composable gradient transformations, which is what makes differentiating through an optimizer natural in JAX. Open optax/_src/alias.py to see optimizers assembled as pure functions. Because updates are just function applications on pytrees, wrapping an inner optimizer in jax.grad requires no patching at all, which is the machinery the JAX MAML above relies on.

huggingface/peft is the production form of modular adaptation, LoRA, adapters, prefix and prompt tuning behind one interface. Open src/peft/tuners/lora/layer.py to see the low-rank update \( W + BA \) in code, the direct descendant of the adapter idea in the multi-task architecture section.

ContinualAI/avalanche is the standard continual-learning framework, with strategies, benchmarks, and the evaluation protocols whose absence made the early literature incomparable. Open avalanche/training/plugins/ewc.py, where the Fisher computation and penalty match the derivation above line for line, including the choice between separate and online (accumulated) penalties.

Common misconceptions

"More tasks always help a multi-task model." The pooling inequality at the top of the page says otherwise. Sharing trades variance for bias, and the bias does not shrink with data. Taskonomy and Standley et al. measured the effect at scale. Transfer is structured, asymmetric, and sometimes negative, and the best grouping of tasks routinely beats training everything together.

"MAML learns weights that are good for all tasks." It learns weights that are good one adaptation step later, which is a different point. Problem 3 exhibits the gap numerically. The joint-training optimum sits at the task-weighted average (\( -0.667 \) in that example) while the MAML optimum sits elsewhere (\( 0.118 \)), and the sinusoid experiment shows the same divergence at network scale, a joint model stuck at MSE 1.5 where the meta-trained model reaches 0.13 from the same amount of test data.

"The second-order term in MAML is negligible, so FOMAML is the same algorithm." FOMAML often matches on benchmarks, and the Reptile expansion explains why the first-order methods keep the gradient-alignment pressure on average. But the Hessian factor \( I - \alpha \nabla^2 \L \) reweights tasks by curvature and can even reverse the meta-update's sign, as Problem 3 shows with two quadratics. The approximation is usually benign and occasionally load-bearing.

"Meta-learning methods beat simple baselines at few-shot classification." On the standard benchmarks, mostly they do not. Chen et al. and Tian et al. showed that a plain pretrained embedding with a linear head fit on the support set matches or beats MAML, matching, prototypical, and relation networks under fair backbones, and ANIL explains why. The episodic machinery was mostly buying representation quality, which ordinary pretraining buys cheaper. The adaptive methods earn their keep only under real distribution shift between meta-train and meta-test.

"EWC computes how important each weight is to the old task." It computes a local, diagonal, quadratic approximation to the old posterior around one mode, with the Fisher standing in for the Hessian, and each adjective is a failure mode, wrong far from the mode, blind to parameter interactions, over-constraining across many tasks. This is why a small replay buffer, which optimizes the true joint objective directly, so often embarrasses it.

"In-context learning is just retrieval from pretraining data." Not just retrieval. Transformers trained from scratch on synthetic function families in-context-learn functions never seen in training, matching least squares on linear regression, and the constructive results show attention can implement optimization steps. But the pure learning story is also wrong for LLMs. Randomized demonstration labels often barely hurt, so much of prompting is task location rather than task learning. The truth is a mixture with unknown, task-dependent proportions.

"Continual-learning numbers are comparable across papers." Only within a fixed scenario. Task-, domain-, and class-incremental protocols differ in what the model is told at test time, and the same method can go from state-of-the-art to near-chance between them. Buffer sizes and head structure move results as much as algorithms do. A claim without its scenario and buffer budget attached is an anecdote, not a result.

Self-check

References

1. Murphy (2023). Probabilistic Machine Learning: Advanced Topics, MIT Press. Hierarchical Bayes and meta-learning chapters. probml.github.io
2. Caruana (1997). Multitask Learning. Machine Learning 28. doi:10.1023/A:1007379606734
3. Ruder (2017). An Overview of Multi-Task Learning in Deep Neural Networks. arXiv:1706.05098
4. Kendall, Gal, Cipolla (2018). Multi-Task Learning Using Uncertainty to Weigh Losses. CVPR. arXiv:1705.07115
5. Chen, Badrinarayanan, Lee, Rabinovich (2018). GradNorm. ICML. arXiv:1711.02257
6. Yu, Kumar, Gupta, Levine, Hausman, Finn (2020). Gradient Surgery for Multi-Task Learning. NeurIPS. arXiv:2001.06782
7. Standley, Zamir, Chen, Guibas, Malik, Savarese (2020). Which Tasks Should Be Learned Together in Multi-task Learning? ICML. arXiv:1905.07553
8. Kurin, De Palma, Kostrikov, Whiteson, Mudigonda (2022). In Defense of the Unitary Scalarization for Deep Multi-Task Learning. NeurIPS. arXiv:2201.04122
9. Ben-David, Blitzer, Crammer, Kulesza, Pereira, Vaughan (2010). A Theory of Learning from Different Domains. Machine Learning 79. doi:10.1007/s10994-009-5152-4
10. Ganin, Ustinova, Ajakan, Germain, Larochelle, Laviolette, Marchand, Lempitsky (2016). Domain-Adversarial Training of Neural Networks. JMLR. arXiv:1505.07818
11. Finn, Abbeel, Levine (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML. arXiv:1703.03400
12. Nichol, Achiam, Schulman (2018). On First-Order Meta-Learning Algorithms. arXiv:1803.02999
13. Raghu, Raghu, Bengio, Vinyals (2020). Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. ICLR. arXiv:1909.09157
14. Rajeswaran, Finn, Kakade, Levine (2019). Meta-Learning with Implicit Gradients. NeurIPS. arXiv:1909.04630
15. Grant, Finn, Levine, Darrell, Griffiths (2018). Recasting Gradient-Based Meta-Learning as Hierarchical Bayes. ICLR. arXiv:1801.08930
16. Vinyals, Blundell, Lillicrap, Kavukcuoglu, Wierstra (2016). Matching Networks for One Shot Learning. NeurIPS. arXiv:1606.04080. Snell, Swersky, Zemel (2017). Prototypical Networks for Few-shot Learning. NeurIPS. arXiv:1703.05175
17. Chen, Liu, Kira, Wang, Huang (2019). A Closer Look at Few-shot Classification. ICLR. arXiv:1904.04232. Tian, Wang, Krishnan, Tenenbaum, Isola (2020). Rethinking Few-Shot Image Classification: A Good Embedding Is All You Need? ECCV. arXiv:2003.11539
18. Hospedales, Antoniou, Micaelli, Storkey (2021). Meta-Learning in Neural Networks: A Survey. TPAMI. arXiv:2004.05439
19. Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel (2016). RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv:1611.02779. Rakelly, Zhou, Quillen, Finn, Levine (2019). Efficient Off-Policy Meta-RL via Probabilistic Context Variables (PEARL). ICML. arXiv:1903.08254
20. Kirkpatrick et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. PNAS. arXiv:1612.00796. Zenke, Poole, Ganguli (2017). Continual Learning Through Synaptic Intelligence. ICML. arXiv:1703.04200
21. Brown et al. (2020). Language Models are Few-Shot Learners. NeurIPS. arXiv:2005.14165
22. Olsson et al. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread. transformer-circuits.pub
23. von Oswald et al. (2023). Transformers Learn In-Context by Gradient Descent. ICML. arXiv:2212.07677. Garg, Tsipras, Liang, Valiant (2022). What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. NeurIPS. arXiv:2208.01066
24. Hendel, Geva, Globerson (2023). In-Context Learning Creates Task Vectors. Findings of EMNLP. arXiv:2310.15916. Todd et al. (2024). Function Vectors in Large Language Models. ICLR. arXiv:2310.15213
25. Triantafillou et al. (2020). Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. ICLR. arXiv:1903.03096