Adversarial generative modeling, from minimax games to stability and image translation

Why this subject matters now

When Goodfellow and coauthors introduced generative adversarial networks in 2014, the idea was startling for a reason that is easy to forget now. They trained a generative model of natural images without ever writing down, evaluating, or lower-bounding a likelihood. Every other generative family on the deep generative models page anchors on a density. Autoregressive models and normalizing flows compute \( \log p_\theta(x) \) exactly, a variational autoencoder bounds it, an energy-based model gives it up to a constant. A GAN gives none of these. It defines only a sampler, \( x = G_\theta(z) \) with \( z \sim \N(0,I) \), and learns by playing a game against a classifier. That the resulting samples were sharp, at a time when likelihood models produced blurry thumbnails, forced the field to take likelihood-free training seriously.

For roughly five years, from DCGAN in 2016 through StyleGAN2 and BigGAN around 2020, GANs were the state of the art in image synthesis by a wide margin. Then diffusion models caught and passed them on fidelity and, more importantly, on the thing GANs never had, stable training with a simple regression loss. By 2022 the frontier of text-to-image generation had moved almost entirely to diffusion and flow matching, and it became common to hear that GANs were finished. That verdict is wrong, and understanding why is the practical payoff of this page. A GAN generates in a single forward pass. A diffusion model needs tens to thousands. When latency is the constraint, in real-time super-resolution, in on-device generation, in the final distillation step that makes a diffusion model fast enough to ship, the adversarial loss is exactly what comes back. The headline systems of 2023 onward that turn a slow diffusion teacher into a one-step or few-step student, adversarial diffusion distillation among them, are GAN training in a new costume. A practitioner who dismissed the adversarial objective missed the mechanism that now makes diffusion deployable.

What a practitioner is expected to know has also sharpened. Five years ago it was enough to run a DCGAN and hope. Today the expectation is that you can explain, from the objective, why a given run collapsed, and whether the fix is a different divergence (Wasserstein), a different constraint on the critic (gradient penalty, spectral normalization, R1), or a different architecture (a style-based generator). You are also expected to read the StyleGAN latent space as a learned disentanglement rather than magic, and to compute and critique an FID number rather than quote it. This page builds that fluency by deriving each piece and checking the arithmetic.

The minimax game and the optimal discriminator

Fix a data distribution \( p_{\text{data}} \) over \( \mathcal{X} = \R^d \). The generator is a differentiable map \( G_\theta : \R^k \to \R^d \) that pushes a fixed noise distribution \( p_z \) (typically \( \N(0, I_k) \)) forward to an implicit distribution \( p_g \) over \( \mathcal{X} \). To sample from \( p_g \) you draw \( z \sim p_z \) and return \( G_\theta(z) \). You cannot evaluate \( p_g(x) \) at a point, because that would require integrating over all \( z \) that map to \( x \) with the change-of-variables Jacobian, which for a generic non-invertible network is intractable. The discriminator is a map \( D_\phi : \mathcal{X} \to (0,1) \) meant to output the probability that its input is real. The value function is

and the game is the saddle-point problem \( \min_G \max_D V(D, G) \). Read the value function as a binary cross-entropy. Label real data \( y=1 \) and generator samples \( y=0 \), and \( V \) is then exactly the log-likelihood of the correct labels under the classifier \( D \). The discriminator wants to raise it, the generator wants to lower it by making its samples indistinguishable from real ones.

Solving the inner maximization in closed form

Rewrite the second expectation as an integral over \( \mathcal{X} \) using the pushforward. For any function \( h \), \( \E_{z}[h(G(z))] = \E_{x \sim p_g}[h(x)] = \int p_g(x)\, h(x)\, dx \). The value function becomes a single integral,

For a fixed \( G \) the integrand at each \( x \) depends on \( D \) only through the scalar value \( D(x) \), and the values at different \( x \) do not interact. Maximizing the integral therefore reduces to maximizing the integrand pointwise. Write \( a = p_{\text{data}}(x) \), \( b = p_g(x) \), \( t = D(x) \in (0,1) \), and maximize \( f(t) = a \log t + b \log(1-t) \). Differentiating gives

The second derivative \( f''(t) = -a/t^2 - b/(1-t)^2 < 0 \) is negative wherever \( a, b \geq 0 \) are not both zero, so this is a maximum. The optimal discriminator is the pointwise density ratio,

which is worth pausing on. The Bayes-optimal classifier between two classes with equal priors is exactly the posterior probability of the real class, and that posterior is the density ratio above. So the discriminator is not learning an arbitrary boundary. At optimality it is estimating \( p_{\text{data}}/(p_{\text{data}}+p_g) \), which is monotone in the likelihood ratio \( p_{\text{data}}/p_g \). This is the same density-ratio trick that underlies noise-contrastive estimation, with the crucial difference that here the "noise" distribution \( p_g \) is itself being trained to be adversarial. Where \( p_{\text{data}} = p_g \), the ratio is \( 1/2 \). A perfect generator drives the discriminator to maximal confusion, output \( 1/2 \) everywhere.

What the generator then minimizes, the Jensen-Shannon divergence

Substitute \( D^\ast_G \) back into \( V \) to get the function of \( G \) alone that the outer minimization sees, and call it \( C(G) = V(D^\ast_G, G) \). Then

The two logarithms are close to KL divergences but the denominator is \( p_{\text{data}}+p_g \), not a normalized distribution. Fix that by inserting a factor of two inside each log, which we compensate by subtracting \( \log 2 \) once per term. Write \( m = \tfrac{1}{2}(p_{\text{data}} + p_g) \), the equal mixture, which is a genuine probability distribution. Then \( \dfrac{p_{\text{data}}}{p_{\text{data}}+p_g} = \dfrac{1}{2}\cdot\dfrac{p_{\text{data}}}{m} \), so

and identically for the second term with \( p_g \) in place of \( p_{\text{data}} \). Adding them,

The Jensen-Shannon divergence is defined as exactly the average of those two KL terms, \( \mathrm{JSD}(p\|q) = \tfrac{1}{2}\KL(p\|m) + \tfrac{1}{2}\KL(q\|m) \) with \( m = \tfrac12(p+q) \), so

because \( 2\log 2 = \log 4 \). The Jensen-Shannon divergence is non-negative and zero only when its arguments coincide, so \( C(G) \geq -\log 4 \) with equality if and only if \( p_g = p_{\text{data}} \). The unique global optimum of the game is the generator that reproduces the data distribution exactly, at which point \( C(G) = -\log 4 \approx -1.386 \) and the discriminator outputs \( 1/2 \) everywhere. This is the central theoretical fact about GANs. At discriminator optimality, the generator minimizes a symmetric, bounded divergence between the data and model distributions, estimated entirely from samples with no density in sight. Every worked number below, and the entire first problem, checks this identity numerically.

Why the minimax loss saturates, and the non-saturating fix

The clean derivation above assumes the discriminator is optimal. Early in training it very nearly is, and that is a problem for the generator, not a help. The generator's contribution to the loss is \( \E_z[\log(1 - D(G(z)))] \), which it wants to minimize by pushing \( D(G(z)) \) toward 1. Consider the gradient signal it receives. Let \( d = D(G(z)) \in (0,1) \) be the discriminator's score on a fake sample. The generator's per-sample loss is \( \ell_{\text{mm}}(d) = \log(1-d) \), and its derivative with respect to \( d \) is

When the generator is doing badly, its samples are obvious fakes and the discriminator confidently outputs \( d \to 0 \). There the gradient magnitude is \( 1/(1-d) \to 1 \), so the signal is bounded and small at exactly the moment the generator most needs a strong push. The loss \( \log(1-d) \) is nearly flat near \( d = 0 \). It saturates. Meanwhile the useful gradient, the one that would move the samples toward the data, is throttled. This is the practical failure Goodfellow flagged in the original paper and again in the 2016 tutorial. Minimizing \( \log(1-D(G(z))) \) gives vanishing gradients whenever the discriminator is winning, which is most of early training.

The fix used in essentially every real GAN is to change the generator's loss without changing the discriminator's. Instead of minimizing \( \log(1-D(G(z))) \), the generator maximizes \( \log D(G(z)) \), equivalently minimizes \( \ell_{\text{ns}}(d) = -\log d \). This is the "non-saturating" heuristic. Its per-sample gradient is

which blows up as \( d \to 0 \). When the generator is losing badly, it now receives a large corrective gradient rather than a vanishing one. Both losses share the same fixed point, the generator wants \( d \) large in either case, and both push in the same direction, but their behavior at \( d \approx 0 \) is opposite. The numbers make this concrete. At \( d = 0.5 \) (a confused discriminator) the two gradients are equal in magnitude, both \( 2 \). At \( d = 0.1 \) the minimax gradient is \( 1/0.9 \approx 1.11 \) while the non-saturating one is \( 1/0.1 = 10 \), nine times larger. At \( d = 0.02 \) the minimax gradient is \( 1.02 \) while the non-saturating one is \( 50 \), a factor of \( 49 \). Problem 4 reproduces these exact figures. The non-saturating loss no longer minimizes the clean \( 2\,\mathrm{JSD} - \log 4 \). It minimizes a different combination of KL terms that Arjovsky and Bottou (2017) analyzed and showed has its own pathologies, unbounded variance and a reverse-KL-like mode-seeking bias, but in exchange it trains.

Non-convergence and mode collapse

The elegant divergence story assumes two things that never both hold, that the inner maximization runs to completion at every generator step, and that \( G \) and \( D \) have unlimited capacity. In practice we alternate a few discriminator steps with a generator step, on finite batches, with finite networks. The object being optimized is then not a fixed function of \( G \) but a moving target, and the dynamics of a two-player game are qualitatively unlike the dynamics of minimizing a fixed loss. A single loss surface has gradient flow that decreases the loss monotonically and settles at a local minimum. A game has no such guarantee. The two players' gradients can point in a rotational, not a descending, direction, and simultaneous updates can orbit a fixed point forever or spiral away from it.

A toy example of oscillation

The cleanest illustration strips away the networks. Consider the scalar bilinear game

whose unique saddle point is the origin \( (0,0) \). The gradients are \( \partial_x f = y \) and \( \partial_y f = x \). Run the natural simultaneous update, where \( x \) descends and \( y \) ascends, both using the value at the current point,

This is a linear map. Its effect on the squared radius \( r_t^2 = x_t^2 + y_t^2 \) is

where the cross terms \( -2\eta x_t y_t \) and \( +2\eta x_t y_t \) cancel exactly. So every step multiplies the distance from the saddle by \( \sqrt{1+\eta^2} > 1 \). The iterate spirals outward no matter how small the learning rate. The game is not just failing to converge, it is actively diverging. With \( \eta = 0.1 \) the per-step multiplier is \( \sqrt{1.01} \approx 1.004988 \), and over 200 steps the radius grows by \( 1.004988^{200} \approx 2.705 \). Problem 5 runs this and confirms the radius grows from \( \sqrt{2} \approx 1.414 \) to \( 3.825 \), matching the prediction to four figures. The lesson transfers directly. Near a GAN equilibrium the interaction between \( G \) and \( D \) has a rotational component, and naive simultaneous gradient updates inherit the same instability. Fixes in the literature all damp the rotation, whether by making the update prescient (extragradient, optimistic gradient), by penalizing the discriminator's gradient (R1 regularization, Mescheder et al. 2018, which provably restores local convergence), or by changing the metric entirely (Wasserstein).

Mode collapse

Mode collapse is the other characteristic failure, and it is a different phenomenon from non-convergence though the two often appear together. The generator maps many, sometimes all, of its noise inputs to a small set of outputs, ignoring whole regions of the data distribution. A face generator produces one convincing face for every \( z \). A digit generator produces only threes and eights. Nothing in the value function directly forbids this. Return to the optimal-discriminator picture. The generator is minimizing a divergence to \( p_{\text{data}} \), and the Jensen-Shannon divergence does penalize missing modes. But that penalty is only felt through the discriminator, and a discriminator trained on finite batches cannot detect that a mode is under-represented if the generator simply never proposes samples there to be caught. Worse, with the non-saturating loss the generator is effectively minimizing a reverse-KL-flavored objective, and reverse KL is mode-seeking. It is happy to place all mass on one high-density region and pay nothing for the modes it drops, because reverse KL only penalizes putting mass where the data has none, not the reverse.

There is a game-theoretic reading too. If the discriminator is held fixed for several generator steps, the generator's best response is to collapse onto the single point the current discriminator scores highest, \( \argmax_x D(x) \). The discriminator then adapts to reject that point, and the generator hops to the next-best point. The result is a generator that cycles through modes one at a time rather than covering them all simultaneously, which is precisely what mode-collapse videos show. Countermeasures attack exactly this. Minibatch discrimination and minibatch standard deviation (Salimans et al. 2016, used in Progressive Growing) let the discriminator see batch-level statistics so a collapsed batch is detectable. Unrolled GANs let the generator anticipate the discriminator's response. And the Wasserstein objective, discussed next, gives a loss that keeps providing gradients even when the supports of \( p_g \) and \( p_{\text{data}} \) barely overlap, which is the regime where JSD-based training stalls.

The Wasserstein GAN

Arjovsky and Bottou (2017) made a sharp diagnosis of why GAN training is unstable, and Arjovsky, Chintala, and Bottou (2017) turned it into a fix. The diagnosis is that \( p_{\text{data}} \) is supported on a low-dimensional manifold in a high-dimensional pixel space, and \( p_g \), the pushforward of a \( k \)-dimensional Gaussian through \( G \), is supported on another such manifold. Two manifolds of dimension much less than \( d \) generically do not overlap. Their intersection has measure zero. On disjoint supports the Jensen-Shannon divergence is pinned at its maximum \( \log 2 \) and, crucially, is locally constant. Nudging \( p_g \) closer to \( p_{\text{data}} \) does not change JSD at all until the supports touch. A constant loss has zero gradient. This is the deep reason a well-trained discriminator kills the generator's learning signal, and no amount of the non-saturating trick fully repairs it, because the underlying divergence itself has no useful slope.

The Earth-Mover distance

The remedy is to measure the distance between distributions with a metric that varies smoothly even when supports are disjoint. The Wasserstein-1 distance, also called the Earth-Mover distance, does this. Informally, if each distribution is a pile of dirt, the distance is the minimum total work, mass times the distance it is moved, to reshape one pile into the other. Formally, with \( \Pi(p_{\text{data}}, p_g) \) the set of all joint distributions (couplings) \( \gamma(x,y) \) whose marginals are \( p_{\text{data}} \) and \( p_g \),

The coupling \( \gamma(x,y) \) is a transport plan, how much mass to move from \( x \) to \( y \). Unlike the KL and JS divergences, \( W_1 \) is finite and continuous even for disjoint supports, and its value decreases smoothly as the supports approach each other. On the canonical example where \( p_0 \) is a point mass at the origin and \( p_\theta \) is a point mass at \( \theta \) on a line, \( \mathrm{JSD} \) is \( \log 2 \) for every \( \theta \neq 0 \) and drops discontinuously to \( 0 \) at \( \theta = 0 \), giving no gradient anywhere, while \( W_1 = |\theta| \), which has gradient \( \mathrm{sign}(\theta) \) everywhere and points straight at the optimum. That difference is the whole argument for Wasserstein GANs.

Kantorovich-Rubinstein duality, stated precisely

The infimum over couplings is intractable to optimize directly, so WGAN uses its dual. The Kantorovich-Rubinstein theorem states that for the Wasserstein-1 distance with the Euclidean ground metric,

where the supremum is over all functions \( f : \mathcal{X} \to \R \) that are 1-Lipschitz, meaning \( |f(x) - f(y)| \leq \lVert x - y \rVert \) for all \( x, y \), written \( \lVert f \rVert_L \leq 1 \). The duality is exact. The primal infimum over transport plans equals this dual supremum over 1-Lipschitz functions. There is a short intuition for why the Lipschitz constraint appears. The dual variable \( f \) is a "potential" whose slope cannot exceed one, because moving a unit of mass a unit of distance costs exactly one unit in the primal, and the dual price of transport cannot exceed that cost or the coupling would route around it. The function \( f \) is called the critic rather than the discriminator, because it no longer outputs a probability in \( (0,1) \). It outputs an unbounded real score, high for real, low for fake, and the difference of its means is the estimated Wasserstein distance.

The WGAN objective follows immediately. Parametrize the critic as a network \( f_w \) constrained to be 1-Lipschitz, and train

The inner maximization estimates \( W_1(p_{\text{data}}, p_g) \), and the generator minimizes that estimate. Because \( W_1 \) is continuous and differentiable almost everywhere in the generator's parameters, the critic can be trained close to optimality without destroying the generator's gradient, the exact opposite of the JS-based discriminator. In fact WGAN recommends training the critic more, several critic steps per generator step, which would be self-defeating in a standard GAN.

Enforcing the Lipschitz constraint by clipping, then by gradient penalty

The entire difficulty of WGAN is enforcing \( \lVert f_w \rVert_L \leq 1 \). The original paper used weight clipping. After each update, clamp every weight into \( [-c, c] \) for a small \( c \). This bounds the Lipschitz constant crudely, since the Lipschitz constant of a composition is at most the product of the operator norms of the layers, and clipping bounds those norms. But it is a blunt instrument. Arjovsky himself noted the pathologies. If \( c \) is too large the constraint is not enforced and if too small the gradients vanish through many layers, and clipping pushes weights to the two extremes \( \pm c \), throwing away critic capacity and biasing it toward very simple functions. WGAN with weight clipping trains, but the weight histogram collapses to two spikes and the critic underfits.

Gulrajani et al. (2017) replaced clipping with a soft constraint, WGAN-GP, derived from a property of the optimal critic. The key fact is that a differentiable function is 1-Lipschitz if and only if its gradient has norm at most one everywhere, \( \lVert \nabla_x f(x) \rVert \leq 1 \). Moreover the optimal WGAN critic has gradient norm exactly one on the transport geodesics between the two distributions, a consequence of the optimal-transport structure. So instead of clipping weights, penalize the critic whenever its gradient norm departs from one. The penalty is evaluated not on all of \( \mathcal{X} \), which is intractable, but on samples \( \hat{x} \) drawn along straight lines between real and fake points, \( \hat{x} = \epsilon\, x_{\text{real}} + (1-\epsilon)\, x_{\text{fake}} \) with \( \epsilon \sim U[0,1] \), because those interpolants concentrate near the geodesics that matter. The full critic loss is

with \( \lambda = 10 \) the value the paper found robust across architectures. The penalty is two-sided. It pushes the gradient norm toward one from both above and below, which the authors found worked better than a one-sided \( \max(0, \lVert \nabla \rVert - 1)^2 \). WGAN-GP removed the need for weight clipping, trained stably across a range of architectures where standard GANs failed, including deep residual critics, and was for years the default recipe for stable adversarial training. Its cost is a second backward pass, because the penalty is a gradient of a gradient. The implementation section shows how to compute it with a double backward in both PyTorch and JAX. The measured cost is modest. A full WGAN-GP discriminator plus generator step on a 784-dimensional MLP with batch 256 runs in 1.914 ms on an NVIDIA H100 80GB, most of which is the double backward through the penalty.

Spectral normalization

Miyato et al. (2018) enforce the Lipschitz constraint a third way, cheaper than a gradient penalty and applicable to any GAN, not just Wasserstein. The idea is to control the Lipschitz constant of the network directly through its weights. For a linear layer \( x \mapsto Wx \) the Lipschitz constant with respect to the Euclidean norm is exactly the spectral norm of \( W \), its largest singular value,

The Lipschitz constant of a feedforward network is bounded by the product of the spectral norms of its linear layers times the Lipschitz constants of its activations, and common activations (ReLU, leaky ReLU) are 1-Lipschitz. So if every weight matrix is rescaled to have spectral norm one, \( W_{\text{SN}} = W / \sigma(W) \), the whole network is 1-Lipschitz (up to the activations, which do not increase it). Spectral normalization applies this rescaling to every layer of the discriminator at every forward pass.

Estimating the spectral norm by power iteration

Computing \( \sigma(W) \) with a full singular value decomposition every forward pass would be far too expensive. Power iteration estimates only the largest singular value, at the cost of two matrix-vector products per step, and one step per forward pass suffices because the weights change slowly. The method rests on a simple fact. The largest singular value of \( W \) is the square root of the largest eigenvalue of \( W\T W \), and power iteration on \( W\T W \) converges to its top eigenvector. Written in terms of the left and right singular vectors \( u, v \) (so \( Wv = \sigma u \) and \( W\T u = \sigma v \) at the top pair), the iteration alternates

To see why this converges, expand the current estimate of \( v \) in the right singular vectors \( \{v_i\} \) of \( W \) with singular values \( \sigma_1 > \sigma_2 \geq \dots \). One round of the map \( v \mapsto W\T W v \) multiplies the coefficient of \( v_i \) by \( \sigma_i^2 \). After \( t \) rounds the ratio of the second coefficient to the first is scaled by \( (\sigma_2/\sigma_1)^{2t} \), which decays geometrically. The estimate therefore converges to \( \sigma_1 \) at a linear rate governed by the gap \( \sigma_2/\sigma_1 \). In a convolutional or linear GAN layer the top singular value is usually well separated, so a handful of iterations, or even one carried across training steps, tracks it closely. On the small worked example \( W = \big[\begin{smallmatrix}3&1&0\\0&2&1\\1&0&1\end{smallmatrix}\big] \), power iteration from a random start reaches the true top singular value \( 3.42479 \) to five decimals in seven iterations (\( 2.8779, 3.3989, 3.4218, 3.4244, 3.4247, 3.4248, 3.4248 \)). The SVD confirms \( \sigma_{\max} = 3.4247893 \). Problem 3 works this through and analyzes the rate. Because spectral normalization needs only these two matrix-vector products, it adds negligible cost, which is why it became standard in large GANs including BigGAN and SN-GAN, and appears in the discriminators of many diffusion decoders as well.

Spectral normalization and the gradient penalty are not mutually exclusive and enforce Lipschitzness in different senses. The penalty constrains the gradient norm on the data-fake interpolants (a local, data-dependent constraint), while spectral normalization bounds the operator norm of each layer (a global, data-independent constraint). Spectral normalization is cheaper and needs no tuning of \( \lambda \). The gradient penalty can fit a tighter, less conservative critic because it does not bound every direction, only the ones that matter. Modern StyleGAN2 uses neither and instead relies on the R1 penalty \( \frac{\gamma}{2}\E_{p_{\text{data}}}[\lVert \nabla_x D(x)\rVert^2] \) (Mescheder et al. 2018), a one-sided gradient penalty on real data only, which is cheaper still and provably stabilizes the local dynamics of the game.

The f-GAN view of GANs as variational divergence minimization

The JSD result is one instance of a general pattern that Nowozin, Cseke, and Tomioka (2016) made explicit. Any f-divergence

for a convex \( f \) with \( f(1) = 0 \), admits a variational lower bound through the convex (Fenchel) conjugate \( f^\ast(t) = \sup_u \{ ut - f(u) \} \). Because \( f(u) = \sup_t \{ ut - f^\ast(t) \} \) by biconjugacy, substituting \( u = p/q \) and pulling the supremum outside the integral gives

where \( T : \mathcal{X} \to \R \) ranges over a family of functions (a neural network, in practice) and the bound is tight when \( T(x) = f'(p(x)/q(x)) \). This is exactly the adversarial template. A maximizing "critic" \( T \) estimates the divergence, and a generator minimizes it. Choosing \( f \) recovers specific GANs. The Jensen-Shannon choice reproduces the original GAN up to the \( \log 4 \) shift. The reverse-KL, forward-KL, Pearson \( \chi^2 \), and squared Hellinger choices give the other members of the f-GAN family, each with a different \( f^\ast \) and therefore a different final-layer activation on the critic. The value of the f-GAN view is conceptual. It says the adversarial game is a generic machine for minimizing a variational estimate of a divergence, and the specific divergence is a design choice, not a fixed feature of "the" GAN. It also clarifies why the Wasserstein distance sits apart. \( W_1 \) is not an f-divergence, its duality is Kantorovich-Rubinstein rather than Fenchel, which is why WGAN needs a Lipschitz constraint rather than a conjugate activation.

Conditional GANs and image-to-image translation

All of the above generates unconditional samples. Most applications want control, the ability to generate an image of a given class or an image corresponding to this input. Mirza and Osindero (2014) added conditioning in the most direct way. Feed a condition \( y \), a class label or any side information, to both players. The generator becomes \( G(z, y) \) and the discriminator \( D(x, y) \), and the value function conditions every expectation on \( y \),

The discriminator now judges not just whether \( x \) is realistic but whether it is realistic and consistent with \( y \), so the generator must respect the condition. Modern conditional GANs inject the label through conditional batch normalization or a projection discriminator rather than concatenation, but the principle is unchanged.

pix2pix, a conditional GAN plus an L1 term

Isola et al. (2017) specialized conditional GANs to image-to-image translation with paired data. Each training example is a pair \( (x, y) \), an input image \( x \) (an edge map, a semantic label map, a grayscale photo) and its target \( y \) (the photograph, the colorized image). The generator maps \( x \mapsto G(x) \) and the discriminator judges pairs \( D(x, y) \). The insight that made pix2pix work is that the adversarial loss alone produces sharp but unfaithful outputs, and a pure regression loss produces faithful but blurry ones, so combine them. The objective adds an \( L_1 \) reconstruction term to the conditional adversarial loss,

The \( L_1 \) term (not \( L_2 \), because \( L_1 \) tolerates a bit of blur less and pushes toward sharper medians) forces \( G(x) \) to match the paired target on average and captures low-frequency correctness, while the discriminator, implemented as a PatchGAN that classifies overlapping \( 70{\times}70 \) patches rather than the whole image, forces high-frequency sharpness and realistic texture. The division of labor, regression for structure, adversary for texture, is the same one that reappears years later inside diffusion autoencoders and in super-resolution networks.

CycleGAN and the cycle-consistency loss

pix2pix needs paired data, which for most translation tasks does not exist. There is no photograph of the same horse as both a horse and a zebra, no summer and winter photo of the identical scene from the identical angle. Zhu et al. (2017) removed the pairing requirement with CycleGAN. Given two unpaired collections, domain \( X \) (horses) and domain \( Y \) (zebras), learn two generators, \( G : X \to Y \) and \( F : Y \to X \), and two discriminators, \( D_Y \) judging whether an image looks like domain \( Y \) and \( D_X \) for domain \( X \). Two adversarial losses push \( G(x) \) to look like a real zebra and \( F(y) \) to look like a real horse.

The problem is that adversarial losses alone are badly underconstrained. Infinitely many maps \( G \) send horse images to the set of realistic zebra images. Nothing ties a particular horse to a particular zebra, and in the worst case \( G \) could map every horse to the same one zebra (mode collapse) and still satisfy the discriminator. The fix is cycle consistency. Translating to the other domain and back should return the original image. Formally, \( F(G(x)) \approx x \) for every horse \( x \), and \( G(F(y)) \approx y \) for every zebra \( y \). This is enforced with an \( L_1 \) penalty,

and the full objective sums the two adversarial losses and the cycle loss with a weight \( \lambda \) (the paper uses \( \lambda = 10 \)),

The reason cycle consistency enables unpaired translation is worth stating carefully, because it is the conceptual core of the method. The adversarial losses constrain the marginals. They force \( G_\#p_X \) to match \( p_Y \) and \( F_\#p_Y \) to match \( p_X \) in distribution. The cycle loss constrains the coupling. It forces \( G \) and \( F \) to be approximate inverses, which means the translation must be a nearly bijective correspondence rather than a many-to-one collapse. A collapsed \( G \) that sent all horses to one zebra could not be inverted by any \( F \), so \( F(G(x)) \) could not recover the specific \( x \), and the cycle loss would be large. The two constraints together, right marginals plus invertibility, pin down a structured, content-preserving map without ever seeing a single aligned pair. The residual ambiguity, which visual attribute counts as "content" to preserve versus "style" to change, is exactly where CycleGAN's known failure modes live (it may hallucinate texture onto the wrong regions, or bake in a color shift as an invertible watermark to satisfy the cycle loss cheaply), and it is the reason later work added semantic or contrastive constraints on top.

Progressive growing and StyleGAN

The architectural line that made GANs produce megapixel photorealism runs through NVIDIA's work. Karras et al. (2018) introduced progressive growing. Start by training the generator and discriminator at \( 4{\times}4 \) resolution, then incrementally fade in new layers that double the resolution, \( 8{\times}8 \), \( 16{\times}16 \), up to \( 1024{\times}1024 \). Each new block is faded in with a smooth interpolation weight so the previously trained layers are not shocked. Growing the resolution progressively stabilizes training, because the model first learns coarse structure on an easy low-dimensional problem and only later refines detail, and it cut training time substantially. Progressive growing also introduced the minibatch standard deviation layer in the discriminator, which appends the standard deviation of features across the batch as an extra feature map, giving the discriminator a direct view of whether the batch has collapsed to low diversity.

The style-based generator

StyleGAN (Karras et al. 2019) rethought the generator itself. A traditional generator feeds the latent \( z \) in at the bottom and propagates it up through convolutions. StyleGAN does two things differently. First, it passes \( z \) through a mapping network, an 8-layer MLP \( f : \mathcal{Z} \to \mathcal{W} \), producing an intermediate latent \( w \in \mathcal{W} \). The synthesis network then does not take \( z \) at all. It starts from a learned constant \( 4{\times}4 \) tensor and is modulated at every layer by \( w \). Second, \( w \) controls the image through adaptive instance normalization (AdaIN). For a feature map channel \( x_i \), AdaIN normalizes it to zero mean and unit variance across spatial positions and then re-scales and re-shifts it with per-channel style parameters \( (y_{s,i}, y_{b,i}) \) computed from \( w \) by a learned affine map,

Injecting the style at every resolution, with independently sampled \( w \) allowed at different layers (style mixing), separates coarse attributes (pose, face shape, controlled by early low-resolution layers) from fine attributes (hair texture, freckles, controlled by late high-resolution layers). Adding per-pixel Gaussian noise at each layer supplies the stochastic detail (exact hair placement) that the deterministic style should not have to encode.

Why the W space disentangles

The disentanglement of \( \mathcal{W} \) is the most cited and most misunderstood property of StyleGAN, so it is worth the derivation in words. The input space \( \mathcal{Z} \) is a fixed Gaussian, so its density is spherically symmetric. But the distribution of real image attributes is not. Some combinations of attributes are common and some are rare or impossible. To match data, a generator that took \( z \) directly would have to warp the round Gaussian heavily to carve out the empty regions, and that warping forces attributes to become entangled, moving along one axis of \( z \) changes several attributes at once because the axis has been bent. The mapping network's job is to absorb that warping. Because \( w = f(z) \) is produced by a learned nonlinear map with no requirement that \( \mathcal{W} \) be Gaussian or even fill space uniformly, \( \mathcal{W} \) can take whatever shape makes the subsequent affine style transformations act more linearly on image attributes. The paper measures this with perceptual path length and linear separability metrics and finds \( \mathcal{W} \) markedly more disentangled than \( \mathcal{Z} \). Disentanglement is not imposed. It emerges because a less warped, more factorized intermediate space is an easier target for the synthesis network to use, and the mapping network is free to provide it.

Weight demodulation in StyleGAN2

StyleGAN images had a characteristic artifact, blob-like distortions that moved with the subject. Karras et al. (2020) traced this to AdaIN. Instance normalization removes per-feature-map mean and variance information, and the generator learned to smuggle signal past that normalization by creating a strong localized spike, a blob, whose statistics dominate the normalization and survive it. StyleGAN2 removes the explicit normalization and replaces it with weight demodulation, which achieves the same scale control as AdAIN but operates on the convolution weights rather than the activations, so there is no spatial statistic for the generator to exploit. Given a style scale \( s_i \) for input channel \( i \), the convolution weights \( w_{ijk} \) (input channel \( i \), output channel \( j \), spatial tap \( k \)) are first modulated,

which scales each input channel by its style. Modulation changes the output variance. To restore it, assume unit-variance inputs and compute the resulting output standard deviation as the \( L_2 \) norm of the modulated weights over the input and spatial dimensions, then divide it out (demodulate),

This restores unit output variance in expectation without ever computing an activation statistic, so the blob exploit disappears. Weight demodulation is a statistical approximation to instance normalization, exact under the assumption of independent unit-variance inputs, and it is cheaper because it acts on weights, which are the same for the whole batch, rather than per-sample activations. StyleGAN2 with this change, plus path-length regularization and the removal of progressive growing in favor of a skip/residual architecture, set the image-quality bar that diffusion would have to clear.

Evaluation with the Inception Score and FID

A GAN has no likelihood, so it cannot be evaluated by held-out log-likelihood the way explicit models are. Evaluation instead compares samples to data through the lens of a fixed pretrained classifier, almost always an Inception-v3 network trained on ImageNet, on the theory that its features capture perceptually meaningful structure. Two metrics dominate.

Inception Score

Salimans et al. (2016) proposed the Inception Score. Push each generated image \( x \) through Inception to get a conditional label distribution \( p(y \mid x) \) over the 1000 ImageNet classes. Two things should hold for good samples. Each image should be recognizable as some object, so \( p(y \mid x) \) should be low-entropy (peaked on one class). Across many images the model should produce all classes, so the marginal \( p(y) = \E_x[p(y \mid x)] \) should be high-entropy (uniform over classes). Both are captured at once by the expected KL divergence between the conditional and the marginal, exponentiated for readability,

The KL is large exactly when each conditional is peaked (low entropy) and the marginal is spread (high entropy), so a high IS rewards both sharpness and diversity. Its flaws are serious, well documented, and worth knowing for interviews. It never looks at the real data at all, so it cannot detect that the generated distribution differs from the target. It is trivially gamed by a model that produces one perfect image per class, it is defined only for ImageNet-like class structure, and it is sensitive to the specific Inception weights and preprocessing. It has largely been retired in favor of FID.

FID as a Frechet distance between Gaussians

Heusel et al. (2017) introduced the Frechet Inception Distance, which fixes the "never looks at data" flaw by comparing feature statistics of real and generated images directly. Take the 2048-dimensional activations of the final Inception pooling layer for a large set of real images and a large set of generated images. Model each set as a multivariate Gaussian, real features \( \sim \N(\mu_r, \Sigma_r) \), generated features \( \sim \N(\mu_g, \Sigma_g) \), with means and covariances estimated from the samples. FID is the Frechet distance, equivalently the squared Wasserstein-2 distance, between these two Gaussians. That distance has a closed form. For two Gaussians the 2-Wasserstein distance is

The sketch of the derivation is that the optimal transport map between two Gaussians is affine, and matching first and second moments under the squared-Euclidean ground cost gives a mean term \( \lVert \mu_r - \mu_g \rVert^2 \) plus a covariance term. The covariance term is the Bures metric between positive-definite matrices, \( \tr(\Sigma_r + \Sigma_g - 2(\Sigma_r^{1/2}\Sigma_g \Sigma_r^{1/2})^{1/2}) \). The matrix \( (\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2} \) has the same trace as \( (\Sigma_r\Sigma_g)^{1/2} \), which is the form the standard FID implementation computes with a matrix square root. Because FID compares to real data, it catches mode collapse (a collapsed \( \Sigma_g \) that is too small inflates the trace term) and blur (which shifts the mean and covariance), and it correlates far better with human judgment than IS. It has its own limitations. The Gaussian assumption discards all non-Gaussian structure in the feature distribution. It is biased upward for small sample sizes, so numbers computed on 5k images are not comparable to numbers on 50k. And it inherits whatever blind spots the Inception features have, so a model that exploits those features can score well while looking wrong to a human. FID is the standard anyway, because it is the least bad option that looks at both distributions.

Two worked computations make the formula concrete. In one dimension with a real \( \N(0, 1^2) \) and a generated \( \N(1, 1.5^2) \), the mean term is \( (0-1)^2 = 1 \) and the variance term is \( 1^2 + 1.5^2 - 2\sqrt{1^2 \cdot 1.5^2} = 1 + 2.25 - 2(1.5) = 0.25 \), so \( \mathrm{FID} = 1.25 \). Problem 2 does the two-dimensional case by hand with full covariance matrices and verifies it against SciPy's matrix square root, obtaining \( \mathrm{FID} = 10.575 \). Both are reproduced exactly in the implementation section.

GANs versus diffusion, honestly

The diffusion page owns the derivation of denoising diffusion and the deep generative models page places both families in the taxonomy. This section is the direct comparison from the adversarial side. The honest summary is that diffusion won the general-purpose text-to-image race and GANs kept a set of niches that are growing again.

Where GANs still win comes down to that first row. When you need a sample now, in one pass, the GAN is structurally advantaged. Real-time and interactive generation, on-device synthesis under a tight compute budget, and especially single- image super-resolution (SRGAN, ESRGAN, and their descendants), where an adversarial loss on top of a regression backbone produces perceptually sharp detail that an MSE loss alone blurs, remain GAN territory. The 2022 StyleGAN-XL result (Sauer et al.) showed style-based GANs scaling to ImageNet at competitive FID, closing much of the gap on the axis where diffusion had pulled ahead.

The more important story is convergence. The expensive part of diffusion is the many-step sampling loop, and the way to remove it is distillation, training a one-step or few-step student to match the many-step teacher. The most effective distillation objectives are adversarial. Adversarial diffusion distillation (Sauer et al. 2023) adds a discriminator that judges the student's one-step output against real images, recovering the sharpness that pure regression distillation loses. Distribution-matching and consistency-style distillation land in the same place. Consistency models (Song et al. 2023) reach the one-step regime from the diffusion side by learning a direct map from any noise level to the clean image, and their adversarially-augmented variants are, mechanically, GANs whose generator happens to have a diffusion pedigree. The two families are meeting in the middle. Diffusion supplies a stable way to learn the distribution, and the adversarial loss supplies the one-step sharpness and the speed. A practitioner in 2026 should expect the production image system to be a diffusion model trained with a regression objective and then distilled to one or two steps with an adversarial loss, which is to say, both.

Worked problems

Implementation

The code below is organized as the assignment specifies. It gives the non-saturating GAN losses in PyTorch and JAX, the WGAN-GP gradient penalty in PyTorch and JAX, and standalone Python for the FID computation and the spectral-norm power iteration. Every snippet corresponds to a derivation above, and the numbers the Python snippets print are the numbers used in the worked problems.

Non-saturating GAN losses

The discriminator loss is the binary cross-entropy of the two-sample classification. The generator loss uses the non-saturating form \( -\log D(G(z)) \), implemented as a BCE against the "real" label so that the gradient is \( -1/d \) rather than the saturating \( -1/(1-d) \). Both frameworks operate on discriminator logits and use the numerically stable logits-BCE, which is why the code never applies a sigmoid explicitly.

import torch
import torch.nn.functional as F

def bce_logits(logits, target):
    # target is 1.0 for "real", 0.0 for "fake"; stable log-sigmoid form
    return F.binary_cross_entropy_with_logits(logits, target)

def d_loss(d_real_logits, d_fake_logits):
    # discriminator maximizes log D(x) + log(1 - D(G(z)))
    # -> minimize BCE(real=1) on data and BCE(real=0) on fakes
    ones = torch.ones_like(d_real_logits)
    zeros = torch.zeros_like(d_fake_logits)
    return bce_logits(d_real_logits, ones) + bce_logits(d_fake_logits, zeros)

def g_loss_nonsaturating(d_fake_logits):
    # generator maximizes log D(G(z))  ==  minimize BCE(fake logits, target=1)
    # gradient wrt d is -1/d (strong when d small), not -1/(1-d) (saturating)
    ones = torch.ones_like(d_fake_logits)
    return bce_logits(d_fake_logits, ones)

def g_loss_saturating(d_fake_logits):
    # the original minimax generator loss: minimize log(1 - D(G(z)))
    # shown only for contrast; its gradient vanishes as d -> 0
    zeros = torch.zeros_like(d_fake_logits)
    return -bce_logits(d_fake_logits, zeros)  # = log(1 - sigmoid(logits)) up to sign

# one optimization step (shapes: x_real [B, D], z [B, k])
def step(G, D, opt_g, opt_d, x_real, z):
    # --- discriminator ---
    opt_d.zero_grad()
    fake = G(z).detach()                    # [B, D]
    ld = d_loss(D(x_real), D(fake))
    ld.backward(); opt_d.step()
    # --- generator (non-saturating) ---
    opt_g.zero_grad()
    lg = g_loss_nonsaturating(D(G(z)))      # note: not detached
    lg.backward(); opt_g.step()
    return ld.item(), lg.item()

import jax, jax.numpy as jnp
import optax

def bce_logits(logits, target):
    # stable binary cross-entropy from logits, matching torch's formulation
    # = max(z,0) - z*target + log(1 + exp(-|z|))
    z = logits
    return jnp.mean(jnp.maximum(z, 0) - z * target + jnp.log1p(jnp.exp(-jnp.abs(z))))

def d_loss(d_real_logits, d_fake_logits):
    return (bce_logits(d_real_logits, jnp.ones_like(d_real_logits))
            + bce_logits(d_fake_logits, jnp.zeros_like(d_fake_logits)))

def g_loss_nonsaturating(d_fake_logits):
    # minimize BCE(fake logits, target=1) == maximize log D(G(z))
    return bce_logits(d_fake_logits, jnp.ones_like(d_fake_logits))

# functional-style step: params carry G and D; apply_G / apply_D are pure fns
def step(pg, pd, opt_g, opt_d, st_g, st_d, x_real, z, apply_G, apply_D):
    def d_obj(pd):
        fake = jax.lax.stop_gradient(apply_G(pg, z))     # [B, D]
        return d_loss(apply_D(pd, x_real), apply_D(pd, fake))
    ld, gd = jax.value_and_grad(d_obj)(pd)
    updates, st_d = opt_d.update(gd, st_d, pd); pd = optax.apply_updates(pd, updates)

    def g_obj(pg):
        return g_loss_nonsaturating(apply_D(pd, apply_G(pg, z)))
    lg, gg = jax.value_and_grad(g_obj)(pg)
    updates, st_g = opt_g.update(gg, st_g, pg); pg = optax.apply_updates(pg, updates)
    return pg, pd, st_g, st_d, ld, lg

The WGAN-GP gradient penalty

The penalty requires a gradient of the critic with respect to its input, evaluated on random interpolants between real and fake, and then a gradient of the whole loss with respect to the critic's parameters, hence a double backward. In PyTorch this is torch.autograd.grad(..., create_graph=True). In JAX it is simply nesting jax.grad inside the loss, which the transform composes automatically. Both compute \( \lambda\,\E[(\lVert\nabla_{\hat x} f(\hat x)\rVert_2 - 1)^2] \) with \( \lambda = 10 \).

import torch

def gradient_penalty(critic, x_real, x_fake, lam=10.0):
    # x_real, x_fake: [B, D]; interpolate along random lines between them
    B = x_real.size(0)
    eps = torch.rand(B, 1, device=x_real.device)          # [B, 1] in [0,1)
    x_hat = (eps * x_real + (1 - eps) * x_fake).requires_grad_(True)
    d_hat = critic(x_hat)                                  # [B, 1]
    grad = torch.autograd.grad(
        outputs=d_hat.sum(), inputs=x_hat,
        create_graph=True)[0]                              # d f / d x_hat, [B, D]
    gnorm = grad.norm(2, dim=1)                            # [B]
    return lam * ((gnorm - 1.0) ** 2).mean()

def critic_loss(critic, x_real, x_fake, lam=10.0):
    # WGAN critic MAXIMIZES E[f(real)] - E[f(fake)], so we minimize the negative,
    # plus the two-sided gradient penalty that enforces 1-Lipschitzness.
    wass = critic(x_fake).mean() - critic(x_real).mean()  # negative W1 estimate
    return wass + gradient_penalty(critic, x_real, x_fake.detach(), lam)

def generator_loss(critic, x_fake):
    # generator minimizes -E[f(fake)] (pushes critic score up on its samples)
    return -critic(x_fake).mean()

import jax, jax.numpy as jnp

def gradient_penalty(apply_critic, pc, x_real, x_fake, key, lam=10.0):
    B = x_real.shape[0]
    eps = jax.random.uniform(key, (B, 1))                 # [B, 1]
    x_hat = eps * x_real + (1 - eps) * x_fake             # [B, D]
    # per-sample gradient of the scalar critic wrt its input, via jacobian-vector:
    def critic_scalar(x):                                 # x: [D] -> scalar
        return apply_critic(pc, x[None, :])[0, 0]
    grads = jax.vmap(jax.grad(critic_scalar))(x_hat)      # [B, D]
    gnorm = jnp.linalg.norm(grads, axis=1)                # [B]
    return lam * jnp.mean((gnorm - 1.0) ** 2)

def critic_loss(apply_critic, pc, x_real, x_fake, key, lam=10.0):
    wass = jnp.mean(apply_critic(pc, x_fake)) - jnp.mean(apply_critic(pc, x_real))
    gp = gradient_penalty(apply_critic, pc,
                          x_real, jax.lax.stop_gradient(x_fake), key, lam)
    return wass + gp

def generator_loss(apply_critic, pc, x_fake):
    return -jnp.mean(apply_critic(pc, x_fake))

FID and the spectral-norm power iteration

These two Python snippets are the exact code used to verify Problems 2 and 3. The FID function is the standard Frechet-distance-between- Gaussians implementation, with the matrix square root computed by SciPy and a sanity check that squaring it reproduces the product to machine precision (a necessary habit on any host, per the numerical note in the authoring guide). The power-iteration function estimates the top singular value and is compared against the full SVD.

import numpy as np
from scipy.linalg import sqrtm

def fid(mu_r, cov_r, mu_g, cov_g):
    # Frechet distance between N(mu_r, cov_r) and N(mu_g, cov_g)
    # = ||mu_r - mu_g||^2 + tr(cov_r + cov_g - 2 (cov_r cov_g)^{1/2})
    diff = mu_r - mu_g
    covmean = sqrtm(cov_r @ cov_g)
    if np.iscomplexobj(covmean):        # numerical roundoff can add tiny imag part
        covmean = covmean.real
    # sanity check the matrix sqrt before trusting the trace term
    assert np.abs(covmean @ covmean - cov_r @ cov_g).max() < 1e-8
    return float(diff @ diff + np.trace(cov_r + cov_g - 2 * covmean))

# Problem 2 values
mu_r, mu_g = np.array([0., 0.]), np.array([3., 1.])
cov_r = np.array([[2., 0.5], [0.5, 1.]])
cov_g = np.array([[1., -0.3], [-0.3, 2.]])
print("FID (2-D) =", fid(mu_r, cov_r, mu_g, cov_g))    # -> 10.5754...

def spectral_norm(W, n_iter=7, seed=0):
    # estimate sigma_max(W) by power iteration on W^T W
    rng = np.random.default_rng(seed)
    u = rng.standard_normal(W.shape[0]); u /= np.linalg.norm(u)
    for _ in range(n_iter):
        v = W.T @ u; v /= np.linalg.norm(v)     # right singular vector estimate
        u = W @ v; sigma = np.linalg.norm(u); u /= sigma
    return sigma

W = np.array([[3., 1., 0.], [0., 2., 1.], [1., 0., 1.]])
print("power iteration sigma_max =", spectral_norm(W))          # 3.42479
print("true sigma_max (svd)      =", np.linalg.svd(W, compute_uv=False)[0])  # 3.4247893

How it is done in practice

The gap between the derivation and a GAN that produces publishable images is mostly engineering discipline around the instability the theory predicts. A few practices recur across every strong system.

Balance, not a fixed ratio. The clean derivation wants the discriminator optimal at every generator step, but a too- strong discriminator saturates the generator and a too-weak one gives a meaningless signal. Standard GANs alternate one discriminator step per generator step and rely on the non-saturating loss. WGAN-GP runs several critic steps (the paper uses five) per generator step, because the Wasserstein critic can be trained near-optimal without harming the generator's gradient. The two-timescale update rule (Heusel et al. 2017, the same paper that introduced FID) gives the discriminator a higher learning rate than the generator and proves convergence to a local Nash equilibrium under that separation. It is standard in modern recipes.

Lipschitz control is not optional at scale. Spectral normalization on the discriminator became the default stabilizer because it costs almost nothing, one power-iteration step per layer per forward pass, and needs no tuning. BigGAN (Brock et al. 2019) combined spectral normalization with very large batches (2048 and up) and the truncation trick at sampling time, where latents are sampled from a truncated Gaussian to trade diversity for fidelity, and pushed class-conditional ImageNet to then-unmatched quality. StyleGAN2 instead uses the R1 penalty on real data. The common thread is that some mechanism must bound how fast the discriminator's output can change with its input, or the game diverges.

The cost profile favors GANs at inference. On an NVIDIA H100 80GB, a full WGAN-GP discriminator-plus-generator training step on a 784-dimensional MLP with batch 256 runs in 1.914 ms, and most of that time is the double backward through the gradient penalty. A plain non-saturating step without the penalty is cheaper still. Inference is a single forward pass through the generator, microseconds to milliseconds for the sizes used in interactive applications. That single-pass sampling is the structural reason GANs remain the tool of choice when latency is the binding constraint, and it is measured on the same hardware where a diffusion sampler needs tens to hundreds of network evaluations for one image.

Evaluation is a moving target. FID is reported everywhere but is only comparable when the sample count, the reference statistics, and the Inception weights match exactly. A paper reporting FID on 50k samples against the training-set statistics is not comparable to one using 10k against a held-out set. Serious comparisons fix the evaluation harness (the pytorch-fid or clean-fid implementations) and quote precision and recall metrics alongside FID to separate fidelity from coverage, because a single scalar cannot distinguish a sharp low-diversity model from a diverse slightly-blurry one.

The current research frontier

Three threads define where adversarial modeling sits in 2026. The first is GANs scaling back up. After diffusion took the headline, NVIDIA's StyleGAN-XL (Sauer et al. 2022) showed that a style-based GAN, with a projected discriminator built on pretrained features and progressive growing done carefully, scales to ImageNet at all resolutions with FID competitive with diffusion, refuting the claim that GANs cannot scale. GigaGAN (Kang et al. 2023, from Adobe and CMU/POSTECH collaborators) scaled a GAN to a billion parameters for text-to-image at a fraction of diffusion's sampling cost, demonstrating that the architecture, not the adversarial objective, had been the limiting factor. StyleGAN3 (Karras et al. 2021) solved the "texture sticking" aliasing problem to make truly equivariant, animation-quality generators.

The second thread is adversarial distillation of diffusion, which is where most industrial attention now goes. The many-step diffusion sampler is the bottleneck for deployment, and the fastest distillation objectives are adversarial. Adversarial diffusion distillation (Sauer et al. 2023, Stability AI) trains a one-step or few-step student with a discriminator judging its output against real images, plus a distillation term from the teacher, recovering sharpness that pure regression distillation loses and matching the multi-step teacher in human preference. Distribution-matching distillation and latent-space adversarial variants land in the same regime. The upshot is that the adversarial loss, thought to be superseded, turned out to be the missing ingredient that makes diffusion fast.

The third thread is the theoretical convergence of the families. Consistency models (Song et al. 2023, OpenAI) learn a direct map from any point on the diffusion trajectory to its clean endpoint, achieving one- or two-step generation from the diffusion side. Their adversarially-regularized versions are, structurally, GANs with a diffusion-derived generator and training target. Optimal-transport and flow-matching perspectives now describe GANs, diffusion, and consistency models in one language of learning a transport map from noise to data, so that the choice between an adversarial loss, a denoising loss, and a consistency loss is understood as a choice of how to fit that map rather than a choice of fundamentally different models. The practical synthesis, a distribution learned stably by diffusion and made fast by an adversarial one-step head, is the shape of production image generation going forward, and it is why the material on this page remains load-bearing even though the pure unconditional GAN is no longer the frontier system on its own.

Open source to read

The following repositories are the canonical implementations. Each entry says what it is best for and which file to open first.

• NVlabs/stylegan2-ada-pytorch is the reference StyleGAN2 with adaptive discriminator augmentation for limited data. Open training/networks.py to read the weight-demodulation convolution and the mapping network exactly as derived above, and training/loss.py for the R1 penalty and path-length regularization.
• junyanz/pytorch-CycleGAN-and-pix2pix is the original authors' implementation of both translation methods. Start with models/cycle_gan_model.py to see the two generators, two discriminators, and the cycle-consistency \( L_1 \) loss assembled, and models/networks.py for the PatchGAN discriminator.
• pytorch/examples (dcgan) is the minimal, readable DCGAN, a single main.py with the non-saturating loss and the convolutional generator and discriminator. The best first GAN to run end to end.
• mseitzer/pytorch-fid is the standard FID implementation. Read src/pytorch_fid/fid_score.py for the exact Frechet-distance-between-Gaussians computation, including the matrix-square-root numerics and the small-sample bias handling that the derivation above only sketches.
• lucidrains/stylegan2-pytorch is a compact, well-commented reimplementation that is easier to read end to end than the official one, and a good second read after the NVIDIA repository for understanding the moving parts.
• PythonOT/POT (Python Optimal Transport) computes Wasserstein distances and transport plans directly. Use it to build the toy that shows \( W_1 \) varies smoothly where JSD is flat, and to sanity-check the Kantorovich-Rubinstein dual against the primal transport cost.
• NVlabs/stylegan3 is the alias-free generator. Read it when the artifact you care about is texture sticking under animation, and to see how careful signal processing removes it.

Common misconceptions

"The discriminator learns an arbitrary decision boundary." At optimality it does not. It learns the density ratio \( p_{\text{data}}/(p_{\text{data}}+p_g) \), which is a specific, calibrated posterior. The generator's loss at that optimum is a real divergence, the Jensen-Shannon divergence, not a heuristic. The adversarial framing hides a principled objective.

"The non-saturating loss just flips a sign." It changes the objective. The minimax generator loss minimizes \( 2\,\mathrm{JSD} - \log 4 \) at discriminator optimality. The non-saturating loss minimizes a different, reverse-KL-flavored combination of divergences that Arjovsky and Bottou analyze. They share a fixed point but differ everywhere else, and the difference is precisely the gradient behavior at \( d \to 0 \) that makes one train and the other stall.

"Wasserstein GANs are stable because the loss is Wasserstein." They are stable because the Wasserstein distance has a useful gradient even when the supports of \( p_g \) and \( p_{\text{data}} \) are disjoint, which is the generic case for image manifolds. The distance is a means and the smooth gradient is the end. And the whole benefit evaporates if the Lipschitz constraint is not enforced, which is why weight clipping, gradient penalty, and spectral normalization exist.

"Weight clipping and the gradient penalty do the same thing." Both aim at 1-Lipschitzness but by opposite means. Clipping bounds each weight, which bounds operator norms crudely and pushes weights to two spikes, wasting critic capacity. The gradient penalty constrains the input-gradient norm on the data-fake interpolants, a data-dependent, tighter constraint that lets the critic keep its capacity. Spectral normalization is a third option that bounds the operator norm exactly per layer without either pathology.

"A low FID means the samples are good." FID assumes the Inception features are Gaussian and only compares first and second moments, so it is blind to any structure those two moments miss and to anything the Inception features themselves ignore. It is biased upward at small sample sizes, so numbers computed on different sample counts are not comparable, and it can be gamed by a model that matches feature statistics while looking wrong to a human. Report it with a fixed harness and alongside precision/recall, never alone.

"StyleGAN's W space disentangles because it is designed to." Nothing in StyleGAN imposes disentanglement. It emerges because the mapping network can unwarp the Gaussian \( \mathcal{Z} \) into a space \( \mathcal{W} \) whose shape makes attributes act more linearly under the affine style transforms, and the synthesis network prefers such a space. Disentanglement is a learned convenience, measured after the fact by perceptual path length, not a hard constraint.

"Diffusion made GANs obsolete." Diffusion won general text-to-image on quality and stability, but GANs sample in one pass, and that keeps them in super-resolution, real-time and on-device generation, and, most importantly, in the adversarial distillation step that makes diffusion itself fast enough to deploy. The families are converging, and the adversarial loss is on both sides of the convergence.

Self-check

References

1. Goodfellow, Bengio, Courville. Deep Learning. MIT Press, 2016. Chapter 20 covers generative models including GANs. deeplearningbook.org
2. Peyre, Cuturi. Computational Optimal Transport. Foundations and Trends in Machine Learning, 2019. arXiv:1803.00567
3. Villani. Optimal Transport: Old and New. Springer, 2009. The reference for Wasserstein distances and Kantorovich-Rubinstein duality.
4. Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, Bengio. Generative Adversarial Nets. NeurIPS 2014. arXiv:1406.2661
5. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. 2016. arXiv:1701.00160
6. Arjovsky, Bottou. Towards Principled Methods for Training Generative Adversarial Networks. ICLR 2017. arXiv:1701.04862
7. Arjovsky, Chintala, Bottou. Wasserstein GAN. ICML 2017. arXiv:1701.07875
8. Gulrajani, Ahmed, Arjovsky, Dumoulin, Courville. Improved Training of Wasserstein GANs (WGAN-GP). NeurIPS 2017. arXiv:1704.00028
9. Miyato, Kataoka, Koyama, Yoshida. Spectral Normalization for Generative Adversarial Networks. ICLR 2018. arXiv:1802.05957
10. Nowozin, Cseke, Tomioka. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. NeurIPS 2016. arXiv:1606.00709
11. Mirza, Osindero. Conditional Generative Adversarial Nets. 2014. arXiv:1411.1784
12. Radford, Metz, Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (DCGAN). ICLR 2016. arXiv:1511.06434
13. Karras, Aila, Laine, Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. ICLR 2018. arXiv:1710.10196
14. Karras, Laine, Aila. A Style-Based Generator Architecture for Generative Adversarial Networks (StyleGAN). CVPR 2019. arXiv:1812.04948
15. Karras, Laine, Aittala, Hellsten, Lehtinen, Aila. Analyzing and Improving the Image Quality of StyleGAN (StyleGAN2). CVPR 2020. arXiv:1912.04958
16. Karras, Aittala, Hellsten, Laine, Lehtinen, Aila. Training Generative Adversarial Networks with Limited Data (ADA). NeurIPS 2020. arXiv:2006.06676
17. Karras, Aittala, Laine, Harkonen, Hellsten, Lehtinen, Aila. Alias-Free Generative Adversarial Networks (StyleGAN3). NeurIPS 2021. arXiv:2106.12423
18. Isola, Zhu, Zhou, Efros. Image-to-Image Translation with Conditional Adversarial Networks (pix2pix). CVPR 2017. arXiv:1611.07004
19. Zhu, Park, Isola, Efros. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (CycleGAN). ICCV 2017. arXiv:1703.10593
20. Salimans, Goodfellow, Zaremba, Cheung, Radford, Chen. Improved Techniques for Training GANs (Inception Score). NeurIPS 2016. arXiv:1606.03498
21. Heusel, Ramsauer, Unterthiner, Nessler, Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID). NeurIPS 2017. arXiv:1706.08500
22. Mescheder, Geiger, Nowozin. Which Training Methods for GANs do actually Converge? (R1 regularization). ICML 2018. arXiv:1801.04406
23. Brock, Donahue, Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis (BigGAN). ICLR 2019. arXiv:1809.11096
24. Ledig, Theis, Huszar, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (SRGAN). CVPR 2017. arXiv:1609.04802
25. Sauer, Schwarz, Geiger. StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. SIGGRAPH 2022. arXiv:2202.00273
26. Kang, Zhu, Zhang, et al. Scaling up GANs for Text-to-Image Synthesis (GigaGAN). CVPR 2023. arXiv:2303.05511
27. Song, Dhariwal, Chen, Sutskever. Consistency Models. ICML 2023. arXiv:2303.01469
28. Sauer, Lorenz, Blattmann, Rombach. Adversarial Diffusion Distillation. 2023. arXiv:2311.17042