Generative Models

Models that learn to generate new data — not just classify existing data. Full derivations of the ELBO, GAN minimax, and the diffusion forward/reverse process.

Table of contents

1. Taxonomy of generative models
2. Variational Autoencoders (VAE)
3. Generative Adversarial Networks (GAN)
4. Normalizing flows (overview)
5. Diffusion models
6. Evaluation of generative models
7. Comparison and when to use what

1. Taxonomy of generative models

A generative model learns the data distribution $p(\mathbf{x})$ and can sample from it.

2. Variational Autoencoders (VAE)

2.1 Latent variable model

The VAE (Kingma & Welling, 2013) defines a joint model:

with prior $p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$.

Marginal likelihood (what we want to maximize):

This integral is intractable for deep decoder networks.

2.2 Variational inference and ELBO derivation

Introduce an encoder (recognition model) $q_\phi(\mathbf{z}|\mathbf{x})$ to approximate the intractable posterior $p_\theta(\mathbf{z}|\mathbf{x})$.

KL decomposition:

Since $D_\text{KL} \geq 0$:

ELBO = Evidence Lower BOund. Maximizing ELBO ≡ maximizing marginal likelihood while minimizing KL to prior.

2.3 Decomposition of the ELBO

• Reconstruction: how well the decoder reconstructs $\mathbf{x}$ from the latent code $\mathbf{z}$.
• KL regularization: how close the encoder distribution is to the prior. Prevents the encoder from using a wildly non-standard latent space. Encourages a smooth, regular latent space useful for generation.

2.4 Gaussian encoder and closed-form KL

Encoder: $q_\phi(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\boldsymbol{\mu}_\phi(\mathbf{x}), \text{diag}(\boldsymbol{\sigma}^2_\phi(\mathbf{x})))$.

Prior: $p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$.

Closed-form KL (for Gaussian):

Only the reconstruction term requires sampling; KL is computed analytically.

2.5 Reparameterization trick

The reconstruction term requires taking gradients through a sampling operation $\mathbf{z} \sim q_\phi(\mathbf{z}|\mathbf{x})$, which is non-differentiable.

Trick: reparameterize as:

Now $\mathbf{z}$ is a deterministic function of $\boldsymbol{\mu}_\phi, \boldsymbol{\sigma}_\phi$, and independent noise $\boldsymbol{\varepsilon}$. Gradients w.r.t. $\phi$ flow through $\boldsymbol{\mu}_\phi$ and $\boldsymbol{\sigma}_\phi$.

2.6 VAE architecture and training

import torch
import torch.nn as nn
import torch.nn.functional as F

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=20):
        super().__init__()
        self.enc_fc1 = nn.Linear(input_dim, 400)
        self.enc_mu  = nn.Linear(400, latent_dim)
        self.enc_logvar = nn.Linear(400, latent_dim)
        self.dec_fc1 = nn.Linear(latent_dim, 400)
        self.dec_out = nn.Linear(400, input_dim)

    def encode(self, x):
        h = F.relu(self.enc_fc1(x))
        return self.enc_mu(h), self.enc_logvar(h)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h = F.relu(self.dec_fc1(z))
        return torch.sigmoid(self.dec_out(h))

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def vae_loss(recon_x, x, mu, logvar, beta=1.0):
    recon = F.binary_cross_entropy(recon_x, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon + beta * kl

2.7 $\beta$-VAE and disentanglement

Higher $\beta$ forces more independent, disentangled latent dimensions. Each latent dimension captures one factor of variation.

2.8 VAE for generation

Sample $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, then decode $\mathbf{x} = \text{Dec}_\theta(\mathbf{z})$. The smooth latent space enables interpolation between data points.

3. Generative Adversarial Networks (GAN)

3.1 The idea (Goodfellow et al., 2014)

Frame generation as a two-player minimax game:

• Generator $G_\theta$: maps noise $\mathbf{z} \sim p(\mathbf{z})$ to fake samples $G_\theta(\mathbf{z}) \approx p_\text{data}$.
• Discriminator $D_\phi$: outputs probability that input is real vs. fake.

Each tries to beat the other:

• $D$ tries to distinguish real from fake.
• $G$ tries to fool $D$.

3.2 Minimax objective (derivation)

Optimal discriminator (for fixed $G$): minimize $-V$ over $D$:

Substitute $D^*_G$ back into $V$:

This equals $-\log 4 + 2 \cdot D_\text{JS}(p_\text{data} \| p_G)$, where $D_\text{JS}$ is the Jensen-Shannon divergence.

Global minimum: $V(D^*_G, G^*) = -\log 4$ when $p_G = p_\text{data}$ (generator perfectly matches data).

3.3 Training algorithm

for each training iteration:
  1. Sample real batch: x ~ p_data
  2. Sample noise: z ~ p(z), generate fakes: x_fake = G(z)
  3. Update D:  maximize log D(x) + log(1 - D(G(z)))
                (or: minimize BCE on real=1, fake=0)
  4. Update G:  minimize log(1 - D(G(z)))
                (in practice: maximize log D(G(z)) — avoids vanishing gradients early)

Non-saturating GAN loss for generator:

3.4 Training instabilities

Mode collapse: $G$ maps all inputs to a few modes (ignores noise). Discriminator learns to reject those modes → $G$ shifts to other modes. Never covers full distribution.

Training instability (oscillation): $D$ and $G$ take turns outcompeting each other without converging.

Gradient vanishing in $D$: if $D$ is too good, $\log(1-D(G(\mathbf{z}))) \approx 0$ → no gradient for $G$.

3.5 Wasserstein GAN (WGAN)

Replace JS divergence with Earth Mover's (Wasserstein-1) distance:

Critic $D$ (no longer a probabilistic discriminator) approximates this distance with Lipschitz constraint.

WGAN objective:

Benefits:

• Meaningful loss (correlates with sample quality, unlike JS divergence).
• More stable training; mode collapse less severe.

Enforcing Lipschitz: weight clipping (original WGAN) or gradient penalty (WGAN-GP): penalize $(\|\nabla D\| - 1)^2$ at interpolated samples.

3.6 Conditional GAN (cGAN)

Condition both $G$ and $D$ on class label $\mathbf{y}$:

Enables class-conditional generation, image-to-image translation (pix2pix), text-to-image (early methods).

3.7 StyleGAN (brief)

Progressive growing: train at low resolution first, gradually increase. Stable training of high-resolution generators.

Style-based generator: separate style (appearance) from content. Inject style via AdaIN (Adaptive Instance Normalization) at each resolution level.

4. Normalizing flows (overview)

4.1 Idea

Learn a bijective, differentiable function $f_\theta: \mathbf{x} \to \mathbf{z}$ between data space and a simple latent space ($\mathbf{z} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$).

Exact likelihood via change-of-variables:

The log-determinant of the Jacobian accounts for volume change.

Challenge: the Jacobian determinant is $O(d^3)$ in general. Flows design architectures where the Jacobian is triangular or block-diagonal → $O(d)$ determinant.

4.2 Coupling layers (RealNVP, Glow)

Split $\mathbf{x}$ into $[\mathbf{x}_1, \mathbf{x}_2]$. Coupling layer:

where $s, t$ are arbitrary networks. Jacobian is triangular → $\det = \exp(\sum s(\mathbf{x}_1))$. Invertible: $\mathbf{x}_2 = (\mathbf{z}_2 - t(\mathbf{x}_1)) / \exp(s(\mathbf{x}_1))$.

Glow (Kingma & Dhariwal, 2018) applies flows to image generation with learnable permutations and achieved high-quality faces.

5. Diffusion models

5.1 Overview

Diffusion models (DDPM, Ho et al., 2020; score matching, Song & Ermon, 2019) are currently the dominant generative model for images, audio, and video.

Idea: learn to reverse a gradual noising process. Add noise to data step by step until it becomes pure Gaussian noise; train a neural network to reverse each noising step.

5.2 Forward process (adding noise)

Define a Markov chain that gradually adds Gaussian noise over $T$ steps:

where $\beta_t \in (0,1)$ is the noise schedule (small at first, increases).

Define $\alpha_t = 1 - \beta_t$, $\bar\alpha_t = \prod_{s=1}^t \alpha_s$.

Key property — forward in one step:

So:

As $t \to T$ and $\bar\alpha_T \to 0$: $\mathbf{x}_T \approx \mathcal{N}(\mathbf{0},\mathbf{I})$ (pure noise).

5.3 Reverse process (denoising)

The reverse process $p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)$ is also Gaussian (for small $\beta_t$):

A neural network $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ predicts the noise $\boldsymbol{\varepsilon}$ that was added at step $t$. Then:

5.4 Training objective (simplified)

The variational lower bound (VLB) simplifies to:

Training algorithm:

for each training step:
  1. Sample x_0 ~ q(x_0)  (real data)
  2. Sample t ~ Uniform({1,...,T})
  3. Sample ε ~ N(0, I)
  4. x_t = sqrt(ᾱ_t) * x_0 + sqrt(1-ᾱ_t) * ε
  5. Loss = ||ε - ε_θ(x_t, t)||²
  6. Gradient step on θ

Remarkably simple: the model just learns to predict the noise given a noisy image and a time step.

5.5 Sampling (reverse diffusion)

Start from $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ and iteratively denoise:

for t = T, T-1, ..., 1:
  z ~ N(0,I)  if t > 1, else z = 0
  x_{t-1} = (1/sqrt(α_t)) * (x_t - β_t/sqrt(1-ᾱ_t) * ε_θ(x_t, t)) + σ_t * z

This requires $T$ (typically 1000) forward passes through $\boldsymbol{\epsilon}_\theta$ → slow generation.

5.6 DDIM (Denoising Diffusion Implicit Models)

Song et al. (2020): derive a non-Markovian sampling process with the same marginals as DDPM but with fewer steps (10–100 instead of 1000):

With $\sigma_t = 0$: deterministic sampling (DDIM). Enables 10–50 step generation.

5.7 Latent diffusion models (LDM / Stable Diffusion)

Problem: running diffusion in pixel space is expensive (high-dimensional $\mathbf{x}$).

Rombach et al. (2022): encode images into a low-dimensional latent space with a pretrained VAE, run diffusion there:

1. Encode: $\mathbf{z}_0 = \mathcal{E}(\mathbf{x}_0)$ ($\sim 8\times$ spatial compression).
2. Add noise to $\mathbf{z}_0$ in forward process.
3. Train diffusion model $\boldsymbol{\epsilon}_\theta$ on latent space.
4. Decode: $\hat{\mathbf{x}}_0 = \mathcal{D}(\hat{\mathbf{z}}_0)$.

Much cheaper (smaller latent space) while maintaining image quality.

5.8 Classifier-free guidance (CFG)

Condition the diffusion model on a text prompt (or class) $\mathbf{c}$. Train with and without conditioning (randomly drop $\mathbf{c}$ with probability 0.1):

Guidance scale $w > 0$ amplifies the conditional signal. Higher $w$ → more prompt-aligned but less diverse.

5.9 Score matching connection

The noise-prediction model $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ is related to the score function (gradient of log density):

Score-based generative models (Song & Ermon) and DDPM are equivalent formulations.

6. Evaluation of generative models

6.1 Fréchet Inception Distance (FID)

Compute Inception-v3 features of real ($\mathbf{m}_r, \boldsymbol{\Sigma}_r$) and generated ($\mathbf{m}_g, \boldsymbol{\Sigma}_g$) samples:

Lower FID = more realistic and diverse samples. Standard metric for image generation.

6.2 Inception Score (IS)

High IS = samples are clearly classifiable AND diverse. Doesn't compare to real data distribution (FID is preferred).

6.3 Precision and Recall

Precision: fraction of generated samples that are realistic (near real data manifold). Recall: fraction of real data manifold covered by generated samples.

High precision + low recall: mode coverage poor (generated images look real but limited variety). High recall + low precision: diverse but unrealistic samples.

7. Comparison and when to use what

Current state (2026):

• Images: Diffusion (Stable Diffusion, DALL-E 3, Imagen).
• Text: Autoregressive (GPT-4, LLaMA, Gemini).
• Audio: Diffusion + autoregressive (AudioLM, MusicGen).
• Video: Diffusion (Sora, Stable Video Diffusion).
• Fast generation needed: GAN still competitive. Flow Matching (continuous normalizing flows) emerging as faster alternative to diffusion.

*File: notes/16_generative_models.md — all DL notes complete.*