DDPM — Denoising Diffusion Probabilistic Models

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256×256 LSUN, we obtain sample quality similar to ProgressiveGAN.

Deep generative models of all kinds have recently exhibited high quality samples in a wide variety of data modalities. Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples, and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable to those of GANs.

This paper presents progress in diffusion probabilistic models. A diffusion probabilistic model is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time. Transitions of this chain are learned to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed. When the diffusion consists of small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians too, allowing for a particularly simple neural network parameterization.

Diffusion models are straightforward to define and efficient to train, but to the best of our knowledge, there has been no demonstration that they are capable of generating high quality samples. We show that diffusion models actually are capable of generating high quality samples, sometimes better than the published results on other types of generative models. In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling. We obtained our best sample quality results using this parameterization, so we consider this equivalence to be one of our primary contributions.

Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models (our models do, however, have log likelihoods better than the large estimates annealed importance sampling has been reported to produce for energy based models and score matching). We find that the majority of our models' lossless codelengths are consumed to describe imperceptible image details. We present a more refined analysis of this phenomenon in the language of lossy compression, and we show that the sampling procedure of diffusion models is a type of progressive decoding that resembles autoregressive decoding along a bit ordering that vastly generalizes what is normally possible with autoregressive models.

Diffusion models are latent variable models of the form $p_\theta(x_0) := \int p_\theta(x_{0:T})\,dx_{1:T}$, where $x_1,\dots,x_T$ are latents of the same dimensionality as the data $x_0 \sim q(x_0)$. The joint distribution $p_\theta(x_{0:T})$ is called the reverse process, and it is defined as a Markov chain with learned Gaussian transitions starting at $p(x_T) = \mathcal{N}(x_T; \mathbf{0}, \mathbf{I})$.

What distinguishes diffusion models is that the approximate posterior $q(x_{1:T}\mid x_0)$, called the forward process or diffusion process, is fixed to a Markov chain that gradually adds Gaussian noise according to a variance schedule $\beta_1,\dots,\beta_T$.

Training is performed by optimizing the usual variational bound on negative log likelihood. A notable property of the forward process is that it admits sampling $x_t$ at an arbitrary timestep $t$ in closed form: using $\alpha_t := 1-\beta_t$ and $\bar\alpha_t := \prod_{s=1}^{t}\alpha_s$, we have $q(x_t\mid x_0) = \mathcal{N}(x_t; \sqrt{\bar\alpha_t}\,x_0, (1-\bar\alpha_t)\mathbf{I})$. Efficient training is therefore possible by optimizing random terms of $L$ with stochastic gradient descent. Further improvements come from variance reduction by rewriting $L$ as a sum of KL divergences between tractable Gaussian posteriors.

We establish a new explicit connection between diffusion models and denoising score matching that leads to a simplified, weighted variational bound objective. We propose predicting the noise $\epsilon$ rather than the posterior mean $\tilde\mu$, motivated by the analysis of $L_t$. With fixed isotropic variance $\sigma_t^2 \mathbf{I}$, the reverse mean can be expressed in terms of $\epsilon$, linking training to multi-scale denoising.

We found it beneficial to sample quality to train on the following variant of the variational bound: $L_\text{simple}(\theta) := \mathbb{E}_{t,x_0,\epsilon}\big[\lVert \epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\, t)\rVert^2\big]$, where $t$ is uniform between $1$ and $T$. Since our simplified objective discards the weighting in the true bound, it is a weighted variational bound that emphasizes different aspects of reconstruction. Our diffusion process setup causes the simplified objective to down-weight loss terms corresponding to small $t$, so the network can focus on more difficult denoising tasks at larger $t$.

We set $T = 1000$ for all experiments. We set the forward process variances to constants increasing linearly from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$. To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ with group normalization. Parameters are shared across time, specified using Transformer sinusoidal position embedding. We use self-attention at the 16×16 feature map resolution.

On CIFAR10, with our FID score of 3.17, our unconditional model achieves better sample quality than most models in the literature, including class conditional models. Our Inception score is 9.46 ± 0.11. We find that training on the true variational bound yields better codelengths than training on the simplified objective, as expected, but the latter yields the best sample quality.

In ablation studies, we find that predicting $\tilde\mu$ works well only when trained on the true variational bound. Learning reverse process variances leads to unstable training and poorer sample quality. Predicting $\epsilon$ performs approximately as well as predicting $\tilde\mu$ under the true bound, but much better when trained with $L_\text{simple}$ (IS 9.46 / FID 3.17 vs. baselines).

We have presented high quality image samples using diffusion models, and we have found connections among diffusion models and variational inference for training Markov chains, denoising score matching and annealed Langevin dynamics, autoregressive models, and progressive lossy compression. Since diffusion models seem to have excellent inductive biases for image data, we look forward to investigating their utility in other data modalities and as components in other types of generative models and machine learning systems.