Boris van Breugel
Nicolas Huynh
Yangming LiDiffusion models are a type of generative model in machine learning that generate data by simulating a reverse diffusion process. They start with random noise and gradually transform it into structured data, such as images or text, by learning the reverse of a predefined forward process that adds noise to the data. This approach allows for generating high-quality, realistic data samples.
Diffusion models are an exciting topic in ML because they have shown remarkable success in generating high-fidelity images, outperforming many traditional generative models like GANs.
For our upcoming Inspiration Exchange session on 13 June, our PhD students will dive into the most recent work of the van der Schaar lab, published at ICML and ICLR 2024, showcasing cutting-edge tools and advanced topics in Diffusion Models.
First, Nicolas Huynh will talk about Time Series Diffusion in the Frequency Domain. Compared to generative modelling for ‘static’ modalities, such as images, text, or tabular data, not much research has been invested in time-series. We approach this overlooked problem due to the importance of generative modelling for privacy concerns, fairness, and data augmentation in downstream tasks in this area.
Our key question is whether we can improve time series diffusion using Fourier analysis by operating in the frequency domain. We investigate how to perform diffusion for time series in the frequency domain and whether it provides any advantages over time domain diffusion.
We theoretically demonstrate that a diffusion process in the time domain is equivalent to diffusion in the frequency domain, providing a method to perform diffusion in the frequency domain. Empirically, we show across various datasets that frequency domain diffusion outperforms time domain diffusion. We then offer an intuition for this improvement using the concept of time and frequency localisation.
Time Series Diffusion in the Frequency Domain
Jonathan Crabbé*, Nicolas Huynh*, Jan Stanczuk, Mihaela van der Schaar
ICML 2024
Abstract
Fourier analysis has been an instrumental tool in the development of signal processing. This leads us to wonder whether this framework could similarly benefit generative modelling. In this paper, we explore this question through the scope of time series diffusion models. More specifically, we analyze whether representing time series in the frequency domain is a useful inductive bias for score-based diffusion models.
By starting from the canonical SDE formulation of diffusion in the time domain, we show that a dual diffusion process occurs in the frequency domain with an important nuance: Brownian motions are replaced by what we call mirrored Brownian motions, characterized by mirror symmetries among their components. Building on this insight, we show how to adapt the denoising score matching approach to implement diffusion models in the frequency domain. This results in frequency diffusion models, which we compare to canonical time diffusion models. Our empirical evaluation on real-world datasets, covering various domains like healthcare and finance, shows that frequency diffusion models better capture the training distribution than time diffusion models.
We explain this observation by showing that time series from these datasets tend to be more localized in the frequency domain than in the time domain, which makes them easier to model in the former case. All our observations point towards impactful synergies between Fourier analysis and diffusion models.
Yangming Li will then discuss two additional recent advances in diffusion modelling. First, he will talk about Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models. Diffusion models are expensive and slow when generating, because of their long iterative process. Reducing the number of iterations would be highly beneficial for reducing compute, cost, and energy consumption, but this usually leads to a degradation in generation quality. The cause of this degradation is the gaussian posterior assumption in the denoising process, which becomes more inaccurate for fewer time steps. Our study demonstrates the impact of the incorrect gaussian posterior assumption on model performance and proposes a new denoising approach, soft mixture denoising, which could help speed up generation.
We show that assuming a simple Gaussian leads to unbounded denoising errors in diffusion models, causing them to fail in approximating data distributions. We introduce soft mixture denoising, a relaxed Gaussian mixture approach that bounds these errors. Experiments on benchmark datasets demonstrate that our method outperforms previous baselines in image generation, particularly with fewer backward iterations.
Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models
Yangming Li, Boris van Breugel, Mihaela van der Schaar
ICLR 2024
Abstract
Because diffusion models have shown impressive performances in a number of tasks, such as image synthesis, there is a trend in recent works to prove (with certain assumptions) that these models have strong approximation capabilities.
In this paper, we show that current diffusion models actually have an expressive bottleneck in backward denoising and some assumption made by existing theoretical guarantees is too strong. Based on this finding, we prove that diffusion models have unbounded errors in both local and global denoising. In light of our theoretical studies, we introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising. SMD not only permits diffusion models to well approximate any Gaussian mixture distributions in theory, but also is simple and efficient for implementation.
Our experiments on multiple image datasets show that SMD significantly improves different types of diffusion models (e.g., DDPM), espeically in the situation of few backward iterations.
We will then move on to our paper On Error Propagation of Diffusion Models. While some anticipate that diffusion models might suffer from error propagation due to their long chains, there is no solid explanation. For instance, conditional random fields (CRFs) have a similar cascade structure but do not face this issue. This paper provides a thorough analysis of error propagation in diffusion models and introduces a simple regularisation method to address it.
Under mild assumptions, we show that cumulative error in diffusion models includes the prediction error of each denoising module and does not diminish over time, confirming error propagation. We then propose a regularisation method based on a tight upper bound of the cumulative error. Experiments demonstrate that our method reduces error propagation and improves performance in image generation.
On Error Propagation of Diffusion Models
Yangming Li, Mihaela van der Schaar
ICLR 2024
Abstract
Although diffusion models (DMs) have shown promising performances in a number of tasks (e.g., speech synthesis and image generation), they might suffer from error propagation because of their sequential structure. However, this is not certain because some sequential models, such as Conditional Random Field (CRF), are free from this problem. To address this issue, we develop a theoretical framework to mathematically formulate error propagation in the architecture of DMs.
The framework contains three elements, including modular error, cumulative error, and propagation equation. The modular and cumulative errors are related by the equation, which interprets that DMs are indeed affected by error propagation.
Our theoretical study also suggests that the cumulative error is closely related to the generation quality of DMs. Based on this finding, we apply the cumulative error as a regularization term to reduce error propagation. Because the term is computationally intractable, we derive its upper bound and design a bootstrap algorithm to efficiently estimate the bound for optimization. We have conducted extensive experiments on multiple image datasets, showing that our proposed regularization reduces error propagation, significantly improves vanilla DMs, and outperforms previous baselines.
We are very much looking forward to discuss our Diffusion Model research with you! Join us at our Inspiration Exchange session at 4pm BST on Thursday, 13 June – sign up here.
Marika Niihori
