van der Schaar Lab

Advanced Topics in Diffusion Models

Diffusion models are a type of generative model in machine learning that generate data by simulating a reverse diffusion process. They start with random noise and gradually transform it into structured data, such as images or text, by learning the reverse of a predefined forward process that adds noise to the data. This approach allows for generating high-quality, realistic data samples.

Diffusion models are an exciting topic in ML because they have shown remarkable success in generating high-fidelity images, outperforming many traditional generative models like GANs.

First, Nicolas Huynh will talk about Time Series Diffusion in the Frequency Domain. Compared to generative modelling for ‘static’ modalities, such as images, text, or tabular data, not much research has been invested in time-series. We approach this overlooked problem due to the importance of generative modelling for privacy concerns, fairness, and data augmentation in downstream tasks in this area.

Our key question is whether we can improve time series diffusion using Fourier analysis by operating in the frequency domain. We investigate how to perform diffusion for time series in the frequency domain and whether it provides any advantages over time domain diffusion.

We theoretically demonstrate that a diffusion process in the time domain is equivalent to diffusion in the frequency domain, providing a method to perform diffusion in the frequency domain. Empirically, we show across various datasets that frequency domain diffusion outperforms time domain diffusion. We then offer an intuition for this improvement using the concept of time and frequency localisation.

Time Series Diffusion in the Frequency Domain

Jonathan Crabbé*, Nicolas Huynh*, Jan Stanczuk, Mihaela van der Schaar

ICML 2024

Abstract

Yangming Li will then discuss two additional recent advances in diffusion modelling. First, he will talk about Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models. Diffusion models are expensive and slow when generating, because of their long iterative process. Reducing the number of iterations would be highly beneficial for reducing compute, cost, and energy consumption, but this usually leads to a degradation in generation quality. The cause of this degradation is the gaussian posterior assumption in the denoising process, which becomes more inaccurate for fewer time steps. Our study demonstrates the impact of the incorrect gaussian posterior assumption on model performance and proposes a new denoising approach, soft mixture denoising, which could help speed up generation.

We show that assuming a simple Gaussian leads to unbounded denoising errors in diffusion models, causing them to fail in approximating data distributions. We introduce soft mixture denoising, a relaxed Gaussian mixture approach that bounds these errors. Experiments on benchmark datasets demonstrate that our method outperforms previous baselines in image generation, particularly with fewer backward iterations.

Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models

Abstract

We will then move on to our paper On Error Propagation of Diffusion Models. While some anticipate that diffusion models might suffer from error propagation due to their long chains, there is no solid explanation. For instance, conditional random fields (CRFs) have a similar cascade structure but do not face this issue. This paper provides a thorough analysis of error propagation in diffusion models and introduces a simple regularisation method to address it.

Under mild assumptions, we show that cumulative error in diffusion models includes the prediction error of each denoising module and does not diminish over time, confirming error propagation. We then propose a regularisation method based on a tight upper bound of the cumulative error. Experiments demonstrate that our method reduces error propagation and improves performance in image generation.

On Error Propagation of Diffusion Models

Abstract

We are very much looking forward to discuss our Diffusion Model research with you! Join us at our Inspiration Exchange session at 4pm BST on Thursday, 13 June – sign up here.

Boris van Breugel

Boris van Breugel most recently completed a MSc in Machine Learning at University College London. His study was supported under a Young Talent Award by Prins Bernhard Cultuurfonds, and a VSBfonds scholarship. Prior to this, he received a MASt in Applied Mathematics from the University of Cambridge.

Reflecting his broad research background, Boris’ current research interests range from model interpretability to learning from missing data, from modelling treatment effects to high-dimensional omics data.

While studying for his MSc in Machine Learning at UCL, Boris developed a model to detect Alzheimer’s disease in different forms of medical imaging data, potentially enabling diagnosis at an earlier stage and thereby aiding the development of more effective treatment plans. He found the healthcare domain uniquely challenging and rewarding, and decided to continue research in the domain.

As a PhD student with the van der Schaar Lab, Boris aims to develop methods for finding meaningful structure in omics data—in essence, he says, “the amount of omics data is increasing at a huge speed, and machine learning methods can allow us to interpret and make sense of all this data.”

Boris’ studentship is funded by the Office of Naval Research (ONR).

Nicolas Huynh

Yangming Li