van der Schaar Lab

van der Schaar Lab to present the most papers of any lab in Europe at ICLR 2024

Researchers at the van der Schaar Lab have had 10 papers accepted to this year’s International Conference on Learning Representations (ICLR), the most of any lab in Europe.

ICLR, one of three leading machine learning (ML) conferences, features important contributions from researchers working in both the public and private sectors. The van der Schaar Lab’s prominence at ICLR reaffirms its status as one of the world’s best ML research outfits, following as it does a similarly strong showing at NeurIPS (another of the three big ML conferences), where the lab ranked second in contributions.

The papers themselves offer technical and conceptual advances in a variety of areas, including the technology underpinning generative AI and the transparent and interpretable models required in high-stakes industries like medicine and finance. They are also notable for their quality: two of the papers received peer review ratings that place them in the top five percent of all accepted papers, while a further three papers are in the top 30 percent.

Taken together, the papers are a powerful demonstration of the van der Schaar Lab’s vision for reality-centric AI, a research agenda which demands a reorientation away from solving simple-to-define yet challenging-to-solve problems (like playing games or making chemical discoveries) towards an approach that puts the unavoidable complexity of the real world at the heart of designing, training, testing, and deploying AI models.

Mihaela van der Schaar, the John Humphrey Plummer Professor of Machine Learning, Artificial Intelligence and Medicine at the University of Cambridge, said: “I am very proud of my students and the work they have done on these papers. Their out-of-the-box thinking has produced some exceptional research, and it’s extremely gratifying to see that fact recognised by ICLR. When done right – when it is turned to real-world situations that are messy and complex – machine learning has the potential to greatly increase what humanity can achieve. In this regard, my students’ work on reality-centric AI is a beacon that I hope more of the machine learning community will follow.” 

Generative AI research

Five of the accepted research papers lay out innovative advancements around large language models (LLMs) and diffusion models, which underpin the generative AI tools used to create text and images, respectively.

The most notable of these papers is L2MAC: Large Language Model Automatic Computer for Extensive Code Generation (full paper). As its title indicates, the paper unveils an LLM architecture that is capable of generating an entire software application’s worth of high-quality, self-tested and editable code. Current public LLMs, such as ChatGPT, are structurally incapable of generating anything like this amount of code due to foundational architectural constraints.

Samuel Holt, lead author of the paper, explained: “LLM frameworks have a goldfish-like memory: after generating a certain number of characters, they reach what is known as the context window constraint. From that point on, they have no memory of the characters outside their context window. This is an insurmountable problem when it comes to tasks that involve the generation of large cohesive outputs, such as code bases where each line of code could crucially depend on any other line of code.”

Remarkably, in a global ranking of LLM-generated code bases, L2MAC currently places fourth, alongside and above models with far bigger budgets behind them.

Two other LLM-related papers have been accepted to ICLR: LLMs to Enhance Bayesian Optimization (full paper) and Query-Dependent Prompt Evaluation and Optimization with Offline Inverse Reinforcement Learning (full paper). The papers investigate how existing ML techniques can be enhanced by LLMs, and vice versa. They join a thriving body of research pushing the capabilities and applications of LLMs far beyond the superficial interactions of everyday users. 

Diffusion models, the technology behind AI image generation tools such as OpenAI’s DALL-E, are the subjects of two of the lab’s ICLR papers. Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models (full paper) and On Error Propagation of Diffusion Models (full paper) put forward technical proposals that limit errors in the generation processes of diffusion models, yielding higher-quality outputs.

Machine learning research

The lab’s five remaining ICLR papers, although highly varied in their approaches, each find new methods for deepening understanding of datasets.

A Neural Framework for Generalized Causal Sensitivity Analysis (full paper) tackles the issue of unobserved confounding – that is, when unmeasured variables have had unknown effects on actions and outcomes. Unobserved confounding is a pervasive issue in ML and is particularly limiting in datasets of medical treatments and their effects. The paper outlines a neural framework that reveals unobserved confounding and leverages it to better draw more accurate conclusions about treatment effects. It is a perfect example of reality-centric AI in action.

Inferring the effect of treatments is also the subject of ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inference (full paper). Where previous inference machines have all employed neural networks, the lab has opted for a new type of solution based on a sophisticated mathematical equation. This solution type has substantial advantages over predecessors, especially in a medical setting, by virtue of its greater interpretability and flexibility.

Towards Transparent Time Series Forecasting (full paper) is another sophisticated contribution to transparent ML. For a time series model to forecast accurately, it must work on two levels: both the properties of individual datum and the trends in the data as a whole must be understood. The paper introduces the concept of “bi-level transparency,” which provides a holistic understanding of a trajectory at both the level of properties and trends. The outlined model has many real-world applications in domains such as finance, medicine and physics.

The final two papers accepted to ICLR explore expertise and hardness. Defining Expertise: Applications to Treatment Effect Estimation (full paper) greatly enriches how to conceive of the different types of expertise (ie, the domain knowledge of decision makers) encoded in datasets. Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI (full paper) adds similar precision to the definition of how hard a dataset might be to train an ML model. 

Find the lab at ICLR 2024

This year, ICLR will take place 7–11 May in Vienna, Austria. The authors of the papers look forward to presenting and discussing their research, but will also be using the conference as an opportunity to sample Vienna’s famous coffee culture. They encourage any fellow attendees – including researchers, students and members of the press – to get in touch ahead of time so that a rendez-vous at one of the city’s many superb cafes can be arranged.

About the lab

The van der Schaar Lab is a world-leading research group led by Mihaela van der Schaar, John Humphrey Plummer Professor of Machine Learning, AI and Medicine at the University of Cambridge. It develops cutting-edge machine learning & AI theory and methods, with the goal of developing reality-centric AI.