AI4Science, a rapidly evolving field at the intersection of artificial intelligence and scientific research, is revolutionising how scientists tackle complex challenges. By harnessing machine learning algorithms and data-driven methodologies, AI4Science accelerates the discovery of underlying patterns and principles in natural phenomena across disciplines like physics, chemistry, and biology. One particularly exciting frontier in this field is equation discovery—where machine learning models infer underlying mathematical relationships directly from data. This capability empowers researchers to uncover the hidden laws governing complex systems, producing models that are not just accurate but inherently interpretable.
Our lab has made significant strides in this area by developing techniques that extract a variety of mathematical equations, such as partial and ordinary differential equations, from data. Moreover, we are pioneering new approaches that go beyond traditional closed-form equations, aiming for models that remain transparent yet offer enhanced flexibility in modelling complex observations.
Equation Discovery
The area of machine learning that aims to discover equations from data is called symbolic regression. Unlike traditional regression which fits data to a pre-defined equation, symbolic regression seeks the best-fitting mathematical expression from a space of potential solutions. This means generating closed-form mathematical expressions composed of a finite set of variables, constants, common operations (e.g., addition, multiplication), and functions (trigonometric, exponential, etc.). Symbolic regression is crucial for unveiling concise, human-readable models.
Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models
S Holt, Z Qian, T Liu, J Weatherall, M van der Schaar
NeurIPS 2024
Abstract
The discovery of dynamical systems is crucial across a range of fields, including pharmacology, epidemiology, and physical sciences. Accurate and interpretablemodeling of these systems is essential for understanding complex temporal processes, optimizing interventions, and minimizing adverse effects. In pharmacology, for example, precise modeling of drug dynamics is vital to maximize therapeutic efficacy while minimizing patient harm, as in chemotherapy.
However, current models, often developed by human experts, are limited by high cost, lack of scalability, and restriction to existing human knowledge. In this paper, we present the Data-Driven Discovery (D3)framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable models of dynamical systems, demonstrated here with pharmacological applications. Unlike traditional methods, D3 enables the LLM to propose, acquire, and integrate new features, validate, and compare dynamical systems models, uncovering new insights into pharmacokinetics.
Experiments on a pharmacokinetic Warfarin dataset reveal that D3 identifies a new plausible model that is well-fitting, highlighting its potential for precision dosing in clinical applications.
ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inference
K Kacprzyk*, S Holt*, J Berrevoets*, Z Qian, M van der Schaar
ICLR 2024
Abstract
Inferring unbiased treatment effects has received widespread attention in the machine learning community. In recent years, our community has proposed numerous solutions in standard settings, high-dimensional treatment settings, and even longitudinal settings.
While very diverse, the solution has mostly relied on neural networks for inference and simultaneous correction of assignment bias. New approaches typically build on top of previous approaches by proposing new (or refined) architectures and learning algorithms. However, the end result — a neural-network-based inference machine — remains unchallenged. In this paper, we introduce a different type of solution in the longitudinal setting: a closed-form ordinary differential equation (ODE). While we still rely on continuous optimization to learn an ODE, the resulting inference machine is no longer a neural network. Doing so yields several advantages such as interpretability, irregular sampling, and a different set of identification assumptions.
Above all, we consider the introduction of a completely new type of solution to be our most important contribution as it may spark entirely new innovations in treatment effects in general. We facilitate this by formulating our contribution as a framework that can transform any ODE discovery method into a treatment effects method.
D-CIPHER: Discovery of Closed-form Partial Differential Equations
K Kacprzyk, Z Qian, M van der Schaar
NeurIPS 2023
Abstract
Closed-form differential equations, including partial differential equations and higher-order ordinary differential equations, are one of the most important tools used by scientists to model and better understand natural phenomena. Discovering these equations directly from data is challenging because it requires modeling relationships between various derivatives that are not observed in the data (equation-data mismatch) and it involves searching across a huge space of possible equations.
Current approaches make strong assumptions about the form of the equation and thus fail to discover many well-known systems. Moreover, many of them resolve the equation-data mismatch by estimating the derivatives, which makes them inadequate for noisy and infrequently sampled systems.
To this end, we propose D-CIPHER, which is robust to measurement artifacts and can uncover a new and very general class of differential equations. We further design a novel optimization procedure, CoLLie, to help D-CIPHER search through this class efficiently. Finally, we demonstrate empirically that it can discover many well-known equations that are beyond the capabilities of current methods.
Deep Generative Symbolic Regression
S Holt, Z Qian, M van der Schaar
ICLR 2023
Abstract
Symbolic regression (SR) aims to discover concise closed-form mathematical equations from data, a task fundamental to scientific discovery. However, the problem is highly challenging because closed-form equations lie in a complex combinatorial search space. Existing methods, ranging from heuristic search to reinforcement learning, fail to scale with the number of input variables.
We make the observation that closed-form equations often have structural characteristics and invariances (e.g., the commutative law) that could be further exploited to build more effective symbolic regression solutions. Motivated by this observation, our key contribution is to leverage pre-trained deep generative models to capture the intrinsic regularities of equations, thereby providing a solid foundation for subsequent optimization steps.
We show that our novel formalism unifies several prominent approaches of symbolic regression and offers a new perspective to justify and improve on the previous ad hoc designs, such as the usage of cross-entropy loss during pre-training. Specifically, we propose an instantiation of our framework, Deep Generative Symbolic Regression (DGSR). In our experiments, we show that DGSR achieves a higher recovery rate of true equations in the setting of a larger number of input variables, and it is more computationally efficient at inference time than state-of-the-art RL symbolic regression solutions.
D-CODE: Discovering Closed-form ODEs from Observed Trajectories
Z Qian, K Kacprzyk, M van der Schaar
ICLR 2022 Spotlight
Abstract
For centuries, scientists have manually designed closed-form ordinary differential equations (ODEs) to model dynamical systems. An automated tool to distill closed-form ODEs from observed trajectories would accelerate the modeling process. Traditionally, symbolic regression is used to uncover a closed-form prediction function with label-feature pairs as training examples.
However, an ODE models the time derivative of a dynamical system, e.g. , and the “label” is usually *not* observed. The existing ways to bridge this gap only perform well for a narrow range of settings with low measurement noise, frequent sampling, and non-chaotic dynamics. In this work, we propose the Discovery of Closed-form ODE framework (D-CODE), which advances symbolic regression beyond the paradigm of supervised learning. D-CODE leverages a novel objective function based on the variational formulation of ODEs to bypass the unobserved time derivative.
For formal justification, we prove that this objective is a valid proxy for the estimation error of the true (but unknown) ODE. In the experiments, D-CODE successfully discovered the governing equations of a diverse range of dynamical systems under challenging measurement settings with high noise and infrequent sampling.
Learning outside the Black-Box: The pursuit of interpretable models
J Crabbé, Y Zhang, W R Zame, M van der Schaar
NeurIPS 2020
Abstract
Machine Learning has proved its ability to produce accurate models but the deployment of these models outside the machine learning community has been hindered by the difficulties of interpreting these models. This paper proposes an algorithm that produces a continuous global interpretation of any given continuous black-box function.
Our algorithm employs a variation of projection pursuit in which the ridge functions are chosen to be Meijer G-functions, rather than the usual polynomial splines. Because Meijer G-functions are differentiable in their parameters, we can tune the parameters of the representation by gradient descent; as a consequence, our algorithm is efficient. Using five familiar data sets from the UCI repository and two familiar machine learning algorithms, we demonstrate that our algorithm produces global interpretations that are both highly accurate and parsimonious (involve a small number of terms).
Our interpretations permit easy understanding of the relative importance of features and feature interactions. Our interpretation algorithm represents a leap forward from the previous state of the art.
Demystifying Black-box Models with Symbolic Metamodels
A M Alaa, M van der Schaar
NeurIPS 2019
Abstract
Understanding the predictions of a machine learning model can be as crucial as the model’s accuracy in many application domains. However, the black-box nature of most highly-accurate (complex) models is a major hindrance to their interpretability. To address this issue, we introduce the symbolic metamodeling framework — a general methodology for interpreting predictions by converting “black-box” models into “white-box” functions that are understandable to human subjects.
A symbolic metamodel is a model of a model, i.e., a surrogate model of a trained (machine learning) model expressed through a succinct symbolic expression that comprises familiar mathematical functions and can be subjected to symbolic manipulation. We parameterize symbolic metamodels using Meijer G-functions — a class of complex-valued contour integrals that depend on scalar parameters, and whose solutions reduce to familiar elementary, algebraic, analytic and closed-form functions for different parameter settings.
This parameterization enables efficient optimization of metamodels via gradient descent, and allows discovering the functional forms learned by a machine learning model with minimal a priori assumptions. We show that symbolic metamodeling provides an all-encompassing framework for model interpretation — all common forms of global and local explanations of a model can be analytically derived from its symbolic metamodel.
Beyond Equations
While closed-form equations offer clear, interpretable representations that enhance our understanding of underlying phenomena, our lab is exploring models that go beyond these constraints. We are pursuing shape arithmetic expressions and other advanced structures that retain transparency while providing more adaptable ways to model complex behaviours.
Towards Transparent Time Series Forecasting
K Kacprzyk, T Liu, M van der Schaar
ICLR 2024
Abstract
Transparent machine learning (ML) models are essential for ensuring interpretability and trustworthiness in decision-making systems, particularly in high-stakes domains such as healthcare, finance, and criminal justice. While transparent machine learning models have been proposed for classification and regression, time series forecasting presents some unique challenges for ensuring transparency.
In particular, currently used bottom-up approaches that focus on the values of the time series at specific time points (usually regularly spaced) do not provide a holistic understanding of the entire time series. This limits the applicability of ML in many critical areas. To open up these domains for ML, we propose a top-down framework of bi-level transparency, which involves understanding the higher-level trends and the lower-level properties of the predicted time series.
Applying this framework, we develop TIMEVIEW, a transparent ML model for time series forecasting based on static features, complemented with an interactive visualization tool. Through a series of experiments, we demonstrate the efficacy and interpretability of our approach, paving the way for more transparent and reliable applications of ML in various domains.
Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations
K Kacprzyk, M van der Schaar
AISTATS 2024
Abstract
Symbolic regression has excelled in uncovering equations from physics, chemistry, biology, and related disciplines. However, its effectiveness becomes less certain when applied to experimental data lacking inherent closed-form expressions. Empirically derived relationships, such as entire stress-strain curves, may defy concise closed-form representation, compelling us to explore more adaptive modeling approaches that balance flexibility with interpretability.
In our pursuit, we turn to Generalized Additive Models (GAMs), a widely used class of models known for their versatility across various domains. Although GAMs can capture non-linear relationships between variables and targets, they cannot capture intricate feature interactions. In this work, we investigate both of these challenges and propose a novel class of models, Shape Arithmetic Expressions (SHAREs), that fuses GAM’s flexible shape functions with the complex feature interactions found in mathematical expressions. SHAREs also provide a unifying framework for both of these approaches. We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints based on the model’s size.
AI4Science in our Engagement Sessions
Inspiration Exchange Session 16
Inspiration Exchange Session 36
Related Research Areas
Research Pillar on Interpretable ML
