What is Data-Centric AI?
AI-powered applications are becoming increasingly widespread across many areas and industries, including e-commerce, finance, manufacturing, medicine, and many more. However, there are many considerations necessary to successfully develop robust and reliable ML systems, which can often be overlooked.
We’re aiming to change that with a data-centric AI lens. But, what does it mean?
The current paradigm in machine learning is model-centric AI. The data is considered a fixed and static asset (e.g. tabular data in a .csv file, a database, a language corpus or an image repository). The data is often considered somewhat external to the machine learning process. The focus is then on model iteration, whether it is new model architectures, novel loss functions or optimisers – with the goal of improving predictive performance for a fixed benchmark.
Of course, this agenda is important — but we need more reliable ML systems. We believe that the current focus on models and architectures as a panacea in the ML community is often a source of brittleness in real-world applications. We believe that the data work, often undervalued as merely operational, is key to unlocking reliable ML systems.
In data-centric AI, we seek to give data centre stage. Data-centric AI views model or algorithmic refinement as less important (and in certain settings, algorithmic development is even considered as a solved problem), and instead seeks to systematically improve the data used by ML systems.
We go further and call for an expanded definition of data-centric AI such that a data-centric lens is applicable to end-to-end pipelines.
DEFINITION
Data-centric AI encompasses methods and tools to systematically: characterise, evaluate and generate the underlying data used to train and evaluate models
At the ML pipeline level, this means that the considerations at each stage should be informed in a data-driven manner.
We term this a data-centric lens. Since data is the fuel for any ML system, we should keep a sharp focus on the data, yet rather than ignoring the model, we should leverage the data-driven insights as feedback to systematically improve the model.
DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems.
DC-Check is an actionable checklist-style framework to elicit data-centric considerations through the different stages of the ML pipeline: Data, Training, Testing, and Deployment. This data-centric lens on development of ML systems aims to promote thoughtfulness and transparency prior to system development.
Data-Centric AI Research Area
Our approach is to build a reference guide for both clinical practitioners and researchers, highlighting specific data-centric AI challenges and research opportunities.
Here, we have created a comprehensive overview graphic in order to communicate the design of data-centric ML pipelines.
Click on the specific areas in the graphic to learn more about our current work.
Data quality - Data characterisation
Unlock the power of data characterisation: Learn how techniques like Data-IQ, TRIAGE, and Datagnosis help audit datasets, boost model efficiency, and improve training by identifying easy, ambiguous, and unlearnable samples.
Explore these tools to enhance your machine learning projects.
Synthetic data for data improvement
Discover how synthetic data can enhance real datasets by augmenting small samples and ensuring fairness. By applying data characterisation techniques, we can guide synthetic data generation to reflect real-world complexities while using novel metrics to ensure quality.
Uncertainty Estimation
Uncertainty estimation is vital to ensure trustworthy predictions from ML models. In particular, we desire that models ‘know what they don’t know’.
Read more: https://www.vanderschaar-lab.com/trustworthy-predictions/
Data monitoring
Identification and appropriate handling of inconsistencies in data at deployment time is crucial to using machine learning models reliably.
Read more: https://www.vanderschaar-lab.com/data-monitoring/
Model testing — stress test scenarios and evaluations beyond subgroups
Machine learning models fail in real-world scenarios! This is especially worrying in high-stakes settings like healthcare where we might have unreliable model performance for different subgroups or across different distributions.
MICCAI 2024 Tutorial: Data-centric & Dynamic learning
IJCAI 2023 Tutorial: Data-Centric AI
NeurIPS 2023 Tutorial: Data-Centric AI for reliable and responsible AI
Our most recent work
ICLR
2025
Going Beyond Static: Understanding Shifts with Time-Series Attribution
This paper propose a time-series shift attribution framework that attributes performance degradation from various types of shifts to each temporal data property in a detailed manner, supported by theoretical analysis and empirical results.
In review
2025
SynthCraft: an AI partner for synthetic data generation to support data access and augmentation in healthcare
Code: https://github.com/vanderschaarlab/SynthCraft
This paper introduces SynthCraft, a human-in-the-loop framework that uses LLMs together with reinforcement-learning–based reasoning to orchestrate synthetic data generation workflows. It enables principled evaluation of privacy, statistical fidelity, and downstream utility across tabular and genomic datasets, supporting transparent and flexible use of state-of-the-art synthetic data methods.
Journal of Data-centric Machine Learning Research
2025
DMLR: Towards Human-Guided, Data-Centric LLM Co-Pilots
This paper proposes CliMB-DC, a human-guided, data-centric framework for LLM-based ML co-pilots. By combining domain expert input with structured multi-agent reasoning and data-centric tools, it enables robust handling of real-world data issues and turns messy datasets into ML-ready inputs, outperforming existing co-pilot approaches on healthcare data.
NeurIPS
2024
Matchmaker: Self-Improving Large Language Model Programs for Schema Matching
This paper proposes Matchmaker, a language-model–based framework for automatically matching fields across different data schemas. It works without labeled training data, self-improves in a zero-shot setting, and outperforms existing methods on real-world medical datasets, enabling faster data integration for ML-ready systems.
NeurIPS
2024
Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models
In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism.
NeurIPS
2024
TRIAGE: Characterizing and auditing training data for improved regression
We introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples’ training dynamics and characterize samples as under-, over-, or well-estimated by the model.
Journal of Data-centric Machine Learning Research
2024
DMLR: You can’t handle the (dirty) truth: Data-centric insights improve pseudo-labeling
This paper challenges the assumption that labeled data used in pseudo-labeling is always reliable, showing that label noise and ambiguity can significantly affect performance. It introduces DIPS, a data-centric framework that analyzes learning dynamics to select high-quality labeled and pseudo-labeled samples, improving robustness and data efficiency across tabular and image datasets.
ICML
2024
Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes
We introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. However, not all the data generated by LLMs will improve downstream utility, as for any generative model. Consequently, we introduce a principled curation mechanism, leveraging learning dynamics, coupled with confidence and uncertainty metrics, to obtain a high-quality dataset.
ICLR
2024
Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI
We propose the Hardness Characterization Analysis Toolkit (H-CAT), which supports comprehensive and quantitative benchmarking of HCMs across the hardness taxonomy and can easily be extended to new HCMs, hardness types, and datasets. We use H-CAT to evaluate 13 different HCMs across 8 hardness types.
Journal of Data-centric Machine Learning Research
2024
DMLR: Data-centric Machine Learning Research – Past, Present and Future
We outline the relevance of community engagement and infrastructure development for the creation of next-generation public datasets that will advance machine learning science. We chart a path forward as a collective effort to sustain the creation and maintenance of these datasets and methods towards positive scientific, societal and business impact.
Journal of Data-centric Machine Learning Research
2024
DMLR: When is Off-Policy Evaluation Useful? A Data-Centric Perspective
This paper proposes DataCOPE, a framework for off-policy evaluation. It assesses whether and how reliably a logged dataset can be used to evaluate a target policy, highlighting potential failure cases before real-world deployment.
AISTATS
2024
DAGnosis: Localized Identification of Data Inconsistencies using Structures
We solve two fundamental limitations using directed acyclic graphs (DAGs) to encode the training set’s features probability distribution and independencies as a structure. Our method, called DAGnosis, leverages these structural interactions to bring valuable and insightful data-centric conclusions…
NeurIPS
2023
Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data
In this work, we introduce 3S Testing, a deep generative modeling framework to facilitate model evaluation by generating synthetic test sets for small subgroups and simulating distributional shifts. Our experiments demonstrate that 3S Testing outperforms traditional baselines — including real test data alone — in estimating model performance on minority subgroups and under plausible distributional shifts.
Inspiration Exchange
2 October 2023
Inspiration Exchange 30
In this jubilee session, we explore Data-Centric AI, share recent advances and discuss new, unexplored directions that could yet provide new breakthroughs with our guests we are excited to be joined by: Luis Oala (Head of Machine Learning, Dotphoton), Dr Manil Maskey (Senior Research Scientist, Office of Chief Science Data Officer, NASA), and Dr Peter Mattson (Senior Staff Engineer, Google; President of MLCommons.org).
AISTATS
2023
Improving Adaptive Conformal Prediction Using Self-Supervised Learning
We solve two fundamental limitations using directed acyclic graphs (DAGs) to encode the training set’s features probability distribution and independencies as a structure. Our method, called DAGnosis, leverages these structural interactions to bring valuable and insightful data-centric conclusions…
NeurIPS
2022
Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data
we propose Data-IQ, a framework to systematically stratify examples into subgroups with respect to their outcomes. We do this by analyzing the behavior of individual examples during training, based on their predictive confidence and, importantly, the aleatoric (data) uncertainty.
ICML
2022
HyperImpute:
Generalized Iterative Imputation with Automatic Model Selection
We propose HyperImpute, a generalized iterative imputation framework for adaptively and automatically configuring column-wise models and their hyperparameters. Practically, we provide a concrete implementation with out-of-the-box learners, optimizers, simulators, and extensible interfaces.
ICML
2022
Data-SUITE: Data-centric identification of in-distribution incongruous examples
We propose a paradigm shift with Data-SUITE: a datacentric framework to identify these regions, independent of a task-specific model. DATA-SUITE leverages copula modeling, representation learning, and conformal prediction to build featurewise confidence interval estimators based on a set of training instances.
ACM Transactions on Computing for Healthcare
2022
MARS: Assisting Human with Information Processing Tasks Using Machine Learning
The main contribution is a novel framework called Machine Assisted Record Selection (MARS). Instead of today’s standard practice of relying on human experts to manually decide the order of records for processing, MARS learns the optimal record selection via an online learning algorithm.
ICML
2020
Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift
In this paper, we develop an approximate Bayesian inference scheme based on posterior regularisation, wherein unlabelled target data are used as “pseudo-labels” of model confidence that are used to regularise the model’s loss on labelled source data. We show that this approach significantly improves the accuracy of uncertainty quantification on covariate-shifted data sets, with minimal modification to the underlying model architecture.
arXiv
2019
Lifelong Bayesian Optimization
In this paper, we present Lifelong Bayesian Optimization (LBO), an online, multitask Bayesian optimization (BO) algorithm designed to solve the problem of model selection for datasets arriving and evolving over time. In LBO, we exploit the correlation between black-box functions by using components of previously learned functions to speed up the learning process for newly arriving datasets.
MC4HC
2019
ASAC: Active Sensing using Actor-Critic models
In this paper, we propose a novel deep learning framework, which we call ASAC (Active Sensing using Actor-Critic models) to address this problem. ASAC consists of two networks: a selector network and a predictor network. The selector network uses previously selected observations to determine what should be observed in the future. The predictor network uses the observations selected by the selector network to predict a label, providing feedback to the selector network (well-selected variables should be predictive of the label).
ICLR
2018
Deep Sensing:
Active Sensing using Multi-directional Recurrent Neural Networks
We develop a novel deep learning architecture: Deep Sensing. At training time, Deep Sensing learns how to issue predictions at various cost-performance points. To do this, it creates multiple representations at various performance levels associated with different measurement rates (costs).
