Why Tabular Foundation Models Should Be a Research Priority

Recent text and image foundation models are incredibly impressive, and these models are attracting an ever-increasing portion of research resources (see figure below, representing different modalities in foundation model research across recent ML conferences).

In our ICML 2024 position paper, we argue why foundation model research should explore other modalities more, in particular tabular data. We believe the time is now to start developing tabular foundation models, or what we coin a Large Tabular Model (LTM). LTMs could revolutionise the way science and ML use tabular data: not as single datasets that are analysed in a vacuum but contextualised with respect to related datasets.

Why we should care about tabular data

Tabular data is ubiquitous in the real world, from electronic healthcare records to census data, from finance to natural sciences. These datasets are the fundament for the progress of scientific knowledge and to influence public policy. And yet, the tabular domain offers uniquely exciting, large, unsolved challenges for researchers. For example, in contrast to other domains, deep learning models still often perform on par with tree-based models.

At last, humans are very limited at understanding and analysing tabular data themselves (cf. image and text), hence a good LTM could truly extend human capabilities.

Why the tabular domain is so challenging

In our paper, we discuss the requirements for an LTM to be adaptable to a wide range of different tasks. We consider four requirements:

Mixed-type Columns – LTMs should be able to handle the different types of data that are common in tables, e.g. numerical, categorical, date-time, and missing data.

Cross-dataset Modelling – Tabular datasets only cover a specific topic and are often moderately sized. Consequently, to enable large-scale training and broad applicability we need one LTM to be trained on different datasets. This requires a capability to model heterogeneous feature spaces—i.e. feature spaces with different (numbers of) features that could carry very different meanings.

Textual Context – The meaning of tabular data is often dependent on contextual metadata, e.g. the dataset description, column names, and category names. An LTM should leverage this information.

Invariance/Equivariance w.r.t. column order – Column order is usually arbitrary for tables. We want an LTM to reflect this and have output that is invariant or equivariant to input permutations.

Because of these requirements, building an LTM is conceptually and architecturally complicated and poses unique and exciting challenges for researchers. We discuss how current work on LTMs falls short of this, and why LLM-based approaches are not very suitable.

Building generative LTMs presents significant challenges due to the complexity of modelling diverse datasets and types with context. These models are currently limited in scale and generalisation, often trained on small, varied datasets that can be noisy or incomplete. The diversity and quality of data, particularly in the tabular domain, complicate the creation of broadly applicable LTMs. Evaluation is difficult due to the intrinsic and extrinsic metrics required, and these models must also be assessed for privacy and bias, as tabular data can contain discriminatory features and biases similar to those found in other data types.

Why solving the challenge is worth it

Tabular foundation models can significantly enhance machine learning by improving the representation and inclusiveness of underrepresented groups and scarce data domains such as healthcare. They could enable both direct adaptation for specific tasks and indirect adaptation through the generation of synthetic data, which can augment real datasets and address data scarcity. LTMs also have the potential to support responsible AI by improving robustness, privacy, and data democratisation. In scientific applications, LTMs may facilitate meta-analyses by consolidating heterogeneous datasets and serve as powerful assistants for data scientists, enhancing productivity and automating complex data tasks.

To find out more about tabular foundation models and why we think they deserve to be a stronger research priority than they are today, read our ICML 2024 position paper here.