Tabular Deep Learning: From Reliable Evaluation to State-of-the-Art Results on Academic Benchmarks

Tabular Deep Learning: From Reliable Evaluation to State-of-the-Art Results on Academic Benchmarks

Rubachev I.V. (NRU HSE, Moscow, Russia)

Abstract

Reliable progress in tabular deep learning starts with a difficult comparison problem. When simple fully connected multilayer perceptrons (MLPs), specialized tabular architectures, and gradient-boosted decision trees are evaluated under a unified and thorough protocol on academic benchmarks, well-tuned MLP models are already hard to improve upon, while GBDT models remain strong competitors that set a high bar for the field. In this work, we use this demanding evaluation setting to demonstrate two mechanisms that improve tabular neural networks beyond strong MLP-based models and often bring them to the level of tree-based methods. The first one is an improved training methodology: two-stage training with input corruptions, auxiliary self-prediction objectives, and longer optimization schedules. The second one is retrieval: neural models equipped with a nonparametric component that, at prediction time, uses representations and labels of similar training objects. On the considered public academic benchmarks, both directions consistently improve tuned parametric neural networks. Moreover, some variants of the two approaches often reach or exceed GBDT quality. These results show that tabular deep learning can be competitive with tree-based methods when evaluated against strong simple models under a rigorous protocol.

Keywords

tabular data; deep learning; benchmarking; pretraining; retrieval; gradient boosting.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 5, 2026, pp. 287-304

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(5)-16

For citation

Rubachev I.V. Tabular Deep Learning: From Reliable Evaluation to State-of-the-Art Results on Academic Benchmarks. Proceedings of the Institute for System Programming, vol. 38, issue 5, 2026, pp. 287-304 DOI: 10.15514/ISPRAS-2026-38(5)-16.

Full text of the paper in pdf (in Russian) Back to the contents of the volume