Trompt: Towards a Better Deep Neural Network for Tabular Data
Tabular data is arguably one of the most commonly used data structures in
various practical domains, including finance, healthcare and e-commerce. The
inherent heterogeneity allows tabular data to store rich information. However,
based on a recently published tabular benchmark, we can see deep neural
networks still fall behind tree-based models on tabular datasets. In this
paper, we propose Trompt--which stands for Tabular Prompt--a novel architecture
inspired by prompt learning of language models. The essence of prompt learning
is to adjust a large pre-trained model through a set of prompts outside the
model without directly modifying the model. Based on this idea, Trompt
separates the learning strategy of tabular data into two parts. The first part,
analogous to pre-trained models, focus on learning the intrinsic information of
a table. The second part, analogous to prompts, focus on learning the
variations among samples. Trompt is evaluated with the benchmark mentioned
above. The experimental results demonstrate that Trompt outperforms
state-of-the-art deep neural networks and is comparable to tree-based models.
Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, Tien-Hao Chang
# Trompt: Towards a Better Deep Neural Network for Tabular Data

###### Abstract

Tabular data is arguably one of the most commonly used data structures in various practical domains, including finance, healthcare and e-commerce. However, based on a recently published tabular benchmark, we can see deep neural networks still fall behind tree-based models on tabular datasets (Grinsztajn et al., 2022). In this paper, we propose _Trompt_-which stands for **T**abular **Prompt_-a novel architecture inspired by prompt learning of language models. The essence of prompt learning is to adjust a large pre-trained model through a set of prompts outside the model without directly modifying the model. Based on this idea, Trompt separates the learning strategy of tabular data into two parts for the intrinsic information of a table and the varied information among samples. Trompt is evaluated with the benchmark mentioned above. The experimental results demonstrate that Trompt outperforms state-of-the-art deep neural networks and is comparable to tree-based models (Figure 1).

Machine Learning, tabular data, tabular data, tabular data, tabular data, tabular data, tabular data, tabular data

## 1 Introduction

Tabular data plays a vital role in many real world applications, such as financial statements for banks to evaluate the credibility of a company, diagnostic reports for doctors to identify the aetiology of a patient, and customer records for e-commerce platforms to discover the potential interest of a customer. In general, tabular data can be used to record activities consisting of heterogeneous features and has many practical usages.

On the other hand, deep learning has achieved a great success in various domains, including computer vision, natural language processing (NLP) and robotics (He et al., 2016; Redmon et al., 2016; Gu et al., 2017; Devlin et al., 2018). Besides extraordinary performance, there are numerous benefits of the end-to-end optimization nature of deep learning, including (i) online learning with streaming data (Sahoo et al., 2017), (ii) multi-model integration that incorporates different types of input, e.g., image and text (Ramachandran and Taylor, 2017) and (iii) representation learning that realizes semi-supervised learning and generative modeling (Van Engelen and Hoos, 2020; Goodfellow et al., 2020).

Consequently, researchers have been dedicated to apply deep learning on tabular data, either through (i) transformer (Huang et al., 2020; Somepalli et al., 2021; Gorishniy et al., 2021) or (ii) inductive bias investigation (Katzir et al., 2020; Arik and Pfister, 2021).

Though many of the previous publications claimed that they have achieved the state of the art, further researches pointed that previous works were evaluated on favorable datasets and tree-based models still show superior performances in the realm of tabular data (Borisov et al., 2021; Gorishniy et al., 2021; Shwartz-Ziv and Armon, 2022). For a fair comparison

Figure 1: Benchmark results.
between different algorithms, a standard benchmark for tabular data was proposed by (Grinsztajn et al., 2022). The benchmark, denoted as _Grinsztajn45_ in this work, consists of 45 curated datasets from various domains.

In this paper, we propose a novel prompt-inspired architecture, _Trompt_, which abbreviates **T**abular **Prompt**. Prompt learning has played an important role in the recent development of language models. For example, GPT-3 can well handle a wide range of tasks with an appropriate prompt engineering (Radford et al., 2018; Brown et al., 2020). In Trompt, prompt is utilized to derive feature importances that vary in different samples. Trompt consists of multiple _Trompt Cells_ and a shared _Trompt Downstream_ as Figure 2. Each Trompt Cell is responsible for feature extraction, while the Trompt Downstream is for prediction.

The performance of Trompt is evaluated on the Grinsztajn45 benchmark and compared with three deep learning models and five tree-based models. Figure 1 illustrates the overall evaluation results on Grinsztajn45. The x-axis is the number of hyperparameter search iterations and y-axis is the normalized performance. In Figure 1, Trompt is consistently better than state-of-the-art deep learning models (SAINT and FT-Transformer) and the gap between deep learning models and tree-based models is narrowed.

Our key contributions are summarized as follows:

* The experiments are conducted on a recognized tabular benchmark, Grinsztajn45. Additionally, we add two well-performed tree-based models, LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018) to baselines.
* Trompt achieves state-of-the-art performance among deep learning models and narrows the performance gap between deep learning models and tree-based models.
* Thorough empirical studies and ablation tests were conducted to verify the design of Trompt. The results further shed light on future research directions of the architecture design of tabular neural network.

## 2 Related Work

In this section, we first discuss the prompt learning of language models. Secondly, we discuss two research branches of tabular neural networks, transformer and inductive bias investigation. Lastly, we discuss the differences between Trompt and the related works and highlight the uniqueness of our work.

### Prompt Learning

The purpose of prompt learning is to transform the input and output of downstream tasks to the original task used to build a pre-trained model. Unlike fine-tuning that changes the task and usually involves updating model weights, a pre-train model with prompts can dedicate itself to one task. With prompt learning, a small amount of data or even zero-shot can achieve good results (Radford et al., 2018; Brown et al., 2020). The emergence of prompt learning substantially improves the application versatility of pre-trained models that are too large for common users to fine-tune.

To prompt a language model, one can insert a task-specific prompt before a sentence and hint the model to adjust its responses for different tasks (Brown et al., 2020). Prompts can either be discrete or soft. The former are composed of discrete tokens from the vocabulary of natural languages (Radford et al., 2018; Brown et al., 2020), while the latter are learned representations (Li and Liang, 2021; Lester et al., 2021).

### Tabular Neural Network

**Transformer.** Self-attention has revolutionized NLP since 2017 (Vaswani et al., 2017), and soon been adopted by other domains, such as computer vision, reinforcement learning and speech recognition (Dosovitskiy et al., 2020; Chen et al., 2021; Zhang et al., 2020). The intention of transformer blocks is to capture the relationships among features, which can be applied on tabular data as well.

TabTransformer (Huang et al., 2020) is the first transformer-based tabular neural network. However, TabTransformer only fed categorical features to transformer blocks and ignored the potential relationships among categorical and numerical features. FT-Transformer (Gorishniy et al., 2021) fixed this issue through feeding both categorical and numerical features to transformer blocks. SAINT (Sompalli et al., 2021) further improved FT-Transformer through applying attentions on not only the feature dimensions but also the sample dimensions.

**Inductive Bias Investigation.** Deep neural networks perform well on tasks with clear inductive bias. For example, Convolutional Neural Network (CNN) works well on images. The kernel of CNN is designed to capture local patterns since neighboring pixels usually relate to each other (LeCun et al., 1995). Recurrent Neural Networks (RNN) is widely used in language understanding because the causal relationship among words is well encapsulated through recurrent units (Rumelhart et al., 1986). However, unlike other popular tasks, the inductive bias of tabular data has not been well discovered.

Given the fact that tree-based model has been the solid state of the art for tabular data (Borisov et al., 2021; Gorishniy et al., 2021; Shwartz-Ziv and Armon, 2022), Net-DNF (Katzir et al., 2020) and TabNet (Arik and Pfister, 2021) hypothesized that the inductive bias for tabular data might be the learning
strategy of tree-based model. The strategy is to find the optimal root-to-leaf decision paths by selecting a portion of the features and deriving the optimal split from the selected features in non-leaf nodes. To emulate the learning strategy, TabNet utilized sequential attention and sparsity regularization. On the other hand, Net-DNF theoretically proved that decision tree is equivalent to some disjunctive normal form (DNF) and proposed disjunctive neural normal form to emulate a DNF formula.

### The Uniqueness of Trompt

We argue that the column importances of tabular data are not invariant for all samples and can be grouped into multiple modalities. Since prompt learning is born to adapt a model to multiple tasks, the concept is used in Trompt to handle multiple modalities. To this end, Trompt separates the learning strategy of tabular data into two parts. The first part, analogous to pre-trained models, focus on learning the intrinsic column information of a table. The second part, analogous to prompts, focus on diversifying the feature importances of different samples.

As far as our understanding, Trompt is the first prompt-inspired tabular neural network. Compared to transformer-based models, Trompt learns separated column importances instead of focusing on the interactions among columns. Compared to TabNet and Net-DNF, Trompt handle multiple modalities by emulating prompt learning instead of the branch split of decision tree.

## 3 Trompt

In this section, we elaborate on the architecture design of Trompt. As Figure 2 shows, Trompt consists of multiple Trompt Cells and a shared Trompt Downstream. Each Trompt Cell is responsible for feature extraction and providing diverse representations, while the Trompt Downstream is for prediction. The details of Trompt Cell and Trompt Downstream are discussed in Section 3.1 and Section 3.2, respectively. In Section 3.3, we further discuss the prompt learning of Trompt.

### Trompt Cell

Figure 3 illustrates the architecture of a Trompt Cell, which can be divided into three parts. The first part derives feature importances (\(\mathbf{M}_{\text{importance}}\)) based on column embeddings (\(\mathbf{E}_{\text{column}}\)), the previous cell's output (\(\mathbf{O}_{\text{prev}}\)) and prompt embeddings (\(\mathbf{E}_{\text{prompt}}\)). The second part transforms the input into feature embeddings (\(\mathbf{E}_{\text{feature}}\)) with two paths for categorical and numerical columns, respectively. The third part expands \(\mathbf{E}_{\text{feature}}\) for the later multiplication.

The details of the first part are illustrated in Section 3.1.1 and the details of the second and third parts are illustrated in Section 3.1.2. Lastly, the generation of the output of a Trompt Cell is illustrated in Section 3.1.3.

#### 3.1.1 Derive Feature Importances

Let \(\mathbf{E}_{\text{column}}\in\mathbb{R}^{C\times d}\) be column embeddings and \(\mathbf{E}_{\text{prompt}}\in\mathbb{R}^{P\times d}\) be prompt embeddings. \(C\) is the number of columns of a table defined by the dataset, while \(P\) and \(d\) are hyperparameters for the number of prompts and the hidden dimension, respectively. Both \(\mathbf{E}_{\text{column}}\) and \(\mathbf{E}_{\text{prompt}}\) are input independent and trainable. Let \(\mathbf{O}_{\text{prev}}\in\mathbb{R}^{B\times P\times d}\) be the previous cell's output and \(B\) be the batch size.

\(\mathbf{O}_{\text{prev}}\) is fused with the prompt embeddings as Equations (1) and (2). Since \(\mathbf{E}_{\text{prompt}}\) is input independent and lack a batch dimension, \(\mathbf{E}_{\text{prompt}}\) is expanded to \(\mathbf{SE}_{\text{prompt}}\) through the stack operation as Equation (1). Later, we concatenate \(\mathbf{SE}_{\text{prompt}}\) and \(\mathbf{O}_{\text{prev}}\) and then reduce the dimension of the concatenated tensor back to \(\mathbb{R}^{B\times P\times d}\) for the final addition as Equation (2).

Figure 2: Overall architecture of the proposed Trompt.
For the same reason as \(\mathbf{E}_{\text{prompt}}\), the \(\mathbf{E}_{\text{column}}\) is expanded to \(\mathbf{SE}_{\text{column}}\) as Equation (3). Subsequently, feature importances are derived through Equation (4), where \(\otimes\) is the batch matrix multiplication, \(\mathsf{T}\) is the batch transpose, and the softmax is applied to the column axis.

\[\mathbf{SE}_{\text{prompt}} =\mathtt{stack}(\mathbf{E}_{\text{prompt}})\in\mathbb{R}^{B\times P \times d} \tag{1}\] \[\hat{\mathbf{SE}}_{\text{prompt}}= \mathtt{dense}(\mathtt{concat}(\mathbf{SE}_{\text{prompt}}, \mathbf{O}_{\text{prev}}))\] \[+\mathbf{SE}_{\text{prompt}}\] (2) \[+\mathbf{O}_{\text{prev}}\] \[\in\mathbb{R}^{B\times P\times d}\] \[\mathbf{SE}_{\text{column}} =\mathtt{stack}(\mathbf{E}_{\text{column}})\in\mathbb{R}^{B \times C\times d} \tag{3}\]

\[\mathbf{M}_{\text{importance}}=\mathtt{softmax}(\hat{\mathbf{SE}}_{\text{ prompt}}\otimes\mathbf{SE}_{\text{column}}^{\intercal})\in\mathbb{R}^{B \times P\times C} \tag{4}\]

The output of the first part is \(\mathbf{M}_{\text{importance}}\in\mathbb{R}^{B\times P\times C}\), which accommodates the feature importances yielded by \(P\) prompts. Notice that the column embeddings are not connected to the input and the prompt embeddings are fused with the previous cell's output. In Section 3.3, we further discuss these designs and their connections to the prompt learning of NLP.

#### 3.1.2 Construct and Expand Feature Embeddings

In Trompt, categorical features are embedded through a embedding layer and numerical features are embedded through a dense layer as previous works (Sompelalli et al., 2021; Gorishniy et al., 2021). The embedding construction procedure is illustrated in part two of Figure 3, where \(\mathbf{E}_{\text{feature}}\in\mathbb{R}^{B\times C\times d}\) is the feature embeddings of the batch.

The shapes of \(\mathbf{M}_{\text{importance}}\) and \(\mathbf{E}_{\text{feature}}\) are \(\mathbb{R}^{B\times P\times C}\) and \(\mathbb{R}^{B\times C\times d}\), respectively. Since \(\mathbf{E}_{\text{feature}}\) lacks the prompt dimension, Trompt expands \(\mathbf{E}_{\text{feature}}\) into \(\mathbf{\hat{E}}_{\text{feature}}\in\mathbb{R}^{B\times P\times C\times d}\) to accommodate the \(P\) prompts by a dense layer in part three of Figure 3.

#### 3.1.3 Generate Output

The output of Trompt Cell is the column-wise sum of the element-wise multiplication of \(\mathbf{\hat{E}}_{\text{feature}}\) and \(\mathbf{M}_{\text{importance}}\) as Equation (5), where \(\odot\) is element-wise multiplication. Notice that, during element-wise multiplication, the shape of \(\mathbf{M}_{\text{importance}}\) is considered \(\mathbb{R}^{B\times P\times C\times 1}\). In addition, since column is the third axis, the shape is reduced from \(\mathbb{R}^{B\times P\times C\times d}\) to \(\mathbb{R}^{B\times P\times d}\) after column-wise summation.

\[\mathbf{O}=\sum_{i=1}^{C}(\mathbf{\hat{E}}_{\text{feature}}\odot\mathbf{M}_ {\text{importance}}),:,i,:\in\mathbb{R}^{B\times P\times d} \tag{5}\]

### Trompt Downstream

A Trompt Downstream makes a prediction based on a Trompt Cell's output, which contains representations corresponding to \(P\) prompt embeddings. To aggregate these representations, the weight for each prompt is first derived through a dense layer and a softmax activation function as Equation (6). Afterwards, the weighted sum is calculated as Equation (7).

The prediction is subsequently made through two dense layers as Equation (8), where \(T\) is the target dimension.

Figure 3: Architecture of a Trompt Cell.
For classification tasks, \(T\) is the number of target classes. For regression tasks, \(T\) is set to 1. As Figure 2 shows, a sample gets a prediction through a Trompt Cell and thus multiple predictions through all cells. During training, the loss of each prediction is separately calculated and the loss is summed up to update model weights. During inference, on the other hand, predictions through all cells are simply averaged as the final prediction as Equation (9), where \(L\) is the number of Trompt Cells.

\[\mathbf{W_{prompt}}=\mathtt{softmax}(\mathtt{dense}(\mathbf{O}))\in\mathbb{R}^{ B\times P} \tag{6}\]

\[\mathbf{\hat{O}}=\sum_{i=1}^{P}(\mathbf{W_{prompt}}\odot\mathbf{O})_{:,i,:} \in\mathbb{R}^{B\times d} \tag{7}\]

\[\mathbf{P}=\mathtt{dense}(\mathtt{relu}(\mathtt{dense}(\mathbf{\hat{O}}))) \in\mathbb{R}^{B\times T} \tag{8}\]

\[\begin{split} loss=\sum_{i=1}^{L}\mathtt{loss}_{\mathtt{fn}}( \mathbf{P}_{i},y)\\ pred=\sum_{i=1}^{L}\mathbf{P}_{i}/L\end{split} \tag{9}\]

### Prompt Learning of Trompt

Trompt's architecture is specifically designed for tabular data, taking into account the unique characteristics of this type of data and the impressive performance of tree-based models. Unlike conventional operations, the design may appear unconventional and detached from tabular data features. In this section, we explain the rationale behind Trompt's network design and how we adapted prompt learning to a tabular neural network.

Tabular data is structured, with each column representing a specific dataset property that remains constant across individual samples. The success of tree-based models relies on assigning feature importances to individual samples. This concept has been explored in models such as TabNet (Arik and Pfister, 2021) and Net-DNF (Katzir et al., 2020). However, tree-based algorithms do not explicitly assign feature importances to individual samples. Instead, importances vary implicitly along the path from the root to a leaf node. Only the columns involved in this path are considered important features for the samples reaching the corresponding leaf node, representing sample-specific feature importances.

Given the fundamental characteristic of tabular data and the learning strategy of tree-based models, Trompt aims to combine the intrinsic properties of columns with sample-specific feature importances using a prompt learning-inspired architecture from NLP (Radford et al., 2018; Brown et al., 2020). Trompt employs column embeddings to represent the intrinsic properties of each column and prompt embeddings to prompt column embeddings, generating feature importances for given prompts. Both column embeddings and prompt embeddings are invariant across samples. However, before prompting column embeddings with prompt embeddings, the prompt embeddings are fused with the output of the previous Trompt Cell as shown in Equation (2), enabling input-related representations to flow through and derive sample-specific feature importances. The "prompt" mechanism in Trompt is implemented as a matrix multiplication in Equation (4).

A conceptual analogy of Trompt's prompt learning approach to NLP is presented in Table 1. It's important to note that the implementation details of prompt learning differ substantially between tabular data and NLP tasks due to the fundamental differences between the two domains. Therefore, appropriate adjustments must be made to bridge these two domains.

## 4 Experiments

In this section, the experimental results and analyses are presented. First, we elaborate on the settings of experiments and the configurations of Trompt in Section 4.1. Second, the performance of Trompt on Grinsztajn45 is reported in

\begin{table}
\begin{tabular}{c|c|c} \hline \hline
**Problem** & **Implemented by** & **Inspired by** \\
**Identification** & & \\ \hline Sample-invariant & \(\mathbf{E}_{\text{column}}\) & Fixed Large \\ Intrinsic Properties & & Language Model \\ \hline Sample-specific & \(\mathbf{M}_{\text{importance}}\) & Task-specific \\ Feature Importances & & Predictions \\ \hline \hline \end{tabular}
\end{table}
Table 1: Analogy of the prompt learning of Trompt to that of NLP.

Figure 4: Architecture of a Trompt Downstream.
Section 4.2. Third, ablation studies regarding the hyperparameters and the architecture of Trompt are studied in Section 4.3. Lastly, the interpretability of Trompt is investigated using synthetic and real-world datasets in Section 4.4.

### Setup

The performance and ablation study of Trompt primarily focus on the Grinsztajn45 benchmark (Grinsztajn et al., 2022)1. This benchmark comprises datasets from various domains and follows a unified methodology for evaluating different models, providing a fair and comprehensive assessment. Furthermore, we evaluate the performance of Trompt on datasets selected by FT-Transformer and SAINT to compare it with state-of-the-art tabular neural networks.

Footnote 1: [https://github.com/LeoGrin/tabular-benchmark](https://github.com/LeoGrin/tabular-benchmark)

For interpretability analysis, we follow the experimental settings of TabNet (Arik and Pfister, 2021). This involves using two synthetic datasets (Syn2 and Syn4) and a real-world dataset (mushroom) to visualize attention masks.

The settings of Grinsztajn45 are presented in Section 4.1.1 and the implementation details of Trompt are presented in Section 4.1.2. Furthermore, the settings of datasets chosen by FT-Transformer and SAINT are provided in Appendix B.2 and Appendix B.3, respectively.

#### 4.1.1 Settings of Grinsztajn45

To fairly evaluate the performance, we follow the configurations of Grinsztajn45, including train test data split, data preprocessing and evaluation metric. Grinsztajn45 comprises two kinds of tasks, classification tasks and regression tasks. Please see Appendix A.1 and Appendix A.2 for the dataset selection criteria and dataset normalization process of Grinsztajn45. The tasks are further grouped according to (i) the size of datasets (medium-sized and large-sized) and (ii) the inclusion of categorical features (numerical only and heterogeneous).

In addition, we make the following adjustments: (i) models with incomplete experimental results in (Grinsztajn et al., 2022) are omitted, (ii) two well-performed tree-based models are added for comparison, and (iii) Trompt used a hyperparameter search space smaller than its opponents. The details of the adjustments are described in Appendix A.3 and Appendix A.4.

#### 4.1.2 Implementation Details

Trompt is implemented using PyTorch. The default hyperparameters are shown in Table 2. The size of embeddings and the hidden dimension of dense layers are configured \(d\). Note that only the size of column and prompt embeddings must be the same by the architecture design. The hidden dimension of dense layers is set as \(d\) to reduce hyperparameters and save computing resources. On the other hand, the number of prompts and the number of Trompt Cells are set to \(P\) and \(L\). Please refer to Appendix F for the hyperparameter search spaces for all baselines and Trompt.

### Evaluation Results

The results of classification tasks are discussed in Section 4.2.1 and the results of regression tasks are discussed in Section 4.2.2. The evaluation metrics are accuracy and r2-score for classification and regression tasks, respectively. In this section, we report an overall result and leave results of individual datasets in Appendix B.1. In addition, the evaluation results on datasets chosen by FT-Transformer and SAINT are provided in Appendix B.2 and Appendix B.3, respectively.

#### 4.2.1 Classification

On the medium-sized classification tasks, Figure 5 shows that Trompt outperforms DNN models. The curve of Trompt is consistently above deep neural networks (SAINT, FT-Transformer and ResNet) on tasks with and without categorical features. Additionally, Trompt narrows the gap between deep neural networks and tree-based models, especially on the tasks with heterogeneous features. In Figure 4(b), Trompt seems to be a member of the leading cluster with four tree-based models. The GradientBoostingTree starts slow but catches up the leading cluster in the end of search. The other deep neural networks forms the second cluster and have a gap to the leading one.

On the large-sized classification tasks, tree-based models remain the leading positions but the gap to deep neural networks is obscure. This echoes that deep neural networks requires more samples for training (LeCun et al., 2015). Figure 5(a) shows that Trompt outperforms ALL models on the task with numerical features and Figure 5(b) shows that Trompt achieves a comparable performance to FT-Transformer on tasks with heterogeneous features.

With the small hyperparameter search space, the curve of Trompt is relatively flat. The flat curve also suggests that Trompt performs well with its default hyperparameters. Its

\begin{table}
\begin{tabular}{l|c|c} \hline \hline \multicolumn{1}{c|}{**Hyperparameter**} & \multicolumn{1}{c|}{**Symbol**} & \multicolumn{1}{c}{**Value**} \\ \hline Feature Embeddings & & \\ Prompt/Column Embeddings & \(d\) & 128 \\ Hidden Dimension & & \\ \hline Prompts & \(P\) & 128 \\ \hline Layer & \(L\) & 6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Default hyperparameters of Trompt.
performance after an exhausted search is worthy of future exploring.

#### 4.2.2 Regression

On the medium-sized regression tasks, Figure 7 shows that Trompt outperforms deep neural networks as the curves of Trompt are consistently higher than SAINT, FT-Transformer and ResNet on tasks with and without categorical features. The gap between deep neural networks and tree-based models is less obvious in Figure 6(a) than that in Figure 6(b). On the tasks with numerical features only, Trompt achieves a comparable performance with random forest. On the tasks with heterogeneous features, Trompt narrows the gap but is below all the tree-based models.

On the large-sized regression tasks with numerical features only, Figure 7(a) shows that Trompt is slightly worse than SAINT and FT-Transformer in the end of search. On the large-sized regression tasks with heterogeneous features, Figure 7(b) shows that Trompt outperforms deep neural networks with a large margin.

In general, deep learning models are not good at handling categorical features. Trompt alleviates this weakness as shown in all tasks with heterogeneous features in Figure 5- Figure 8. Trompt achieves superior performance over state-of-the-art deep neural networks except on the large-sized regression tasks with numerical features only.

### Ablation Study

In this subsection, we discuss the ablation study results of Trompt regarding hyperparameters and architecture design. Please refer to Appendix C for the settings of the ablation study. In the main article, we report two major ablations on (i) the number of prompts and (ii) the necessity of expanding feature embeddings by a dense layer. Other ablations can be found in Appendix D.

**Ablations on the number of prompts.** Prompt embeddings (\(\mathbf{E}_{\text{prompt}}\)) stand a vital role to derive the feature importances. Here we discuss the effectiveness of adjusting the number of prompts.

As shown in Table 3, setting the number of prompts to one results in the worse results. However, halving and doubling the default number (128) do not have much effect on the performance. The results demonstrate that Trompt is not sensitive to the number of prompts, as long as the number of prompts is enough to accommodate the modalities of the dataset.

**Ablations on expanding feature embeddings by a dense layer.** Part three of Figure 3 uses a dense layer to expand feature embeddings to accommodate \(P\) prompts. Here we discuss the necessity of the dense layer.

As you can see in Table 4, adding a dense layer really leads to better results and is a one of the key architecture designs of Trompt. By design, adding the dense layer enables

Figure 5: Benchmark on **medium-sized classification** datasets.

Figure 8: Benchmark on **large-sized regression** datasets.

Figure 6: Benchmark on **large-sized classification** datasets.

Figure 7: Benchmark on **medium-sized regression** datasets.
Trompt to generate different feature embeddings for each prompt. Without the dense layer, Trompt is degraded to a simplified situation where each prompt uses the same feature embeddings. The results of Table 3 and Table 4 suggest that the variation of feature importances, which comes from both the prompt embedding and the expansion dense layer, is the key to the excellent performance of Trompt.

### Interpretability

Besides outstanding performance, tree-based models are well-known for their interpretability. Here we explore whether Trompt can also provide concise feature importances that highlighted salient features. To investigate this, we conduct experiments on both synthetic datasets and real-world datasets, following the experimental design of TabNet (Arik and Pfister, 2021). To derive the feature importances of Trompt for each sample, \(\mathbf{M}_{\text{importance}}\in\mathbb{R}^{B\times P\times C}\) is reduced to \(\mathbf{\hat{M}}_{\text{importance}}\in\mathbb{R}^{B\times C}\) as Equation (10), where the weight of \(\mathbf{M}_{\text{importance}}\) is the \(\mathbf{W}_{\text{prompt}}\) of Equation (6).

Notice that all Trompt Cells derive separated feature importances. We demonstrate the averaged results of all cells here and leave the results of each cell in Appendix E.1.

\[\mathbf{\hat{M}}_{\text{importance}}=\sum_{i=1}^{P}(\mathbf{W}_{\text{ prompt}}\odot\mathbf{M}_{\text{importance}})_{:,i,:}\in\mathbb{R}^{B\times C} \tag{10}\]

**Synthetic datasets.** The Syn2 and Syn4 datasets are used to study the feature importances learned by each model (Chen et al., 2018). A model is trained on oversampled training set (10k to 100k) using default hyperparameters and evaluated on 20 randomly picked testing samples. The configuration is identical to that in TabNet (Arik and Pfister, 2021).

Figure 9 and Figure 10 compare the important features of the dataset and those learned by Trompt. In the Syn2 dataset, features 2-5 are important (Figure 8(a)) and Trompt excellently focuses on them (Figure 8(b)). In the Syn4 dataset, either features 0-1 or 2-5 could be important based on the value of feature 10 (Figure 9(a)). As Figure 10 shows, Trompt still properly focuses on features 0-5 and discovers the influence of feature 10.

**Real-world datasets.** The mushroom dataset (Dua and Graff, 2017) is used as the real-world dataset for visualization as TabNet (Arik and Pfister, 2021). With only the _Odor_ feature, most machine learning models can achieve \(>95\%\) test accuracy (Arik and Pfister, 2021). As a result, a high feature importance is expected on Odor.

Table 5 shows the three most important features of Trompt and five tree-based models. As shown, all models place Odor in their top three. The second and third places of Trompt, _gill-size_ and _gill-color_, also appear in the top three of the other models. Actually, _cap-color_ is selected only by XGBoost. If it is excluded, the union of the top important features of all models comes down to four features. The one Trompt missed is _spore-print-color_, which is the fifth place of Trompt. Overall speaking, the important features selected by Trompt are consistent with those by tree-based models, and can therefore be used in various analyses that are familiar in the field of machine learning.

To further demonstrate that the experimental results were not ad-hoc, we repeat the experiments on additional real-world datasets. Please see Appendix E.2 for the details and

\begin{table}
\begin{tabular}{l|c|c} \hline \hline  & **w (default)** & **w/o** \\ \hline Classification & \(81.81\%\) & \(80.76\%\) \\ \hline Regression & \(74.15\%\) & \(73.73\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: The performance of with and without applying feature transformation on Input Transform.

Figure 10: Attention mask on Syn4 dataset (synthetic).

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline  & **1** & **64** & **128 (default)** & **256** \\ \hline Classification & \(79.74\%\) & \(81.76\%\) & \(81.81\%\) & \(81.85\%\) \\ \hline Regression & \(72.07\%\) & \(74.11\%\) & \(74.15\%\) & \(74.14\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The performance of different number of prompts.

Figure 9: Attention mask on Syn2 dataset (synthetic).
experimental results.

## 5 Discussion

In this section, we further explore the "prompt" mechanism of Trompt. Section 5.1 clarifies the underlying hypothesis of how the prompt learning of Trompt fits for tabular data. In addition, as Trompt is partially inspired by the learning strategy of tree-based models, we further discussed the difference between Trompt and tree-based models in Section 5.2.

### Further exploration of the "prompt" mechanism in Trompt

The "prompt" mechanism in Trompt is realized as Equation (4). This equation involves a matrix multiplication of expanded prompt embeddings (\(\mathbf{SE}_{\text{prompt}}\in\mathbb{R}^{B\times P\times d}\)) and the transpose of expanded column embeddings (\(\mathbf{SE}_{\text{column}}\in\mathbb{R}^{B\times C\times d}\)). It results in \(\mathbf{M}_{\text{importance}}\in\mathbb{R}^{P\times C}\), which represents prompt-to-column feature importances. The matrix multiplication calculates the cosine-based distance between \(\mathbf{SE}_{\text{prompt}}\) and \(\mathbf{SE}_{\text{column}}\), and favors high similarity between the sample-specific representations and sample-invariant intrinsic properties.

To make it clearer, \(\mathbf{SE}_{\text{prompt}}\) consists of \(P\) embeddings that are specific to individual samples, except for the first Trompt Cell where \(\mathbf{O}_{\text{prev}}\) is a zero tensor since there is no previous Trompt Cell, as stated in Equations (1) and (2). On the other hand, \(\mathbf{SE}_{\text{column}}\) consists of \(C\) embeddings that represent intrinsic properties specific to a tabular dataset as stated in Equation (3).

Unlike self-attention, which calculates the distance between queries and keys and derives token-to-token similarity measures, Trompt calculates the distance between \(\mathbf{SE}_{\text{prompt}}\) and \(\mathbf{SE}_{\text{column}}\) in Equation (4) to derive sample-to-intrinsic-property similarity measures. The underlying idea of the calculation is to capture the distance between each sample and intrinsic property of a tabular dataset and we hypothesize that incorporating the intrinsic properties into the modeling of a tabular neural network might help making good predictions.

### The differences between Trompt and Tree-based Models

As discussed in Section 3.3, the idea of using prompt learning to derive feature importances, is inspired by the learning algorithm of tree-based models and the intrinsic properties of tabular data. As a result, Trompt and tree-based models share a common characteristic in that they enable sample-dependent feature importances. However, there are two main differences between them. First, to incorporate the intrinsic properties of tabular data, Trompt uses column embeddings to share the column information across samples, while the learning strategy of tree-based models learn column information in their node-split nature. Second, Trompt and tree-based models use different techniques to learn feature importance. Trompt derives feature importances explicitly through prompt learning, while tree-based models vary the feature importances implicitly in the root-to-leaf path.

## 6 Conclusion

In this study, we introduce Trompt, a novel network architecture for tabular data analysis. Trompt utilizes prompt learning to determine varying feature importances in individual samples. Our evaluation shows that Trompt outperforms state-of-the-art deep neural networks (SAINT and FT-Transformer) and closes the performance gap between deep neural networks and tree-based models.

The emergence of prompt learning in deep learning is promising. While the design of Trompt may not be intuitive or perfect for language model prompts, it demonstrates the potential of leveraging prompts in tabular data analysis. This work introduces a new strategy for deep neural networks to challenge tree-based models and future research in this direction can explore more prompt-inspired architectures.

\begin{table}
\begin{tabular}{l|c|c|c} \hline \hline  & **1st** & **2nd** & **3rd** \\ \hline RandomForest & odor (\(15.11\%\)) & gill-size (\(12.37\%\)) & gill-color (\(10.42\%\)) \\ \hline XGBoost & spore-print-color (\(29.43\%\)) & odor (\(22.71\%\)) & cap-color (\(14.07\%\)) \\ \hline LightGBM & spore-print-color (\(22.08\%\)) & gill-color (\(14.95\%\)) & odor (\(12.96\%\)) \\ \hline CatBoost & odor (\(72.43\%\)) & spore-print-color (\(10.57\%\)) & gill-size (\(2.71\%\)) \\ \hline GradientBoostingTree & gill-color (\(31.08\%\)) & spore-print-color (\(19.89\%\)) & odor (\(17.44\%\)) \\ \hline Trompt (ours) & odor (\(24.93\%\)) & gill-size (\(8.13\%\)) & gill-color (\(5.73\%\)) \\ \hline \hline \end{tabular}
\end{table}
Table 5: The top-3 importance score ratio on the mushroom dataset.