TANGOS: Regularizing Tabular Neural Networks through Gradient
  Orthogonalization and Specialization
Despite their success with unstructured data, deep neural networks are not
yet a panacea for structured tabular data. In the tabular domain, their
efficiency crucially relies on various forms of regularization to prevent
overfitting and provide strong generalization performance. Existing
regularization techniques include broad modelling decisions such as choice of
architecture, loss functions, and optimization methods. In this work, we
introduce Tabular Neural Gradient Orthogonalization and Specialization
(TANGOS), a novel framework for regularization in the tabular setting built on
latent unit attributions. The gradient attribution of an activation with
respect to a given input feature suggests how the neuron attends to that
feature, and is often employed to interpret the predictions of deep networks.
In TANGOS, we take a different approach and incorporate neuron attributions
directly into training to encourage orthogonalization and specialization of
latent attributions in a fully-connected network. Our regularizer encourages
neurons to focus on sparse, non-overlapping input features and results in a set
of diverse and specialized latent units. In the tabular domain, we demonstrate
that our approach can lead to improved out-of-sample generalization
performance, outperforming other popular regularization methods. We provide
insight into why our regularizer is effective and demonstrate that TANGOS can
be applied jointly with existing methods to achieve even greater generalization
performance.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, Mihaela van der Schaar
# Tangos: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization

###### Abstract

Despite their success with unstructured data, deep neural networks are not yet a panacea for structured tabular data. In the tabular domain, their efficiency crucially relies on various forms of regularization to prevent overfitting and provide strong generalization performance. Existing regularization techniques include broad modelling decisions such as choice of architecture, loss functions, and optimization methods. In this work, we introduce Tabular Neural Gradient Orthogonalization and Specialization (TANGOS), a novel framework for regularization in the tabular setting built on latent unit attributions. The gradient attribution of an activation with respect to a given input feature suggests how the neuron _attends_ to that feature, and is often employed to interpret the predictions of deep networks. In TANGOS, we take a different approach and incorporate neuron attributions directly into training to encourage orthogonalization and specialization of _latent attributions_ in a fully-connected network. Our regularizer encourages neurons to focus on sparse, non-overlapping input features and results in a set of diverse and specialized latent units. In the tabular domain, we demonstrate that our approach can lead to improved out-of-sample generalization performance, outperforming other popular regularization methods. We provide insight into _why_ our regularizer is effective and demonstrate that TANGOS can be applied jointly with existing methods to achieve even greater generalization performance.

## 1 Introduction

Despite its relative under-representation in deep learning research, tabular data is ubiquitous in many salient application areas including medicine, finance, climate science, and economics. Beyond raw performance gains, deep learning provides a number of promising advantages over non-neural methods including multi-modal learning, meta-learning, and certain interpretability methods, which we expand upon in depth in Appendix C. Additionally, it is a domain in which general-purpose regularizers are of particular importance. Unlike areas such as computer vision or natural language processing, architectures for tabular data generally do not exploit the inherent structure in the input features (i.e. locality in images and sequential text, respectively) and lack the resulting inductive biases in their design. Consequentially, improvement over non-neural ensemble methods has been less pervasive. Regularization methods that implicitly or explicitly encode inductive biases thus play a more significant role. Furthermore, adapting successful strategies from the ensemble literature to neural networks may provide a path to success in the tabular domain (e.g. Wen et al., 2020). Recent work in Kadra et al. (2021) has demonstrated that suitable regularization is essential to
outperforming such methods and, furthermore, a balanced _cocktail_ of regularizers results in neural network superiority.

Regularization methods employed in practice can be categorized into those that prevent overfitting through data augmentation (Krizhevsky et al., 2012; Zhang et al., 2018), network architecture choices (Hinton et al., 2012; Ioffe and Szegedy, 2015), and penalty terms that explicitly influence parameter learning (Hoerl and Kennard, 1970; Tibshirani, 1996; Jin et al., 2020), to name just a few. While all such methods are unified in attempting to improve out-of-sample generalization, this is often achieved in vastly different ways. For example, \(L1\) and \(L2\) penalties favor sparsity and shrinkage, respectively, on model weights, thus choosing more parsimonious solutions. Data perturbation techniques, on the other hand, encourage smoothness in the system assuming that small perturbations in the input should not result in large changes in the output. Which method works best for a given task is generally not known _a priori_ and considering different classes of regularizer is recommended in practice. Furthermore, combining multiple forms of regularization simultaneously is often effective, especially in lower data regimes (see e.g. Brigato and Iocchi, 2021 and Hu et al., 2017).

Neuroscience research has suggested that neurons are both _selective_(Johnston and Dark, 1986) and have _limited capacity_(Cowan et al., 2005) in reacting to specific physiological stimuli. Specifically, neurons selectively choose to focus on a few chunks of information in the input stimulus. In deep learning, a similar concept, commonly described as a _receptive field_, is employed in convolutional layers (Luo et al., 2016). Here, each convolutional unit has multiple filters, and each filter is only sensitive to specialized features in a local region. The output of the filter will activate more strongly if the feature is present. This stands in contrast to fully-connected networks, where the all-to-all relationships between neurons mean each unit depends on the entire input to the network. We leverage this insight to propose a regularization method that can encourage artificial neurons to be more specialized and orthogonal to each other.

**Contributions. (1) Novel regularization method for deep tabular models.** In this work, we propose TANGOS, a novel method based on regularizing neuron attributions. A visual depiction is given in Figure 1. Specifically, each neuron is more _specialized_, attending to sparse input features while its attributions are more _orthogonal_ to those of other neurons. In effect, different neurons pay attention to non-overlapping subsets of input features resulting in better generalization performance. We demonstrate that this novel regularization method results in excellent generalization performance on tabular data when compared to other popular regularizers. **(2) Distinct regularization objective.** We explore how TANGOS results in distinct emergent characteristics in the model weights. We further show that its improved performance is linked to increased diversity among weak learners in an ensemble of latent units, which is generally in contrast to existing regularizers. **(3) Combination with other regularizers.** Based upon these insights, we demonstrate that deploying TANGOS _in tandem_ with other regularizers can further improve generalization of neural networks in the tabular setting beyond that of any individual regularizer.

Figure 1: **TANGOS encourages specialization and orthogonalization. TANGOS penalizes neuron attributions during training. Here, indicates strong positive attribution and indicates strong negative attribution, while interpolating colors reflect weaker attributions. Neurons are regularized to be _specialized_ (attend to sparser features) and _orthogonal_ (attend to non-overlapping features).**
## 2 Related Work

**Gradient Attribution Regularization.** A number of methods exist which incorporate a regularisation term to penalize the network gradients in some way. Penalizing gradient attributions is a natural approach for achieving various desirable properties in a neural network. Such methods have been in use at least since Drucker & Le Cun (1992), where the authors improve robustness by encouraging invariance to small perturbations in the input space. More recently, gradient attribution regularization has been successfully applied across a broad range of application areas. Some notable examples include encouraging the learning of robust features in auto-encoders (Rifai et al., 2011), improving stability in the training of generative adversarial networks (Gulrajani et al., 2017), and providing robustness to adversarial perturbations (Moosavi-Dezfooli et al., 2019). While many works have applied a shrinkage penalty (L2) to input gradients, Ross et al. (2017) explore the effects of encouraging sparsity by considering an L1 penalty term. Gradient penalties may also be leveraged to compel a network to _attemd_ to particular human-annotated input features (Ross et al., 2017). A related line of work considers the use of gradient aggregation methods such as Integrated Gradients (Sundararajan et al., 2017) and, typically, penalizes their deviation from a given target value (see e.g. Liu & Avici (2019) and Chen et al. (2019)). In contrast to these works, we do not require manually annotated regions upon which we constrain the network to attend. Similarly, Erion et al. (2021) provide methods for encoding domain knowledge such as smoothness between adjacent pixels in an image. We note that while these works have investigated penalizing a predictive model's output attributions, we are the first to regularize attributions on latent neuron activations. We provide an extended discussion of related works on neural network regularization more generally in Appendix A.

## 3 Tangos

### Problem Formulation

We operate in the standard supervised learning setting, with \(d_{X}\)-dimensional input variables \(X\in\mathcal{X}\subseteq\mathbb{R}^{d_{X}}\) and target output variable \(Y\in\mathcal{Y}\subseteq\mathbb{R}\). Let \(P_{XY}\) denote the joint distribution between input and target variables. The goal of the supervised learning algorithm is to find a predictive model, \(f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}\) with learnable parameters \(\theta\in\Theta\). The predictive model belongs to a hypothesis space \(f_{\theta}\in\mathcal{H}\) that can map from the input space to the output space.

The predictive function is usually learned by optimizing a loss function \(\mathcal{L}:\Theta\rightarrow\mathbb{R}\) using _empirical risk minimization_ (ERM). The empirical risk cannot be directly minimized since the data distribution \(P_{XY}\) is not known. Instead, we use a finite number of iid samples \((x,y)\sim P_{XY}\), which we refer to as the training data \(\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}\).

Once the predictive model is trained on \(\mathcal{D}\), it should ideally predict well on out-of-sample data generated from the same distribution. However, overfitting can occur if the hypothesis space \(\mathcal{H}\) is to complex and the sampling of training data does not fully represent the underlying distribution \(P_{XY}\). Regularization is an approach that reduces the complexity of the hypothesis space so that more generalized functions are learned to explain the data. This leads to the following ERM:

\[\theta^{*}=\operatorname*{arg\,min}_{\theta\in\Theta}\frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}}\mathcal{L}(f_{\theta}(x),y)+\mathcal{R}(\theta,x,y), \tag{1}\]

that includes an additional regularization term \(\mathcal{R}\) which, generally, is a function of input \(x\), the label \(y\), the model parameters \(\theta\), and reflects prior assumptions about the model. For example, \(L1\) regularization reflects the belief that sparse solutions in parameter space are more desirable.

### Neuron Attributions

Formally, attribution methods aim to uncover the importance of each input feature of a given sample to the prediction of the neural network. Recent works have demonstrated that feature attribution methods can be incorporated into the training process (Lundberg & Lee, 2017; Erion et al., 2021). These _attribution priors_ optimize attributions to have desirable characteristics, including interpretability as well as smoothness and sparsity in predictions. However, these methods have exclusively investigated _output_ attributions, i.e., contributions of input features to the output of a model. To the best of our knowledge, we are the first work to investigate regularization of _latent attributions_.
We rewrite our predictive function \(f\) using function composition \(f=l\circ g\). Here \(g:\mathcal{X}\to\mathcal{H}\) maps the input to a representation \(h=g(x)\in\mathcal{H}\), where \(\mathcal{H}\subseteq\mathbb{R}^{d_{H}}\) is a \(d_{H}\)-dimensional latent space. Additionally, \(l:\mathcal{H}\to\mathcal{Y}\) maps the latent representation to a label space \(y=l(h)\in\mathcal{Y}\). We let \(h_{i}=g_{i}(x)\), for, \(i\in[d_{H}]\) denote the \(i^{th}\) neuron in the hidden layer of interest. Additionally, we use \(a^{i}_{j}(x)\in\mathbb{R}\) to denote the attribution of the \(i^{th}\) neuron w.r.t. the feature \(x_{j}\). With this notation, upper indices correspond to latent units and lower indices to features. In some cases, it will be convenient to stack all the feature attributions together in the attribution vector \(a^{i}(x)=[a^{i}_{j}(x)]_{j=1}^{d_{X}}\in\mathbb{R}^{d_{X}}\).

Attribution methods work by using gradient signals to evaluate the contributions of the input features. In the most simplistic setting:

\[a^{i}_{j}(x)\equiv\frac{\partial h_{i}(x)}{\partial x_{j}}. \tag{2}\]

This admits a simple interpretation through a first-order Taylor expansion: if the input feature \(x_{j}\) were to increase by some small number \(\epsilon\in\mathbb{R}^{+}\), the neuron activation would change by \(\epsilon\cdot a^{i}_{j}(x)+\mathcal{O}(\epsilon^{2})\). The larger the absolute value of the gradient, the stronger the effect of a change in the input feature. We emphasize that our method is _agnostic_ to the gradient attribution method, as different methods may be more appropriate for different tasks. For a comprehensive review of different methods, assumptions, and trade-offs, see Ancona et al. (2017). For completeness, we also note another category of attribution methods is built around _perturbations_: this class of methods evaluates contributions of individual features through repeated perturbations. Generally speaking, they are more computationally inefficient due to the multiple forward passes through the neural network and are difficult to include directly in the training objective.

### Rewarding Orthogonalization and Specialization

The main contribution of this work is proposing regularization on neuron attributions. In the most general sense, any function of any neuron attribution method could be used as a regularization term, thus encoding prior knowledge about the properties a model should have.

Specifically, the regularization term is a function of the network parameters \(\theta\) and \(x\), i.e., \(\mathcal{R}(\theta,x)\), and encourages prior assumptions on desired behavior of the learned function. Biological sensory neurons are highly specialized. For example, certain visual neurons respond to a specific set of visual features including edges and orientations within a single receptive field. They are thus highly _selective_ with _limited capacity_ to react to specific physiological stimuli (Johnston & Dark, 1986; Cowan et al., 2005). Similarly, we hypothesize that neurons that are more specialized and pay attention to sparser signals should exhibit better generalization performance. We propose the following desiderata and corresponding regularization terms:

* **Specialization.** The contribution of input features to the activation of a particular neuron should be sparse, i.e., \(||a^{i}(x)||\) is small for all \(i\in[d_{H}]\) and \(x\in\mathcal{X}\). Intuitively, in higher-dimensional settings, a few features should account for a large percentage of total attributions while others are near zero, resulting in more _specialized_ neurons. We write this as a regularization term for mini-batch training: \[\mathcal{L}_{\mathrm{spec}}(x)=\frac{1}{B}\sum_{b=1}^{B}\frac{1}{d_{H}}\sum_{i =1}^{d_{H}}\lVert a^{i}(x_{b})\rVert_{1},\] where \(b\in[B]\) is the batch index of \(x_{b}\in\mathcal{X}\) and \(\lVert\cdot\rVert_{1}\) denotes the \(l_{1}\) norm.
* **Orthogonalization.** Different neurons should attend to non-overlapping subsets of input features given a particular input sample. To encourage this, we penalize the correlation between neuron attributions \(\rho[a^{i}(x),a^{j}(x)]\) for all \(i\neq j\) and \(x\in\mathcal{X}\). In other words, for each particular input, we

Figure 2: **Method illustration.** TANGOS regularizes the gradients with respect to each of the latent units.
want to discipline the latent units to attend to different aspects of the input. Then, expressing this as a regularization term for mini-batch training, we obtain:

\[\mathcal{L}_{\mathrm{orth}}(x)=\frac{1}{B}\sum_{b=1}^{B}\frac{1}{C}\sum_{i=2}^{d _{H}}\sum_{j=1}^{i-1}\rho\left[a^{i}(x_{b}),a^{j}(x_{b})\right].\]

Here, \(C\) is the number of pairwise correlations, \(C=\frac{d_{H}\cdot(d_{H}-1)}{2}\), and \(\rho[a^{i}(x_{b}),a^{j}(x_{b})]\in[0,1]\) is calculated using the cosine similarity \(\frac{|a^{i}(x_{b})\ a^{j}(x_{b})|}{||a^{i}(x_{b})||||a^{j}(x_{b})||_{2}}\) where \(\|\cdot\|_{2}\) denotes the \(l_{2}\) norm.

These terms can be combined into a single regularization term and incorporated into the training objective. The resulting TANGOS regularizer can be expressed as:

\[\mathcal{R}_{\texttt{TANGOS}}(x)=\lambda_{1}\mathcal{L}_{\mathrm{spec}}(x)+ \lambda_{2}\mathcal{L}_{\mathrm{orth}}(x),\]

where \(\lambda_{1}\), \(\lambda_{2}\in\mathbb{R}\) act as weighting terms. As this expression is computed using gradient signals, it can be efficiently implemented and minimized in any auto-grad framework.

## 4 _How_ and _Why_ Does TANGOS Work?

To the best of our knowledge, TANGOS is the only work to explicitly regularize latent neuron attributions. A natural question to ask is (1) _How is_ TANGOS _different from other regularization?_ While intuitively it makes sense to enforce _specialization_ of each unit and _orthogonalization_ between units, we empirically investigate if other regularizers can achieve similar effects, revealing that our method regularizes a unique objective. Having established that the TANGOS objective is unique, the next question is (2) _Why does it work?_ To investigate this question, we frame the set of neurons as an ensemble, and demonstrate that our regularization improves diversity among _weak learners_, resulting in improved out-of-sample generalization.

### TANGOS Regularizes a Unique Objective

TANGOS encourages generalization by explicitly decorrelating and sparsifying the attributions of latent units. A reasonable question to ask is if this is unique, or if other regularizers might achieve the same objective implicitly. Two alternative regularizers that one might consider are \(L2\) weight regularization and \(Dropout\). Like TANGOS, weight regularization methods implicitly and partially penalize the gradients by shrinking the weights in the neural network. Additionally, \(Dropout\) trains an ensemble of learners by forcing each neuron to be more independent. In Figure 3, we provide these results on the UCI temperature forecast dataset (Cho et al., 2020), in which data from 25 weather stations in South Korea is used to predict next-day peak temperature. We train a fully connected neural network for each regularization method. Specifically, we plot \(\mathcal{L}_{spec}\) and \(\mathcal{L}_{orth}\) for neurons in the penultimate layers and the corresponding generalization performance. We supply an extended selection of these results on additional datasets and regularizers in Appendix J.

First, we observe that TANGOS significantly decreases correlation between different neuron attributions while other regularization terms, in fact, increase them. For \(L2\) weight regularization, this suggests that as the neural network weights are made smaller, the neurons increasingly attend to the same input features. A similar effect is observed for \(Dropout\) - which has a logical explanation. Indeed, \(Dropout\) creates redundancy by forcing each latent unit to be independent of others. Naturally, this encourages individual neurons to attend to overlapping features. In contrast, TANGOS aims to achieve specialization, such that neurons pay attention to sparse, non-overlapping features.

Additionally, we note that no alternative regularizers achieve greater attribution sparsity. This does not come as a surprise for \(Dropout\), where the aim to induce redundancy in each neuron will naturally encourage individual neurons to attend to more features. While \(L2\) does achieve a similar level of sparsity, this is paired with a high \(\mathcal{L}_{orth}\) term indicating that, although the latent units do attend to sparse features, they appear to collapse to a solution in which they all attend to the same weighted subset of the input features. This, as we will discover in SS4.2, is unlikely to be optimal for out-of-sample generalization.

Therefore, we conclude that the pairing of the specialization and orthogonality objectives in TANGOS regularizes a unique objective.
### TANGOS Generalizes by Increasing Diversity Among Latent Units

Having established how TANGOS differs from existing regularization objectives, we now turn to answer _why_ it works. In this section, we provide an alternative perspective on the effect of TANGOS regularization in the context of ensemble learning. A predictive model \(f(x)\) may be considered as an ensemble model if it can be written in the form \(f(x)=\sum_{T_{k}\in\mathcal{T}}\alpha_{k}T_{k}(x)\), where \(\mathcal{T}\) represents a set of basis functions sometimes referred to as _weak learners_ and the \(\alpha_{k}\)'s represent their respective scalar weights. It is therefore clear that each output of a typical neural network may be considered an ensemble predictor with every latent unit in its penultimate layer acting as a weak learner in their contribution to the model's output. More formally, in this setting \(T_{k}(x)\) is the activation of latent unit \(k\) with respect to an input \(x\) and \(\alpha_{k}\) is the subsequent connection to the output activation. With this in mind, we present the following definition.

**Definition 4.1**.: _Consider an ensemble regressor \(f(x)=\sum_{T_{k}\in\mathcal{T}}\alpha_{k}T_{k}(x)\) trained on \(\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}\) where each \((x,y)\) is drawn randomly from \(P_{XY}\). Additionally, the weights are constrained such that \(\sum_{k}\alpha_{k}=1\). Then, for a given input-label pair \((x,y)\), we define:_

1. _The overall ensemble error as:_ \(\mathrm{Err}=(f(x)-y)^{2}\)_._
2. _The weighted errors of the weak learners as:_ \(\overline{\mathrm{Err}}=\sum_{k}\alpha_{k}(T_{k}(x)-y)^{2}\)_._
3. _The ensemble diversity as:_ \(\mathrm{Div}=\sum_{k}\alpha_{k}(T_{k}(x)-f(x))^{2}\)_._

Intuitively, \(\overline{\mathrm{Err}}\) provides a measure of the strength of the ensemble members while \(\mathrm{Div}\) measures the diversity of their outputs. To understand the relationship between these two terms and the overall ensemble performance, we consider Proposition 1.

**Proposition 1** (Krogh & Vedelsby 1994).: _The overall ensemble error for an input-label pair \((x,y)\) can be decomposed into the weighted errors of the weak learners and the ensemble diversity such that:_

\[\mathrm{Err}=\overline{\mathrm{Err}}-\mathrm{Div}. \tag{3}\]

This decomposition provides a fundamental insight into the success of ensemble methods: an ensemble's overall error is reduced by decreasing the average error of the individual weak learners and increasing the diversity of their outputs. Successful ensemble methods explicitly increase ensemble diversity when training weak learners by, for example, sub-sampling input features (random forest, Breiman, 2001), sub-sampling from the training data (bagging, Breiman, 1996) or error-weighted input importance (boosting, Buhlmann, 2012).

Returning to the specific case of neural networks, it is clear that TANGOS provides a similar mechanism of increasing diversity among the latent units that act as weak learners in the penultimate

Figure 3: **Comparison of regularization objectives. (Top) Generalization performance of key regularization techniques, (Bottom) corresponding neuron attributions evaluated on the test set. \(L2\) and \(DO\) can reduce overfitting, but neuron attributions are in fact becoming more correlated. TANGOS achieves the best generalization performance by penalizing a different objective.**
layer. By forcing the latent units to attend to sparse, uncorrelated selections of features, the learned ensemble is encouraged to produce diverse learners whilst maintaining coverage of the entire input space in aggregate. In Figure 4, we demonstrate this phenomenon in practice by returning to the UCI temperature forecast regression task. We provide extended results in Appendix J. We train a fully connected neural network with two hidden layers with the output layer weights constrained such that they sum to 1. We observe that regularizing with TANGOS increases diversity of the latent activations resulting in improved out-of-sample generalization. This is in contrast to other typical regularization approaches which also improve model performance, but exclusively by attempting to reduce the error of the individual ensemble members. This provides additional motivation for applying TANGOS in the tabular domain, an area where traditional ensemble methods have performed particularly well.

## 5 Experiments

In this section, we empirically evaluate TANGOS as a regularization method for improving generalization performance. We present our benchmark methods and training architecture, followed by extensive results on real-world datasets. There are a few main aspects that deserve empirical

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline Dataset & Baseline & L1 & L2 & DO & BN & IN & MU & TANGOS \\ \hline \multicolumn{10}{|c|}{Regression (Mean Squared Error)} \\ \hline FB & 0.037 & 0.081 & **0.029** & 0.060 & 0.699 & 0.043 & 0.147 & 0.032 \\ BH & 0.192 & 0.197 & 0.183 & 0.209 & 0.190 & 0.215 & 0.286 & **0.166** \\ WE & 0.118 & 0.096 & 0.099 & 0.097 & **0.090** & 0.101 & 0.146 & 0.093 \\ BC & 0.323 & 0.263 & 0.277 & 0.282 & 0.294 & 0.308 & 0.323 & **0.244** \\ WQ & 0.673 & 0.641 & 0.644 & 0.658 & 0.639 & 0.669 & 0.713 & **0.637** \\ SC & 0.422 & 0.408 & 0.411 & 0.423 & 0.410 & 0.434 & 0.547 & **0.387** \\ FF & 1.274 & 1.280 & 1.274 & 1.266 & 1.330 & **1.201** & 1.289 & 1.276 \\ PR & 0.624 & 0.611 & 0.580 & 0.592 & 0.647 & 0.591 & 0.745 & **0.573** \\ ST & 0.419 & 0.416 & 0.418 & 0.387 & 0.461 & 0.539 & **0.380** & 0.382 \\ AB & 0.345 & 0.319 & 0.332 & **0.312** & 0.348 & 0.355 & 0.366 & 0.325 \\ \hline \hline \multicolumn{10}{|c|}{Avg Rank} & 5.4 & 3.8 & 3.4 & 4.0 & 5.0 & 5.5 & 7.1 & **1.9** \\ \hline \multicolumn{10}{|c|}{Classification (Mean Negative Log-Likelihood)} \\ \hline \hline HE & 0.490 & 0.472 & 0.431 & 0.428 & 0.459 & 0.435 & **0.416** & 0.426 \\ BR & 0.074 & 0.070 & 0.070 & 0.078 & 0.080 & 0.071 & 0.095 & **0.069** \\ CE & 0.519 & **0.395** & 0.407 & 0.436 & 0.604 & 0.457 & 0.472 & 0.408 \\ CR & 0.464 & 0.405 & 0.402 & 0.456 & 0.460 & 0.481 & 0.448 & **0.369** \\ HC & 0.320 & 0.222 & 0.226 & 0.237 & 0.257 & 0.312 & 0.248 & **0.215** \\ AU & 0.448 & 0.442 & 0.385 & 0.405 & 0.549 & 0.479 & 0.478 & **0.379** \\ TU & 1.649 & 1.633 & 1.613 & 1.621 & **1.484** & 1.646 & 1.657 & 1.495 \\ EN & 1.040 & 1.040 & 1.042 & 1.058 & 1.098 & 1.072 & 1.065 & **0.974** \\ TH & 0.700 & 0.506 & **0.500** & 0.714 & 0.785 & 0.638 & 0.618 & 0.513 \\ SO & 0.606 & **0.238** & 0.382 & 0.567 & 0.484 & 0.540 & 0.412 & 0.371 \\ \hline \hline Avg Rank & 6.4 & 3.0 & 2.7 & 4.8 & 6.3 & 6.0 & 5.2 & **1.7** \\ \hline \end{tabular}
\end{table}
Table 1: **Stand-Alone Regularization. Comparison of regularizers on regression and classification in terms of test MSE and NLL. All models are trained on real-world datasets using 5-fold cross-validation and final evaluation reported on a held-out test set. **Bold** indicates the best performance. The average rank of each method across both regression and classification is included in the final row of the respective tables.**

Figure 4: **Neuron Diversity. Overall ensemble error and decomposition in terms of diversity and average error of the weak learners. Note while all methods achieve low overall error, TANGOS is the only method that does so by increasing the diversity among the latent units.**
investigation, which we investigate in turn: \(\blacktriangleright\)**Stand-alone performance.** SS5.1 Comparing the performance of TANGOS, where the focus is on applying it as a stand-alone regularizer, to a variety of benchmarks on a suite of real-world datasets. \(\blacktriangleright\)**In tandem performance.** SS5.2 Motivated by our unique regularization objective and our analysis in SS4, we demonstrate that applying TANGOS_in conjunction_ with other regularizers can lead to even greater gains in generalization performance. \(\blacktriangleright\)**Modern architectures.** SS5.3 We evaluate performance on a state-of-the-art tabular architecture and compare to boosting. All experiments were run on NVIDIA RTX A4000 GPUs. Code is provided on Github12.

Footnote 1: [https://github.com/alanjeffares/TANGOS](https://github.com/alanjeffares/TANGOS)

**TANGOS.** We train TANGOS regularized models as described in Algorithm 1 in Appendix F. For the specialization parameter we search for \(\lambda_{1}\in\{1,10,100\}\) and for the orthogonalization parameter we search for \(\lambda_{2}\in\{0.1,1\}\). For computational efficiency, we apply a sub-sampling scheme where 50 neuron pairs are randomly sampled for each input (for further details see Appendix F).

**Benchmarks.** We evaluate TANGOS against a selection of popular regularizer benchmarks. First, we consider weight decay methods **L1** and **L2** regularization, which sparsify and shrink the learnable parameters. For the regularizers coefficients, we search for \(\lambda\in\{0.1,0.01,0.001\}\) where regularization is applied to all layers. Next, we consider Dropout **(DO)**, with drop rate \(p\in\{10\%,25\%,50\%\}\), and apply DO after every dense layer during training. We also consider implicit regularization in batch normalization (**BN**). Lastly, we evaluate data augmentation techniques Input Noise (**IN**), where we use additive Gaussian noise with mean 0 and standard deviation \(\sigma\in\{0.1,0.05,0.01\}\) and MixUp (**MU**). Furthermore, each training run applies early stopping with patience of 30 epochs. In all experiments, we use 5-fold cross-validation to train and validate each benchmark. We select the model which achieves the lowest validation error and provide a final evaluation on a held-out test set.

### Generalization: Stand-Alone Regularization

For the first set of experiments, we are interested in investigating the individual regularization effect of TANGOS. To ensure a fair comparison, we evaluate the generalization performance on held-out test sets across a variety of datasets.

**Datasets.** We employ \(20\) real-world tabular datasets from the UCI machine learning repository. Each dataset is split into \(80\%\) for cross-validation and the remaining \(20\%\) for testing. The splits are standardized on just the training data, such that features have mean \(0\) and standard deviation \(1\) and categorical variables are one-hot encoded. See Appendix L for further details on the \(20\) datasets used.

**Training and Evaluation.** To ensure a fair comparison, all regularizers are applied to an MLP with two ReLU-activated hidden layers, where each hidden layer has \(d_{H}+1\) neurons. The models are trained using Adam optimizer with a dataset-dependent learning rate from \(\{0.01,0.001,0.0001\}\) and are trained for up to a maximum of \(200\) epochs. For regression tasks, we report the average Mean Square Error (MSE) and, on classification tasks, we report the average negative log-likelihood (NLL).

**Results.** Table 1 provides the benchmarking results for individual regularizers. We observe that TANGOS achieves the best performance on \(10/20\) of the datasets. We also observe that on 6 of the remaining datasets, TANGOS ranks second. This is also illustrated by the ranking plot in Appendix H. There we also provide a table displaying standard errors. As several results have overlapping error intervals, we also assess the magnitude of improvement by performing a non-parametric Wilcoxon signed-rank sum test (Wilcoxon, 1992) paired at the dataset level. We compare TANGOS to the best-performing baseline method (L2) as a one-tailed test for both the regression and classification results obtaining p-values of 0.006 and 0.026 respectively. This can be interpreted as strong evidence to suggest the difference is statistically significant in both cases. Note that a single regularizer is seldom used by itself. In addition to a stand-alone method, it remains to be shown that TANGOS brings value when used with other regularization methods. This is explored in the next section.

### Generalisation: In Tandem Regularization

Motivated by the insights described in SS4, a natural next question is whether TANGOS can be applied in conjunction with existing regularization to unlock even greater generalization performance. In this set of experiments, we investigate this question.
**Setup.** The setting for this experiment is identical to SS5.1 except now we consider the six baseline regularizers _in tandem_ with TANGOS. We examine if pairing our proposed regularizer with existing methods results in even greater generalization performance. We again run 5-fold cross-validation, searching over the same hyperparameters, with the final models evaluated on a held-out test set.

**Results.** We summarize the aggregated results over the datasets for each of the six baseline regularizers in combination with TANGOS in Figure 5. Consistently across all regularizers in both the regression and the classification settings, we observe that adding TANGOS regularization improves test performance. We provide the full table of results in the supplementary material. We also note an apparent interaction effect between certain regularizers (i.e. input noise for regression and dropout for classification), where methods that seemed to not be particularly effective as stand-alone regularizers become the best-performing method when evaluated in tandem. The relationship between such regularizers provides an interesting direction for future work.

### Closing the Gap on Boosting

In this experiment, we apply TANGOS regularization to a state-of-the-art deep learning architecture for tabular data (Gorishniy et al., 2021) and evaluate its contribution towards producing competitive performance against leading boosting methods. We provide an extended description of this experiment in Appendix B and results in Table 2. We find that TANGOS provides moderate gains in this setting, improving performance relative to state-of-the-art boosting methods. Although boosting approaches still match or outperform deep learning in this setting, in Appendix C we argue that deep learning may also be worth pursuing in the tabular modality for its other distinct advantages.

## 6 Discussion

In this work, we have introduced TANGOS, a novel regularization method that promotes specialization and orthogonalization among the gradient attributions of the latent units of a neural network. We showed _how_ this regularization objective is distinct from other popular methods and motivated _why_ it provides out-of-sample generalization. We empirically demonstrated TANGOS utility with extensive experiments. This work raises several exciting avenues for **future work** including (1) developing TANGOS beyond the tabular setting (e.g. images), (2) investigating alternative efficient methods for achieving specialization and orthogonalization, (3) proposing other latent gradient attribution regularizers, (4) augmenting TANGOS for specific applications such as multi-modal learning or increased interpretability (see Appendix C).

\begin{table}
\begin{tabular}{|c|c|c c|c c|} \hline \multirow{2}{*}{Setting} & \multirow{2}{*}{Dataset} & \multicolumn{2}{c|}{FT-Transformer} & \multicolumn{2}{c|}{Boosting} \\ \cline{3-6}  & & Baseline & \(\mathbf{+}\)TANGOS & XGBoost & CatBoost \\ \hline \multirow{2}{*}{Default} & Jannis & \(0.714\pm 0.002\) & \(\mathbf{0.720}\pm 0.000\) & \(0.711\pm 0.000\) & \(\mathbf{0.724}\pm 0.001\) \\  & Higgs & \(0.721\pm 0.002\) & \(\mathbf{0.723}\pm 0.000\) & \(0.717\pm 0.000\) & \(\mathbf{0.728}\pm 0.001\) \\ \hline \multirow{2}{*}{Tuned} & Jannis & \(0.720\pm 0.001\) & \(\mathbf{0.727}\pm 0.001\) & \(0.724\pm 0.000\) & \(\mathbf{0.727}\pm 0.001\) \\  & Higgs & \(0.727\pm 0.002\) & \(\mathbf{0.729}\pm 0.002\) & \(0.728\pm 0.001\) & \(\mathbf{0.729}\pm 0.002\) \\ \hline \end{tabular}
\end{table}
Table 2: **FT-Transformer Architecture and Boosting. Adding TANGOS regularization can contribute to closing the gap between state-of-the-art tabular architectures and leading boosting methods. We report mean accuracy \(\pm\) standard deviation.**

Figure 5: **In Tandem Regularization. Aggregated errors across the 10 regression datasets (left) and the 10 classification datasets (right). In all cases, the addition of TANGOS provides superior performance over the standalone regularizer.**
## Acknowledgments

We thank the anonymous ICLR reviewers as well as members of the van der Schaar lab for many insightful comments and suggestions. Alan Jeffares is funded by the Cystc Fibrosis Trust. Tennison Liu would like to thank AstraZeneca for their sponsorship and support. Fergus Imrie and Mihaela van der Schaar are supported by the National Science Foundation (NSF, grant number 1722516). Mihaela van der Schaar is additionally supported by the Office of Naval Research (ONR).

## Reproducibility Statement

We have attempted to make our experimental results easily reproducible by both a detailed description of our experimental procedure and providing the code used to produce our results ([https://github.com/alanjeffares/TANGOS](https://github.com/alanjeffares/TANGOS)). Experiments are described in Section 5 with further details in Appendices F and L. All datasets used in this work can be freely downloaded from the UCI repository (Dua et al., 2017) with specific details provided in Appendix L.
