Hidden Classification Layers: Enhancing linear separability between
  classes in neural networks layers
In the context of classification problems, Deep Learning (DL) approaches
represent state of art. Many DL approaches are based on variations of standard
multi-layer feed-forward neural networks. These are also referred to as deep
networks. The basic idea is that each hidden neural layer accomplishes a data
transformation which is expected to make the data representation "somewhat more
linearly separable" than the previous one to obtain a final data representation
which is as linearly separable as possible. However, determining the
appropriate neural network parameters that can perform these transformations is
a critical problem. In this paper, we investigate the impact on deep network
classifier performances of a training approach favouring solutions where data
representations at the hidden layers have a higher degree of linear
separability between the classes with respect to standard methods. To this aim,
we propose a neural network architecture which induces an error function
involving the outputs of all the network layers. Although similar approaches
have already been partially discussed in the past literature, here we propose a
new architecture with a novel error function and an extensive experimental
analysis. This experimental analysis was made in the context of image
classification tasks considering four widely used datasets. The results show
that our approach improves the accuracy on the test set in all the considered
cases.
Andrea Apicella, Francesco IsgrÃ², Roberto Prevete
Hidden Classification Layers: a study on Data Hidden Representations with a Higher Degree of Linear Separability between the Classes

###### Abstract

In the context of classification problems, Deep Learning (DL) approaches represent state of art. Many DL approaches are based on variations of standard multi-layer feed-forward neural networks. These are also referred to as deep networks. The basic idea is that each hidden neural layer accomplishes a data transformation which is expected to make the data representation "somewhat more linearly separable" than the previous one to obtain a final data representation which is as linearly separable as possible. However, determining the appropriate neural network parameters that can perform these transformations is a critical problem. In this paper, we investigate the impact on deep network classifier performances of a training approach favouring solutions where data representations at the hidden layers have a higher degree of linear separability between the classes with respect to standard methods. To this aim, we propose a neural network architecture which induces an error function involving the outputs of all the network layers. Although similar approaches have already been partially discussed in the past literature, here we propose a new architecture with a novel error function and an extensive experimental analysis. This experimental analysis was made in the context of image classification tasks considering four widely used datasets. The results show that our approach improves the accuracy on the test set in all the considered cases.

keywords: neural networks; hidden layers, hidden representations, linearly separable +
Footnote †: journal: Pattern Recognition Letters

## 1 Introduction

Nowadays, the success of Deep Learning (DL) approaches has led to an increase in interest in Multi-Layer Feed-Forward (MLFF) neural networks (LeCun et al., 2015) insofar as a successful class of deep neural networks consists of MLFF networks with more than one hidden layer and possibly some specific architectural choices. In the rest of the paper we will refer to such Deep Neural Networks as DNNs. In a nutshell, DNN networks are computational architectures organised as \(L\) consecutive layers or levels of elementary computing units, called _neurons_. The last layer \(L\) is the _output_ layer, and the remaining layers are usually called _hidden_ or _internal_ layers.

In a DNN network each hidden layer \(l\) performs a non-linear functional map \(\Phi_{\theta_{l}}^{l}\) from the output of the previous layer \(\mathbf{z}^{l-1}\) (and possibly other previous layers) to the output of the layer itself. Where \(\theta_{l}\) are the weights associated to the connections incoming into the layer \(l\), plus the biases of the layer. By contrast, the output layer may also perform a linear transformation. In other words, the whole computation of a DNN can be viewed as a non-linear parametric functional mapping \(\mathbf{y}=M(\mathbf{x};\theta)\) from a \(d\)-dimensional space to a \(c\)-dimensional space, where \(d\) is the number of input variables and \(c=m_{L}\) is the number of neurons in the output layer. The parameters \(\theta\) are the weights and biases of the network, and \(\mathbf{y}\) are the output values of the output layer.

Although from a theoretical point of view the DNNs capability of being universal approximators has been extensively discussed (Cohen et al., 2016; Huang et al., 2000; Longstaff and Cross, 1987), together with the inducted hidden feature representation spaces (Lerner et al., 1999), it is important to notice that the difficult to effectively find the most suitable \(\theta\) remains. In particular, when DNNs are applied in the context of classification problems, one has to find the parameters \(\theta_{l}\) such that the composition of \(L-1\) non-linear transformations \(\mathbf{z}^{L-1}=\Phi_{L-1}\big{(}\Phi_{L-2}(\ldots\Phi_{1}(\mathbf{x}))\big{)}\) maps each input \(\mathbf{x}\) from a _non-linearly_ separable space into a _linearly separable_ one. In fact, in the context of classification problems,
one of the main goals is to find a suitable data representation which allows to obtain a linearly separable classification problem. Plausibly, when a DNN is used, each internal representation \(\mathbf{z}_{l}\) can be expected to make the representations of \(\mathbf{x}\) "somewhat more linearly separable" than the previous one \(\mathbf{z}_{l-1}\). We underline that the complexity of a classification problem can be measured with respect different aspects, however class separability is a key aspect and different levels of class separability can be quantified (Loreena et al., 2019). In particular, in (Schilling et al., 2021) the Generalized Discrimination Value (GDV) to measure the separability between two dataset is introduced. The GDV is defined as the gap between the mean intra-cluster and the mean inter-cluster distances, computed on a set of labeled data represented in some space. More in detail, the GDV compares in a quantitative way the degree of class separability between two data representations. Since GDV can be computed on different types of representations, it can be also used to compare the separability of the same data represented in different spaces, such as the different representations returned by different neural networks' layers.

However, we again emphasise that how to determine the appropriate parameters \(\theta\) from a data set by a supervised learning process minimizing an error (or loss) function is still a critical problem. We notice that error functions usually depend on the final network output values only, without taking care about the results obtained in the hidden layers. Thus, starting from the previous considerations, in this paper we investigate the possibility to achieve a supervised learning approach which favours solutions where \(\mathbf{x}\)'s representations at the hidden levels have a higher degree of linear separability between the classes with respect to standard approaches. To this aim, we propose a DNN architecture which induces an error function involving the output values of all the network layers. More specifically, as we will discuss in more detail in Section 2, the output of each hidden layer \(l\) is sent to an additional linear output layer which is trained to classify the input \(x\) on the basis of the input representation encoding in the layer \(l\) (see Figure 1). From now on, we named this architecture Hidden Classification Layer network (HCL). We investigated the impact of this type of solution in a series of experimental scenarios as we will discuss in more detail in Section 3.

Although similar approaches have already been partially discussed in the past literature (see, for example, (Lee et al., 2015)), here we propose both a different version in terms of both neural architecture and error function, and a more extensive experimental analysis (see Sections 2 and 3). In particular, in (Lee et al., 2015) the supervision of the hidden layer was made by SVMs instead of linear neural layers as in our case. In (Wang et al., 2020) a cascade of Convolutional Neural Networks (C-CNN) was proposed. C-CNN is composed of hidden layers combined together through dilated convolutions and trained using a proposed progress optimisation algorithm. Also in this case, our approach proposes a simpler architecture to favour hidden representations with a higher degree of class separability. The rest of the paper is organised as follow: in Section 2 the proposed method is described; Section 3 describes the experimental setup and the evaluation methods; in Section 4 the results are reported and discussed; finally, Section 5 contains final remarks.

## 2 Model description

A neural network, as reminded before, is structured in L layers of neurons. Each neuron \(i\) belonging to the \(l\)-th layer, achieves a two-step computation (see (Bishop and Nasrabadi, 2006), chapter 4): a linear combination \(a_{i}^{l}\) of the neuron's inputs is computed first, and then the neuron output \(z_{i}^{l}\) is computed by an activation function \(f_{l}(\cdot)\), i.e., \(z_{i}^{l}=f_{l}(a_{i}^{l})\). Usually, activation functions are nonlinear function (see (Apicella et al., 2021) for a review). The activation function input \(a_{i}^{l}\) is usually computed on the basis of real values, said _weights_, associated with the connections coming from the neurons belonging to the layer \(l-1\) (and possibly from other previous layers) and a bias value associated to the neuron \(i\). Each layer \(l\) is composed of \(m_{l}\) neurons, and the flow of computation proceeds from the the first hidden layer to the output layer in a forward-propagation fashion.

In this research work, we focus on \(C\)-classes classification problems, with \(C\geq 2\). In this context, Cross-Entropy (CE) loss (Wang et al., 2022) is one of the most common loss function to be optimised. Given a dataset of N samples, DS=\(\{(\mathbf{x}^{n},\mathbf{t}^{n})\}_{n=1}^{N}\), CE for the n-th sample can be expressed as follows:

\[CE^{(n)}(\theta;\mathbf{y}^{n},\mathbf{t}^{n})=-\sum_{c=1}^{C}t_{c}^{n}\log(y _{c}^{n})\]

where \(\mathbf{t}^{n}\in\{0,1\}^{C}\) is the one-hot encoding representation of the class label of the \(n\)-th sample of the dataset, and \(\mathbf{y}^{n}=\mathbf{y}(\mathbf{x}^{n};\theta)\) is the output of the neural network when it is fed with the input \(\mathbf{x}^{n}\). Finally, \(\theta\) corresponds to all the network parameters. The total CE loss is equal to the sum of the single \(CE^{(n)}\) over the dataset samples,
i.e., \(CE=\sum\limits_{n=1}^{N}CE^{(n)}\). As previously said, this loss formulation takes into account only the classification reported by the final layer of the network, without considering how the intermediate network levels affect the final classification scores. By contrast, in our model, HCL network, the data representation corresponding to the output of each hidden layer is used as input of a linear classifier so as to favour a data representation for each level as separable as possible.

More formally, given a DNN composed of \(l_{1},l_{2},\ldots,l_{L-1}\) hidden layers and a final layer \(l_{L}\) having \(C\) neurons, we connect each hidden layer \(l_{j},\ 1\leq j\leq L-2\) with a new layer \(\overline{l}_{j}\) acting as an independent classifier. Adopting a proper loss function to train each classifier \(\overline{l}_{j}\), we expect that the features learned by the associate layer \(l_{j}\) are the most discriminating as possible. In other words, additional \(L-2\) layers \(\{\overline{l}_{1},\overline{l}_{2},\ldots\overline{l}_{L-2}\}\) composed of \(C\) neurons are added, and each \(\overline{l}_{j}\) layer receives connections from the hidden layer \(l_{j}\) only, making each \(\overline{l}_{j}\) as an independent linear classifier. Therefore, given a DNN \(M\), we obtain an HCL network \(\overline{M}\) which will be composed of two distinct sets of layers: i) standard neural network layers \(\{l_{1},l_{2},\ldots,l_{L}\}\), composing M, and ii) _hidden classification layers_\(\{\tilde{l}_{1},\tilde{l}_{2},\ldots,\tilde{l}_{L-2}\}\), composing a set of layers where each layer \(\overline{l}_{i}\) favours more separable data representations in the respective hidden layer \(l_{i}\), independently from the subsequent layers. Each hidden classification layer \(\tilde{l}_{i}\) has a set of parameters \(\theta^{i}\). Each \(\theta^{i}\) is composed of a set of distinct parameters, corresponding to the connections incoming in the layer \(\tilde{l}_{i}\), and a set of shared parameters with the other \(\theta^{i}\), with \(j<i\), which correspond to the parameters of the DNN \(M\) down to the layer \(l_{i}\). In figure 1 a general scheme of the proposed approach is reported.

Denoting with \(\overline{\mathbf{z}}^{n,j}\) the scores returned by the hidden classification layer \(\overline{l}_{j}\) on the \(n\)-th input sample, we propose the following Weighted Cross Entropy (WCE) loss formulation:

\[WCE^{(n)}(\theta;\mathbf{y}^{n},\mathbf{t}^{n})=CE^{(n)}(\theta _{M};\mathbf{y}^{n},\mathbf{t}^{n})+\\ +\sum\limits_{j=1}^{L-1}\lambda_{j}\cdot CE^{(n)}(\theta^{j}; \overline{\mathbf{z}}^{n,j},\mathbf{t}^{n}) \tag{1}\]

where \(\mathbf{y}^{n}\) is the score returned by the final layer of the classifier \(M,\overline{\mathbf{z}^{n,j}}\) is the score returned by the hidden classification layer \(\overline{l}_{j}\) tied to the \(l_{j}\) layer, \(\theta^{M}\) are the parameters of the model \(M\), and \(\{\lambda_{1},\lambda_{2},\ldots,\lambda_{L}\}\) is a set of regularisation coefficients greater than or equal to 0. Setting \(\lambda_{1}=\lambda_{2}=\cdots=\lambda_{L-1}=0\) results in standard CE loss applied to the final classification layer only, while different values give different weights to the hidden classification layers \(\{\tilde{l}_{j}\}_{j=1}^{L-1}\).

## 3 Experimental assessment

### Data and neural network models

The performance of the HCM network architecture is assessed on image classification tasks considering four well-known datasets: _MNIST_, _Fashion MNIST_, _CIFAR 10_, and _CIFAR 100_. The MNISTLeCun et al. (1998) dataset consists of 70,000 grayscale images at a resolution of \(28\times 28\) representing 10 different classes (the digits from 0 to 9). It is divided in two sets: the former composed of 60,000 images usually used as training samples and the latter of the remaining 10,000 images usually used as test samples. Fashion-MNIST is a dataset of images representing fashion articles Xiao et al. (2017). Fashion-MNIST was proposed as a replacement for the original MNIST dataset for benchmarking machine learning algorithms, sharing the same image size and structure of training and testing splits. Indeed, it provides a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, representing one of the following items: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot. CIFAR-10 dataset consists of 60,000 colour images of 10 different classes, that are _airplane_, _automobile_, _bird_, _cat_, _deer_, _dog_, _frog_, _horse_, _ship_, and _truck_. The dataset provides 50,000 training images and 10,000 test images at a resolution of \(32\times 32\).

Concerning the neural network models which are used as baseline to evaluate the classification enhancement of HCL network architecture, we considered LeNet-5 LeCun et al. (1998), Hinton network Hinton et al. (2012), and ResNet18 He et al. (2016). They are among the most famous networks exploiting convolutional layers for image classification.

In its standard formulation, LeNet-5 is composed of a sequence of 3 convolutional layers interspersed by 2 sub-sampling layers, followed by 2 final full-connected layers (the latest one for classification). Instead, Hinton network is composed of three convolutional hidden layers interleaved with three maxpooling layers. Finally, the main characteristic of a ResNet is the presence of shortcuts between non-consecutive layers that allows deep networks to be easily trained. In this work, we use the 18 layer residual network (ResNet18) described in He et al. (2016).
Figure 1: a scheme of the proposed approach. Each hidden layer \(l^{l}\) of the main branch of the network produces an output \(vec_{l}^{l}\), and the final classification layer \(l^{L}\) produces the output \(vec_{l}\). For each hidden layer \(\vec{l}^{l}\), \(1\leq i<L\), a further classification layer \(\vec{l}^{l}\) is added. Each \(\vec{l}^{l}\) is fed with the respective \(\mathbf{z}^{l}\), producing an output \(\mathbf{\bar{z}}^{l}\). Therefore, all outputs \(\mathbf{\bar{z}}^{l}\), \(\forall 1\leq i<L\), are used together with the network classification output \(\mathbf{y}\) to compute the final WCE loss.
### Evaluation

Each model is evaluated on both the original version (_vanilla_) proposed in their respective works and on its modified instance as described in Section 2. All the models were trained using the same experimental setup reported in their reference papers, except for Learning Rate \(LR\), the number of Max Epochs \(ME\), and the Patience Epochs \(PE\), which can be strongly dependent by the network architecture. \(ME\) and \(PE\) are experimentally set to 1000 and 200 respectively, since we experimentally noticed that these values are enough to converge in all the analysed cases, while optimal \(LR\) and \(\lambda\) values are found through a grid-search approach. For the \(LR\), the search space was \(LR\in[10^{-5},10^{-1}]\), instead different combinations are considered for \(\lambda\) parameters. Experiments on ResNet involving CIFAR 10 and CIFAR 100 dataset were made both considering only original data and augmented data, using 4 pixels zero padding, corner cropping, and random flipping.

Importantly, in order to experimentally show that the proposed method leads toward more easily separable data representations we computed Generalized Discrimination Value (GDV) measure Schilling et al. (2021) for each vanilla network's layer and its corresponding version equipped with hidden classification layers. We expect that, as the depth of the network increases, the data representations obtained with the proposed network's layout are more easily separable respect to the representations obtained by the respective models without additional layers. Note that GDV is a measure of how well different data classes separate.GDV values result 0.0 for data points with randomly shuffled classes, and \(-1.0\) in the case of perfectly separable classes. More in detail, GDV on a data representation \(\mathbf{z}\) is defined as

\[GDV(\mathbf{z})=\frac{1}{\sqrt{D}}\Big{(}\frac{1}{L}\sum_{c=1}^ {C}d^{intra}(\mathbf{z}_{c})+\\ -\frac{2}{C(C-1)}\sum_{c=1}^{C-1}\sum_{m=c+1}^{C}d^{inter}( \mathbf{z}_{c},\mathbf{z}_{m})\Big{)}\]

where \(d^{intra}(\mathbf{z}_{c})\) is the mean intra-class distance on the data representations of the data \(\mathbf{z}_{c}\) belonging to the class \(c\), and \(d^{inter}(\mathbf{z}_{c},\mathbf{z}_{m})\) is the mean inter-class distance on the data representations \(\mathbf{z}_{c},\mathbf{z}_{m}\) belonging to the \(c\) and \(m\) classes.

## 4 Results

In Tab. 1 the test set accuracy, which was obtained by both the HCL network architecture and the vanilla networks, is reported. It is shown that the adoption of the HCL architecture improves the accuracy in all the cases, especially in the cases where a low accuracy for vanilla networks was obtained. In these cases, in fact, HCL network architecture appears to give a more significant improvement. In Fig. 3 the GDV values for each layer of each model and for CIFAR10 and CIFAR100 dataset are reported.

## 5 Conclusion

In this research work, we experimentally investigated the impact of constraining the classification complexity of the intermediate input representations with respect to their linear separability on the performances of DNNs in classification tasks. To this aim, we proposed a novel DNN architecture, which we named Hidden Classification Layer (HCL) network, where the output of each standard hidden layer is sent to a hidden classification layer trained to classify the input \(x\) based on the \(x\) representation given by the standard layer itself. HCL network architecture allows obtaining solutions with input representations at the hidden levels having a lower classification complexity with respect to their linear separability. Note that our approach can be applied in slightly different ways: 1) given an already known neural network architecture, one can, first, augment it by hidden classification layers and, then, train the whole system from scratch; 2) given an already trained neural network architecture, one can, first, augment it by hidden classification layers and, then, tune the whole system; 3) one can design and train a new neural architecture equipped with hidden classification layers. In this study, we used the first approach to test our proposal by considering three successful neural network models (LeNet-5,

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Model** & **Baseline** & **Proposed** \\ \hline
**MNIST** & LeNet5 & 99.0 & **99.2** \\  & Hinton & 98.7 & **99.4** \\ \hline
**FMNIST** & LeNet5 & 90.6 & **90.8** \\  & Hinton & 92.1 & **92.6** \\  & ResNet & 92.3 & **93.3** \\ \hline
**CIFAR10** & LeNet5 & 66.2 & **71.5** \\  & Hinton & 81.2 & **83.3** \\  & ResNet & 86.3 & **89.0** \\  & ResNet (augmented) & 94.4 & **94.6** \\ \hline
**CIFAR100** & LeNet5 & 34.9 & **38.4** \\  & Hinton & 52.5 & **53.3** \\  & ResNet & 60.1 & **63.6** \\  & ResNet (augmented) & 74.7 & **75.4** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results of the evaluation stage. For each dataset, accuracy on the test set obtained on both the vanilla models (_baseline_) and the proposed ones are reported. ResNet has not been tested on MNIST dataset due to the already very high accuracies obtained with the other models. For CIFAR10 and CIFAR100, performance was evaluated also using augmented data (_augmented_) for ResNet
Figure 3: GDV values obtained with the proposed approach (LAT) compared with the vanilla networks on the CIFAR100 dataset. On the x axis, the layer of the model and on the y axis the respective GDV value.

Figure 2: GDV values obtained with the proposed approach (LAT) compared with the vanilla networks on the CIFAR10 dataset. On the x axis, the layer of the model and on the y axis the respective GDV value.
Hinton network, and ResNet18). These models were trained with and without hidden classification layers to evaluate the impact of HCL on the model performances experimentally. Each model was trained and tested on four datasets (MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100). The results show that the HCL network has a positive impact uniformly (see Table 1). It is interesting that the proposed approach leads to a GDV improvement in almost all cases, suggesting that the HCL network architecture can help the model build more separable inner representations. Moreover, it is worth noting that, in all the cases and with and without hidden classifier layers, the GDV values exhibit only a slight decrement for the initial network layers or they have even a wavering behaviour. Just only for the last layers, there is a sharp decrement (this decrease is particularly pronounced for HCL networks). Thus, these results are consistent with (Balduzzi et al., 2017), where the authors show experimentally that, during the learning phase, the loss derivatives with respect to the network parameters behave very similarly to a random walk on the first weight layers. In fact, in our case, data separability occurs mainly in the last layers of the network at the end of the learning process.

## Funding

This work is supported by the European Union - FSE-REACT-EU, PON Research and Innovation 2014-2020 DM1062/2021 contract number 18-I-15350-2 and by the Ministry of University and Research, PRIN research project "BRIO - BIAS, RISK, OPACITY in AI: design, verification and development of Trustworthy AI.", Project no. 2020SSKZ7R.
