Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks
  via Aggregated Early Exits
Today, artificial neural networks are the state of the art for solving a
variety of complex tasks, especially in image classification. Such
architectures consist of a sequence of stacked layers with the aim of
extracting useful information and having it processed by a classifier to make
accurate predictions. However, intermediate information within such models is
often left unused. In other cases, such as in edge computing contexts, these
architectures are divided into multiple partitions that are made functional by
including early exits, i.e. intermediate classifiers, with the goal of reducing
the computational and temporal load without extremely compromising the accuracy
of the classifications. In this paper, we present Anticipate, Ensemble and
Prune (AEP), a new training technique based on weighted ensembles of early
exits, which aims at exploiting the information in the structure of networks to
maximise their performance. Through a comprehensive set of experiments, we show
how the use of this approach can yield average accuracy improvements of up to
15% over traditional training. In its hybrid-weighted configuration, AEP's
internal pruning operation also allows reducing the number of parameters by up
to 41%, lowering the number of multiplications and additions by 18% and the
latency time to make inference by 16%. By using AEP, it is also possible to
learn weights that allow early exits to achieve better accuracy values than
those obtained from single-output reference models.
Simone Sarti, Eugenio Lomurno, Matteo Matteucci
# Anticipate, Ensemble and Prune:

###### Abstract

Today, artificial neural networks are the state of the art for solving a variety of complex tasks, especially in image classification. Such architectures consist of a sequence of stacked layers with the aim of extracting useful information and having it processed by a classifier to make accurate predictions. However, intermediate information within such models is often left unused. In other cases, such as in edge computing contexts, these architectures are divided into multiple partitions that are made functional by including early exits, i.e. intermediate classifiers, with the goal of reducing the computational and temporal load without extremely compromising the accuracy of the classifications. In this paper, we present Anticipate, Ensemble and Prune (AEP), a new training technique based on weighted ensembles of early exits, which aims at exploiting the information in the structure of networks to maximise their performance. Through a comprehensive set of experiments, we show how the use of this approach can yield average accuracy improvements of up to 15% over traditional training. In its hybrid-weighted configuration, AEP's internal pruning operation also allows reducing the number of parameters by up to 41%, lowering the number of multiplications and additions by 18% and the latency time to make inference by 16%. By using AEP, it is also possible to learn weights that allow early exits to achieve better accuracy values than those obtained from single-output reference models.

 Early Exits, Ensemble, Pruning, AEP, Image Classification, Convolutional Neural Networks

## I Introduction

Over the last decade, deep learning has emerged as one of the dominant disciplines in computer science, thanks to the research conducted and the remarkable results obtained. Among the main fields of application, that of visual recognition, and in particular that of image classification, has undoubtedly been the catalyst for a veritable revolution, rooted in the development and refinement of architectures known as convolutional neural networks (ConvNets). Such artificial neural networks involve numerous convolutional layers that extract the information contained in input images and process it to obtain high-level features.

The development of the first ConvNet, called AlexNet [1], was made possible by the increasing production and storage of data and, in particular, by the public release of the ImageNet benchmark [2]. From then on, the succession of discoveries of increasingly high-performance models accelerated. Among the major milestones, architectures such as VGG [3], Inception [4], ResNet [5], DenseNet [6], MobileNet [7], EfficientNet [8], and recently ConvNeK [9] excelled in setting new levels in terms of accuracy, scalability, efficiency and design quality.

Recently, the study of neural networks with early exits has gained importance. An early exit of a neural network is a classifier placed at an intermediate level between the input layer and the traditional single output layer. The objectives of such a design pattern are many and varied, including exploiting the information contained in the intermediate layers of the models, streamlining their overall weight by cutting them, or for purposes related to distributed systems and edge computing [10]. The optimal number of branches with early exits and their positioning represent an important choice in this context, especially for very deep ConvNets or for architectures that are not strictly sequential. Another fundamental step lies in the choice of updating the weights of such architectures with multiple outputs, their individual or joint use, and the management of outputs with degraded performance.

This work shows the analysis of the behaviour of the main ConvNets presented in the literature modified through the proposed early exits technique named Anticipate, Ensemble and Prune (AEP). In particular, AEP is presented as an early exits-weighted ensemble technique. Outputs aggregation strategies for both loss function and inference are discussed, in order to understand which conditions favour an improvement or lead to a loss in classification performance compared to the basic single-output version of the neural networks examined. Finally, through the adoption of a pruning step, it is shown how it is possible to reduce the number of parameters, operations and network latency, further increasing accuracy through the extraction of the optimal sub-ensemble network. The experiments are conducted on a large set of ConvNets and datasets and both in a traditional training context and through the tuning of pre-trained architectures. Unlike the main reference works in the literature, which aim to reduce latency as much as possible without sacrificing model accuracy or to develop techniques for edge computing [11, 12, 13], this work aims to quantify and maximise the accuracy gain that
the use of early exits' ensemble can provide.

The document is further divided into five sections. The Section II summarises related work that has been proposed in the literature and is useful for understanding the rest of the document. The Section III describes the steps that make up the AEP technique. The section IV describes the details of the experiments and the configurations with which they were performed. The section V presents and discusses the results of the experiments. Finally, the section VI concludes the work and suggests some possible research directions.

## II Related Works

Early exits is a deep learning technique that aims to improve the efficiency of neural network models by allowing them to make predictions before the input data is processed by all the available layers. This is done by training multiple sub-models within the backbone model, each with a different level of complexity. One of the first works in which the early exits technique is used was carried out by Panda _et al._ and applied to convolutional neural networks (ConvNets) [14]. In particular, the aim of this study was to create an algorithm capable of identifying the optimal depth within the classification network under examination, so that the computational expense could be dynamically adjusted without losing accuracy. During the same period, Teerapittayanon _et al._ presented their work on early exits aimed at demonstrating the predictive properties of classifiers placed in the intermediate layers of ConvNets, such that the easiest samples to predict are processed by fewer hidden layers, while the most difficult ones traverse the entire architecture [15].

A more recent approach aimed at reducing the energy cost and complexity of single-output ConvNets has been proposed by Wang _et al._ achieving an extremely beneficial trade-off between accuracy and flops [16]. The technique implemented involves the use of early exits and weighted loss functions applied to architectures containing skip connections. Pacheco _et al._ demonstrated how the use of early exit architectures is incredibly beneficial in the context of edge computing [17]. In particular, they shown how the ability to classify non-anomalous samples at the shallow levels of a ConvNet allows not losing performance compared to a single-exit classification.

Having several available classifiers, some of them not necessarily high-performing, the most intuitive step is to identify an intelligent aggregation strategy to exploit their joint potential. Ensembling, largely seen as the natural solution in many classification problems, is a machine learning technique that consists of training several different models to solve the same task and then exploiting the knowledge derived from all of them at inference time to make the best choice. The ensemble technique works because the different models have different weaknesses that are compensated by the others' strength points. Ensembling of early exits consists in classifiers sharing part of their structure and parameters, but still working on different features given the same input data. This technique has been exploited by Wolczyk _et al._ to produce an early exit-based approach in which each prediction is reused by subsequent exits, combining previous results in an ensemble-like manner [11]. The goal of this work was to minimise the prediction latency without sacrificing the accuracy performance of the proposed models.

For what concerns the training of early exit networks, joint training is the most common approach and consists in formulating a single optimization problem whose loss depends on all branches, in particular the network loss is often calculated as the weighted sum of the branches losses [15, 18, 19]. Strategies have also been devised to dynamically adjust the exits losses weights during training [16] and to improve the ensemble by combining the prediction loss with a a diversity loss [12, 20]. When early exits are trained jointly with the backbone network, they favor the learning of more discriminative features at each layer and lead to faster convergence, while also acting as regularization [15, 21].

The output of early exits ensemble can also be computed in a multitude of ways, e.g., as arithmetic mean of the predictions [19, 20], via geometric ensemble [11] or through voting strategies [12]. The effectiveness of early exits ensembles is not limited to image classification, as they've also recently been used for image captioning [22], natural language processing [12], for uncertainty quantification and biosignal classification [13, 20] and to improve robustness against adversarial attacks [19]. Moreover, early exits ensembles were employed to produce a teacher-free knowledge distillation technique by treating the aggregated predictions as the teacher predictions [23].

## III Method

The technique of ensemble based on early exits proposed in this work, and called Anticipate, Ensemble and Prune (AEP), has been fully tested in the computer vision field to solve image classification tasks, but can easily be extended to other types of data and purposes. Given a ConvNet, it is practically always possible to identify the presence of repeating blocks or stages stacked within it. Regardless of the architecture in question, it is therefore possible to replicate the classifier present at the end of the ConvNet, i.e. after the last stage, immediately after each intermediate stage, as can be seen in Figure 1.

The training of an AEP model proceeds as follows: for each batch of images with dimension \(B\), given \(C\) target classes, the algorithm lets the images flow through the network such that every classifier outputs a tensor of \(B\) elements, each representing one image and containing \(C\) values, one for each class \(c\). In this setting, the network loss is computed as the weighted sum of the categorical cross entropy loss \(L_{i}\) of each exit \(i\in[1,N]\), in a joint training fashion, as in Equation 1.

\[L_{i}=L_{cee,i}=-\frac{1}{B}\sum_{b=1}^{B}\sum_{c=1}^{C}(p_{b,c}\log(y_{b,c})) \tag{1}\]

Unlike the other approaches, in AEP the last exit is not treated differently from the intermediate ones, since its contribution to the final loss is weighted following the same weight
assignment strategy as the others. The Equation 2 represents the network loss, where \(\alpha_{i}\) is the weight assigned to the loss associated with output \(i\).

\[L=\sum_{i=1}^{N}\alpha_{i}\cdot L_{i} \tag{2}\]

The predictions obtained after the ensemble, from now on referred to as \(\hat{y}\), are calculated as a weighted sum of the outputs of each i-th output, namely \(O_{i}\). The calculation of each classification metric is performed after applying the argmax operator to the vector thus obtained. The Equation 3 represents the prediction step, where \(\beta_{i}\) is the weight assigned to the outputs associated with exit \(i\).

\[\hat{y}=\mathrm{argmax}(\sum_{i=1}^{N}\beta_{i}\cdot O_{i}) \tag{3}\]

The strategy for selecting weights \([\alpha_{1},...,\alpha_{N}]\) and \([\beta_{1},...,\beta_{N}]\) is a crucial step in AEP. To make the selection independent of the choice of backbone neural network and dataset, the weights of the loss function as well as of the prediction ensemble were tested in linearly increasing or decreasing form. Furthermore, the values of these weights were chosen such that they were strictly positive and with sum equal to one. In this way, it is possible to maintain the desired properties by having adaptive scales of weights regardless of the number of early exits within the neural network.

In the initial set of experiments, the same weights were used to compute both the network loss and the output of the ensemble. Specifically, the weights were either always increasing or always decreasing. To differentiate between these two cases, the experiments in which the weights were assigned in a descending order for both the losses and the outputs were referred to as "EEdesc", while those in which the weights were assigned in an ascending order for both the losses and the outputs were referred to as "EEasc". Ablation studies demonstrated that, while the use of decreasing weights generally led to better exits when considered individually, particularly for earlier exits, networks with ascending weights performed better in terms of ensemble prediction. To address this, the two sets of weights were decoupled, and "EEmix" networks were tested, in which the exits losses weights were assigned in descending order and the outputs weights were assigned in ascending order. Additionally, a uniform weight mode was tested, in which all exits were given the same weight. These experiments were referred to as "EEunif". A summary of the different weight modes can be found in Table I.

Upon completion of training the multi-output network, a pruning step is applied as a post-processing technique to optimize its performance. The resulting ConvNet is loaded and validated in all possible combinations of exit activation states, using the same exit weights used in training. The best sub-network is then extracted and evaluated on the test set. This selection process can aid in further improving accuracy in comparison to both the full ensemble and the single-exit network, while also reducing the number of parameters, operations, and latency as entire sections of the original network can be removed.

## IV Experiments Setup

The experiments conducted to evaluate the effectiveness of the AEP technique included several ConvNet architectures and many of the major image classification benchmarks. For an analysis characterized by high completeness, all experiments were evaluated with two different input scales, and each ConvNet was trained both through training from scratch and through fine-tuning from the weights learned on ImageNet.

### _Networks_

The AEP technique was tested on 5 well known network architectures, thus encompassing many of ConvNets most relevant architectural trends and patterns. The goal was to span different network sizes, design patterns and inner components. An abstract and general architecture was constructed as a theoretical model for the various ConvNets considered,

Fig. 1: Outline of the AEP technique. \(O_{i}\) represents the output of the i-th exit stage with linear activation. \(\not\)\(\not\)\(\not\)\(\not\)\(\not\)\(\sim\) represents the loss function, i.e., the categorical cross-entropy, while \(\not\)\(\sim\) represents the argmax function used to obtain the class prediction. \(\alpha_{i}\) and \(\beta_{i}\) represent the weights assigned to the loss and output of exit stage \(i\), respectively. The activation of the outputs is relative to the pruning step with which the proposed method terminates.
observable in Figure 1. By identifying the recurring cells in these architectures, it was possible to identify the list of stages to which early exits should be attached. At this point, each network could be enriched with the function to extract a subnetwork by specifying which exits to preserve. Below the list of networks identified and used in the experiments:

* **ResNet50**[5]: In terms of size, it is in the middle of the ResNet family of architectures. ResNets are characterized by the use of skip/residual connections between layers, which should help in improving gradient flow through the network.
* **VGG16**[3]: It is a purely sequential ConvNet, and while smaller than other VGG models, it is still by far the most demanding in terms of operations required.
* **DenseNet169**[6]: It is found in the middle of the DenseNet family of architectures, they are characterized by a dense blocks structure, in which features obtained after a layers are not only passed to the successive layers but also concatenated to the outputs of said layers, so that the information in a block is better preserved and exploited.
* **MobileNetV3Small**[7]: Designed to work under mobile settings, it was a good candidate to see what would happen with tiny architectures. It is also characterized by the use of a particular type of block called the Residual Inverted Bottleneck, characterized by high memory efficiency.
* **EfficientNetB5**[8]: Part of the EfficientNet family, currently one of the best performing architecture families, this model in particular was chosen because with its more than 500 layers and, in contrast with MobileNetV3Small, allowed to study a huge architecture.

For each of the 5 networks, the classifier present in the original single-exit architecture was extracted and replicated for each of the early exits. The only exit stage altered from the original structure was that of the VGG16 model, which by default is extremely large and over-parameterized. It was substituted it with a simpler Global Average Pooling layer followed by a dense layer with a number of units equal to the number of possible classes.

The experiments conducted considered each network both with a number of early exits set at 4, for comparison purposes, and with a number of early exits equal to the number of stages in the original model. Specifically, while the experiments on Resnet50 and DenseNet169 did not need to be repeated since 4 was already the correct number of exits to match the number of stages, VGG16 and MobileNetV3Small required also a set of experiments concerning 5 exits, while EfficientNetB5 required also tests involving 7 exits. We will refer to these 3 networks by adding the "full" keyword to their name: **VGG16full**, **MobileNetv3Smallfull** and **EfficientNetB5full**, bringing the total number of networks tested to 8.

### _Datasets and Metrics_

In this study, a thorough experimental evaluation of the AEP algorithm was conducted using six diverse image classification datasets. These datasets were chosen to cover a range of characteristics, including variations in class imbalance and image modality, in order to assess the generalizability and robustness of the proposed algorithm. The datasets used in this study spanned a range of class sizes, including datasets with tens to hundreds of classes, and are summarized in Table II.

The evaluation included comparing the performance of baseline and ensemble networks under different scenarios, including training from scratch and fine-tuning from ImageNet weights for all networks as well as using images of sizes 224x224 and 64x64 for all datasets. For each experiment, a set of performance metrics were collected, including Top1 accuracy, number of parameters, number of MACs (or Multiply-ACcumulate operations), latency, and training time. Additionally, for multi-output networks, the Top1 accuracy for each exit was also recorded.

### _Hyperparameters Setting_

The goal of this research was to evaluate the performance of single-exit networks in comparison to early-exit ensembles rather than matching or beating state of the art results. To accomplish this, a simple training approach was employed, using 100 training epochs, a batch size of 64, and a learning rate of \(10^{-4}\) with the Adam optimizer. The other training parameters were left at their default values in PyTorch. An early stopping technique was applied, with a patience of 12 epochs, based on the validation loss. No data augmentation or learning rate schedules were utilized. The only preprocessing applied to the images was resizing to 224 or 64 through bicubic interpolation and normalization. The small learning rate was chosen to ensure that the same hyperparameters could be used in fine-tuning, thereby ensuring comparable results. The models were implemented using PyTorch and executed on an NVIDIA Quadro RTX 6000 GPU.

## V Results and Discussion

This section describes the results of the experiments conducted in the present research. The analysis begins by comparing the variations in accuracy between single-output architectures and the proposed configurations. Next, the benefits of the pruning step are shown, in terms of accuracy gain, parameter reduction, optimisation of MACs and faster inference time. Finally, the accuracy performance of early exits without ensemble compared to single-output models is analysed. The symbol '*' indicates early exits experiments whose networks
have been subjected to pruning. In contrast, the absence of the symbol '*' represents full-ensemble networks.

### _Classification Accuracy_

Table III shows the results of the experiments performed in terms of the percentage change in Top1 accuracy compared to the single-output experiments, grouped by neural network in the left column and by dataset in the right column.

With regard to the experiments performed with 224x224 images and traditional training (TRAIN-224), excellent average improvements can first of all be appreciated regardless of the strategy with which AEP was applied. For each neural network
and each dataset, it is possible to identify a configuration that improves the accuracy obtained. In particular, as far as the datasets are concerned, the greatest advantages are found in CIFAR100 and TinyImageNet, which respectively obtain an average accuracy improvement of 16.23% with the EEmix* configuration and 13.94% with the EEdesc* configuration. With regard to network configurations, the improvement achieved by ResNet50 and both MobileNetV3-small versions is noteworthy.

The general considerations made for the TRAIN-224 scenario are confirmed and underlined by the results of the experiments with 64x64 images and traditional training (TRAIN-64). In this case, with the same dataset and neural network, the problem to be solved by the algorithm considered is more complex due to the reduced amount of input information. In this context, it is possible to observe how beneficial AEP is overall, increasing the quality of the models regardless of their configuration and dataset, in some cases making such improvements as to make the difference between a poorly performing model and a winning one. In this scenario, the EEdesc* technique seems to be definitely the favourite, capable of improving accuracy performance by 15% on average.

Moving from the training scenario to the fine-tuning scenario, and in particular to the one with 224x224 images (FINETUNE-224), a different behavior can be observed. Since these are pre-trained algorithms on ImageNet images with similar size, the addition of early exits and ensembles seems to have a negative overall impact on average. This is reasonable in light of the fact that the starting models have an optimized set of weights available to facilitate the production of accurate predictions from the single output at the bottom of the network. This assumption is confirmed by the counterexample represented by the results obtained from the EEasc and EEasc* configurations. In fact, it can be observed from Table III how the use of ascending weights in the loss and inference phase turns out to be more suitable by favoring strongly the contribution of the outputs close to the single output of the original model.

Finally, going to look at the results of the last pool of experiments,i.e., those with fine-tuning and 64x64 images (FINETUNE-64), it is possible to see a different behavior from the previous case. The change of spatial domain resulted in an overall improvement in the accuracy of the AEP-based models compared to the single-output counterpart. This is probably due to the higher complexity of the problem inversely proportional to the size of the input, which thus allows better exploitation of classifiers trained on lower-level and thus less elaborate features, as is not the case with large images that require more complex transformations. However, the use of ascending weights turns out to be the winning move here as well, capable of improving models accuracy by an average of 4.79%.

In general, it can be argued that ensemble networks with early exits improve much more than the baseline when trained from scratch than when fine-tuned. This is reasonable because when training from scratch by jointly optimising the exits, the network is able to improve the features of each layer to achieve a good classification result, producing better features in the layers closer to the network input. Fine-tuning, on the other hand, starts from the weights obtained after training single-output networks, which means that the features of the initial layers are not good classification features by themselves, but do produce good classification features in the final layers. Adding early exits therefore leads to contradictory behaviour, as if the optimisation were trying to override the initial weights to solve a different task. In fact, by assigning higher weights to exits closer to the input, you end up assigning high classification importance to layers that should not be able to do so, which can lead to worse performance than the baseline.

Regarding the effect of the pruning step, the same behaviour as described in full ensembles can be observed with slight improvements in accuracy. Table IV collects and compares the accuracy performance of pruned versions of early exits ensemble networks with their full-ensemble counterparts. Among the different types of networks, EEdesc networks seem to be the ones that benefit the most from the pruning phase, particularly in fine-tuning, the same learning context in which they perform the worst.

### _Parameters, Operations and Latency_

The average effect in terms of parameters of applying AEP is summarised in Table V. As can be seen from the first column, the addition of early exits has a minimal impact on the increase in parameters, which on average increases by only 2.88% compared to single-exit architectures. The key aspect is the impact of the pruning step, which in general brings considerable benefits. Paying attention to the TRAINING-64 scenario, it can be seen that the EEmix* configuration results in an average parameter reduction of 41.02% and in parallel an average accuracy gain of 13.31%, as described in Table III.

With regard to the number of MACs, the average results of the experiments conducted are summarised in Table VI. Also for this metric, the addition of early exits alone has a negative impact, albeit with an average computational increase of only 5%. On the contrary, it is possible to observe the beneficial effect of pruning, which in addition to providing the previously discussed benefits, goes as far as reducing the
number of operations by up to a maximum of 17.95% in the EEmix* case.

A similar behaviour can be observed in the inference latency domain, as shown by the results of the experiments summarised in Table VII. Adding more outputs and calculating their weighted sum is clearly disadvantageous from a time point of view. In percentage terms, this phenomenon is more pronounced in architectures with input sizes 64x64. The cut of unnecessary outputs implemented by the pruning operation inevitably benefits inference times, as the image input to the networks will on average have to travel through a portion of the architecture and not its entirety. Fine-tuning scenarios tend to benefit less from the pruning step precisely because, as can be deduced from Table V and Table VI, on average only the very last outputs, or none at all, are cut out. Consequently, although mitigated, the addition of early exits is only partially beneficial, as opposed to scenarios characterised by training from scratch.

### _Early Exits Analysis_

An interesting aspect is certainly that of the quality of the classifiers obtained prior to the pruning operation. In fact, the proposed training, whether from scratch or fine-tuning, has the purpose of optimising the ensemble to be pruned, but has the secondary effect of computing weights that can be used directly by the classifiers obtained thanks to early exits. For each of the scenarios considered, Table VIII presents the average percentage change in accuracy of the early outputs common to all configurations compared to the average performance achieved by single-output networks.

The first notable result is the presence of early exits performing better than the single output network. This occurs in every scenario except FINETUNE-224, managing to achieve average accuracy improvements of up to 9.96%. This confirms not only the effectiveness of AEP as an ensemble-based training technique, but also opens up the possibility of realising shallower and at the same time more performant architectures, particularly useful in mobile or edge or distributed setups. Focusing on the TRAIN-64 and TRAIN-224 scenarios, it is evident that both third and fourth outputs represent an advantageous alternative to the traditional single output regardless of the AEP configuration used. This result is perhaps the most significant. It is important to reflect on the fact that sometimes
good human intuitions, such as increasing the complexity of a model, do not necessarily translate into improvements in performance. On the contrary, better results can be achieved by optimising the learning process and exploiting various levels of abstraction of the input data, as is done through the use of multiple outputs.

## VI Conclusion

This paper presented Anticipate, Ensemble and Prune (AEP), the early exits and ensemble-based technique for improving the performance of single-output artificial neural networks. The extensive series of experiments conducted demonstrated how this approach can be applied in the context of image classification, achieving remarkable improvements in terms of accuracy, parametric complexity, number of operations and inference time. The experiments were also replicated in the context of fitting pre-trained networks using fine-tuning techniques, with excellent results, especially in the absence of high-resolution input images. It was also shown how AEP can optimise the performance of individual early exits, exceeding the accuracy of single-exit models even without using the ensemble technique. We believe this can be an important step towards optimising the efficiency of neural architectures, especially for mobile and edge scenarios. In a follow-up study, AEP will include an automatic optimisation process to find the best loss weights and a search process to find the best output weights, which is likely to lead to further improvements and potentially reveal undiscovered design patterns.

## Acknowledgment

The European Commission has partially funded this work under the H2020 grant N. 101016577 AI-SPRINT: AI in Secure Privacy-pReserving computlNg conTinuum.
