Aggregated f-average Neural Network applied to Few-Shot Class
  Incremental Learning
Ensemble learning leverages multiple models (i.e., weak learners) on a common
machine learning task to enhance prediction performance. Basic ensembling
approaches average the weak learners outputs, while more sophisticated ones
stack a machine learning model in between the weak learners outputs and the
final prediction. This work fuses both aforementioned frameworks. We introduce
an aggregated f-average (AFA) shallow neural network which models and combines
different types of averages to perform an optimal aggregation of the weak
learners predictions. We emphasise its interpretable architecture and simple
training strategy, and illustrate its good performance on the problem of
few-shot class incremental learning.
Mathieu Vu, Emilie Chouzenoux, Ismail Ben Ayed, Jean-Christophe Pesquet
# Aggregated \(f\)-average Neural Network for Interpretable Ensembling

###### Abstract

Ensemble learning leverages multiple models (i.e., weak learners) on a common machine learning task to enhance prediction performance. Basic ensembling approaches average the weak learners outputs, while more sophisticated ones stack a machine learning model in between the weak learners outputs and the final prediction. This work fuses both aforementioned frameworks. We introduce an _aggregated_\(f\)-_average_ (AFA) shallow neural network which models and combines different types of averages to perform an optimal aggregation of the weak learners predictions. We emphasise its interpretable architecture and simple training strategy, and illustrate its good performance on the problem of few-shot class incremental learning.

 ensemble learning, estimator aggregation, weakly supervised learning, few-shot learning, incremental learning.

## I Introduction

### _Ensemble learning_

Ensemble learning (or ensembling) is a set of methods which leverage an ensemble of models (also called weak learners), instead of relying on a single learner to perform a given machine learning task (e.g., classification). While ensembling is obviously more demanding in terms of computing resources, it can achieve better accuracy and generalisation, improve overall stability, and reduce prediction variance and bias. Two main phases are identified in the process of building an ensemble model, namely the training of weak learners and the fusion of outputs [1]. The former focuses on producing an ensemble of diverse models, which is a crucial step in ensemble learning [2, 3, 4, 5]. For example, bootstrap aggregating (or bagging) [6] trains each model on a different subset of the training data to produce diverse weak learners. Output fusion gathers outputs from each weak learner of the ensemble and combines them to produce the final prediction [7, 8, 9].

One could distinguish two categories of methods for output fusion in ensemble learning [1]. The most basic one is to average weak learners outputs or, in the case of classification, to use a majority voting scheme [10]. Different types of averages could be used (e.g., arithmetic, geometric, harmonic, etc) and weights could be included to further refine results. Those weights can be set using various kinds of criteria, for example based on weak learners isolated performance [11]. The second category of methods uses meta-learners. They consist in plugging an additional model, responsible for taking advantage of the weak learners. Mixture of experts is a popular variant of meta-learners, where a gating network selects the weak learner that is most suited to produce the correct prediction given a certain input [12, 13]. A more straightforward output fusion based on meta-learners is the stacking of an additional learning model. Taking weak learners output as input, it learns the best combination to assemble the unified prediction [14, 15].

### _Contribution_

Our contribution, in this work, is to introduce _aggregated_\(f\)-_average_ (AFA) neural networks (NNs), based on a novel architecture for the output fusion phase of ensemble learning. It consists of a shallow neural network modelling different types of averages (arithmetic, geometric, harmonic, etc.) and is able, through supervised learning, to combine and/or select them optimally. Thanks to a specific architecture including original nonlinear activations and constrained weights, it is easily interpretable. To illustrate the performance of AFA neural networks upon the state-of-the-art, we describe their application and implementation in the currently popular setting of few-shot class incremental learning (FSCIL).

The paper is organised as follows. First, in section II, we present the architecture of our AFA model along with its training process. We then introduce, in section III, the FSCIL problem and describe our ensembling approach in this context. Experiments on several datasets highlight the benefits of our model, when compared with other ensemble output fusion methods.

## II Methodology

### _Ensembling through averaging_

Let \(K\) machine learning models trained for a common task (e.g., classification), produce \(K\) outputs \((x_{k})_{1\leq k\leq K}\), assumed to be vectors in \(\mathbb{R}^{N}\). In ensemble learning, those \(K\) outputs are combined during an output fusion phase in order to produce a single, expectably better, prediction for the task at hand. A naive method is to average the outputs. We summarize in Table I common expressions for weighted averages, with \((\omega_{k})_{1\leq k\leq K}\) nonnegative reals such that \(\sum_{k=1}^{K}\omega_{k}=1\).
###### Abstract

We consider the \(f\)-average of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted averages of weighted weighted averages of weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted averages of weighted weighted weighted averages of weighted weighted weighted averages of
columnwise the inputs \((x_{k})_{1\leq k\leq K}\). The resulting joint aggregate estimate of the \(J\) outputs \((\widetilde{x}_{k})_{1\leq j\leq J}\) is defined as

\[\widetilde{x}=\sum_{j=1}^{J}A_{j}\tilde{x}_{j}=A\begin{bmatrix}\tilde{x}_{1}\\ \vdots\\ \tilde{x}_{J}\end{bmatrix}, \tag{10}\]

with, for every \(j\in\{1,\ldots,J\}\), \(A_{j}\in[0,+\infty)^{N\times N}\), and \(A\in[0,+\infty)^{N\times NJ}\) is the rowwise stacking of \((A_{j})_{1\leq j\leq J}\) matrices. Operations (8)-(10) are equivalent to plug \(\mathbf{x}\) as the input of a neural network with \(J\) sub-networks of the form presented in Figure 1, operating in parallel, followed by a linear layer involving the weight matrix \(A\). We further add a final activation function, \(g\colon\mathbb{R}^{N}\to\mathbb{R}^{N}\), to control the domain of the output. For instance, a softmax activation can be used to get nonnegative outputs summing to one, in a classification context. The resulting network, called _aggregated \(f\)-average_, is displayed in Figure 2. It has a limited number \(JN^{2}(K+1)\) of linear parameters, namely the entries of matrices \(W_{1},\ldots,W_{J}\), and \(A\). The training of these parameters can follow a classical supervised learning approach. Given a sample and its ground truth, the task model loss is computed (e.g., a cross-entropy loss for classification), before updating the weights from all layers using a backpropagation algorithm (e.g., Adam optimizer [17]). Constraints on weight matrices \((W_{j})_{1\leq j\leq J}\) can simply be imposed through a projection step of each row of these matrices on the convex unit simplex set [18], performed after each backpropagation update. Let us remark that this model maintains the interpretability properties of the \(J\)\(f\)-average sub-models it contains. Furthermore, weights from matrix \(A\) can be viewed as the contribution level of each type of average model.

## III Aggregated \(f\)-average for few shot incremental learning

### _FSCIL framework and datasets_

We now illustrate the potential of the proposed AFA framework by applying it to the problem of few-shot class incremental learning. FSCIL, recently introduced in [19], focuses on designing machine learning methods that can deal with both incremental [20, 21] and few-shot situations [22, 23, 24, 25]. Specifically, FSCIL aims at including an increasing number of categories in a classification problem with the extra constraint that only a small number of training samples are available for upcoming classes. This is motivated by the frequent practical situation in computer vision when a model, built to classify certain categories seen during training phase, must be adjusted to classify images belonging to new classes with only (very) few annotations. The main difficulty revolves around the ability to learn new classes while preventing _catastrophic forgetting_ of classes previously learned, yielding poor performance of standard incremental learning methods.

The FSCIL setting considered here consists of \(K\) successive sessions that incrementally provide new categories of images to be classified. First session (\(k=1\)), also called base session, is a standard classification problem with a large number \(n_{\text{train}}\) of training samples for each of the \(n_{\text{class\_base}}\) base classes. In subsequent sessions (\(k\in\{2,\ldots,K\}\)), only a small additional number \(n_{\text{shots}}\) of training samples is provided for a limited number \(n_{\text{way}}\) of novel classes. At each session \(k\), the number of classes to predict is

\[N_{k}=n_{\text{class\_base}}+(k-1)n_{\text{way}}. \tag{11}\]

Our experiments focus on four typical FSCIL datasets, which characteristics are summarized in Table III.

Fig. 2: Structure of the proposed aggregated \(f\)-average neural network. It aggregates \(J\)\(f\)-averages for ensembling, with \(A\in[0,+\infty)^{N\times NJ}\). The activation function \(g:\mathbb{R}^{N}\to\mathbb{R}^{N}\) is selected according to the task (e.g, softmax for classification, linear for regression).
### _Application of aggregated \(f\)-average_

We address FSCIL as the ensembling of successive few-shot learning problems [30]. For each session, we train a few-shot learning classifier that serves as a weak classifier (i.e., weak learner) specialised on its own session in our ensemble model. During base session, the weak classifier is a standard classifier with a ResNet-18 architecture [31] trained thanks to the larger training set. Then, for the remaining sessions (\(k\geq 2\)), its backbone (i.e., all layers except the last fully connected layer) is frozen and used as feature extractor. For these sessions, that last fully connected layer (specialised on base session) is replaced by a few-shot learning method. We use a state-of-the-art nearest-neighbour classifier that determines the estimated class of a test image by retrieving the closest mean centroid in the feature space [32]. It is trained to classify the \(n_{\text{way}}\) new classes provided by the current session.

As described in Figure 5, for a given \(K\), our aggregated \(f\)-average model takes as input the outputs from sessions \(k\in\{1,\ldots,K\}\), of size \(N=N_{K}\) (i.e., the total number of classes, defined in (11)), to form a vector \(\boldsymbol{x}\in\mathbb{R}^{KN_{K}}\). As the weak classifiers from earlier sessions are trained to predict a fewer number of classes, a zero-padding is performed for its prediction vector in order to reach output size \(N_{K}\). Our AFA model is trained for \(100\) epochs, minimising a cross-entropy loss with the Adam algorithm [17], using an initial learning rate of \(10^{-1}\) decreased by a factor \(10\) at epochs \(40\) and \(70\). The training uses a _prototype rehearsal_ strategy [33] which consists of collecting predictions from all previous weak classifiers on training sets from all previous sessions to form the training set of the ensemble learning model. It is then fed to the model as mini-batches of size 64. We set \(J=4\) and \((f_{j})_{1\leq j\leq J}\) chosen so as to aggregate four different types of means, namely arithmetic, geometric, harmonic, and quadratic (power-2), using the activation functions presented in Table II with \(\epsilon=10^{-4}\). Weights of matrices \((W_{j})_{1\leq j\leq J}\) and matrix \(A\) are initialised with entries sampled randomly following a uniform distribution between \(0\) and \(\frac{1}{KN_{K}}\) before being projected onto the simplex to comply with the sum-to-one constraint. Our model and all experiments were implemented using Tensorflow [34].

### _Results_

We compare our proposed model performance, for increasing values of \(K\), against classic ensembling methods for classification performing output fusion. We try out different types of averages (namely, arithmetic, geometric, harmonic), as well as a majority vote scheme. Performance were also compared against three different types of ensembling neural network: 1) a shallow neural network with similar number of parameters and layers as our AFA model, 2) a deeper neural network with five fully connected layers, and 3) a neural network specifically designed for ensembling including

Fig. 3: Comparison of ensemble learning output fusion methods, on FSCIL datasets, in terms of averaged \(F_{1}\) score over all classes, for various sessions \(k\). The closer the \(F_{1}\) score is to 1, the better. As sessions are incrementally added, the classification task becomes increasingly more difficult because of the additional categories to be handled.
a weighted average layer followed by a fully connected layer for the output [35]. All neural network models, including the AFA model, were trained with the same process with only slight adjustments on learning rate parameters to adapt to each model architecture. The results, in terms of accuracy and \(F_{1}\) scores (the larger, the better), are summarized in Table IV and Figure 3. Note that the mean accuracy over all classes might hide performance discrepancies between base and new classes [36], as can be seen in Table IV. Typically, a model trained on base session only (called _No Fine-tuning_ in Table IV), reaches a decent mean accuracy thanks to its performance on base session classes (\(k=1\)), while it yields very poor performance on new classes (\(k>1\)). In contrast, our approach AFA performs an effective incremental learning, with good accuracy on both base and new classes.

Classic averaging shows inconsistent results (see Figure 3), as the optimal type of average depends on the session number and on the dataset. The majority vote scheme seems to produce worse results in early sessions (low \(K\)), when few voters are available, and a gain in performance with an increasing number of voters. By design, the proposed ensemble method models those different types of average, and learns their optimal combination, thus producing significantly better performance in every session.
inconsistent results; its performance catastrophically drop in later sessions (higher \(K\)) showing training issues. Indeed, these later sessions drastically increase its architecture size due to the higher number \(K\) of models to ensemble and the higher number of classes \(N_{K}\) to predict, while not providing a significantly larger training set because of the few-shot constraint of our task at hand.

Figure 4 illustrates these behaviours. It presents training losses of the different stacked neural network models, including the proposed AFA, on a specific example. The plots demonstrate our model' stability and fast convergence to a lower loss value than other methods which exhibit a plateau at a higher value. The deep NN loss behavior shows the inability to train a large number of parameters on such limited amount of data, with a validation loss diverging.

## IV Conclusion

In this article, we presented a novel output fusion method for ensemble learning. Inspired by basic methods, namely averaging and model stacking, our method relies on an original neural network architecture that models multiple types of averages. It is trained to learn an optimal and interpretable selection and/or combination of the multiple weak learners inputs. We illustrated its operability and efficiency on the problem of few-shot class incremental learning where it significantly outperforms standard ensemble learning output fusion methods.
