Can we avoid Double Descent in Deep Neural Networks?
Finding the optimal size of deep learning models is very actual and of broad
impact, especially in energy-saving schemes. Very recently, an unexpected
phenomenon, the ``double descent'', has caught the attention of the deep
learning community. As the model's size grows, the performance gets first
worse, and then goes back to improving. It raises serious questions about the
optimal model's size to maintain high generalization: the model needs to be
sufficiently over-parametrized, but adding too many parameters wastes training
resources. Is it possible to find, in an efficient way, the best trade-off? Our
work shows that the double descent phenomenon is potentially avoidable with
proper conditioning of the learning problem, but a final answer is yet to be
found. We empirically observe that there is hope to dodge the double descent in
complex scenarios with proper regularization, as a simple $\ell_2$
regularization is already positively contributing to such a perspective.
Victor Quétu, Enzo Tartaglione
# Can We Avoid Double Descent in Deep Neural Networks?

###### Abstract

Finding the optimal size of deep learning models is very actual and of broad impact, especially in energy-saving schemes. Very recently, an unexpected phenomenon, the "double descent", has caught the attention of the deep learning community. As the model's size grows, the performance gets first worse, and then goes back to improving. It raises serious questions about the optimal model's size to maintain high generalization: the model needs to be sufficiently over-parametrized, but adding too many parameters wastes training resources. Is it possible to find, in an efficient way, the best trade-off?

Our work shows that the double descent phenomenon is potentially avoidable with proper conditioning of the learning problem, but a final answer is yet to be found. We empirically observe that there is hope to dodge the double descent in complex scenarios with proper regularization, as a simple \(\ell_{2}\) regularization is already positively contributing to such a perspective.

Victor Queu  Enzo Tartaglione LTCI, Telecom Paris, Institut Polytechnique de Paris Double descent, pruning, deep learning, regularization

## 1 The Race towards Generalization

In the recent few years, the research community has witnessed a race towards achieving higher and higher performance (in terms of an error on unseen data), proposing very large architectures like, for example, transformers [1]. From big architectures come big responsibilities: learning strategies to avoid the over-fitting urge to be developed.

The most straightforward approach would be to provide more data: deep learning methods are notoriously data hunger. Since they typically optimize some objective function through gradient descent, having more data in the training set helps the optimization process in selecting the most appropriate set of features (to oversimplify, the most recurrent ones). This allows us to have high performance on unseen data. Such an approach has the big drawbacks of requiring enormous computational power for training and, most importantly, large annotated datasets. While undertaking the first drawback is an actual research topic [2, 3], the second is broadly explored with approaches like transfer learning [4] or self-supervised learning [5].

In more realistic cases, large datasets are typically unavailable, and in those cases approaches working with small data, in the context of _frugal AI_, need to be employed. This poses research questions on how to enlarge the available datasets or to transfer knowledge from similar tasks; however it poses also questions on how to optimally dimension the deep learning model to be trained. Contrarily to what is expected from the bias-variance trade-off, the phenomenon of _double descent_ can be observed in a very over-parameterized network: given some optimal set of parameters for the model \(\mathbf{w}^{opt}\) with loss value \(\mathcal{L}^{opt}\), adding more parameters will worse the performance until a local maximum \(\mathcal{L}^{*}\) beyond which, adding even more parameters, the trend goes back decreasing. This phenomenon, named _double descent_[6] is displayed in Fig. 1, and is consistently reported in the literature [7, 8]. Double descent poses the serious problem of finding the best set of parameters, in order not to fall into an over-parametrized

Figure 1: The double descent phenomenon (dashed line): is it possible to constrain the learning problem to a minimum such that the loss, in over-parametrized regimes, remains close to \(\mathcal{L}^{opt}\) (continuous line)?
(or under-parametrized) regime. The possible approaches to tackle this are two: finding \(\mathbf{w}^{opt}\), which is a problem requiring a lot of computation, or extremely over-sizing the model. Unfortunately, both the roads are not compatible with a frugal setup: is there a solution to this problem?

In this work, we show that the double descent phenomenon is potentially avoidable. Having a sufficiently large regularization on the model's parameters drives the deep models in a configuration where the set of parameters in excess \(\mathbf{w}^{exc}\) are essentially producing no perturbation on the output of the model. Nevertheless, as opposed to Nakkiran et al. [9], who showed in regression tasks that such a regularization could help in dodging double descent, we observe that, in classification tasks, this regularization is insufficient in complex scenarios: an ingredient is still missing, although we are on the right path.

## 2 Double descent and its implications

**Double descent in machine learning models.** The double descent phenomenon has been highlighted in various machine learning models, like decision trees, random features [10], linear regression [11] and deep neural networks [12]. Based on the calculation of the precise limit of the excess risk under the high dimensional framework where the training sample size, the dimension of data, and the dimension of random features tend to infinity proportionally, Meng et al. [10] demonstrate that the risk curves of double random feature models can exhibit double and even multiple descents. The double descent risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. Muthukumar et al. [11] provide a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. By defining the effective model complexity, Nakkiran et al. [13] showed that the double descent phenomenon is not limited to varying the model size, but it is also observed as a function of training time or epochs and also identified certain regimes were increasing the number of train samples hurts test performance.

**Double descent in regression tasks.** It has been recently shown that, for certain linear regression models with isotropic data distribution, optimally-tuned \(\ell_{2}\) regularization can achieve monotonic test performance as either the sample size or the model size is grown. Nakkiran et al. [9] demonstrated it analytically and established that optimally-tuned \(\ell_{2}\) regularization can mitigate double descent for general models, including neural networks like Convolutional Neural Networks. Endorsing such a result, Yilmaz et al. [12] indicated that regularization-wise double descent can be explained as a superposition of bias-variance trade-offs on different features of the data (for a linear model) or parts of the neural network and that double descent can be eliminated by scaling the regularization strengths accordingly.

**Double descent for classification tasks.** Although many signs of progress in regression models have been done, in classification tasks, the problem of avoiding, or formally characterizing, the double descent phenomenon, is much harder to tackle. The test error of standard deep networks, like the ResNet architecture, trained on standard image classification datasets, consistently follows a double descent curve both when there is label noise (CIFAR-10) and without any label noise (CIFAR-100) [12]. Double descent of pruned models concerning the number of original model parameters has been studied by Chang et al, which reveals a double descent behavior also in model pruning [14]. Model-wise, the double descent phenomenon has been studied a lot under the spectrum of over-parametrization: a recent work also confirmed that sparsification via network pruning can cause double descent in presence of noisy labels [15]. He et al. [15] proposed a novel learning distance interpretation, where they observe a correlation between the model's configurations before and after training with the sparse double descent curve, emphasizing the flatness of the minimum reached after optimization. Our work differs from this study by enlightening some cases in which the double descent phenomenon is not evident. We show, in particular, that by imposing some constraints on the learning process, we can avoid the double descent. The experimental setup follows the same as He et al.'s.

## 3 DODGING THE DOUBLE descent

**A regularization-based approach.** In the previous section we have presented all the main contributions achieved in the literature around the double descent phenomenon. It is a known result that, given an over-parametrized model, without special constraints on the training, this phenomenon can be consistently observed. However, it is also known that, for an optimally parametrized model, double descent should not occur. Let us say, the target, optimal output for the model is \(\mathbf{y}_{opt}\). For instance, there is some subset \(\mathbf{w}_{exc}\in\mathbf{w}\) of parameters belonging to the model which are in excess, namely the ones contributing to the double descent phenomenon. Since these are not essential in the learning/inference steps, they can be considered as _noisy parameters_, which deteriorate the
performance of the whole model and make the whole learning problem more difficult. They generate a perturbation in the output of the model which we can quantify as

\[\mathbf{y}^{exc}=\sum_{w_{i}\in\mathbf{w}^{exc}}\text{ forward}\left[\phi(\mathbf{x}_{i}\cdot w_{i})\right], \tag{1}\]

where \(\mathbf{x}_{i}\) is the input(s) processed by \(w_{i}\), \(\phi(\cdot)\) is some non-linear activation for which \(\phi(z)\approx z\) when \(z\to 0\), and \(\text{forward}(\cdot)\) simply forward-propagates the signal to the output of the neural network. As the output of the model is given by \(\mathbf{y}=\mathbf{y}^{opt}+\mathbf{y}^{exc}\), in the optimal case we would require that \(\mathbf{y}^{exc}=\mathbf{0}\). To satisfy such a condition, we have two possible scenarios:

* \(\exists w_{i}\neq 0\in\mathbf{w}^{exc}\). In this case, the optimizer finds a minimum loss such that the algebraic sum of the noisy contribution is zero. When a subset of these is removed, it is possible that \(\mathbf{y}^{exc}\neq\mathbf{0}\), which results in a performance loss and, for instance, the double descent phenomenon is observed.
* \(w_{i}=0,\forall w_{i}\in\mathbf{w}^{exc}\). In this case, there is no contribution of the noisy terms and we are in the optimal scenario, where the parameters are de-facto removed from the model.

Let us focus on the second case. For numerical reasons, this scenario is unrealistic during continuous optimization; hence we can achieve a similar outcome by satisfying two conditions:

1. \(w_{i}x_{i}\approx 0,\forall w_{i}\in\mathbf{w}^{exc}\);
2. \(\|\text{forward}\left[\phi(\mathbf{x}_{i}\cdot w_{i})\right]\|_{1}\leq\|\phi(\mathbf{x }_{i}\cdot w_{i})\|_{1}\), with \(\|\phi(\mathbf{x}_{i}\cdot w_{i})\|_{1}\approx 0\).

We can achieve these conditions with a sufficiently large regularization on the parameters \(\mathbf{w}\). The first condition is achievable employing any common weight penalty, as we assume that, for local minima of the loss where \(\frac{\partial L}{\partial w_{i}}=0\), we have some weight penalty \(C\) pushing its magnitude towards zero. The second condition, on the contrary, requires more careful consideration. Indeed, for the propriety of the function's composability, we need to ensure that the activation function in every layer does not amplify the signal (true in the most common scenarios) and that all the parameters have the lowest magnitude possible. For this reason, ideally, the regulation to employ should be \(\ell_{\infty}\); on the other hand, however, we are also required to enforce some sparsity. Towards this end, recent works in the field suggest \(\ell_{2}\) regulation is a fair compromise[16]. Hence, we need a sufficiently large \(\ell_{2}\) regularization.

**How can we observe we have avoided double descent?** We present an algorithm to sketch the (eventual) double descent phenomenon in Alg. 1. After training for the first time the model on the learning task \(\Xi\), eventually with \(\ell_{2}\) regularization weighted by \(\lambda\) (line 2), a magnitude pruning stage is set up (line 4). Neural network pruning, whose goal is to reduce a large network to a smaller one without altering accuracy, removes irrelevant weights, filters, or other structures from neural networks. An unstructured pruning method called magnitude-based pruning, popularized by [16], adopted a process in which weights, below some specific threshold \(T\), are pruned (line 4). We highlight that more complex pruning approaches exist, but magnitude-based pruning shows their competitiveness despite very low complexity [17]. Towards this end, the hyper-parameter \(T^{iter}\) sets the relative pruning percentage, or in other words, how many parameters will be removed at every pruning stage. Once pruned, the accuracy of the model typically decreases. To recover perfor

Figure 2: Test accuracy in the function of sparsity with different amounts of symmetric noise \(\varepsilon\). Dashed lines correspond to vanilla and solids lines to \(\ell_{2}\) regularization.

**Left:** LeNet-300-100 on MNIST. **Middle:** ResNet-18 on CIFAR-10. **Right:** ResNet-18 on CIFAR-100.
mance, the lottery ticket rewinding technique, proposed by Frankle & Carbin [18], is used. It consists of retraining the subset of parameters that are surviving the pruning stage to their initialization value (line 5) and then training the model (line 6). This approach allows us to state whether a lowly-parametrized model from initialization can, in the best case, learn a target task. We end our sketching once we reach a sparsity higher or equal than \(T^{end}\) (line 3).

**Experimental setup.** For the experimental setup, we follow the same approach as He et al. [15]. The first model we train is a LeNet-300-100 on MNIST, for 200 epochs, optimized with SGD with a fixed learning rate of 0.1. The second model is a ResNet-18, trained on CIFAR-10 & CIFAR-100, for 160 epochs, optimized with SGD, having momentum 0.9 and a learning rate of 0.1 decayed by a factor 0.1 at milestones 80 and 120. For each dataset, a percentage \(\varepsilon\) of symmetric, noisy labels are introduced: the labels of a given proportion of training samples are flipped to one of the other class labels, selected with equal probability [19]. In our experiments, we test with \(\varepsilon\in\{10\%,20\%,50\%\}\). When the regularization \(\ell_{2}\) is employed, we set \(\lambda\) to \(1\times 10^{-4}\). In all experiments, we use batch sizes of 128 samples, \(T^{iter}\) to 20% and \(T^{end}\) is 99.9%.

**Results.** Fig. 2 displays our results. As in He et al. [15] work, looking at LeNet-300-100 with \(\varepsilon=10\%\), the double descent consists of 4 phases. First, at low sparsities, the network is over-parameterized, thus pruned network can still reach similar accuracy to the dense model. The second phase is a phase near the interpolation threshold, where training accuracy is going to drop, and test accuracy is about to first decrease and then increase as sparsity grows. The third phase is located at high sparsities, where test accuracy is rising. The final phase happens when both training and test accuracy drop significantly. However, while we can observe the double descent in the test accuracy without \(\ell_{2}\) regularization, the phenomenon fades when the regularization is added. Indeed, the existence of the second phase is questioned: the test accuracy which is expected to decrease in this phase reaches a plateau before rising when regularization is added. In this simple setup, the double descent is here dodged. However, Fig. 2 also portrays the result of ResNet-18 on CIFAR-10 and CIFAR-100 experiences, with different percentages of noisy labels. Whether the regularization is used or not, on average, and for every value of \(\varepsilon\), the double descent phenomenon occurs and is present in both cases. Those experiments, which can be considered more complex than the previous one, highlight some limits of the use of the standard regularization to avoid the double descent phenomenon, and suggest that a specific regularizer should be designed.

**Ablation on \(\boldsymbol{\lambda}\).** In the previous experiments in Fig. 2, we have proposed solutions with a \(\ell_{2}\)-regularization hyper-parameter which provides a good trade-off between performance in terms of validation error and avoidance of the double descent. However, ablating the regularization parameter is a matter of interest to ensure that with larger values, the double descent can not be avoided either. Hence, we propose in Fig. 3 an ablation study on \(\lambda\) for CIFAR-10 with \(\varepsilon=10\%\). We observe that, even for extremely high \(\lambda\), the double descent is not dodged, and the overall performance of the model drops: the imposed regularization becomes too high, and the training set is not entirely learned anymore. This indicates us that, while for regression tasks \(\ell_{2}\) regularization is the key to dodge the double descent, in classification tasks the learning scenario can be very different, and some extra ingredient is still missing.

## 4 Is Double Descent Avoidable?

The problem of finding the best-fitting set of parameters for deep neural networks, which has evident implications for both theoretical and applicative cases, is currently a subject of great interest in the research community. In particular, the phenomenon of double descent prioritizes the research around finding the optimal size for deep neural networks: if a model is not extremely over-parametrized it may fall into a sub-optimal local minimum, harming the generalization performance.

In this paper, we have moved some first steps, in a traditional classification setup, towards avoiding the double descent. If we successfully achieve a local minimum where, regardless of its over-parametrization, the performance of the model is consistently high at varying the cardinality of its parameters, there would not be a strict need in finding the optimal subset of parameters for the trained model. Standard regularization

Figure 3: Train and test loss at varying \(\lambda\) for CIFAR-10, with \(\varepsilon=10\%\).
approaches like \(\ell_{2}\), which have proven their effectiveness in regression tasks, evidenced some limits in more complex scenarios while dodging effectively the double descent in simpler setups. This result gives us hope: a custom regularization towards avoidance of double descent can be designed, and will be the subject of future research.
