Recent empirical results on over-parameterized deep networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks. Researchers are actively working on bridging this discrepancy by proposing better complexity measures. Instead, we directly measure prediction bias and variance for four classification and regression tasks on modern deep networks. We find that both bias and variance can decrease as the number of parameters grows. Qualitatively, the phenomenon persists over a number of gradient-based optimizers. To better understand the role of optimization, we decompose the total variance into variance due to training set sampling and variance due to initialization. Variance due to initialization is significant in the under-parameterized regime. In the over-parameterized regime, total variance is much lower and dominated by variance due to sampling. We provide theoretical analysis in a simplified setting that is consistent with our empirical findings. Despite a few notable exceptions, such as boosting (Schapire, 1990; BID13 BID5 , the dogma in machine learning has been: "the price to pay for achieving low bias is high variance" BID14 . This balance between underfitting (high bias) and overfitting (high variance) is commonly known as the biasvariance tradeoff FIG0 . Statistical learning theory (Vapnik, 1998) identifying a notion of model capacity, understood as the main parameter controlling this tradeoff. Complex (high capacity) models achieve low prediction bias at the expense of high variance. In their landmark work that highlighted this dilemma, BID14 suggest that bias decreases and variance increases with network size.However, there is a growing amount of empirical evidence that wider networks generalize better than their smaller counterparts (Neyshabur et al., 2015; Zagoruyko & Komodakis, 2016; Novak et al., 2018; BID8 BID2 Spigler et al., 2018; Liang et al., 2017; BID6 . In those cases the U-shaped test error curve is not observed. Researchers have identified classic measures of complexity as a culprit. The idea is that, once we have identified the right complexity measure, we will again be able to observe this fundamental tradeoff.We bypass this important, ongoing discussion by measuring prediction bias and variance directly-something that has not been done in related literature since BID14 , to the best of our knowledge. These measurements allow us to reason directly about the existence of a tradeoff with respect to network width. We find evidence that both bias and variance can decrease at the same time as network width increases in common classification and regression settings with deep networks.We observe this qualitative behavior with a number of gradient-based optimizers. In order to get a closer look at the role of optimization and sampling, we propose a simple decomposition of total prediction variance. We use the law of total variance to get a term that corresponds to variance due to training set sampling and another that corresponds to variance due to initialization. Variance due to initialization is significant in the under-parameterized regime and monotonically decreases with width in the over-parameterized regime. There, total variance is much lower and dominated by variance due to sampling (Fig. 2) .We provide theoretical analysis, consistent with our empirical findings, in simplified analysis settings: i) prediction variance does not grow arbitrarily in linear models; ii ) variance due to initialization diminishes in deep networks under strong assumptions. On the left is an illustration of the common intuition for the bias-variance tradeoff BID12 . We find that variance decreases along with bias when increasing network width (right). These results seem to contradict the traditional intuition. Our empirical results demonstrate that in the practical setting, variance due to initialization decreases with network width while variance due to sampling levels off. Here, we take inspiration from linear models (Hastie et al., 2009, Section 7. 3) to provide arguments for the behavior of variance in increasingly wide neural networks. First, we provide evidence against BID14 's claim that "the price to pay for achieving low bias is high variance," finding that both bias and variance decrease with width. Second, we find variance due to sampling (analog of regular variance in simple settings) does not appear to be dependent on width, once sufficiently over-parameterized. Third, variance due to initialization decreases with width. We see further theoretical treatment of variance as a fruitful direction for better understanding complexity and generalization abilities of neural networks. We made strong assumptions, but there is some support for them in the literature. The existence of a subspace M ⊥ in which no learning occurs was also conjectured by BID0 and shown to hold in linear neural networks under a simplifying assumption that decouples the dynamics of the weights in different layers. Li et al. (2018) empirically showed the existence of a critical number d(N ) = d of relevant parameters for a given learning task, independent of the size of the model. Sagun et al. (2017) showed that the spectrum of the Hessian for over-parameterized networks splits into (i) a bulk centered near zero and (ii) a small number of large eigenvalues; and Gur-Ari et al. FORMULA2 recently gave evidence that the small subspace spanned by the Hessian's top eigenvectors is preserved over long periods of training. These results suggest that learning occurs mainly in a small number of directions.
We provide evidence against classical claims about the bias-variance tradeoff and propose a novel decomposition for variance.