On the Variance of Neural Network Training with respect to Test Sets and
  Distributions
Typical neural network trainings have substantial variance in test-set
performance between repeated runs, impeding hyperparameter comparison and
training reproducibility. In this work we present the following results towards
understanding this variation. (1) Despite having significant variance on their
test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have
little variance in performance on the underlying test-distributions from which
their test-sets are sampled. (2) We show that these trainings make
approximately independent errors on their test-sets. That is, the event that a
trained network makes an error on one particular example does not affect its
chances of making errors on other examples, relative to their average rates
over repeated runs of training with the same hyperparameters. (3) We prove that
the variance of neural network trainings on their test-sets is a downstream
consequence of the class-calibration property discovered by Jiang et al.
(2021). Our analysis yields a simple formula which accurately predicts variance
for the binary classification case. (4) We conduct preliminary studies of data
augmentation, learning rate, finetuning instability and distribution-shift
through the lens of variance between runs.
Keller Jordan
# Calibrated Chaos: Variance Between Runs of Neural Network Training is Harmless and Inevitable

###### Abstract

Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. We present the following results towards understanding this variation. (1) Despite having significant variance on their test-_sets_, we demonstrate that standard CIFAR-10 and ImageNet trainings have very little variance in their performance on the test-_distributions_ from which their test-sets are sampled, suggesting that variance is less of a practical issue than previously thought. (2) We present a simplifying statistical assumption which closely approximates the structure of the test-set accuracy distribution. (3) We argue that test-set variance is inevitable in the following two senses. First, we show that variance is largely caused by high sensitivity of the training process to initial conditions, rather than by specific sources of randomness like the data order and augmentations. Second, we prove that variance is unavoidable given the observation that ensembles of trained networks are well-calibrated. (4) We conduct preliminary studies of distribution-shift, fine-tuning, data augmentation and learning rate through the lens of variance between runs.

## 1 Introduction

Modern neural networks (Krizhevsky et al., 2012; He et al., 2016; Vaswani et al., 2017) are trained using stochastic gradient-based algorithms (Rumelhart et al., 1986; Kingma and Ba, 2014), involving randomized weight initialization, data/batch ordering, and data augmentations. Because of this stochasticity, each independent run of training produces a different network with better or worse performance than average.

The difference between such independent runs is often substantial. Picard (2021) finds that for a standard CIFAR-10 (Krizhevsky et al., 2009) training configuration, there exist random seeds which differ by 1.3% in terms of test-set accuracy. In comparison, the gap between the top two methods competing for state-of-the-art on CIFAR-10 has been less than 1% throughout the majority of the benchmark's lifetime1. Prior works therefore view this variance as an obstacle which impedes comparisons between training configurations (Bouthillier et al., 2021; Picard, 2021) and reproducibility (Bhojanapalli et al., 2021; Zhuang et al., 2022). To mitigate stochasticity, Zhuang et al. (2022) study deterministic tooling, Bhojanapalli et al. (2021) develop regularization methods, and several recent works (Wightman et al., 2021; Liu et al., 2022) report the average of validation metrics across multiple runs when comparing training configurations.

Footnote 1: [https://paperswithcode.com/sota/image-classification-on-cifar-10](https://paperswithcode.com/sota/image-classification-on-cifar-10)

In this paper, we begin with the following series of questions regarding the variance in test-set accuracy between independent, identically-configured runs of neural network training:

1. What causes this variance? Can we isolate which of source of randomness is most responsible?
2. If we train many times, and take the network which performed best on the test-_set_, should we also expect it to perform above average on the test-_distribution_?
3. Which hyperparameters affect variance? For example, do there exist training configurations with the same average accuracy, but different amounts of variance?
Our contributions towards resolving these questions come in two flavors. First, we utilize large-scale data (\(\approx 350,000\) trained networks) in order to obtain a series of empirical results, for the case of standard trainings on CIFAR-10 and ImageNet (Deng et al., 2009). Second, we present theoretical results regarding distribution-wise variance, and establish a connection between variance and the calibration of neural network ensembles (Lakshminarayanan et al., 2017). Summarized, our contributions are as follows:

1. We demonstrate that no single source of randomness contributes independently to the overall variance between runs (Section 3.1). Instead, we argue that variance is caused by the extreme sensitivity of the training process to initial conditions (Section 3.2). In particular we show, confirming the results of Summers and Dinneen (2021), that varying just a single weight at initialization produces only 1% less churn than all three sources of randomness combined.
2. We argue that the observable variance between runs of training in terms of test-_set_ accuracy is a form of calibrated finite-sample noise, which does not imply variance with respect to the test-_distribution_. We begin by demonstrating that when training to convergence, disjoint splits of test data become decorrelated with respect to trained model performance (Section 4.1). Next we present a simplifying statistical assumption (Hypothesis 1) which closely approximates the accuracy distribution (Section 4.2). We derive an estimator for the standard deviation of distribution-wise accuracy, which predicts a value of 0.033% for CIFAR-10 and 0.034% for ImageNet. This is 10-20\(\times\) less variance than what we observe on the test-set (Section 4.3). We prove that a quantity of test-set variance is inevitable, given the observation that ensembles of independently trained networks are well-calibrated (Lakshminarayanan et al., 2017). For binary classification problems we derive a lower bound on the variance which we demonstrate is a good approximation to the true value (Section 4.4).
3. We obtain a variety of results connecting variance to other observed phenomena in deep learning. First, we strengthen the results of prior works (Devlin et al., 2018; Dodge et al., 2020; Mosbach et al., 2020) by showing that BERT-Large finetuning exhibits 99\(\times\) more distribution-wise variance than BERT-Base (Section 5.1). Next, we demonstrate that distribution-shifted test-sets experience excess variance between runs of training, compared to an in-domain test-set (Section 5.2). We find that data augmentation significantly reduces variance between runs (Section 5.3). When increasing the learning rate, we show that excess distribution-wise variance appears at the same point at which accuracy begins to decline (Section 5.4). Finally, using statistical correlations between the predicted logits of many independently trained networks, we obtain a new kernel function which reveals structure more effectively than distance in the penultimate feature-space of much larger models (Section 5.5).

### Related work

A number of prior works report results pertaining to our first question about variance - namely, whether we can determine which source of training stochasticity is most responsible. Fort et al. (2019) observe that when using a below-optimal learning rate, randomized data ordering has a smaller impact than model initialization on the churn of predictions between runs. Bhojanapalli et al. (2021) similarly find that fixing the data ordering has no effect, while fixing the model initialization reduces churn. Bouthillier et al. (2021) report that data ordering has a larger impact than model initialization. In contrast, we demonstrate in Section 3 that most variation can be attributed to the high sensitivity of the training process to initial conditions. We recently became aware of Summers and Dinneen (2021), who reach the same conclusions; our results further confirm their findings via several new experiments.

Dodge et al. (2020) study variation between runs of BERT finetuning. The authors report achieving substantial performance increases relative to previously reported results, via the strategy of re-running finetuning many times and taking the best-performing result. Other recent works have picked up this strategy, _e.g._, Margatina et al. (2021) evaluate active learning strategies using the best result out of five runs, citing Dodge et al. (2020) as motivation. We demonstrate (Section 5.1) that for the case of BERT-Base, these performance gains only amount to overfitting the validation set, and cannot be expected to generalize to new samples of data from the test-distribution.
Several prior works study the effects of varying the choice of which examples are included in the training and test datasets. Neal et al. (2018) study the behavior of two-layer MLPs as the data used to train them is varied. Baldock et al. (2021) define a metric called the "consistency score", which is the probability that an example is predicted correctly over randomness induced by varying the training data, and relates it to other notions of example difficulty. Bouthillier et al. (2021) explore re-sampling both the train and test sets, and find that across bootstrap re-samples of the test-set, the variance in accuracy of a trained model conforms to the theoretically expected binomial approximation.

Ilyas et al. (2022) demonstrate that across varying sub-samplings of the training dataset, the trained network's test-example predictions can be approximately modeled as linear functions ("datamodels") of the vector of indicator-variables corresponding to the subset of examples that was used to train. In total, the authors use over 3,000,000 runs of training on CIFAR-10 in order to learn these datamodels. The authors show that datamodels can be used to generate a high-quality feature embedding of CIFAR-10. We demonstrate that a similar such embedding can even be recovered from simple between-run statistical correlations, using a fixed training set (Section 5.5). Overall, our work shares the large-scale empirical approach of Ilyas et al. (2022), and we also use the same software package, FFCV (Lecler et al., 2022), to perform high-throughput CIFAR-10 training. We use this approach to study the simpler topic of between-run variation given fixed training and test datasets.

Broadly, our work is part of a line of research which aims to understand the relationship between pairs of neural network weights produced by repeated independent runs of stochastic gradient descent. This topic is of both theoretical and practical interest, and has been studied from a variety of angles, including the similarity of internal representations (Li et al., 2015; Kornblith et al., 2019), degree of correlation between predictions (Fort et al., 2019; Jiang et al., 2021), similarity of decision boundaries (Sompalli et al., 2022), path-connectivity in weight-space (Draxler et al., 2018; Garipov et al., 2018), linear mode connectivity (Frankle et al., 2020), and linear mode connectivity modulo weight-space permutation symmetries (Tatro et al., 2020; Entezari et al., 2021; Ainsworth et al., 2022; Jordan et al., 2022).

## 2 Setup

We study supervised classification problems in which test examples are sampled from a distribution \(\mathcal{D}\) over \(\mathcal{X}\times\mathcal{Y}\). We make no assumptions on the training distribution. We view a given stochastic training configuration \(\mathcal{A}\), with fixed hyperparameters other than the random seed, as inducing a distribution over trained network weights \(\theta\sim\mathcal{A}\). We write \(f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}\) to denote the prediction function computed by the network which has weights \(\theta\). In particular, for our networks which end in a softmax layer, \(f_{\theta}(x)\) computes

Figure 1: **Measuring variance. Test-set accuracy distributions across four training durations on CIFAR-10. Each distribution is a histogram of test-set accuracy across 60,000 independent runs of training. The difference between the “luckiest” and most unlucky run (max and min accuracy) is 13.2%, 6.6%, 1.7%, and 1.4% for the 0, 4, 16, and 64-epoch training durations, respectively. The standard deviations are 1.87%, 0.56%, 0.19%, and 0.15%.**
the argmax over predicted probabilities. For each example \((x,y)\) we define \(C_{x,y}(\theta)=\mathbf{1}_{f_{\theta}(x)=y}\) to be equal to one when the network \(f_{\theta}\) correctly predicts the example. We view \(C_{x,y}\) as a Bernoulli random variable, whose mean \(\overline{C}_{x,y}=\mathbb{E}_{\theta\sim\mathcal{A}}[C_{x,y}(\theta)]\) is the probability that \((x,y)\) is correctly predicted by a newly trained network. For a test set \(S=((x_{1},y_{1}),\ldots,(x_{n},y_{n}))\) we write \(A_{S}(\theta)=\frac{1}{n}\sum_{i=1}^{n}C_{x_{i},y_{i}}(\theta)\) to denote the test-set accuracy of network \(f_{\theta}\). We assume that the test-set is sampled IID, so that, putting things together, we can write \(\mathbb{E}_{S\sim\mathcal{D}^{n}}[\text{Var}_{\theta\sim\mathcal{A}}(A_{S}( \theta))]\) to denote the expected (over re-sampling of the test-set) variance (over training stochasticity) of test-set accuracy. Finally, we define the distribution-wise accuracy of \(f_{\theta}\) as \(A(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}[C_{x,y}(\theta)]\), and the average error of training configuration \(\mathcal{A}\) as \(\text{err}(\mathcal{A})=\mathbb{E}_{\theta\sim\mathcal{A}}[1-A(\theta)]\).

For our core experiments (Section 4), we train ResNets on CIFAR-10. We additionally conduct experiments on ImageNet in Section 5.2. For CIFAR-10 we use the 9-layer ResNet architecture developed by Page (2019). Derivatives of this architecture have become standard for use in fast CIFAR-10 training, _e.g._, in the DAWNBench competition (Coleman et al., 2017). We train using SGD-momentum with weight decay, and use random flipping and translation data augmentations. We study a wide range of training durations, and always linearly ramp the learning rate down to zero by the end of training. The 64-epoch training configuration achieves a mean accuracy of 93.78%, which is increased to 94.42% when we additionally use Cutout (DeVries and Taylor, 2017). In Figure 1 we show histograms of test-set accuracy for this configuration across four training durations. The zero-epoch case corresponds to evaluating the network at initialization; this naturally has an average accuracy of 10% which is equivalent to random chance, but some random initializations reach as high as 14% and as low as 6% accuracy. As we train for longer, the average accuracy improves and the distribution becomes more concentrated. Further details of our training configurations are provided in Appendix A.

## 3 Isolating sources of randomness

### The three typical sources

Training neural networks typically involves three sources of stochasticity, namely, model initialization, data ordering, and data augmentations. In this section we investigate how each of these sources contributes to the final variance between runs that we observe at the end of training.

We develop a CIFAR-10 training framework2 that allows each source to be independently controlled by one of three different seeds. For example, when the data-augmentation seed is fixed and the data-order seed is varied, the set of augmented images seen by the network throughout training will remain the same, but be presented in a different order. When all three seeds are fixed, training is deterministic, so that repeated runs produce the same network. Standard training is equivalent to allowing all three seeds to vary.

Footnote 2: [https://github.com/KellerJordan/CIFAR10-isolated-rng](https://github.com/KellerJordan/CIFAR10-isolated-rng)

Figure 2: **One source of stochasticity suffices for long training. When training for only 1 epoch, varying all three sources of randomness induces a standard deviation of 1.33% in test-set accuracy between runs, while each source alone induces 25-40% less variance. But when training for 64 epochs, varying any one source induces as much variance as all three together. Each distribution corresponds to 4,000 runs of training.**
We fix two seeds, and vary just the third (_e.g._, varying only the data order while keeping the model initialization and data augmentations fixed). Our naive intuition is that each factor contributes some part to the overall variance, so that this should decrease variance relative to the baseline of varying all three seeds.

Our results show that for short trainings of 1-16 epochs, this intuition is correct (Figure 2). For example, when training for 4 epochs, if we fix the data order and augmentations, while varying only the model initialization, then variance in test-set accuracy is reduced by 26%, with the standard deviation going from \(0.45\pm 0.01\%\) to \(0.38\pm 0.01\%\).

However, for longer trainings of 32 epochs or more, varying only one of the three random factors produces approximately the same variance as the baseline of varying all three. For example, across 8,000 runs of training for 64 epochs, varying just the model initialization (with data ordering and augmentation fixed) produces a standard deviation of 0.158%, almost the same as the baseline, which has 0.160%. At \(n=8,000\) this is not a statistically significant difference; it is possible that the true values are the same, or that they differ by a small amount. We conclude that for this training regime, any single random factor suffices to generate the full quantity of variance, rather than each factor contributing to overall variance.

### Sensitivity to initial conditions

In the previous section, we showed that when training to convergence, varying just the model initialization (or just the data ordering, or augmentations) produces approximately the same quantity of variance between runs as a baseline fully random setup. In this section we find that even varying a single weight at initialization suffices. Our findings confirm the work of Summers and Dinneen (2021), who reach similar conclusions.

Consider multiplying a single random weight in the network by 1.001. We call this "poking" the network. This is a tiny change; recent work in quantization (_e.g._, Dettmers et al., 2022) implies that trained models can typically have _all_ their weights modified more than this without losing accuracy.

Nevertheless, in Figure 3 we demonstrate that poking the network early in training produces a large difference in the final result. Our experiment is to run two trainings with the same random seed, but with one network being "poked" at some point during training. We measure the disagreement rate between the two networks, _i.e._, the fraction of their test-set predictions that differ. For short trainings, poking induces much less disagreement than changing the random seed. But when training for 128 epochs, poking alone produces an average disagreement of 5.14%, barely less than the 5.19% produced by using two different random seeds. In our experiments, we also observe that varying just the first batch of data, or the numerical precision of the first step (_e.g._, fp16 vs. fp32) has a similar effect. We conclude that almost all variation between runs is not produced by specific sources of randomness like model initialization, data ordering, etc., but is instead intrinsic to the training process, which has extreme sensitivity to initial conditions.

Figure 3: **Training has high sensitivity to initial conditions.** (Left:) For short trainings, pairs of runs which differ only by one network having been “poked” (_i.e._, had a single weight changed slightly at initialization) disagree on 7.0-7.5% of predictions. Pairs of runs with fully different random seeds disagree more, on \(\sim\)8.5% of predictions. For long trainings, there is almost no difference. (Right:) The earlier a network is poked during the training process, the more its predictions will disagree with the network that trained unperturbed with the same random seed.
## 4 Delving into variance

### Do lucky random seeds generalize?

In Figure 1 we observed that our standard CIFAR-10 training configuration has significant variation between runs. Even when training for a long duration, we found pairs of random seeds which produce trained networks whose test-set accuracy differs by more than 1%. In this section, we argue that this variance is merely a form of finite-sample noise caused by the limited size of the test-set, and does not imply almost any genuine fluctuation in the quality of the trained network.

Suppose we view the random seed as a training hyperparameter. Then we have observed that it can be effectively "tuned" to obtain improved performance on the test-set - on average, our training configuration attains an accuracy of 94.42%, but we found random seeds which reach above 95%, which is more than 10% fewer errors. However, this improvement on the test-set alone is not enough to conclude that the random seed genuinely affects model quality. What remains to be seen is whether these performance improvements can generalize to unseen data, or if we are merely over-fitting the random seed to the observed test-set.

To find out, we perform the following experiment. First, we split the CIFAR-10 test-set into two halves of 5,000 examples each. We can view the first half as the hyperparameter-validation split and second as the held-out test split. Next, we execute many independent runs of training, with identical configurations other than the varying random seed. We measure the performance of each trained network on both splits of data. If lucky random seeds do generalize, then we should observe that runs of training which perform well on the first split also perform better than average on the second split.

To additionally determine the effect of training duration, we repeat this experiment for trainings of 0, 4, 16, and 64 epochs, using 60,000 independently trained networks for each duration. We view the results in Figure 4. For short trainings, the two splits are indeed highly correlated, such that runs which perform well on the first split also tend to do well on the second. But when training for longer, this correlation nearly disappears. When training for 64 epochs, for example, our highest-performing network on the first split does not even perform better than average on the second split. And on average, the top 1/4 of runs with respect to the first split only perform 0.02% better than average on the second split.

This result has the following practical implication. Suppose we want to obtain a good CIFAR-10 model. Noticing significant variation between runs (Figure 1), we might be tempted to re-run training many times, in order to obtain networks with better test-set performance. However, according to Figure 4, this would be useless, because improvements on the test-set due to re-training will have near-zero correlation with improvements on unseen data. These networks would be "better" only in the sense of attaining higher test-set accuracy, but not in the sense of being more accurate on unseen data from the same distribution.

Figure 4: **Disjoint splits of test data become decorrelated when training to convergence. We evaluate a large number of independently trained networks on two splits of the CIFAR-10 test-set. When under-training, there is substantial correlation, so that a “lucky” run which over-performs on the first split is also likely to achieve higher-than-average accuracy on the second. As we increase the training duration, this correlation shrinks.**
### Example-wise independence

In the previous section we showed that when training to convergence, disjoint splits of test data become nearly decorrelated, in the sense that networks which randomly perform well on one split do not perform better than average on another. In this section we argue that this property also extends to individual examples, which become decorrelated as training progresses.

We begin by randomly choosing a pair of test-set examples to check for dependence: say, examples 776 and 796. In Figure 6 we observe that this pair is indeed independent; _i.e._, using the notation introduced in Section 2, we observe the relation \(C_{x_{776},y_{776}}\perp\!\!\!\perp C_{x_{796},y_{796}}\) (up to statistical significance across 60,000 runs). Given this small positive result, we next hypothesize that _all_ test-set examples vary independently between runs:

**Hypothesis 1** (Examplewise independence).: \[C_{x_{i},y_{i}}\perp\!\!\!\perp C_{x_{j},y_{j}}\quad\forall\,i\neq j\] (1)

Using our large collection of trained networks, we have accurate estimates for each example's mean \(\overline{C}_{x_{i},y_{i}}\). This enables us to sample from the hypothesis distribution by simulating the average of 10,000 independent coin flips with probabilities \(\overline{C}_{x_{1},y_{1}}\) through \(\overline{C}_{x_{10,000},y_{10,000}}\). In Figure 5 we demonstrate that this simulated distribution becomes a close fit with reality as we train to convergence. For short trainings, the empirical distribution contains excess variance which is unexplained by Hypothesis 1, but for long trainings this excess disappears, so that the hypothesis becomes true in the aggregate sense of predicting the shape and standard deviation of the empirical distribution. In Figure 14 we also show that the hypothesis compares favorably to the binomial approximation. And in Section 5.5 we investigate example-wise deviations, finding only a small number of visually similar pairs which violate the hypothesis. In the next section we will explore how deviations from Hypothesis 1 can be used to estimate variance with respect to the test-distribution.

Figure 5: **Examples independence explains variance when training to convergence. (Left:) We compare the empirical test-set accuracy distributions with those generated by an equal number of samples from our hypothesized statistical model (Equation 1). The hypothesis is clearly wrong for short trainings, but becomes a close fit as training progresses. (Right:) Assuming Hypothesis 1, the standard deviation of test-set accuracy should be \(\sqrt{\frac{1}{n^{2}}\sum_{i=1}^{n}\overline{C}_{x_{i},y_{i}}(1-\overline{C}_ {x_{i},y_{i}})}\). This formula becomes a good approximation when training to convergence.**

Figure 6: **Independent example-wise variation. (Left) is image 776 of the CIFAR-10 test-set. Out of 60,000 64-epoch trainruns, 21,736 (36.2%) correctly predict this example. (Right) is image 796, which is correctly predicted by 36,392 (60.7%). If the two are statistically independent, then they should be simultaneously correct \(0.362\cdot 0.607=21.97\%\) of the time. In fact, 13,103 (21.84% \(\pm 0.33\%\)) of our trained networks do correctly predict both.**
### Estimating distribution-wise variance

In Section 4.1 we showed that accuracy is decorrelated between disjoint splits of test-data, and argued that this implies small genuine variation in model quality between runs of training. In this section we clarify our notion of model quality, and present a method of directly estimating variance on a test-distribution.

We first discuss our notion of model quality. Neural networks are typically evaluated by their performance on a test-set. However, what really matters is performance on the test-_distribution_\(\mathcal{D}\), because this is what determines the expected performance on new batches of unseen data. Therefore, our notion of model quality is the distribution-wise accuracy \(A(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathbf{1}_{f_{\theta}(x)=y}]\). The test-set is a finite sample from \(\mathcal{D}\), so that test-set accuracy is a noisy approximation to this.

Estimating the _mean_ of distribution-wise accuracy \(\mu:=\mathbb{E}_{\theta\sim\mathcal{A}}[A(\theta)]\) is relatively easy, because test-set accuracy is an unbiased estimator, as \(\mathbb{E}_{S\sim\mathcal{D}^{n}}[\mathbb{E}_{\theta\sim\mathcal{A}}[A_{S}( \theta)]]=\mu\). Estimating the _variance_\(\sigma^{2}:=\operatorname{Var}_{\theta\sim\mathcal{A}}(A(\theta))\) is more challenging. We prove that test-set variance is an overestimate for this quantity (proof in Appendix B.1).

**Theorem 1**.: _In expectation, variance in test-set accuracy overestimates distribution-wise variance._

\[\mathbb{E}_{S\sim\mathcal{D}^{n}}\left[\operatorname*{Var}_{\theta\sim \mathcal{A}}(A_{S}(\theta))\right]\geq\operatorname*{Var}_{\theta\sim \mathcal{A}}(A(\theta))\]

We aim to obtain an unbiased estimate of \(\operatorname{Var}_{\theta\sim\mathcal{A}}(A(\theta))\). We recall the results of Section 4.2: When training to convergence, we found that test-set accuracy follows a distribution which can be approximately recovered by assuming that all test examples vary independently from each other, in terms of the event that the model classifies them correctly (Hypothesis 1, Figure 5). The distribution-wise variance in this case should be essentially zero. On the other hand, for shorter trainings we observed substantial test-set variance in excess of that predicted by Hypothesis 1, _i.e._, \(\operatorname{Var}_{\theta\sim\mathcal{A}}(A_{S}(\theta))>\frac{1}{n^{2}}\sum_ {i=1}^{n}\overline{C}_{x_{i},y_{i}}(1-\overline{C}_{x_{i},y_{i}})\). For example, the hypothesis predicted that our 4-epoch configuration should have a standard deviation of \(0.22\%\), but the observed value was much larger at around \(0.56\%\). This suggests that these shorter trainings may have significant true variance between runs.

Our intuition is therefore that distribution-wise variance is related to this excess of test-set variance over that predicted by Hypothesis 1. We prove (Appendix B.2) that with a small rescaling, this is true:

**Theorem 2**.: _The distribution-wise variance between runs is equal to the following expectation._

\[\operatorname*{Var}_{\theta\sim\mathcal{A}}(A(\theta))=\frac{n}{n-1}\cdot \mathbb{E}_{S\sim\mathcal{D}^{n}}\left[\operatorname*{Var}_{\theta\sim \mathcal{A}}(A_{S}(\theta))\,-\,\frac{1}{n^{2}}\sum_{i=1}^{n}\overline{C}_{x_{ i},y_{i}}(1-\overline{C}_{x_{i},y_{i}})\right] \tag{2}\]

In particular, the quantity \(\hat{\sigma}_{S}^{2}:=\frac{n}{n-1}\cdot\left(\operatorname*{Var}_{\theta\sim \mathcal{A}}(A_{S}(\theta))\,-\,\frac{1}{n^{2}}\sum_{i=1}^{n}\overline{C}_{x_{ i},y_{i}}(1-\overline{C}_{x_{i},y_{i}})\right)\) is our desired unbiased estimator for distribution-wise variance. We calculate this estimate using many runs of training, with a fixed test-set. In Figure 7 we compare \(\hat{\sigma}_{S}^{2}\) to the simple test-set variance \(\operatorname{Var}_{\theta\sim\mathcal{A}}(A_{S}(\theta))\) across a range of training durations. When training for 4 epochs, we estimate the standard deviation of test-distribution accuracy to be \(\sqrt{\hat{\sigma}_{S}^{2}}=0.52\%\), indicating significant differences in model quality between trainings of this duration. In contrast, when training for 64 epochs, we estimate the distribution-wise standard deviation to be only \(0.033\%\). We obtain a similarly small estimate for ImageNet trainings in Section 5.2. These findings suggest that when training to convergence, there is little variation in model quality (_i.e._, expected performance with respect to new batches of data from the test-distribution) between runs of training.

Having confirmed that distribution-wise variance is small, it still remains to explain why there is high variance on the finite test-set in the first place. In the next section, we investigate the reasons for this.

Figure 7: **Test-set variance overestimates true variance.** We use Equation 2 to calculate unbiased estimates of the distribution-wise variance \(\operatorname{Var}_{\theta\sim\mathcal{A}}(A(\theta))\), which become \(22\times\) smaller than the test-set variance when training to convergence.
### Consequences of calibration

In this section we prove that variance in the test-set accuracy between runs of training is an inevitable consequence of the observation that ensembles of trained networks are well-calibrated. Our analysis additionally leads us to derive a simple formula which accurately estimates variance for binary classification problems.

In comparison to neural networks, regularized linear models have little variance between runs of training. For example, when training a linear model on CIFAR-10 with optimized choice of weight decay, we observe in Figure 9 (left) that the standard deviation of test-set accuracy is below 0.01%. This is because \(L^{2}\)-regularized linear models have a single optimum, so that variance with respect to both the test-set and test-distribution tends towards zero as training progresses. For neural networks, we found in the previous section that distribution-wise variance drops to a small value when training for long enough. But, unlike the linear model, the standard deviation of test-set accuracy plateaus around 0.14%, even for long trainings. It remains to explain this excessive test-set variation in neural network trainings, relative to linear models.

Lakshminarayanan et al. (2017) observe that neural networks have the following property: if we collect an ensemble of many independently trained networks, then the uncertainty estimates of this ensemble will be roughly calibrated. That is, if we let \(S^{\prime}\) be the subset of test images which are classified by 30-40% of independently trained networks as "dog", then approximately 30-40% of the images in \(S^{\prime}\) really will be dogs. Mathematically, ensemble-calibration can be defined as the following hypothesized property of training.

**Hypothesis 2** (Ensemble calibration).: _For every \(j\in\mathcal{Y}\) and \(p\in[0,1]\),_

\[\underset{(x,y)\sim\mathcal{D}}{P}\left(y=j\mid P_{\theta\sim\mathcal{A}}(f_{ \theta}(x)=j)=p\right)=p\]

To see how this property relates to variance, suppose we have a zero-variance training configuration which makes the same test-set predictions every run. This will produce a subset of examples which are predicted as "dog" by 100% of trained networks. But unless the network has zero error, not all of these examples really will be dogs. So the ensemble must be overconfident, and cannot satisfy Hypothesis 2. Therefore, ensemble-calibration seems to imply a degree of variation between runs. We prove the following results in support of this claim.

**Theorem 3**.: _For binary classification, test-set variance is bounded below as follows._

\[\underset{S\sim\mathcal{D}^{n}}{\mathbb{E}}\left[\underset{\theta\sim \mathcal{A}}{\mathrm{Var}}(A_{S}(\theta))\right]\geq\frac{\mathrm{err}( \mathcal{A})-\mathrm{ECE}(\mathcal{A})}{2n}\]

**Theorem 4**.: _For binary classification, if we have ensemble-calibration and zero distribution-wise variance, then the test-set variance is given by the following formula._

\[\underset{S\sim\mathcal{D}^{n}}{\mathbb{E}}\left[\underset{\theta\sim \mathcal{A}}{\mathrm{Var}}(A_{S}(\theta))\right]=\frac{\mathrm{err}(\mathcal{ A})}{2n}\]

**Theorem 5**.: _For \(k\)-way classification, assuming ensemble-calibration, test-set variance is lower-bounded as follows._

\[\underset{S\sim\mathcal{D}^{n}}{\mathbb{E}}\left[\underset{\theta\sim \mathcal{A}}{\mathrm{Var}}(A_{S}(\theta))\right]\geq\frac{\mathrm{err}( \mathcal{A})}{nk^{2}}\]

In these theorems, we first consider binary classification. In this setting, we derive a lower-bound (Theorem 3) on the test-set variance, as a function of the expected calibration error (ECE). The definition of ECE can be found with the proof in Appendix B.3; it is equal to

Figure 8: **Test-set variance can be estimated by a simple formula for binary classification tasks.** Across hundreds of tasks, the formulas of Theorem 3 and 4 approximately recover the true stddevs of test-set accuracy. In comparison, the binomial approximation is inaccurate.
zero when Hypothesis 2 is satisfied. We additionally derive an exact formula (Theorem 4), which holds when assuming both ensemble-calibration and zero distribution-wise variance. Proof is provided in Appendix B.4.

In Figure 8 we apply these formulas to predicting the variance in test-set accuracy, for each of hundreds of binary classification tasks derived from CIFAR-10. We evaluate all \(\binom{10}{5}/2\) tasks corresponding to classifying each subset of five classes versus the other five. These tasks have a wide range of error rates and variances. As theoretically expected, the formula \((\operatorname{err}(\mathcal{A})-\operatorname{ECE}(\mathcal{A}))/2n\) is a lower bound. Empirically, it looks to be close to the true values. The downside of this formula is that it involves the calculation of \(\operatorname{ECE}(\mathcal{A})\), which requires sampling many runs of training. The next formula we evaluate is \(\operatorname{err}(\mathcal{A})/2n\), which does not have this weakness, since only a single run is needed to approximately estimate the average error. When assuming both Hypothesis 1 and Hypothesis 2, this formula is theoretically exact (Theorem 4). Empirically, it is also a good fit, with \(R^{2}=0.996\) across this collection of tasks. As a baseline, we also include the variance estimate produced by the binomial approximation, which is \(\operatorname{err}(\mathcal{A})(1-\operatorname{err}(\mathcal{A}))/n\). This estimate is around two times larger than that of Theorem 4 for these tasks, and significantly overestimates the true variance.

Finally, we consider \(k\)-way classification for arbitrary \(k\). In this setting, assuming ensemble-calibration, we derive a lower-bound for the test-set variance in terms of the average error (Theorem 5), with proof in Appendix B.5. We note that this bound is loose, and in our experiments not a good approximation to the true variance. From these theorems, we conclude that ensemble-calibration (Hypothesis 2) inevitably implies positive variance in test-set accuracy between runs of training, for all classification tasks.

## 5 Additional experiments

### BERT finetuning

In this section we study BERT (Devlin et al., 2018) finetuning, and use the tools developed in Section 4 to clearly differentiate the behavior of BERT-Large from BERT-Base. For our experiment, we finetune pretrained checkpoints of both models 1,000 times each on the MRPC (Dolan and Brockett, 2005) task. MRPC contains 5,801 sentence pairs labeled yes/no for whether the second sentence paraphrases the first, and has a training set of 3,668 examples, validation set of 408 examples, and test set of 1,725 examples.

Previous works (Devlin et al., 2018; Dodge et al., 2020; Mosbach et al., 2020) report and investigate training instability for BERT finetuning, noting particular instability for BERT-Large. In our experiment, we find that both BERT-Large and BERT-Base have substantial variance in validation-set performance between runs. BERT-Base has a standard deviation of 0.80% and BERT-Large has 2.24%, which seems to imply that both models are unstable, with BERT-Large being only somewhat moreso. However, we find that the tools of Section 4 yield a more complete picture which clearly differentiates the two models. In

Figure 9: (Far left:) Training a regularized linear model has very little variance between runs. (Center left:) Removing either data augmentation, or 80% of training data from CIFAR-10 training, reduces the average accuracy to around 87.5%. But the former produces much more variance between runs than the latter. (Right two:) When finetuning BERT-Base on MRPC, variations between runs in terms of performance on the validation and test sets are not strongly correlated. On the other hand, BERT-Large has significant instability.
Figure 9 (right) we show that for BERT-Base, the validation and test splits are close to decorrelated in terms of finetuned model performance. The top 15% of seeds in terms of validation-set performance achieve only 0.09% higher performance than average on the test-set, implying that most of the observed variation does not generalize. In contrast, for BERT-Large there is a clear correlation. Furthermore, Equation 2 estimates a distribution-wise standard deviation of 2.08% for BERT-Large compared to only 0.21% for BERT-Base, meaning that the former has 99\(\times\) more variance. This provides a strong confirmation that BERT-Large finetuning is significantly more unstable compared to BERT-Base; as to the reasons underlying this instability, we defer to prior and potentially future work.

### ImageNet distribution shifts

In this section we show that shifted distributions of test data have increased variance in their accuracy distributions between runs of training. We additionally confirm that the findings of Section 4 generalize to standard ImageNet training.

Our experiment is as follows. We independently train 1,000 ResNet-18s on ImageNet using a standard configuration (see Appendix A.2), attaining an average accuracy of 71.0%. We study the predictions of these networks on the ImageNet validation set, ImageNet-V2, and three distribution-shifted datasets.

We first look at the ImageNet validation set. In Figure 10 (rightmost) we observe that the true distribution closely matches the one predicted by Hypothesis 1. The observed standard deviation is 0.118%, but using Equation 2 we estimate distribution-wise variance to be 12\(\times\) smaller, at 0.034%. This value is close to what we found for CIFAR-10, confirming that both training scenarios adhere to our conclusions from Section 4.

Next we consider ImageNet-V2 (Recht et al., 2019). This dataset is intended to have the same distribution of examples as ImageNet, and we demonstrate that its accuracy distribution has similar statistical properties as well. Specifically, we find that the distribution predicted by Hypothesis 1 also closely matches the true distribution, and Equation 2 estimates a distribution-wise standard deviation of 0.071%, which is larger than what we found on the ImageNet validation set, but still relatively small. We note that the accuracy distribution for this dataset is wider, but under Hypothesis 1 this can be explained simply by the fact that it has 5\(\times\) fewer examples than the ImageNet validation set.

By contrast, ImageNet-R (Hendrycks et al., 2021), ObjectNet (Barbu et al., 2019) and ImageNet-Sketch (Wang et al., 2019) have different statistical behavior compared to the first two datasets. These datasets are constructed to have shifted distributions relative to ImageNet. We find that their accuracy distributions have variance significantly in excess of that predicted by Hypothesis 1. We estimate using Equation 2 that these three test-sets have large distribution-wise standard deviations of 0.181%, 0.179%, and 0.257% respectively, indicating significant differences between runs of training.

We additionally investigate correlations between pairs of these five datasets (Figure 15). The strongest correlation is between ImageNet-R and ImageNet-Sketch, with \(R^{2}=0.14\) (\(p<10^{-8}\)). Manual inspection

Figure 10: **Distribution shift produces excess distribution-wise variance between runs.** Across 1,000 runs of ImageNet training, both the ImageNet validation set and ImageNet-V2 have accuracy distributions close to that predicted by Hypothesis 1. On the other hand, distribution-shifted sets have significant excess variance, which indicates genuine differences between trained models, in light of Theorem 2.
shows that both ImageNet-Sketch and ImageNet-R contain many sketch-like images, suggesting that similar features may induce correlation between distributions. All other pairs have \(R^{2}<0.01\). For example, ImageNet-Sketch is decorrelated from ObjectNet, with \(R^{2}=0.001\) (\(p=0.336\)).

Overall, our findings suggest that training instability is in some sense a relative notion. ImageNet training is stable when evaluated on the main distribution, with a small standard deviation of \(0.034\%\) on the distribution of the ImageNet validation set. But it is unstable on shifted distributions, with ImageNet-Sketch having \(58\times\) as much distribution-wise variance, at a standard deviation of \(0.257\%\). This serves as a caveat to the title: from the perspective of the main training distribution, variance is harmless in that every trained network has almost the same performance, but from the perspective of shifted distributions, there are significant differences between runs.

### The effect of data augmentation

In this section we look at the effect of data augmentation on variance. In Figure 9 (center left) we compare two ablations: first, removing a fixed \(80\%\) of training data, and second, removing data augmentation. While both configurations achieve a similar mean accuracy of \(87.5\%\), the augmentation-free training has over \(3.5\times\) more variance between runs. We also observe that the ensemble accuracy of the augmentation-free networks is higher, reaching \(91.2\%\), compared to the reduced-data ensemble, which achieves only \(89.8\%\). Based on these observations, we speculate that one role of data augmentation may be to reduce variance between runs.

### The effect of the learning rate

In this section we investigate the relationship between learning rate and variance. Our experiment is to execute 1,000 64-epoch CIFAR-10 trainings for each binary-power learning rate between \(2^{-10}\) and \(2^{2}\). For each setting, we measure the mean and variance of test-set accuracy. We observe three distinct regimes (Figure 11). For small learning rates below \(2^{-5}\), performance is low and variance is high. Between \(2^{-5}\) and \(0.5\), the performance increases with the learning rate, and variance is close to the level predicted by Hypothesis 1, indicating little distribution-wise variance. Above \(0.5\), which is the optimal learning rate, performance drops and variance rapidly increases.

We observe that a learning rate of \(0.5\) produces both the highest mean accuracy and the lowest variance. Raising it to \(1.0\) causes the standard deviation of test-set accuracy to increase from \(0.148\%\) to \(0.168\%\). This may seem insignificant, but Equation 2 estimates that this corresponds to the distribution-wise standard deviation rising from \(0.033\%\) to \(0.075\%\), which is a significant \(5\times\) increase in distribution-wise variance. We therefore speculate that, as a general property of neural network trainings, the optimal learning rate is the largest one which does not induce significant distribution-wise variance.

Figure 11: **Accuracy peaks at the largest learning rate without excess variance. We show the mean and standard deviation of test-set accuracy across a range of learning rates. The highest accuracy is attained at learning rate \(0.5\), which also has the smallest excess variance relative to the prediction of Hypothesis 1. When raising the learning rate above that, variance rises and accuracy drops.**
### Neural posterior correlation kernel

In Section 4.2, we showed that the shape and standard deviation of the test-set accuracy distribution are closely approximated by the hypothesis that all test-example predictions vary independently between runs of training. In this section, we investigate whether there exist individual pairs of examples which deviate from this hypothesis, and introduce a new kernel based on correlations between predicted logits.

Hypothesis 1 predicts that each pair of test-set examples \((x_{i},y_{i}),(x_{j},y_{j})\) varies independently in terms of the events that each are classified correctly by the trained model. That is, a trained model \(f_{\theta}\) which correctly predicts \(f_{\theta}(x_{i})=y_{i}\) should be no more likely than chance to also correctly predict \(f_{\theta}(x_{j})=y_{j}\).

To find deviations from this hypothesis, we repeat the experiment of Figure 6 across all pairs of test-set examples. That is, for a pair \((x_{i},y_{i}),(x_{j},y_{j})\), let \(C_{1}:=C_{x_{i},y_{i}}\) and \(C_{2}:=C_{x_{j},y_{j}}\). Then we estimate the values of \(P(C_{1})\), \(P(C_{2})\), and \(P(C_{1}C_{2})\) across our 60,000 runs of training, and compute \(\delta_{ij}=|\hat{P}(C_{1}C_{2})-\hat{P}(C_{1})\hat{P}(C_{2})|\) in order to measure deviation from Hypothesis 1.

Across all \(\binom{10,000}{2}\) pairs, we find that there exist only five for which \(\delta_{ij}\geq 0.02\) (Figure 13), and an additional 19 with \(\delta_{ij}\geq 0.01\). The remaining 49,994,976 approximately conform the hypothesis with \(|\hat{P}(C_{1}C_{2})-\hat{P}(C_{1})\hat{P}(C_{2})|<0.01\). Therefore, on a fine-grained level Hypothesis 1 also seems to be a good approximation for all but a small number of pairs.

We next look for pairs of examples whose predicted _logits_ are correlated between runs of training. For \(K\)-way classification, let \(F_{\theta}(x)\) be the vector of \(K\) logits produced by the

Figure 12: **The neural posterior correlation kernel.** (Left:) TriMap projection of the CIFAR-10 test-set according to the NPCK, which groups examples together based on correlations of their predicted logits between repeated runs of training. (Right:) From the top, ten pairs with \(\kappa_{NPCK}(x,x^{\prime})\approx 0.75\), then 0.50, then 0.25. The reader is invited to interactively explore this visualization at [https://observablehq.com/d/24e96d5a104e383f](https://observablehq.com/d/24e96d5a104e383f).

Figure 13: **Anomalous pairs.** Out of all pairs of examples in the CIFAR-10 test-set, only these five deviate by more than 2% from Hypothesis 1.
trained network \(\theta\) on input \(x\). For a pair of inputs \((x,x^{\prime})\), we define the "neural posterior correlation kernel" (NPCK) as follows:

\[\kappa_{NPCK}(x,x^{\prime})=\frac{1}{K}\sum_{k=1}^{K}\operatorname*{ corr}_{\theta\sim\mathcal{A}}(F_{\theta}(x)_{k},F_{\theta}(x^{\prime})_{k}) \tag{3}\]

We compute an approximation of this kernel using the logit outputs from 60,000 networks trained for 64 epochs each. We find that the NPCK is effective at revealing structure in the dataset; _e.g._, pairs of inputs with high NPCK values are almost all visually near-duplicates. We present a sample of such pairs in Figure 12; there are 11 pairs with \(\kappa_{NPCK}(x,x^{\prime})\geq 0.75\) which are almost exact duplicates, 100 pairs with \(\kappa_{NPCK}(x,x^{\prime})\geq 0.5\) which are near-duplicates, and 670 pairs with \(\kappa_{NPCK}(x,x^{\prime})\geq 0.25\), which share visual features. We additionally use TriMap (Amid and Warmuth, 2019) to produce a 2D projection of the test-set according to the kernel, which the reader is invited to explore at the URL provided in Figure 12. One can find clusters of near-duplicate images and various intriguing subpopulations by inspecting this projection.

Compared to distances in penultimate layer feature-spaces, the NPCK performs favorably. We compare it to the penultimate layer features of two models: one of our small trained ResNets, and a much larger finetuned ViT (Dosovitskiy et al., 2020) which attains over 98% accuracy on CIFAR-10. Out of the 100 most similar pairs for each metric, the NPCK recovers a far higher fraction of near-duplicates (Figure 16 and Figure 17). Only a specially trained duplicate detection model (Yokoo, 2021) performs comparably.

The NPCK bears a close similarity to the neural network prior kernel (NNPK), which is defined by Amid et al. (2022) as \(\kappa_{NNPK}(x,x^{\prime})=\mathbb{E}_{\theta\sim\pi}[\langle F_{\theta}(x), F_{\theta}(x^{\prime})\rangle]\), that is, the expected inner product between predicted logits as the network weights vary over the initialization distribution \(\pi\). The NPCK is equivalent to the NNPK with two modifications: first, normalizing the logits, and second, passing from the weight distribution at initialization to the weight distribution after training. That is, let \(G_{\theta}(x)_{k}=\frac{F_{\theta}(x)_{k}-\mathbb{E}_{\theta\sim\mathcal{A}} [F_{\theta}(x)_{k}]}{\operatorname*{std}_{\theta\sim\mathcal{A}}(F_{\theta}( x)_{k})}\) be the normalized logits, then the NPCK satisfies the equality \(\kappa_{NPCK}(x,x^{\prime})=\frac{1}{K}\mathbb{E}_{\theta\sim\mathcal{A}}[ \langle G_{\theta}(x),G_{\theta}(x^{\prime})\rangle]\).

We can view the NPCK as inducing an embedding via the map \(\varphi(x)=\{\kappa_{NPCK}(x,x_{j})\}_{j\in[10,000]}\). This map shares similar properties to the _datamodel embeddings_ proposed by Ilyas et al. (2022). Both are computed via thousands of runs of training, and reveal global structure in the dataset more effectively than penultimate-layer feature embeddings. And both embeddings have a high "effective dimension": the top 500 principal components of the datamodel embeddings of the training set explain less than 50% of their variation, and similarly, the top 500 components of our NPCK-based embeddings of the test set explain only 55%. In comparison, just 10 components explain 90% of variation in penultimate layer features (Ilyas et al., 2022). The NPCK-based embeddings are somewhat simpler to compute than datamodel embeddings, as they do not depend on varying the training data, but of course, this makes them lack the specific meaning of datamodels. Overall, both embeddings can be used effectively to reveal structure in the training and test datasets.

## 6 Conclusions

In this paper we present a method of estimating the variance between repeated runs of training with respect to performance on a test-_distribution_. This form of variance implies genuine differences between trained networks in terms of their expected performance on new batches of data, whereas we show that the variance in performance on a test-_set_ often does not. In particular, we find that standard CIFAR-10 and ImageNet trainings have significant variance on their respective test-sets, but almost no variance with respect to their test-distributions, so that all networks generated by repeated runs of such trainings have almost the same expected performance on new samples of test data.

We demonstrate that for these trainings, variation in the test-set predictions approximately satisfies _examplewise independence_ (Hypothesis 1), where the event that a trained network correctly predicts a particular example can be modeled as a (biased) coin flip which varies independently from the network's predictions on other examples. On the other hand, at the level of predicted logits there do exist more significant dependencies between examples, which we show can be used to reveal structure in the dataset via the neural posterior correlation kernel.
Prior works have observed that ensembles of networks from repeated runs of training make predictions that are roughly _calibrated_ (Hypothesis 2). We prove that this calibration inevitably implies a degree of variance in test-set accuracy between runs of training. And for binary classification tasks, when additionally assuming Hypothesis 1, we prove that the variance in test-set accuracy is given by \(\text{err}/2n\), where \(\text{err}\) is the average error and \(n\) is the size of the test-set. We demonstrate that this simple formula is indeed a good approximation across hundreds of binary classifications tasks derived from CIFAR-10.

We use our estimator for distribution-wise variance to obtain the following new or strengthened conclusions connecting variance to other phenomena in deep learning. (1) Distribution-wise variance shrinks as we increase the training duration, reaching a small standard deviation of below 0.04% for standard CIFAR-10 and ImageNet trainings. (2) The optimal learning rate is the largest one that does not induce excess distribution-wise variance. (3) Data augmentation reduces variance between runs. (4) Test-sets which are distribution-shifted relative to the training set incur excess variance compared to in-domain test-data. We demonstrate each of these claims in a limited context, for benchmark tasks. Each therefore forms a partially validated hypothesis about the empirical behavior of neural network training. We are interested in future work which finds counterexamples to these hypotheses, or expands their demonstrated range of applicability. We also hope that our work will lead to the discovery of other such regularities in the variance between runs of neural network training.

## Acknowledgements

We are grateful to Behnam Neyshabur for his guidance on a preliminary version of this work. We thank Ehsan Amid, Luke Johnston, and Ryan Weber each for their insightful comments on the draft.
