On Emergence of Clean-Priority Learning in Early Stopped Neural Networks
When random label noise is added to a training dataset, the prediction error
of a neural network on a label-noise-free test dataset initially improves
during early training but eventually deteriorates, following a U-shaped
dependence on training time. This behaviour is believed to be a result of
neural networks learning the pattern of clean data first and fitting the noise
later in the training, a phenomenon that we refer to as clean-priority
learning. In this study, we aim to explore the learning dynamics underlying
this phenomenon. We theoretically demonstrate that, in the early stage of
training, the update direction of gradient descent is determined by the clean
subset of training data, leaving the noisy subset has minimal to no impact,
resulting in a prioritization of clean learning. Moreover, we show both
theoretically and experimentally, as the clean-priority learning goes on, the
dominance of the gradients of clean samples over those of noisy samples
diminishes, and finally results in a termination of the clean-priority learning
and fitting of the noisy samples.
Chaoyue Liu, Amirhesam Abedsoltan, Mikhail Belkin
# On Emergence of Clean-Priority Learning in Early Stopped Neural Networks

###### Abstract

When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset initially improves during early training but eventually deteriorates, following a U-shaped dependence on training time. This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as _clean-priority learning_. In this study, we aim to explore the learning dynamics underlying this phenomenon. We theoretically demonstrate that, in the early stage of training, the update direction of gradient descent is determined by the clean subset of training data, leaving the noisy subset has minimal to no impact, resulting in a prioritization of clean learning. Moreover, we show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes, and finally results in a termination of the clean-priority learning and fitting of the noisy samples.

Machine Learning, ICML

## 1 Introduction

Recent studies suggest that Neural Network (NN) models tend to first learn the patterns in the clean data and over-fit the noise at a later stage (Arpit et al., 2017; Li et al., 2020). We refer to this phenomena as _clean-priority learning_. Real datasets may have intrinsic label noise, which is why early stopping can be useful in practice, saving a significant amount of unnecessary computation cost.

To study _clean-priority learning_ phenomena, we intentionally add label noise to the training data set and leave the test data set untouched, this is a common setting in literature (see, for example, (Zhang et al., 2021; Nakkiran et al., 2021; Belkin et al., 2018)). Figure 1 illustrate this for a MNIST classification task using CNN. The test prediction error exhibits a U-shaped dependence on training time, with an initial decrease followed by an increase after the early stopping point. The observation is that in the intermediate steps, especially around the early stopping point, the test performance can be significantly better than the label noise level added to the training set (below the dashed line).

To further explore this phenomenon, we address the following fundamental questions:

1. _What is the underlying mechanism by which neural networks learn the clean data first and fit the noise in later stages?_
2. _How does the model performance deteriorate after the early stopping point?_

At the outset, we analyze the configuration of sample-wise gradients (or its variant, for multi-class classification) on the training dataset, at the neural network initialization. Our objective is to examine whether there exists any pattern among the gradients that can explain the clean-priority learning phenomenon. Our analysis reveals that, at initialization, samples within the same class (before label corruption), which are presumably more similar to each other, tend to have their sample-wise gradients relatively closer in vector directions (compared to the samples from different

Figure 1: Classification errors on training and test datasets of MNIST using CNN. Test error exhibits a U-shaped curve, and can be significantly lower than the noise level during training.
classes). The label corruption, which flips the label to a different class, flips the corresponding sample-wise gradient to its opposite direction. Consequently, the sum of the noisy sample gradients is in sharp opposite direction of that of the clean sample gradients.

The key observation is that, due to the dominance of the population of the clean samples, in the early stage of learning, the gradient of noisy subset is cancelled out, and essentially makes no contribution on the gradient descent (GD) update direction1. It is also worth noting that almost all clean sample-wise gradient vectors "agree" with the GD update (i.e., have positive projection), while almost all noisy sample-wise gradients are "against" the GD update. As a result, the individual loss on each clean sample is decreased, and that on each noisy sample is increased. Hence, we see that, in the early stage, the GD algorithm is determined by the clean samples and exhibits the clean-priority learning.

Footnote 1: To be more precise, the only effect of the noisy subset gradient is resulting in a smaller GD step size.

We further show that as the clean-priority learning process continues, the clean subset gradient's dominance over the noisy subset gradually diminishes. This is particularly evident around the early stopping point, at which the noisy subset gradient begins to make a meaningful contribution, causing the model to fit the noisy samples along with the clean ones. This new trend in learning behavior is expected to hurt the model's performance, which was previously based primarily on the clean samples.

In summary, we make the following contributions:

* **learning dynamics.** In the early stage of learning of neural networks, the noisy samples contribution to the GD update is cancelled out by that of the clean samples, which is the key mechanism underlying clean-priority learning. However, this clean-priority learning behavior gradually fades as the dominance of the clean subset diminishes, particularly around the early stopping point. We experimentally verify our findings on deep neural networks on various classification problems.
* For fully connected networks with mild assumption on data we theoretically prove our empirical observation.
* In addition, we find for neural networks, at initialization, sample-wise gradients from the same class tend to have relatively similar directions, when there is no label noise.

The paper is organized as follows: in Section 2, we describe the setup of the problems and introduce necessary concepts and notations. In Section 3, we analyze the sample-wise gradients at initialization of neural network, for binary classification. In Section 4, we show the learning dynamics, especially the clean-priority learning, on binary classification. In Section 5, we extend our study and findings to multi-class classification problems.

### Related works

Early stopping is often considered as a regularization technique and is widely used in practice to obtain good performance for machine learning models (Zhang & Wallace, 2017; Gal & Ghahramani, 2016; Graves et al., 2013). Early stopping also received a lot theoretical analyses, both on non-neural network models, especially linear regression and kernel regression (Yao et al., 2007; Ali et al., 2019; Xu et al., 2022; Shen et al., 2022), and on neural networks (Zhang et al., 2021; Ji et al., 2021).

Recent studies suggest that, when random label noise presents, neural networks fit the clean data first and "overfit" the noise later on (Arpit et al., 2017; Li et al., 2020; Bai et al., 2021). For example, based on experimental observations that maximum validation set accuracy is achieved before good training set accuracy, the work (Arpit et al., 2017) conjectures that the neural network learns clean patterns first. However, there is no explanation on why and how the clean-priority learning happens. In another work (Li et al., 2020), assuming (almost) perfectly cluster-able data and uniform conditioning on Jacobian matrices, proves that clean data are fit by two-layer neural networks in an early stage. However, this data assumption requires, at the same location of each noisy sample, there must exist several (at least \(1/\delta\), with \(\delta\) being the label noise level) clean samples. This assumption is often not met by actual datasets.

To the best of our knowledge, our work is the first to elucidate the mechanism underlying the clean-priority phenomenon, offering a fresh perspective on its dynamics.

## 2 Problem setup and preliminary

In this paper, we consider supervised classification problems.

Datasets.There is a training dataset \(\mathcal{D}\triangleq\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n}\) of size \(|\mathcal{D}|=n\). In each sample \((\mathbf{x}_{i},y_{i})\), there are input features \(\mathbf{x}_{i}\in\mathbb{R}^{d}\) and a label \(y_{i}\). For binary (2-class) classification problems, the label \(y_{i}\in\{0,1\}\) is binary; for multi-class classification problems, the label is one-hot encoded, \(y_{i}\in\mathbb{R}^{C}\), where \(C\) is the total number of classes. We further denote \(\mathcal{D}^{(c)}\), \(c\in\{1,2,\cdots,C\}\), as the subset of \(\mathcal{D}\) that is composed of samples from the \(c\)-th class. It is easy to see, \(\mathcal{D}=\cup_{c=1}^{C}\mathcal{D}^{(c)}\).

We assume the labels in \(\mathcal{D}\) are randomly corrupted. Specif
ically, if denote \(\hat{y}\) as the ground truth label of \((\mathbf{x}_{i},y_{i})\in\mathcal{D}\), there exists a non-empty set

\[\mathcal{D}_{noise}\triangleq\{(\mathbf{x}_{i},y_{i})\in\mathcal{D}:y_{i}\neq \hat{y}_{i}\}. \tag{1}\]

Furthermore, the labels \(y_{i}\) in \(\mathcal{D}_{noise}\) is uniformly randomly distributed across all the class labels except \(\hat{y}_{i}\). We call \(\mathcal{D}_{noise}\) as the _noisy subset_ and its elements as _noisy samples_. We also define the _clean subset_\(\mathcal{D}_{clean}\) as the compliment, i.e., \(\mathcal{D}_{clean}=\mathcal{D}\backslash\mathcal{D}_{noise}\), and call its elements as _clean samples_. The _noise level_\(\delta\) is defined as the ratio \(|\mathcal{D}_{noise}|/|\mathcal{D}|\). In this paper, we set \(\delta<0.5\), i.e., the majority of training samples are not corrupted. We also denote \(\tilde{\mathcal{D}}\) as the ground-truth-labeled dataset: \(\tilde{\mathcal{D}}\triangleq\{(\mathbf{x}_{i},\hat{y}_{i})\}_{i=1}^{n}\).

In addition, there is a test dataset \(\tilde{\mathcal{D}}\) which is i.i.d. drawn from the same data distribution as the training set \(\mathcal{D}\), except that the labels of test set \(\tilde{\mathcal{D}}\) are not corrupted.

Optimization.Given an arbitrary dataset \(\mathcal{S}\) and a model \(f\) which is parameterized by \(\mathbf{w}\) and takes an input \(\mathbf{x}\), we define the loss function as

\[L(\mathbf{w};\mathcal{S})=\frac{1}{|\mathcal{S}|}\sum_{(\mathbf{x},y)\in \mathcal{S}}l(\mathbf{w};\mathbf{x}_{i},y_{i}), \tag{2}\]

with \(l(\mathbf{w};\mathbf{x}_{i},y_{i})\triangleq l(f(\mathbf{w};\mathbf{x}),y)\) is evaluated on a single sample and is a function of the model output \(f(\mathbf{w};\mathbf{x})\) and label \(y\). We use the logistic loss for binary classification problems, and the cross entropy loss for multi-class problems. In this paper, we use neural networks as the model. In case of binary classification problems, the output layer of \(f\) has only one neuron, and there exists a _sigmoid_ function on top such that the output \(f(\mathbf{w};\mathbf{x})\in(0,1)\). In case of multi-class classification problems, the neural network \(f\) has \(C\) output neurons, and there exists a _softmax_ function to normalize the outputs. Throughout the paper, we assume that the neural network is large enough so that the training data can be exactly fit. This is usually satisfied when the neural network is over-parameterized (Liu et al., 2022).

The optimization goal is to minimize the empirical loss function (i.e., loss function on the training dataset \(\mathcal{D}\)):

\[L(\mathbf{w})\triangleq L(\mathbf{w};\mathcal{D})=\frac{1}{n}\sum_{i=1}^{n}l (\mathbf{w};\mathbf{x}_{i},y_{i}). \tag{3}\]

The above loss function is usually optimized by gradient descent (or its stochastic variants) which has the following update form:

\[\mathbf{w}_{t+1} =\mathbf{w}_{t}-\eta\nabla L(\mathbf{w}_{t};\mathcal{D}) \tag{4}\] \[=\mathbf{w}_{t}-\eta\frac{1}{n}\sum_{(\mathbf{x}_{i},y_{i})\in \mathcal{D}}\nabla l(\mathbf{w}_{t};\mathbf{x}_{i},y_{i}).\]

Here, \(\nabla l(\mathbf{w};\mathbf{x}_{i},y_{i})\) is the gradient of loss \(l(\mathbf{w};\mathbf{x}_{i},y_{i})\) w.r.t. the neural network parameters \(\mathbf{w}\).

Sample-wise gradients.We call \(\nabla l(\mathbf{w};\mathbf{x}_{i},y_{i})\)_sample-wise gradient_, as it is evaluated on a single sample, and denote it as \(\nabla l_{i}(\mathbf{w})\) for short. As there are \(n\) samples in \(\mathcal{D}\), at each point \(\mathbf{w}\) in the parameter space, we have \(n\) sample-wise gradients, each of which is of dimension \(p\).

Denote \(h(\mathbf{w};\mathbf{x})\) as the pre-activation output neuron(s) before activation, which is in \(\mathbb{R}\) for binary classification, and is in \(\mathbb{R}^{C}\) for multi-class classification. The sample-wise gradient for a given sample \((\mathbf{x}_{i},y_{i})\) has the following form (Bishop and Nasrabadi, 2006):

\[\nabla l_{i}(\mathbf{w})=(f(\mathbf{w};\mathbf{x}_{i})-y_{i})\nabla h(\mathbf{ w};\mathbf{x}_{i}). \tag{5}\]

Note that the above expression is a scalar-vector multiplication for binary classification, and is a vector-matrix multiplication for multi-class classification.

We note that the sample-wise gradient will be one of our major quantities, and play a fundamental role in the analysis through out this paper.

## 3 Sample-wise gradients at initialization for binary classification

In this section, we analyze the configuration of the \(n\) sample-wise gradients at randomly initialization of neural networks for binary classification.

We start with the expression Eq.(5) of the sample-wise gradient \(\nabla l_{i}(\mathbf{w})\). For binary classification, \(\nabla l_{i}(\mathbf{w})\) is proportional to the model derivative \(\nabla h(\mathbf{w};\mathbf{x}_{i})\), up to a scalar factor \(f(\mathbf{w};\mathbf{x}_{i})-y_{i}\). In Section 3.1, we focus on the directions of \(\nabla h(\mathbf{w};\mathbf{x}_{i})\), which is label independent. Then, in Section 3.2, we discuss the effects of the label-related factor \(f(\mathbf{w};\mathbf{x}_{i})-y_{i}\) and label noise on the direction of sample-wise gradients.

### Direction of the model derivative \(\nabla h\)

Given any two inputs \(\mathbf{x},\mathbf{z}\in\mathbb{R}^{d}\), we denote the angle between \(\mathbf{x}\) and \(\mathbf{z}\) in the data space by \(\theta_{d}(\mathbf{x},\mathbf{z})\), and denote the angle between the two vectors \(\nabla h(\mathbf{w}_{0};\mathbf{x})\) and \(\nabla h(\mathbf{w}_{0};\mathbf{z})\) by \(\theta_{h}(\mathbf{x},\mathbf{z})\). In this subsection, we are concerned with the connection between \(\theta_{h}(\mathbf{x},\mathbf{z})\) and \(\theta_{d}(\mathbf{x},\mathbf{z})\) at the network initialization.

Consider the following two types of neural networks: a two-layer linear network \(h_{1}\), which is defined as

\[h_{1}(\mathbf{w};\mathbf{x})=\frac{1}{\sqrt{m}}\mathbf{v}^{T}A\mathbf{x}, \tag{6}\]

and a two-layer ReLU network \(h_{2}\), which is defined as

\[h_{2}(\mathbf{w};\mathbf{x})=\frac{1}{\sqrt{m}}\mathbf{v}^{T}\sigma(A\mathbf{x }). \tag{7}\]

Here, \(\sigma(\cdot)=\max(\cdot,0)\) is the element-wise ReLU activation function, \(\mathbf{v}\in\mathbb{R}^{m}\) and \(A\in\mathbb{R}^{m\times d}\) are the first layer
and second layer parameters, respectively. We denoted \(\mathbf{w}\) as the collection of all the parameters. We use the NTK parameterization Jacot et al. (2018) for the networks; namely, each parameter is i.i.d. initialized using the normal distribution \(\mathcal{N}(0,1)\) and there exist a scaling factor \(1/\sqrt{m}\) explicitly on each hidden layer.

The following theorem and its corollary show the relation between \(\theta_{h}(\mathbf{x},\mathbf{z})\) and \(\theta_{d}(\mathbf{x},\mathbf{z})\). (Proofs in Appendix A.1).

**Theorem 3.1**.: _Consider the two-layer neural networks, \(h_{1}\) and \(h_{2}\), defined in Eq.(6) and (7), with infinite width \(m\). Given any two inputs \(\mathbf{x}\) and \(\mathbf{z}\), the angles \(\theta_{h_{1}}(\mathbf{x},\mathbf{z})\) and \(\theta_{h_{2}}(\mathbf{x},\mathbf{z})\) have the following relations with \(\theta_{d}(\mathbf{x},\mathbf{z})\), at network initialization \(\mathbf{w}_{0}\): for the linear neural network \(h_{1}\),_

\[\theta_{h_{1}}(\mathbf{x},\mathbf{z})=\theta_{d}(\mathbf{x},\mathbf{z});\]

_for the ReLU neural network \(h_{2}\),_

\[\cos\theta_{h_{1}}(\mathbf{x},\mathbf{z})=\frac{\pi-\theta_{d}(\mathbf{x}, \mathbf{z})}{\pi}\cos\theta_{d}(\mathbf{x},\mathbf{z})+\frac{1}{2\pi}\sin \theta_{d}(\mathbf{x},\mathbf{z}).\]

**Corollary 3.2**.: _Consider the same networks \(h_{1}\) and \(h_{2}\) as in Theorem 3.1. For both networks, the following holds: for any inputs \(\mathbf{x}\), \(\mathbf{z}\) and \(\mathbf{z}^{\prime}\), if \(0\leq\theta_{d}(\mathbf{x},\mathbf{z})\leq\theta_{d}(\mathbf{x},\mathbf{z}^{ \prime})\leq\frac{\pi}{2}\), then \(0\leq\theta_{h_{i}}(\mathbf{x},\mathbf{z})\leq\theta_{h_{i}}(\mathbf{x}, \mathbf{z}^{\prime})\leq\frac{\pi}{2}\), for \(i\in\{1,2\}\)._

The theorem and corollary suggest that: _similar inputs (small angle \(\theta_{d}\)) induce similar model derivatives (small angle \(\theta_{h}\))._

**Remark 3.3** (Not just at initialization).: As discussed in Liu et al. (2020, 2022), the model derivative \(\nabla h(\mathbf{w};\mathbf{x})\) is constant during optimization for infinitely wide neural networks. Hence, the angle \(\theta_{h}\) between model derivatives is also constant, and Theorem 3.1 and Corollary 3.2 apply to any time stamp of the network training.

Experimental verification.We experimentally verify the above theoretical results on neural networks with large width. Specifically, we consider six neural networks: three linear networks with \(2\), \(3\) and \(5\) layers, respectively; and three ReLU networks with \(2\), \(3\) and \(5\) layers. Each hidden layer of each neural network has \(512\) neurons. For each network, we compute the model derivatives \(\nabla h\) on the \(1\)-sphere \(\mathcal{S}^{1}=\{(\cos\theta_{d},\sin\theta_{d}):\theta_{d}\in[0,2\pi)\}\), at the network initialization. Figure 2 shows the relations between the angle \(\theta_{h}\) and \(\theta_{d}\).2 We observe that the curves for the \(2\)-layer networks match Theorem 3.1. More importantly, the experiments suggest that the same or similar relations, as well as Corollary 3.2, still hold for deep neural networks, although our analysis is conducted on shallow networks.

Footnote 2: The curves for the three linear networks are almost identical and not visually distinguishable, we only present the one for \(2\)-layer linear network in Figure 2.

Consider the following synthetic dataset (also shown in the left panel of Figure 3): two separated data clusters in a \(2\)-dimensional space. We use a \(3\)-layer ReLU network of width \(m=512\) at its initialization to compute the sample-wise model derivatives \(\nabla h\). The right panel of Figure 3 shows the distributions of angle \(\theta_{h}\) for data pairs from the same cluster ("within") and from different clusters ("between"). It can be easily seen that the "within" distribution has smaller angles \(\theta_{h}\) than the "between" distribution, which is expected as the data from the same clusters are more similar.

### Directions of the sample-wise gradients

Now, we consider the sample-wise gradients \(\nabla l(\mathbf{w}_{0})\) at initialization, using Eq.(5). We note that the direction of \(\nabla l(\mathbf{w}_{0})\) is determined by \(\nabla h(\mathbf{w}_{0};\mathbf{x})\) and the label \(y\). This is because only the sign (not the magnitude) of \(y-f(\mathbf{w}_{0};\mathbf{x})\) may affect the direction, and the post-activation output \(f(\mathbf{w}_{0};\mathbf{x})\) is always in \((0,1)\) and label \(y\) is either \(0\) or \(1\).

Motivated by this observation, for a fixed class \(c\in\{0,1\}\), we consider the following subsets: clean subset \(\mathcal{D}^{(c)}_{clean}\triangleq\mathcal{D}^{(c)}\cap\mathcal{D}_{clean}\), noisy subset \(\mathcal{D}^{(c)}_{noise}\triangleq\mathcal{D}^{(c)}\cap\mathcal{D}_{noise}\), and

Figure 3: **(Left) Data visualization: two separated data clusters in \(2\)-dimensional space. (Right) Distributions of angle \(\theta_{h}\) for sample pairs from the same cluster (“within”) and from different clusters (“between”).**

Figure 2: Relation between \(\theta_{h}\) and \(\theta_{d}\). For both shallow and deep neural networks, similar inputs (small angle \(\theta_{d}\)) induce similar model derivatives (small angle \(\theta_{h}\)).
\(\mathcal{D}^{(c)}_{other}\triangleq\mathcal{D}_{clean}\backslash\mathcal{D}^{(c)}\). We note that \(\mathcal{D}^{(c)}_{noise}\) and \(\mathcal{D}^{(c)}_{clean}\) have the same input distribution but different labels \(y\), while \(\mathcal{D}^{(c)}_{noise}\) and \(\mathcal{D}^{(c)}_{other}\) have the same labels \(y\) but different input distributions.

For these subsets, we denote their corresponding sets of sample-wise gradients as \(\mathcal{G}^{(c)}_{clean}(\mathbf{w})\), \(\mathcal{G}^{(c)}_{noise}(\mathbf{w})\) and \(\mathcal{G}^{(c)}_{other}(\mathbf{w})\), respectively. We also define the corresponding _subset gradients_ as the sum of all sample-wise gradients in the subset: \(g^{(c)}_{k}(\mathbf{w})\triangleq\sum_{\nabla l(\mathbf{w})\in\mathcal{G}^{(c )}_{k}}\nabla l(\mathbf{w})\), for \(k\in\{clean,\ noise,\ other\}\). the direction of \(g^{(c)}_{k}(\mathbf{w})\) is the same as that of the average gradient in the subset.

**Direction of sample-wise gradients.** We consider the angles between sample-wise gradients, and denote by \(\theta_{g}\).

First we consider the clean subset. Within \(\mathcal{D}^{(c)}_{clean}\), the factor \(f(\mathbf{w};\mathbf{x})-y\) always have the same sign, as the label \(y\) is the same. Thus, the directional distribution of \(\mathcal{G}^{(c)}_{clean}(\mathbf{w}_{0})\) is identical to the \(\nabla h\) distribution, which has been analyzed in Section 3.1. Presumably, inputs from the same ground truth class tend to be more similar (with small angles \(\theta_{d}\)), compared to others. Applying Corollary 3.2, we expect the angles \(\theta_{h}\), and therefore \(\theta_{g}\) also, within this subset are relatively small. The green plots of Figure 4 show numerical verification of the \(\theta_{g}\) distributions for the clean subset \(\mathcal{D}^{(c)}_{clean}\).3 We see that \(\theta_{g}\) tends to concentrate around relatively small angles.

Footnote 3: For illustration purpose, we compare each sample-wise gradient with the average direction, represented by \(g^{(c)}_{clean}(\mathbf{w}_{0})\).

The interesting part is about the noisy subset \(\mathcal{D}_{noise}\). This subset shares the same input distribution, hence \(\nabla h\) distribution as well, with the clean subset. However, due to different label \(y\), the factor \(f(\mathbf{w}_{0};\mathbf{x})-y\) has different sign from that of clean samples, which flips all \(\nabla l(\mathbf{w}_{0})\in\mathcal{G}^{(c)}_{noise}(\mathbf{w}_{0})\) to the opposite direction of those in \(\mathcal{G}^{(c)}_{noise}(\mathbf{w}_{0})\). As a consequence, sample-wise gradients between \(\mathcal{G}^{(c)}_{noise}(\mathbf{w}_{0})\) and \(\mathcal{G}^{(c)}_{clean}(\mathbf{w}_{0})\) makes large angles \(\theta_{g}\); and the noisy subset gradient \(g^{(c)}_{noise}(\mathbf{w}_{0})\) is sharply opposite to \(g^{(c)}_{clean}(\mathbf{w}_{0})\). Figure 4 experimentally verifies this phenomenon. The red histograms, representing the \(\theta_{g}\) distribution for noisy subset, are symmetric to the green ones and locate at large angles. The red dash lines representing the angle \(\theta_{g}\) between \(g^{(c)}_{noise}(\mathbf{w}_{0})\) and \(g^{(c)}_{clean}(\mathbf{w}_{0})\) is almost close to \(180^{\circ}\).

Lastly, the subset \(\mathcal{D}^{(c)}_{other}\), having different ground truth labels with the other two subsets, has different input distributions. By Corollary 3.2, the sample-wise gradients \(\nabla l(\mathbf{w}_{0})\) of this subset are expected to be not align with those of the other two subsets (as shown by the blue histograms in Figure 4). Moreover, the subset gradient \(g^{(c)}_{other}(\mathbf{w}_{0})\) should have a significant component orthogonal to \(g^{(c)}_{noise}(\mathbf{w}_{0})\) and \(g^{(c)}_{clean}(\mathbf{w}_{0})\) (as shown by the blue dash lines in Figure 4).

**Magnitudes of subset gradients.** We are interested in the magnitudes of \(g^{(c)}_{clean}(\mathbf{w}_{0})\) and \(g^{(c)}_{noise}(\mathbf{w}_{0})\). By definition, for \(k\in\{clean,\ noise\}\),

\[g^{(c)}_{k}(\mathbf{w}_{0})=|D^{(c)}_{k}|\mathbb{E}[\nabla l]=|D^{(c)}_{k}| \mathbb{E}[f(\mathbf{w}_{0};\mathbf{x})-y]\mathbb{E}[\nabla h]\]

where the expectation is taken over the corresponding data subset. We know that \(\mathbb{E}[\nabla h]\) is the same for clean and noisy subsets. In addition, \(\mathbb{E}[f(\mathbf{w}_{0};\mathbf{x})-y]\) are opposite for these two subsets, as \(\mathbb{E}[f(\mathbf{w}_{0};\mathbf{x})]\) is \(0.5\) by random guess and \(y=1\) for one subset and \(y=0\) for the other. Hence, we see that the magnitudes \(\|g^{(c)}_{k}\|\) are determined by the subset population, and we have

\[\|g^{(c)}_{clean}(\mathbf{w}_{0})\|/\|g^{(c)}_{noise}(\mathbf{w}_{0})\|=(1- \delta)/\delta>1. \tag{8}\]

## 4 Learning dynamics of binary classification

In this section, we analyze the learning dynamics of binary classification with label noise in the training dataset.

Specifically, we show that in the early stage of training, the dynamics exhibits a clean-priority learning characteristic, due to a dominance of the clean subset in first-order information, i.e., sample-wise gradients. We further show that in later stage of training, this dominance fades away and clean-priority learning terminates, resulting in a fitting of the noisy samples and worsening of the test performance.

We partition the optimization procedure into two stages: early stage which happens before the early stopping point; and later stage which is after the early stopping point.

### Initialization & early stage

In Section 3, we have seen that, at initialization,

\[g^{(c)}_{noise}(\mathbf{w}_{0})=-\alpha_{0}g^{(c)}_{clean}(\mathbf{w}_{0}), \tag{9}\]

with \(\alpha_{0}\triangleq\delta/(1-\delta)\in(0,1)\). We note that, during training, the model derivative \(\nabla h\) for a wide neural network is found

Figure 4: The distributions of \(\theta_{g}\). **Left**: synthetic data in Figure 3 (\(\delta=0.3\)), **Right**: two classes MNIST ((“0” and 1”, \(\delta=0.3\)). Dash lines represent subset gradients. Both cases use a 2-layer ReLU neural network.
to barely change Liu et al. (2020):

\[\nabla h(\mathbf{w}_{t})=\nabla h(\mathbf{w}_{0}),\;\forall t>0.\]

By Eq.(5), this implies that each sample-wise gradient \(\nabla l_{i}\) keeps its direction unchanged during training (but changes in magnitude through the factor \(f(\mathbf{w};\mathbf{x})-y\)). Therefore, we make the following assumption:

**Assumption 4.1**.: There exist a time \(T>0\) and a sequence \(\{\alpha_{t}\}_{t=0}^{T}\), with each \(\alpha_{t}\in(0,1)\), such that, for all \(t\in[0,T]\) and \(c\in\{0,1\}\), the following holds \(g_{noise}^{(c)}(\mathbf{w}_{t})=-\alpha_{t}g_{clean}^{(c)}(\mathbf{w}_{t})\).

Define \(\hat{g}^{(c)}(\mathbf{w})\) as the summation of the sample-wise gradients with ground truth labels, i.e., \(\hat{g}^{(c)}(\mathbf{w})=\sum_{(\mathbf{x},\hat{y})\in\hat{\mathcal{D}}^{(c) }}\nabla l(\mathbf{w};\mathbf{x},\hat{y})\). By the assumption, we have for all \(0\leq t\leq T\) and \(c\in\{0,1\}\),

\[g_{clean}^{(c)}(\mathbf{w}_{t})=\frac{1}{\alpha_{t}+1}\hat{g}^{(c)}(\mathbf{ w}_{t}). \tag{10}\]

On the other hand, by definition, we have for full gradient

\[\nabla L(\mathbf{w}_{t};\mathcal{D})=\sum_{c}\left(g_{clean}^{(c)}(\mathbf{w} _{t})+g_{noise}^{(c)}(\mathbf{w}_{t})\right). \tag{11}\]

Combining Assumption 4.1 and Eqs.(10) and (11), we get that the full gradient on the training data \(\mathcal{D}\) has the same direction with that on the ground-truth-labeled data \(\hat{\mathcal{D}}\). Hence, we have the following proposition.

**Proposition 4.2** (Update rules).: _Suppose Assumption 4.1 holds with time \(T>0\) and sequence \(\{\alpha_{t}\}_{t=0}^{T}\in(0,1)^{T}\). Then, the gradient descent (with learning rate \(\eta\)), Eq.(4), has the following equivalent update rule_

\[\mathbf{w}_{t+1}=\mathbf{w}_{t}-\eta_{t}^{\prime}\nabla L(\mathbf{w}_{t};\hat {\mathcal{D}}),\;\;\mathrm{for}\;t\leq T, \tag{12}\]

_with \(\eta_{t}^{\prime}=\frac{1-\alpha_{t}}{1+\alpha_{t}}\eta>0\) and \(\nabla L(\mathbf{w}_{t};\hat{\mathcal{D}})\) being the gradient evaluated on the ground-truth-labeled dataset \(\hat{\mathcal{D}}\)._

**Remark 4.3** (mini-batch scenario).: In mini-batch SGD, similar relation of Eq.(12) also holds for a mini-batch estimation \(\nabla L(\mathbf{w}_{0};\mathcal{B})\), as long as the sampling of the mini-batch is independent of the label noise and the batch size \(|\mathcal{B}|\) is not too small such that the majority of samples are clean in the batches. Hence, in the following, we do not explicitly write out the dependence on the mini-batches.

The theorem states that, after adding label noise to the training dataset, the gradient descent update is equivalent to the one without label noise (except a different learning rate \(\eta_{t}^{\prime}<\eta\)). In another word, the gradient descent does not essentially "see" the noisy data and its update direction is determined only by the clean samples.

Clean-priority learning.This theorem implies the following learning characteristics of what we call _clean-priority learning_, as we described below.

_Training loss and accuracy on subsets._ The loss \(L(\mathbf{w};\mathcal{D}_{clean})\) on the clean subset keeps decreasing, while the loss \(L(\mathbf{w};\mathcal{D}_{noise})\) on the noisy subset is increasing, as formally stated in the following Theorem (see the proof in Appendix A.4):

**Theorem 4.4**.: _Suppose Assumption 4.1 holds with time \(T>0\) and sequence \(\{\alpha_{t}\}_{t=0}^{T}\), \(\alpha_{t}\in(0,1)\). We have, for all \(t\in[0,T]\) and sufficiently small \(\eta\),_

\[L(\mathbf{w}_{t+1};\mathcal{D}_{clean}) <L(\mathbf{w}_{t};\mathcal{D}_{clean});\] \[L(\mathbf{w}_{t+1};\mathcal{D}_{noise}) >L(\mathbf{w}_{t};\mathcal{D}_{noise}).\]

Accordingly, the training accuracy on the clean subset is increased, and that on the noisy subset is decreased.

_Residual magnitude:_\(|f(\mathbf{w};\mathbf{x})-y|\). As a consequence of the decreasing clean subset loss \(L(\mathbf{w};\mathcal{D}_{clean})\), the clean training samples are learned, in the sense that the network output \(f(\mathbf{w};\mathbf{x})\) moves towards its corresponding label \(y\), i.e., \(|f(\mathbf{w};\mathbf{x})-y|\) decreases on the clean subset. On the

Figure 5: Learning dynamics on two classes (“7” and “9”) of MNIST (noise level \(\delta=0.4\)) with FCN. **Left**: in the early stage (before the vertical dash line), clean subset error decreases, while noisy subset error increases. **Middle**: In the early stage, the clean subset average residual \(\mathbb{E}_{(\mathbf{x},y)\in\mathcal{D}_{clean}}[|f(\mathbf{w};\mathbf{x})-y|]\) decreases, i.e., on average the network outputs of clean subset move towards the labels, indicating a “learning” on the clean subset. One the other hand, the noisy subset average residual, \(\mathbb{E}_{(\mathbf{x},\hat{y})\in\mathcal{D}_{noise}}[|f(\mathbf{w}; \mathbf{x})-y|]\), monotonically increases, indicating that the noisy subset is not-learned. **Right**: total test error and total training error.
other hand, the increase of the \(L(\mathbf{w};\mathcal{D}_{noise})\) results in that, on the noisy subset, the network output \(f(\mathbf{w};\mathbf{x})\) moves away from its corresponding label \(y\), but towards its ground truth label \(\hat{y}\). Namely, the noisy subset is not learnt.

_Test loss._ As the test dataset \(\bar{\mathcal{D}}\) is not label-corrupted and is drawn from the same data distribution as \(\hat{\mathcal{D}}\), it is expected that the update rule in Eq.(12) decreases the test loss \(L(\mathbf{w};\bar{\mathcal{D}})\).

Figure 5 shows the clean-priority learning phenomenon on a binary classification of two classes of MNIST. The relevant part is the early stage, i.e., before the early stopping point (left of the vertical dash line). As one can see, in this stage, the prediction error and noisy subset loss \(L(\mathbf{w};\mathcal{D}_{noise})\) keep increasing (See Appendix B for subset loss curves). Especially, the prediction error increases from a random guess (error \(=0.5\)) at initialization towards \(100\%\). Meanwhile, the clean subset loss and prediction error keep decreasing. Moreover, the average residual magnitude \(|f(\mathbf{w};\mathbf{x})-y|\) decreases on the clean subset, but increases on the noisy subset, implying that only clean subset is learnt. These behaviors illustrate that in the early stage the learning dynamics prioritize the clean samples.

In short, in the early stage, the _clean-priority learning_ prioritizes the learning on clean training samples. The interesting point is that, although it seems impossible to distinguish the clean from the noisy directly from the data, this prioritization is possible because the model have access to the first-order information, i.e., sample-wise gradients. Importantly, it is this awareness of the clean samples and this prioritization in the early stage that allow the possibility of achieving test performances better than the noisy level.

### Early stopping point & later stage

As we have seen in the above subsection, the dominance of the magnitude \(\|g_{clean}^{(c)}\|\) over \(\|g_{noise}^{(c)}\|\) is one of the key reasons to maintain the clean-priority learning in the early stage. However, we shall see shortly that this dominance diminish as the training goes on, resulting in a final termination of the clean-priority learning.

Diminishing dominance of the clean gradient.Recall that the sample-wise gradient is proportional to the magnitude of the residual: \(\nabla l(\mathbf{w})\propto y-f(\mathbf{w};\mathbf{x})\). The learning of a sample, i.e., decreased \(|y-f(\mathbf{w};\mathbf{x})|\), results in a decrease in the magnitude \(|\nabla l(\mathbf{w})|\). As an effect of the clean-priority learning, the residuals magnitude \(|f(\mathbf{w};\mathbf{x})-y|\) evolves differently for different data subsets: _decreases_ on the clean subset \(\mathcal{D}_{clean}\), but _increases_ on the noisy subset \(\mathcal{D}_{noise}\). This difference leads to the diminishment of the dominance of clean subset \(\|g_{clean}^{(c)}(\mathbf{w})\|\), which originates from the dominance of the population of clean training samples.

**Theorem 4.5** (Diminishing dominance of the clean gradient).: _Assume the neural network is infinitely wide and the learning rate \(\eta\) of the gradient descent is sufficiently small. Suppose Assumption 4.1 holds with time \(T>0\) and sequence \(\{\alpha_{t}\}_{t=0}^{T}\in(0,1)^{T}\). The sequence \(\{\alpha_{t}\}_{t=0}^{T}\) monotonically increases: for all \(t\in[0,T]\), \(\alpha_{t+1}>\alpha_{t}\)._

Please find the proof in Appendix A.5. As \(\alpha_{t}\) measures this clean dominance (\(\alpha_{t}\) close to \(1\) means less dominant), this theorem indicates that the dominance diminishes as the training goes on.

Figure 6 illustrates this diminishing dominance on the two class MNIST classification problem. In the early stage, the ratio \(\|g_{clean}^{(c)}(\mathbf{w})\|/\|g_{noise}^{(c)}(\mathbf{w})\|\) starts with a value around the ratio of population \((1-\delta)/\delta=1.5\), and monotonically decrease to around \(1\) at or before the early stopping point, indicating that the dominance vanishes.

Learning the noisy samples.In the later stage (i.e., after the early stopping point), the magnitudes of \(\|g_{clean}^{(c)}(\mathbf{w})\|\) and \(\|g_{noise}^{(c)}(\mathbf{w})\|\) are similar, and there is no apparent dominance of one over the other. Then, the model and algorithm do not distinguish the clean subset from the noisy one, and there will be no clean-priority learning. In this stage, the model learns both the clean and noisy subsets, aiming at achieving exact fitting of the training data. Ultimately, training errors of both subsets converge to zero.

It is expected that in this stage the loss and prediction error on the test dataset \(\bar{\mathcal{D}}\) become worse, as the learning on the noisy subset contaminates the performance achieved by the clean-priority learning in the earlier stage.

As illustrated in Figure 5, after the early stopping point, the noisy subset starts to be learnt. Specifically, both training loss and error on this subset turn to decrease towards zero; the average residual magnitude \(|f(\mathbf{w};\mathbf{x})-y|\) turn to de

Figure 6: Diminishing dominance of clean gradient. The ratio \(\|g_{clean}^{(c)}(\mathbf{w})\|/\|g_{noise}^{(c)}(\mathbf{w})\|\) monotonically decreases in the early stage (before the vertical dash line), as a consequence of the clean-priority learning, as predicted by Theorem 4.5. Experiment setting is the same as in Figure 5.
crease, indicating that the network output \(f(\mathbf{w};\mathbf{x})\) is learnt to move towards its (corrupted) label. It is worth to note that the learning on the clean subset is still ongoing, as both training loss and error on this subset keeps decreasing.

In high level, before the first stage, the learning procedure prioritizes the clean training samples, allowing the superior-noise-level performance on the test dataset; in later stage, the learning procedure picks up the noisy samples, worsening the test performance toward the noise-level.

## 5 Multi-class classification

In this section we show that multi-class classification problems exhibit the same learning dynamics, especially the clean-priority learning, as described in Section 4.

For multi-class classification, we consider a variant of the sample-wise gradient, _single-logit sample-wise gradient_.

Single-logit sample-wise gradients.In a \(C\)-class classification problem, the neural network \(f\) has \(C\) output logits, and the labels are a \(C\)-dimensional one-hot encoded vectors. One can view the neural network as \(C\) co-existing binary classifiers. Specifically, for each \(c\in\{1,2,\cdots,C\}\), the \(c\)-th logit \(f_{c}\) is a binary classifier, and the \(c\)-th component of the label \(y_{c}\in\{0,1\}\) is the binary label for \(f_{c}\).

By Eq.(5), the sample-wise gradient can be written as \(\nabla l(\mathbf{w})=\sum_{c=1}^{C}\nabla l_{c}(\mathbf{w}),\) where

\[\nabla l_{c}(\mathbf{w})\triangleq(f_{c}(\mathbf{w};\mathbf{x})-y_{c})\nabla h _{c}(\mathbf{w};\mathbf{x}) \tag{13}\]

is the _single-logit sample-wise gradient_, which only depends on quantities of the corresponding single logit.

We point out that, the cleanness of a sample is only well defined with respect to each single logit, but not to the whole output. For example, consider a sample with ground truth label \(0\) but is incorrectly labeled as class \(1\). For all the rest binary classifiers, except the \(0\)-th and \(1\)-st, this sample is always considered as the negative class, as \(y_{c}=0\) for all \(c\neq 0\) or \(1\); hence, the noisy sample is considered "clean", for these \(C-2\) binary classifiers. Therefore, a noisy sample is not necessarily noisy for all the \(C\) binary classifiers.

With this observation, we consider the single-logit sample-wise gradient \(\nabla l_{c}(\mathbf{w})\) instead.

At initialization.Given \(c\in\{1,2,\cdots,C\}\), the \(c\)-logit sub-network \(h_{c}\) (before _softmax_) is the same as the network \(h\) discussed in Section 3, and the output \(f_{c}\in(0,1)\). Hence, all the directional analysis for binary case (Section 3) still applies to the single-logit sample-wise gradient \(\nabla l_{c}(\mathbf{w})\). See Appendix C for numerical verification.

Different from the _sigmoid_ output activation which tends to predict an average of \(0.5\) before training, the _softmax_ has an average output \(f_{c}\) around \(1/C\) with random guess at initialization. This leads to \(\mathbb{E}[f_{c}(\mathbf{w}_{0};\mathbf{x})-y_{c}]=1-1/C\) when \(y_{c}=1\), and \(\mathbb{E}[f_{c}(\mathbf{w}_{0};\mathbf{x})-y_{c}]=1/C\) when \(y_{c}=0\). Recalling that \(\mathcal{D}_{clean}^{(c)}\) and \(\mathcal{D}_{noise}^{(c)}\) (hence the corresponding \(\nabla h\)) have the same distribution, using Eq.(13) we have

\[g_{noise}^{(c)}(\mathbf{w}_{0}) \approx-\delta\hat{g}^{(c)}(\mathbf{w}_{0})/(C-1), \tag{14a}\] \[g_{clean}^{(c)}(\mathbf{w}_{0}) \approx(1-\delta)\hat{g}^{(c)}(\mathbf{w}_{0}). \tag{14b}\]

Therefore, we have the dominance of \(\|g_{clean}^{(c)}\|\) over \(\|g_{noise}^{(c)}\|\) at initialization, with a ratio

\[\|g_{clean}^{(c)}(\mathbf{w}_{0})\|/\|g_{clean}^{(c)}(\mathbf{w}_{0})\| \approx(C-1)(1-\delta)/\delta.\]

Figure 7: Learning dynamics on multi-class classification. **Left**: in the early stage (before the vertical dash line), clean subset error decreases, while noisy subset error increases. **Middle**: In the early stage, the clean subset average residual \(\mathbb{E}_{(\mathbf{x},\mathbf{y})\in\mathcal{D}_{clean}}[\|f(\mathbf{w}; \mathbf{x})-y\|]\) decreases, i.e., on average the network outputs of clean subset move towards the labels, indicating a “learning” on the clean subset. One the other hand, the noisy subset average residual, \(\mathbb{E}_{(\mathbf{x},\mathbf{y})\in\mathcal{D}_{noise}}[\|f(\mathbf{w}; \mathbf{x})-y\|]\), monotonically increases, indicating that the noisy subset is not-learned. **Right**: total test error and total training error. See subset loss curves in Appendix C.
Learning dynamics.As the configuration of \(\nabla l_{c}\) is similar to that of a binary classification, we expect similar learning dynamics as discussed in Section 4, especially the clean-priority learning, happen for multi-class classification.

We conduct experiments to classify the MNIST (with added label noise \(\delta=0.3\)) and CIFAR-10 (with added label noise \(\delta=0.4\)) datasets using a CNN and a ResNet, respectively. As is shown in Figure 8 and Figure 7, in most of the early stage, the clean subset has clean dominance over the noise subset and the dynamics shows the clean-priority learning characteristic, decreasing the clean subset error and residual, but increasing the noisy subset error and residual. Furthermore, the dominance of the clean subset monotonically decreases (Figure 8) until the early stopping point. In the later stage, the networks start to learn the noisy subsets. See the detailed experimental setup in Appendix B.

## 6 Discussion and Future work

In this section, we aim to provide some insights into the relationship between the clean-priority phenomena described in this paper and previous works, as well as how this phenomena may manifest in other gradient descent-based learning algorithms.

Firstly, previous studies observed that, for certain very large models, the test classification error may exhibit a second descent in the later stages of training (Nakkiran et al., 2021). In such scenarios, our analysis still holds for the first descent and the subsequent ascent that follows it. Regarding the second descent, we hypothesize that it may be connected to certain types of underlying feature learning dynamics, for example, the Expected Gradient Outer Product (EGOP) (Radhakrishnan et al., 2022). However, further investigation is needed to confirm this hypothesis, and we leave it as a topic for future research.

Secondly, as can be seen in Figure 1, 5, and 7, This occurs sometime after the early stopping point. It suggests that while the model is fitting the noise, the test performance at convergence is not catastrophic, and still outperforms the random guess. This, in turn, implies that neural networks demonstrate a tempered over-fitting behavior, as described in (Mallinar et al., 2022).

#### Acknowledgments

We are grateful for the support from the National Science Foundation (NSF) and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning ([https://deepfoundations.ai/](https://deepfoundations.ai/)) through awards DMS-2031883 and #814639 and the TI-LOS institute (NSF CCF-2112665). This work used NVIDIA V100 GPUs NVLINK and HDR IB (Expanse GPU) at SDSC Dell Cluster through allocation TG-CIS220009 and also, Delta system at the National Center for Supercomputing Applications through allocation bbj-delta-gpu from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
