Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4–8.2 percent points under existence of real-world noise. By virtue of massive labeled data, deep neural networks (DNNs) have achieved a remarkable success in numerous machine learning tasks, such as image classification (Krizhevsky et al., 2012) and object detection (Redmon et al., 2016) . However, owing to their high capacity to memorize any label noise, the generalization performance of DNNs drastically falls down when noisy labels are contained in the training data (Jiang et al., 2018; Han et al., 2018; Song et al., 2019) . In particular, Zhang et al. (2017) have shown that a standard convolutional neural network (CNN) can easily fit the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data. Thus, it is challenging to train a DNN robustly even when noisy labels exist in the training data. A popular approach to dealing with noisy labels is "sample selection" that selects true-labeled samples from the noisy training data (Jiang et al., 2018; Ren et al., 2018; Han et al., 2018; Yu et al., 2019; Song et al., 2019) . Here, (1−τ )×100% of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly, where τ ∈ [0, 1] is a noise rate. This loss-based separation is well known to be justified by the memorization effect (Arpit et al., 2017) that DNNs tend to learn easy patterns first and then gradually memorize all samples. In practice, Han et al. (2018) empirically proved that training on such small-loss samples yields a much better generalization performance under artificial noise scenarios. Despite its great success, Song et al. (2019) have recently argued that the performance of the lossbased separation becomes considerably worse depending on the type of label noise. For instance, Figure 1: Loss distributions at the training accuracy of 50% using DenseNet (L=40, k=12): (a) and (b) show those on CIFAR-100 with two types of synthetic noises of 40%, where "symmetric noise" flips a true label into other labels with equal probability, and "pair noise" flips a true label into a specific false label; (c) shows those on FOOD-101N (Lee et al., 2018) with real-world noise of 18.4%. ) show how many true-labeled and false-labeled samples are memorized when training DenseNet (L=40, k=12) 1 on CIFAR-100 with two types of synthetic noises of 40%. "Default" is a standard training method, and "Prestopping" is our proposed one; (c) contrasts the convergence of test error between the two methods. in symmetric noise (Figure 1(a ) ), the loss-based approach well separates true-labeled samples from false-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones. On the other hand, in pair and real-world noises (Figures 1(b) and 1(c)) , many false-labeled samples are misclassified as true-labeled ones because the two distributions overlap closely; this overlap confirms that the loss-based separation still accumulates severe label noise from many misclassified cases, especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise (Ren et al., 2018; Yu et al., 2019) . This limitation definitely calls for a new approach that supports any type of label noise for practical use. In this regard, as shown in Figure 2 (a), we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows: • A noise type affects the memorization rate for false-labeled samples: The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise. That is, the red portion in Figure 2 (a) starts to appear earlier in pair noise than in symmetric noise. This observation supports the significant overlap of true-labeled and false-labeled samples in Figure 1(b ) . Thus , the loss-based separation performs well only if the false-labeled samples are scarcely learned at an early stage of training, as in symmetric noise. • There is a period where the network accumulates the label noise severely: Regardless of the noise type, the memorization of false-labeled samples significantly increases at a late stage of training. That is, the red portion in Figure 2 (a) increases rapidly after the dashed line, in which we call the error-prone period. (See Section 3.2.1 for the details of estimating the error-prone period). We note that the training in that period brings no benefit. The generalization performance of "Default" deteriorates sharply, as shown in Figure 2 (c). Based on these findings, we contend that eliminating this error-prone period should make a profound impact on robust optimization. In this paper, we propose a novel approach, called Prestopping, that achieves noise-free training based on the early stopping mechanism. Because there is no benefit from the error-prone period, Prestopping early stops training before that period begins. This early stopping effectively prevents a network from overfitting to false-labeled samples, and the samples memorized until that point are added to a maximal safe set because they are true-labeled (i.e., blue in Figure 2 (a)) with high precision. Then, Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training. Notably, our proposed merger of "early stopping" and "learning from the maximal safe set" indeed eliminates the error-prone period from the training process, as shown in Figure 2 (b). As a result, the generalization performance of a DNN remarkably improves in both noise types, as shown in Figure 2 (c). To validate the superiority of Prestopping, DenseNet (Huang et al., 2017) and VGG-19 (Simonyan & Zisserman, 2015) were trained on both simulated and real-world noisy data sets, including CIFAR-10, CIFAR-100, ANIMAL-10N, and Food-101N. Compared with four state-of-the-art methods, Prestopping significantly improved test error by up to 18.1pp 2 in a wide range of noise rates. In this paper, we proposed a novel two-phase training strategy for the noisy training data, which we call Prestopping. The first phase, "early stopping," retrieves an initial set of true-labeled samples as many as possible, and the second phase, "learning from a maximal safe set," completes the rest training process only using the true-labeled samples with high precision. Prestopping can be easily applied to many real-world cases because it additionally requires only either a small clean validation set or a noise rate. Furthermore, we combined this novel strategy with sample refurbishment to develop Prestopping+. Through extensive experiments using various real-world and simulated noisy data sets, we verified that either Prestopping or Prestopping+ achieved the lowest test error among the seven compared methods, thus significantly improving the robustness to diverse types of label noise. Overall, we believe that our work of dividing the training process into two phases by early stopping is a new direction for robust training and can trigger a lot of subsequent studies. A Prestopping WITH NOISE-RATE HEURISTIC Algorithm 2 describes the overall procedure of Prestopping with the noise-rate heuristic, which is also self-explanatory. Compared with Algorithm 1, only the way of determining the best stop point in Lines 7-9 has changed. Algorithm 2 Prestopping with Noise-Rate Heuristic INPUT:D: data, epochs: total number of epochs, q: history length, τ : noise rate OUTPUT: θ t : network parameters 1: t ← 1; θ t ← Initialize the network parameter; 2: θ tstop ← ∅; /* The parameter of the stopped network */ 3: for i = 1 to epochs do /* Phase I: Learning from a noisy training data set */ 4: Draw a mini-batch B t fromD; 6: θ tstop ← θ t ; break; 10: t ← t + 1; 11: θ t ← θ tstop ; /* Load the network stopped at t stop */ 12: for i = stop_epoch to epochs do /* Phase II: Learning from a maximal safe set */ 13: Draw a mini-batch B t fromD;
We propose a novel two-phase training approach based on "early stopping" for robust training on noisy labels. Paper proposes to study how early stopping in optimization helps find confident examples This paper proposes a two-phase training method for learning with label noise.