Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures. The acquisition of large quantities of a high-quality human annotation is a frequent bottleneck in applying DNNs. There are two cheap but imperfect alternatives to collect annotation at large scale: crowdsourcing from non-experts and web annotations, particularly for image data where the tags and online query keywords are treated as valid labels. Both these alternatives typically introduce noisy (wrong) labels. While Rolnick et al. (2017) empirically demonstrated that DNNs can be surprisingly robust to label noise under certain conditions, Zhang et al. (2017) has shown that DNNs have the capacity to memorize the data and will do so eventually when being confronted with too many noisy labels. Consequently, training DNNs with traditional learning procedures on noisy data strongly deteriorates their ability to generalize -a severe problem. Hence, limiting the influence of label noise is of great practical importance. A common approach to mitigate the negative influence of noisy labels is to eliminate them from the training data and train deep learning models just with the clean labels (Frénay & Verleysen, 2013) . Employing semi-supervised learning can even counteract the noisy labels (Laine & Aila, 2016; Luo et al., 2018) . However, the decision which labels are noisy and which are not is decisive for learning robust models. Otherwise, unfiltered noisy labels still influence the (supervised) loss and affect the task performance as in these previous works. They use the entire label set to compute the loss and severely lack a mechanism to identify and filter out the erroneous labels from the labels set. In this paper, we propose a self-ensemble label filtering (SELF) framework that identifies potentially noisy labels during training and keeps the network from receiving supervision from the filtered noisy labels. This allows DNNs to gradually focus on learning from undoubtedly correct samples even with an extreme level of noise in the labels (e.g., 80% noise ratio) and leads to improved performance as the supervision become less noisy. The key contribution of our work is progressive filtering, i.e., leverage the knowledge provided in the network's output over different training iterations to form a consensus of predictions (self-ensemble predictions) to progressively identify and filter out the noisy labels from the labeled data. When learning under label noise, the network receives noisy updates and hence fluctuates strongly. Such conduct of training would impede to learn stable neural representations and further mislead the consensus of the predictions. Therefore, it is essential to incorporate a model with stable training behavior to obtain better estimates from the consensus. Concretely, we employ the semi-supervised technique as a backbone to our framework to stabilize the learning process of the model. Correctly, we maintain the running average model, such as proposed by Tarvainen & Valpola (2017) , a.k.a. the Mean-Teacher model. This model ensemble learning provides a more stable supervisory signal than the noisy model snapshots and provides a stable ground for progressive filtering to filter out potential noisy labels. Note that this is different from just a mere combination of semi-supervised techniques with a noisy label filtering method. We call our approach self-ensemble label filtering (SELF) -that establishes model ensemble learning as a backbone to form a solid consensus of the self-ensemble predictions to filter out the noisy labels progressively. Our framework allows to compute supervised loss on cleaner subsets rather than the entire noisy labeled data as in previous works. It further leverages the entire dataset, including the filtered out erroneous samples in the unsupervised loss. To best of our knowledge, we are the first to identify and propose self-ensemble as a principled technique against learning under noisy labels. Our motivation stems from the observation that DNNs start to learn from easy samples in initial phases and gradually adapt to hard ones during training. When trained on wrongly labeled data, DNNs learn from clean labels at ease and receive inconsistent error signals from the noisy labels before over-fitting to the dataset. The network's prediction is likely to be consistent on clean samples and inconsistent or oscillates strongly on wrongly labeled samples over different training iterations. Based on this observation, we record the outputs of a single network made on different training epochs and treat them as an ensemble of predictions obtained from different individual networks. We call these ensembles that are evolved from a single network self-ensemble predictions. Subsequently, we identify the correctly labeled samples via the agreement between the provided label set and our running average of self-ensemble predictions. The samples of ensemble predictions that agree with the provided labels are likely to be consistent and treated as clean samples. In summary, our SELF framework stabilizes the training process and improves the generalization ability of DNNs. We evaluate the proposed technique on image classification tasks using CI-FAR10, CIFAR100 & ImageNet. We demonstrate that SELF consistently outperforms the existing approaches on asymmetric and symmetric noise at all noise levels, as shown in Fig. 1 . Besides, SELF remains robust towards the choice of the network architecture. Our work is transferable to other tasks without the need to modify the architecture or the primary learning objective. Figure 2: Overview of the self-ensemble label filtering (SELF) framework. The model starts in iteration 0 with training from the noisy label set. During training, the model maintains a selfensemble, a running average of itself (Tarvainen & Valpola, 2017) to provide a stable learning signal. Also, the model collects a self-ensemble prediction (moving-average) for the subsequent filtering. Once the best model is found, these predictions identify and filter out noisy labels using the original label set L 0 . The model performs this progressive filtering until there is no more better model. For details see Algorithm 1. 2 SELF-ENSEMBLE LABEL FILTERING 2.1 OVERVIEW Fig. 2 shows an overview of our proposed approach. In the beginning, we assume that the labels of the training set are noisy. The model attempts to identify correct labels progressively using selfforming ensembles of models and predictions. Since wrong labels cause strong fluctuations in the model's predictions, using ensembles is a natural way to counteract noisy labels. Concretely, in each iteration, the model learns from a detected set of potentially correct labels and maintains a running average of model snapshots (realized by the Mean Teacher model Tarvainen & Valpola (2017) ). This ensemble model is evaluated on the entire dataset and provides an additional learning signal for training the single models. Additionally, our framework maintains the runningaverage of the model's predictions for the filtering process. The model is trained until we find the best model w.r.t. the performance on the validation set (e.g., by early-stopping). The set of correct labels is detected based on the strategy defined in Sec. 2.2. In the next iteration, we again use all data and the new filtered label set as input for the model training. The iterative training procedure stops when no better model can be found. In the following, we give more details about the combination of this training and filtering procedure. We propose a simple and easy to implement a framework to train robust deep learning models under incorrect or noisy labels. We filter out the training samples that are hard to learn (possibly noisy labeled samples) by leveraging ensemble of predictions of the single network's output over different training epochs. Subsequently, we allow clean supervision from the non-hard samples and further leverage additional unsupervised loss from the entire dataset. We show that our framework results in DNN models with superior generalization performance on CIFAR-10, CIFAR-100 & ImageNet and outperforms all previous works under symmetric (uniform) and asymmetric noises. Furthermore, our models remain robust despite the increasing noise ratio and change in network architectures.
We propose a self-ensemble framework to train more robust deep learning models under noisy labeled datasets. This paper proposed "self-ensemble label filtering" for learning with noisy labels where the label noise is instance-independent, which yield more accurate identification of inconsistent predictions.  This paper proposes an algorithm for learning from data with noisy labels which alternates between updating the model and removing samples that look like they have noisy labels.