Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%. Recent advances in deep unsupervised learning, such as generative adversarial networks (GANs) BID8 , have led to an explosion of interest in semi-supervised learning. Semisupervised methods make use of both unlabeled and labeled training data to improve performance over purely supervised methods. Semi-supervised learning is particularly valuable in applications such as medical imaging, where labeled data may be scarce and expensive BID23 .Currently the best semi-supervised results are obtained by consistency-enforcing approaches BID2 BID17 BID31 BID21 BID24 . These methods use unlabeled data to stabilize their predictions under input or weight perturbations. Consistency-enforcing methods can be used at scale with state-of-the-art architectures. For example, the recent Mean Teacher BID31 model has been used with the Shake-Shake BID7 architecture and has achieved the best semi-supervised performance on the consequential CIFAR benchmarks.This paper is about conceptually understanding and improving consistency-based semi-supervised learning methods. Our approach can be used as a guide for exploring how loss geometry interacts with training procedures in general. We provide several novel observations about the training objective and optimization trajectories of the popular ⇧ BID17 and Mean Teacher BID31 consistency-based models. Inspired by these findings , we propose to improve SGD solutions via stochastic weight averaging (SWA) BID12 , a recent method that averages weights of the networks corresponding to different training epochs to obtain a single model with improved generalization. On a thorough empirical study we show that this procedure achieves the best known semi-supervised results on consequential benchmarks. In particular:• We show in Section 3.1 that a simplified ⇧ model implicitly regularizes the norm of the Jacobian of the network outputs with respect to both its inputs and its weights, which in turn encourages flatter solutions. Both the reduced Jacobian norm and flatness of solutions have been related to generalization in the literature BID29 BID22 BID3 BID27 BID13 BID12 . Interpolating between the weights corresponding to different epochs of training we demonstrate that the solutions of ⇧ and Mean Teacher models are indeed flatter along these directions ( FIG0 ).• In Section 3.2, we compare the training trajectories of the ⇧, Mean Teacher, and supervised models and find that the distances between the weights corresponding to different epochs are much larger for the consistency based models. The error curves of consistency models are also wider ( FIG0 ), which can be explained by the flatness of the solutions discussed in section 3.1. Further we observe that the predictions of the SGD iterates can differ significantly between different iterations of SGD.• We observe that for consistency-based methods , SGD does not converge to a single point but continues to explore many solutions with high distances apart. Inspired by this observation, we propose to average the weights corresponding to SGD iterates, or ensemble the predictions of the models corresponding to these weights. Averaging weights of SGD iterates compensates for larger steps, stabilizes SGD trajectories and obtains a solution that is centered in a flat region of the loss (as a function of weights). Further, we show that the SGD iterates correspond to models with diverse predictions -using weight averaging or ensembling allows us to make use of the improved diversity and obtain a better solution compared to the SGD iterates. In Section 3.3 we demonstrate that both ensembling predictions and averaging weights of the networks corresponding to different training epochs significantly improve generalization performance and find that the improvement is much larger for the ⇧ and Mean Teacher models compared to supervised training. We find that averaging weights provides similar or improved accuracy compared to ensembling, while offering the computational benefits and convenience of working with a single model. Thus, we focus on weight averaging for the remainder of the paper.• Motivated by our observations in Section 3 we propose to apply Stochastic Weight Averaging (SWA) BID12 to the ⇧ and Mean Teacher models. Based on our results in Section 3.3 we propose several modifications to SWA in Section 4. In particular, we propose fast-SWA, which (1) uses a learning rate schedule with longer cycles to increase the distance between the weights that are averaged and the diversity of the corresponding predictions; and (2) averages weights of multiple networks within each cycle (while SWA only averages weights corresponding to the lowest values of the learning rate within each cycle). In Section 5, we show that fast-SWA converges to a good solution much faster than SWA.• Applying weight averaging to the ⇧ and Mean Teacher models we improve the best reported results on CIFAR-10 for 1k, 2k, 4k and 10k labeled examples, as well as on CIFAR-100 with 10k labeled examples. For example, we obtain 5.0% error on CIFAR-10 with only 4k labels, improving the best result reported in the literature BID31 ) by 1.3%. We also apply weight averaging to a state-of-the-art domain adaptation technique BID6 closely related to the Mean Teacher model and improve the best reported results on domain adaptation from CIFAR-10 to STL from 19.9% to 16.8% error.• We release our code at https://github.com/benathi/fastswa-semi-sup 2 BACKGROUND
Consistency-based models for semi-supervised learning do not converge to a single point but continue to explore a diverse set of plausible solutions on the perimeter of a flat region. Weight averaging helps improve generalization performance. The paper proposes to apply Stochastic Weight Averaging to the semi-supervised learning context, arguing that the semi-supervised MT/Pi models are especially amenable to SWA and propose fast SWA to speed up training.