We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL. Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements. We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning. Current machine learning algorithms require enormous amounts of training data to learn new tasks. This is a problem for many practical problems across domains such as biology and medicine where labeled data is hard to come by. In contrast, we humans can quickly learn new concepts from limited training data. We are able to do this by relying on our past "visual experience". Recent work attempts to emulate this by training a feature representation to classify a training dataset of "base" classes with the hope that the resulting representation generalizes not just to unseen examples of the same classes but also to novel classes, which may have very few training examples (called fewshot learning). However, training for base class classification can force the network to only encode features that are useful for distinguishing between base classes. In the process, it might discard semantic information that is irrelevant for base classes but critical for novel classes. This might be especially true when the base dataset is small or when the class distinctions are challenging. One way to recover this useful semantic information is to leverage representation learning techniques that do not use class labels, namely, unsupervised or self-supervised learning. The key idea is to learn about statistical regularities (e.g., spatial relationship between patches, orientation of an images) that might be a cue to semantics. Despite recent advances, these techniques have only been applied to a few domains (e.g., entry-level classes on internet imagery), and under the assumption that large amounts of unlabeled images are available. Their applicability to the general few-shot scenario described above is unclear. In particular, can these techniques help prevent overfitting to base classes and improve performance on novel classes in the few-shot setting? If so, does the benefit generalize across domains and to more challenging tasks? Moreover, can we use self-supervised training to boost performance in domains where even unlabeled images are hard to get? This paper seeks to answer these questions. We show that with no additional training data, adding a self-supervised task as an auxiliary task (Figure 1 ) improves the performance of existing few-shot techniques on benchmarks across a multitude of domains ( Figure 2 ). Intriguingly, we find that the benefits of self-supervision increase with the difficulty of the task, for example when training from a smaller base dataset, or with degraded inputs such as low resolution or greyscale images (Figure 3 ). One might surmise that as with traditional SSL, additional unlabeled images might improve performance further. But what unlabeled images should we use for novel problem domains where unlabeled data is not freely available? To answer this, we conduct a series of experiments with additional unlabeled data from different domains. We find that adding more unlabeled images improves performance only when the images used for self-supervision are within the same domain as the base classes ( Figure 4a ); otherwise they can even negatively impact the performance of the few-shot learner (Figure 4b, 4c, 4d) . Based on this analysis we present a simple approach that uses a domain classifier to pick similar-domain unlabeled images for self-supervision from a large, generic pool of images. The resulting method improves over the performance of a model trained with self-supervised learning from images within the domain (Figure 4c ). Taken together, this results in a powerful, general and practical approach for improving few-shot learning from small datasets in novel domains. Finally, these benefits are also observed on standard classification tasks (Appendix A.3). We have shown that self-supervision improves transferability of representations on few-shot learning tasks across a range of different domains. Surprisingly, we found that self-supervision is more beneficial for more challenging problems, especially when the number of images used for selfsupervision is small, orders of magnitude smaller than previously reported results. This has a practical benefit that the images within small datasets can be used for self-supervision without relying on a large-scale external dataset. We have also shown that additional unlabeled images can improve performance only if they are from the same or similar domains. Finally, for domains where unlabeled data is limited, we present a novel, simple approach to automatically identify such similar-domain images from a larger pool. Table 1 : Example images and dataset statistics. For few-shot learning experiments the classes are split into base, val, and novel set. Image representations learned on base set are evaluated on the novel set while val set is used for cross-validation. These datasets vary in the number of classes but are orders of magnitude smaller than ImageNet dataset. A.2 RESULTS ON FEW-SHOT LEARNING Table 2 shows the performance of ProtoNet with different self-supervision on seven datasets. We also test the accuracy of the model on novel classes when trained only with self-supervision on the base set of images. Compared to the randomly initialized model ("None" rows), training the network to predict rotations gives around 2% to 21% improvements on all datasets, while solving jigsaw puzzles only improves on aircrafts and flowers. However, these numbers are significantly worse than learning with supervised labels on the base set, in line with the current literature. Table 3 shows the performance of ProtoNet with jigsaw puzzle loss on harder benchmarks. The results on degraded version of the datasets are shown in the top part, and the bottom part shows the results of using only 20% of the images in the base categories. The gains using SSL are higher in this setting. Table 4 : Performance on few-shot transfer task using different meta-learners. Using jigsaw puzzle loss gives improvements across different meta-learners on most of the datasets. ProtoNet with jigsaw loss performs the best on all five datasets.
Self-supervision improves few-shot recognition on small and challenging datasets without relying on extra data; Extra data helps only when it is from the same or similar domain. An empirical study of different self-supervised learning (SSL) methods, showing SSL helps more when the dataset is harder, that domain matters for training, and a method to choose samples from an unlabeled dataset. 