We introduce MultiGrain, a neural network architecture that generates compact image embedding vectors that solve multiple tasks of different granularity: class, instance, and copy recognition. MultiGrain is trained jointly for classification by optimizing the cross-entropy loss and for instance/copy recognition by optimizing a self-supervised ranking loss. The self-supervised loss only uses data augmentation and thus does not require additional labels. Remarkably, the unified embeddings are not only much more compact than using several specialized embeddings, but they also have the same or better accuracy. When fed to a linear classifier, MultiGrain using ResNet-50 achieves 79.4% top-1 accuracy on ImageNet, a +1.8% absolute improvement over the the current state-of-the-art AutoAugment method. The same embeddings perform on par with state-of-the-art instance retrieval with images of moderate resolution. An ablation study shows that our approach benefits from the self-supervision, the pooling method and the mini-batches with repeated augmentations of the same image.
 Image recognition is central to computer vision, with dozens of new approaches being proposed every year, each optimized for particular aspects of the problem. From coarse to fine, we may distinguish the recognition of (a) classes, where one looks for a certain type of object regardless of intra-class variations, (b) instances, where one looks for a particular object despite changes in the viewing conditions, and (c) copies, where one looks for a copy of a specific image despite edits. While these problems are in many ways similar, the standard practice is to use specialized, and thus incompatible, image representations for each case. Consider for example image retrieval, where the goal is to match a query image to a large database of other images, whose applications include detection of copyrighted images and exemplar-based recognition of unseen objects. Often one would like to search the same collection with multiple granularities, by matching the query by class, instance, or copy. Adopting multiple image embeddings, narrowly optimized for each granularity, means multiplying the resource usage. Using a single embedding relevant to all these tasks reduces both the computing time and the storage space. However, this might come at the cost of a reduced accuracy. In this paper we introduce MultiGrain, a compact embedding that, as illustrated in fig. 1 , can solve recognition tasks of different granularities while maintaining or surpassing the accuracy of specialized embeddings. MultiGrain is obtained by training a Convolutional Neural Network (CNN) jointly on the different tasks. CNNs trained for image classification are known to be good universal features extractors. However, authors (Babenko & Lempitsky, 2015) have noted that the intermediate layers of such CNNs are generally better for low-level tasks such as instance and copy recognition. In contrast, our work extracts a single global embedding at the top of the network. The key is to optimize this embedding simultaneously for classification and instance retrieval. In this manner, the same representation integrates different degrees of invariance. Indeed, by definition, copies of the same image contain the same instance, and images that contain the same instance also contain the same class. Figure 1 : Top: Our goal is to extract an image descriptor incorporating different levels of granularity, so that we can solve, classification and particular object recognition tasks: The descriptor is either fed to a linear classifier, or directly compared with cosine similarity. Right: The MultiGrain architecture. As an additional contribution, we show that MultiGrain can be learned using only class-level labels via self-supervised learning (Caron et al., 2018) . The instance recognition is learned for free, without labels specific to instance recognition: we use the identity of arbitrary images as labels, and data augmentation to generate different versions of each image. We also find that, unexpectedly, forming batches with multiple augmentations of the same image, improves the classifier performance, even for models trained only for classification. This contradicts the common knowledge that training batches should maximize diversity. Finally, we incorporate in MultiGrain a pooling layer inspired by image retrieval that boosts the classification accuracy for high-resolution images. Overall, MultiGrain offers compelling performance both for classification and image retrieval, including outperforming the SoTA classification accuracy on ImageNet for ResNet-50. MultiGrain is a unified embedding for image classification and instance retrieval. It relies on a classical CNN trunk, with a GeM pooling layer, topped with two heads at training time. We have discovered that this pooling layer allows us to increase the resolution of images used at inference time, while maintaining a small resolution at training time. We have shown that MultiGrain embeddings can perform well on classification and retrieval. Interestingly, MultiGrain also sets a new state of the art on pure classification compared to all results obtained with the same convolutional trunk. Our approach will be open-sourced. We report a few details and additional experiments that did not fit in the main paper. Appendix A outlines the repeated augmentation sampling algorithm. Appendix B illustrates the effect of GeM pooling on activation maps. Appendix C studies the effect of the loss weighting parameter. Appendix D shows the effect of data-augmented batches when training a simple toy model. Appendix E lists the values of a few hyper-parameters used in our method. Appendix F gives a some more ablation results in the retrieval setting. Finally, Appendix G shows how to use the ingredients of MultiGrain to improve the accuracy of an off-the-shelf pre-trained ConvNet at almost no additional training cost. It obtains what appear to be the best reported classification results on imagenet-2012 for a convnet with publicly available weights.
Combining classification and image retrieval in a neural network architecture, we obtain an improvement for both tasks. This paper proposes a unified embedding for image classification and instance retrieval to enhance the performance for both tasks. The paper proposes to jointy train a deep neural net for image classification, instance, and copy recognition.