In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). 
 In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.
 The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning. A crucial property of a useful model is to generalize, that is to perform well on test settings given learning on a training setting. While what is most commonly meant by generalization is being robust to having a limited number of training examples in distributionally-matched settings , many tasks are designed to address variations in the data between when a model is trained and when it is evaluated. For instance, some classification tasks address distributional changes in the input: from lacking guarantees of distributional match between train and test (e.g., covariate shift, Shimodaira, 2000) to having fundamental domain differences (e.g., domain adaptation, Ben-David et al., 2007) . A number of tasks have also been designed specifically to understand models in terms of their ability to generalize to test situations that are poorly represented during training (e.g., Few-Show learning, Li et al., 2006) , or even consist of a diverse and entirely novel set of sub-tasks (Zamir et al., 2018) . For supervised classification, Zero-Shot Learning (ZSL, Larochelle et al., 2008) is among the most difficult of these tasks, as it requires the model to make useful inferences about (e.g., correctly label) unseen concepts, given parameters learned only from seen training concepts and additional high-level semantic information. The fundamental question we wish to address in this work is: What are the principles that contribute to learning good representations for ZSL? While the most successful ZSL models (Atzmon & Chechik, 2019; Wang et al., 2019) use pretrained features from Imagenet (Krizhevsky et al., 2012; Russakovsky et al., 2015) , we wish to understand how these features can emerge given only the data provided from the ZSL task. Specifically, we explore the role of compositionality and locality (Tokmakov et al., 2018; Stone et al., 2017) as two principles that lead to good generalization. Our study focuses on image representations, so we explore various means of learning representations that are local and compositional for convolutional neural networks (CNNs). We also leverage the structure of CNNs and available annotations from ZSL datasets as a means of interpreting various models in terms of these factors. Overall, our results support the hypothesis that compositionality and locality are crucial principles for training models that generalize well. Finally, in order to provide a cleaner framework for understanding the relationship between the above principles and generalization, we re-introduce Zero-Shot Learning from scratch (ZFS). In this setting, the model is not allowed to be pretrained on another dataset, such as Imagenet, and is evaluated on its ability to perform classification using auxiliary attributes and labels trained only using the data available from the training split of the target dataset. We believe that ZFS will provide researchers with a better experimental framework to understand which principles are important for Zero-Shot generalization. The contributions of our work are as follows: • We introduce Zero-Shot Learning from scratch (ZFS), an extension to ZSL, which we believe will be an important benchmark for understanding which learning principles lead to better generalization. • We evaluate several supervised and unsupervised methods on their ability to learn features that generalize in the ZFS setting by training a prototypical network on top of those features (in a similar way to what was done in Snell et al., 2017 , with Imagenet features). We then relate this generalization performance with different proxies for locality and compositionality of the given representations, and show that both concepts contribute heavily. • We introduce a novel version of Deep InfoMax (DIM, Hjelm et al., 2018) which draws local patch representations from other images with the same label as positive samples. • We introduce a novel visualization technique based on Mutual Information, that allows to investigate local properties of the learned representations. In this section, we describe in more detail our experiments and analyize the results. The full numerical results and plots for all considered models can be found in the Appendix. Motivated by the need for more realistic evaluation settings for Zero-Shot Learning methods, we proposed a new evaluation framework where training is strictly performed only on the benchmark data, with no pre-training on additional datasets. In the proposed setting, we hypothesize that locality and compositionality are fundamental ingredients for successful zero-shot generalization. We perform a series of tests of the relationship between these two aspects and zero-shot performance of a diverse set of representations. We find that models that encourage both these aspects, either explicitly (through a penalty per instance) or implicitly by construction, tend to perform better at zero-shot learning. We also find that models that focus on reconstruction tasks fail at capturing the semantic information necessary for good generalization, calling into question their applicability as general representation learning methods.
An analysis of the effects of compositionality and locality on representation learning for zero-shot learning. Proposes evaluation framework for ZSL where the model is not allowed to be pretrained and instead, model parameters are randomly initialized for better understanding of what's happening in ZSL.