Title: Invariance Makes a Difference: Disentangling the Role of Invariance and Equivariance in Representations. Abstract: Representations learned by deep neural networks are the foundation that enables their tremendous success and consequently a lot of work has been invested into understanding their properties. Most of this work, however, focuses on the relationships between representations and features in the input without explicitly characterizing their nature, i.e. whether they are invariances or equivariances. In this work, we concretely define and disentangle these relationships and show with carefully controlled experiments that, in fact, invariance is of central importance in achieving high generalization on downstream tasks, often more so than equivariance. To this end, we investigate the properties and performance of image classification models on synthetic datasets that we introduce and which allow us to precisely control factors of variation in the models' training and test data. With this method we explore a) the role of invariance in enabling high performance when transferring to target tasks and b) the factors that influence which invariances a model learns. We highlight the importance of representational invariance by showing that the representations learned by classification models transfer well to new classes but perform poorly when the required invariances change, and that learning the wrong invariances can be harmful. Additionally, we find that the invariances learned by models are primarily determined by the relationship of features in the training data with the training objective and that there are inductive biases that make certain invariances more difficult to learn than others.