Title: Learning Useful Representations for Shifting Tasks and Distributions . Abstract: Representation learning in deep models usually happens as a side effect of minimizing the expected risk using back-propagation. However, one of the challenges of modern deep learning is the increasingly recognized need to deal with multiple tasks and varying data distributions, as illustrated, for instance, by the value of transfer learning and the risks of shortcut learning. Are the representations learned by back-propagation up to the task?
This work presents and empirically evaluates two methods that combine the feature extractors learned during multiple training episodes and construct a representation that is richer than those usually obtained through a single expected risk minimization episode. Comprehensive experiments in supervised transfer learning, self-supervised transfer learning, few-shot learning, and out-of-distribution robust learning scenarios, show that such rich representations can match and often exceed the performance of those obtained by training an equivalently sized network, with usually a far less computational burden.