We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the model parameters with respect to a task-specific loss given an input sample. Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network. We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight. Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients. Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures. We demonstrate strong results in all settings. And our results are well-aligned with our theoretical insight. Despite tremendous success of deep models, training deep neural networks requires a massive amount of labeled data and computing resources. The recent development of representation learning holds great promises for improving data efficiency of training, and enables an easy adaption to different tasks using the same feature representation. These features can be learned via either unsupervised learning using deep generative models (Kingma & Welling, 2013; Dumoulin et al., 2016) , or self-supervised learning with "pretext" tasks and pseudo labels (Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018) , or transfer learning from another large-scale dataset (Yosinski et al., 2014; Oquab et al., 2014; Girshick et al., 2014) . After learning, the activations of the deep network are considered as generic features. By leveraging these features, simple classifiers, e.g., linear models, can be build for different tasks. However, given sufficient amount of training data, the performance of representation learning methods lack behind fully-supervised deep models. As a step to bridge this gap, we propose to make use of gradient-based features from a pre-trained network, i.e., gradients of the model parameters relative to a task-specific loss given an input sample. Our key intuition is that these per-sample gradients contain task-relevant discriminative information. More importantly, we design a novel linear model that accounts for both gradient-based and activation-based features. The design of our linear model stems from the recent advances in the theoretical analysis of deep models. Specifically, our gradient-based features are inspired by the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019b ) modified for finite-width networks. Therefore, our model provides a local approximation of fine-tuning a underlying deep model, and the accuracy of the approximation is controlled by the semantic gap between the representation learning and the target tasks. Finally, the specific structure of the gradient-based features and the linear model allows us to derive an efficient and scalable algorithm for training the linear model with these features. To evaluate our method, we focus on visual representation learning in this paper, although our model can be easily modified for natural language processing or speech recognition. To this end, we consider a number of learning tasks in vision, including unsupervised, self-supervised and transfer learning. Our method was evaluated across tasks, datasets and architectures and compared against a set of baseline methods. We observe empirically that our model with gradient-based features outperforms the traditional activation-based features by a significant margin in all settings. Moreover, our results compare favorably against those produced by fine-tuning of network parameters. Our main contributions are thus summarized as follows. • We propose a novel representation learning method. At the core of our method lies in a linear model that builds on gradients of model parameters as the feature representation. • From a theoretical perspective, we show that our linear model provides a local approximation of fine-tuning an underlying deep model. From a practical perspective, we devise an efficient and scalable algorithm for the training and inference of our method. • We demonstrate strong results of our method across various representation learning tasks, different network architectures and several datasets. Furthermore, these empirical results are well-aligned with our theoretical insight. In this paper, we presented a novel method for deep representation learning. Specifically, given a pre-trained deep model, we explored the per-sample gradients of the model parameters relative to a task-specific loss, and constructed a linear model that combines gradients of model parameters and the activation of the model. We showed that our model can be very efficient in training and inference, and provides a local linear approximation to an underlying deep model. Through a set of experiments, we demonstrated that these gradient-based features are highly discriminative for the target task, and our method can significantly improve over the baseline method of representation learning across tasks, datasets and network architectures. We believe that our work provides a step forward towards deep representation learning. We summarize our main conclusions from the ablation studies. 1. Pre-training is required for the representation power of the gradient feature. This holds for both the gradient model and the full model.
Given a pre-trained model, we explored the per-sample gradients of the model parameters relative to a task-specific loss, and constructed a linear model that combines gradients of model parameters and the activation of the model. This paper proposes to use the gradients of specific layers of convolutional networks as features in a linearized model for transfer learning and fast adaptation.