A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. 
 We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations. Transfer learning has driven a number of recent successes in computer vision and NLP. Computer vision tasks like image captioning BID52 and visual question answering typically use CNNs pretrained on ImageNet BID24 BID41 to extract representations of the image, while several natural language tasks such as reading comprehension and sequence labeling BID25 have benefited from pretrained word embeddings BID28 BID37 that are either fine-tuned for a specific task or held fixed.Many neural NLP systems are initialized with pretrained word embeddings but learn their representations of words in context from scratch, in a task-specific manner from supervised learning signals. However, learning these representations reliably from scratch is not always feasible, especially in low-resource settings, where we believe that using general purpose sentence representations will be beneficial.Some recent work has addressed this by learning general-purpose sentence representations BID49 BID17 BID11 BID27 BID20 BID33 BID34 . However, there exists no clear consensus yet on what training objective or methodology is best suited to this goal.Understanding the inductive biases of distinct neural models is important for guiding progress in representation learning. BID40 and BID4 demonstrate that neural machine translation (NMT) systems appear to capture morphology and some syntactic properties. BID40 also present evidence that sequence-to-sequence parsers more strongly encode source language syntax. Similarly, BID0 probe representations extracted by sequence autoencoders, word embedding averages, and skip-thought vectors with a multi-layer perceptron (MLP) classifier to study whether sentence characteristics such as length, word content and word order are encoded.To generalize across a diverse set of tasks, it is important to build representations that encode several aspects of a sentence. Neural approaches to tasks such as skip-thoughts, machine translation, natural language inference, and constituency parsing likely have different inductive biases. Our work exploits this in the context of a simple one-to-many multi-task learning (MTL) framework, wherein a single recurrent sentence encoder is shared across multiple tasks. We hypothesize that sentence representations learned by training on a reasonably large number of weakly related tasks will generalize better to novel tasks unseen during training, since this process encodes the inductive biases of multiple models. This hypothesis is based on the theoretical work of BID3 . While our work aims at learning fixed-length distributed sentence representations, it is not always practical to assume that the entire "meaning" of a sentence can be encoded into a fixed-length vector. We merely hope to capture some of its characteristics that could be of use in a variety of tasks.The primary contribution of our work is to combine the benefits of diverse sentence-representation learning objectives into a single multi-task framework. To the best of our knowledge, this is the first large-scale reusable sentence representation model obtained by combining a set of training objectives with the level of diversity explored here, i.e. multi-lingual NMT, natural language inference, constituency parsing and skip-thought vectors. We demonstrate through extensive experimentation that representations learned in this way lead to improved performance across a diverse set of novel tasks not used in the learning of our representations. Such representations facilitate low-resource learning as exhibited by significant improvements to model performance for new tasks in the low labelled data regime -achieving comparable performance to a few models trained from scratch using only 6% of the available training set on the Quora duplicate question dataset. In this section, we describe our approach to evaluate the quality of our learned representations, present the results of our evaluation and discuss our findings. We present a multi-task framework for learning general-purpose fixed-length sentence representations. Our primary motivation is to encapsulate the inductive biases of several diverse training signals used to learn sentence representations into a single model. Our multi-task framework includes a combination of sequence-to-sequence tasks such as multi-lingual NMT, constituency parsing and skip-thought vectors as well as a classification task -natural language inference. We demonstrate that the learned representations yield competitive or superior results to previous general-purpose sentence representation methods. We also observe that this approach produces good word embeddings. Table 5 : Evaluation of sentence representations by probing for certain sentence characteristics and syntactic properties. Sentence length, word content & word order from BID0 and sentence active/passive, tense and top level syntactic sequence (TSS) from BID40 . Numbers reported are the accuracy with which the models were able to predict certain characteristics.In future work, we would like understand and interpret the inductive biases that our model learns and observe how it changes with the addition of different tasks beyond just our simple analysis of sentence characteristics and syntax. Having a rich, continuous sentence representation space could allow the application of state-of-the-art generative models of images such as that of BID32 to language. One could also consider controllable text generation by directly manipulating the sentence representations and realizing it by decoding with a conditional language model.
A large-scale multi-task learning framework with diverse training objectives to learn fixed-length sentence representations This paper is about learning sentence embeddings by combining several training signals: skip-thought, predicting translation, classifying entailment relationships, and predicting the constituent parse.