This paper proposes a new model for document embedding. Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize. We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model. This allows us to train deeper architectures that are fully parallelizable. Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document. Empirically we demonstrate superior results on two publicly available benchmarks. Full code will be released with the final version of this paper. A typical approach is to develop a document embedding model which produces fixed length vector representations that preserve relevant semantic information. These models are trained in unsupervised fashion on unlabeled text, and the resulting embeddings can be used as input for a variety of NLP tasks such as sentiment analysis and information retrieval BID2 BID21 BID19 . Despite significant research effort in this area the most commonly used methods are still based on the bag-of-words (n-grams) representations.However, recent work has shown that remarkably accurate embedding models can be learned using distributed representations of words BID27 . Within this category two popular approaches are doc2vec BID21 and skip-thought BID19 . doc2vec extends the distributed word model word2vec BID27 by attaching document-specific vectors to word2vec and learning them jointly with word representations. While accurate, this model requires iterative optimization to be conducted for each new document making it challenging to deploy in high volume production environments. Furthermore, doc2vec is trained using localized contexts of very small size (typically 5 to 10 words) and never sees the whole document. This makes it difficult to capture long range semantic relationships within the document.Skip-thought uses a recurrent neural network (RNN) to sequentially ingest the document one word at a time. Last layer activations after the last word are then taken as document embedding. RNN models have been gaining popularity and a number of other approaches have been proposed BID10 BID22 . Recurrent architecture addresses both problems of the doc2vec approach. During inference only a forward pass through the network is required to produce an embedding that is based on the entire content of the document. However, the sequential nature of the RNN makes it difficult to leverage the full benefits of modern hardware such as GPUs that offer highly scalable parallel execution. This can significantly slow down both training and inference. Consequently most RNN models including skip-thought are relatively shallow with only a few hidden layers. Moreover, many of the commonly used RNN achitectures such as LSTM BID11 and GRU BID3 , gate information form already seen input at each recurrence step. Repeated gating has an effect where more weight is placed on latter words and the network can "forget" earlier parts of the document BID20 . This is not ideal for document embedding where long range relationships that can occur anywhere in the document need to modeled.In this work we propose an embedding model that addresses the aforementioned problems. We show that there is a direct connection between language and embedding models. We then use recent advances in language modeling to derive a convolutional neural network (CNN) embedding model. Similarly to skip-thought, inference in our model is done via a forward pass through the CNN. However, the CNN architecture allows to process the entire document in parallel significantly accelerating both learning and inference. We show that the variable length input problem can be effectively dealt with using either padding or global pooling in the last convolutional layer. Moreover significant gains can be achieved using deeper architectures where each successive layer captures increasingly longer range dependencies in the document. We presented a CNN model for document embedding. In this approach successive layers of convolutions are applied to distributed word representations to model increasingly longer range semantic relationships within the document. We further proposed a stochastic forward prediction learning algorithm where the model is trained to predict the successive words for randomly chosen subsequences within the document. This learning procedure has few hyper parameters to tune and is straightforward to implement. Our model is able to take full advantage of parallel execution, and achieves better performance while also being significantly faster than current state-of-the-art RNN models.
Convolutional neural network model for unsupervised document embedding. Introduces a new model for the general task of inducing document representations (embeddings) which uses a CNN architecture to improve computational efficiency. This paper proposes using CNNs with a skip-gram like objective as a fast way to output document embeddings