Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method comprising: obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that follows a last word in the sequence of words in the new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent. 2. The method of claim 1, wherein the trained neural network system comprises an embedding layer configured to map the input document and each word in the sequence of words from the input document to respective vector representations, a combining layer configured to combine the vector representations into a combined representation, and a classifier layer configured to generate the word scores using the combined representation. 3. The method of claim 2, wherein the embedding layer maps the words in the sequence of words to vector representations in accordance with a first set of parameters, and wherein the classifier layer generates the word scores from the combined representation in accordance with a second set of parameters. 4. The method of claim 3, wherein the values of the first set of parameters and the values of the second set of parameters are fixed from training the neural network system to generate the word scores. 5. The method of claim 2, wherein determining the vector representation for the new document using the trained neural network system comprises performing a respective iteration of gradient descent for each of the plurality of sequences of words to adjust the vector representation of the new document from a previous iteration of gradient descent. 6. The method of claim 5, wherein the performing the respective iteration of gradient descent for each of the plurality of sequences comprises: mapping each of the words in the sequence to a vector representation using the embedding layer; combining the vector representation for the words in the sequence with the vector representation for the new document from the previous iteration to generate a combined representation; generating word scores from the combined representation; computing a gradient using the word scores and the word that follows the sequence in the new document; and adjusting the vector representation for the new document from the previous iteration using the gradient. 7. The method of claim 2, wherein the combining layer is configured to concatenate the vector representations of the words in the sequence with the vector representation of the input document. 8. The method of claim 2, wherein the combining layer is configured to average the vector representations of the words in the sequence and the vector representation of the input document. 9. The method of claim 1, wherein each of the plurality of sequences of words contains a fixed number of words. 10. A method comprising: obtaining a plurality of training documents, wherein each document in the plurality of training documents includes a plurality of training sequences of words, and, for each sequence of words, a word that follows a last word in the training sequence of words in the training document; and training a neural network system on each of the training documents using gradient descent and backpropagation, wherein the neural network system is configured to receive data identifying an input document and an input sequence of words from the input document and to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence of words in the input document, and wherein training the neural network system on each of the training documents comprises, for each training document, performing a respective iteration of gradient descent for each sequence of words in the training document. 11. The method of claim 10, wherein the neural network system comprises an embedding layer configured to map the input document and each word in the sequence of words from the input document to respective vector representations, a combining layer configured to combine the vector representations into a combined representation, and a classifier layer configured to generate the word scores using the combined representation. 12. The method of claim 11, wherein the embedding layer maps the words in the sequence of words to vector representations in accordance with a first set of parameters, and wherein the classifier layer generates the word scores from the combined representation in accordance with a second set of parameters. 13. The method of claim 12, wherein performing the respective iteration of gradient descent for each of the plurality of sequences in the training document comprises: mapping each of the words in the sequence to a vector representation using the embedding layer; mapping the data identifying the training document to a vector representation using the embedding layer; combining the vector representation for the words in the sequence with the vector representation for the training document from the previous iteration to generate a combined representation; generating word scores from the combined representation; computing a gradient using the word scores and the word that follows the sequence in the new document; and adjusting values of the second set of parameters using the gradient. 14. The method of claim 13, wherein performing the respective iteration of gradient descent further comprises adjusting values of the first set of parameters using backpropagation. 15. The method of claim 10, wherein the combining layer is configured to concatenate the vector representations of the words in the sequence with the vector representation of the input document. 16. The method of claim 10, wherein the combining layer is configured to average the vector representations of the words in the sequence and the vector representation of the input document. 17. The method of claim 10, wherein each of the plurality of sequences of words contains a fixed number of words. 18. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that follows a last word in the sequence of words in the new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent. 19. The system of claim 18, wherein the trained neural network system comprises an embedding layer configured to map the input document and each word in the sequence of words from the input document to respective vector representations, a combining layer configured to combine the vector representations into a combined representation, and a classifier layer configured to generate the word scores using the combined representation. 20. The system of claim 19, wherein determining the vector representation for the new document using the trained neural network system comprises performing a respective iteration of gradient descent for each of the plurality of sequences of words to adjust the vector representation of the new document from a previous iteration of gradient descent.