Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A system comprising: a compressed recurrent neural network (RNN) implemented by one or more computers, the compressed RNN comprising a plurality of recurrent layers, wherein each of the plurality of recurrent layers of the RNN is configured to, for each of a plurality of time steps, receive a respective layer input for the time step and process the layer input for the time step to generate a respective layer output for the time step, wherein each of the plurality of recurrent layers has a respective recurrent weight matrix, Wh, and a respective inter-layer weight matrix, Wx, and wherein at least one of the plurality of recurrent layers is compressed such that a respective recurrent weight matrix, Whl, of the compressed layer is defined by a first compressed weight matrix, Zhl, and a projection matrix, Pl, and a respective inter-layer weight matrix, Wxl, of the compressed layer is defined by a second compressed weight matrix, Zxl, and the projection matrix, Pl. 2. The system of claim 1, wherein each of the plurality of recurrent layers is configured to, for each time step: generate the respective layer output for the time step by applying an inter-layer weight matrix for a previous layer to a current input to the layer and applying a recurrent weight matrix for the layer to a recurrent input to the layer. 3. The system of claim 1, wherein the at least one of the plurality of recurrent layers that is compressed is configured to, for each time step: generate the respective layer output for the time step in part by applying the first compressed weight matrix and the projection matrix to a respective recurrent input to the layer. 4. The system of claim 1, wherein the respective recurrent weight matrix is defined by the product of the first compressed weight matrix and the projection matrix and wherein the respective inter-layer weight matrix is defined by the product of the second compressed matrix and the projection matrix. 5. The system of claim 1, wherein the RNN is an acoustic model. 6. The system of claim 1, wherein the RNN is a speech recognition model. 7. The system of claim 1, wherein the RNN is at least a 68% compression of an uncompressed version of the RNN. 8. The system of claim 7, wherein a word error rate of the RNN is within 5% of a word error rate of an uncompressed version of the RNN. 9. A method for compressing a recurrent neural network (RNN), wherein the RNN is implemented by one or more computers and comprises a plurality of recurrent layers, wherein each of the plurality of recurrent layers of the RNN is configured to, for each of a plurality of time steps, receive a respective layer input for the time step and process the layer input to generate a respective layer output for the time step, and wherein each recurrent layer has a respective recurrent weight matrix, Wh, and a respective inter-layer weight matrix, Wx, the method comprising, for one of the plurality of recurrent layers: generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, wherein the product of the first compressed weight matrix and the projection matrix approximates the recurrent weight matrix of the one of the plurality of recurrent layers; and generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl, wherein the product of the second compressed weight matrix and the projection matrix approximates the inter-layer weight matrix of the one of the plurality of recurrent layers. 10. The method of claim 9, further comprising, for the one of the plurality of recurrent layers: replacing the respective recurrent weight matrix with the product of the first compressed weight matrix and the projection matrix; and replacing the respective inter-layer weight matrix with the product of the second compressed weight matrix and the projection matrix. 11. The method of claim 9, wherein the respective ranks Zh and P of are less than the rank of Wh. 12. The method of claim 9, wherein the respective ranks Zx and P of are less than the rank of Wx. 13. The method of claim 9, wherein generating the second compressed weight matrix, Zx, comprises inverting the projection matrix and multiplying the inverted projection matrix by the inter-layer weight matrix. 14. The method of claim 9, wherein singular value decomposition (SVD) compression techniques are used to generate the first compressed weight matrix, Zhl, and the projection matrix, P. 15. The method of claim 9, wherein the one of the plurality of layers is an l-th layer, and wherein the output for the l-th layer can be expressed by htl=σ(Wxl−1htl−1+ZhlPlht−1l+bl), wherein htl represents a hidden layer activation output of the l-th layer at time t, Wxl−1 represents an inter-layer weight matrix from a previous, (l−1)-th, layer bl represents an l-th layer bias vector, and σ(·) denotes a non-linear activation function. 16. The method of claim 9, wherein the RNN is a long short-term memory (LSTM) RNN. 17. The method of claim 9, further comprising training the RNN before generating the first compressed weight matrix, Zh, the projection matrix, P, and the second compressed weight matrix, Zx; and re-training the RNN after generating the first compressed weight matrix, Zh, the projection matrix, P, and the second compressed weight matrix, Zx. 18. The method of claim 9, wherein the RNN is an acoustic model. 19. The method of claim 9, wherein the RNN is a speech recognition model. 20. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations for compressing a recurrent neural network (RNN), wherein the RNN is implemented by one or more computers and comprises a plurality of recurrent layers, wherein each of the plurality of recurrent layers of the RNN is configured to, for each of a plurality of time steps, receive a respective layer input for the time step and process the layer input to generate a respective layer output for the time step, and wherein each recurrent layer has a respective recurrent weight matrix, Wh, and a respective inter-layer weight matrix, Wx, the operations comprising, for one of the plurality of recurrent layers: generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, wherein the product of the first compressed weight matrix and the projection matrix approximates the recurrent weight matrix of the one of the plurality of recurrent layers; and generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl, wherein the product of the second compressed weight matrix and the projection matrix approximates the inter-layer weight matrix of the one of the plurality of recurrent layers.