Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using a neural network system that includes a whitened neural network layer. One of the methods includes receiving an input activation generated by a layer before the whitened neural network layer in the sequence; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A neural network system implemented by one or more computers and configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input, the sequence of neural network layers comprising: a whitened neural network layer, wherein the neural network layer is configured to perform operations comprising: receiving an input activation generated by a layer before the whitened neural network layer in the sequence; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence. 2. The neural network system of claim 1, wherein the set of whitening parameters includes a mean parameter vector and a whitening weight matrix, and wherein processing the received activation in accordance with the set of whitening parameters to generate the whitened activation comprises: subtracting the mean parameter vector from the input activation to generate an intermediate whitened activation; and applying the whitening weight matrix to the intermediate whitened activation to generate the whitened activation. 3. The neural network system of claim 2, wherein the whitening weight matrix is a matrix having elements that are derived from an eigen decomposition of a matrix of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence during training of the neural network layers. 4. The neural network system of claim 2, wherein the mean parameter vector is derived from a mean of input activations generated by the layer before the whitened neural network layer in the sequence during training of the neural network layers. 5. The neural network system of claim 1, wherein the layer parameters comprise a bias vector d and a layer weight matrix V and the output activation h satisfies: h=f(Vhw+d), wherein f is an element-wise non-linear activation function and hw is the whitened activation. 6. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a neural network system that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input, the sequence of neural network layers comprising: a whitened neural network layer, wherein the neural network layer is configured to perform operations comprising: receiving an input activation generated by a layer before the whitened neural network layer in the sequence; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence. 7. The computer storage media of claim 6, wherein the set of whitening parameters includes a mean parameter vector and a whitening weight matrix, and wherein processing the received activation in accordance with the set of whitening parameters to generate the whitened activation comprises: subtracting the mean parameter vector from the input activation to generate an intermediate whitened activation; and applying the whitening weight matrix to the intermediate whitened activation to generate the whitened activation. 8. The computer storage media of claim 7, wherein the whitening weight matrix is a matrix having elements that are derived from an eigen decomposition of a matrix of a covariance of input activations generated by the layer before the whitened neural network layer in the sequence during training of the neural network layers. 9. The computer storage media of claim 7, wherein the mean parameter vector is derived from a mean of input activations generated by the layer before the whitened neural network layer in the sequence during training of the neural network layers. 10. The computer storage media of claim 6, wherein the layer parameters comprise a bias vector d and a layer weight matrix V and the output activation h satisfies: h=f(Vhw+d), wherein f is an element-wise non-linear activation function and hw is the whitened activation. 11. A method of training a neural network that is configured to receive a network input and to process the network input through each of a sequence of neural network layers to generate a network output from the network input, wherein the sequence of neural network layers comprises a whitened neural network layer, wherein the neural network layer is configured to receive an input activation generated by a layer before the whitened neural network layer in the sequence, process the received activation in accordance with a set of whitening parameters to generate a whitened activation, process the whitened activation in accordance with a set of layer parameters to generate an output activation, and provide the output activation as input to a neural network layer after the whitened neural network layer in the sequence, and wherein the method comprises: training the neural network on training data to adjust values of the set of layer parameters of the whitened neural network layer using a gradient descent technique; and during the training, adjusting the values of the whitening parameters using statistics of input activations generated by the layer before the whitened neural network layer in the sequence. 12. The method of claim 11, further comprising: in response to adjusting the values of the whitening parameters using statistics of input activations generated by the layer before the whitened neural network layer in the sequence, further adjusting the values of the set of layer parameters so that output activations generated by the whitened neural network layer remain unchanged. 13. The method of claim 11, wherein adjusting the values of the whitening parameters comprises, during the training: adjusting the values of the whitening weight matrix based on an estimated covariance of input activations generated by the layer before the whitened neural network layer; and adjusting the values of the mean parameter vector based on an estimated mean of input activations generated by the layer before the whitened neural network layer. 14. The method of claim 13, wherein adjusting the values of the whitening weight matrix based on the estimated covariance comprises adjusting the values of the whitening weight matrix based on an eigen decomposition of a matrix of the estimated covariance of input activations generated by the layer before the whitened neural network layer. 15. The method of claim 11, wherein adjusting the values of the set of layer parameters of the whitened neural network layer using the gradient descent technique comprises treating the values of the whitening parameters as constants when updating the values of the set of layers parameters using the gradient descent technique. 16. The method of claim 11, wherein the layer parameters comprise a bias vector d and a layer weight matrix V and the output activation h satisfies: h=f(Vhw+d), wherein f is an element-wise non-linear activation function and hw is the whitened activation. 17. The method of claim 11, wherein adjusting the values of the whitening parameters comprises: determining that T gradient descent updates to the values of the layer parameters have occurred since a preceding update to the values of the whitening parameters.