Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using normalized target outputs. One of the methods includes updating current values of the normalization parameters to account for the target output for the training item; determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values; processing the training item using the neural network to generate a normalized output for the training item in accordance with current values of main parameters of the neural network; determining an error for the training item using the normalized target output and the normalized output; and using the error to adjust the current values of the main parameters of the neural network.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for training a neural network on training data to generate normalized outputs that are mappable to un-normalized outputs in accordance with a set of normalization parameters, wherein the training data comprises a sequence of training items and, for each training item in the sequence, a respective target output, the method comprising, for each training item in the sequence: updating current values of the normalization parameters to account for the target output for the training item; determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values; processing the training item using the neural network to generate a normalized output for the training item in accordance with current values of main parameters of the neural network; determining an error for the training item using the normalized target output and the normalized output; and using the error to adjust the current values of the main parameters of the neural network. 2. The method of claim 1, wherein the normalization parameters comprise a shift parameter and a scale parameter of the normalization. 3. The method of claim 2, wherein determining the normalized target output comprises applying the updated values of the scale parameter and the shift parameter to the target output. 4. The method of claim 1, wherein updating the current normalization parameter values to account for the target output for the training item comprises updating the current normalization parameter values so that the normalized target outputs for the training items up to and including the training item in the sequence have a specified distribution. 5. The method of claim 1, wherein the normalized outputs generated by the neural network are mappable to un-normalized outputs in accordance with the normalization parameters and a set of auxiliary parameters. 6. The method of claim 5, further comprising: updating current values of the auxiliary parameters to preserve the mapping between the un-normalized outputs and normalized outputs despite the updating of the current values of the normalization parameters. 7. The method of claim 6, wherein updating the current values of the auxiliary parameters to preserve the mapping between the un-normalized outputs and normalized outputs despite the updating of the current values of the normalization parameters comprises updating the current values of the auxiliary parameters to cancel out the effect of updating the current values of the normalization parameters on the mapping. 8. The method of claim 6, wherein determining the error comprises: processing the training input in accordance with the current values of the main parameters to generate an initial output; and normalizing the initial output in accordance with the updated values of the auxiliary parameters. 9. The method of claim 8, wherein using the error to adjust the current values of the parameters of the neural network comprises: performing an iteration of a neural network training technique to adjust the current values of the main parameters of the neural network. 10. The method of claim 9, further comprising: adjusting the updated values of the auxiliary parameters as part of performing the iteration of the neural network training technique. 11. The method of claim 9, wherein the neural network training technique is stochastic gradient descent (SGD). 12. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network on training data to generate normalized outputs that are mappable to un-normalized outputs in accordance with a set of normalization parameters, wherein the training data comprises a sequence of training items and, for each training item in the sequence, a respective target output, the operations comprising, for each training item in the sequence: updating current values of the normalization parameters to account for the target output for the training item; determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values; processing the training item using the neural network to generate a normalized output for the training item in accordance with current values of main parameters of the neural network; determining an error for the training item using the normalized target output and the normalized output; and using the error to adjust the current values of the main parameters of the neural network. 13. The system of claim 12, wherein the normalization parameters comprise a shift parameter and a scale parameter of the normalization. 14. The system of claim 13, wherein determining the normalized target output comprises applying the updated values of the scale parameter and the shift parameter to the target output. 15. The system of claim 12, wherein updating the current normalization parameter values to account for the target output for the training item comprises updating the current normalization parameter values so that the normalized target outputs for the training items up to and including the training item in the sequence have a specified distribution. 16. The system of claim 12, wherein the normalized outputs generated by the neural network are mappable to un-normalized outputs in accordance with the normalization parameters and a set of auxiliary parameters. 17. The system of claim 16, the operations further comprising: updating current values of the auxiliary parameters to preserve the mapping between the un-normalized outputs and normalized outputs despite the updating of the current values of the normalization parameters. 18. The system of claim 17, wherein updating the current values of the auxiliary parameters to preserve the mapping between the un-normalized outputs and normalized outputs despite the updating of the current values of the normalization parameters comprises updating the current values of the auxiliary parameters to cancel out the effect of updating the current values of the normalization parameters on the mapping. 19. The system of claim 17, wherein determining the error comprises: processing the training input in accordance with the current values of the main parameters to generate an initial output; and normalizing the initial output in accordance with the updated values of the auxiliary parameters. 20. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network on training data to generate normalized outputs that are mappable to un-normalized outputs in accordance with a set of normalization parameters, wherein the training data comprises a sequence of training items and, for each training item in the sequence, a respective target output, the operations comprising, for each training item in the sequence: updating current values of the normalization parameters to account for the target output for the training item; determining a normalized target output for the training item by normalizing the target output for the training item in accordance with the updated normalization parameter values; processing the training item using the neural network to generate a normalized output for the training item in accordance with current values of main parameters of the neural network; determining an error for the training item using the normalized target output and the normalized output; and using the error to adjust the current values of the main parameters of the neural network.