Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks to generate additional outputs. One of the systems includes a neural network and a sequence processing subsystem, wherein the sequence processing subsystem is configured to perform operations comprising, for each of the system inputs in a sequence of system inputs: receiving the system input; generating an initial neural network input from the system input; causing the neural network to process the initial neural network input to generate an initial neural network output for the system input; and determining, from a first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. An augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising: a neural network, wherein the neural network is configured to receive neural network inputs and process each of the neural network inputs to generate a respective neural network output for each neural network input; and a sequence processing subsystem, wherein the sequence processing subsystem is configured to perform operations comprising, for each of the system inputs in the sequence of system inputs: receiving the system input; generating an initial neural network input from the system input; causing the neural network to process the initial neural network input to generate an initial neural network output for the system input; and determining, from a first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input. 2. The augmented neural network system of claim 1, the operations further comprising: in response to determining not to cause the neural network to generate one or more additional neural network outputs for the system input, generating a system output for the system input from a second portion of the initial neural network output for the system input. 3. The augmented neural network system of claim 1, the operations further comprising: in response to determining to cause the neural network to generate one or more additional neural network outputs for the system input, causing the neural network to generate the one or more additional neural network outputs for the system input, wherein causing the neural network to generate each of the one or more additional neural network outputs for the system input comprises: generating an additional neural network input; causing the neural network to process the additional neural network input to generate the additional neural network output; and determining, from a first portion of the additional neural network output, whether or not to cause the neural network to generate a further additional neural network output. 4. The augmented neural network system of claim 3, the operations further comprising: in response to determining not to cause the neural network to generate a further additional neural network output, generating a system output for the system input from a second portion of a last additional neural network output generated by the neural network for the system input. 5. The augmented neural network system of claim 3, the operations further comprising: in response to determining not to cause the neural network to generate a further additional neural network output, generating a system output for the system input from second portions of the initial neural network output and the one or more additional neural network outputs. 6. The augmented neural network system of claim 5, wherein generating the system output comprises: determining a respective weight for the initial neural network output and for each of the additional neural network outputs from the first portions of the corresponding initial neural network output and the additional neural network outputs; and combining the second portions of the initial neural network output and the additional neural network outputs in accordance with the respective weights. 7. The augmented neural network system of claim 3, wherein generating the additional neural network input comprises generating the additional neural network input from the system input. 8. The augmented neural network system of claim 3, wherein generating the additional neural network input comprises generating a default neural network input. 9. The augmented neural network system of claim 1, wherein determining, from the first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input comprises: determining a probability from the first portion of the initial neural network output; and determining whether or not to cause the neural network to generate the one or more additional neural network outputs based on the probability. 10. The augmented neural network system of claim 1, wherein determining, from the first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input comprises: determining whether or not the first portion of the initial neural network output exceeds a threshold value; and determining not to cause the neural network to generate the one or more additional neural network outputs when the first portion of the initial neural network output exceeds the threshold value. 11. A method for processing a sequence of system inputs to generate a sequence of system outputs, the method comprising, for each of the system inputs: receiving the system input; generating an initial neural network input from the system input; causing a neural network to process the initial neural network input to generate an initial neural network output for the system input, wherein the neural network is configured to receive neural network inputs and process each of the neural network inputs to generate a respective neural network output for each neural network input; and determining, from a first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input. 12. The method of claim 11, further comprising: in response to determining not to cause the neural network to generate one or more additional neural network outputs for the system input, generating a system output for the system input from a second portion of the initial neural network output for the system input. 13. The method of claim 11, further comprising: in response to determining to cause the neural network to generate one or more additional neural network outputs for the system input, causing the neural network to generate the one or more additional neural network outputs for the system input, wherein causing the neural network to generate each of the one or more additional neural network outputs for the system input comprises: generating an additional neural network input; causing the neural network to process the additional neural network input to generate the additional neural network output; and determining, from a first portion of the additional neural network output, whether or not to cause the neural network to generate a further additional neural network output. 14. The method of claim 13, further comprising: in response to determining not to cause the neural network to generate a further additional neural network output, generating a system output for the system input from a second portion of a last additional neural network output generated by the neural network for the system input. 15. The method of claim 13, further comprising: in response to determining not to cause the neural network to generate a further additional neural network output, generating a system output for the system input from second portions of the initial neural network output and the one or more additional neural network outputs. 16. The method of claim 15, wherein generating the system output comprises: determining a respective weight for the initial neural network output and for each of the additional neural network outputs from the first portions of the corresponding initial neural network output and the additional neural network outputs; and combining the second portions of the initial neural network output and the additional neural network outputs in accordance with the respective weights. 17. The method of claim 13, wherein generating the additional neural network input comprises generating the additional neural network input from the system input. 18. The method of claim 11, wherein determining, from the first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input comprises: determining a probability from the first portion of the initial neural network output; and determining whether or not to cause the neural network to generate the one or more additional neural network outputs based on the probability. 19. The method of claim 11, wherein determining, from the first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input comprises: determining whether or not the first portion of the initial neural network output exceeds a threshold value; and determining not to cause the neural network to generate the one or more additional neural network outputs when the first portion of the initial neural network output exceeds the threshold value. 20. A computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations implement an augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising: a neural network, wherein the neural network is configured to receive neural network inputs and process each of the neural network inputs to generate a respective neural network output for each neural network input; and a sequence processing subsystem, wherein the sequence processing subsystem is configured to perform operations comprising, for each of the system inputs in the sequence of system inputs: receiving the system input; generating an initial neural network input from the system input; causing the neural network to process the initial neural network input to generate an initial neural network output for the system input; and determining, from a first portion of the initial neural network output for the system input, whether or not to cause the neural network to generate one or more additional neural network outputs for the system input.