Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing operations using data from a data source. In one aspect, a method includes a neural network system including a controller neural network configured to: receive a controller input for a time step and process the controller input and a representation of a system input to generate: an operation score distribution that assigns a respective operation score to an operation and a data score distribution that assigns a respective data score in the data source. The neural network system can also include an operation subsystem configured to: perform operations to generate operation outputs, wherein at least one of the operations is performed on data in the data source, and combine the operation outputs in accordance with the operation score distribution and the data score distribution to generate a time step output for the time step.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A neural network system for generating a system output from a system input by repeatedly performing operations from a set of operations using data from a data source, the neural network system comprising: a controller neural network configured to, for each of a plurality of time steps: receive a controller input for the time step; and process the controller input and a representation of the system input to generate: an operation score distribution that assigns a respective operation score to each operation in the set of operations, and a data score distribution that assigns a respective data score to each of a plurality of locations in the data source; and an operation subsystem configured to, for each of the plurality of time steps: perform operations from the set of operations to generate a plurality of operation outputs, wherein at least one of the operations is performed on data in the plurality of locations in the data source, and combine the operation outputs in accordance with the operation score distribution and the data score distribution to generate a time step output for the time step. 2. The neural network system of claim 1, wherein the operation subsystem is further configured to, for each of the plurality of time steps: store the time step output for the time step for use in any following time steps. 3. The neural network system of claim 1, wherein the system output is the time step output for the last time step in the plurality of time steps. 4. The neural network system of claim 1, wherein the controller input for the first time step in the plurality of time steps is a predetermined initial input, and wherein, for each subsequent time step, the operation subsystem is configured to: generate the controller input for the time step from the operation score distribution and the data score distribution for the preceding time step. 5. The neural network system of claim 4, wherein the controller input for the time step comprises a weighted sum of vectors representing locations in the data source and a weighted sum of vectors representing the operations in the set of operations. 6. The neural network system of claim 1, wherein the controller neural network comprises: a recurrent neural network configured to, for each time step, process the controller input for the time step to update a current hidden state of the recurrent neural network; one or more operation neural network layers configured to, for each time step, process the representation of the system input and the updated hidden state of the recurrent neural network to generate the operation score distribution; and one or more data neural network layers configured to, for each time step, process the representation of the system input and the updated hidden state of the recurrent neural network to generate the data score distribution. 7. The neural network system of claim 1, wherein the system input is a sequence of inputs, and wherein the neural network system further comprises: an encoding recurrent neural network configured to process each input in the sequence of inputs to generate the representation of the input. 8. The neural network system of claim 7, wherein the system input is a natural language question that relates to the data in the data source, and wherein the system output is an estimate of an answer to the natural language question. 9. The neural network system of claim 1, wherein one or more of the operations take as input a value of a mask variable that points to a subset of a location in the data source and wherein the operation subsystem is configured to, for each of the plurality of time steps: determine a current value of the mask variable by combining the operation outputs in accordance with the operation score distribution. 10. A method for generating a system output from a system input by repeatedly performing operations from a set of operations using data from a data source, the method comprising, for each of a plurality of time steps: processing a controller input for the time step and a representation of the system input using a controller recurrent neural network to generate: an operation score distribution that assigns a respective operation score to each operation in the set of operations, and a data score distribution that assigns a respective data score to each of a plurality of locations in the data source; and performing operations from the set of operations to generate a plurality of operation outputs, wherein at least one of the operations is performed on data in the plurality of locations in the data source; and combining the operation outputs in accordance with the operation score distribution and the data score distribution to generate a time step output for the time step. 11. The method of claim 10, further comprising, for each of the plurality of time steps: storing the time step output for the time step for use in any following time steps. 12. The method of claim 10, wherein the system output is the time step output for the last time step in the plurality of time steps. 13. The method of claim 10, wherein the controller input for the first time step in the plurality of time steps is a predetermined initial input, and wherein, each subsequent time step comprises: generating the controller input for the time step from the operation score distribution and the data score distribution for the preceding time step. 14. The method of claim 13, wherein the input for the time step comprises a weighted sum of vectors representing locations in the data source and a weighted sum of vectors representing the operations in the set of operations. 15. The method of claim 10, wherein the controller neural network comprises: a recurrent neural network configured to, for each time step, process the controller input for the time step to update a current hidden state of the recurrent neural network; one or more operation neural network layers configured to, for each time step, process the representation of the system input and the updated hidden state of the recurrent neural network to generate the operation score distribution; and one or more data neural network layers configured to, for each time step, process the representation of the system input and the updated hidden state of the recurrent neural network to generate the data score distribution. 16. The method of claim 10, wherein the system input is a sequence of inputs, and further comprising: processing, by an encoding recurrent neural network of a neural network system, each input in the sequence of inputs to generate the representation of the input. 17. The method of claim 16, wherein the system input is a natural language question that relates to the data in the data source, and wherein the system output is an estimate of an answer to the natural language question. 18. The method of claim 10, wherein one or more of the operations take as input a value of a mask variable that points to a subset of a location in the data source and wherein each of the plurality of time steps comprises: determining a current value of the mask variable by combining the operation outputs in accordance with the operation score distribution. 19. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating a system output from a system input by repeatedly performing operations from a set of operations using data from a data source, and wherein the operations comprise, for each of the plurality of time steps: processing a controller input for the time step and a representation of a system input using a controller recurrent neural network to generate: an operation score distribution that assigns a respective operation score to each operation in the set of operations, and a data score distribution that assigns a respective data score to each of a plurality of locations in the data source; and performing operations from the set of operations to generate a plurality of operation outputs, wherein at least one of the operations is performed on data in the plurality of locations in the data source; and combining the operation outputs in accordance with the operation score distribution and the data score distribution to generate a time step output for the time step. 20. The computer storage medium of claim 19, the operations further comprising, for each of the plurality of time steps: storing the time step output for the time step for use in any following time steps.