Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A memory-augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the memory-augmented neural network system comprising: a controller neural network configured to receive a neural network input at each of a plurality of time steps and to process the controller neural network input to generate a neural network output for the time step, wherein each neural network output includes: a read key, and a write vector; an external memory; and a sparse memory access subsystem that is configured to, for each of the plurality of time steps: generate a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in the external memory using the read key, read data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generate a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and write the write vector to the plurality of locations in the external memory in accordance with the writing weights. 2. The system of claim 1, wherein the sparse set of reading weights includes only K non-zero weights, wherein K is a small predetermined integer. 3. The system of claim 2, wherein the sparse memory access subsystem is configured to generate the sparse set of reading weights by: determining the K locations of the plurality of locations that are most similar to the read key according to a similarity measure; assigning a zero reading weight to each location of the plurality of locations other than the K locations; and determining the reading weight for each of the K locations from the similarity measure between the location and the read key. 4. The system of claim 3, wherein the sparse memory access subsystem is configured to determine the K locations using a k-nearest neighbors technique. 5. The system of claim 3, wherein the sparse memory access subsystem is configured to determine the K locations using an approximate nearest neighbor technique. 6. The system of claim 1, wherein the set of writing weights is a sparse set of weights that includes only J non-zero weights, wherein J is a small predetermined integer. 7. The system of claim 6, wherein the sparse memory access subsystem is configured to generate the set of writing weights by: determining a least recently accessed location from the plurality of locations; determining the writing weight for each location other than the least recently accessed location from the reading weight for the location at the preceding time step and an interpolation parameter included in the neural network output; and determining the writing weight for the least recently accessed location by performing an interpolation between the reading weight for the location at the preceding time step and a constant value in accordance with the interpolation parameter. 8. The system of claim 7, wherein the sparse memory access subsystem is configured to determine the least recently accessed location by: maintaining a respective usage value for each of the plurality of locations; and selecting the location having the smallest usage value as the least recently accessed location. 9. The system of claim 8, wherein the respective usage value for each of the plurality of locations is a time discounted sum of the write weights for the location over preceding time steps. 10. The system of claim 8, wherein the respective usage value for each of the plurality of locations is a number of time steps since a non-negligible memory access of the location. 11. The system of claim 7, wherein the sparse memory access subsystem is further configured to, for each of the plurality of time steps: erase the data stored in the least recently accessed location. 12. The system of claim 1, wherein the system further includes one or more output layers, and wherein the one or more output layers are configured to, for each of the plurality of time steps: process the data read from the external memory for the time step to generate a system output for the time step. 13. The system of claim 1, wherein the neural network output for each of the plurality of time steps further includes a system output portion. 14. The system of claim 13, wherein the sparse memory access subsystem is further configured to, for each of the plurality of time steps, provide an output derived from the system output portion as a system output for the time step. 15. The system of claim 13, wherein the system further includes one or more output layers, and wherein the one or more output layers are configured to, for each of the plurality of time steps: process the data read from the external memory and the system output portion for the time step to generate a system output for the time step. 16. The system of claim 1, wherein the controller neural network input for each time step is a respective system input from the sequence of system inputs. 17. The system of claim 1, wherein the sparse memory access subsystem is further configured to, for each of the plurality of time steps: combine the data read from the external memory and a respective system input from the sequence of system inputs to generate a subsequent neural network input for a subsequent time step. 18. One or more non-transitory storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a memory-augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the memory-augmented neural network system comprising: a controller neural network configured to receive a neural network input at each of a plurality of time steps and to process the controller neural network input to generate a neural network output for the time step, wherein each neural network output includes: a read key, and a write vector; an external memory; and a sparse memory access subsystem that is configured to, for each of the plurality of time steps: generate a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in the external memory using the read key, read data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generate a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and write the write vector to the plurality of locations in the external memory in accordance with the writing weights. 19. The computer storage media of claim 18, wherein the sparse set of reading weights includes only K non-zero weights, wherein K is a small predetermined integer. 20. A method for processing a sequence of system inputs to generate a sequence of system outputs, the method comprising, at each of a plurality of time steps: processing a neural network input at the time step using a controller neural network, wherein the controller neural network is configured to receive a neural network input at each of the plurality of time steps and to process the controller neural network input to generate a neural network output for the time step, wherein each neural network output includes: a read key, and a write vector; generating a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in an external memory using the read key in the neural network output for the time step; reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights; generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory; and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.