Systems, methods, devices, and other techniques are disclosed for using an augmented neural network system to generate a sequence of outputs from a sequence of inputs. An augmented neural network system can include a controller neural network, a hierarchical external memory, and a memory access subsystem. The controller neural network receives a neural network input at each of a series of time steps processes the neural network input to generate a memory key for the time step. The external memory includes a set of memory nodes arranged as a binary tree. To provide an interface between the controller neural network and the external memory, the system includes a memory access subsystem that is configured to, for each of the series of time steps, perform one or more operations to generate a respective output for the time step. The capacity of the neural network system to account for long-range dependencies in input sequences may be extended. Also, memory access efficiency may be increased by structuring the external memory as a binary tree.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. An augmented neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the augmented neural network system comprising: a controller neural network configured to receive a neural network input at each of a plurality of time steps and to process the neural network input to generate a neural network output for the time step, wherein each neural network output includes: a memory key; a hierarchical external memory comprising a plurality of memory nodes arranged as a binary tree; and a memory access subsystem that is configured to, for each of the plurality of time steps: traverse the binary tree to generate a path from a root memory node in the binary tree to a particular leaf memory node in the binary tree using the memory key, read the data stored in the particular leaf memory node, and update the data stored in the particular leaf memory node using the memory key. 2. The system of claim 1, wherein updating the data stored in the particular leaf memory node comprises: processing the data stored in the particular leaf memory node and the memory key using a write neural network, wherein the write neural network is configured to process the data stored in the particular leaf memory node and the memory key to generate updated memory node data; and writing the updated memory node data to the particular leaf memory node. 3. The system of claim 2, wherein the memory access subsystem is further configured to, for each of the plurality of time steps: update the data stored in the memory nodes on the path other than the particular leaf memory node in reverse traversal order by, for each of the other nodes: processing the updated data stored in the child node of the node that is on the path and the data stored in the child node that is not on the path using a join neural network, wherein the join neural network is configured to process data stored in one child node and data stored in another child node and to generate updated data for the parent node of the one child node and the other child node; and writing the updated data to the node. 4. The system of claim 1, wherein traversing the binary tree using the memory key comprises, at each memory node on the path until the particular leaf memory node: processing the data stored in the memory node and the memory key using a search neural network, wherein the search neural network is configured to process the data stored in the memory node and the memory key to generate a probability value p; following a first branch from the memory node to add a new child node to the path with probability p; and following a second branch from the memory to add a new child node to the path with probability 1−p. 5. The system of claim 1, wherein the memory access subsystem is further configured to: initialize the data stored in the leaf memory nodes of the binary tree using the sequence of system inputs. 6. The system of claim 5, wherein the memory access subsystem is further configured to: initialize the data stored in the memory nodes in the tree other than the leaf memory nodes in bottom up order by, for each of the other nodes: processing the updated data stored in the child nodes of the node using a join neural network, wherein the join neural network is configured to process data stored in one child node and data stored in another child node and to generate updated data for the parent node of the one child node and the other child node; and writing the updated data to the node. 7. The system of claim 1, wherein the system further includes one or more output layers, and wherein the one or more output layers are configured to, for each of the plurality of time steps: process the data read from the external memory for the time step to generate a system output for the time step. 8. The system of claim 1, wherein the neural network output for each of the plurality of time steps further includes a system output portion. 9. The system of claim 8, wherein the memory access subsystem is further configured to, for each of the plurality of time steps, provide an output derived from the system output portion as a system output for the time step. 10. The system of claim 8, wherein the system further includes one or more output layers, and wherein the one or more output layers are configured to, for each of the plurality of time steps: process the data read from the external memory and the system output portion for the time step to generate a system output for the time step. 11. The system of claim 1, wherein the neural network input for each time step is a respective system input from the sequence of system inputs. 12. The system of claim 1, wherein the memory access subsystem is further configured to, for each of the plurality of time steps: combine the data read from the external memory and a respective system input from the sequence of system inputs to generate a subsequent neural network input for a subsequent time step. 13. A computer-implemented method for processing a sequence of inputs that includes a respective input at each of a plurality of time steps to generate a sequence of outputs that includes a respective output at each of the plurality of time steps, the method comprising: obtaining the sequence of inputs; for each time step in the plurality of time steps: processing the respective input at the time step using a controller neural network to generate a memory key; identifying data stored in a hierarchical memory that is distinct from the controller neural network and includes a plurality of memory nodes arranged in a binary tree, wherein identifying the data includes: (i) traversing the binary tree to generate a path from a root memory node in the binary tree to a particular leaf memory node in the binary tree using the memory key, and (ii) reading the data stored in the particular leaf memory node; and providing a respective output for the sequence of outputs at the time step using the data stored in the particular leaf memory node. 14. The method of claim 13, further comprising for each time step in the plurality of time steps: updating the data stored in the particular leaf memory node by processing the data currently stored in the particular leaf memory node and the memory key using a write neural network to generate updated memory node data; and writing the updated memory node data to the particular leaf memory node. 15. The method of claim 13, wherein traversing the binary tree using the memory key comprises, at each memory node on the path until the particular leaf memory node: processing the data stored in the memory node and the memory key using a search neural network, wherein the search neural network is configured to process the data stored in the memory node and the memory key to generate a probability value p; following a first branch from the memory node to add a new child node to the path with probability p; and following a second branch from the memory to add a new child node to the path with probability 1−p. 16. The method of claim 13, further comprising: processing the sequence of inputs, or a sequence of system inputs from which the sequence of inputs is derived, using an embedding neural network to generate a sequence of embeddings that includes a respective embedding at each time step in the plurality of time steps; and initializing the data stored in the plurality of memory nodes of the binary tree using the sequence of embeddings. 17. The method of claim 13, wherein the controller neural network is one of a feedforward neural network or a recurrent neural network. 18. One or more computer-storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for processing a sequence of inputs that includes a respective input at each of a plurality of time steps to generate a sequence of outputs that includes a respective output at each of the plurality of time steps, the operations comprising: obtaining the sequence of inputs; for each time step in the plurality of time steps: processing the respective input at the time step using a controller neural network to generate a memory key; identifying data stored in a hierarchical memory that is distinct from the controller neural network and includes a plurality of memory nodes arranged in a binary tree, wherein identifying the data includes: (i) traversing the binary tree to generate a path from a root memory node in the binary tree to a particular leaf memory node in the binary tree using the memory key, and (ii) reading the data stored in the particular leaf memory node; and providing a respective output for the sequence of outputs at the time step using the data stored in the particular leaf memory node. 19. The one or more computer-storage media of claim 18, wherein the operations further comprise, for each time step in the plurality of time steps: updating the data stored in the particular leaf memory node by processing the data currently stored in the particular leaf memory node and the memory key using a write neural network to generate updated memory node data; and writing the updated memory node data to the particular leaf memory node. 20. The one or more computer-storage media of claim 18, wherein traversing the binary tree using the memory key comprises, at each memory node on the path until the particular leaf memory node: processing the data stored in the memory node and the memory key using a search neural network, wherein the search neural network is configured to process the data stored in the memory node and the memory key to generate a probability value p; following a first branch from the memory node to add a new child node to the path with probability p; and following a second branch from the memory to add a new child node to the path with probability 1−p.