Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A system for training a deep neural network used to select actions to be performed by an agent that interacts with an environment by performing actions selected from a predetermined set of actions, the system comprising: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network, and wherein each worker is further configured to repeatedly perform operations comprising: determining current values of the parameters of the deep neural network; receiving a current observation characterizing a current state of the environment replica interacted with by the actor associated with the worker; selecting a current action to be performed by the actor associated with the worker in response to the current observation in accordance with a respective action selection policy for the worker and using one or more outputs generated by the deep neural network in accordance with the current values of the parameters; identifying an actual reward resulting from the actor performing the current action when the environment replica is in the current state; receiving a next observation characterizing a next state of the environment replica interacted with by the actor, wherein the environment replica transitioned into the next state from the current state in response to the actor performing the current action; performing an iteration of a reinforcement learning technique to determine a current gradient using the actual reward and the next observation; updating an accumulated gradient with the current gradient to determine an updated accumulated gradient; determining whether criteria for updating the current values of the parameters of the deep neural network have been satisfied; and when the criteria for updating the current values of the parameters of the deep neural network have been satisfied: determining updated values of the parameters using the updated accumulated gradient, and storing the updated values in a shared memory accessible to all of the plurality of workers. 2. The system of claim 1, wherein each worker executes independently of each other worker on the same computer. 3. The system of claim 1, wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker. 4. The system of claim 1, the operations further comprising, when the criteria for updating the current values of the parameters of the deep neural network have been satisfied, clearing the updated accumulated gradient. 5. The system of claim 1, wherein determining whether the criteria for updating the current values of the parameters of the deep neural network have been satisfied comprises determining that a specified number of iterations of the reinforcement learning technique have been performed since a preceding update of the values of the parameters of the deep neural network. 6. The system of claim 1, wherein the deep neural network is a Q network that is configured to receive an input observation characterizing an input state of the environment and an input action and to generate a Q value that is an estimate of a long-term reward resulting from the input action being performed when the environment is in the input state, and wherein selecting the current action to be performed by the actor in response to the current observation comprises: for each action in the predetermined set of actions, processing the current observation and the action using the deep neural network to generate a Q value for the action in accordance with the current values of the parameters of the deep neural network; and selecting an action from the predetermined set of actions in accordance with the action selection policy for the worker using the Q values for the actions. 7. The system of claim 6, wherein the reinforcement learning technique is a Q-learning technique, and wherein performing the iteration comprises: determining a maximum output of a target network for any of the actions in the predetermined set of actions when the environment replica is in the next state, wherein the target network is the same neural network as the Q network but with possibly different parameter values; determining an error from the maximum output of the target network, the actual reward, and the Q value for the current action; and determining the current gradient using the error. 8. The system of claim 6, wherein the reinforcement learning technique is a SARSA technique, and wherein performing the iteration comprises: selecting a next action to be performed by the actor associated with the worker in response to the next observation in accordance with the respective action selection policy for the worker; processing the next action and the next observation using a target network to determine a target network output for the next action, wherein the target network is the same neural network as the Q network but with possibly different parameter values; determining an error from the target network output for the next action, the actual reward, and the Q value for the current action; and determining the current gradient using the error. 9. One or more non-transitory storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a system for training a deep neural network used to select actions to be performed by an agent that interacts with an environment by performing actions selected from a predetermined set of actions, the system comprising: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network, and wherein each worker is further configured to repeatedly perform operations comprising: determining current values of the parameters of the deep neural network; receiving a current observation characterizing a current state of the environment replica interacted with by the actor associated with the worker; selecting a current action to be performed by the actor associated with the worker in response to the current observation in accordance with a respective action selection policy for the worker and using one or more outputs generated by the deep neural network in accordance with the current values of the parameters; identifying an actual reward resulting from the actor performing the current action when the environment replica is in the current state; receiving a next observation characterizing a next state of the environment replica interacted with by the actor, wherein the environment replica transitioned into the next state from the current state in response to the actor performing the current action; performing an iteration of a reinforcement learning technique to determine a current gradient using the actual reward and the next observation; updating an accumulated gradient with the current gradient to determine an updated accumulated gradient; determining whether criteria for updating the current values of the parameters of the deep neural network have been satisfied; and when the criteria for updating the current values of the parameters of the deep neural network have been satisfied: determining updated values of the parameters using the updated accumulated gradient, and storing the updated values in a shared memory accessible to all of the plurality of workers. 10. The computer-storage media of claim 9, wherein each worker executes independently of each other worker on the same computer. 11. The computer-storage media of claim 9, wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker. 12. The computer-storage media of claim 9, the operations further comprising, when the criteria for updating the current values of the parameters of the deep neural network have been satisfied, clearing the updated accumulated gradient. 13. The computer-storage media of claim 9, wherein determining whether the criteria for updating the current values of the parameters of the deep neural network have been satisfied comprises determining that a specified number of iterations of the reinforcement learning technique have been performed since a preceding update of the values of the parameters of the deep neural network. 14. The computer-storage media of claim 9, wherein the deep neural network is a Q network that is configured to receive an input observation characterizing an input state of the environment and an input action and to generate a Q value that is an estimate of a long-term reward resulting from the input action being performed when the environment is in the input state, and wherein selecting the current action to be performed by the actor in response to the current observation comprises: for each action in the predetermined set of actions, processing the current observation and the action using the deep neural network to generate a Q value for the action in accordance with the current values of the parameters of the deep neural network; and selecting an action from the predetermined set of actions in accordance with the action selection policy for the worker using the Q values for the actions. 15. The computer-storage media of claim 14, wherein the reinforcement learning technique is a Q-learning technique, and wherein performing the iteration comprises: determining a maximum output of a target network for any of the actions in the predetermined set of actions when the environment replica is in the next state, wherein the target network is the same neural network as the Q network but with possibly different parameter values; determining an error from the maximum output of the target network, the actual reward, and the Q value for the current action; and determining the current gradient using the error. 16. The computer-storage media of claim 14, wherein the reinforcement learning technique is a SARSA technique, and wherein performing the iteration comprises: selecting a next action to be performed by the actor associated with the worker in response to the next observation in accordance with the respective action selection policy for the worker; processing the next action and the next observation using a target network to determine a target network output for the next action, wherein the target network is the same neural network as the Q network but with possibly different parameter values; determining an error from the target network output for the next action, the actual reward, and the Q value for the current action; and determining the current gradient using the error. 17. A method of training a deep neural network used to select actions to be performed by an agent that interacts with an environment by performing actions selected from a predetermined set of actions, the method comprising: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network, and wherein each worker is further configured to repeatedly perform operations comprising: determining, by a first worker of a plurality of workers current values of the parameters of the deep neural network, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network; receiving, by the first worker, a current observation characterizing a current state of the environment replica interacted with by the actor associated with the first worker; selecting, by the first worker, a current action to be performed by the actor associated with the first worker in response to the current observation in accordance with a respective action selection policy for the first worker and using one or more outputs generated by the deep neural network in accordance with the current values of the parameters; identifying, by the first worker, an actual reward resulting from the actor performing the current action when the environment replica is in the current state; receiving, by the first worker, a next observation characterizing a next state of the environment replica interacted with by the actor, wherein the environment replica transitioned into the next state from the current state in response to the actor performing the current action; performing, by the first worker, an iteration of a reinforcement learning technique to determine a current gradient using the actual reward and the next observation; updating, by the first worker, an accumulated gradient with the current gradient to determine an updated accumulated gradient; determining, by the first worker, whether criteria for updating the current values of the parameters of the deep neural network have been satisfied; and when the criteria for updating the current values of the parameters of the deep neural network have been satisfied: determining, by the first worker, updated values of the parameters using the updated accumulated gradient, and storing, by the first worker, the updated values in a shared memory accessible to all of the plurality of workers. 18. The method of claim 17, wherein each worker executes independently of each other worker on the same computer. 19. The method of claim 17, wherein the respective action selection policy for each worker is different from the respective action selection policy for each other worker. 20. The method of claim 17, wherein the deep neural network is a Q network that is configured to receive an input observation characterizing an input state of the environment and an input action and to generate a Q value that is an estimate of a long-term reward resulting from the input action being performed when the environment is in the input state, and wherein selecting the current action to be performed by the actor in response to the current observation comprises: for each action in the predetermined set of actions, processing the current observation and the action using the deep neural network to generate a Q value for the action in accordance with the current values of the parameters of the deep neural network; and selecting an action from the predetermined set of actions in accordance with the action selection policy for the worker using the Q values for the actions.