Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions. In one aspect, a system includes a value subnetwork configured to receive an observation characterizing a current state of the environment and process the observation to generate a value estimate; a policy subnetwork configured to receive the observation and process the observation to generate an ideal point in the continuous action space; and a subsystem configured to receive a particular point in the continuous action space representing a particular action; generate an advantage estimate for the particular action; and generate a Q value for the particular action that is an estimate of an expected return resulting from the agent performing the particular action when the environment is in the current state.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A reinforcement learning system for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions, the system comprising: a value subnetwork configured to: receive an observation characterizing a current state of the environment; and process the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state; a policy subnetwork configured to: receive the observation, and process the observation to generate an ideal point in the continuous action space; and a subsystem configured to: receive a particular point in the continuous action space representing a particular action; generate an advantage estimate for the particular action from a distance between the ideal point and the particular point; and generate a Q value for the particular action that is an estimate of an expected return resulting from the agent performing the particular action when the environment is in the current state by combining the advantage estimate and the value estimate. 2. The system of claim 1, wherein the agent is a mechanical agent interacting with a real-world environment. 3. The system of claim 1, wherein the subsystem is further configured to: receive a reward and a subsequent observation resulting from the agent performing the particular action, wherein the subsequent observation characterizes a subsequent state of the environment; generate a new value estimate for the subsequent state by providing the subsequent observation as input to the value subnetwork, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state; and combine the reward and the new value estimate to generate a target output value for the particular action. 4. The system of claim 1, wherein generating the advantage estimate comprises: applying a function having state-dependent parameters to the distance. 5. The system of claim 4, wherein the system further comprises: a function parameter subnetwork configured to: receive the observation; and process the observation to generate an output that defines values of the state-dependent parameters. 6. The system of claim 5, wherein the advantage estimate satisfies: A(x, u|θA)=−1÷2(u−μ(x|θμ))TP(x|θP) (u−μ(x|θμ)) where the first term is the transpose of the distance, P is a state-dependent parameter matrix that has entries defined by the state-dependent parameter values, and the third term is the distance. 7. The system of claim 6, wherein P is a positive definite matrix, and wherein P(x|θP)=LLT, where L is a lower-triangular matrix having entries derived from the output of the function parameter subnetwork. 8. The system of any claim 1, wherein the system further comprises: a replay memory storing experience tuples used to train the policy subnetwork and the value subnetwork. 9. The system of claim 8, wherein the subsystem is further configured to, during the training of the policy subnetwork and the value subnetwork: generate imagination rollouts, wherein each imagination rollout is a synthetic experience tuple; and add the generated imagination rollouts to the replay memory. 10. The system of claim 9, wherein generating an imagination rollout comprises: identifying a particular observation characterizing a state transitioned into by the environment in response to an action performed by the agent; selecting an action different from the action performed by the agent in response to the particular observation; processing the particular observation and the selected action using a state transition model to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selection action in response to the particular observation; and generating a synthetic experience tuple that includes the particular observation, the selected action, and the next observation. 11. A method of training a policy subnetwork of a reinforcement learning system that is configured to compute Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions comprising: obtaining an experience tuple identifying a training observation characterizing a training state of the environment, an action performed by the agent in response to the training observation, a reward received as a result of the agent performing the action in response to the training observation, and a subsequent observation characterizing a subsequent state of the environment; processing the training observation using a value subnetwork to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state; processing, using the policy subnetwork and in accordance with current values of the parameters of the policy subnetwork, the training observation to generate an ideal point in the continuous action space for the training observation; generating an advantage estimate for the training action from a distance between the ideal point and a particular point representing the training action; generating a Q value for the particular action by combining the advantage estimate and the value estimate; processing the subsequent observation using the value subnetwork to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state; combining the reward and the new value estimate to generate a target Q value for the particular action; and determining an update to the current values of the parameters of the policy subnetwork using an error between the Q value for the particular action and the target Q value. 12. A method for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions, the method comprising: receiving, by a value subnetwork of a reinforcement learning system, an observation characterizing a current state of the environment; processing, by the value subnetwork of the reinforcement learning system, the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state; processing, by a policy subnetwork of the reinforcement learning system, the observation to generate an ideal point in the continuous action space; generating, by the subsystem of the reinforcement learning system, an advantage estimate for the particular action from a distance between the ideal point and the particular point; and generating, by the subsystem of the reinforcement learning system, a Q value for the particular action that is an estimate of an expected return resulting from the agent performing the particular action when the environment is in the current state by combining the advantage estimate and the value estimate. 13. The method of claim 12, wherein the agent is a mechanical agent interacting with a real-world environment. 14. The method of claim 12, further comprising: receiving, by the subsystem of the reinforcement learning system, a reward and a subsequent observation resulting from the agent performing the particular action, wherein the subsequent observation characterizes a subsequent state of the environment; generating, by the subsystem of the reinforcement learning system, a new value estimate for the subsequent state by providing the subsequent observation as input to the value subnetwork, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state; and combining, by the subsystem of the reinforcement learning system, the reward and the new value estimate to generate a target output value for the particular action. 15. The method of claim 12, wherein generating the advantage estimate comprises: applying a function having state-dependent parameters to the distance. 16. The method of claim 15, further comprising: processing, by a function parameter subnetwork of the reinforcement learning system, the observation to generate an output that defines values of the state-dependent parameters. 17. The method of claim 16, wherein the advantage estimate satisfies: A(x, u|θA)=−1÷2(u−μ(x|θμ))TP(x|θP) (u−μ(x|θμ)) where the first term is the transpose of the distance, P is a state-dependent parameter matrix that has entries defined by the state-dependent parameter values, and the third term is the distance. 18. The method of claim 17, wherein P is a positive definite matrix, and wherein P(x|θP)=LLT, where L is a lower-triangular matrix having entries derived from the output of the function parameter subnetwork. 19. The method of claim 12, further comprising: generating, by the subsystem of the reinforcement learning system, imagination rollouts, wherein each imagination rollout is a synthetic experience tuple; and adding, by the subsystem of the reinforcement learning system, the generated imagination rollouts to a replay memory of the reinforcement learning system storing experience tuples used to train the policy subnetwork and the value subnetwork. 20. The method of claim 19, wherein generating an imagination rollout comprises: identifying a particular observation characterizing a state transitioned into by the environment in response to an action performed by the agent; selecting an action different from the action performed by the agent in response to the particular observation; processing the particular observation and the selected action using a state transition model to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selection action in response to the particular observation; and generating a synthetic experience tuple that includes the particular observation, the selected action, and the next observation.