Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting action slates using reinforcement learning. One of the methods includes receiving an observation characterizing a current state of an environment; selecting an action slate by processing the observation and a plurality of candidate action slates using a deep neural network, wherein each candidate action slate comprises a respective plurality of actions from the set of actions, and wherein the deep neural network is configured to, for each of the candidate action slates, process the observation and the actions in the candidate action slate to generate a slate Q value for the candidate action slate that is an estimate of a long-term reward resulting from the candidate action slate being provided to the action selector in response to the observation; and providing the selected action slate to an action selector in response to the observation.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method of providing a slate of actions to an action selector that interacts with an environment by selecting and performing actions, wherein the slate of actions includes a plurality of actions selected from a predetermined set of actions, and wherein the environment transitions states in response to actions performed by the action selector, the method comprising: receiving an observation characterizing a current state of the environment; selecting an action slate comprising a plurality of actions by processing the observation and a plurality of candidate action slates using a deep neural network, wherein each candidate action slate comprises a respective plurality of actions from the set of actions, and wherein the deep neural network is configured to, for each of the candidate action slates, process the observation and the actions in the candidate action slate to generate a slate Q value for the candidate action slate that is an estimate of a long-term reward resulting from the candidate action slate being provided to the action selector in response to the observation; and providing the selected action slate to the action selector in response to the observation. 2. The method of claim 1, wherein, in response to receiving the action slate, the action selector performs either (i) an action selected from the action slate or (ii) a null action that is not included in the set of actions. 3. The method of claim 1, wherein the action slate comprises a predetermined number of slots, and wherein selecting the action slate comprises, for a given subset of the slots in the action state: generating a plurality of candidate slates for the given subset of slots, each candidate slate for the given subset of slots including: in any slot for which an action has already been selected, the action already selected for the slot, a respective candidate action in each of the slots in the given subset, wherein each candidate slate has a different combination of candidate actions in the slots in the given subset from each other candidate slate, and a respective placeholder action in any slot in the action slate other than the slots in the given subset and the slots for which an action has already been selected; processing each of the plurality of candidate slates with the observation using the deep neural network to generate a respective slate Q value for each candidate slate; and selecting as the actions in the given subset of slots in the action slate the candidate actions that are in the slots in the given subset in the candidate slate having the highest slate Q value. 4. The method of claim 3, wherein the given subset of slots has a predetermined number of slots that is greater than one. 5. The method of claim 3, wherein the given subset of slots has one slot. 6. The method of claim 3, wherein the slots in the action slate are ordered from a highest slot in the action slate to a lowest slot in the action slate when provided to the action selector, and wherein selecting an action slate comprises selecting an action for each subset of slots in the action slate in sequence based on the ordering of the slots in the action slate when provided to the action selector. 7. The method of claim 3, further comprising: generating a random ordering of subsets of slots in the action slate, wherein selecting an action slate comprises selecting an action for each subset of slots in the action slate in sequence according to the random ordering. 8. The method of claim 3, wherein, for each candidate slate, the placeholder action is the same as one of the candidate actions in the given subset. 9. The method of claim 3, wherein, for each candidate slate, the placeholder actions are actions suggested by an external action recommendation system. 10. The method of claim 1, wherein the candidate actions are selected from a subset of the actions in the predetermined set of actions generated by an external process. 11. The method of claim 1, further comprising: receiving a reward in response to providing the action slate to the action selector; and using the reward in updating values of the parameters of the deep neural network. 12. The method of claim 11, wherein using the reward in updating values of the parameters of the deep neural network comprises: receiving a next observation characterizing a next state of the environment; and using the current observation, the selected action slate, the reward, and the next observation to update the values of the parameters of the deep neural network. 13. The method of claim 12, wherein using the current observation, the selected action slate, the reward, and the next observation to update the values of the parameters of the deep neural network comprises: modifying the reward to generate a modified reward; and using the modified reward in place of the reward in updating the values. 14. The method of claim 13, wherein the modified reward satisfies r̂a, wherein r is the reward and a is a constant value greater than one. 15. The method of claim 1, wherein the environment is a content item presentation setting provided by a content item recommendation system, wherein the action selector is a user of the content item recommendation system, wherein the actions in the set of actions are recommendations of content items, and wherein each action in the action slate is a recommendation of a distinct action to the user of the content item recommendation system. 16. The method of claim 1, wherein the observation characterizing the current state includes data characterizing a preceding action that was selected by the action selector from a preceding slate of actions provided to the action selector in response to a preceding observation. 17. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for providing a slate of actions to an action selector that interacts with an environment by selecting and performing actions, wherein the slate of actions includes a plurality of actions selected from a predetermined set of actions, and wherein the environment transitions states in response to actions performed by the action selector, the operations comprising: receiving an observation characterizing a current state of the environment; selecting an action slate comprising a plurality of actions by processing the observation and a plurality of candidate action slates using a deep neural network, wherein each candidate action slate comprises a respective plurality of actions from the set of actions, and wherein the deep neural network is configured to, for each of the candidate action slates, process the observation and the actions in the candidate action slate to generate a slate Q value for the candidate action slate that is an estimate of a long-term reward resulting from the candidate action slate being provided to the action selector in response to the observation; and providing the selected action slate to the action selector in response to the observation. 18. The system of claim 17, wherein, in response to receiving the action slate, the action selector performs either (i) an action selected from the action slate or (ii) a null action that is not included in the set of actions. 19. The system of claim 17, wherein the action slate comprises a predetermined number of slots, and wherein selecting the action slate comprises, for a given subset of the slots in the action state: generating a plurality of candidate slates for the given subset of slots, each candidate slate for the given subset of slots including: in any slot for which an action has already been selected, the action already selected for the slot, a respective candidate action in each of the slots in the given subset, wherein each candidate slate has a different combination of candidate actions in the slots in the given subset from each other candidate slate, and a respective placeholder action in any slot in the action slate other than the slots in the given subset and the slots for which an action has already been selected; processing each of the plurality of candidate slates with the observation using the deep neural network to generate a respective slate Q value for each candidate slate; and selecting as the actions in the given subset of slots in the action slate the candidate actions that are in the slots in the given subset in the candidate slate having the highest slate Q value. 20. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for providing a slate of actions to an action selector that interacts with an environment by selecting and performing actions, wherein the slate of actions includes a plurality of actions selected from a predetermined set of actions, and wherein the environment transitions states in response to actions performed by the action selector, the operations comprising: receiving an observation characterizing a current state of the environment; selecting an action slate comprising a plurality of actions by processing the observation and a plurality of candidate action slates using a deep neural network, wherein each candidate action slate comprises a respective plurality of actions from the set of actions, and wherein the deep neural network is configured to, for each of the candidate action slates, process the observation and the actions in the candidate action slate to generate a slate Q value for the candidate action slate that is an estimate of a long-term reward resulting from the candidate action slate being provided to the action selector in response to the observation; and providing the selected action slate to the action selector in response to the observation.