Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for optimizing machine learning systems. In one aspect a method includes determining an average error of a machine learning system (“MLS”). An evaluation function that provides a result that would have been achieved using a specified value of a given parameter is defined. An expected outcome function that provides expected results for prior events based on the error of the MLS is defined. For each of multiple prior events, a target value of the given parameter is determined, e.g., using the expected outcome function. A model is generated using the MLS based on features of the prior events and the determined target values of the given parameter for the prior events. A value is assigned to the given parameter for a new event based on application of the model to features of the new event.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A system for optimizing a machine learning model, comprising: a third-party corpus database storing information related to a plurality of third-party content; a set of computing devices that interact with the third-party corpus database and perform operations comprising: determining an average error of a machine learning system; defining an evaluation function that provides a result that would have been achieved using a specified value of a given parameter in prior events; defining an expected outcome function that provides expected results for prior events based on the error of the machine learning system; determining, for each of multiple prior events, a target value of the given parameter that causes the expected outcome function to provide a specified output for the prior event; generating a model using the machine learning system based on features of the prior events and the determined target values of the given parameter for the prior events; assigning a value to the given parameter for a new event based on application of the model to features of the new event; selecting third-party content for distribution to a client device based on the assigned value of the given parameter and selection values submitted by third-party content providers; and distributing, over a network, the selected third-party content to the client device. 2. The system of claim 1, wherein defining the evaluation function comprises defining the evaluation function to provide an output that specifies an amount of gain that would have been realized if a specified threshold eligibility value had been used to select third-party content. 3. The system of claim 2, wherein the set of computing devices perform operations further comprising evaluating selection values submitted by third-parties for each of one or more prior requests, wherein, for each request, the evaluation function provides an output of zero when no third-party has submitted a selection value that meets the threshold eligibility value, provides an output of the threshold eligibility value when a single third-party submitted a submission value meeting the threshold eligibility value, and provides an output that is greater than the threshold eligibility value when multiple third-parties submitted a submission value meeting the threshold eligibility value. 4. The system of claim 1, wherein defining the expected outcome function comprises defining the expected outcome function that outputs an amount of gain that would have been realized for a given request when the error of the machine learning system causes the actual threshold eligibility value to be higher or lower than a given threshold eligibility value for that given request, but the error does not prevent distribution of third-party content in response to the given request. 5. The system of claim 1, wherein determining the target value of the given parameter comprises determining a threshold eligibility value that maximizes the gain output by the expected outcome function. 6. The system of claim 1, wherein assigning the value to the given parameter comprises outputting, from the model, the threshold eligibility value that will be used for selection of third-party content that is provided in response to the request. 7. The system of claim 6, wherein selecting third-party content for distribution comprises selecting content having a selection value that equals or exceeds the threshold eligibility value output by the model. 8. A method of optimizing a machine learning system comprising: determining an average error of a machine learning system; defining an evaluation function that provides a result that would have been achieved using a specified value of a given parameter in prior events; defining an expected outcome function that provides expected results for prior events based on the error of the machine learning system; determining, for each of multiple prior events, a target value of the given parameter that causes the expected outcome function to provide a specified output for the prior event; generating, by one or more computing devices, a model using the machine learning system based on features of the prior events and the determined target values of the given parameter for the prior events; assigning, by one or more computing devices, a value to the given parameter for a new event based on application of the model to features of the new event; selecting, by one or more computing devices, third-party content for distribution to a client device based on the assigned value of the given parameter and selection values submitted by third-party content providers; and distributing, over a network, the selected third-party content to the client device. 9. The method of claim 8, wherein defining the evaluation function comprises defining the evaluation function to provide an output that specifies an amount of gain that would have been realized if a specified threshold eligibility value had been used to select third-party content. 10. The method of claim 9, further comprising evaluating selection values submitted by third-parties for each of one or more prior requests, wherein, for each request, the evaluation function provides an output of zero when no third-party has submitted a selection value that meets the threshold eligibility value, provides an output of the threshold eligibility value when a single third-party submitted a submission value meeting the threshold eligibility value, and provides an output that is greater than the threshold eligibility value when multiple third-parties submitted a submission value meeting the threshold eligibility value. 11. The method of claim 8, wherein defining the expected outcome function comprises defining the expected outcome function that outputs an amount of gain that would have been realized for a given request when the error of the machine learning system causes the actual threshold eligibility value to be higher or lower than a given threshold eligibility value for that given request, but the error does not prevent distribution of third-party content in response to the given request. 12. The method of claim 8, wherein determining the target value of the given parameter comprises determining a threshold eligibility value that maximizes the gain output by the expected outcome function. 13. The method of claim 8, wherein assigning the value to the given parameter comprises outputting, from the model, the threshold eligibility value that will be used for selection of third-party content that is provided in response to the request. 14. The method of claim 13, wherein selecting third-party content for distribution comprises selecting content having a selection value that equals or exceeds the threshold eligibility value output by the model. 15. A non-transitory computer readable medium storing instructions that upon execution by one or more data processing apparatus cause the one or more data processing apparatus to perform operations comprising: determining an average error of a machine learning system; defining an evaluation function that provides a result that would have been achieved using a specified value of a given parameter in prior events; defining an expected outcome function that provides expected results for prior events based on the error of the machine learning system; determining, for each of multiple prior events, a target value of the given parameter that causes the expected outcome function to provide a specified output for the prior event; generating a model using the machine learning system based on features of the prior events and the determined target values of the given parameter for the prior events; assigning a value to the given parameter for a new event based on application of the model to features of the new event; selecting third-party content for distribution to a client device based on the assigned value of the given parameter and selection values submitted by third-party content providers; and distributing, over a network, the selected third-party content to the client device. 16. The computer readable medium of claim 15, wherein defining the evaluation function comprises defining the evaluation function to provide an output that specifies an amount of gain that would have been realized if a specified threshold eligibility value had been used to select third-party content. 17. The computer readable medium of claim 16, further comprising evaluating selection values submitted by third-parties for each of one or more prior requests, wherein, for each request, the evaluation function provides an output of zero when no third-party has submitted a selection value that meets the threshold eligibility value, provides an output of the threshold eligibility value when a single third-party submitted a submission value meeting the threshold eligibility value, and provides an output that is greater than the threshold eligibility value when multiple third-parties submitted a submission value meeting the threshold eligibility value. 18. The computer readable medium of claim 15, wherein defining the expected outcome function comprises defining the expected outcome function that outputs an amount of gain that would have been realized for a given request when the error of the machine learning system causes the actual threshold eligibility value to be higher or lower than a given threshold eligibility value for that given request, but the error does not prevent distribution of third-party content in response to the given request. 19. The computer readable medium of claim 15, wherein determining the target value of the given parameter comprises determining a threshold eligibility value that maximizes the gain output by the expected outcome function. 20. The computer readable medium of claim 15, wherein assigning the value to the given parameter comprises outputting, from the model, the threshold eligibility value that will be used for selection of third-party content that is provided in response to the request, and wherein selecting third-party content for distribution comprises selecting content having a selection value that equals or exceeds the threshold eligibility value output by the model.