Methods, systems, and apparatus, including computer programs encoded on computer storage media, for batching inputs to machine learning models. One of the methods includes receiving a stream of requests, each request identifying a respective input for processing by a first machine learning model; adding the respective input from each request to a first queue of inputs for processing by the first machine learning model; determining, at a first time, that a count of inputs in the first queue as of the first time equals or exceeds a maximum batch size and, in response: generating a first batched input from the inputs in the queue as of the first time so that a count of inputs in the first batched input equals the maximum batch size, and providing the first batched input for processing by the first machine learning model.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method comprising: receiving a stream of requests, each request identifying a respective input for processing by a first machine learning model; adding the respective input from each request to a first queue of inputs for processing by the first machine learning model; determining, at a first time, that a count of inputs in the first queue as of the first time equals or exceeds a maximum batch size and, in response: generating a first batched input from the inputs in the queue as of the first time so that a count of inputs in the first batched input equals the maximum batch size, and providing the first batched input for processing by the first machine learning model; and determining, at a second time, that (i) a count of inputs in the queue as of the second time is less than that the maximum batch size but (ii) an oldest input in the queue is older than a latency parameter, and in response: generating a second batched input from the inputs in the queue as of the second time so that a count of inputs in the second batched input is less than the maximum batch size, and providing the second batched input for processing by the first machine learning model. 2. The method of claim 1, wherein generating the second batched input comprises: determining that the count of inputs in the second batched input is more than a first acceptable batch size and less than a second acceptable batch size and, in response, adding placeholder inputs to the second batched input so that the count of inputs in the second batched input equals the second acceptable batch size. 3. The method of claim 2, wherein each placeholder input is a copy of a respective input already in the second batched input. 4. The method of claim 2, further comprising: receiving a second machine learning output generated by the machine learning model for the second batched input; determining, for each input in the second batched input, a respective portion of the second machine learning output that corresponds to the input; for each placeholder input in the second batched input, discarding the respective corresponding portion of the second machine learning output; and for each input in the second batched input that is not a placeholder input, providing the respective corresponding portion of the second machine learning output in a response to the request that identified the input. 5. The method of claim 1, wherein the processing of inputs by the first machine learning model is managed by a computational graph system that represents the operations of the first machine learning model during processing of a given batched input as computational graph, wherein the computational graph comprises a plurality of nodes and directed edges, wherein each node represents a respective operation, and wherein each directed edge connects a respective first node to a respective second node that represents an operation that receives, as input, an output of an operation represented by the respective first node. 6. The method of claim 5, wherein the computational graph system processes batched inputs by assigning the operations represented by the nodes in the computational graph among a plurality of devices. 7. The method of claim 5, wherein providing the first batched input for processing by the first machine learning model comprises providing the first batched input to the computational graph system, and wherein providing the second batched input for processing by the first machine learning model comprises providing the second batched input to the computational graph system. 8. The method of claim 1, further comprising: maintaining a plurality of queues that includes the first queue, wherein each queue corresponds to a respective machine learning model from a plurality of machine learning models. 9. The method of claim 8, further comprising: determining that a thread has become available for processing a batched input by one of the plurality of machine learning models; and selecting the first machine learning model from the plurality of machine learning models and, in response, providing the first batched input for processing on the thread that has become available. 10. The method of claim 1, further comprising: determining the maximum batch size based on characteristics of hardware resources available for processing inputs to the machine learning model and an architecture of the machine learning model. 11. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving a stream of requests, each request identifying a respective input for processing by a first machine learning model; adding the respective input from each request to a first queue of inputs for processing by the first machine learning model; determining, at a first time, that a count of inputs in the first queue as of the first time equals or exceeds a maximum batch size and, in response: generating a first batched input from the inputs in the queue as of the first time so that a count of inputs in the first batched input equals the maximum batch size, and providing the first batched input for processing by the first machine learning model; and determining, at a second time, that (i) a count of inputs in the queue as of the second time is less than that the maximum batch size but (ii) an oldest input in the queue is older than a latency parameter, and in response: generating a second batched input from the inputs in the queue as of the second time so that a count of inputs in the second batched input is less than the maximum batch size, and providing the second batched input for processing by the first machine learning model. 12. The system of claim 11, wherein generating the second batched input comprises: determining that the count of inputs in the second batched input is more than a first acceptable batch size and less than a second acceptable batch size and, in response, adding placeholder inputs to the second batched input so that the count of inputs in the second batched input equals the second acceptable batch size. 13. The system of claim 12, wherein each placeholder input is a copy of a respective input already in the second batched input. 14. The system of claim 12, the operations further comprising: receiving a second machine learning output generated by the machine learning model for the second batched input; determining, for each input in the second batched input, a respective portion of the second machine learning output that corresponds to the input; for each placeholder input in the second batched input, discarding the respective corresponding portion of the second machine learning output; and for each input in the second batched input that is not a placeholder input, providing the respective corresponding portion of the second machine learning output in a response to the request that identified the input. 15. The system of claim 11, wherein the processing of inputs by the first machine learning model is managed by a computational graph system that represents the operations of the first machine learning model during processing of a given batched input as computational graph, wherein the computational graph comprises a plurality of nodes and directed edges, wherein each node represents a respective operation, and wherein each directed edge connects a respective first node to a respective second node that represents an operation that receives, as input, an output of an operation represented by the respective first node. 16. The system of claim 15, wherein the computational graph system processes batched inputs by assigning the operations represented by the nodes in the computational graph among a plurality of devices. 17. The system of claim 15, wherein providing the first batched input for processing by the first machine learning model comprises providing the first batched input to the computational graph system, and wherein providing the second batched input for processing by the first machine learning model comprises providing the second batched input to the computational graph system. 18. The system of claim 11, the operations further comprising: maintaining a plurality of queues that includes the first queue, wherein each queue corresponds to a respective machine learning model from a plurality of machine learning models. 19. The system of claim 18, the operations further comprising: determining that a thread has become available for processing a batched input by one of the plurality of machine learning models; and selecting the first machine learning model from the plurality of machine learning models and, in response, providing the first batched input for processing on the thread that has become available. 20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving a stream of requests, each request identifying a respective input for processing by a first machine learning model; adding the respective input from each request to a first queue of inputs for processing by the first machine learning model; determining, at a first time, that a count of inputs in the first queue as of the first time equals or exceeds a maximum batch size and, in response: generating a first batched input from the inputs in the queue as of the first time so that a count of inputs in the first batched input equals the maximum batch size, and providing the first batched input for processing by the first machine learning model; and determining, at a second time, that (i) a count of inputs in the queue as of the second time is less than that the maximum batch size but (ii) an oldest input in the queue is older than a latency parameter, and in response: generating a second batched input from the inputs in the queue as of the second time so that a count of inputs in the second batched input is less than the maximum batch size, and providing the second batched input for processing by the first machine learning model.