Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a respective neural network output for each of a plurality of inputs, the method comprising, for each of the neural network layers: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs up to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than or equal to the respective associated batch size of a subsequent layer in the sequence; and processing the number of the one or more batches of inputs to generate the respective neural network layer output.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for generating a respective neural network output for each of a plurality of inputs, wherein the generating comprises processing each input through each of a plurality of neural network layers to generate the respective neural network output for the input, wherein the neural network layers are arranged in a directed graph structure, and wherein each neural network layer has a respective batch size, the method comprising, for each of the neural network layers: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs equal to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than, less than, or equal to the respective associated batch size of a subsequent layer in the directed graph structure; and processing the number of the one or more batches of inputs to generate the respective neural network layer output. 2. The method of claim 1, where the respective batch size is based at least on a weight reuse value, the weight reuse value representing a number of times that weight inputs need to be reused for a compute time of output values using the weight inputs at a matrix computation unit to be longer than a load time of the weight inputs from memory. 3. The method of claim 2, where the weight reuse value is based at least on a clock rate of the memory storing the weight inputs. 4. The method of claim 2, where each batch size is based at least on the weight reuse value divided by a number of times that weight inputs for the respective layer are reused. 5. The method of claim 2, where the plurality of neural network layers is processed at a matrix processing unit, where processing the number of the one or more batches of inputs comprises computing accumulated values for each input using the matrix computation unit. 6. The method of claim 1, where each input corresponds to a distinct image resource. 7. The method of claim 1, where each input corresponds to an audio sample. 8. The method of claim 1, further comprising forming a batch from the one or more layer outputs for processing at the subsequent layer. 9. The method of claim 1, further comprising generating, for each output, a corresponding inference. 10. A system for generating a respective neural network output for each of a plurality of inputs, wherein the generating comprises processing each input through each of a plurality of neural network layers to generate the respective neural network output for the input, wherein the neural network layers are arranged in a directed graph structure, and wherein each neural network layer has a respective batch size, the system comprising: one or more computers; and computer-readable medium coupled to the one or more computers and having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to, for each of the neural network layers, perform operations comprising: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs equal to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than, less than, or equal to the respective associated batch size of a subsequent layer in the directed graph structure; and processing the number of the one or more batches of inputs to generate the respective neural network layer output. 11. The system of claim 10, where the respective batch size is based at least on a weight reuse value, the weight reuse value representing a number of times that weight inputs need to be reused for a compute time of output values using the weight inputs at a matrix computation unit to be longer than a load time of the weight inputs from memory. 12. The system of claim 11, where the weight reuse value is based at least on a clock rate of the memory storing the weight inputs. 13. The system of claim 11, where each batch size is based at least on the weight reuse value divided by a number of times that weight inputs for the respective layer are reused. 14. The system of claim 11, where the plurality of neural network layers is processed at a matrix processing unit, where processing the number of the one or more batches of inputs comprises computing accumulated values for each input using the matrix computation unit. 15. The system of claim 10, where each input corresponds to a distinct image resource. 16. The system of claim 10, where each input corresponds to an audio sample. 17. The system of claim 10, further comprising forming a batch from the one or more layer outputs for processing at the subsequent layer. 18. The system of claim 10, further comprising generating, for each output, a corresponding inference. 19. A computer-readable medium having instructions stored thereon, which, when executed by one or more computers, cause the one or more computers to perform operations for generating a respective neural network output for each of a plurality of inputs, wherein the generating comprises processing each input through each of a plurality of neural network layers to generate the respective neural network output for the input, wherein the neural network layers are arranged in a directed graph structure, and wherein each neural network layer has a respective batch size, the operations comprising, for each of the neural network layers: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs equal to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than, less than, or equal to the respective associated batch size of a subsequent layer in the directed graph structure; and processing the number of the one or more batches of inputs to generate the respective neural network layer output. 20. The computer-readable medium of claim 19, where the respective batch size is based at least on a weight reuse value, the weight reuse value representing a number of times that weight inputs need to be reused for a compute time of output values using the weight inputs at a matrix computation unit to be longer than a load time of the weight inputs from memory. 21. The computer-readable medium of claim 20, where the weight reuse value is based at least on a clock rate of the memory storing the weight inputs. 22. The computer-readable medium of claim 20, where each batch size is based at least on the weight reuse value divided by a number of times that weight inputs for the respective layer are reused. 23. The computer-readable medium of claim 20, where the plurality of neural network layers is processed at a matrix processing unit, where processing the number of the one or more batches of inputs comprises computing accumulated values for each input using the matrix computation unit. 24. The computer-readable medium of claim 19, where each input corresponds to a distinct image resource. 25. The computer-readable medium of claim 19, where each input corresponds to an audio sample. 26. The computer-readable medium of claim 19, further comprising forming a batch from the one or more layer outputs for processing at the subsequent layer. 27. The computer-readable medium of claim 19, further comprising generating, for each output, a corresponding inference.