A system and method provides efficient parallel training of a neural network model on multiple graphics processing units. A training module reduces the time and communication overhead of gradient accumulation and parameter updating of the network model in a neural network by overlapping processes in an advantageous way. In a described embodiment, a training module overlaps backpropagation, gradient transfer and accumulation in a Synchronous Stochastic Gradient Decent algorithm on a convolution neural network. The training module collects gradients of multiple layers during backpropagation of training from a plurality of graphics processing units (GPUs), accumulates the gradients on at least one processor and then delivers the gradients of the layers to the plurality of GPUs during the backpropagation of the training. The whole model parameters can then be updated on the GPUs after receipt of the gradient of the last layer.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. An apparatus comprising: a central processing unit (CPU) communicatively coupled to a plurality of graphics processing units (GPUs) and a memory unit; a training module residing in the memory unit and executed by the CPU that overlaps backpropagation, gradient transfer and accumulation to efficiently train the neural network model by: collecting gradients of each layer during backpropagation of training sent from the GPUs; accumulating the gradients of each layer concurrently during backpropagation on the CPU; and delivering the gradients of each layer during the backpropagation from the CPU to GPUs after summing up all gradients of each layer. 2. The apparatus of claim 1 wherein at least one of the GPUs further updates whole model parameters of the neural network model after the receipt the gradient of the last layer. 3. The apparatus of claim 1 wherein the backpropagation is part of a synchronous stochastic gradient descent algorithm for training the neural network model. 4. The apparatus of claim 1 wherein the neural network model is part of a convolutional neural network (CNN). 5. The apparatus of claim 1 further comprising an array in the memory coupled to the CPU for storing partial gradients of each layer collected from the GPUs. 6. The apparatus of claim 5 wherein the array further stores accumulated gradients of each layer collected from the GPUs. 7. The apparatus of claim 1 wherein the GPUs transfer gradients of each layer to be collected by the CPU while continuing execution of subsequent layers. 8. The apparatus of claim 1 wherein the GPUs create a communication stream to send the gradient for each layer to the host for accumulation. 9. A computer-implemented method executed by at least one central processing unit (CPU) for efficiently training a neural network model comprising: collecting gradients of each layer during backpropagation of training sent from a plurality of GPUs; accumulating the gradients of each layer on the CPU concurrently while the GPUs continue backpropagation to overlap backpropagation, gradient transfer and accumulation; and delivering the gradients of each layer during the backpropagation from the CPU to GPUs after summing up all gradients of each layer. 10. The method of claim 9 wherein at least one of the GPUs further updates whole model parameters of the neural network model after the receipt the gradient of the last layer. 11. The method of claim 10 wherein after receiving the update of the whole model parameters the GPUs start forward computation for the next training iteration. 12. The method of claim 9 wherein the backpropagation is part of a synchronous stochastic gradient descent algorithm for training the neural network model. 13. The method of claim 9 wherein the neural network model is part of a convolutional neural network (CNN). 14. The method of claim 9 further comprising storing partial gradients of each layer collected from the GPUs in an array in a memory coupled to the CPU. 15. The method of claim 14 wherein the array further stores accumulated gradients of each layer collected from the GPUs. 16. The method of claim 9 further comprising transferring gradients of each layer to be collected by the CPU while continuing execution of subsequent layers. 17. The method of claim 9 further comprising creating a communication stream to send the gradient for each layer to the host for accumulation. 18. A computer-implemented method executed by at least one central processing unit (CPU) for efficiently training a neural network model comprising: collecting gradients of each layer during backpropagation of training sent from a plurality of GPUs; accumulating the gradients of each layer on the CPU concurrently while the GPUs continue backpropagation execution of subsequent layers to overlap backpropagation, gradient transfer and accumulation; storing partial gradients of each layer collected from the GPUs in an array in a memory coupled to the CPU and further storing accumulated gradients of each layer collected from the GPUs in the array; creating a communication stream to send the gradient for each layer to the host for accumulation and delivering the gradients of each layer during the backpropagation from the CPU to GPUs after summing up all gradients of each layer; wherein at least one of the GPUs further updates whole model parameters of the neural network model after the receipt the gradient of the last layer; and wherein the backpropagation is part of a synchronous stochastic gradient descent algorithm for training the neural network model on a convolutional neural network (CNN).