Techniques and constructs can reduce the time required to determine solutions to optimization problems such as training of neural networks. Modifications to a computational model can be determined by a plurality of nodes operating in parallel. Quantized modification values can be transmitted between the nodes to reduce the volume of data to be transferred. The quantized values can be as small as one bit each. Quantization-error values can be stored and used in quantizing subsequent modifications. The nodes can operate in parallel and overlap computation and data transfer to further reduce the time required to determine solutions. The quantized values can be partitioned and each node can aggregate values for a corresponding partition.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method comprising: determining gradient matrices of a computational model of an optimization problem; quantizing the gradient matrices using corresponding stored error matrices; updating the error matrices using the corresponding quantized gradient matrices; and exchanging the quantized gradient matrices with a number of nodes. 2. The method as claim 1 recites, wherein the determining, quantizing, and updating are performed by individual ones of the nodes for respective ones of the gradient matrices and the error matrices corresponding to the respective gradient matrices. 3. The method as claim 1 recites, further comprising repeating the determining, quantizing, updating, and exchanging steps for each of a plurality of minibatches of the computational model, the exchanging for a first of the minibatches being performed in parallel with the determining, quantizing, or updating for a second of the minibatches. 4. The method as claim 3 recites, the exchanging comprising exchanging the quantized gradient matrices. 5. The method as claim 3 recites, wherein the nodes exchange the quantized gradient matrices synchronously. 6. The method as claim 3 recites, the exchanging comprising: partitioning the quantized gradient matrices; providing individual ones of the partitions of the quantized gradient matrices to respective ones of the nodes; aggregating the received partitions at the nodes; and transmitting the aggregated data from individual ones of the nodes to other ones of the nodes. 7. The method as claim 1 recites, further including adjusting a parallelization factor as a function of batch size based at least in part on time measurements. 8. The method as claim 1 recites, the quantizing comprising determining an approximate single-bit representation for each element of the gradient matrices. 9. The method as claim 1 recites, further including reconstructing the quantized gradient matrices after the exchanging. 10. The method as claim 1 recites, wherein the computational model includes a neural network model, the method further including updating the neural network model using the gradient matrices. 11. A computer-readable medium having thereon computer-executable instructions, the computer-executable instructions upon execution configuring a computer to carry out steps including: determining gradient matrices of a computational model of an optimization problem; quantizing the gradient matrices using corresponding stored error matrices; updating the error matrices using the corresponding quantized gradient matrices; and exchanging the quantized gradient matrices with a number of nodes. 12. A system comprising: one or more computer-readable media having thereon a plurality of modules and a computational model of an optimization problem; and a plurality of nodes, each including at least one processing unit, each processing unit operably coupled to at least one of the computer-readable media, the processing units adapted to intercommunicate and to execute modules of the plurality of modules comprising: an update-determining module configured to determine modification values of the computational model; a quantization module configured to quantize the determined modification values using stored error values and to update the stored error values using the determined modification values and the quantized modification values; a transferring module configured to transmit at least some of the quantized modification values to at least one other of the processing units; and an updating module configured to modify the stored computational model according to received quantized modification values. 13. The system as claim 12 recites, wherein at least one of the nodes transmits the quantized modification values directly to at least one other of the nodes. 14. The system as claim 12 recites, wherein each node includes a respective memory coupled to the respective at least one processing unit and configured to store respective private quantization state including the stored error values. 15. The system as claim 12 recites, wherein the computational model includes a neural-network model and the update-determining module is configured to determine the modification values using a stochastic gradient descent algorithm. 16. The system as claim 12 recites, wherein each of the nodes includes a plurality of processing units operably coupled to the respective computer-readable medium and configured to execute at least the update-determining module. 17. The system as claim 12 recites, further including a crossbar communicatively connecting the nodes, wherein the nodes are configured to execute the transferring module to transmit the at least some of the quantized modification values via the crossbar in parallel with executing the update-determining module. 18. The system as claim 12 recites, wherein the at least one processing unit of each node includes a general-purpose graphics processing unit (GPGPU) configured to execute the updating and quantization modules and a central processing unit (CPU) configured to execute the transferring module. 19. The system as claim 18 recites, wherein the transferring module is configured to transfer second quantized modification values from the GPGPU to the CPU in parallel with transferring the at least some of the quantized modification values to the at least one other of the processing units. 20. The system as claim 12 recites, the quantization module further configured to reconstruct modification values using the transferred quantized gradient modification values and the updating module configured to modify the stored computational model according to the reconstructed modification values.