Aspects of the present invention are directed to computer-implemented techniques for improving the training of artificial neural networks using a reduced precision (e.g., float16) data format. Embodiments of the present invention rescale tensor values prior to performing matrix operations (such as matrix multiplication or matrix addition) to prevent overflow and underflow. To preserve accuracy throughout the performance of the matrix operations, the scale factors are defined using a novel data format to represent tensors, wherein a matrix is represented by the tuple X, where X=(a, v[.]), wherein a is a float scale factor and v[.] are scaled values stored in the float16 format. The value of any element X[i] according to this data format would be equal to a*v[i].
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A computer-implemented method of performing matrix computations, said method comprising: accessing a first matrix, X; accessing a second matrix, Y, wherein said matrices X and Y are memory resident and comprise values represented in a reduced precision floating point format and wherein said reduced precision floating point format represents values from a low threshold to a high threshold; determining a first scaling factor, a, for matrix X; determining a second scaling factor, b, for matrix Y, wherein said determinings are performed to avoid overflow and underflow for matrix computations involving a*X and b*Y; and performing a matrix computation that takes a*X and b*Y as inputs and that generates an output. 2. The method as described in claim 1, wherein said reduced precision floating point format comprises the float16 floating point format. 3. The method as described in claim 1, wherein said matrix computation comprises matrix multiplication. 4. The method as described in claim 1, wherein said matrix computation comprises matrix convolution. 5. The method as described in claim 1, wherein said matrix computation comprises matrix addition. 6. The method as described in claim 1, wherein said matrix computation comprises at least one of: matrix multiplication; matrix convolution; and matrix addition and wherein further said matrix computation is part of a learning network training process comprising gradient computations. 7. The method as described in claim 1, wherein said matrix computation is performed by a hardware computation unit of a graphics processing unit. 8. The method as described in claim 1, wherein said determining said first scaling factor, a, comprises: computing a maximum absolute value, max(X), of values of said matrix, X; computing a mean value, mean(X), of non-zero values of said matrix, X; and computing a range value, range(X) as [max(X)/mean(X)]. 9. The method as described in claim 8, wherein said determining said second scaling factor, b, comprises: computing a maximum absolute value, max(Y), of values of said matrix, Y; computing a mean value, mean(Y), of non-zero values of said matrix, Y; and computing a range value, range(Y) as [max(Y)/mean(Y)]. 10. The method as described in claim 9, wherein said matrix computation comprises matrix multiplication and wherein said determining said first scaling factor, a, and wherein said determining said second scaling factor, b, further comprise: selecting said a and said b wherein [a*mean(X)*b*mean(Y)] is equal to or greater than said low threshold; and selecting said a and said b wherein: [a*mean(X)*b*max(Y)] is equal to or less than said high threshold; [b*mean(Y)*a*max(X)] is equal to or less than said high threshold; K*[a*mean(X)*b*mean(Y)] is equal to or less than said high threshold, wherein K is a constant; a*max(X) is equal to or less than said high threshold; and b*max(Y) is equal to or less than said high threshold. 11. The method as described in claim 9, wherein said matrix computation comprises matrix convolution and wherein said determining said first scaling factor, a, and wherein said determining said second scaling factor, b, further comprise: selecting said a and said b wherein [a*mean(X)*b*mean(Y)] is equal to or greater than said low threshold; and selecting said a and said b wherein: [a*mean(X)*b*max(Y)] is equal to or less than said high threshold; [a*max(X)*b*max(Y)] is equal to or less than said high threshold; K*[a*mean(X)*b*mean(Y)] is equal to or less than said high threshold, wherein K is a constant; a*max(X) is equal to or less than said high threshold; and b*max(Y) is equal to or less than said high threshold. 12. A computer-implemented method of performing training for a deep learning network, said method comprising: accessing a first matrix, X; accessing a second matrix, Y, wherein said matrices X and Y are memory resident and comprise values represented in a float16 floating point format and wherein said float16 format represents values from a low threshold to a high threshold; determining a first scaling factor, a, for matrix X; determining a second scaling factor, b, for matrix Y, wherein said determinings are performed to avoid overflow and underflow for matrix computations involving a*X and b*Y; and performing said training by performing computations of a forward propagation layer, a convolutional layer, an inner product layer and a backward propagation layer, wherein said convolutional layer comprises a gradient computation that comprises a matrix computation that takes a*X and b*Y as inputs and that generates a gradient output. 13. The method as described in claim 12 wherein said matrix computation is performed by a hardware computation unit of a graphics processing unit. 14. The method as described in claim 13 wherein said determining said first scaling factor, a, comprises: computing a maximum absolute value, max(X), of values of said matrix, X; computing a mean value, mean(X), of non-zero values of said matrix, X; and computing a range value, range(X) as [max(X)/mean(X)]. 15. The method as described in claim 14 wherein said determining said second scaling factor, b, comprises: computing a maximum absolute value, max(Y), of values of said matrix, Y; computing a mean value, mean(Y), of non-zero values of said matrix, Y; and computing a range value, range(Y) as [max(Y)/mean(Y)]. 16. The method as described in claim 15 wherein said matrix computation comprises matrix multiplication and wherein said determining said first scaling factor, a, and wherein said determining said second scaling factor, b, further comprise: selecting said a and said b wherein [ a*mean(X)*b*mean(Y)] is equal to or greater than said low threshold; and selecting said a and said b wherein: [a*mean(X)*b*max(Y)] is equal to or less than said high threshold; [b*mean(Y)*a*max(X)] is equal to or less than said high threshold; K*[a*mean(X)*b*mean(Y)] is equal to or less than said high threshold, wherein K is a constant; a*max(X) is equal to or less than said high threshold; and b*max(Y) is equal to or less than said high threshold. 17. The method as described in claim 15 wherein said matrix computation comprises matrix convolution and wherein said determining said first scaling factor, a, and wherein said determining said second scaling factor, b, further comprise: selecting said a and said b wherein [a*mean(X)*b*mean(Y)] is equal to or greater than said low threshold; and selecting said a and said b wherein: [a*mean(X)*b*max(Y)] is equal to or less than said high threshold; [a*max(X)*b*max(Y)] is equal to or less than said high threshold; K*[a*mean(X)*b*mean(Y)] is equal to or less than said high threshold, wherein K is a constant; a*max(X) is equal to or less than said high threshold; and b*max(Y) is equal to or less than said high threshold. 18. A non-transitory computer readable medium comprising a plurality of programmed instructions stored thereon, wherein, the plurality of programmed instructions are configured to be executed by a processing device to perform training for a deep learning network, said plurality of programmed instructions comprising: instructions to access a first matrix, X; instructions to access a second matrix, Y, wherein said matrices X and Y are memory resident and comprise values represented in a float16 floating point format and wherein said float16 format represents values from a low threshold to a high threshold; instructions to determine a first scaling factor, a, for matrix X; instructions to determine a second scaling factor, b, for matrix Y, wherein said instructions to determine said first and said second scaling factors are performed to avoid overflow and underflow for matrix computations involving a*X and b*Y; and instructions to perform said training by performing computations of a forward propagation layer, a convolutional layer, an inner product layer and a backward propagation layer, wherein said convolutional layer comprises a gradient computation that comprises a matrix computation that takes a*X and b*Y as inputs and that generates a gradient output. 19. The non-transitory computer readable medium as described in claim 18 wherein said instructions to determine said first scaling factor, a, comprises: instructions to compute a maximum absolute value, max(X), of values of said matrix, X; instructions to compute a mean value, mean(X), of non-zero values of said matrix, X; and instructions to compute a range value, range(X) as [max(X)/mean(X)]. 20. The non-transitory computer readable medium as described in claim 19 wherein said instructions to determine said second scaling factor, b, comprises: instructions to compute a maximum absolute value, max(Y), of values of said matrix, Y; instructions to compute a mean value, mean(Y), of non-zero values of said matrix, Y; and instructions to compute a range value, range(Y) as [max(Y)/mean(Y)].