Apparatuses and methods of manufacturing same, systems, and methods for performing network parameter quantization in deep neural networks are described. In one aspect, diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network are determined and then used to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters. In another aspect, the neural network is trained using first and second moment estimates of gradients of the network parameters and then the second moment estimates are used to weight the network parameters as part of quantizing the network parameters. In yet another aspect, network parameter quantization is performed by using an entropy-constrained scalar quantization (ECSQ) iterative algorithm. In yet another aspect, network parameter quantization is performed by quantizing the network parameters of all layers of a deep neural network together at once.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method, comprising: determining diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network; and using the determined diagonals to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters. 2. The method of claim 1, wherein the network parameters are quantized by Hessian-weighted k-means clustering of the network parameters, by using Hessian-weighted means as cluster centers. 3. The method of claim 2, wherein the quantized network parameters are encoded by binary encoding. 4. The method of claim 1, wherein network parameters are clustered by uniform quantization, using Hessian-weighted means as cluster centers, and the quantized network parameters are encoded by variable-length binary encoding. 5. The method of claim 1, wherein the neural network is a deep neural network having multiple layers, and quantization is performed on all layers at the same time. 6. The method of claim 1, wherein entropy coding is used, and quantization clustering is performed by using entropy-constrained scalar quantization (ECSQ). 7. The method of claim 6, wherein network parameters are clustered by uniform quantization, using Hessian-weighted means as cluster centers. 8. The method of claim 6, wherein network parameters are clustered by using iterative operations to perform ECSQ. 9. The method of claim 8, wherein clustering comprises: defining a Lagrangian cost function in terms of a distortion measure of the clusters and the entropy of the network parameters after clustering; and iteratively assigning and updating clusters until the Lagrangian cost function reduces less than a threshold. 10. The method of claim 8, wherein Hessian-weighting is used in performing the assigning and updating. 11. A method, comprising: training a neural network using first and second moment estimates of gradients of the network parameters; and using the second moment estimates to weight the network parameters as part of quantizing the network parameters. 12. The method of claim 11, wherein training is performed by a stochastic gradient descent (SGD) optimizer. 13. An apparatus in a neural network, comprising: one or more non-transitory computer-readable media; and at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of: determining diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network; and using the determined diagonals to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters. 14. The apparatus of claim 13, wherein the quantized network parameters are encoded by binary encoding. 15. The apparatus of claim 13, wherein the network parameters are quantized by Hessian-weighted k-means clustering of the network parameters, by using Hessian-weighted means as cluster centers, and are encoded by fixed-length binary encoding. 16. The apparatus of claim 13, wherein network parameters are clustered by uniform quantization, using Hessian-weighted means as cluster centers, and the quantized network parameters are encoded by variable-length binary encoding. 17. The apparatus of claim 13, wherein the neural network is a deep neural network having multiple layers, and quantization is performed on all layers at the same time. 18. The apparatus of claim 13, wherein entropy coding (optimal variable-length binary encoding) is used, and quantization clustering is performed by using entropy-constrained scalar quantization (ECSQ). 19. An apparatus in a neural network, comprising: one or more non-transitory computer-readable media; and at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of: training a neural network using first and second moment estimates of gradients of the network parameters; and using the second moment estimates to weight the network parameters as part of quantizing the network parameters. 20. A method, comprising: manufacturing a chipset comprising: at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of: determining diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network; and using the determined diagonals to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters; and the one or more non-transitory computer-readable media which store the instructions and the determined diagonals. 21. A method of testing an apparatus, comprising: testing whether the apparatus has at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of: determining diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network; and using the determined diagonals to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters; and testing whether the apparatus has the one or more non-transitory computer-readable media which store the instructions.