The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a compression method considering load balance for deep neural networks and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks in an efficient way so as to improve utilization of resources of the hardware platform.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for compressing a neural network, wherein the connection relations between the neurons of said neural network are characterized by a plurality of matrices, said method comprising: dividing step, for dividing at least one of said plurality of matrices into a plurality of submatrices; compression step, for compressing said submatrices into sparse submatrices; and encoding step, for encoding said compressed sparse submatrices. 2. The method of claim 1, wherein said dividing step further includes: dividing said matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices. 3. The method of claim 2, wherein said N is determined by the number of processing elements used for running the neural network, and said processing elements compute in a parallel manner. 4. The method of claim 1, wherein said compression step further includes: compressing each submatrix divided from the same matrix according to the same compression ratio. 5. The method of claim 4, wherein said compression step further includes: raking all elements in each submatrix from small to large according to their absolute values; reserving a percentage of elements with larger absolute values, said percentage corresponds to the compression ratio; and discarding other elements by setting their values to zero. 6. The method of claim 1, wherein said encoding step further includes: encoding said compressed sparse submatrices using the Compressed Row Storage (CRS) or Compressed Column Storage (CCS) format. 7. The method according to claim 1, further includes: providing the encoded sparse matrices to a plurality of parallel processing elements. 8. A method for compressing a neural network, wherein the connection relations between the neurons of said neural network are characterized by a plurality of matrices, said method comprising: dividing step, for dividing at least one of said plurality of matrices into a plurality of submatrices; sensitivity analysis step, for analyzing the sensitivity of each of said plurality of matrices, so as to determine an initial compression ratio for each of said plurality of matrices; compression step, for compressing the plurality of submatrices of respective matrix according to its corresponding initial compression ratio, so as to obtain a compressed neural network; fine-tuning step, for fine-tuning said compressed neural network, so as to obtain a final neural network. 9. The method of claim 8, further includes: repeating said dividing step, sensitivity analysis step, compression step and fine-tuning step iteratively. 10. The method of claim 8, wherein said neural network is LSTM neural network. 11. The method of claim 8, wherein said dividing step further includes: dividing each matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices. 12. The method of claim 11, wherein said N is determined by the number of processing elements used for running the neural network, and said processing elements compute in a parallel manner. 13. The method of claim 8, wherein said sensitivity analysis step further includes: obtaining an initial word error ratio WERinitial of the neural network; obtaining the word error ratios WERd1, WERd2 . . . WERdn of the neural network compressed according to different compression ratios d1,d2, . . . dn, respectively; and selecting said initial compression ratio from said different compression ratios d1, d2, . . . dn based on said word error ratio WERd1, WERd2 . . . WERdn of the compressed neural network. 14. The method of claim 13, wherein selecting said initial compression ratio from said different compression ratios further includes: calculating respective ΔWERs, which are the differences between WERinitial and WERd1, WERd2, . . . , WERdn; selecting the maximum compression ratio among all the compression ratios with a ΔWER below a predetermined threshold. 15. The method of claim 8, wherein said compression step further includes: pruning said submatrix based on its initial compression ratio. 16. The method of claim 8, wherein said compression step further includes: first compression step, for compressing each submatrix based on its initial compression ratio; adjusting step, for adjusting said initial compression ratio of each submatrix based on the WER of the neural network obtained after said first compression step, so as to obtain an adjusted compression ratio for each submatrix; second compression step, for compressing each submatrix based on corresponding adjusted compression ratio, so as to obtain said compressed neural network. 17. The method according to claim 8, wherein said fine-tuning step further includes: training step, for training said compressed neural network using a training dataset and updating the weight matrix; deciding step, for determining whether said weight matrix has converged; if not, returning to said training step; and if yes, the resultant neural network is regarded as said final neural network. 18. The method according to claim 17, wherein said training step further includes: inputting a training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix; updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix; calculating the loss of the neural network based on said updated weight matrix; deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating training step based on the previous valid iteration; if the present iteration is valid, it goes to said deciding step for determining whether said weight matrix has converged. 19. A method for compressing a neural network, wherein the weights between the neurons of said neural network are characterized by a plurality of matrices, said method comprising: sensitivity analysis step, for analyzing the sensitivity of each of said plurality of matrices, and determining an initial compression ratio for each of said plurality of matrices; compression step, for compressing each of said plurality of matrices based on said initial compression ratio, so as to obtain a compressed neural network, including dividing step, for dividing each of said plurality of matrices into a plurality of submatrices; pruning step, for pruning said submatrices into sparse submatrices; and fine-tuning step, for fine-tuning said compressed neural network. 20. The method of claim 19, further includes: repeating said sensitivity analysis step, compression step and fine-tuning step in at least one iteration. 21. The method of claim 19, wherein said dividing step further includes: dividing each matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices. 22. A device for compressing a neural network, wherein the connection relations between the neurons of said neural network are characterized by a plurality of matrices, said device comprising: a dividing unit, for dividing at least one of said plurality of matrices into a plurality of submatrices; a compression unit, for compressing said submatrices into sparse submatrices; and an encoding unit, for encoding said compressed sparse submatrices. 23. A device for compressing a neural network, wherein the connection relations between the neurons of said neural network are characterized by a plurality of matrices, said device comprising: a dividing unit, for dividing at least one of said plurality of matrices in to a plurality of submatrices; a sensitivity analysis unit, for analyzing the sensitivity of each of said plurality of matrices, so as to determine an initial compression ratio for each of said plurality of matrices; a compression unit, for compressing each submatrix according to said initial compression ratio, so as to obtain a compressed neural network; a fine-tuning unit, for fine-tuning said compressed neural network, so as to obtain a final neural network.