The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a multi-iteration compression method for deep neural networks and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks without degrading the accuracy of the neural
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for compressing a neural network, wherein the weights between the neurons of said neural network are characterized by a plurality of matrices, said method comprising: sensitivity analysis step, for analyzing the sensitivity of each of said plurality of matrices, and determining an initial compression ratio for each of said plurality of matrices; compression step, for compressing each of said plurality of matrices based on said initial compression ratio, so as to obtain a compressed neural network; and fine-tuning step, for fine-tuning said compressed neural network. 2. The method of claim 1, further includes: repeating said sensitivity analysis step, compression step and fine-tuning step for at least one iterations. 3. The method of claim 1, wherein said sensitivity analysis step further includes: obtaining an initial word error ratio WERinitial of the neural network; obtaining word error ratios WERd1, WERd2 . . . WERdn of the neural network compressed with different compression ratios d1, d2, . . . dn, respectively; and selecting the initial compression ratio from said different compression ratios based on said word error ratio WERd1, WERd2 . . . WERdn of the compressed neural network. 4. The method of claim 3, wherein selecting said initial compression ratio from said different compression ratios further includes: calculating respective ΔWERs, which are the differences between WERinitial and WERd1, WERd2 . . . , WERdn, respectively; selecting the maximum compression ratio among all the compression ratios with a ΔWER below a predetermined threshold. 5. The method of claim 1, wherein said compression step further includes: pruning corresponding matrix based on said compression ratio of each matrix. 6. The method of claim 5, wherein said pruning further includes: ranking all elements in each matrix from small to large according to their absolute values; keeping a percentage of elements of each matrix, said percentage corresponds to the compression ratio of said matrix, wherein said percentage of elements being kept having larger absolute values; and setting other elements into zero, wherein other elements have smaller absolute values. 7. The method of claim 1, wherein said compression step further includes: first compression step, for compressing each matrix based on its initial compression ratio; adjusting step, for adjusting said initial compression ratio of each matrix based on the WER of the neural network obtained by said first compression step, so as to obtain an adjusted compression ratio for each matrix; second compression step, for compressing each matrix based on said adjusted compression ratio, so as to obtain said compressed neural network. 8. The method according to claim 7, wherein said adjusting step further includes: compression ratio modifying step, for modifying said initial compression ratio of each sensitive matrix and compressing the corresponding matrix according the modified compression ratio; deciding step, for deciding whether the WER of the neural network compressed according to said modified compression ratio meets a predetermined requirement; if not, repeating said compression ratio modifying step in order to further modify the compression ratio of each sensitive matrix; and if yes, said modified compression ratio of each sensitive matrix is regarded as the adjusted compression ratio of the corresponding matrix. 9. The method according to claim 8, wherein said compression ratio modifying step further includes: reducing the compression ratio of each sensitive matrix by a predetermined step based on the WER of the neural network. 10. The method according to claim 7, wherein said adjusting step further includes: compression ratio modifying step, for modifying said initial compression ratio of each insensitive matrix and compressing the corresponding matrix according the modified compression ratio; deciding step, for deciding whether the WER of the neural network compressed according to said modified compression ratio meets a predetermined requirement; if not, repeating said compression ratio modifying step in order to further adjust the compression ratio of each insensitive matrix; and if yes, said modified compression ratio of each insensitive matrix is regarded as the adjusted compression ratio of the corresponding matrix. 11. The method according to claim 10, wherein said compression ratio modifying step further includes: reducing the compression ratio of each insensitive matrix by a predetermined step based on the WER of the neural network. 12. The method according to claim 1, wherein said fine-tuning step further includes: training step, for training said compressed neural network using a training dataset and updating the weight matrix; deciding step, for determining whether said weight matrix has converged; if not, returning to said training step; and if yes, said compressed neural network is regarded as the final neural network. 13. The method according to claim 12, wherein said training step further includes: inputting said training dataset and calculating the derivative of a loss function of the neural network so as to obtain a gradient matrix; updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix; calculating the loss of the neural network based on said updated weight matrix; deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating the training step based on the previous valid iteration; if the present iteration is valid, it goes to said deciding step for determining whether said weight matrix has converged. 14. The method according to claim 12, wherein said training step further includes: updating said weight matrix using a mask matrix. 15. The method according to claim 12, wherein said deciding step further includes: testing the WER of the neural network obtained in said training step, and if the WER meets a predetermined requirement, the neural network is considered as been converged. 16. The method according to claim 12, wherein said deciding step further includes: testing the relative improvement of the loss of the present valid iteration over that of the previous valid iteration, and if the relative improvement meets a predetermined requirement, the neural network is considered as been converged. 17. A device for compressing a neural network, wherein the weights between the neurons of said neural network is characterized by a plurality of matrices, said device comprising: a sensitivity analysis unit, configured for analyzing the sensitivity of each of said plurality of matrices, and determining an initial compression ratio for each of said plurality of matrix; a compression unit, configured for compressing each of said plurality of matrices based on said initial compression ratio, so as to obtain a compressed neural network; and a fine-tuning unit, configured for fine-tuning said compressed neural network. 18. The device of claim 17, wherein said sensitivity analysis unit further includes: a WER obtaining unit, configured for obtaining an initial word error ratio WERinitial of the neural network and word error ratios WERd1, WERd2 . . . WERdn of the neural network compressed with different compression ratios d1, d2, . . . dn, respectively; and an electing unit, configured for selecting the initial compression ratio from said different compression ratios based on said word error ratio WERd1, WERd2 . . . WERdn of the compressed neural network. 19. The device of claim 17, wherein said compression unit further includes: a first compression unit, configured for compressing each matrix based on its initial compression ratio; an adjusting unit, configured for adjusting said initial compression ratio of each matrix based on the WER of the neural network obtained by said first compression unit, so as to obtain an adjusted compression ratio for each matrix; a second compression unit, configured for compressing each matrix based on said adjusted compression ratio, so as to obtained said compressed neural network. 20. The device according to claim 17, wherein said fine-tuning unit further includes: a training unit, configured for training said compressed neural network using a training dataset and updating the weight matrix; a deciding unit, configured for determining whether said weight matrix has converged.