The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a compression method for deep neural networks with proper use of mask and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks while maintaining or even improving the accuracy of the neural networks after compression.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for compressing a neural network with a mask matrix, said neural network is characterized by a plurality of matrices, said method comprising: mask matrix obtaining step of obtaining a mask matrix M, wherein said mask matrix M records the distribution of non-zero elements in a plurality of sparse matrices of the neural network; first pruning step of pruning the neural network nnet0 using said mask matrix M to obtain a pruned neural network nneti1; first retraining step of retraining the pruned neural network nneti1 without a mask, wherein the pruned neural network nneti1 is retrained without using said mask matrix M to obtain a dense neural network nneto1; second pruning step of pruning said dense neural network nneto1 using said mask matrix M to obtain a sparse neural network nneti2; second retraining step of retraining said sparse neural network nneti2 using said mask matrix M to obtain a final neural network nneto2. 2. The method of claim 1, wherein said mask matrix obtaining step further includes: pruning said neural network to be compressed nnet0 to obtain a plurality of sparse matrices containing zero and non-zero elements; obtaining said mask matrix M by using elements “1” to represent non-zero elements in corresponding sparse matrix. 3. The method of claim 1, wherein said first pruning step further includes: point multiplying said neural network to be compressed nnet0 with said mask matrix M to obtain said pruned neural network nneti1:nneti1=M ⊚ nnet0. 4. The method of claim 1, wherein said first retraining step further includes: training step, for training said pruned neural network nneti1 using a training dataset and updating its weight matrix without using said mask matrix M; deciding step, for determining whether the updated weight matrix has converged; if not, returning to said training step; and if yes, the resultant neural network is regarded as said dense neural network nneto1. 5. The method according to claim 4, wherein said training step further includes: inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix; updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix; calculating the loss of the neural network based on said updated weight matrix; deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating the training step based on the previous valid iteration; if the present iteration is valid, it goes to said deciding step for determining whether said updated weight matrix has converged. 6. The method of claim 1, wherein said second pruning step further includes: point multiplying said dense neural network nneto1 with said mask matrix M to obtain said sparse neural network nneti2:nneti2=M ⊚ nneto1. 7. The method of claim 1, wherein said second retraining step further includes: training step, for training said sparse neural network nneti2 using a training dataset and updating its weight matrix using said mask matrix M; deciding step, for determining whether the updated weight matrix has converged; if not, returning to said training step; and if yes, the resultant neural network is regarded as said final neural network nneto2. 8. The method according to claim 7, wherein said training step further includes: inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix; updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix*said mask matrix; calculating the loss of the neural network based on said updated weight matrix; deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating the training step based on the previous valid iteration; if the present iteration is valid, it goes to said deciding step for determining whether said updated weight matrix has converged. 9. The method according to claim 1, wherein said mask matrix M contains elements “0” and “1”, wherein: elements “1” mean that elements in corresponding position of the weight matrix are remained, while elements “0” mean that elements in corresponding position of the weight matrix are set to zero. 10. A device for compressing a neural network with a mask matrix, said neural network is characterized by a plurality of matrix, said device comprising: a mask matrix obtaining unit, configured for obtaining a mask matrix M, wherein said mask matrix M records the distribution of non-zero elements in a plurality of sparse matrices of the neural network; a first pruning unit, configured for pruning the neural network nnet0 using said mask matrix M to obtain a pruned neural network nneti1; a first retraining unit, configured for retraining the pruned neural network nneti1 obtained from said first pruning unit without using said mask matrix M to obtain a dense neural network nneto1; a second pruning unit, configured for pruning said dense neural network nneto1 using said mask matrix M to obtain a sparse neural network nneti2; a second retraining unit, configured for retraining said sparse neural network nneti2 using said mask matrix M to obtain a final neural network nneto2. 11. The device of claim 10, wherein said mask matrix obtaining unit further includes: a pruning unit, configured for pruning said neural network to be compressed nnet0 to obtain a plurality of sparse matrices containing zero and non-zero elements; an obtaining unit, configured for obtaining said mask matrix M by using elements “1” to represent non-zero elements in corresponding sparse matrix. 12. The device of claim 10, wherein said first pruning unit further includes: a first point multiplying unit, configured for point multiplying said neural network to be compressed nnet0 with said mask matrix M to obtain said pruned neural network nneti1:nneti1=M ⊚ nnet0. 13. The device of claim 10, wherein said first retraining unit further includes: a first training unit, configured for training said pruned neural network nneti1 using a training dataset and updating its weight matrix without using said mask matrix M; a first deciding unit, configured for determining whether the updated weight matrix has converged; if not, said first training unit repeats the training; and if yes, the resultant neural network is regarded as said dense neural network nneto1. 14. The device according to claim 13, wherein said first training unit further includes: a first inputting unit, configured for inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix; a first updating unit, configured for updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix; a first calculating unit, configured for calculating the loss of the neural network based on said updated weight matrix; a first decision unit, configured for deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, said first training unit adjusts the learning rate and repeats the training based on the previous valid iteration; if the present iteration is valid, said first training unit outputs said updated weight matrix to said first deciding unit for determining whether said updated weight matrix has converged. 15. The device of claim 10, wherein said second pruning unit further includes: a second point multiplying unit, configured for point multiplying said dense neural network nneto1 with said mask matrix M to obtain said sparse neural network nneti2:nneti2=M ⊚ nneto1. 16. The device of claim 10, wherein said second retraining unit further includes: a second training unit, configured for training said sparse neural network nneti2 using a training dataset and updating its weight matrix using said mask matrix M; a second deciding unit, configured for determining whether the updated weight matrix has converged; if not, said second training unit repeats the training; and if yes, the resultant neural network is regarded as said final neural network nneto2. 17. The device according to claim 16, wherein said second training unit further includes: a second inputting unit, configured for inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix; a second updating unit, configured for updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating−learning rate*gradient matrix*said mask matrix; a second calculating unit, configured for calculating the loss of the neural network based on said updated weight matrix; a second decision unit, configured for deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid; if the present iteration is invalid and the total number of iterations does not reach its maximum, said second training unit adjusts the learning rate and repeats the training based on the previous valid iteration; if the present iteration is valid, said second training unit outputs said updated weight matrix to said second deciding unit for determining whether said updated weight matrix has converged. 18. The device according to claim 10, wherein said mask matrix M contains elements “0” and “1”, wherein: elements “1” mean that elements in corresponding position of the weight matrix are remained, while elements “0” mean that elements in corresponding position of the weight matrix are set to zero.