Compressing a machine learning network, such as a neural network, includes replacing one layer in the neural network with compressed layers to produce the compressed network. The compressed network may be fine-tuned by updating weight values in the compressed layer(s).
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method of compressing a neural network, comprising: replacing at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; inserting nonlinearity between compressed layers of the compressed network; and fine-tuning the compressed network by updating weight values in at least one of the compressed layers. 2. The method of claim 1, in which the inserting nonlinearity comprises applying a nonlinear activation function to neurons of the compressed layers. 3. The method of claim 2, in which the nonlinear activation function is a rectifier, absolute value function, hyperbolic tangent function or a sigmoid function. 4. The method of claim 1, in which the fine-tuning is performed by updating the weight values in the compressed neural network. 5. The method of claim 4, in which the fine-tuning comprises updating weight values in at least one of a subset of the compressed layers or in a subset of uncompressed layers. 6. The method of claim 4, in which the fine-tuning is performed using training examples, the training examples comprising at least one of a first set of examples used to train an uncompressed network or a new set of examples. 7. The method of claim 1, further comprising: initializing the neural network by repeatedly applying compression, insertion of nonlinear layers, and the fine-tuning as a method for initializing deeper neural networks. 8. A method of compressing a neural network, comprising: replacing at least one layer in the neural network with multiple compressed layers to produce a compressed neural network such that a receptive field size of the compressed layers combined matches a receptive field of uncompressed layers; and fine-tuning the compressed network by updating weight values in at least one compressed layer. 9. The method of claim 8, in which a kernel size of uncompressed layers is equal to the receptive field size. 10. The method of claim 8, in which the replacing comprises replacing at least one layer in the neural network having a kernel size kx×ky with multiple compressed layers of a same type with the kernel sizes k1x×k1y, k2x×k2y . . . kLx×kLy to produce the compressed network in which properties (k1x−1)+(k2x−1)+ . . . =(kx−1) and (k1y−1)+(k2y−1)+ . . . =(ky−1) are satisfied. 11. The method of claim 10, in which a convolutional layer with the kernel size kx×ky is replaced with three convolutional layers with the kernel sizes 1×1, kx×ky and 1×1, respectively. 12. A method of compressing a neural network, comprising: replacing at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; and determining weight matrices of compressed layers by applying an alternating minimization process. 13. The method of claim 12, further comprising fine-tuning the compressed network by updating weight values in at least one of the compressed layers. 14. The method of claim 13, in which the fine-tuning includes updating weight values in at least one of a subset of the compressed layers, or a subset of uncompressed layers. 15. The method of claim 13, in which the fine-tuning is performed in multiple stages, in which in a first stage the fine-tuning is performed on a subset of compressed layers, and in a second stage the fine-tuning is performed on a subset of compressed and uncompressed layers. 16. An apparatus for compressing a neural network, comprising: a memory; and at least one processor coupled to the memory, the at least one processor configured: to replace at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; to insert nonlinearity between compressed layers of the compressed network; and to fine-tune the compressed network by updating weight values in at least one compressed layers. 17. The apparatus of claim 16, in which the at least one processor is further configured to insert nonlinearity by applying a nonlinear activation function to neurons of the compressed layers. 18. The apparatus of claim 17, in which the nonlinear activation function is a rectifier, absolute value function, hyperbolic tangent function or a sigmoid function. 19. The apparatus of claim 16, in which the at least one processor is further configured to perform the fine-tune by updating the weight values in the compressed neural network. 20. The apparatus of claim 19, in which the at least one processor is further configured to perform the fine-tuning by updating weight values in at least one of a subset of compressed layers or in a subset of uncompressed layers. 21. The apparatus of claim 19, in which the at least one processor is further configured to perform the fine-tuning by using training examples, the training examples comprising at least one of a first set of examples used to train an uncompressed network or a new set of examples. 22. The apparatus of claim 16, in which the at least one processor is further configured to initialize the neural network by repeatedly applying compression, insertion of nonlinear layers, and the fine-tuning as a method for initializing deeper neural networks. 23. An apparatus for compressing a neural network, comprising: a memory; and at least one processor coupled to the memory, the at least one processor configured: to replace at least one layer in the neural network with multiple compressed layers to produce a compressed neural network such that a receptive field size of the compressed layers combined matches a receptive field size of uncompressed layers; and to fine-tune the compressed network by updating weight values in at least one of compressed layers. 24. The apparatus of claim 23, in which a kernel size of uncompressed layers is equal to the receptive field size. 25. The apparatus of claim 23, in which the at least one processor is further configured to replace at least one layer in the neural network having a kernel size kx×ky with multiple compressed layers of a same type with the kernel sizes k1x×k1y, k2x×k2y . . . kLx×kLy to produce the compressed network in which properties (k1x−1)+(k2x−1)+ . . . =(kx−1) and (k1y−1)+(k2y−1)+ . . . =(ky−1) are satisfied. 26. The apparatus of claim 25, in which a convolutional layer with the kernel size kx×ky is replaced with three convolutional layers with the kernel sizes 1×1, kx×ky and 1×1 respectively. 27. An apparatus for compressing a neural network, comprising: a memory; and at least one processor coupled to the memory, the at least one processor configured: to replace at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; and to determine weight matrices of compressed layers by applying an alternating minimization process. 28. The apparatus of claim 27, in which the at least one processor is further configured to fine-tune the compressed network by updating weight values in at least one of the compressed layers. 29. The apparatus of claim 28, in which the at least one processor is configured to perform the fine-tuning by updating weight values in at least one of a subset of the compressed layers, or a subset of uncompressed layers. 30. The apparatus of claim 28, in which the at least one processor is further configured to perform the fine-tuning in multiple stages, in which in a first stage the fine-tuning is performed on a subset of compressed layers, and in a second stage the fine-tuning is performed on a subset of compressed and uncompressed layers. 31. An apparatus for compressing a neural network, comprising: means for replacing at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; means for inserting nonlinearity between compressed layers of the compressed network; and means for fine-tuning the compressed network by updating weight values in at least one of the compressed layers. 32. An apparatus for compressing a neural network, comprising: means for replacing at least one layer in the neural network with multiple compressed layers to produce a compressed neural network such that a receptive field size of the compressed layers combined matches a receptive field of uncompressed layers; and means for fine-tuning the compressed network by updating weight values in at least one compressed layer. 33. An apparatus for compressing a neural network, comprising: means for replacing at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; and means for determining weight matrices of compressed layers by applying an alternating minimization process. 34. A non-transitory computer readable medium having encoded thereon program code for compressing a neural network, the program code being executed by a processor and comprising: program code to replace at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; program code to insert nonlinearity between compressed layers of the compressed network; and program code to fine-tune the compressed network by updating weight values in at least one compressed layers. 35. A non-transitory computer readable medium having encoded thereon program code for compressing a neural network, the program code being executed by a processor and comprising: program code to replace at least one layer in the neural network with multiple compressed layers to produce a compressed neural network such that a receptive field size of the compressed layers combined matches a receptive field size of uncompressed layers; and program code to fine-tune the compressed network by updating weight values in at least one of compressed layers. 36. A non-transitory computer readable medium having encoded thereon program code for compressing a neural network, the program code being executed by a processor and comprising: program code to replace at least one layer in the neural network with a plurality of compressed layers to produce a compressed neural network; and program code to determine weight matrices of compressed layers by applying an alternating minimization process.