Methods and systems for training a neural network include sampling multiple local sub-networks from a global neural network. The local sub-networks include a subset of neurons from each layer of the global neural network. The plurality of local sub-networks are trained at respective local processing devices to produce trained local parameters. The trained local parameters from each local sub-network are averaged to produce trained global parameters.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for training a neural network, comprising: sampling a plurality of local sub-networks from a global neural network, wherein the local sub-networks comprise a subset of neurons from each layer of the global neural network; training the plurality of local sub-networks at respective local processing devices to produce trained local parameters; and averaging the trained local parameters from each local sub-network to produce trained global parameters. 2. The method of claim 1, wherein sampling the plurality of local sub-networks comprises fixed sampling. 3. The method of claim 2, wherein training the plurality of local sub-networks comprises updating local parameters at each local processing device according to: ( W k ) t + 1 = argmin W k   F  ( W K ) + ρ 2   W k - ( W G k ) t + ( Λ k ) t  2 where F(.) is the objective function, Wk is the set of local parameters, WGk is the set of global parameters, Λk is the dual, k is an index of a particular local processing device, t is a particular iteration, and ρ is a user-specified tuning parameter. 4. The method of claim 1, wherein sampling the plurality of local sub-networks comprises dynamic sampling. 5. The method of claim 4, wherein training the plurality of local sub-networks comprises updating local parameters at each local processing device according to: ( W k ) t s + 1 = 1 ρ + 1 [ ( W k ) t s - η  ∇ F  ( W k ) t s ] + 1 ρ + 1  [ ( W G k ) t + ( Λ k ) t ] where Wk is the set of local parameters, WGk is the set of global parameters, Λk is the dual, k is an index of a particular local processing device, t is a particular iteration, ρ is a user-specified tuning parameter, ts is a number of iterations of updates on a local device s, and η is a user-tuned learning rate. 6. The method of claim 1, wherein training the plurality of local sub-networks comprises sharing local parameters between local processing devices when parameters overlap in respective sub-networks. 7. The method of claim 1, wherein the global parameters are stored at a single global parameter server. 8. The method of claim 1, further comprising performing a classification task using the trained global parameters. 9. The method of claim 8, wherein performing the classification task comprises performing distributed matrix multiplication. 10. A system for training a neural network, comprising: a plurality of local sub-network processing devices, each comprising: a neural network module comprising neurons that represent a subset of neurons from each layer of a global neural network, the neural network module being configured to train a local sub-network to produce trained local parameters; and a global parameter server configured to average the trained local parameters from each local sub-network to produce trained global parameters. 11. The system of claim 10, wherein the neurons of each neural network module represent a local sub-network that is sampled from the global neural network based on fixed sampling. 12. The system of claim 11, wherein each neural network module is further configured to updating local parameters at according to: ( W k ) t + 1 = argmin  W k  F  ( W K ) + ρ 2   W k - ( W G k ) t + ( Λ k ) t  2 where F(.) is the objective function, Wk is the set of local parameters, WGk is the set of global parameters, Λk is the dual, k is an index of a particular local processing device, t is a particular iteration, and ρ is a user-specified tuning parameter 13. The system of claim 10, wherein the neurons of each neural network module represent a local sub-network that is sampled from the global neural network based on dynamic sampling. 14. The system of claim 13, wherein each neural network module is further configured to updating local parameters at according to: ( W k ) t s + 1 = 1 ρ + 1 [ ( W k ) t s - η  ∇ F  ( W k ) t s ] + 1 ρ + 1  [ ( W G k ) t + ( Λ k ) t ] where Wk is the set of local parameters, WGk is the set of global parameters, Λk is the dual, k is an index of a particular local processing device, t is a particular iteration, ρ is a user-specified tuning parameter, ts is a number of iterations of updates on a local device s, and η is a user-tuned learning rate. 15. The system of claim 10, wherein the neural network module of each local sub-network processing device is further configured to share local parameters between local processing devices when parameters overlap in respective sub-networks. 16. The system of claim 10, wherein the plurality of local sub-network processing devices are further configured to perform a classification task using the trained global parameters. 17. The system of claim 16, wherein the plurality of local sub-network processing devices are further configured to perform distributed matrix multiplication.