Artificial neural networks (ANNs) are a distributed computing model in which computation is accomplished using many simple processing units (called neurons) and the data embodied by the connections between neurons (called synapses) and the strength of these connections (called synaptic weights). An attractive implementation of ANNs uses the conductance of non-volatile memory (NVM) elements to code the synaptic weight. In this application, the non-idealities in the response of the NVM (such as nonlinearity, saturation, stochasticity and asymmetry in response to programming pulses) lead to reduced network performance compared to an ideal network implementation. Disclosed is a method that improves performance by implementing a learning rate parameter that is local to each synaptic connection, a method for tuning this local learning rate, and an implementation that does not compromise the ability to train many synaptic weights in parallel during learning.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A neuron-centric method implemented in an artificial neural network (ANN) to mitigate network freeze out, the ANN comprising a plurality of neurons arranged in layers with the outputs of one layer connected to the inputs of each neuron of the subsequent layer, where neurons are connected to each other via a synapse, each synapse having a synaptic weight encoding a connection strength between two connected neurons, where a plurality of training examples are serially input to the matrix while observing its output, where a backpropagation algorithm updates the synaptic weight in response to a difference between the output from a given layer and a desired output from said given layer, the neuron-centric method comprising: (a) storing a local learning rate value in each neuron, where an effective learning rate applied to each synapse is determined jointly by local learning rates of two connected neurons; (b) identifying two connected neurons whose synaptic weight dithers during the network freeze out; (c) suppressing activity of the two connected neurons of (b) whose synaptic weight dithers, by decreasing the effective learning rate of it; and (d) increasing local learning rates of at least one other connected neurons; wherein the suppressing step (c) and the increasing step (d) stabilize network performance and mitigate the network freeze out. 2. The method of claim 1, wherein the synaptic weights in ANN are implemented using a non-volatile memory (NVM). 3. The method of claim 2, wherein a pair of NVM conductances represent the synaptic weights, with one NVM conductance in the pair providing a positive current contribution and the other NVM conductance in the pair providing a negative current contribution. 4. The method of claim 2, wherein the NVM is selected from the group consisting of resistive random access memory (RRAM) and phase-change memory (PCM). 5. The method of claim 1, wherein the neurons in ANN are represented using CMOS circuitry. 6. The method of claim 1, wherein the ANN is used as part of an application that is selected from the group consisting of: pattern recognition application, image recognition application, speech recognition application, and language processing application. 7. The method of claim 1, wherein the plurality of neurons comprises a plurality of upstream neurons and a plurality of downstream neurons, where each neuron in the ANN determines (i) a number of programming pulses to send downstream based on its own local learning rate, (ii) its output value xi for the upstream neurons, and (iii) a backpropagated error δj for the downstream neurons. 8. The method of claim 7, wherein the suppressing step (c) and the increasing step (d) constrain the output value xi to always be positive, and where changes in sign of the backpropagated error δj capture changes in sign of weight updates with regards to the dithering, and where a rule adjusts local learning rates by reducing local learning rate when the sign of δj changes from one training example to the next, and the rule adjusts local learning rates by increasing it when the sign is the same. 9. The method of claim 8, wherein each downstream neuron decides how many pulses to send to upstream neurons based on the product η·gjδ·δj where η is a global learning rate for the ANN, and gjδ is a local learning rate for neuron j, which is adjusted based on the behavior of the backpropagated error, δj. 10. The method of claim 8, wherein the local learning rate is constrained to be between 0.1 and 10.0, and when sgn(δj(t))=sgn(δi(t−1)) the local learning rate is increased according to gjδ(t)=gjδ(t−1)+ε where ε is a predetermined incremental value, and when the sign of the error differs from that in the previous step, the local learning rate is decreased by according to gjδ(t)=gjδ(t−1)·λ where λ is a predetermined fraction, with λ>>ε. 11. An article of manufacture having a computer usable medium having computer readable program code embodied therein which implements a neuron-centric method implemented in an artificial neural network (ANN) to mitigate network freeze out, the ANN comprising a plurality of neurons arranged in layers with the outputs of one layer connected to the inputs of each neuron of the subsequent layer, where neurons are connected to each other via a synapse, each synapse having a synaptic weight encoding a connection strength between two connected neurons, where a plurality of training examples are serially input to the matrix while observing its output, where a backpropagation algorithm updates the synaptic weight in response to a difference between the output from a given layer and a desired output from said given layer, said medium comprising: (a) computer readable program code storing a local learning rate value in each neuron where an effective learning rate applied to each synapse is determined jointly by local learning rates of two connected neurons; (b) computer readable program code identifying two connected neurons whose synaptic weight dithers during the network freeze out; (c) computer readable program code suppressing activity of the two connected neurons of (b) whose synaptic weight dithers, by decreasing the effective learning rate of it; and (d) computer readable program code increasing local learning rates of at least one other connected neurons; wherein the suppressing step (c) and the increasing step (d) stabilizes network performance and mitigates the network freeze out. 12. The article of manufacture of claim 11, wherein the synaptic weights in ANN are implemented using a non-volatile memory (NVM). 13. The article of manufacture of claim 12, wherein a pair of NVM conductances represent the synaptic weights, with one NVM conductance in the pair providing a positive current contribution and the other NVM conductance in the pair providing a negative current contribution. 14. The article of manufacture of claim 12, wherein the NVM is selected from the group consisting of resistive random access memory (RRAM) and phase-change memory (PCM). 15. The article of manufacture of claim 11, wherein the neurons in ANN are represented using CMOS circuitry. 16. The article of manufacture of claim 11, wherein the ANN is used as part of an application that is selected from the group consisting of: pattern recognition application, image recognition application, speech recognition application, and language processing application. 17. The article of manufacture of claim 11, wherein the plurality of neurons comprises a plurality of upstream neurons and a plurality of downstream neurons, where each neuron in the ANN determines (i) a number of programming pulses to send downstream based on its own local learning rate, (ii) its output value xi for the upstream neurons, and (iii) a backpropagated error δj for the downstream neurons. 18. The article of manufacture of claim 17, wherein the suppressing step (c) and the increasing step (d) constrain the output value xi to always be positive, and where changes in sign of the backpropagated error δj capture changes in sign of weight updates with regards to the dithering, and where a rule adjusts local learning rates by reducing local learning rate when the sign of δj changes from one training example to the next, and the rule adjusts local learning rates by increasing it when the sign is the same. 19. The article of manufacture of claim 18, wherein each downstream neuron decides how many pulses to send to upstream neurons based on the product η·gjδ·δj where η is a global learning rate for the ANN, and gjδ is a local learning rate for neuron j, which is adjusted based on the behavior of the backpropagated error, δj. 20. The article of manufacture of claim 18, wherein the local learning rate is constrained to be between 0.1 and 10.0, and when sgn(δj(t))=sgn(δi(t−1)) the local learning rate is increased according to gjδ(t)=gjδ(t−1)+ε where ε is a predetermined incremental value, and when the sign of the error differs from that in the previous step, the local learning rate is decreased by according to gjδ(t)=gjδ(t−1)·λ where λ is a predetermined fraction, with λ>>ε. 21. A method of enhancing the performance of an artificial neural network using local learning rates, the network comprising a plurality of simple processing units called neurons where neurons are connected to each other via a synapse, each synapse having a synaptic weight encoding a connection strength between two connected neurons, the plurality of neurons arranged in layers with the outputs of one layer connected to the inputs of each neuron of the subsequent layer; wherein a plurality of training examples are serially input to the network while observing its output; and wherein a back-propagation algorithm is used to update the synaptic weights in response to a difference between the output from a given layer and a desired output from said given layer; wherein the method comprises: modifying a magnitude of a synaptic weight of the update by one or two local learning rates stored in at least one of two connected neurons that are connected by an individual synapse; determining local learning rates using rules implemented by the two connected neurons where the local learning rates are stored; and iterating the training examples until the output approaches the desired output within a desired accuracy. 22. The method of claim 21, wherein the rule for adjusting the local learning rate is based on the change of sign of the synaptic weight updates, and the neurons are constrained to unipolar outputs having outputs that are always one sign (unipolar outputs). 23. The method of claim 21, wherein the local learning rate determines the programming pulses that are applied for synaptic weight updates in such a way that programming pulses are applied to many synapses in parallel. 24. The method of claim 21, wherein each synaptic weight is represented by the conductance of one or two non-volatile memory elements.