Training neural networks by constructing a neural network model having neurons each associated with a quantized activation function adapted to output a quantized activation value. The neurons are arranged in layers and connected by connections associated quantized connection weight functions adapted to output quantized connection weight values. During a training process a plurality of weight gradients are calculated during backpropagation sub-processes by computing neuron gradients, each of an output of a respective the quantized activation function in one layer with respect to an input of the respective quantized activation function. Each neuron gradient is calculated such that when an absolute value of the input is smaller than a positive constant threshold value, the respective neuron gradient is set as a positive constant output value and when the absolute value of the input is smaller than the positive constant threshold value the neuron gradient is set to zero.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for training neural networks, comprising: constructing a neural network model having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight function adapted to output a quantized connection weight value selected from a second finite set; receiving a training set dataset; using said training set dataset to train said neural network model according to respective said quantized connection weight values, said training includes computing a plurality of weight gradients for backpropagation sub-processes by: computing a plurality of neuron gradients, each said neuron gradient is of an output of a respective said quantized activation function in one layer of said plurality of layers with respect to an input of said respective quantized activation function and is calculated such that when an absolute value of said input is smaller than a positive constant threshold value, said respective neuron gradient is set as a positive constant value and when the absolute value of said input is larger than said positive constant threshold value said neuron gradient is set to zero, and updating a plurality of floating point connection weight values according to said plurality of weight gradients; outputting a trained quantized neural network formed as an outcome of said training process. 2. The method of claim 1, wherein each of said plurality of connections is associated with one of said plurality of floating point connection weight values. 3. The method of claim 1, wherein said neural network model is a convolutional neural network (CNN) model and said trained quantized neural network is a trained quantized CNN. 4. The method of claim 1, wherein said quantized activation function is a binary function and said quantized activation value is a binary value. 5. The method of claim 1, wherein said quantized connection weight function is a binary function and said quantized connection weight value is a binary value. 6. The method of claim 1, wherein quantized connection weight value and said quantized activation value are binary values. 7. The method of claim 1, wherein said quantized activation function and said quantized connection weight function are deterministic functions. 8. The method of claim 1, wherein said quantized activation function and said quantized connection weight function are stochastic functions. 9. The method of claim 8, wherein said stochastic functions are hard sigmoid functions. 10. The method of claim 1, wherein said input is batch normalized. 11. The method of claim 1, wherein said input is scaled and shifted according to at least one constant value. 12. The method of claim 1, wherein said input is scaled and shifted according to a batch normalization calculated by approximating variance of values of at least one minibatch of said training set and normalizing said values of said at least one minibatch of said training set. 13. The method of claim 12, wherein multiplications of said batch normalization are performed using left and right bit-shift operations. 14. The method of claim 2, further comprising updating said plurality of floating-point connection weight values during said backpropagation sub-processes by performing multiplication operations using left and right bit-shift operations. 15. A system for training neural networks, comprising: a storage comprising a neural network model having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight function adapted to output a quantized connection weight value selected from a second finite set; at least one processor coupled to the storage for executing a code comprising: using a training set dataset to train said neural network model according to respective said quantized connection weight values, said training includes computing a plurality of weight gradients for backpropagation sub-processes by: computing a plurality of neuron gradients, each said neuron gradient is of an output of a respective said quantized activation function in one layer of said plurality of layers with respect to an input of said respective quantized activation function and is calculated such that when an absolute value of said input is smaller than a positive constant threshold value, said respective neuron gradient is set as a positive constant value and when the absolute value of said input is smaller than said positive constant threshold value said neuron gradient is set to zero, and updating a plurality of floating point connection weight values according to said plurality of weight gradients; and outputting a trained quantized neural network formed as an outcome of said training process. 16. A method for using a trained neural network, comprising: providing a neural network trained by a training set and having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight value selected from a second finite set; receiving new data element; using said neural network trained for inferencing a conclusion regarding said data element by at least one forward pass process during which only bitwise operations are held to multiply respective said quantized connection weight values and respective said quantized activation values; and outputting said conclusion. 17. The method of claim 16, wherein said bitwise operations are XNOR-count operations and population count, operations.