<p>Keras correctly implements L1 regularization. In the context of neural networks, L1 regularization simply adds the L1 norm of the parameters to the loss function (see <a href="http://cs231n.github.io/neural-networks-2/#reg" rel="nofollow noreferrer">CS231</a>).</p>

<p>While L1 regularization does encourages sparsity, it does not guarantee that output will be sparse. The parameter updates from stochastic gradient descent are inherently noisy. Thus, the probability that any given parameter is exactly 0 is vanishingly small.</p>

<p>However, many of the parameters of an L1 regularized network are often close to 0. A rudimentary approach would be to threshold small values to 0. There has been research to explore more advanced methods of generating sparse neural network. In <a href="https://arxiv.org/abs/1611.06694" rel="nofollow noreferrer">this paper</a>, the authors simultaneously prune and train a neural network to achieve 90-95% sparsity on a number of well known network architectures. </p>
Is the L1 regularization in Keras/Tensorflow *really* L1-regularization?
<p>I am employing L1 regularization on my neural network parameters in Keras with <code>keras.regularizers.l1(0.01)</code> to obtain a sparse model. I am finding that, while many of my coefficients are <em>close</em> to zero, few of them are actually zero.</p>

<p>Upon looking at <a href="https://github.com/fchollet/keras/blob/master/keras/regularizers.py" rel="noreferrer">the source code for the regularization</a>, it suggests that Keras simply adds the L1 norm of the parameters to the loss function.</p>

<p>This would be incorrect because the parameters would almost certainly never go to zero (within floating point error) as intended with L1 regularization. The L1 norm is not differentiable when a parameter is zero, so subgradient methods need to be used where the parameters are set to zero if close enough to zero in the optimization routine. See the soft threshold operator <code>max(0, ..)</code> <a href="https://en.wikipedia.org/wiki/Lasso_(statistics)#Orthonormal_covariates" rel="noreferrer">here</a>.</p>

<p>Does Tensorflow/Keras do this, or is this impractical to do with stochastic gradient descent?</p>

<p>EDIT: Also <a href="http://jocelynchi.com/soft-thresholding-operator-and-the-lasso-solution" rel="noreferrer">here</a> is a superb blog post explaining the soft thresholding operator for L1 regularization.</p>