<p>Note that the formula you show does not actually present true "weight decay", but instead L2-regularization. Many people mix these up, including well-known professors. Let me explain.</p>

<p>When using pure SGD (without momentum) as an optimizer, weight decay is the same thing as adding a L2-regularization term to the loss. <strong>When using any other optimizer, including Momentum, this is not true.</strong></p>

<p>Weight decay (don't know how to TeX here, so excuse my pseudo-notation):</p>

<pre><code>w[t+1] = w[t] - learning_rate * dw - weight_decay * w
</code></pre>

<p>L2-regularization:</p>

<pre><code>loss = actual_loss + lambda * 1/2 sum(||w||_2 for w in network_params)
</code></pre>

<p>Computing the gradient of the extra term in L2-regularization gives <code>lambda * w</code> and thus inserting it into the SGD update equation</p>

<pre><code>dloss_dw = dactual_loss_dw + lambda * w
w[t+1] = w[t] - learning_rate * dw
</code></pre>

<p>gives the same as weight decay, but mixes <code>lambda</code> with the <code>learning_rate</code>. Any other optimizer, even SGD with momentum, gives a different update rule for weight decay as for L2-regularization! See the paper <a href="/https://arxiv.org/abs/1711.05101">Fixing weight decay in Adam</a> for more details. (Edit: AFAIK, <a href="http://www.cs.toronto.edu/~hinton/absps/parle.pdf" rel="nofollow noreferrer">this 1987 Hinton paper</a> introduced "weight decay", literally as "each time the weights are updated, their magnitude is also decremented by 0.4%" at page 10)</p>

<p>That being said, there doesn't seem to be support for "proper" weight decay in TensorFlow yet. There are a few issues discussing it, specifically because of above paper.</p>

<p>One possible way to implement it is by writing an op that does the decay step manually after every optimizer step. A different way, which is what I'm currently doing, is using an additional SGD optimizer just for the weight decay, and "attaching" it to your <code>train_op</code>. Both of these are just crude work-arounds, though. My current code:</p>

<pre><code># In the network definition:
with arg_scope([layers.conv2d, layers.dense],
               weights_regularizer=layers.l2_regularizer(weight_decay)):
    # define the network.

loss = # compute the actual loss of your problem.
train_op = optimizer.minimize(loss, global_step=global_step)
if args.weight_decay not in (None, 0):
    with tf.control_dependencies([train_op]):
        sgd = tf.train.GradientDescentOptimizer(learning_rate=1.0)
        train_op = sgd.minimize(tf.add_n(tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)))
</code></pre>

<p>This somewhat makes use of TensorFlow's provided bookkeeping. Note that the <code>arg_scope</code> takes care of appending an L2-regularization term for every layer to the <code>REGULARIZATION_LOSSES</code> graph-key, which I then all sum up and optimize using SGD which, as shown above, corresponds to actual weight-decay.</p>

<p>Hope that helps, and if anyone gets a nicer code snippet for this, or TensorFlow implements it better (i.e. in the optimizers), please share.</p>

<p><strong>Edit:</strong> see also <a href="https://github.com/tensorflow/tensorflow/pull/17438" rel="nofollow noreferrer">this PR</a> which just got merged into TF.</p>
How to set weight cost strength in TensorFlow?
<p>I'm trying to use TensorFlow with my deep learning project.  </p>

<p>When I use Momentum Gradient Descent, how is the weight cost strength set?<br>
(The λ in this <a href="http://i.stack.imgur.com/305RE.jpg" rel="noreferrer">formula</a>.) </p>