<p>Yes, Adam and AdamW weight decay are different.</p>
<blockquote>
<p>Hutter pointed out in their paper (<a href="https://arxiv.org/abs/1711.05101" rel="noreferrer">Decoupled Weight Decay Regularization</a>) that the way weight decay is implemented in Adam in every library seems to be wrong, and proposed a simple way (which they call AdamW) to fix it.</p>
</blockquote>
<p>In Adam, the weight decay is usually implemented by adding <code>wd*w</code> (<code>wd</code> is weight decay here) to the gradients (Ist case), rather than actually subtracting from weights (IInd case).</p>
<pre><code># Ist: Adam weight decay implementation (L2 regularization)
final_loss = loss + wd * all_weights.pow(2).sum() / 2
# IInd: equivalent to this in SGD
w = w - lr * w.grad - lr * wd * w
</code></pre>
<blockquote>
<p>These methods are same for vanilla SGD, but as soon as we add momentum, or use a more sophisticated optimizer like Adam, L2 regularization (first equation) and weight decay (second equation) become different.</p>
</blockquote>
<p>AdamW follows the second equation for weight decay.</p>
<p>In Adam</p>
<blockquote>
<p>weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)</p>
</blockquote>
<p>In AdamW</p>
<blockquote>
<p>weight_decay (float, optional) – weight decay coefficient (default: 1e-2)</p>
</blockquote>
<p>Read more on the <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" rel="noreferrer">fastai blog</a>.</p>
AdamW and Adam with weight decay
<p>Is there any difference between <code>torch.optim.Adam(weight_decay=0.01)</code> and <code>torch.optim.AdamW(weight_decay=0.01)</code>?</p>
<p>Link to the docs: <a href="https://pytorch.org/docs/stable/optim.html" rel="noreferrer">torch.optim</a></p>