<p>Be careful when using weight decay with the vanilla Adam optimizer, as it appears that the vanilla Adam formula is wrong when using weight decay, as pointed out in the article <em>Decoupled Weight Decay Regularization</em> <a href="https://arxiv.org/abs/1711.05101" rel="nofollow noreferrer">https://arxiv.org/abs/1711.05101</a> .</p>
<p>You should probably use the AdamW variant when you want to use Adam with weight decay.</p>
Should we do learning rate decay for adam optimizer
<p>I'm training a network for image localization with Adam optimizer, and someone suggest me to use exponential decay. I don't want to try that because Adam optimizer itself decays learning rate. But that guy insists and he said he did that before. So should I do that and is there any theory behind your suggestion?</p>