<p>In my experience it usually not necessary to do learning rate decay with Adam optimizer.</p>

<p>The theory is that Adam already handles learning rate optimization (<a href="http://arxiv.org/pdf/1412.6980v8.pdf" rel="noreferrer">check reference</a>) :</p>

<blockquote>
  <p>"We propose Adam, a method for efficient stochastic optimization that
  only requires first-order gradients with little memory requirement.
  The method <strong>computes individual adaptive learning rates</strong> for different
  parameters from estimates of first and second moments of the
  gradients; the name Adam is derived from adaptive moment estimation."</p>
</blockquote>

<p>As with any deep learning problem YMMV, one size does not fit all, you should try different approaches and see what works for you, etc. etc.</p>
Should we do learning rate decay for adam optimizer
<p>I'm training a network for image localization with Adam optimizer, and someone suggest me to use exponential decay. I don't want to try that because Adam optimizer itself decays learning rate. But that guy insists and he said he did that before. So should I do that and is there any theory behind your suggestion?</p>