title: How is learning rate decay implemented by Adam in keras
body: <p>I set learning rate decay in my optimizer Adam, such as </p>
  
  <pre class="lang-py prettyprint-override"><code>LR = 1e-3
  LR_DECAY = 1e-2
  OPTIMIZER = Adam(lr=LR, decay=LR_DECAY)
  </code></pre>
  
  <p>As the keras document <a href="https://github.com/keras-team/keras/blob/c10d24959b0ad615a21e671b180a1b2466d77a2b/keras/optimizers.py#L433" rel="noreferrer">Adam</a> states, after each epoch learning rate would be </p>
  
  <pre><code>lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))
  </code></pre>
  
  <p>If I understand correctly, learning rate be like this,</p>
  
  <pre><code>lr = lr * 1 / ( 1 + num_epoch * decay)
  </code></pre>
  
  <p>But I don't see the learning rate decay come into effect, after seeing that printed out.
  Is there any problem when I use this ?</p>
  
  <p><strong>Edit</strong><br>
  I print out the learning by setting the verbose of the <code>ReduceLROnPlateau</code>, such as,  </p>
  
  <pre><code>ReduceLROnPlateau(monitor='val_loss', factor=0.75, patience=Config.REDUCE_LR_PATIENCE, verbose=1, mode='auto', epsilon=0.01, cooldown=0, min_lr=1e-6
  </code></pre>
  
  <p>And that would monitor the val-loss and reduce the learning rate by multiplying the <code>factor</code>.
  The printed learning rate is like this, </p>
  
  <pre><code>Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0007500000356230885.
  </code></pre>
  
  <p>And I set the initial learning rate to be 1e-3. Therefore, it appears that the learning rate change from 1e-3 to 1e-3 * 0.75, so I doubt that the <code>decay</code> I set in Adam isn't working.</p>