<p>As this is very specific question, I wouldn't go to any mathematical details of Adam. I guess in the article, the line <strong>it computes individual learning rates for different parameters</strong> got you off.</p>

<p>This is the screenshot of the actual Adam algorithm proposed in the paper <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1412.6980.pdf</a></p>

<p><a href="https://i.stack.imgur.com/UmcK2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UmcK2.png" alt="enter image description here"></a></p>

<p>Adam keeps an exponentially decaying average of past gradients so it behaves like a heavy ball with friction which helps it faster convergence and stability.</p>

<p>But, if you look into the algorithm there's an <strong>alpha</strong> (step size), this is the keras equivalent of learning rate = 0.001 we provide. So, the algorithm needs a step size to update the parameters (simply, it's a scaling factor for the weight update). As for the varying learning rate (or update), you can see the last equation (it uses <strong>m_t</strong> and <strong>v_t</strong>, these are updated in the loop) but the <strong>alpha</strong> stays fixed in the whole algorithm. This is the keras learning rate that we have to provide.</p>

<p>As, alpha stays same, we sometimes have to use learning rate scheduling where we actually decrease the learning rate after few epochs. There are other variations where we increase the learning rate first then decrease.</p>
How does the Keras Adam optimizer learning rate hyper-parameter relate to individually computed learning rates for network parameters?
<p>So through my limited understanding of Adam (mainly through this post: <a href="https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c" rel="nofollow noreferrer">https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c</a>) I gather that the Adam optimizer computes individual learning rates for each parameter in a network.</p>

<p>But in the Keras docs (<a href="https://keras.io/optimizers/" rel="nofollow noreferrer">https://keras.io/optimizers/</a>) the Adam optimizer takes a learning rate parameter.</p>

<p>My question is how does the learning rate parameter taken by the Adam object correlate to these computed learning rates? As far as I can tell, this isn't covered in the post linked (Or it is but it went over my head).</p>