<p>I was asking myself the exact same question, and wondering why wouldn't it change. By looking at the <a href="https://arxiv.org/pdf/1412.6980v8.pdf" rel="nofollow noreferrer">original paper</a> (page 2), one sees that the <code>self._lr</code> stepsize (designed with <code>alpha</code> in the paper) is required by the algorithm, but never updated. We also see that there is an <code>alpha_t</code> that is updated for every <code>t</code> step, and should correspond to the <code>self._lr_t</code> attribute. But in fact, as you observe, <strong>evaluating the value for the <code>self._lr_t</code> tensor at any point during the training returns always the initial value, that is, <code>_lr</code></strong>.</p>

<p>So your question, as I understood it, is <strong>how to get the <code>alpha_t</code> for TensorFlow's AdamOptimizer as described in section 2 of the paper and in the corresponding <a href="https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/train/AdamOptimizer" rel="nofollow noreferrer">TF v1.2 API page</a></strong>:</p>

<blockquote>
  <p><code>alpha_t = alpha * sqrt(1-beta_2_t) / (1-beta_1_t)</code></p>
</blockquote>

<h3>BACKGROUND</h3>

<p>As you observed, the <code>_lr_t</code> tensor doesn't change thorough the training, which may lead to the false conclusion that the optimizer doesn't adapt (this can be easily tested by switching to the <em>vanilla</em> <code>GradientDescentOptimizer</code> with the same <code>alpha</code>). And, in fact, other values do change: a quick look at the optimizer's <code>__dict__</code> shows the following keys: <code>['_epsilon_t', '_lr', '_beta1_t', '_lr_t', '_beta1', '_beta1_power', '_beta2', '_updated_lr', '_name', '_use_locking', '_beta2_t', '_beta2_power', '_epsilon', '_slots']</code>.</p>

<p>By inspecting them through training, I noticed that <strong>only <code>_beta1_power</code>, <code>_beta2_power</code> and the <code>_slots</code> get updated</strong>.</p>

<p>Further inspecting <a href="https://github.com/tensorflow/tensorflow/blob/927f811b0303e51126531c135f8b093383de2d6d/tensorflow/python/training/adam.py#L211" rel="nofollow noreferrer">the optimizer's code</a>, in line 211, we see the following update:</p>

<pre><code>update_beta1 = self._beta1_power.assign(
        self._beta1_power * self._beta1_t,
        use_locking=self._use_locking)
</code></pre>

<p>Which basically means that <code>_beta1_power</code>, which is <a href="https://github.com/tensorflow/tensorflow/blob/927f811b0303e51126531c135f8b093383de2d6d/tensorflow/python/training/adam.py#L120" rel="nofollow noreferrer">initialized with <code>_beta1</code></a>, will be multiplied by <code>_beta_1_t</code> after every iteration, which <a href="https://github.com/tensorflow/tensorflow/blob/927f811b0303e51126531c135f8b093383de2d6d/tensorflow/python/training/adam.py#L133" rel="nofollow noreferrer">is also initialized with <code>beta_1_t</code></a>.</p>

<p>But here comes the confusing part: <strong><code>_beta1_t</code> and <code>_beta2_t</code> never get updated</strong>, so effectively they hold the initial values (<code>_beta1</code>and <code>_beta2</code>) through the whole training, contradicting the notation of the paper in a similar fashion as <code>_lr</code> and <code>lr_t</code> do. <em>I guess this is for a reason but I personally don't know why, in any case this are protected/private attributes of the implementation (as they start with an underscore) and don't belong to the public interface (they may even change among TF versions).</em></p>

<p>So after this small background we can see that <code>_beta_1_power</code> and <code>_beta_2_power</code> are the original beta values exponentiated to the current training step, that is, the equivalent to the variables referred with <code>beta_t</code>in the paper. Going back to the definition of <code>alpha_t</code> in the section 2 of the paper, we see that, with this information, it should be pretty straightforward to implement:</p>

<h3>SOLUTION</h3>

<pre><code>optimizer = tf.train.AdamOptimizer()
# rest of the graph...

# ... somewhere in your session
# note that a0 comes from a scalar, whereas bb1 and bb2 come from tensors and thus have to be evaluated
a0, bb1, bb2 = optimizer._lr, optimizer._beta1_power.eval(), optimizer._beta2_power.eval()
at = a0* (1-bb2)**0.5 /(1-bb1)
print(at)
</code></pre>

<p>The variable <code>at</code> holds the <code>alpha_t</code> for the current training step.</p>

<h3>DISCLAIMER</h3>

<p>I couldn't find a cleaner way of getting this value by just using the optimizer's interface, but please let me know if it exists one! I guess there is none, which actually puts into question the usefulness of plotting <code>alpha_t</code>, since <strong>it does not depend on the data</strong>.</p>

<p>Also, to complete this information, section 2 of the paper also gives the formula for the weight updates, which is much more telling, but also more plot-intensive. For a very nice and good-looking implementation of that, you may want to take a look at <a href="https://stackoverflow.com/a/44688307/4511978">this nice answer</a> from the post that you linked.</p>

<p>Hope it helps! Cheers,<br>
Andres</p>
Learning rate doesn't change for AdamOptimizer in TensorFlow
<p>I would like to see how the learning rate changes during training (print it out or create a summary and visualize it in tensorboard).</p>

<p>Here is a code snippet from what I have so far:</p>

<pre class="lang-py prettyprint-override"><code>optimizer = tf.train.AdamOptimizer(1e-3)
grads_and_vars = optimizer.compute_gradients(loss)
train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)

sess.run(tf.initialize_all_variables())

for i in range(0, 10000):
   sess.run(train_op)
   print sess.run(optimizer._lr_t)
</code></pre>

<p>If I run the code I constantly get the initial learning rate (1e-3) i.e. I see no change.</p>

<p>What is a correct way for getting the learning rate at every step?</p>

<p>I would like to add that <a href="https://stackoverflow.com/questions/36990476/getting-the-current-learning-rate-from-a-tf-train-adamoptimizer">this question</a> is really similar to mine. However, I cannot post my findings in the comment section there since I do not have enough rep.</p>