<blockquote>
  <p>ADAM or adaptive momentum works as follows:</p>
</blockquote>

<p><a href="https://i.stack.imgur.com/3boio.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3boio.jpg" alt="Adam algorithm"></a></p>

<p>The velocity <code>v</code> accumulates the gradient Elements.</p>

<p>When you watch the Adam equations in this <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">paper</a> you will see, that <strong>the stepsize has an upper bound at the learning rate. In the paper they call this characteristics of Adam: "its careful choice of stepsize"</strong> (discussed in Section 2.1 of the paper).
This is exactly what you observe here as "essential equal rates" during the first 5 steps, the rates in Adam are building up (accumulating) over multiple previous gradients, while the stepsize is being restricted to the learning rate itself. </p>

<p>On more Information on how the variable is calculated and updated in Tensorflow (see equations <a href="https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer" rel="nofollow noreferrer">here</a>).</p>

<p><em>Additional remarks on Adam:</em> </p>

<p>The larger <code>α</code> is relative to the learning rate, the more previous gradients aﬀect the current direction.</p>

<p>In sgd, the size of the step was simply the norm of the gradient multiplied by the learning rate. </p>

<p>In Adam, the size of the step depends on how large and how
aligned a sequence of gradients are. The step size is largest when many successive
gradients point in exactly the same direction. If the momentum algorithm always
observes Gradient <code>g</code>, then it will eventually accelerate in the direction of <code>−g</code>.</p>

<p>This is from the Deep Learning Book by Ian Goodfellow, in more Detail you can read about Adam <a href="http://www.deeplearningbook.org/contents/optimization.html" rel="nofollow noreferrer">here</a>.</p>
Why does AdamOptimizer seem not apply the correct gradient?
<p>It's possible that I just misunderstood how Adam works, but why is this happening:</p>

<pre><code>x = tf.Variable([0.0, 0.0]) # variable
y = tf.constant([5.0, 1.0]) # target
cost = tf.abs(x-y)**2
</code></pre>

<p>As the first dimension of <code>y</code> is larger than the second, the gradient in the first dimension is larger than the second (as it should be) and each dimension of <code>x</code> approaches its target value at its own rate:</p>

<pre><code>sgd = tf.train.GradientDescentOptimizer(0.001)
train = sgd.minimize(cost)

with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for _ in range(5):
    sess.run(train)
    grad,variable = sess.run(opt.compute_gradients(cost))[0]
    print(grad,variable)

#[-9.98  -1.996] [0.01  0.002]
#[-9.96004  -1.992008] [0.01998  0.003996]
#[-9.94012  -1.988024] [0.02994004 0.00598801]
#[-9.920239  -1.9840479] [0.03988016 0.00797603]
#[-9.900399  -1.9800799] [0.0498004  0.00996008]
</code></pre>

<p>Why are the rates essentially equal if we use Adam, even though the gradients have quite different values?</p>

<pre><code>adam = tf.train.AdamOptimizer(0.001)
train = adam.minimize(cost)

with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for _ in range(5):
    sess.run(train)
    grad,variable = sess.run(opt.compute_gradients(cost))[0]
    print(grad,variable)

#[-9.998 -1.998] [0.001 0.001]
#[-9.996 -1.996] [0.00199999 0.00199997]
#[-9.994     -1.9940002] [0.00299997 0.00299989]
#[-9.992001  -1.9920005] [0.00399994 0.00399976]
#[-9.99     -1.990001] [0.0049999  0.00499955]
</code></pre>