<p>Although the expression "<em>Adam is RMSProp with momentum</em>" is widely used indeed, it is just a very rough shorthand description, and it should not be taken at face value; already in the original <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">Adam paper</a>, it was explicitly clarified (p. 6):</p>

<blockquote>
  <p>There are a few important differences between RMSProp with momentum and Adam: RMSProp with momentum generates its parameter updates using a momentum on the rescaled gradient, whereas Adam updates are directly estimated using a running average of first and second moment of the gradient.</p>
</blockquote>

<p>Sometimes, authors make clear that the subject expression is just a loose description, e.g. in the (highly recommended) <a href="https://ruder.io/optimizing-gradient-descent/index.html#adam" rel="nofollow noreferrer">Overview of gradient descent optimization algorithms</a> (emphasis added):</p>

<blockquote>
  <p>Adam also keeps an exponentially decaying average of past gradients mt, <strong>similar</strong> to momentum.</p>
</blockquote>

<p>or in <a href="https://cs231n.github.io/neural-networks-3/#ada" rel="nofollow noreferrer">Stanford CS231n: CNNs for Visual Recognition</a> (again, emphasis added):</p>

<blockquote>
  <p>Adam is a recently proposed update that <strong>looks a bit</strong> like RMSProp with momentum.</p>
</blockquote>

<p>That said, it's true that some other frameworks actually include a <code>momentum</code> parameter for Adam, but this is actually the <code>beta1</code> parameter; here is <a href="https://cntk.ai/pythondocs/cntk.learners.html#cntk.learners.adam" rel="nofollow noreferrer">CNTK</a>:</p>

<blockquote>
  <p><strong>momentum</strong> (float, list, output of <code>momentum_schedule()</code>) – momentum schedule. Note that this is the beta1 parameter in the Adam paper. For additional information, please refer to the <a href="https://docs.microsoft.com/en-us/cognitive-toolkit/BrainScript-SGD-Block#converting-learning-rate-and-momentum-parameters-from-other-toolkits" rel="nofollow noreferrer">this CNTK Wiki article</a>.</p>
</blockquote>

<p>So, don't take this too literally, and don't loose your sleep over it.</p>
Is Adam optimizer really RMSprop plus momentum? If yes, why it doesn't have a momentum parameter?
<p>Here is a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers" rel="nofollow noreferrer">link</a> to tensorflow optimizers. And there you can see, that RMSprop takes momentum as argument while Adam does not do this. So I am confuzed. Adam optimization pretends to be a RMSprop optimization with momentum, like this:  </p>

<p><em>Adam = RMSprop + Momentum</em>  </p>

<p>But why then RMSprop does have the momentum parameter and Adam does not?</p>