<p><a href="https://stackoverflow.com/a/44225502/5728089">Salvador Dali's answer</a> already explains about the differences between some popular methods (i.e. optimizers), but I would try to elaborate on them some more.<br/>
(Note that our answers disagree about some points, especially regarding ADAGRAD.)</p>

<h3>Classical Momentum (CM) vs Nesterov's Accelerated Gradient (NAG)</h3>

<p>(Mostly based on section 2 in the paper <a href="http://proceedings.mlr.press/v28/sutskever13.pdf" rel="noreferrer">On the importance of initialization and momentum in deep learning</a>.)<br/></p>

<p>Each step in both CM and NAG is actually composed of two sub-steps:</p>

<ul>
<li>A momentum sub-step - This is simply a fraction (typically in the range <code>[0.9,1)</code>) of the last step.</li>
<li>A gradient dependent sub-step - This is like the usual step in <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="noreferrer">SGD</a> - it is the product of the learning rate and the vector opposite to the gradient, while the gradient is computed where this sub-step starts from.</li>
</ul>

<p>CM takes the gradient sub-step first, while NAG takes the momentum sub-step first.</p>

<p>Here is a demonstration from <a href="https://stats.stackexchange.com/a/368179/215801">an answer about intuition for CM and NAG</a>:<br/></p>

<p><a href="https://i.stack.imgur.com/pAwIf.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pAwIf.png" alt="CM vs NAG example"></a></p>

<p>So NAG seems to be better (at least in the image), but why?<br/></p>

<p>The important thing to note is that it doesn't matter when the momentum sub-step comes - it would be the same either way. Therefore, we might as well behave is if the momentum sub-step has already been taken.<br/></p>

<p>Thus, the question actually is: Assuming that the gradient sub-step is taken after the momentum sub-step, should we calculate the gradient sub-step as if it started in the position before or after taking the momentum sub-step?<br/></p>

<p>"After it" seems like the right answer, as generally, the gradient at some point <code>θ</code> roughly points you in the direction from <code>θ</code> to a minimum (with the relatively right magnitude), while the gradient at some other point is less likely to point you in the direction from <code>θ</code> to a minimum (with the relatively right magnitude).<br/></p>

<p>Here is a demonstration (from the gif below):</p>

<p><a href="https://i.stack.imgur.com/T7Ori.png" rel="noreferrer"><img src="https://i.stack.imgur.com/T7Ori.png" alt="CM vs NAG in a specific moment in the awesome gif"></a></p>

<ul>
<li>The minimum is where the star is, and the curves are <a href="https://en.wikipedia.org/wiki/Contour_line" rel="noreferrer">contour lines</a>. (For an explanation about contour lines and why they are perpendicular to the gradient, see videos <a href="https://www.youtube.com/watch?v=WsZj5Rb6do8" rel="noreferrer">1</a> and <a href="https://www.youtube.com/watch?v=ZTbTYEMvo10" rel="noreferrer">2</a> by the legendary <a href="https://www.youtube.com/c/3blue1brown" rel="noreferrer">3Blue1Brown</a>.)</li>
<li>The (long) purple arrow is the momentum sub-step.</li>
<li>The transparent red arrow is the gradient sub-step if it starts before the momentum sub-step.</li>
<li>The black arrow is the gradient sub-step if it starts after the momentum sub-step.</li>
<li>CM would end up in the target of the dark red arrow.</li>
<li>NAG would end up in the target of the black arrow.</li>
</ul>

<p>Note that this argument for why NAG is better is independent of whether the algorithm is close to a minimum.<br/>
In general, both NAG and CM often have the problem of accumulating more momentum than is good for them, so whenever they should change direction, they have an embarrassing "response time". The advantage of NAG over CM that we explained doesn't prevent the problem, but only makes the "response time" of NAG less embarrassing (but embarrassing still).<br/></p>

<p>This "response time" problem is beautifully demonstrated in the gif by <a href="https://twitter.com/AlecRad" rel="noreferrer">Alec Radford</a> (which appeared in <a href="https://stackoverflow.com/a/44225502/5728089">Salvador Dali's answer</a>):<br/>
<a href="https://i.stack.imgur.com/ihlS2.gif" rel="noreferrer"><img src="https://i.stack.imgur.com/ihlS2.gif" alt="An example of the embarrassing response time of both momentum methods"></a></p>

<h3>ADAGRAD</h3>

<p>(Mostly based on section 2.2.2 in <a href="https://arxiv.org/abs/1212.5701" rel="noreferrer">ADADELTA: An Adaptive Learning Rate Method</a> (the original ADADELTA paper), as I find it much more accessible than <a href="http://www.jmlr.org/papers/v12/duchi11a.html" rel="noreferrer">Adaptive Subgradient Methods for Online Learning and Stochastic Optimization</a> (the original ADAGRAD paper).)<br/></p>

<p>In <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="noreferrer">SGD</a>, the step is given by <code>- learning_rate * gradient</code>, while <code>learning_rate</code> is a hyperparameter.<br/>
ADAGRAD also has a <code>learning_rate</code> hyperparameter, but the actual learning rate for each component of the gradient is calculated individually.<br/>
The <code>i</code>-th component of the <code>t</code>-th step is given by:<br/></p>

<pre><code>              learning_rate 
- --------------------------------------- * gradient_i_t
  norm((gradient_i_1, ..., gradient_i_t))
</code></pre>

<p>while:</p>

<ul>
<li><code>gradient_i_k</code> is the <code>i</code>-th component of the gradient in the <code>k</code>-th step</li>
<li><code>(gradient_i_1, ..., gradient_i_t)</code> is a vector with <code>t</code> components. This isn't intuitive (at least to me) that constructing such a vector makes sense, but that's what the algorithm does (conceptually).</li>
<li><code>norm(vector)</code> is the <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm" rel="noreferrer">Eucldiean norm</a> (aka <code>l2</code> norm) of <code>vector</code>, which is our intuitive notion of length of <code>vector</code>.</li>
<li>Confusingly, in ADAGRAD (as well as in some other methods) the expression that is multiplied by <code>gradient_i_t</code> (in this case, <code>learning_rate / norm(...)</code>) is often called "the learning rate" (in fact, I called it "the actual learning rate" in the previous paragraph). I guess this is because in <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="noreferrer">SGD</a> the <code>learning_rate</code> hyperparameter and this expression are one and the same.</li>
<li>In a real implementation, some constant would be added to the denominator, to prevent a division by zero.</li>
</ul>

<p>E.g. if:</p>

<ul>
<li>The <code>i</code>-th component of the gradient in the first step is <code>1.15</code></li>
<li>The <code>i</code>-th component of the gradient in the second step is <code>1.35</code></li>
<li>The <code>i</code>-th component of the gradient in the third step is <code>0.9</code></li>
</ul>

<p>Then the norm of <code>(1.15, 1.35, 0.9)</code> is the length of the yellow line, which is:<br/>
<code>sqrt(1.15^2 + 1.35^2 + 0.9^2) = 1.989</code>.<br/>
And so the <code>i</code>-th component of the third step is: <code>- learning_rate / 1.989 * 0.9</code><br/></p>

<p><a href="https://i.stack.imgur.com/pjyp7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pjyp7.png" alt="l2 norm example"></a></p>

<p>Note two things about the <code>i</code>-th component of the step:</p>

<ol>
<li>It is proportional to <code>learning_rate</code>.</li>
<li>In the calculations of it, the norm is increasing, and so the learning rate is decreasing.</li>
</ol>

<p>This means that ADAGRAD is sensitive to the choice of the hyperparameter <code>learning_rate</code>.<br/>
In addition, it might be that after some time the steps become so small, that ADAGRAD virtually gets stuck.</p>

<h3>ADADELTA and RMSProp</h3>

<p>From the <a href="https://arxiv.org/abs/1212.5701" rel="noreferrer">ADADELTA paper</a>:</p>

<blockquote>
  <p>The idea presented in this paper was derived from ADAGRAD in order to improve upon the two main drawbacks
  of the method: 1) the continual decay of learning rates
  throughout training, and 2) the need for a manually selected
  global learning rate.</p>
</blockquote>

<p>The paper then explains an improvement that is meant to tackle the first drawback:</p>

<blockquote>
  <p>Instead of accumulating the sum of squared gradients over all
  time, we restricted the window of past gradients that are accumulated
  to be some fixed size <code>w</code> [...]. This ensures that learning continues
  to make progress even after many iterations of updates have
  been done.<br/>
  Since storing <code>w</code> previous squared gradients is inefficient,
  our methods implements this accumulation as an exponentially
  decaying average of the squared gradients.</p>
</blockquote>

<p>By "exponentially decaying average of the squared gradients" the paper means that for each <code>i</code> we compute a weighted average of all of the squared <code>i</code>-th components of all of the gradients that were calculated.<br/>
The weight of each squared <code>i</code>-th component is bigger than the weight of the squared <code>i</code>-th component in the previous step.</p>

<p>This is an approximation of a window of size <code>w</code> because the weights in earlier steps are very small.</p>

<p>(When I think about an exponentially decaying average, I like to visualize a <a href="https://en.wikipedia.org/wiki/Comet" rel="noreferrer">comet's</a> trail, which becomes dimmer and dimmer as it gets further from the comet:<br/></p>

<p><a href="https://i.stack.imgur.com/UIzkI.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UIzkI.png" alt="a comet&#39;s trail as an intuition for a moving average"></a>)</p>

<p>If you make only this change to ADAGRAD, then you will get RMSProp, which is a method that was proposed by Geoff Hinton in <a href="http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf" rel="noreferrer">Lecture 6e of his Coursera Class</a>.</p>

<p>So in RMSProp, the <code>i</code>-th component of the <code>t</code>-th step is given by:<br/></p>

<pre><code>                   learning_rate
- ------------------------------------------------ * gradient_i_t
  sqrt(exp_decay_avg_of_squared_grads_i + epsilon)
</code></pre>

<p>while:</p>

<ul>
<li><code>epsilon</code> is a hyperparameter that prevents a division by zero.</li>
<li><code>exp_decay_avg_of_squared_grads_i</code> is an exponentially decaying average of the squared <code>i</code>-th components of all of the gradients calculated (including <code>gradient_i_t</code>).</li>
</ul>

<p>But as aforementioned, ADADELTA also aims to get rid of the <code>learning_rate</code> hyperparameter, so there must be more stuff going on in it.<br/></p>

<p>In ADADELTA, the <code>i</code>-th component of the <code>t</code>-th step is given by:<br/></p>

<pre><code>  sqrt(exp_decay_avg_of_squared_steps_i + epsilon)
- ------------------------------------------------ * gradient_i_t
  sqrt(exp_decay_avg_of_squared_grads_i + epsilon) 
</code></pre>

<p>while <code>exp_decay_avg_of_squared_steps_i</code> is an exponentially decaying average of the squared <code>i</code>-th components of all of the steps calculated (until the <code>t-1</code>-th step).<br/>
<code>sqrt(exp_decay_avg_of_squared_steps_i + epsilon)</code> is somewhat similar to momentum, and according to <a href="https://arxiv.org/abs/1212.5701" rel="noreferrer">the paper</a>, it "acts as an acceleration term". (The paper also gives another reason for why it was added, but my answer is already too long, so if you are curious, check out section 3.2.)</p>

<h3>Adam</h3>

<p>(Mostly based on <a href="https://arxiv.org/abs/1412.6980" rel="noreferrer">Adam: A Method for Stochastic Optimization</a>, the original Adam paper.)</p>

<p>Adam is short for Adaptive Moment Estimation (see <a href="https://stats.stackexchange.com/a/367987/215801">this answer</a> for an explanation about the name).<br/>
The <code>i</code>-th component of the <code>t</code>-th step is given by:<br/></p>

<pre><code>                   learning_rate
- ------------------------------------------------ * exp_decay_avg_of_grads_i
  sqrt(exp_decay_avg_of_squared_grads_i) + epsilon
</code></pre>

<p>while:</p>

<ul>
<li><code>exp_decay_avg_of_grads_i</code> is an exponentially decaying average of the <code>i</code>-th components of all of the gradients calculated (including <code>gradient_i_t</code>).</li>
<li>Actually, both <code>exp_decay_avg_of_grads_i</code> and <code>exp_decay_avg_of_squared_grads_i</code> are also corrected to account for a bias toward <code>0</code> (for more about that, see section 3 in <a href="https://arxiv.org/abs/1412.6980" rel="noreferrer">the paper</a>, and also <a href="https://stats.stackexchange.com/a/234686/215801">an answer in stats.stackexchange</a>).</li>
</ul>

<p>Note that Adam uses an exponentially decaying average of the <code>i</code>-th components of the gradients where most <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="noreferrer">SGD</a> methods use the <code>i</code>-th component of the current gradient. This causes Adam to behave like "a heavy ball with friction", as explained in the paper <a href="https://arxiv.org/abs/1706.08500" rel="noreferrer">GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium</a>.<br/>
See <a href="https://datascience.stackexchange.com/a/39224/56981">this answer</a> for more about how Adam's momentum-like behavior is different from the usual momentum-like behavior.</p>
Gradient Descent vs Adagrad vs Momentum in TensorFlow
<p>I'm studying <em>TensorFlow</em> and how to use it, even if I'm not an expert of neural networks and deep learning (just the basics).</p>

<p>Following tutorials, I don't understand the real and practical differences between the three optimizers for loss. I look at the <a href="https://www.tensorflow.org/versions/r0.7/api_docs/python/train.html#optimizers" rel="noreferrer">API</a> and I understand the principles, but my questions are:</p>

<p><strong>1. When is it preferable to use one instead of the others ?</strong></p>

<p><strong>2. Are there important differences to know ?</strong></p>