<p>See <a href="https://arxiv.org/pdf/1707.09725.pdf#page=96" rel="nofollow noreferrer">my masters thesis</a> for a very similar list:</p>
<h2>Optimization algorithms for neural networks</h2>
<ul>
<li>Gradient based
<ul>
<li>Flavours of gradient descent (only first order gradient):
<ul>
<li>Stochastic gradient descent: <a href="https://i.stack.imgur.com/SZ72p.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SZ72p.png" alt="enter image description here" /></a></li>
<li>Mini-Batch gradient descent: <img src="https://i.imgur.com/JktJp2O.png" alt="" /></li>
<li>Learning Rate Scheduling:
<ul>
<li>Momentum: <img src="https://i.imgur.com/MGWost4.png" alt="" /></li>
<li><a href="https://en.wikipedia.org/wiki/Rprop" rel="nofollow noreferrer">RProp</a> and the mini-batch version RMSProp</li>
<li><a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#AdaGrad" rel="nofollow noreferrer">AdaGrad</a></li>
<li>Adadelta (<a href="http://arxiv.org/pdf/1212.5701v1.pdf" rel="nofollow noreferrer">paper</a>)</li>
<li>Exponential Decay Learning Rate</li>
<li>Performance Scheduling</li>
<li>Newbob Scheduling</li>
</ul>
</li>
<li><a href="https://en.wikipedia.org/wiki/Quickprop" rel="nofollow noreferrer">Quickprop</a></li>
<li>Nesterov Accelerated Gradient (NAG): <a href="http://cs231n.github.io/neural-networks-3/#sgd" rel="nofollow noreferrer">Explanation</a></li>
</ul>
</li>
<li>Higher order gradients
<ul>
<li><a href="https://en.wikipedia.org/wiki/Newton%27s_method" rel="nofollow noreferrer">Newton's method</a>: <a href="https://stats.stackexchange.com/a/253636/25741">Typically not possible</a></li>
<li>Quasi-Newton method
<ul>
<li>BFGS</li>
<li>L-BFGS</li>
</ul>
</li>
</ul>
</li>
<li>Unsure how it works
<ul>
<li>Adam (Adaptive Moment Estimation)
<ul>
<li>AdaMax</li>
</ul>
</li>
<li>Conjugate gradient</li>
</ul>
</li>
</ul>
</li>
<li>Alternatives
<ul>
<li>Genetic algorithms</li>
<li>Simulated Annealing</li>
<li><a href="https://martin-thoma.com/twiddle/" rel="nofollow noreferrer">Twiddle</a></li>
<li>Markov random fields (graphcut/mincut)</li>
<li>The <strong>Simplex algorithm</strong> is used for linear optimization in a operations research setting, but aparently also for neural networks (<a href="https://visualstudiomagazine.com/articles/2014/10/01/simplex-optimization.aspx" rel="nofollow noreferrer">source</a>)</li>
</ul>
</li>
</ul>
<p>You might also want to have a look at my article about <a href="https://martin-thoma.com/optimization-basics" rel="nofollow noreferrer">optimization basics</a> and at Alec Radfords nice gifs: <a href="https://imgur.com/a/Hqolp" rel="nofollow noreferrer">1</a> and <a href="https://imgur.com/s25RsOr" rel="nofollow noreferrer">2</a>, e.g.</p>
<p><img src="https://i.imgur.com/2dKCQHh.gif?1" alt="" /></p>
<p>Other interesting resources are:</p>
<ul>
<li><a href="http://sebastianruder.com/optimizing-gradient-descent/" rel="nofollow noreferrer">An overview of gradient descent optimization algorithms</a></li>
</ul>
<h2>Trade-Offs</h2>
<p>I think all of the posted optimization algorithms have some scenarios where they have advantages. The general trade-offs are:</p>
<ul>
<li>How much of an improvement do you get in one step?</li>
<li>How fast can you calculate one step?</li>
<li>How much data can the algorithm deal with?</li>
<li>Is it guaranteed to find a local minimum?</li>
<li>What requirements does the optimization algorithm have for your function? (e.g. to be once, twice or three times differentiable)</li>
</ul>
What are alternatives of Gradient Descent?
<p>Gradient Descent has a problem of Local Minima. We need run gradient descent exponential times for to find global minima. </p>

<p>Can anybody tell me about any alternatives of gradient descent with their pros and cons.</p>

<p>Thanks.</p>