<p>It depends on your datasets and NN models, but generally, I would start with Adam. Figure 2 in this paper (<a href="http://arxiv.org/abs/1412.6980" rel="noreferrer">http://arxiv.org/abs/1412.6980</a>) shows Adam works well.</p>

<p><a href="https://i.stack.imgur.com/cWsLk.png" rel="noreferrer"><img src="https://i.stack.imgur.com/cWsLk.png" alt="enter image description here"></a></p>

<p>Also, you can see a very nice animation from
<a href="http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html" rel="noreferrer">http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html</a>.</p>

<p><a href="https://i.stack.imgur.com/88sSR.gif" rel="noreferrer"><img src="https://i.stack.imgur.com/88sSR.gif" alt="enter image description here"></a></p>
How do I choose an optimizer for my tensorflow model?
<p>Tensorflow seems to have a large collection of optimizers, is there any high level guideline (or review paper) on which one is best adapted to specific classes of loss functions ?</p>