<p>Here is a good review paper on this topic "A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay" by Leslie N. Smith 
<a href="https://arxiv.org/pdf/1803.09820.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.09820.pdf</a></p>
Is there a thorough exploration of the effect of momentum on Stochastic Gradient Descent?
<p>Many CNN papers use momentum=0.9 when using Stochastic Gradient Descent in weight update. There is a good logic for using it, but what I am looking for is a thorough exploration of effects of that parameter. I've searched across many papers, and there are some insights here and there, but I have not been able a comprehensive exploration. Also, does it usefulness vary across different computer vision tasks like classification, segmentation, detection?</p>