<p>Since the hyperparameters we're talking about are related to <strong>backpropagation</strong>, which is a gradient-based approach, I believe the main reference is <a href="http://arxiv.org/abs/1206.5533" rel="nofollow">Y. Bengio</a>, along with the more classic <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow">Lecun et al.</a>.</p>

<p>There are three main approaches to find out the optimal value for an hyperparameter. The first two are well explained in the first paper I linked.</p>

<ul>
<li>Manual search. The researcher choose the optimal value through try-and-error.</li>
<li>Automatic search. The researcher relies on an automated routine in order to speed up the search.</li>
<li>Bayesian Optimization. You can find a video presenting it <a href="https://www.youtube.com/watch?v=VG2uCpKJkSg" rel="nofollow">here</a>.</li>
</ul>
Navigating hyper-parameters
<p>I was just wondering if someone could provide a good source for me to read on how I should approach choosing hyper-parameters of the solver based on the complexity of my problem.</p>

<p>Basically, I understand that many feel that they are "shooting around in the dark" when it comes to setting and then modifying these parameters and a system or benchmark for choosing parameters based on specific problem/data complexity has escaped me.</p>

<p>If you care to explain your own methodology or simply provide commentary on your source, it would be much appreciated.</p>