<p>I have applied Bayesian Optimization for hyper-parameter tuning in production, so I've faced similar issues. </p>

<p>Different Bayesian methods have different characteristics in terms of exploration/exploitation trade off, for example probability of improvement (PI) tends to exploit more by selecting the next point close to the known extremum, while <a href="http://www.jmlr.org/papers/volume3/auer02a/auer02a.pdf" rel="nofollow noreferrer">upper confidence bound</a> (UCB) on the contrary prefers exploration. The problem is the first local maximum is often not good enough to exploit, but "exploratory" methods take too much time and luck to use them alone.</p>

<p>The method that demonstrated the best results for me was a <a href="https://arxiv.org/pdf/1009.5419.pdf" rel="nofollow noreferrer">portfolio strategy</a> (also known as a mixed strategy), that essentially makes an ensemble out of other methods. For example, on each step, it can pick UCB with 40% probability, PI with 40% and plain random point with 20% probability. What is important is that all methods share the outcomes, for instance if at some point a random method selects a good candidate, it then changes the GP model for UCB and PI, so from this moment PI is more likely to exploit that point. I have sometimes noticed that even the negative yet unexpected result changed the shape of a GP significantly, which in turn affected UCB and how it explores.</p>

<p>Clearly, a portfolio distribution itself can also change over time. It makes sense to start off by exploring more and then shift to exploitation, but still leaving some chance to explore (ε-greedy in the limit).</p>

<p>As for the range selection, I preferred to make it as large as possible and let Bayesian Optimization decide which values deserve more attempts, at least in the beginning. Note that PI method doesn't care much how big your range is. UCB tends to take more attempts as the range grows. In my experience, often the correlation between certain ranges (e.g. a regularizer is less than 0.01) and the outcome (severe overfitting) became obvious after several runs, and that allowed to narrow the range for all methods. But I don't recommend "premature optimizations" like that right from the start.</p>

<p>In the end, I wrote my own library for Bayesian Optimization. If you're interested in the code, please <a href="https://github.com/maxim5/hyper-engine" rel="nofollow noreferrer">check it out on GitHub</a>.</p>
Controlled Bayesian Optimization for Hyperparameter Tuning
<p>What is the best way to use hyperparameter tuning using Bayesian Optimization with some heuristic selections to explore too?</p>

<p>In packages such as <a href="https://github.com/kuz/caffe-with-spearmint" rel="nofollow noreferrer">spearmint</a> or <a href="https://github.com/hyperopt/hyperopt" rel="nofollow noreferrer">hyperopt</a> you can specify a range to explore but I want to also explore some heuristic values that do not necessarily belong to the range. Any suggestions what' the best practice to do this?</p>