<p>I'd like to add that <a href="https://en.wikipedia.org/wiki/Bayesian_optimization" rel="nofollow noreferrer">Bayesian optimization</a> is a perfect example of an <em>adaptive random search</em>, so looks like it's exactly what you want to apply.</p>

<p>The idea of Bayesian optimization is to model the target function using <a href="https://en.wikipedia.org/wiki/Gaussian_process" rel="nofollow noreferrer">Gaussian Processes</a> (GP), select the best next point according to the current model and update the model after seeing the actual outcome. So, effectively, Bayesian optimization starts like a random search, gradually builds a picture of what the function looks like and shifts its focus to the most promising areas (note that "promising" can be defined differently by different particular methods - PI, EI, UCB, etc). There are further techniques to help it to find a right balance between exploration and exploitation, for example <a href="https://arxiv.org/pdf/1009.5419.pdf" rel="nofollow noreferrer">portfolio strategy</a>. If that's what you mean by <em>adaptive</em>, then Bayesian optimization is your choice .</p>

<p>If you'd like to extend your code without external libraries, it's totally possible because Bayesian optimization is not that hard to implement. You can take a look at sample code that I used in <a href="https://github.com/maxim5/hyper-engine" rel="nofollow noreferrer">my research</a>, for example <a href="https://github.com/maxim5/hyper-engine/blob/master/hyperengine/bayesian/utility.py" rel="nofollow noreferrer">here</a> is the bulk of GP-related code.</p>
Improvements of Random Search for Hyperparameter Optimization
<p>Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. Additional to the continuous Cost and gamma parameter, I have one discrete parameter and also an equality constraint over some parameters.</p>

<p>Now, I would like to develop random search further, e.g. through adaptive random search. That means for example adaptation of the search direction or of the search range.</p>

<p>Does somebody have an idea how this can be done or could reference to some existing work on this? Other ideas for improving random search are also welcome.</p>