<p>Here is the <a href="https://github.com/paulknysh/blackbox" rel="nofollow">solution</a> to your problem.</p>

<p>A method behind it is described in this <a href="http://arxiv.org/pdf/1605.00998.pdf" rel="nofollow">paper</a>.</p>
Finding Optimal Parameters In A "Black Box" System
<p>I'm developing machine learning algorithms which classify images based on training data.</p>

<p>During the image preprocessing stages, there are several parameters which I can modify that affect the data I feed my algorithms (for example, I can change the Hessian Threshold when extracting SURF features). So the flow thus far looks like:</p>

<p>[param1, param2, param3...] => [black box] => accuracy %</p>

<p>My problem is: <strong>with so many parameters at my disposal, how can I systematically pick values which give me optimized results/accuracy?</strong> A naive approach is to run i nested for-loops (assuming i parameters) and just iterate through all parameter combinations, but if it takes 5 minute to calculate an accuracy from my "black box" system this would take a long, long time.</p>

<p>This being said, are there any algorithms or techniques which can search for optimal parameters in a black box system? I was thinking of taking a course in Discrete Optimization but I'm not sure if that would be the best use of my time.</p>

<p>Thank you for your time and help!</p>

<p><strong>Edit</strong> (to answer comments):
I have 5-8 parameters. Each parameter has its own range. One parameter can be 0-1000 (integer), while another can be 0 to 1 (real number). Nothing is stopping me from multithreading the black box evaluation.</p>

<p>Also, there are some parts of the black box that have some randomness to them. For example, one stage is using k-means clustering. Each black box evaluation, the cluster centers may change. I run k-means several times to (hopefully) avoid local optima. In addition, I evaluate the black box multiple times and find the median accuracy in order to further mitigate randomness and outliers.</p>