<blockquote>
<p>I've always used minimum runtime over multiple runs because any deviation from the minimum will be due to the CPU being busy with unrelated things, but I couldn't find any reliable sources confirming that that's the best practice. Other obvious choices are average or median run time. (Maximum seems odd, as it will probably be dominated by unrelated CPU spikes.) Are there any better ways to make sense of the statistical data gathered from several runs?</p>
</blockquote>
<p><a href="https://arxiv.org/abs/1608.04295" rel="nofollow noreferrer">Chen, J. and Revels, J., 2016. Robust benchmarking in noisy environments. arXiv preprint arXiv:1608.04295.</a> has some moderately extensive and mathematically justified arguments about how to do robust benchmarking.
The simple version is: use minimum.</p>
<p>As you expect because it boils down to:
<em>noise can only ever make things run slower, never faster.</em>
as such <code>minimum</code> not <code>mean</code> or <code>median</code> is the best estimator of the true time.
Since the model is <code>true_time + noise</code> where <code>noise</code> is nonnegative, and thus the sample that is smallest must have the smallest error (since <code>true_time</code> doessn't chane between samples, only noise).</p>
Which metric should be used for benchmarks?
<p>When I'm benchmarking some code, either on a fixed set of input data, or on random input where the randomness doesn't affect the control flow: what is the best metric to use to assess the performance of my code?</p>

<p>I've always used minimum runtime over multiple runs because any deviation from the minimum will be due to the CPU being busy with unrelated things, but I couldn't find any reliable sources confirming that that's the best practice. Other obvious choices are average or median run time. (Maximum seems odd, as it will probably be dominated by unrelated CPU spikes.) Are there any better ways to make sense of the statistical data gathered from several runs?</p>

<p>As paxdiablo points out, if I can measure CPU time directly that would be ideal. But what do I do when I can only benchmark wall time?</p>

<p>As I said I was unable to find anything reliable on this, but maybe I just didn't find the right Google keywords, so if you can point me to anything existing, that would already be a great help. Also, please feel free to migrate this to Programmers.SE, if this question is too general for SO.</p>