title: benchmarking, code reordering, volatile
body: <p>I decide I want to benchmark a particular function, so I naïvely write code like this:</p>
  
  <pre><code>#include &lt;ctime&gt;
  #include &lt;iostream&gt;
  
  int SlowCalculation(int input) { ... }
  
  int main() {
      std::cout &lt;&lt; "Benchmark running..." &lt;&lt; std::endl;
      std::clock_t start = std::clock();
      int answer = SlowCalculation(42);
      std::clock_t stop = std::clock();
      double delta = (stop - start) * 1.0 / CLOCKS_PER_SEC;
      std::cout &lt;&lt; "Benchmark took " &lt;&lt; delta &lt;&lt; " seconds, and the answer was "
                &lt;&lt; answer &lt;&lt; '.' &lt;&lt; std::endl;
      return 0;
  }
  </code></pre>
  
  <p>A colleague pointed out that I should declare the <code>start</code> and <code>stop</code> variables as <code>volatile</code> to avoid code reordering.  He suggested that the optimizer could, for example, effectively reorder the code like this:</p>
  
  <pre><code>    std::clock_t start = std::clock();
      std::clock_t stop = std::clock();
      int answer = SlowCalculation(42);
  </code></pre>
  
  <p>At first I was skeptical that such extreme reordering was allowed, but after some research and experimentation, I learned that it was.</p>
  
  <p>But volatile didn't feel like the right solution; isn't volatile really just for memory mapped I/O?</p>
  
  <p>Nevertheless, I added <code>volatile</code> and found that not only did the benchmark take significantly longer, it also was wildly inconsistent from run to run.  Without volatile (and getting lucky to ensure the code wasn't reordered), the benchmark consistently took 600-700 ms.  With volatile, it often took 1200 ms and sometimes more than 5000 ms.  The disassembly listings for the two versions showed virtually no difference other than a different selection of registers.  This makes me wonder if there is another way to avoid the code reordering that doesn't have such overwhelming side effects.</p>
  
  <p>My question is:</p>
  
  <blockquote>
    <p>What is the best way to prevent code reordering in benchmarking code like this?</p>
  </blockquote>
  
  <p>My question is similar to <a href="https://stackoverflow.com/questions/6130100/using-volatile-to-prevent-compiler-optimization-in-benchmarking-code">this one</a> (which was about using volatile to avoid elision rather than reordering), <a href="https://stackoverflow.com/questions/4247586/benchmarking-using-ctime-and-instruction-reordering">this one</a> (which didn't answer how to prevent reordering), and <a href="https://stackoverflow.com/questions/9712192/can-compiler-reorder-code-over-calls-to-stdchronosystem-clocknow">this one</a> (which debated whether the issue was code reordering or dead code elimination).  While all three are on this exact topic, none actually answer my question.</p>
  
  <p><strong>Update</strong>: The answer appears to be that my colleague was mistaken and that reordering like this isn't consistent with the standard.  I've upvoted everyone who said so and am awarding the bounty to the Maxim.</p>
  
  <p>I've seen one case (based on the code in <a href="https://stackoverflow.com/questions/14965553/difficulties-to-measure-c-c-performance/14981627#14981627">this question</a>) where Visual Studio 2010 reordered the clock calls as I illustrated (only in 64-bit builds).  I'm trying to make a minimal case to illustrate that so that I can file a bug on Microsoft Connect.</p>
  
  <p>For those who said that volatile should be much slower because it forces reads and writes to memory, this isn't quite consistent with the code being emitted.  In my answer on <a href="https://stackoverflow.com/questions/14965553/difficulties-to-measure-c-c-performance/14981627#14981627">this question</a>, I show the disassembly for the the code with and without volatile.  Inside the loop, everything is kept in registers.  The only significant differences appear to be register selection.  I do not understand x86 assembly well enough to know why the performance of the non-volatile version is <em>consistently</em> fast while the volatile version is <em>inconsistently</em> (and sometimes dramatically) slower.</p>