title: When to call cudaDeviceSynchronize?
body: <p>when is calling to the <code>cudaDeviceSynchronize</code> function  really needed?. </p>
  
  <p>As far as I understand from the CUDA documentation, CUDA kernels are asynchronous, so it seems that we should call <code>cudaDeviceSynchronize</code> after each kernel launch.   However, I have tried the same code (training neural networks) with and without any <code>cudaDeviceSynchronize</code>, except one before the time measurement. I have found that I get the same result but with a  speed up between 7-12x (depending on the matrix sizes).  </p>
  
  <p>So, the question is if there are any reasons to use <code>cudaDeviceSynchronize</code> apart of time measurement. </p>
  
  <p>For example:</p>
  
  <ul>
  <li><p>Is it needed before copying data from the GPU back to the host with <code>cudaMemcpy</code>?</p></li>
  <li><p>If I do matrix multiplications like</p>
  
  <pre><code>C = A * B
  D = C * F
  </code></pre></li>
  </ul>
  
  <p>should I put <code>cudaDeviceSynchronize</code> between both? </p>
  
  <p>From my experiment It seems that I don't.</p>
  
  <p>Why does <code>cudaDeviceSynchronize</code> slow the program so much?</p>