title: Strange behavior of PCIe throughput when used in OpenCL
body: <p>Tested the throughput  of PCIe with OpenCL and I am getting strange results.</p>
  
  <p>I am using PCIe 3, x16.
  DATA_SIZE = 2097152 (int)
  Data size bytes = 8.388608 MB</p>
  
  <p>The code is following:</p>
  
  <pre><code>struct timeval  tv1, tv2;
  gettimeofday(&amp;tv1, NULL);
  
  err=clEnqueueWriteBuffer(the_queue, dev_buffer, CL_TRUE, 0, (size_t)sizeof(int)*DATA_SIZE, (void *)host_buffer, 0, NULL, &amp;event);
  if (err != CL_SUCCESS) {
      printf("Error:clEnqueueWriteBuffer:dbuff_in(code %d)\n\n", err );
      exit(0);
  }
  clFinish(the_queue);
  
  gettimeofday(&amp;tv2, NULL);
  
  printf ("Total time = %f seconds\n",
         (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
         (double) (tv2.tv_sec - tv1.tv_sec));
  
  err=clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_END,sizeof(cl_ulong), &amp;end, NULL);
  if (err != CL_SUCCESS) {
      printf("Error in profiling !!!Err code %d\n\n", err );
      exit(0);
  }
  
  err=clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_START,sizeof(cl_ulong), &amp;start, NULL);
  if (err != CL_SUCCESS) {
      printf("Error in profiling !!!Err code %d\n\n", err );
      exit(0);
  }
  
  transfertime=transfertime+(end - start) * 1.0e-6f;
  printf("%f  ",transfertime/1000);
  </code></pre>
  
  <p>When running this code I get the following results:</p>
  
  <p>Total time = 11.2 ms</p>
  
  <p>Transfer time = 0.02 ms</p>
  
  <p>Note: I used two different methods for measure the transfer time one is using OpenCL profiling information, other one is just measure the start of the code and end.</p>
  
  <p>Taking into consideration that PCIe3 x16 has 15.62884615 GB throughput the theoretical time would be ~0.54 ms to transfer my data.</p>
  
  <p>Using the profiling for bench-marking I got the result which is roughly 25 faster then theoretical maximum of the PCIe3, when using the standard bench-marking(start-end of the code) I got the result which is ~ 20 times less than theoretical speed.</p>
  
  <p>Could you please share your experience with this.
  What is the right way of bench-marking? And overall why the behavior is so strange?</p>
  
  <p>PS. The device is FPGA (VU9P)</p>