title: GPU vs CPU end to end latency for dynamic image resizing
body: <p>I have currently used OpenCV and ImageMagick for some throughput benchmarking and I am not finding working with GPU to be much faster than CPUs. Our usecase on site is to resize dynamically to the size requested from a master copy based on a service call and trying to evaluate if having GPU makes sense to resize per  service call dynamically. </p>
  
  <p>Sharing the code I wrote for OpenCV. I am running the following function for all the images stored in a folder serially and Ultimately I am running N such processes to achieve X number of image resizes.<strong>I want to understand if my approach is incorrect to evaluate or if the usecase doesn't fit typical GPU usecases. And what exactly might be limiting GPU performance. I am not even maximizing the utilization to anywhere close to 100%</strong></p>
  
  <p><strong>resizeGPU.cpp</strong>:
  {</p>
  
  <pre><code>    cv::Mat::setDefaultAllocator(cv::cuda::HostMem::getAllocator (cv::cuda::HostMem::AllocType::PAGE_LOCKED));
  
      auto t_start = std::chrono::high_resolution_clock::now();
      Mat src = imread(input_file,CV_LOAD_IMAGE_COLOR);
      auto t_end_read = std::chrono::high_resolution_clock::now();
      if(!src.data){
              std::cout&lt;&lt;"Image Not Found: "&lt;&lt; input_file &lt;&lt; std::endl;
              return;
      }
  
      cuda::GpuMat d_src;
      d_src.upload(src,stream);
      auto t_end_h2d = std::chrono::high_resolution_clock::now();
      cuda::GpuMat d_dst;
  
      cuda::resize(d_src, d_dst, Size(400, 400),0,0, CV_INTER_AREA,stream);
      auto t_end_resize = std::chrono::high_resolution_clock::now();
  
      Mat dst;
      d_dst.download(dst,stream);
      auto t_end_d2h = std::chrono::high_resolution_clock::now();
      std::cout&lt;&lt;"read,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_read-t_start).count()&lt;&lt;",host2device,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_h2d-t_end_read).count()
                              &lt;&lt;",resize,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_resize-t_end_h2d).count()
                              &lt;&lt;",device2host,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_d2h-t_end_resize).count()
              &lt;&lt;",total,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_d2h-t_start).count()&lt;&lt;endl;
  
  
  }
  </code></pre>
  
  <p><strong>resizeCPU.cpp:</strong></p>
  
  <pre><code>    auto t_start = std::chrono::high_resolution_clock::now();
      Mat src = imread(input_file,CV_LOAD_IMAGE_COLOR);
      auto t_end_read = std::chrono::high_resolution_clock::now();
      if(!src.data){
              std::cout&lt;&lt;"Image Not Found: "&lt;&lt; input_file &lt;&lt; std::endl;
              return;
      }
  
      Mat dst;
      resize(src, dst, Size(400, 400),0,0, CV_INTER_AREA);
      auto t_end_resize = std::chrono::high_resolution_clock::now();
  
      std::cout&lt;&lt;"read,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_read-t_start).count()&lt;&lt;",resize,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_resize-t_end_read).count()
          &lt;&lt;",total,"&lt;&lt;std::chrono::duration&lt;double, std::milli&gt;(t_end_resize-t_start).count()&lt;&lt;endl;
  </code></pre>
  
  <p><strong>Compiling</strong> : g++ -std=c++11 resizeCPU.cpp -o resizeCPU <code>pkg-config --cflags --libs opencv</code></p>
  
  <p>I am running each program N number of times controlled by following code : runMultipleGPU.sh</p>
  
  <pre><code>#!/bin/bash
  echo $1
  START=1
  END=$1
  for (( c=$START; c&lt;=$END; c++ ))
  do
  ./resizeGPU "$c" &amp;#&gt;/dev/null #&amp;disown;
  done
  wait
  echo All done
  </code></pre>
  
  <p>Run : ./runMultipleGPU.sh </p>
  
  <p>Those timers around lead to following aggregate data </p>
  
  <pre><code>No_processes    resizeCPU   resizeGPU   memcpyGPU   totalresizeGPU
  1                 1.51        0.55        2.13         2.68
  10                5.67        0.37        2.43         2.80
  15                6.35        2.30       12.45        14.75
  20                6.30        2.05       10.56        12.61
  30                8.09        4.57       23.97        28.55
  </code></pre>
  
  <p>No of images run per process : 267</p>
  
  <p>Average size of the image: 624Kb</p>
  
  <p>According to data above, as we increase the number of processes(leading to increased number of simultaneous resizes) the resize perform
  ance(which includes actual resize + host to device and device to host copy) increases significantly on GPU vs CPU. </p>
  
  <p>Similar results after using ImageMagick which uses OpenCL beneath</p>
  
  <p><strong>Code</strong> : </p>
  
  <pre><code>setenv("MAGICK_OCL_DEVICE","OFF",1); //Turn in ON to use GPU acceleration
  Image image;
  auto t_start_read = std::chrono::high_resolution_clock::now();
  image.read( full_path );
  auto t_end_read = std::chrono::high_resolution_clock::now();
  image.resize( Geometry(400,400) );
  auto t_end_resize = std::chrono::high_resolution_clock::now();
  </code></pre>
  
  <p><strong>Results</strong> : </p>
  
  <pre><code>No_procs    resizeCPU   resizeGPU
  1            63.23       8.54
  10           76.16      31.04
  15           76.56      50.79
  20           76.58      71.68
  30           86.29     140.17
  </code></pre>
  
  <p>Test Machine configuration: </p>
  
  <p>4 GPU (Tesla P100) - but test only utilizes 1 GPU</p>
  
  <p>64 CPU cores (over Intel Xeon 2680 v4 CPU ) </p>
  
  <p>OpenCV version : 3.4.0</p>
  
  <p>ImageMagick version : 6.9.9-26 Q16 x86_64 2018-01-17</p>
  
  <p>Cuda Toolkit : 9.0</p>