title: How to run a Cuda kernel call and CPU function in parallel?
body: <p>I have a program running on GPU using CUDA with a lot of small kernels, meaning that the kernel call on my CPU needs about the same time as the kernel execution on my GPU. </p>
  
  <p>I would like to add a CPU function to my program loop that needs about the same time as one iteration of all my kernels. I know that after a kernel launch, the CPU can work asynchronous to the GPU but because my last kernel launch is not much ahead of the GPU work being done, this is no option in this case. </p>
  
  <p>So, my idea was to use multiple threads:
  One thread to launch my GPU kernels and another one (or multiple other ones) to execute the CPU function and run those two in parallel.</p>
  
  <p>I created a small example to test this idea:</p>
  
  <pre><code>#include &lt;unistd.h&gt;
  #include &lt;cuda_runtime.h&gt;
  #include &lt;cuda_profiler_api.h&gt;
  
  #define THREADS_PER_BLOCK 64
  
  __global__ void k_dummykernel1(const float* a, const float* b, float* c, const int N)
  {
      const int id = blockIdx.x * blockDim.x + threadIdx.x;
      if(id &lt; N)
      {
          float ai = a[id];
          float bi = b[id];
  
          c[id] = powf(expf(bi*sinf(ai)),1.0/bi);
      }
  }
  
  __global__ void k_dummykernel2(const float* a, const float* b, float* c, const int N)
  {
      const int id = blockIdx.x * blockDim.x + threadIdx.x;
      if(id &lt; N)
      {
          float bi = b[id];
  
          c[id] = powf(c[id],bi);
      }
  }
  
  __global__ void k_dummykernel3(const float* a, const float* b, float* c, const int N)
  {
      const int id = blockIdx.x * blockDim.x + threadIdx.x;
      if(id &lt; N)
      {
          float bi = b[id];
  
          c[id] = logf(c[id])/bi;
      }
  }
  
  __global__ void k_dummykernel4(const float* a, const float* b, float* c, const int N)
  {
      const int id = blockIdx.x * blockDim.x + threadIdx.x;
      if(id &lt; N)
      {
  
          c[id] = asinf(c[id]);
      }
  }
  
  int main()
  {
      int N = 10000;
      int N2 = N/5;
  
      float *a = new float[N];
      float *b = new float[N];
      float *c = new float[N];
  
      float *d_a,*d_b,*d_c;
  
      for(int i = 0; i &lt; N; i++)
      {
          a[i] = (10*(1+i))/(float)N;
          b[i] = (i+1)/50.0;
      }
  
  
  
      cudaMalloc((void**)&amp;d_a,N*sizeof(float));
      cudaMalloc((void**)&amp;d_b,N*sizeof(float));
      cudaMalloc((void**)&amp;d_c,N*sizeof(float));
  
      cudaMemcpy(d_a, a ,N*sizeof(float), cudaMemcpyHostToDevice);
      cudaMemcpy(d_b, b ,N*sizeof(float), cudaMemcpyHostToDevice);
  
  
      cudaProfilerStart();
  
  
      for(int k = 0; k &lt; 100; k++)
      {
  
          k_dummykernel1&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel2&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel3&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel4&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
  
          k_dummykernel1&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel2&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel3&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
          k_dummykernel4&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
  
          for(int i = 0; i &lt; N2; i++)
          {
              c[i] = pow(a[i],b[i]);
          }
  
      }
  
      cudaDeviceSynchronize();
      usleep(40000);
  
      for(int k = 0; k &lt;= 100; k++)
      {
  
  #pragma omp parallel sections num_threads(2)
          {
  #pragma omp section
              {
                  k_dummykernel1&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel2&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel3&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel4&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
  
                  k_dummykernel1&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel2&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel3&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
                  k_dummykernel4&lt;&lt;&lt;(N + THREADS_PER_BLOCK - 1)/THREADS_PER_BLOCK, THREADS_PER_BLOCK&gt;&gt;&gt;(d_a,d_b,d_c,N);
              }
  
  #pragma omp section
              {
                  for(int i = 0; i &lt; N2; i++)
                  {
                      c[i] = pow(a[i],b[i]);
                  }
              }
          }
      }
  
      cudaDeviceSynchronize();
  
      cudaProfilerStop();
  
      delete[] a;
      delete[] b;
      delete[] c;
  
      cudaFree((void*)d_a);
      cudaFree((void*)d_b);
      cudaFree((void*)d_c);
  }
  </code></pre>
  
  <p>I compile using: <code>nvcc main.cu -O3 -Xcompiler -fopenmp</code></p>
  
  <p>First, I run 2x4 kernels and the CPU computation sequential and after that, I tried to do it in parallel using OpenMP sections.</p>
  
  <p>This is the result in the profiler:
  <a href="https://i.stack.imgur.com/wgG3U.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wgG3U.png" alt="Complete Profiler Timeline"></a></p>
  
  <p>The parallel version is much slower than the sequential one...</p>
  
  <p>If I zoom into the sequential part, it looks like this:
  <a href="https://i.stack.imgur.com/JLg3b.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JLg3b.png" alt="Sequential Timeline"></a></p>
  
  <p>One can see that between each 8 kernel launches there is a gap, where the cpu computations are done (this one I would like to close by overlapping it with the kernel calls).</p>
  
  <p>If I zoom into the parallel part (same zoom level!), it looks like this:
  <a href="https://i.stack.imgur.com/akNOR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/akNOR.png" alt="Parallel Timeline"></a></p>
  
  <p>There are no gaps anymore but the kernel launches now need about 15 microseconds (vs 5microseconds before).</p>
  
  <p>I also tried bigger array sizes and <code>std::thread</code> instead of OpenMP but the problem is always the same as before.</p>
  
  <p>Can someone tell me, if this is even possible to get to work and if yes, what am I doing wrong?</p>
  
  <p>Thanks in advance</p>
  
  <p>Cat</p>