title: What is the fastest way to perform vector-by-vector dot products for two MxN matrices with small M in CUDA?
body: <p>I have two matrices, each of which are <code>MxN</code>, where <code>M = 16</code> and <code>N</code> is much larger (say <code>n = 262144</code>, for example).  My goal is to produce a vector of length <code>N</code>, where each element corresponds to the dot product of the <code>nth</code> vector in each of the matrices.</p>
  
  <p>I have attempted the following approach, where <code>cIdx</code> corresponds to the column index of a column vector in each matrix.  Not surprisingly, NVIDIA Visual Profiler is telling me that this approach is mostly memory-bandwidth bound.</p>
  
  <pre><code>    public static void MatrixDotProduct(
          float* matrix1,
          float* matrix2,
          float* dotProduct,
          int2 matrixDimensions)
      {
          int i = blockIdx.x * blockDim.x + threadIdx.x;
          int stride = gridDim.x * blockDim.x;
          float sum;
  
          for (int cIdx = i; cIdx &lt; matrixDimensions.y; cIdx += stride)
          {
              int ci = cIdx * matrixDimensions.x;
              sum = 0f;
  
              for (int j = 0; j &lt; matrixDimensions.x; j++)
              {
                  sum += matrix1[ci + j] * matrix2[ci + j];
              }
  
              dotProduct[cIdx] = sum;
          }
      }
  </code></pre>
  
  <p>I have also found a version of a <a href="http://cuda-programming.blogspot.com/2013/01/vector-dot-product-in-cuda-c.html" rel="nofollow noreferrer">vector-by-vector</a> dot product, which I tried to parallelize as well.  Unfortunately, this ran 20% slower than the above implementation (perhaps it's because my <code>M = 16</code>?).  Is there a better way to approach this problem that I'm missing here?</p>