title: How can I increase memory access speed in a GPU? (CUDA)
body: <p>First I need to describe the application for which I am using CUDA.</p>
  
  <p>It is a fairly standard heat flow simulation. I take a bunch of 3D arrays of floats (mutable temperature and heat, and constant k value, sar_value, and a few others) and copy them into linear arrays allocated on the GPU. All of these are in global memory.</p>
  
  <p>Next, I launch a kernel function that computes heat flow. I launch this kernel with a 2 dimensional block construct and a 1D thread construct. The block corresponds to x and y coordinates of the simulation cube that we are performing heat flow calculations on. The thread corresponds the the z coordinate. All thread/block coordinates are multiples of the total cube size, as to maximize performance.</p>
  
  <p>Next, on each cell I perform a lengthy calculation. All of the arrays are linear, so I have prepped offsets to compute the next cell in the z, y and x directions. The bulk of spatial locality occurs in write/read memory, so texture memory is not an option there. In total, per calculation there are 2 writes to large arrays, 6 constant large array reads (as in one index of a 300 MB array of floats), 8 mutable large array reads, 6 constant small array reads (as in cube root of 300 MB). All of this occurs in two lines of code. I didn't include multiple reads of the same memory location as separate reads, as I am assuming that they are cached.</p>
  
  <p>Second I will describe the results I have been having with this calculation.</p>
  
  <p>I get about 225 million cells/second on a Tesla C1060. On large data sets (40-60 million cells), I see no difference in performance between launching 1 thread per cell vs 1 thread per 2 cells vs 1 thread per 4 cells, all the way up to a certain point. This indicates to me that the limiting factor with the calculation is the actual memory fetching. When I launch 1 thread for multiple blocks, I relieve the memory overload on the system and so each calculation is faster, although the calculations are less parallel - netting to no performance gain, + or - a percent or two.</p>
  
  <p>What have I tried? I have tried putting my most spatially local constant large array into 3D texture memory - disastrous 3-4x slowdown. I have deemed constant memory to be not viable because the data access pattern is such that each index in the large array is only accessed once or twice, and besides, I do not necessarily know the size of the input at compile time. I have tried 1D textures on the large constant arrays; also bad.</p>
  
  <p>Is there anything more I can do? Also, if you can see the number of bytes fetches per second (225 million/sec * 100 or so bytes), it is well within, by roughly a multiple of 10, the memory bandwidth of a Tesla C1060. Why is the memory the limiting factor? I saw somewhere that someone "tiled" their data set for a similar heat flow calculation (I think in a paper by the people behind "Mint"), what does this mean?</p>
  
  <p>Thank you for any answers. Please feel free to ask any questions in the comments section.</p>