title: MPI + GPU : how to mix the two techniques
body: <p>My program is well-suited for MPI.  Each CPU does its own, specific (sophisticated) job, produces a single <code>double</code>, and then I use an <code>MPI_Reduce</code> to multiply the result from every CPU.</p>
  
  <p>But I repeat this many, many times (> 100,000).  Thus, it occurred to me that a GPU would dramatically speed things up.</p>
  
  <p>I have google'd around, but can't find anything concrete.  How do you go about mixing MPI with GPUs?  Is there a way for the program to query and verify <strong>"oh, <em>this</em> rank is the GPU, all other are CPUs" ?</strong>  Is there a recommended tutorial or something?</p>
  
  <p>Importantly, I don't want or need a full set of GPUs.  I really just need a lot of CPUs, and then a single GPU to speed up the frequently-used <code>MPI_Reduce</code> operation.</p>
  
  <p>Here is a schematic example of what I'm talking about:</p>
  
  <p>Suppose I have 500 CPUs.  Each CPU somehow produces, say, 50 <code>double</code>s.  I need to multiply all 250,00 of these <code>double</code>s together.  Then I repeat this between 10,000 and 1 million times.  If I could have one GPU (in addition to the 500 CPUs), this could be really efficient.  Each CPU would compute its 50 <code>double</code>s for all ~1 million "states".  Then, all 500 CPUs would send their <code>double</code>s to the GPU.  The GPU would then multiply the 250,000 <code>double</code>s together for each of the 1 million "states", producing 1 million <code>doubles</code>.<br>
  These numbers are not exact.  The compute is indeed very large.  I'm just trying to convey the general problem.</p>