title: MPI-3 Shared Memory for Array Struct
body: <p>I have a simple C++ struct that basically wraps a standard C array:</p>
  
  <pre><code>struct MyArray {
      T* data;
      int length;
      // ...
  }
  </code></pre>
  
  <p>where <code>T</code> is a numeric type like <code>float</code> or <code>double</code>.  <code>length</code> is the number of elements in the array.  Typically my arrays are very large (tens of thousands up to tens of millions of elements).</p>
  
  <p>I have an MPI program where I would like to expose two instances of <code>MyArray</code>, say <code>a_old</code> and <code>a_new</code>, as shared memory objects via MPI 3 shared memory.  The context is that each MPI rank reads from <code>a_old</code>.  Then, each MPI rank writes to certain indices of <code>a_new</code> (each rank only writes to its own set of indices - no overlap).  Finally, <code>a_old = a_new</code> must be set on all ranks.  <code>a_old</code> and <code>a_new</code> are the same size.  Right now I'm making my code work by syncing (<code>Isend/Irecv</code>) each rank's updated values with other ranks.  However, due to the data access pattern, there's no reason I need to incur the overhead of message passing and could instead have one shared memory object and just put a barrier before <code>a_old = a_new</code>.  I think this would give me better performance (though please correct me if I'm wrong).</p>
  
  <p>I have had trouble finding complete code examples of doing shared memory with MPI 3.  Most sites only provide reference documentation or incomplete snippets.  Could someone walk me through a simple and <em>complete</em> code example that does the sort of thing I'm trying to achieve (updating and syncing a numeric array via MPI shared memory)?  I understand the main concepts of creating shared memory communicators and windows, setting fences, etc., but it would really help my understanding to see one example that puts it all together.</p>
  
  <p>Also, I should mention that I'll only be running my code on one node, so I don't need to worry about needing multiple copies of my shared-memory object across nodes; I just need one copy of my data for the single node on which my MPI processes are running.  Despite this, other solutions like OpenMP aren't feasible for me in this case, since I have a ton of MPI code and can't rewrite everything for the sake of one or two arrays I'd like to share.</p>