title: multithreaded combining of files (shared memory)
body: <p>Lets say I have N files in a format like this:</p>
  
  <p><strong>One File looks like this:</strong><br>
  For each time there is some amount of data with different id</p>
  
  <pre><code>- time 1:
        -  data with id: 10
        -  data with id: 13
        -  data with id: 4
  - time 2:
        -  data with id: 10
        -  data with id: 77
  ...etc
  </code></pre>
  
  <p>(for each time the data with ids from 1-1000 are spreaded <strong>some how</strong> (mixed) over these N files)</p>
  
  <p>I would like to combine all these N files so that I have a single file which is ordered :</p>
  
  <p><strong>Final File:</strong>  </p>
  
  <pre><code>- time 1:
         -  data with id: 1
         -  data with id: 2
         -  data with id: 3
         -  ...
         -  data with id: 1000
  
  - time 2:
         -  data with id: 1
         -  data with id: 2
         -  data with id: 3
         -  ...
         -  data with id: 1000
  ...etc
  </code></pre>
  
  <p>The size of data id 1-1000 is approximately 100mb, but I have a lot of times which accounts for up to 50 Gbytes of data.</p>
  
  <p>My solution for this problem would be so far like this to make this as fast as possible:</p>
  
  <p>I use <strong>T  threads</strong> on a supercomputer node (1 computer with e.g. 24-48 cores) (for example). 
  I have allocated a <strong>shared memory array to hold all datas with ids 1 - 1000 for one time</strong> (can also be more if I like)</p>
  
  <p><strong>Procedure:</strong><br>
  Step 1:  </p>
  
  <ul>
  <li>Each thread has some files it opens and owns. Each thread then fills
  in the data of the ids it has in the files into the shared array.</li>
  </ul>
  
  <p>Step 2:   </p>
  
  <ul>
  <li>When all threads have finally processed one time --> Thread 1 writes
  this array in ordered form to the final file.</li>
  </ul>
  
  <p>asdasd</p>
  
  <ol>
  <li>I would be very much interested if that is efficient? Is the
  parallel read not sequentialized anyway so it is for no use at all?
  I could compute the final file on a local computer with ultra fast SSD or on a cluster node with network storage
  (Lustres or Panasas Filesystems)   </li>
  <li>Could I use all threads again in
  step 2 to write in parallel to the disk lets say with MPI IO (which 
  supports parallel write by offsets), or how else can that be
  achieved? -> the c++ standart library?</li>
  </ol>
  
  <p>Thanks for any inputs!</p>