title: Python multiprocessing pool; wait for iteration to complete
body: <p>I've got a large dataset which I want my script to iterate through, perform a series of operations on each entry, then arrange the results for storage to HDD. Because the datasets can be relatively large (~250 GB), RAM availability requires that the dataset be processed in chunks (what I've called dataBlock in the code below) of 1000 entries at a time. I also use the <code>multiprocessing.Pool</code> class to facilitate use of multiple CPU cores for this task. </p>
  
  <p>I've essentially got things arranged so that each dataBlock is passed to the Pool, the Pool executes the desired calculations on the dataBlock using the <code>imap</code> method, the Pool returns the calculation results, and the results for the data chunk are appended to a list. This list (<code>processed_data</code>) is the desired end product of the set of calculations.</p>
  
  <pre><code>processed_data = []
  
  multiprocessing.Pool(processor_cap) as pool:
  
      for blockIndex, block in enumerate(range(1000, height-remainder, 1000)):
  
          #Read-in 1000 spectra from source dataset
          dataBlock = np.asarray(raw_dset[blockIndex*1000:block][:])
  
          '''
          Pass data block to processor pool, which iterates through data
          block. Each spectrum is handed off to a CPU in the pool,
          which centroids it and appends the result to "processed_block".
          '''
          processed_block = pool.imap(centroid_spectrum, dataBlock)
  
          #Append processed spectra to processed data bin
          for idx, processed_spectrum in enumerate(processed_block):
              processed_data.append(processed_spectrum)
  </code></pre>
  
  <p>What I'd like to know is how to make the script pause after the call of <code>pool.imap()</code> until the full <code>processed_block</code> has been returned without closing the pool. Currently, it progresses right into the <code>for</code> loop which immediately follows in the snippit above without waiting for <code>processed_block</code> to be returned by <code>pool.imap</code>. I've tried calling <code>pool.join()</code> immediately after the <code>pool.imap()</code> call, but it only returns <code>***AssertionError</code> and again continues to the <code>for</code> loop below it. I can eventually successfully call <code>pool.close()</code> and <code>pool.join()</code> later in the script once all dataBlocks have been fed to the pool, just below the end of the outermost <code>for</code> loop above. </p>
  
  <p>Thanks in advance for helping! </p>