title: Read, compress, write with multiprocessing
body: <p>I'm compressing files. A single process is fine for a few of them, but I'm compressing thousands of them and this can (and has) taken several days, so I’d like to speed it up with multiprocessing. I’ve <a href="https://stackoverflow.com/a/2069556">read</a> that I should avoid having multiple processes reading files at the same time, and I’m guessing I shouldn’t have multiple processes writing at once as well. This is my current method that runs singly:</p>
  
  <pre><code>import tarfile, bz2, os
  def compress(folder):
      "compresses a folder into a file"
  
      bz_file = bz2.BZ2File(folder+'.tbz', 'w')
  
      with tarfile.open(mode='w', fileobj = bz_file) as tar:
  
          for fn in os.listdir(folder):
  
              read each file in the folder and do some pre processing
              that will make the compressed file much smaller than without
  
              tar.addfile( processed file )
  
      bz_file.close()
      return
  </code></pre>
  
  <p>This is taking a folder and compressing all its contents into a single file. This makes them easier to handle and more organized. If I just tossed this into a pool, then I’d have several processes reading and writing all at once, so I want to avoid that. I can rework it so only one process is reading the files but I still have multiple ones writing:</p>
  
  <pre><code>import multiprocessing as mp
  import tarfile, bz2, os
  
  def compress(file_list):
      folder = file_list[0]
      bz_file = bz2.BZ2File(folder+'.tbz', 'w')
  
      with tarfile.open(mode='w', fileobj = bz_file) as tar:
  
          for i in file_list[1:]:
              preprocess file data
              tar.addfile(processed data)
  
      bz_file.close()
      return
  
  cpu_count = mp.cpu_count()
  p = mp.Pool(cpu_count)
  
  for subfolder in os.listdir(main_folder):
  
      read all files in subfolder into memory, place into file_list
      place file_list into fld_list until fld_list contains cpu_count
      file lists. then pass to  p.map(compress, fld_list)
  </code></pre>
  
  <p>This still has a number of processes writing compressed files at once. Just the act of telling tarfile what kind of compression to use starts writing to the hard drive. I cannot read all the files I need to compress into memory as I don’t have that amount of RAM to do so – so it also has the issue that I’m restarting Pool.map many times.</p>
  
  <p>How can I read and write files in a single process, yet have all the compression in several processes, while avoiding restarting multiprocessing.Pool multiple times?</p>