title: multiprocessing with python-bioformats reader?
body: <p>I have a code that I want parallelized. It generally goes like this:</p>
  <pre><code>import javabridge as jb
  import bioformats as bf
  from google.cloud import storage
  import tempfile
  
  def process_image(blob):
      storage_client = storage.Client() 
      bucket = storage_client.bucket(&quot;lab_olympus_images&quot;)
  
      with tempfile.TemporaryDirectory() as tmp_dir:
          filename = tmp_dir + '/' + blob.split('/')[-1]
          blob = bucket.blob(blob)
          blob.download_to_filename(filename)
          with bf.ImageReader(filename, perform_init=True) as rdr:
              imagearray = rdr.read(c=0, z=0, t=0)
      eightbit = (255*imagearray).astype(np.uint8)
      # process images with opencv
      results = process(eightbit)
          
      return results
  
  blobs = [blob.name for blob in storage.Client().list_blobs(&quot;bucket_name&quot;)]
  if __name__ == '__main__':
      jb.start_vm(class_path.bf.JARS)
      with get_context(&quot;spawn&quot;).Pool() as pool:
          output = pool.map(process_image, blobs)
      jb.kill_vm()
  </code></pre>
  <p>I think I have boiled down the issue with javabridge.
  What I have tried:</p>
  <p>1.) like mentioned in this github issue: <a href="https://github.com/LeeKamentsky/python-javabridge/issues/176" rel="nofollow noreferrer">https://github.com/LeeKamentsky/python-javabridge/issues/176</a>, I have tried to start and kill the VM inside my function. It works on the first iteration, but fails on the next job.</p>
  <p>2.)I have tried to use the solution from this github issue: <a href="https://github.com/LeeKamentsky/python-javabridge/issues/181" rel="nofollow noreferrer">https://github.com/LeeKamentsky/python-javabridge/issues/181</a>, but shared memory runs only one at a time (I can tell because it only makes one temp folder at a time).</p>
  <pre><code>with parallel_backend(&quot;multiprocessing&quot;, n_jobs=num_cores):
      output = Parallel(require='sharedmem')(delayed(QC)(i) for i in inputs)
  </code></pre>
  <p>3.) I have tried this as well:</p>
  <pre><code>output = Parallel(n_jobs=num_cores)(delayed(QC)(i, jb, bf) for i in inputs)
  </code></pre>
  <p>It fails similarily to #1.</p>
  <p>is their a work around to this issue with javabridge?</p>
  <p>would Jpype work with python-bioformats and how would I use it for multiprocessing?</p>
  <p>I'm on the verge of giving up and trying to rewrite everything in Java.</p>