title: Avoid global variables for unpicklable shared state among multiprocessing.Pool workers
body: <p>I frequently find myself writing programs in Python that construct a large (megabytes) read-only data structure and then use that data structure to analyze a very large (hundreds of megabytes in total) list of small records.  Each of the records can be analyzed in parallel, so a natural pattern is to set up the read-only data structure and assign it to a global variable, then create a <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool" rel="noreferrer"><code>multiprocessing.Pool</code></a> (which implicitly copies the data structure into each worker process, via <code>fork</code>) and then use <code>imap_unordered</code> to crunch the records in parallel.  The skeleton of this pattern tends to look like this:</p>
  
  <pre><code>classifier = None
  def classify_row(row):
      return classifier.classify(row)
  
  def classify(classifier_spec, data_file):
      global classifier
      try:
          classifier = Classifier(classifier_spec)
          with open(data_file, "rt") as fp, \
               multiprocessing.Pool() as pool:
              rd = csv.DictReader(fp)
              yield from pool.imap_unordered(classify_row, rd)
      finally:
          classifier = None
  </code></pre>
  
  <p>I'm not happy with this because of the global variable and the implicit coupling between <code>classify</code> and <code>classify_row</code>.  Ideally, I would like to write</p>
  
  <pre><code>def classify(classifier_spec, data_file):
      classifier = Classifier(classifier_spec)
      with open(data_file, "rt") as fp, \
           multiprocessing.Pool() as pool:
          rd = csv.DictReader(fp)
          yield from pool.imap_unordered(classifier.classify, rd)
  </code></pre>
  
  <p>but this does not work, because the Classifier object usually contains objects which cannot be pickled (because they are defined by extension modules whose authors didn't care about that); I have also read that it would be really slow if it did work, because the Classifier object would get copied into the worker processes on <em>every invocation</em> of the bound method.</p>
  
  <p>Is there a better alternative?  I only care about 3.x.</p>