title: Python multiprocessing with dataframe and multiple arguments
body: <p>According to <a href="https://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments">this answer</a> when multiprocessing with multiple arguments <a href="https://docs.python.org/3/library/multiprocessing.html" rel="nofollow noreferrer">starmap</a> should be used. The problem I am having is that one of my arguments is a constant dataframe. When I create a list of arguments to be used by my function and starmap the dataframe gets stored over and over. I though I could get around this problem using <a href="https://stackoverflow.com/questions/22487296/multiprocessing-in-python-sharing-large-object-e-g-pandas-dataframe-between">namespace</a>, but can't seem to figure it out. My code below hasn't thrown an error, but after 30 minutes no files have written. The code runs in under 10 minutes without using multiprocessing and just calling <code>write_file</code> directly.</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import multiprocessing as mp
  
  def write_file(df, colIndex, splitter, outpath):
      with open(outpath + splitter + ".txt", 'a') as oFile:
          data = df[df.iloc[:,colIndex] == splitter]
          data.to_csv(oFile, sep = '|', index = False, header = False)
  
  mgr = mp.Manager()
  ns = mgr.Namespace()
  df = pd.read_table(file_, delimiter = '|', header = None)
  ns.df = df.iloc[:,1] = df.iloc[:,1].astype(str)
  fileList = list(df.iloc[:, 1].astype('str').unique())
  for item in fileList:
      with mp.Pool(processes=3) as pool:
          pool.starmap(write_file, np.array((ns, 1, item, outpath)).tolist())
  </code></pre>