title: Multiprocessing Pool with a for loop
body: <p>I have a list of files that I pass into a for loop and do a whole bunch of functions. Whats the easiest way to parallelize this? Not sure I could find this exact thing anywhere and I think my current implementation is incorrect because I only saw one file being run. From some reading I've done, I think this should be a perfectly parallel case.</p>
  
  <p>Old code is something like this:</p>
  
  <pre><code>import pandas as pd
  filenames = ['file1.csv', 'file2.csv', 'file3.csv', 'file4.csv']
  for file in filenames:
      file1 = pd.read_csv(file)
      print('running ' + str(file))
      a = function1(file1)
      b = function2(a)
      c = function3(b)
      for d in range(1,6):
              e = function4(c, d)
      c.to_csv('output.csv')
  </code></pre>
  
  <p>(incorrectly) Parallelized code</p>
  
  <pre><code>import pandas as pd
  from multiprocessing import Pool
  filenames = ['file1.csv', 'file2.csv', 'file3.csv', 'file4.csv']
  def multip(filenames):
      file1 = pd.read_csv(file)
      print('running ' + str(file))
      a = function1(file1)
      b = function2(a)
      c = function3(b)
      for d in range(1,6):
              e = function4(c, d)
      c.to_csv('output.csv')
  
  if __name__ == '__main__'
      pool = Pool(processes=4)
      runstuff = pool.map(multip(filenames))
  </code></pre>
  
  <p>What I <strong><em>(think)</em></strong> I want to do is have <strong>one file</strong> be computed <strong>per core</strong> (maybe per process?). I also did</p>
  
  <pre><code>multiprocessing.cpu_count()
  </code></pre>
  
  <p>and got 8 (I have a quad so its probably taking into account threads). Since I have around 10 files total, if I can put one file per process to speed things up that would be great! I would hope the remaining 2 files would find a process after the processes from the first round complete as well.</p>
  
  <p>Edit: 
  for further clarity, the functions (i.e. function1, function2 etc) also feed into other functions (i.e function1a, function1b) inside their respective files. I call function 1 using an import statement. </p>
  
  <p>I get the following error:</p>
  
  <pre><code>OSError: Expected file path name or file-like object, got &lt;class 'list'&gt; type
  </code></pre>
  
  <p>Apparently doesn't like being passed a list but i don't want to do filenames[0] in the if statement because that only runs one file</p>