title: Python multiprocessing Pool.map not faster than calling the function once
body: <p>I have a very large list of strings (originally from a text file) that I need to process using python. Eventually I am trying to go for a map-reduce style of parallel processing.</p>
  
  <p>I have written a "mapper" function and fed it to <code>multiprocessing.Pool.map()</code>, but it takes the same amount of time as simply calling the mapper function with the full set of data. I must be doing something wrong.</p>
  
  <p>I have tried multiple approaches, all with similar results.</p>
  
  <pre><code>def initial_map(lines):
      results = []
      for line in lines:
          processed = # process line (O^(1) operation)
          results.append(processed)
      return results
  
  def chunks(l, n):
      for i in xrange(0, len(l), n):
          yield l[i:i+n]
  
  if __name__ == "__main__":
      lines = list(open("../../log.txt", 'r'))
      pool = Pool(processes=8)
      partitions = chunks(lines, len(lines)/8)
      results = pool.map(initial_map, partitions, 1)
  </code></pre>
  
  <p>So the chunks function makes a list of sublists of the original set of lines to give to the <code>pool.map()</code>, then it should hand these 8 sublists to 8 different processes and run them through the mapper function. When I run this I can see all 8 of my cores peak at 100%. Yet it takes 22-24 seconds.</p>
  
  <p>When I simple run this (single process/thread):</p>
  
  <pre><code>lines = list(open("../../log.txt", 'r'))
  results = initial_map(results)
  </code></pre>
  
  <p>It takes about the same amount of time. ~24 seconds. I only see one process getting to 100% CPU.</p>
  
  <p>I have also tried letting the pool split up the lines itself and have the mapper function only handle one line at a time, with similar results.</p>
  
  <pre><code>def initial_map(line):
      processed = # process line (O^(1) operation)
      return processed
  
  if __name__ == "__main__":
      lines = list(open("../../log.txt", 'r'))
      pool = Pool(processes=8)
      pool.map(initial_map, lines)
  </code></pre>
  
  <p>~22 seconds.</p>
  
  <p>Why is this happening? Parallelizing this should result in faster results, shouldn't it?</p>