title: Progress measuring with python's multiprocessing Pool and map function
body: <p>Following code I'm using for parallel csv processing:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import csv
  from time import sleep
  from multiprocessing import Pool
  from multiprocessing import cpu_count
  from multiprocessing import current_process
  from pprint import pprint as pp
  
  def init_worker(x):
    sleep(.5)
    print "(%s,%s)" % (x[0],x[1])
    x.append(int(x[0])**2)
    return x
  
  def parallel_csv_processing(inputFile, outputFile, header=["Default", "header", "please", "change"], separator=",", skipRows = 0, cpuCount = 1):
    # OPEN FH FOR READING INPUT FILE
    inputFH   = open(inputFile,  "rt")
    csvReader = csv.reader(inputFH, delimiter=separator)
  
    # SKIP HEADERS
    for skip in xrange(skipRows):
      csvReader.next()
  
    # PARALLELIZE COMPUTING INTENSIVE OPERATIONS - CALL FUNCTION HERE
    try:
      p = Pool(processes = cpuCount)
      results = p.map(init_worker, csvReader, chunksize = 10)
      p.close()
      p.join()
    except KeyboardInterrupt:
      p.close()
      p.join()
      p.terminate()
  
    # CLOSE FH FOR READING INPUT
    inputFH.close()
  
    # OPEN FH FOR WRITING OUTPUT FILE
    outputFH  = open(outputFile, "wt")
    csvWriter = csv.writer(outputFH, lineterminator='\n')
  
    # WRITE HEADER TO OUTPUT FILE
    csvWriter.writerow(header)
  
    # WRITE RESULTS TO OUTPUT FILE
    [csvWriter.writerow(row) for row in results]
  
    # CLOSE FH FOR WRITING OUTPUT
    outputFH.close()
  
    print pp(results)
    # print len(results)
  
  def main():
    inputFile  = "input.csv"
    outputFile = "output.csv"
    parallel_csv_processing(inputFile, outputFile, cpuCount = cpu_count())
  
  if __name__ == '__main__':
    main()
  </code></pre>
  
  <p>I would like to somehow measure the progress of the script (just plain text not any fancy ASCII art). The one option that comes to my mind is to compare the lines that were successfully processed by <code>init_worker</code> to all lines in input.csv, and print the actual state e.g. every second, can you please point me to right solution? I've found several articles with similar problematic but I was not able to adapt it to my needs because neither used the <code>Pool</code> class and <code>map</code> method. I would also like to ask about <code>p.close(), p.join(), p.terminate()</code> methods, I've seen them mainly with <code>Process</code> not <code>Pool</code> class, are they necessary with <code>Pool</code> class and have I use them correctly? Using of <code>p.terminate()</code> was intended to kill the process with ctrl+c but this is <a href="https://stackoverflow.com/questions/32160054/keyboard-interrupts-with-pythons-multiprocessing-pool-and-map-function">different</a> story which has not an happy end yet. Thank you.</p>
  
  <p>PS: My input.csv looks like this, if it matters:</p>
  
  <pre><code>0,0
  1,3
  2,6
  3,9
  ...
  ...
  48,144
  49,147
  </code></pre>
  
  <p>PPS: as I said I'm newbie in <code>multiprocessing</code> and the code I've put together just works. The one drawback I can see is that whole csv is stored in memory, so if you guys have better idea do not hesitate to share it.</p>
  
  <p><strong>Edit</strong></p>
  
  <p>in reply to @J.F.Sebastian</p>
  
  <p>Here is my actual code based on your suggestions:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import csv
  from time import sleep
  from multiprocessing import Pool
  from multiprocessing import cpu_count
  from multiprocessing import current_process
  from pprint import pprint as pp
  from tqdm import tqdm
  
  def do_job(x):
    sleep(.5)
    # print "(%s,%s)" % (x[0],x[1])
    x.append(int(x[0])**2)
    return x
  
  def parallel_csv_processing(inputFile, outputFile, header=["Default", "header", "please", "change"], separator=",", skipRows = 0, cpuCount = 1):
  
    # OPEN FH FOR READING INPUT FILE
    inputFH   = open(inputFile,  "rb")
    csvReader = csv.reader(inputFH, delimiter=separator)
  
    # SKIP HEADERS
    for skip in xrange(skipRows):
      csvReader.next()
  
    # OPEN FH FOR WRITING OUTPUT FILE
    outputFH  = open(outputFile, "wt")
    csvWriter = csv.writer(outputFH, lineterminator='\n')
  
    # WRITE HEADER TO OUTPUT FILE
    csvWriter.writerow(header)
  
    # PARALLELIZE COMPUTING INTENSIVE OPERATIONS - CALL FUNCTION HERE
    try:
      p = Pool(processes = cpuCount)
      # results = p.map(do_job, csvReader, chunksize = 10)
      for result in tqdm(p.imap_unordered(do_job, csvReader, chunksize=10)):
        csvWriter.writerow(result)
      p.close()
      p.join()
    except KeyboardInterrupt:
      p.close()
      p.join()
  
    # CLOSE FH FOR READING INPUT
    inputFH.close()
  
    # CLOSE FH FOR WRITING OUTPUT
    outputFH.close()
  
    print pp(result)
    # print len(result)
  
  def main():
    inputFile  = "input.csv"
    outputFile = "output.csv"
    parallel_csv_processing(inputFile, outputFile, cpuCount = cpu_count())
  
  if __name__ == '__main__':
    main()
  </code></pre>
  
  <p>Here is output of <code>tqdm</code>:</p>
  
  <pre><code>1 [elapsed: 00:05,  0.20 iters/sec]
  </code></pre>
  
  <p>what does this output mean? On the page you've referred <code>tqdm</code> is used in loop following way:</p>
  
  <pre><code>&gt;&gt;&gt; import time
  &gt;&gt;&gt; from tqdm import tqdm
  &gt;&gt;&gt; for i in tqdm(range(100)):
  ...     time.sleep(1)
  ... 
  |###-------| 35/100  35% [elapsed: 00:35 left: 01:05,  1.00 iters/sec]
  </code></pre>
  
  <p>This output makes sense, but what does my output mean? Also it does not seems that ctrl+c problem is fixed: after hitting ctrl+c script throws some Traceback, if I hit ctrl+c again then I get new Traceback and so on. The only way to kill it is sending it to background (ctr+z) and then kill it (kill %1)</p>