title: Python multiprocessing map_async ready() and _number_left not working as expected
body: <p>This is a little complicated, and I am sure this is a newbie error, but I cannot for the life for me figure out where to even look.</p>
  
  <p>Trying to do a multiprocessing map_async to process a large number of files.  Essentially, the code gets a list of files, looks in each file for a match on a MAC address and writes out a line to another file if it matches.</p>
  
  <p>nodb is my library....I have not included everything here yet (it's kind of convoluted).  I am hoping that someone can point me where to even look for debugging this.</p>
  
  <p>Here is the problem: the code works perfectly on anything under 60,000 files. However, when I point it at a directory with 595,200 files, the little "while true" loop that checks to see if it is done (using _number_left) stops working....processing appears to continue but the _number_left does not decrease and the ready() function returns TRUE...which it isn't.</p>
  
  <p>And it stops after processing 62111 or 62112 files every single time I run it.  I added the little "dump" function thinking my queue was filling up.</p>
  
  <p>Not sure what else to tell you...am I missing something? (probably)  Please let me know what more I can tell you to figure this out.  I really have no idea what is relevant....</p>
  
  <p>Code is:</p>
  
  <pre><code>import nodb_v09d as nodb
  
  import netaddr
  import sys
  import collections
  from multiprocessing import Pool, Queue
  import itertools
  import time
  
  # Handle CLI arguments
  #
  nmultip = 1
  args = sys.argv[1:]
  nmultip = nodb.parseArgs(args)
  
  # this function just gets the file list in the directory
  todoPif = nodb.getFileList('/data/david/data/2012/05')
  
  filterfields = { 10:set([int(netaddr.EUI('00-00-0a-0e-c9-be')),\
                           int(netaddr.EUI('00:15:ce:de:78:f3')),\
                           int(netaddr.EUI('3c-75-4a-ea-15-01')),\
                           int(netaddr.EUI('00-24-d1-1e-e9-be'))])
                   }
  ff=collections.OrderedDict(sorted(filterfields.items()))
  
  resultfields = [28,29,30,33]
  rf=resultfields.sort
  
  cutoff = 40
  
  todocnt = len(todoPif)
  
  outQ = Queue()
  hdrQ = Queue()
  
  # output file
  gpif = '/media/sf_samplePifs/test2.gpif'
  
  append = 0
  
  if __name__ == '__main__':
  
      # this is a little trick I picked up off the internet for passing multiple queues.  works ok
      todopool = Pool(None,nodb.poolQueueInit,[outQ,hdrQ])
  
      # itertools used to create an arg that contains a constant (ff) for all calls)
      r = todopool.map_async(nodb.deRefCall,itertools.izip(todoPif, itertools.repeat(ff)),1)
  
      while (True):
          nodb.logging.info('number left: ' + str(r._number_left) + '\nready? ' + str(r.ready()))
          nodb.logging.info('queue size: ' + str(outQ.qsize()))
  
          if (r._number_left == 0): break
          if (outQ.qsize() &gt;= cutoff):
              nodb.dumpQueueToGpif(gpif, hdrQ, outQ, append, cutoff)
              if (append == 0): 
                  append = 1
  
          sys.stderr.write('\rPIF Files DONE: ' + str(todocnt-r._number_left) + '/' + str(todocnt))
          print '\n'
          time.sleep(0.2)
      r.wait()  
  
  sys.stderr.write('\rPIF Files DONE: ' + str(todocnt) + '/' + str(todocnt) + '\n')
  
  # dump remainder to file
  nodb.dumpQueueToGpif(gpif, hdrQ, outQ, append,outQ.qsize())
  </code></pre>
  
  <p><strong>MAJOR ADDITION:</strong></p>
  
  <p>At the request of another user, I simplified the code.  No queues, no external private libraries, etc:</p>
  
  <pre><code>import sys
  import os
  import time
  from multiprocessing import Pool
  
  def doPifFile(pifFile):
  
      #readPif = call(['ls','-l',' &gt; /tmp/out'])
      cmd = 'ipdr_dump ' + pifFile + ' | grep "," | wc -l &gt; /tmp/dump'
      readPif = os.system(cmd)
      return readPif
  
  def getFileList(directory):
      flist = list()
      for root, dirs, files in os.walk(directory):
          for piffile in files:
              if piffile.endswith('.pif'):
                  flist.append(os.path.abspath(os.path.join(root,piffile)))
      return flist
  
  todoPif = getFileList('/data/david/data/2012/05')
  
  todocnt = len(todoPif)
  print '# of files to process: ' + str(todocnt)
  
  if __name__ == '__main__':
  
      todopool = Pool()
      r = todopool.map_async(doPifFile,todoPif,1)
  
      while (True):
          print 'number left: ' + str(r._number_left) + '\nready? ' + str(r.ready())
  
          #if (r.ready()): break
          if (r._number_left == 0): break
          sys.stderr.write('\rPIF Files DONE: ' + str(todocnt-r._number_left) + '/' + str(todocnt))
          print '\n'
          time.sleep(0.2)
  
  sys.stderr.write('\rPIF Files DONE: ' + str(todocnt) + '/' + str(todocnt) + '\n')
  </code></pre>
  
  <p>When I ran it, I got something VERY interesting that did NOT show up on the runs with the more complex code but that happened at exactly the same spot, although it claims it happened in my dump program:</p>
  
  <pre><code>number left: 533100
  ready? False
  PIF Files DONE: 62100/595200
  
  number left: 533090
  ready? False
  PIF Files DONE: 62110/595200
  
  *** glibc detected *** ipdr_dump: corrupted double-linked list: 0x0000000001a58370 ***
  ======= Backtrace: =========
  /lib64/libc.so.6[0x36f5c76126]
  /lib64/libc.so.6[0x36f5c78eb4]
  /lib64/libc.so.6(fclose+0x14d)[0x36f5c6678d]
  /lib64/libz.so.1[0x36f6803021]
  ipdr_dump[0x405c0b]
  ipdr_dump[0x40546e]
  ipdr_dump[0x401c2a]
  /lib64/libc.so.6(__libc_start_main+0xfd)[0x36f5c1ecdd]
  ipdr_dump[0x4016b9]
  ======= Memory map: ========
  00400000-0040e000 r-xp 00000000 08:02 2364135                            /home/david/ipdr_dump
  0060d000-0060e000 rw-p 0000d000 08:02 2364135                            /home/david/ipdr_dump
  01a54000-01a75000 rw-p 00000000 00:00 0                                  [heap]
  32e0600000-32e0604000 r-xp 00000000 08:02 3932181                        /lib64/libuuid.so.1.3.0
  32e0604000-32e0803000 ---p 00004000 08:02 3932181                        /lib64/libuuid.so.1.3.0
  32e0803000-32e0804000 rw-p 00003000 08:02 3932181                        /lib64/libuuid.so.1.3.0
  36f5800000-36f5820000 r-xp 00000000 08:02 3932309                        /lib64/ld-2.12.so
  36f5a1f000-36f5a20000 r--p 0001f000 08:02 3932309                        /lib64/ld-2.12.so
  36f5a20000-36f5a21000 rw-p 00020000 08:02 3932309                        /lib64/ld-2.12.so
  36f5a21000-36f5a22000 rw-p 00000000 00:00 0 
  36f5c00000-36f5d8a000 r-xp 00000000 08:02 3932315                        /lib64/libc-2.12.so
  36f5d8a000-36f5f89000 ---p 0018a000 08:02 3932315                        /lib64/libc-2.12.so
  36f5f89000-36f5f8d000 r--p 00189000 08:02 3932315                        /lib64/libc-2.12.so
  36f5f8d000-36f5f8e000 rw-p 0018d000 08:02 3932315                        /lib64/libc-2.12.so
  36f5f8e000-36f5f93000 rw-p 00000000 00:00 0 
  36f6000000-36f6002000 r-xp 00000000 08:02 3932566                        /lib64/libdl-2.12.so
  36f6002000-36f6202000 ---p 00002000 08:02 3932566                        /lib64/libdl-2.12.so
  36f6202000-36f6203000 r--p 00002000 08:02 3932566                        /lib64/libdl-2.12.so
  36f6203000-36f6204000 rw-p 00003000 08:02 3932566                        /lib64/libdl-2.12.so
  36f6400000-36f6417000 r-xp 00000000 08:02 3932564                        /lib64/libpthread-2.12.so
  36f6417000-36f6617000 ---p 00017000 08:02 3932564                        /lib64/libpthread-2.12.so
  36f6617000-36f6618000 r--p 00017000 08:02 3932564                        /lib64/libpthread-2.12.so
  36f6618000-36f6619000 rw-p 00018000 08:02 3932564                        /lib64/libpthread-2.12.so
  36f6619000-36f661d000 rw-p 00000000 00:00 0 
  36f6800000-36f6815000 r-xp 00000000 08:02 3932563                        /lib64/libz.so.1.2.3
  36f6815000-36f6a14000 ---p 00015000 08:02 3932563                        /lib64/libz.so.1.2.3
  36f6a14000-36f6a15000 r--p 00014000 08:02 3932563                        /lib64/libz.so.1.2.3
  36f6a15000-36f6a16000 rw-p 00015000 08:02 3932563                        /lib64/libz.so.1.2.3
  36f6c00000-36f6c83000 r-xp 00000000 08:02 3932493                        /lib64/libm-2.12.so
  36f6c83000-36f6e82000 ---p 00083000 08:02 3932493                        /lib64/libm-2.12.so
  36f6e82000-36f6e83000 r--p 00082000 08:02 3932493                        /lib64/libm-2.12.so
  36f6e83000-36f6e84000 rw-p 00083000 08:02 3932493                        /lib64/libm-2.12.so
  36f7000000-36f7007000 r-xp 00000000 08:02 3935994                        /lib64/librt-2.12.so
  36f7007000-36f7206000 ---p 00007000 08:02 3935994                        /lib64/librt-2.12.so
  36f7206000-36f7207000 r--p 00006000 08:02 3935994                        /lib64/librt-2.12.so
  36f7207000-36f7208000 rw-p 00007000 08:02 3935994                        /lib64/librt-2.12.so
  36f7800000-36f781d000 r-xp 00000000 08:02 3932588                        /lib64/libselinux.so.1
  36f781d000-36f7a1c000 ---p 0001d000 08:02 3932588                        /lib64/libselinux.so.1
  36f7a1c000-36f7a1d000 r--p 0001c000 08:02 3932588                        /lib64/libselinux.so.1
  36f7a1d000-36f7a1e000 rw-p 0001d000 08:02 3932588                        /lib64/libselinux.so.1
  36f7a1e000-36f7a1f000 rw-p 00000000 00:00 0 
  36f7c00000-36f7c16000 r-xp 00000000 08:02 3932572                        /lib64/libresolv-2.12.so
  36f7c16000-36f7e16000 ---p 00016000 08:02 3932572                        /lib64/libresolv-2.12.so
  36f7e16000-36f7e17000 r--p 00016000 08:02 3932572                        /lib64/libresolv-2.12.so
  36f7e17000-36f7e18000 rw-p 00017000 08:02 3932572                        /lib64/libresolv-2.12.so
  36f7e18000-36f7e1a000 rw-p 00000000 00:00 0 
  36f8000000-36f800e000 r-xp 00000000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
  36f800e000-36f820d000 ---p 0000e000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
  36f820d000-36f820e000 r--p 0000d000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
  36f820e000-36f820f000 rw-p 0000e000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
  36f8800000-36f8849000 r-xp 00000000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
  36f8849000-36f8a49000 ---p 00049000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
  36f8a49000-36f8a4b000 r--p 00049000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
  36f8a4b000-36f8a4d000 rw-p 0004b000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
  36f8c00000-36f8c16000 r-xp 00000000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
  36f8c16000-36f8e15000 ---p 00016000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
  36f8e15000-36f8e16000 rw-p 00015000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
  36f9400000-36f9535000 r-xp 00000000 08:02 4206136                        /usr/lib64/libnss3.so
  36f9535000-36f9734000 ---p 00135000 08:02 4206136                        /usr/lib64/libnss3.so
  36f9734000-36f9739000 r--p 00134000 08:02 4206136                        /usr/lib64/libnss3.so
  36f9739000-36f973b000 rw-p 00139000 08:02 4206136                        /usr/lib64/libnss3.so
  36f973b000-36f973d000 rw-p 00000000 00:00 0 
  36f9800000-36f9825000 r-xp 00000000 08:02 4206135                        /usr/lib64/libnssutil3.so
  36f9825000-36f9a24000 ---p 00025000 08:02 4206135                        /usr/lib64/libnssutil3.so
  36f9a24000-36f9a2a000 r--p 00024000 08:02 4206135                        /usr/lib64/libnssutil3.sonumber left: 533078
  ready? False
  PIF Files DONE: 62122/595200
  
  number left: 533068
  ready? False
  PIF Files DONE: 62132/595200
  
  number left: 533056
  ready? False
  PIF Files DONE: 62144/595200
  </code></pre>
  
  <p>What is odd is that it continued on, whereas the previous runs caused a failure of the _number_left and misfire on "ready()" (although the processes still ran in the background).</p>
  
  <p>I have run the dump program manually on the 16 processor box I have, and they run in parallel fine, never seen that glibc error before.  I have to assume that it is associated with the python setup....I just don't know where.</p>
  
  <p>This may be too complex for a forum diagnosis.  Any further ideas on where I might look or how I might be able to seem what happened are welcome.</p>
  
  <p>One more tidbit...printed out pool._success.  It changes to FALSE at the magic moment with the _number_left stops moving.  </p>
  
  <pre><code>INFO  number left: 533167  
  ready? False  
  successful? True  
  INFO  queue size: 424  
  PIF Files DONE: 62033/595200  
  
  INFO  number left: 533117
  ready? False
  successful? True
  INFO  queue size: 424
  PIF Files DONE: 62083/595200
  
  INFO  number left: 533087
  ready? True
  successful? False
  INFO  queue size: 424
  PIF Files DONE: 62113/595200
  
  INFO  number left: 533087
  ready? True
  successful? False
  INFO  queue size: 424
  PIF Files DONE: 62113/595200
  </code></pre>