title: Python multiprocessing pool inside daemon process
body: <p>I opened up a question for this problem and did not get a thorough enough answer to solve the issue (most likely due to a lack of rigor in explaining my issues which is what I am attempting to correct): <a href="https://stackoverflow.com/questions/6428842/zombie-process-in-python-multiprocessing-daemon">Zombie process in python multiprocessing daemon</a></p>
  
  <p>I am trying to implement a python daemon that uses a pool of workers to executes commands using <code>Popen</code>. I have borrowed the basic daemon from <a href="http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/" rel="nofollow noreferrer">http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/</a></p>
  
  <p>I have only changed the <code>init</code>, <code>daemonize</code> (or equally the <code>start</code>) and <code>stop</code> methods. Here are the changes to the <code>init</code> method:</p>
  
  <pre><code>def __init__(self, pidfile):
  #, stdin='/dev/null', stdout='STDOUT', stderr='STDOUT'):
      #self.stdin = stdin
      #self.stdout = stdout
      #self.stderr = stderr
      self.pidfile = pidfile
      self.pool = Pool(processes=4)
  </code></pre>
  
  <p>I am not setting stdin, stdout and stderr so that I can debug the code with print statements. Also, I have tried moving this pool around to a few places but this is the only place that does not produce exceptions. </p>
  
  <p>Here are the changes to the <code>daemonize</code> method:</p>
  
  <pre><code>def daemonize(self):
      ...
  
      # redirect standard file descriptors
      #sys.stdout.flush()
      #sys.stderr.flush()
      #si = open(self.stdin, 'r')
      #so = open(self.stdout, 'a+')
      #se = open(self.stderr, 'a+', 0)
      #os.dup2(si.fileno(), sys.stdin.fileno())
      #os.dup2(so.fileno(), sys.stdout.fileno())
      #os.dup2(se.fileno(), sys.stderr.fileno())
  
      print self.pool
  
      ...
  </code></pre>
  
  <p>Same thing, I am not redirecting io so that I can debug. The print here is used so that I can check the pools location.</p>
  
  <p>And the <code>stop</code> method changes:</p>
  
  <pre><code>def stop(self):
      ...
  
      # Try killing the daemon process
      try:
          print self.pool
          print "closing pool"
          self.pool.close()
          print "joining pool"
          self.pool.join()
          print "set pool to None"
          self.pool = None
          while 1:
              print "kill process"
              os.kill(pid, SIGTERM)
  
      ...
  </code></pre>
  
  <p>Here the idea is that I not only need to kill the process but also clean up the pool. The <code>self.pool = None</code> is just a random attempt to solve the issues which didn't work. At first I thought this was a problem with zombie children which was occurring when I had the <code>self.pool.close()</code> and <code>self.pool.join()</code> inside the while loop with the <code>os.kill(pid, SIGTERM)</code>. This is before I decided to start looking at the pool location via the <code>print self.pool</code>. After doing this, I believe the pools are not the same when the daemon starts and when it stops. Here is some output:</p>
  
  <pre><code>me@pc:~/pyCode/jobQueue$ sudo ./jobQueue.py start
  &lt;multiprocessing.pool.Pool object at 0x1c543d0&gt;
  me@pc:~/pyCode/jobQueue$ sudo ./jobQueue.py stop
  &lt;multiprocessing.pool.Pool object at 0x1fb7450&gt;
  closing pool
  joining pool
  set pool to None
  kill process
  kill process
  ... [ stuck in infinite loop]
  </code></pre>
  
  <p>The different locations of the objects suggest to me that they are not the same pool and that one of them is probably the zombie?</p>
  
  <p>After <code>CTRL+C</code>, here is what I get from <code>ps aux|grep jobQueue</code>:</p>
  
  <pre><code>root     21161  0.0  0.0  50384  5220 ?        Ss   22:59   0:00 /usr/bin/python ./jobQueue.py start
  root     21162  0.0  0.0      0     0 ?        Z    22:59   0:00 [jobQueue.py] &lt;defunct&gt;
  me       21320  0.0  0.0   7624   940 pts/0    S+   23:00   0:00 grep --color=auto jobQueue
  </code></pre>
  
  <p>I have tried moving the <code>self.pool = Pool(processes=4)</code> to a number of different places. If it is moved to the <code>start()' or</code>daemonize()<code>methods,</code>print self.pool` will throw an exception saying that it is NoneType. In addition, the location seems to change the number of zombie process that will pop up.</p>
  
  <p>Currently, I have not added the functionality to run anything via the workers. My problem seems completely related to setting up the pool of workers correctly. I would appreciate any information that leads to solving this issue or advice about creating a daemon service that uses a pool of workers to execute a series of commands using <code>Popen</code>. Since I haven't gotten that far, I do not know what challenges I face ahead. I am thinking I might just need to write my own pool but if there is a nice trick to make the pool work here, it would be amazing.</p>