title: python multiprocessing/threading cleanup
body: <p>I have a python tool, that has basically this kind of setup:</p>
  
  <pre><code>main process (P1) -&gt; spawns a process (P2) that starts a tcp connection
                    -&gt; spawns a thread (T1) that starts a loop to receive 
                       messages that are sent from P2 to P1 via a Queue (Q1)
  
  server process (P2) -&gt; spawns two threads (T2 and T3) that start loops to
                         receive messages that are sent from P1 to P2 via Queues (Q2 and Q3)
  </code></pre>
  
  <p>The problem I'm having is that when I stop my program (with Ctrl+C), it doesn't quit. The server process is ended, but the main process just hangs there and I have to kill it.</p>
  
  <p>The thread loop functions all look the same:</p>
  
  <pre><code>def _loop(self):
      while self.running:
          res = self.Q1.get()
          if res is None:
              break
          self._handle_msg(res)
  </code></pre>
  
  <p>All threads are started as daemon:</p>
  
  <pre><code>t = Thread(target=self._loop)
  t.setDaemon(True)
  t.start()
  </code></pre>
  
  <p>In my main process, I use atexit, to perform clean-up tasks:</p>
  
  <pre><code>atexit.register(self.on_exit)
  </code></pre>
  
  <p>Those clean-up tasks are essentially the following:</p>
  
  <p>1) set <code>self.running</code> in P1 to <code>False</code> and sent <code>None</code> to Q1, so that the Thread T1 should finish</p>
  
  <pre><code>self.running = False
  self.Q1.put(None)
  </code></pre>
  
  <p>2) send a message to P2 via Q2 to inform this process that it is ending</p>
  
  <pre><code>self.Q2.put("stop")
  </code></pre>
  
  <p>3) In P2, react to the "stop" message and do what we did in P1</p>
  
  <pre><code>self.running = False
  self.Q2.put(None)
  self.Q3.put(None)
  </code></pre>
  
  <p>That is it and in my understanding, that should make everything shut down nicely, but it doesn't.</p>
  
  <p>The main code of P1 also contains the following endless loop, because otherwise the program would end prematurely:</p>
  
  <pre><code>while running:
      sleep(1)
  </code></pre>
  
  <p>Maybe that has something to do with the problem, but I cannot see why it should.</p>
  
  <p>So what did I do wrong? Does my setup have major design flaws? Did I forget to shut down something?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Ok, I modified my code and managed to make it shut down correctly most of the time. Unfortunately, from now and then, it still got stuck.</p>
  
  <p>I managed to write a small working example of my code. To demonstrate what happens, you need to simple start the script and then use <code>Ctrl + C</code> to stop it. It looks like the issue appears now usually if you press <code>Ctrl + C</code> as soon as possible after starting the tool.</p>
  
  <pre><code>#!/usr/bin/env python
  # -*- coding: utf-8 -*-
  
  import signal
  import sys
  import logging
  from multiprocessing import Process, Queue
  from threading import Thread
  from time import sleep
  
  
  logger = logging.getLogger("mepy-client")
  
  
  class SocketClientProtocol(object):
  
      def __init__(self, q_in, q_out, q_binary):
          self.q_in = q_in
          self.q_out = q_out
          self.q_binary = q_binary
          self.running = True
          t = Thread(target=self._loop)
          #t.setDaemon(True)
          t.start()
          t = Thread(target=self._loop_binary)
          #t.setDaemon(True)
          t.start()
  
      def _loop(self):
          print "start of loop 2"
          while self.running:
              res = self.q_in.get()
              if res is None:
                  break
              self._handle_msg(res)
          print "end of loop 2"
  
      def _loop_binary(self):
          print "start of loop 3"
          while self.running:
              res = self.q_binary.get()
              if res is None:
                  break
              self._handle_binary(res)
          print "end of loop 3"
  
      def _handle_msg(self, msg):
          msg_type = msg[0]
          if msg_type == "stop2":
              print "STOP RECEIVED"
              self.running = False
              self.q_in.put(None)
              self.q_binary.put(None)
  
      def _put_msg(self, msg):
          self.q_out.put(msg)
  
      def _handle_binary(self, data):
          pass
  
      def handle_element(self):
          self._put_msg(["something"])
  
  def run_twisted(q_in, q_out, q_binary):
      s = SocketClientProtocol(q_in, q_out, q_binary)
      while s.running:
          sleep(2)
          s.handle_element()
  
  
  class MediatorSender(object):
  
      def __init__(self):
          self.q_in = None
          self.q_out = None
          self.q_binary = None
          self.p = None
          self.running = False
  
      def start(self):
          if self.running:
              return
          self.running = True
          self.q_in = Queue()
          self.q_out = Queue()
          self.q_binary = Queue()
          print "!!!!START"
          self.p = Process(target=run_twisted, args=(self.q_in, self.q_out, self.q_binary))
          self.p.start()
          t = Thread(target=self._loop)
          #t.setDaemon(True)
          t.start()
  
      def stop(self):
          print "!!!!STOP"
          if not self.running:
              return
          print "STOP2"
          self.running = False
          self.q_out.put(None)
          self.q_in.put(["stop2"])
          #self.q_in.put(None)
          #self.q_binary.put(None)
  
          try:
              if self.p and self.p.is_alive():
                  self.p.terminate()
          except:
              pass
  
      def _loop(self):
          print "start of loop 1"
          while self.running:
              res = self.q_out.get()
              if res is None:
                  break
              self._handle_msg(res)
          print "end of loop 1"
  
      def _handle_msg(self, msg):
          self._put_msg(msg)
  
      def _put_msg(self, msg):
          self.q_in.put(msg)
  
      def _put_binary(self, msg):
          self.q_binary.put(msg)
  
      def send_chunk(self, chunk):
          self._put_binary(chunk)
  
  running = True
  def signal_handler(signal, frame):
      global running
      if running:
          running = False
          ms.stop()
      else:
          sys.exit(0)
  
  if __name__ == "__main__":
      signal.signal(signal.SIGINT, signal_handler)
      ms = MediatorSender()
      ms.start()
      for i in range(100):
          ms.send_chunk("some chunk of data")
      while running:
          sleep(1)
  </code></pre>