title: multiprocessing GUI schemas to combat the "Not Responding" blocking
body: <p>What are the best ways to create a multiprocessing/ GUI coding system?</p>
  
  <p>I would like to create a place for the internet community to come and find examples on how to use  the <code>multiprocessing</code> module in python.  </p>
  
  <p>I have seen several small examples of <code>multiprocessing</code> processes on the internet of simple global functions which are called in a main module, but I have found that this rarely translates easily into anything that anyone actually does with regard to GUIs. I would think that many programs would have the functions which they want to use in a separate process as methods of objects (which may be aggregates of other objects etc.) and perhaps a single GUI element would have an associated object that needs to call this process, etc. </p>
  
  <p>For example, I have a relatively complex program and I am having problems in getting a responsive GUI for it, which I believed to be due to my lack of understanding in <code>multiprocessing</code> and threading with <code>QThread</code>. However, I do know that the example given below will at least pass information between processes in the manner I desire (due to being able to execute <code>print</code> statements) but my GUI is still locking.  Does anyone know what may be causing this, and if it is still a probelm with my lack of understanding in mutlithreaded/multiprocessing architectures?</p>
  
  <p>Here is a small pseudo code example of what I am doing:</p>
  
  <pre><code>class Worker:
      ...
      def processing(self, queue):
          # put stuff into queue in a loop
  
  # This thread gets data from Worker
  class Worker_thread(QThread):
      def __init__(self):
          ...
          # make process with Worker inside
      def start_processing(self):
          # continuously get data from Worker
          # send data to Tab object with signals/slots
  
  class Tab(QTabWidget):
      # spawn a thread separate from main GUI thread
  
      # update GUI using slot
      def update_GUI()
  </code></pre>
  
  <p>And this code is fully compilable example which embodies the overlying sturcture of my program:</p>
  
  <pre><code>from PyQt4 import QtCore, QtGui
  import multiprocessing as mp
  import numpy as np
  import sys
  import time
  
  # This object can hold several properties which will be used for the processing
  # and will be run in the background, while it updates a thread with all of it's progress
  class Worker:
      def __init__(self, some_var):
          self.some_var = some_var
          self.iteration = 0
  
      def some_complex_processing(self, queue):
          for i in range(0,5000):
              self.iteration += 1
              queue.put(self.iteration)
          queue.put('done with processing')
  
  # This Woker_thread is a thread which will spawn a separate process (Worker).
  # This separate is needed in order to separate the data retrieval
  # from the main GUI thread, which should only quickly update when needed 
  class Worker_thread(QtCore.QThread):
      # signals and slots are used to communicate back to the main GUI thread
      update_signal = QtCore.pyqtSignal(int)
      done_signal = QtCore.pyqtSignal()
  
      def __init__(self, parent, worker):
          QtCore.QThread.__init__(self, parent)
          self.queue = mp.Queue()
          self.worker = worker
          self.parent = parent
          self.process = mp.Process(target=self.worker.some_complex_processing, args=(self.queue,))
  
      # When the process button is pressed, this function will start getting data from Worker
      # this data is then retrieved by the queue and pushed through a signal
      # to Tab.update_GUI
      @QtCore.pyqtSlot()
      def start_computation(self):
          self.process.start()
          while(True):
              try:
                  message = self.queue.get()
                  self.update_signal.emit(message)
              except EOFError:
                  pass
              if message == 'done with processing':
                  self.done_signal.emit()
                  break
              #self.parent.update_GUI(message)
          self.process.join()
          return
  
  # Each tab will start it's own thread, which will spawn a process
  class Tab(QtGui.QTabWidget):
      start_comp = QtCore.pyqtSignal()
      def __init__(self, parent, this_worker):
          self.parent = parent
          self.this_worker = this_worker
          QtGui.QTabWidget.__init__(self, parent)
  
          self.treeWidget = QtGui.QTreeWidget(self)
          self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
          self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
  
          self.thread = Worker_thread(parent=self, worker=self.this_worker)
          self.thread.update_signal.connect(self.update_GUI)
          self.thread.done_signal.connect(self.thread.quit)
          self.start_comp.connect(self.thread.start_computation)
          self.thread.start()
  
      ###############################
      # Here is what should update the GUI at every iteration of Worker.some_complex_processing()
      # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated.
      @QtCore.pyqtSlot(int)
      def update_GUI(self, iteration):
          self.step.setText(0, str(iteration))
          #time.sleep(0.1)
          print iteration
  
      def start_signal_emit(self):
          self.start_comp.emit()
  
  # GUI stuff
  class MainWindow(QtGui.QMainWindow):
      def __init__(self, parent = None):
          QtGui.QMainWindow.__init__(self)
          self.tab_list = []
          self.setTabShape(QtGui.QTabWidget.Rounded)
          self.centralwidget = QtGui.QWidget(self)
          self.top_level_layout = QtGui.QGridLayout(self.centralwidget)
  
          self.tabWidget = QtGui.QTabWidget(self.centralwidget)
          self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)
  
          process_button = QtGui.QPushButton("Process")
          self.top_level_layout.addWidget(process_button, 0, 1)
          QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)
  
          self.setCentralWidget(self.centralwidget)
          self.centralwidget.setLayout(self.top_level_layout)
  
          # Make Tabs in loop from button
          for i in range(0,10):
              name = 'tab' + str(i)
              self.tab_list.append(Tab(self.tabWidget, Worker(name)))
              self.tabWidget.addTab(self.tab_list[-1], name)
  
      # Do the processing
      def process(self):
          for tab in self.tab_list:
              tab.start_signal_emit()
          return
  
  if __name__ == "__main__":
      app = QtGui.QApplication([])
      win = MainWindow()
      win.show()
      sys.exit(app.exec_())
  </code></pre>
  
  <p><em><strong>More Information:</em></strong>
  I am writing a program which I would like to spawn several processes from and have them continuously show their progress throughout their processing.  I would like the program to be multiprocessed in order to get the best speed out of the program as possible.</p>
  
  <p>At the moment, I am trying to use a thread to spawn a process and use signals and slots to update the GUI while the data is continuously retrieved by a queue.  It appears that the <code>queues</code>, <code>signals</code>, and <code>slots</code> work when using <code>print</code> statements, but can not update the GUI.  If anyone has any other suggestions as to how I should structure this in order to keep the program more managable, I would like to learn.</p>
  
  <p><em><strong>EDIT</em></strong>: I have made the adjustments put forth by Min Lin, with the addition of making <code>Worker</code> a <code>QObject</code> so that <code>moveToThread()</code> would work.<br>
  Here is the new code I have at the moment:</p>
  
  <pre><code>from PyQt4 import QtCore, QtGui
  import multiprocessing as mp
  import numpy as np
  import sys
  import time
  
  class Worker(QtCore.QObject):
      update_signal = QtCore.pyqtSignal(int)
      done_signal = QtCore.pyqtSignal()
  
      def __init__(self, some_var):
          QtCore.QObject.__init__(self, parent=None)
          self.some_var = some_var
          self.iteration = 0
          self.queue = mp.Queue()
          self.process = mp.Process(target=self.some_complex_processing, args=(self.queue,))
  
      def some_complex_processing(self, queue):
          for i in range(0,5000):
              self.iteration += 1
              queue.put(self.iteration)
          queue.put('done with processing')
  
      @QtCore.pyqtSlot()
      def start_computation(self):
          self.process.start()
          while(True):
              try:
                  message = self.queue.get()
                  self.update_signal.emit(message)
              except EOFError:
                  pass
              if message == 'done with processing':
                  self.done_signal.emit()
                  break
          self.process.join()
          return
  
  
  
  class Tab(QtGui.QTabWidget):
      start_comp = QtCore.pyqtSignal()
      def __init__(self, parent, this_worker):
          self.parent = parent
          self.this_worker = this_worker
          QtGui.QTabWidget.__init__(self, parent)
  
          self.treeWidget = QtGui.QTreeWidget(self)
          self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
          self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
  
          # Use QThread is enough
          self.thread = QtCore.QThread();
          # Change the thread affinity of worker to self.thread.
          self.this_worker.moveToThread(self.thread);
          self.this_worker.update_signal.connect(self.update_GUI)
          self.this_worker.done_signal.connect(self.thread.quit)
          self.start_comp.connect(self.this_worker.start_computation)
          self.thread.start()
  
      ###############################
      # Here is what should update the GUI at every iteration of Worker.some_complex_processing()
      # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated.
      @QtCore.pyqtSlot(int)
      def update_GUI(self, iteration):
          self.step.setText(0, str(iteration))
          #time.sleep(0.1)
          print iteration
  
      def start_signal_emit(self):
          self.start_comp.emit()
  
  # GUI stuff
  class MainWindow(QtGui.QMainWindow):
      def __init__(self, parent = None):
          QtGui.QMainWindow.__init__(self)
          self.tab_list = []
          self.setTabShape(QtGui.QTabWidget.Rounded)
          self.centralwidget = QtGui.QWidget(self)
          self.top_level_layout = QtGui.QGridLayout(self.centralwidget)
  
          self.tabWidget = QtGui.QTabWidget(self.centralwidget)
          self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)
  
          process_button = QtGui.QPushButton("Process")
          self.top_level_layout.addWidget(process_button, 0, 1)
          QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)
  
          self.setCentralWidget(self.centralwidget)
          self.centralwidget.setLayout(self.top_level_layout)
  
          # Make Tabs in loop from button
          for i in range(0,10):
              name = 'tab' + str(i)
              self.tab_list.append(Tab(self.tabWidget, Worker(name)))
              self.tabWidget.addTab(self.tab_list[-1], name)
  
      # Do the processing
      def process(self):
          for tab in self.tab_list:
              tab.start_signal_emit()
          return
  
  if __name__ == "__main__":
      app = QtGui.QApplication([])
      win = MainWindow()
      win.show()
      sys.exit(app.exec_())
  </code></pre>
  
  <p>Thank you for all of the answers, I appreciate the level of detail that everyone has gone into in describing the idea they believe to be solution, but unfortunately I have not yet been able to perform these types of processes which operate on the object they belong to while displaying the object's attribute on a GUI.<br>
  However, I have learned a decent amount from this post, which allowed me to realize that the threaded version I have at the moment is hanging the GUI since the GUI update function is too large and takes too much processing.</p>
  
  <p>So, I have taken the <code>QTimer()</code> approach to my multi-threaded version and it is performing much better!  I would advise anyone facing similar problems to at least attempt something similar to this. </p>
  
  <p>I was unaware of this approach to solving GUI update problems, and it is now a pseudo or temporary fix to the problem I am facing.</p>