title: Multiprocessing.Queue with hugh data causes _wait_for_tstate_lock
body: <p>An Exception is raised in <code>threading._wait_for_tstate_lock</code> when I transfere hugh data between a <code>Process</code> and a <code>Thread</code> via <code>multiprocessing.Queue</code>.</p>
  <p>My minimal working example looks a bit complex first - sorry. I will explain. The original application loads a lot of (not so important) files into RAM. This is done in a separate process to save ressources. The main gui thread shouldn't freez.</p>
  <ol>
  <li><p>The GUI start a separate <code>Thread</code> to prevent the gui event loop from freezing.</p>
  </li>
  <li><p>This separate <code>Thread</code> then starts one <code>Process</code> which should does the work.</p>
  </li>
  </ol>
  <p>a) This <code>Thread</code> instanciates a <code>multiprocess.Queue</code> (be aware that this is a <code>multiprocessing</code> and not <code>threading</code>!)</p>
  <p>b) This is givin to the <code>Process</code> for sharing data from <code>Process</code> back to the <code>Thread</code>.</p>
  <ol start="3">
  <li><p>The <code>Process</code> does some work (3 steps) and <code>.put()</code> the result into the <code>multiprocessing.Queue</code>.</p>
  </li>
  <li><p>When the <code>Process</code> ends the <code>Thread</code> takes over again and collect the data from the <code>Queue</code>, store it to its own attribute <code>MyThread.result</code>.</p>
  </li>
  <li><p>The <code>Thread</code> tells the GUI main loop/thread to call a callback function if it has time for.</p>
  </li>
  <li><p>The callback function (<code>MyWindow::callback_thread_finished()</code>) get the results from <code>MyWindow.thread.result</code>.</p>
  </li>
  </ol>
  <p>The problem is if the data put to the <code>Queue</code> is to big something happen I don't understand - the <code>MyThread</code> never ends. I have to cancle the application via Strg+C.</p>
  <p>I got some hints from the docs. But my problem is I did not fully understand the documentation. But I have the feeling that the key of my problems can be found there.
  Please see the two red boxex in &quot;<a href="https://docs.python.org/3.5/library/multiprocessing.html#pipes-and-queues" rel="noreferrer">Pipes and Queues</a>&quot; (Python 3.5 docs).
  That is the full output</p>
  <pre><code>MyWindow::do_start()
  Running MyThread...
  Running MyProcess...
  MyProcess stoppd.
  ^CProcess MyProcess-1:
  Exception ignored in: &lt;module 'threading' from '/usr/lib/python3.5/threading.py'&gt;
  Traceback (most recent call last):
    File &quot;/usr/lib/python3.5/threading.py&quot;, line 1288, in _shutdown
      t.join()
    File &quot;/usr/lib/python3.5/threading.py&quot;, line 1054, in join
      self._wait_for_tstate_lock()
    File &quot;/usr/lib/python3.5/threading.py&quot;, line 1070, in _wait_for_tstate_lock
      elif lock.acquire(block, timeout):
  KeyboardInterrupt
  Traceback (most recent call last):
    File &quot;/usr/lib/python3.5/multiprocessing/process.py&quot;, line 252, in _bootstrap
      util._exit_function()
    File &quot;/usr/lib/python3.5/multiprocessing/util.py&quot;, line 314, in _exit_function
      _run_finalizers()
    File &quot;/usr/lib/python3.5/multiprocessing/util.py&quot;, line 254, in _run_finalizers
      finalizer()
    File &quot;/usr/lib/python3.5/multiprocessing/util.py&quot;, line 186, in __call__
      res = self._callback(*self._args, **self._kwargs)
    File &quot;/usr/lib/python3.5/multiprocessing/queues.py&quot;, line 198, in _finalize_join
      thread.join()
    File &quot;/usr/lib/python3.5/threading.py&quot;, line 1054, in join
      self._wait_for_tstate_lock()
    File &quot;/usr/lib/python3.5/threading.py&quot;, line 1070, in _wait_for_tstate_lock
      elif lock.acquire(block, timeout):
  KeyboardInterrupt
  </code></pre>
  <p>This is the minimal working example</p>
  <pre><code>#!/usr/bin/env python3
  
  import multiprocessing
  import threading
  import time
  import gi
  gi.require_version('Gtk', '3.0')
  from gi.repository import Gtk
  from gi.repository import GLib
  
  
  class MyThread (threading.Thread):
      &quot;&quot;&quot;This thread just starts the process.&quot;&quot;&quot;
      def __init__(self, callback):
          threading.Thread.__init__(self)
          self._callback = callback
  
      def run(self):
          print('Running MyThread...')
          self.result = []
  
          queue = multiprocessing.Queue()
          process = MyProcess(queue)
          process.start()
          process.join()
  
          while not queue.empty():
              process_result = queue.get()
              self.result.append(process_result)
          print('MyThread stoppd.')
          GLib.idle_add(self._callback)
  
  
  class MyProcess (multiprocessing.Process):
      def __init__(self, queue):
          multiprocessing.Process.__init__(self)
          self.queue = queue
  
      def run(self):
          print('Running MyProcess...')
          for i in range(3):
              self.queue.put((i, 'x'*102048))
          print('MyProcess stoppd.')
  
  class MyWindow (Gtk.Window):
      def __init__(self):
          Gtk.Window.__init__(self)
          self.connect('destroy', Gtk.main_quit)
          GLib.timeout_add(2000, self.do_start)
  
      def do_start(self):
          print('MyWindow::do_start()')
          # The process need to be started from a separate thread
          # to prevent the main thread (which is the gui main loop)
          # from freezing while waiting for the process result.
          self.thread = MyThread(self.callback_thread_finished)
          self.thread.start()
  
      def callback_thread_finished(self):
          result = self.thread.result
          for r in result:
              print('{} {}...'.format(r[0], r[1][:10]))
  
  if __name__ == '__main__':
      win = MyWindow()
      win.show_all()
      Gtk.main()
  </code></pre>
  <p>Possible duplicate but quite different and IMO without an answer for my situation: <a href="https://stackoverflow.com/q/47147328/4865723">Thread._wait_for_tstate_lock() never returns</a>.</p>
  <h2>Workaround</h2>
  <p>Using a <em>Manager</em> by modifing line 22 to <code>queue = multiprocessing.Manager().Queue()</code> solve the problem. But I don't know why. My intention of this question is to understand the things behind and not only to make my code work. Even I don't really know what a <code>Manager()</code> is and if it has other (problem causing) implications.</p>