title: Does Python GIL need to be taken care when work with multi-thread C++ extension?
body: <p>I'm now implementing a data subscriber with Python, which subscribes to a data publisher (actually a ZeroMQ publisher socket) and will get notified once any new messages are fed. In my subscriber, the messages are dumped to a data processor after received. The subscriber will also get notified by the processor when done. Since the data processor is written in C++, I have to extend the Python code with a simple C++ module. </p>
  
  <p>Below is a simplified runnable code sample of my data subscriber. Code <code>main.py</code>, in which the module proc represents the processor, subscribe to a ZeroMQ socket on <code>localhost:10000</code>, setup the callback, and send the received message to the processor by calling <code>proc.onMsg</code>.</p>
  
  <pre><code>#!/bin/python
  # main.py
  
  import gevent
  import logging
  import zmq.green as zmq
  
  import pub 
  import proc
  
  logging.basicConfig( format='[%(levelname)s] %(message)s', level=logging.DEBUG )
  
  SUB_ADDR = 'tcp://localhost:10000'
  
  def setupMqAndReceive():
      '''Setup the message queue and receive messages.
      '''
      ctx  = zmq.Context()
      sock = ctx.socket( zmq.SUB )
      # add topics
      sock.setsockopt_string( zmq.SUBSCRIBE, 'Hello' )
  
      sock.connect( SUB_ADDR )
  
      while True:
          msg = sock.recv().decode( 'utf-8' )
          proc.onMsg( msg )
  
  def callback( a, b ):
      print( '[callback]',  a, b ) 
  
  def main():
      '''Entrance of the module.
      '''
      pub.start()
      proc.setCallback( callback )
      '''A simple on-liner
      gevent.spawn( setupMqAndReceive ).join()
      works. However, the received messages will not be
      processed by the processor.
      '''
      gevent.spawn( setupMqAndReceive )
      proc.start()
  </code></pre>
  
  <p>Module <code>proc</code> is simplified with three functions exported:</p>
  
  <ul>
  <li><code>setCallback</code> setup the callback function, so that when the message processed, my subscriber could be notified;</li>
  <li><code>onMsg</code> is invoked by the subscriber;</li>
  <li><code>start</code> setups a new worker thread to handle the messages from subscriber and make the main thread join to wait for the worker thread to exit.</li>
  </ul>
  
  <p>The full version of source code could be found on <a href="https://github.com/more-more-tea/python_gil" rel="nofollow">github</a> at <a href="https://github.com/more-more-tea/python_gil" rel="nofollow">https://github.com/more-more-tea/python_gil</a>. Nevertheless, it does not run as my expectation. Once the processor thread is added, the subscriber cannot receive data from the publisher in the gevent loop. If I simply drop the data processor module, the subscriber gevent loop could receive the messages from the publisher.</p>
  
  <p>Is there anything wrong with the code? I suspect the GIL interferes the concurrency of the pthread in the message processor, or the gevent loop is starved. Any hints about the question or about how to debug it will be highly appreciated!</p>