title: How to invoke Python function as a callback inside C++ thread using pybind11
body: <p>I designed a C++ system that invokes user defined callbacks from procedure running in a separate thread. Simplified <code>system.hpp</code> looks like this:</p>
  
  <pre><code>#pragma once
  
  #include &lt;atomic&gt;
  #include &lt;chrono&gt;
  #include &lt;functional&gt;
  #include &lt;thread&gt;
  
  class System
  {
  public:
    using Callback = std::function&lt;void(int)&gt;;
    System(): t_(), cb_(), stop_(true) {}
    ~System()
    {
      stop();
    }
    bool start()
    {
      if (t_.joinable()) return false;
      stop_ = false;
      t_ = std::thread([this]()
      {
        while (!stop_)
        {
          std::this_thread::sleep_for(std::chrono::milliseconds(100));
          if (cb_) cb_(1234);
        }
      });
      return true;
    }
    bool stop()
    {
      if (!t_.joinable()) return false;
      stop_ = true;
      t_.join();
      return true;
    }
    bool registerCallback(Callback cb)
    {
      if (t_.joinable()) return false;
      cb_ = cb;
      return true;
    }
  
  private:
    std::thread t_;
    Callback cb_;
    std::atomic_bool stop_;
  };
  </code></pre>
  
  <p>It works just fine and can be tested with this short example <code>main.cpp</code>:</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include "system.hpp"
  
  int g_counter = 0;
  
  void foo(int i)
  {
    std::cout &lt;&lt; i &lt;&lt; std::endl;
    g_counter++;
  }
  
  int main()
  {
    System s;
    s.registerCallback(foo);
    s.start();
    while (g_counter &lt; 3)
    {
      std::this_thread::sleep_for(std::chrono::milliseconds(1));
    }
    s.stop();
    return 0;
  }
  </code></pre>
  
  <p>which will output <code>1234</code> a few times and then will stop. However I encountered a problem trying to create python bindings for my <code>System</code>. If I register a python function as a callback, my program will deadlock after calling <code>System::stop</code>. I investigated the topic a bit and it seems that I face the issue with <a href="https://wiki.python.org/moin/GlobalInterpreterLock" rel="noreferrer">GIL</a>. Reproducible example:</p>
  
  <p><code>binding.cpp</code>:</p>
  
  <pre><code>#include "pybind11/functional.h"
  #include "pybind11/pybind11.h"
  
  #include "system.hpp"
  
  namespace py = pybind11;
  
  PYBIND11_MODULE(mysystembinding, m) {
    py::class_&lt;System&gt;(m, "System")
      .def(py::init&lt;&gt;())
      .def("start", &amp;System::start)
      .def("stop", &amp;System::stop)
      .def("registerCallback", &amp;System::registerCallback);
  }
  </code></pre>
  
  <p>python script:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import mysystembinding
  import time
  
  g_counter = 0
  
  def foo(i):
    global g_counter
    print(i)
    g_counter = g_counter + 1
  
  s = mysystembinding.System()
  s.registerCallback(foo)
  s.start()
  while g_counter &lt; 3:
    time.sleep(1)
  s.stop()
  </code></pre>
  
  <p>I have read the <a href="https://pybind11.readthedocs.io/en/stable/advanced/misc.html" rel="noreferrer">pybind11 docs</a> section about the possibility to acquire or release GIL on the C++ side. However I did not manage to get rid of the deadlock that occurs in my case:</p>
  
  <pre><code>PYBIND11_MODULE(mysystembinding, m) {
    py::class_&lt;System&gt;(m, "System")
      .def(py::init&lt;&gt;())
      .def("start", &amp;System::start)
      .def("stop", &amp;System::stop)
      .def("registerCallback", [](System* s, System::Callback cb)
        {
          s-&gt;registerCallback([cb](int i)
          {
            // py::gil_scoped_acquire acquire;
            // py::gil_scoped_release release;
            cb(i);
          });
        });
  }
  </code></pre>
  
  <p>If I call <code>py::gil_scoped_acquire acquire;</code> before calling the callback, deadlock occurs anyway.
  If I call <code>py::gil_scoped_release release;</code> before calling the callback, I get</p>
  
  <blockquote>
    <p>Fatal Python error: PyEval_SaveThread: NULL tstate</p>
  </blockquote>
  
  <p>What should I do to register python functions as callbacks and avoid deadlocks?</p>