title: How to tidy/fix PyCXX's creation of new-style Python extension-class?
body: <p>I've nearly finished rewriting a C++ Python wrapper (PyCXX).</p>
  
  <p>The original allows old and new style extension classes, but also allows one to derive from the new-style classes:</p>
  
  <pre><code>import test
  
  // ok
  a = test.new_style_class();
  
  // also ok
  class Derived( test.new_style_class() ):
      def __init__( self ):
          test_funcmapper.new_style_class.__init__( self )
  
      def derived_func( self ):
          print( 'derived_func' )
          super().func_noargs()
  
      def func_noargs( self ):
          print( 'derived func_noargs' )
  
  d = Derived()
  </code></pre>
  
  <p>The code is convoluted, and appears to contain errors (<a href="https://stackoverflow.com/questions/27564257/why-does-pycxx-handle-new-style-classes-in-the-way-it-does">Why does PyCXX handle new-style classes in the way it does?</a>)</p>
  
  <p>My question is: <strong>What is the rationale/justification for PyCXX's convoluted mechanism? Is there a cleaner alternative?</strong></p>
  
  <p>I will attempt to detail below where I am at with this enquiry.  First I will try and describe what PyCXX is doing at the moment, then I will describe what I think could maybe be improved.</p>
  
  <hr>
  
  <p>When the Python runtime encounters <code>d = Derived()</code>, it does <code>PyObject_Call( ob ) where ob is the</code>PyTypeObject<code>for</code>NewStyleClass<code>. I will write</code>ob<code>as</code>NewStyleClass_PyTypeObject`.</p>
  
  <p>That PyTypeObject has been constructed in C++ and registered using <code>PyType_Ready</code></p>
  
  <p><code>PyObject_Call</code> will invoke <code>type_call(PyTypeObject *type, PyObject *args, PyObject *kwds)</code>, returning an initialised Derived instance i.e.</p>
  
  <pre><code>PyObject* derived_instance = type_call(NewStyleClass_PyTypeObject, NULL, NULL)
  </code></pre>
  
  <p>Something like this.</p>
  
  <p>(All of this coming from (<a href="http://eli.thegreenplace.net/2012/04/16/python-object-creation-sequence" rel="nofollow noreferrer">http://eli.thegreenplace.net/2012/04/16/python-object-creation-sequence</a> by the way, thanks Eli!)</p>
  
  <p>type_call does essentially:</p>
  
  <pre><code>type-&gt;tp_new(type, args, kwds);
  type-&gt;tp_init(obj, args, kwds);
  </code></pre>
  
  <p>And our C++ wrapper has inserted functions into the <code>tp_new</code> and <code>tp_init</code> slots of <code>NewStyleClass_PyTypeObject</code> something like this:</p>
  
  <pre><code>typeobject.set_tp_new( extension_object_new );
  typeobject.set_tp_init( extension_object_init );
  
  :
      static PyObject* extension_object_new( PyTypeObject* subtype, 
                                                PyObject* args, PyObject* kwds )
      {
          PyObject* pyob = subtype-&gt;tp_alloc(subtype,0);
  
          Bridge* o = reinterpret_cast&lt;Bridge *&gt;( pyob );
  
          o-&gt;m_pycxx_object = nullptr;
  
          return pyob;
      }
  
      static int extension_object_init( PyObject* _self, 
                                              PyObject* args, PyObject* kwds )
      {
          Bridge* self{ reinterpret_cast&lt;Bridge*&gt;(_self) };
  
          // NOTE: observe this is where we invoke the constructor, 
          //       but indirectly (i.e. through final)
          self-&gt;m_pycxx_object = new FinalClass{ self, args, kwds };
  
          return 0;
      }
  </code></pre>
  
  <p>Note that we need to bind together the Python Derived instance, and it's corresponding C++ class instance. (Why? Explained below, see 'X').  To do that we are using:</p>
  
  <pre><code>struct Bridge
  {
      PyObject_HEAD // &lt;-- a PyObject
      ExtObjBase* m_pycxx_object;
  }
  </code></pre>
  
  <p>Now this bridge raises a question. I'm very suspicious of this design.</p>
  
  <p>Note how memory was allocated for this new PyObject:</p>
  
  <pre><code>        PyObject* pyob = subtype-&gt;tp_alloc(subtype,0);
  </code></pre>
  
  <p>And then we typecast this pointer to <code>Bridge</code>, and use the 4 or 8 (<code>sizeof(void*)</code>) bytes immediately following the <code>PyObject</code> to point to the corresponding C++ class instance (this gets hooked up in <code>extension_object_init</code> as can be seen above).</p>
  
  <p>Now for this to work we require:</p>
  
  <p>a) <code>subtype-&gt;tp_alloc(subtype,0)</code> must be allocating an extra <code>sizeof(void*)</code> bytes
   b) The <code>PyObject</code> doesn't require any memory beyond <code>sizeof(PyObject_HEAD)</code>, because if it did then this would be conflicting with the above pointer</p>
  
  <p>One major question I have at this point is:
  Can we guarantee that the <code>PyObject</code> that the Python runtime has created for our <code>derived_instance</code> does not overlap into Bridge's <code>ExtObjBase* m_pycxx_object</code> field?</p>
  
  <p>I will attempt to answer it: it is US determining how much memory gets allocated. When we create <code>NewStyleClass_PyTypeObject</code> we feed in how much memory we want this <code>PyTypeObject</code> to allocate for a new instance of this type:</p>
  
  <pre><code>template&lt; TEMPLATE_TYPENAME FinalClass &gt;
  class ExtObjBase : public FuncMapper&lt;FinalClass&gt; , public ExtObjBase_noTemplate
  {
  protected:
      static TypeObject&amp; typeobject()
      {
          static TypeObject* t{ nullptr };
          if( ! t )
              t = new TypeObject{ sizeof(FinalClass), typeid(FinalClass).name() };
                     /*           ^^^^^^^^^^^^^^^^^ this is the bug BTW!
                          The C++ Derived class instance never gets deposited
                          In the memory allocated by the Python runtime
                          (controlled by this parameter)
  
                          This value should be sizeof(Bridge) -- as pointed out
                          in the answer to the question linked above
  
          return *t;
      }
  :
  }
  
  class TypeObject
  {
  private:
      PyTypeObject* table;
  
      // these tables fit into the main table via pointers
      PySequenceMethods*       sequence_table;
      PyMappingMethods*        mapping_table;
      PyNumberMethods*         number_table;
      PyBufferProcs*           buffer_table;
  
  public:
      PyTypeObject* type_object() const
      {
          return table;
      }
  
      // NOTE: if you define one sequence method you must define all of them except the assigns
  
      TypeObject( size_t size_bytes, const char* default_name )
          : table{ new PyTypeObject{} }  // {} sets to 0
          , sequence_table{}
          , mapping_table{}
          , number_table{}
          , buffer_table{}
      {
          PyObject* table_as_object = reinterpret_cast&lt;PyObject* &gt;( table );
  
          *table_as_object = PyObject{ _PyObject_EXTRA_INIT  1, NULL }; 
          // ^ py_object_initializer -- NULL because type must be init'd by user
  
          table_as_object-&gt;ob_type = _Type_Type();
  
          // QQQ table-&gt;ob_size = 0;
          table-&gt;tp_name              = const_cast&lt;char *&gt;( default_name );
          table-&gt;tp_basicsize         = size_bytes;
          table-&gt;tp_itemsize          = 0; // sizeof(void*); // so as to store extra pointer
  
          table-&gt;tp_dealloc           = ...
  </code></pre>
  
  <p>You can see it going in as <code>table-&gt;tp_basicsize</code></p>
  
  <p>But now it seems clear to me that PyObject-s generated from <code>NewStyleClass_PyTypeObject</code> will never require additional allocated memory.</p>
  
  <p>Which means that this whole <code>Bridge</code> mechanism is unnecessary.</p>
  
  <p>And PyCXX's original technique for using PyObject as a base class of <code>NewStyleClassCXXClass</code>, and initialising this base so that the Python runtime's PyObject for <code>d = Derived()</code> is in fact this base, this technique is looking good. Because it allows seamless typecasting.</p>
  
  <p>Whenever Python runtime calls a slot from <code>NewStyleClass_PyTypeObject</code>, it will be passing a pointer to d's PyObject as the first parameter, and we can just typecast back to <code>NewStyleClassCXXClass</code>. &lt;-- 'X' (referenced above)</p>
  
  <p>So really my question is: why don't we just do this? <strong><em>Is there something special about deriving from <code>NewStyleClass</code> that forces extra allocation for the PyObject</em></strong>?</p>
  
  <p>I realise I don't understand the creation sequence in the case of a derived class. Eli's post didn't cover that.</p>
  
  <p>I suspect this may be connected with the fact that</p>
  
  <pre><code>    static PyObject* extension_object_new( PyTypeObject* subtype, ...
  </code></pre>
  
  <p>^ this variable name is 'subtype'
  I don't understand this, and I wonder if this may hold the key.</p>
  
  <p>EDIT: I thought of one possible explanation for why PyCXX is using sizeof(FinalClass) for initialisation. It might be a relic from an idea that got tried and discarded. i.e. If Python's tp_new call allocates enough space for the FinalClass (which has the PyObject as base), maybe a new FinalClass can be generated on that exact location using 'placement new', or some cunning reinterpret_cast business. My guess is this might have been tried, found to pose some problem, worked around, and the relic got left behind.</p>