title: Python packaging for functionality on top of binary extension module
body: <p>I have a Python package that's provided using a binary extension module. I want to extend some functionality on top of that binary and package it. The details are as follows:</p>
  <p>I have a file structure, say;</p>
  <pre class="lang-sh prettyprint-override"><code>├── somepackage
  │   ├── __init__.py
  ├── lib
  │   ├── Linux64
  │   │   └── SomePackageExtModule.so
  │   └── Win64
  │       └── SomePackageExtModule.pyd
  └── setup.py
  </code></pre>
  <p>The significant code I have in <code>setup.py</code> and <code>__init__.py</code> to package it are as follows;</p>
  <h3>setup.py</h3>
  <pre class="lang-py prettyprint-override"><code>import setuptools, os, sys, platform, distutils.dir_util
  from pathlib import Path
  
  package = &quot;somepackage&quot;
  
  #Force platform wheel
  try:
      from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
      class bdist_wheel(_bdist_wheel):
        def finalize_options(self):
          _bdist_wheel.finalize_options(self)
        self.root_is_pure = (&quot;--universal&quot; in sys.argv)
  except ImportError:
    bdist_wheel = None  
    
  pkg_dir = os.path.join(os.curdir, pkgname)
  if (&quot;sdist&quot; in sys.argv) or ((&quot;bdist_wheel&quot; in sys.argv) and (&quot;--universal&quot; in sys.argv)):
    #sdist deploys all shared libraries
    distutils.dir_util.copy_tree(&quot;lib&quot;, pkg_dir)  
  else:
    #Deploy the current platform shared library only
    ossys = platform.system()
    platmac = platform.machine()
    libdir = None
    if ossys == &quot;Windows&quot;:
      if (sys.maxsize &gt; 2**32):
        #Win x64
        libdir = &quot;Win64&quot;
      else:
        #Win x86
        libdir = &quot;Win32&quot;
    elif ossys == &quot;Linux&quot;:
      if platmac == &quot;x86_64&quot;:
        #Linux x86_64
        libdir = &quot;Linux64&quot;
  
    if libdir:
      distutils.dir_util.copy_tree(os.path.join(&quot;lib&quot;, libdir), os.path.join(pkg_dir, libdir))  
    else:
      raise ValueError(&quot;Unsupported platform.&quot;)
    
  #Create the package data.   
  pkgdata = []
  for dir_, _, files in os.walk(pkg_dir):
    for file_name in files:
      rel_dir = os.path.relpath(dir_, pkg_dir)
      rel_file = os.path.join(rel_dir, file_name)    
      if ''.join(Path(rel_file).suffixes) in ['.pyd', '.so']:
        pkgdata.append(rel_file)
  
  setuptools.setup(
    name=pkgname,
    # ...
    packages=[pkgname],
    package_data={pkgname: pkgdata},
    # ...
    cmdclass={
      'bdist_wheel': bdist_wheel
    }
  )
  </code></pre>
  <h3>__init__.py</h3>
  <pre class="lang-py prettyprint-override"><code>import sys, os, sys, platform
  import importlib, importlib.machinery, importlib.util
  
  class PyVerNotSupported(Exception):
    pass
  
  def findmodule():
    is_conda = os.path.exists(os.path.join(sys.prefix, 'conda-meta'))
    pyver = f&quot;{sys.version_info.major}.{sys.version_info.minor}&quot;
    ossys = platform.system()
    platmac = platform.machine()  
    libdir = None
    libext = None
  
    if not (pyver in [&quot;3.6&quot;, &quot;3.7&quot;, &quot;3.8&quot;, &quot;3.9&quot;, &quot;3.10&quot;]):
      raise PyVerNotSupported(f&quot;SomePackage doesn't support Python{pyver}.&quot;)
  
    if ossys == &quot;Windows&quot;:
      #Check if the current platform is Windows
      if (sys.maxsize &gt; 2**32):
        #Win x64  
        libdir = &quot;Win64&quot;
      else:
        #Win x86
        libdir = &quot;Win32&quot;
  
      libext = &quot;.pyd&quot;
    elif ossys == &quot;Linux&quot;:
      #Check if the current platform is Linux
      if platmac == &quot;x86_64&quot;:
          #Linux x86_64
          libdir = &quot;Linux64&quot;
  
      libext = &quot;.so&quot;
  
    if libdir:
      sdir = os.path.join(os.path.dirname(os.path.abspath(__file__)), libdir) 
      # print(sdir)
      if not os.path.exists(sdir):
        raise ValueError(&quot;SomePackage module not found. Try to reinstall the somepackage package or check for support compatibility.&quot;)        
      for fname in os.listdir(sdir):      
        if ('SomePackage' in fname) and (fname.endswith(libext)):
          return os.path.join(sdir, os.path.basename(fname))
      raise ValueError(&quot;SomePackage module not found. Try to reinstall the somepackage package.&quot;) 
    else:
      raise ValueError(&quot;Unsupported platform.&quot;)  
  
  def package_import():  
      modulefullpath = findmodule()   
      loader = importlib.machinery.ExtensionFileLoader(&quot;SomePackage&quot;, modulefullpath)
      spec = importlib.util.spec_from_file_location(&quot;SomePackage&quot;, modulefullpath,
          loader=loader, submodule_search_locations=None)
      ld = loader.create_module(spec)
      package = importlib.util.module_from_spec(spec)
      sys.modules[&quot;somepackage&quot;] = package
      spec.loader.exec_module(package)
      return package
      
  #Import the shared lib
  package = package_import()
  </code></pre>
  <p>Now I want to extend the functionality of base classes provided by the extension module and provide as part of the package. I figured out the functionality code using metaclasses but couldn't find how to provide that functionality as part of the package itself when others pip install it.</p>
  <p>Say, the binary extension module - <strong>SomePackage</strong> provides a class <code>SomeClass</code> with a base class implementation of:</p>
  <pre class="lang-py prettyprint-override"><code>class SomeClass:
      def __init__(self):
          self.var = 10
      def test_func(self):
          print(&quot;Inside Test func&quot;)
  </code></pre>
  <p>I want to add a new function/method as part of the <code>SomeClass</code> class which I did using the metaclass implementation. Say,</p>
  <pre class="lang-py prettyprint-override"><code>class MyMeta(type):
      def __new__(cls, clsname, bases, clsdict):
          def some_func(self, x):
              self.var += 2
              print(f&quot;Updated the value of var: {self.var}&quot;)
              self.test_func()
              return self.var + x
  
          # filter the elements which aren't in the __new__ signature
          print(locals().items())
          tmp = {k: v for k, v in locals().items() if k not in (&quot;cls&quot;, &quot;clsname&quot;, &quot;bases&quot;, &quot;clsdict&quot;)}
          for k, v in tmp.items(): clsdict[k] = v
  
          return type.__new__(cls, clsname, bases, clsdict)
  
  
  class SomeClass(SomeClass, metaclass=MyMeta):
      pass
  </code></pre>
  <p>This code works if I add the above metaclass and inherit the base class <code>SomeClass</code> in the external python code after we pip install the <strong>somepackage</strong> package. Say, I created an object of <code>SomeClass</code> and call the newly added <code>some_func</code> function:</p>
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; some_obj = SomeClass()
  &gt;&gt;&gt; some_obj.some_func(2)
  
  ## Output ##
  Updated the value of var: 12
  Inside Test func
  14
  </code></pre>
  <p>I'm thinking to provide some of such useful utility functions on top of the <strong>SomePackage</strong> extension module as part of the package installation process. I thought I might need to this layer 2 code in <code>somepackage</code> directory as follows so that tree structure will be:</p>
  <pre class="lang-sh prettyprint-override"><code>├── somepackage
  │   ├── __init__.py
  │   ├── layer_2_utils.py
  │   ├── layer_2.py
  ├── lib
  │   ├── Linux64
  │   │   └── SomePackageExtModule.so
  │   └── Win64
  │       └── SomePackageExtModule.pyd
  └── setup.py
  </code></pre>
  <p>Where <code>layer_2_utils.py</code> can contain Metaclass implementation to add required utility functions. The <code>layer_2.py</code> can contain the class inheritance extensions - <code>class(SomeClass, metaclass=MyMeta): pass</code> etc.. itself by importing the metaclass from <code>layer_2_utils.py</code>.</p>
  <p>Now, please tell me how to implement this. What should be changed in <code>setup.py</code>, <code>__init__.py</code>, and how to make this import system work to provide this pip package? If I'm thinking wrong in this import system, please point me that and provide any solution. Any help is highly appreciated. Thanks in advance. :)</p>