title: Simulating sub packages using Python import hook
body: <p>I'm going to hack on Python import system.  Suppose we have the following directory structure:</p>
  
  <pre><code>.
  ├── main
  │   ├── main.py
  │   └── parent
  │       └── __init__.py
  └── pkg1
      ├── __init__.py
      ├── sub
      │   ├── __init__.py
      │   └── import_global.py
      └── success.py
  </code></pre>
  
  <p>The launch script would be <code>main.py</code>, so there should be a top-most module <code>parent</code>.  Now, I want to simulate a subpackage, whose fullname is <code>parent.intermediate.pkg1</code>, that indeed refers to the <code>pkg1</code> directory.</p>
  
  <p>There actually does not exist the <code>intermediate</code> module, however, I indeed need to simulate one (in my real project, the name of this intermediate module will be dynamically generated).  So I decide to use the Python import hooks.</p>
  
  <p>First, let me introduce the contents of <code>pkg1</code>.</p>
  
  <p>pkg1/sub/import_global.py:</p>
  
  <pre><code>from operator import add
  Value = add(1, 2)
  </code></pre>
  
  <p>pkg1/success.py:</p>
  
  <pre><code>Value = 'Success'
  </code></pre>
  
  <p>And (part of main.py), I made some test cases:</p>
  
  <pre><code>class MainTestCase(unittest.TestCase):
      def test_success(self):
          from parent.intermediate.pkg1 import success
          self.assertEqual(success.Value, "Success")
  
      def test_import_global(self):
          from parent.intermediate.pkg1.sub import import_global
          self.assertEqual(import_global.Value, 3)
  
      def test_not_found(self):
          def F():
              from parent.intermediate.pkg1 import not_found
          self.assertRaises(ImportError, F)
  
  
  unittest.main()
  </code></pre>
  
  <p>All of the <code>__init__.py</code> are empty.  Now it's going to implement the import hooks. I've drafted two versions, each has some problem.</p>
  
  <p>The first version:</p>
  
  <pre><code>class PkgLoader(object):
      def install(self):
          sys.meta_path[:] = [x for x in sys.meta_path if self != x] + [self]
  
      def find_module(self, fullname, path=None):
          if fullname.startswith('parent.'):
              return self
  
      def load_module(self, fullname):
          if fullname in sys.modules:
              return sys.modules[fullname]
          parts = fullname.split('.')[1:]
          path = os.path.join(os.path.dirname(__file__), '..')
          # intermediate module
          m = None
          ns = 'parent.intermediate'
          if ns in sys.modules:
              m = sys.modules[ns]
          elif parts[0] == 'intermediate':
              m = imp.new_module(ns)
              m.__name__ = ns
              m.__path__ = [ns]
              m.__package__ = '.'.join(ns.rsplit('.', 1)[:-1])
          else:
              raise ImportError("Module %s not found." % fullname)
          # submodules
          for p in parts[1:]:
              ns = '%s.%s' % (ns, p)
              fp, filename, options = imp.find_module(p, [path])
              if ns in sys.modules:
                  m = sys.modules[ns]
              else:
                  m = imp.load_module(ns, fp, filename, options)
                  sys.modules[ns] = m
              path = filename
          return m
  
  loader = PkgLoader()
  loader.install()
  </code></pre>
  
  <p>Where the <code>test_import_global</code> fails:</p>
  
  <pre><code>E..
  ======================================================================
  ERROR: test_import_global (__main__.MainTestCase)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "main.py", line 54, in test_import_global
      from parent.intermediate.pkg1.sub import import_global
    File "main.py", line 39, in load_module
      m = imp.load_module(ns, fp, filename, options)
    File "../pkg1/sub/import_global.py", line 1, in &lt;module&gt;
      from operator import add
    File "main.py", line 35, in load_module
      fp, filename, options = imp.find_module(p, [path])
  ImportError: No module named operator
  
  ----------------------------------------------------------------------
  Ran 3 tests in 0.005s
  
  FAILED (errors=1)
  </code></pre>
  
  <p>Now for the second version, I modified <code>load_module</code>:</p>
  
  <pre><code>def load_module(self, fullname):
      if fullname in sys.modules:
          return sys.modules[fullname]
      parts = fullname.split('.')[1:]
      path = os.path.join(os.path.dirname(__file__), '..')
      # intermediate module
      m = None
      ns = 'parent.intermediate'
      if ns in sys.modules:
          m = sys.modules[ns]
      elif parts[0] == 'intermediate':
          m = imp.new_module(ns)
          m.__name__ = ns
          m.__path__ = [ns]
          m.__package__ = '.'.join(ns.rsplit('.', 1)[:-1])
      else:
          raise ImportError("Module %s not found." % fullname)
      # submodules
      for p in parts[1:]:
          ns = '%s.%s' % (ns, p)
          # ======&gt; The modification starts here &lt;======
          try:
              fp, filename, options = imp.find_module(p, [path])
          except ImportError:
              return None
          # ======&gt; The modification ends here &lt;======
          if ns in sys.modules:
              m = sys.modules[ns]
          else:
              m = imp.load_module(ns, fp, filename, options)
              sys.modules[ns] = m
          path = filename
      return m
  </code></pre>
  
  <p>Where the <code>test_not_found</code> fails:</p>
  
  <pre><code>.F.
  ======================================================================
  FAIL: test_not_found (__main__.MainTestCase)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "main.py", line 65, in test_not_found
      self.assertRaises(ImportError, F)
  AssertionError: ImportError not raised
  
  ----------------------------------------------------------------------
  Ran 3 tests in 0.004s
  
  FAILED (failures=1)
  </code></pre>
  
  <p>So the question is now clear: how can I implement the import hook, so that all these three test cases can pass?</p>