title: Using BeautifulSoup 4 and recursion to capture the structure of HTML nested tags
body: <p>I've been trying to capture the layout of an HTML page using BeautifulSoup4 and recursion. The idea is to have linked data structures of parents to children, for example, a layout like this:</p>
  
  <pre><code>&lt;html&gt;
   &lt;h1&gt;
    &lt;!--Contents--&gt;
   &lt;/h1&gt;
   &lt;div&gt; 
    &lt;div&gt; 
     &lt;!--Contents--&gt;
    &lt;/div&gt;
   &lt;/div&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>Would be stored in a list like so:</p>
  
  <pre><code>html = [ h1 , div ] # Where h1 and div are also lists also containing lists
  </code></pre>
  
  <p>I've had a hard time finding Q&amp;A on this specific problem, so instead I've tried modeling a function off of using recursion to traverse <a href="https://stackoverflow.com/questions/39879638/recursively-go-through-all-directories-until-you-find-a-certain-file-in-python">directories</a> as they're pretty similar.</p>
  
  <p>This is my current function in Python 3 that's supposed to nest tags into lists:</p>
  
  <pre><code>def listGen(l , hObj):
      # Where hObj is a BS4 object and l is a sorted lists containing direct children to the html tag
      z = []
      for x in l:
          z.append(hObj.find(x).children)
  
      def expand(xlist1):
          # Where xlist1 is a list generator 
          for n in xlist1:
              if n.name is not None:
                  print(n.name)
                  for n2 in hObj.find(n.name).children:
                      if n2.name is not None:
                          print(n2.name , "--") #Debugging print              
          return z #Temporary
      for x in z:
          print("------")
          expand(x , 0)
      return z
  </code></pre>
  
  <p>Parsing the <a href="https://www.wikipedia.org/" rel="nofollow noreferrer">Wikipedia Home Page</a> gives me an output of:</p>
  
  <pre><code>------
  h1
  img --
  div --
  div
  div --
  strong --
  div
  div --
  strong --
  div
  div --
  strong --
  div
  div --
  strong --
  div
  div --
  strong --
  hr
  div
  div --
  strong --
  p
  small --
  small --
  small --
  script
  script
  script
  style
  ------
  meta
  title
  meta
  script
  meta
  link
  link
  link
  style
  style
  link
  link
  </code></pre>
  
  <p>Which is exactly what I need, <strong><em>however</em></strong> it takes two for loops and will take many more to get all the children of children. Furthermore I wouldn't know how much tags nest from future websites. So I changed the <code>expand</code> function to:</p>
  
  <pre><code>def expand(xlist1 , depth): 
      l1 = list(xlist1)
      if depth &lt; len(l1):
          for n in l1[depth]:
              if n is not None:
                  if hObj.find(l1[depth].name).children:
                      return expand(hObj.find(l1[depth].name).children , 0)
              if n is None:
                  print(2) # Debugging print
                  return expand(xlist1 , depth + 1)
      if depth &gt;= len(l1):
          return 0 # Temporary
      return 0 # Temporary
  </code></pre>
  
  <p>Only to give me maximum recursion errors, I've tried many other variations of it all to no avail.</p>
  
  <p>I've scoured through the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow noreferrer">BS4 Docs</a> multiple times and there seems to be no built in function for this. Any suggestions or is this not a viable way to achieve what I'm looking for?</p>