title: Web Scraping - recursive calls to links from href, traverse store data
body: <p>So, I am creating a web scraper that visits a website, finds all the div with <code>li elements</code>.
  Then traverses the li's and stores them inside a <code>text file</code>.</p>
  <p>Now, the website has embedded links within some li also.</p>
  <p>So, I need to traverse the links and find the li elements within them and come back to the parent page.
  <img src="https://imgur.com/tyupGeJ.jpg" alt="pageStructure" /></p>
  <p><em><strong>My Code is shared below</strong></em></p>
  <pre><code>import urllib
  import urllib.request
  from bs4 import BeautifulSoup
  
  
  def writeToFile(ul):
      for li in ul:
          with open('path/to/file.txt', 'a+') as f:
              text = li.text
              f.write(text + ',')
              f.close()
  
  
  def searchElements(url):
      print(url)
      response = urllib.request.urlopen(url)
      html = response.read()
  
      soup = BeautifulSoup(html, 'html.parser')
  
      divs = soup.findAll('div', id=lambda x: x and x.startswith('mntl-sc-block_1-0-'))
      for div in divs:
          ul = div.find(&quot;ul&quot;)
          if ul is not None:
              ulVariable = ul.findAll('a')
              for b in ulVariable:
                  if ulVariable is not None:
                      if b is not None:
                          linkItemsList = list()
                          links = (b.get(&quot;href&quot;))
                          linkItemsList.append(links)
                          for link in linkItemsList:
                              searchElements(link)
                              print('link internal data print')
                              writeToFile(ul)
                  else:
                      print('link in not none else')
                      writeToFile(ul)
          print('all non link')
          writeToFile(ul)
  
  
  def main():
      searchElements('https://www.thebalancecareers.com/list-of-information-technology-it-skills-2062410')
  
  
  if __name__ == '__main__':
      main()
  </code></pre>
  <p>I am not getting the proper logic for the recursive calls. I am getting stuck in the child pages.</p>
  <p>So, I would be really appreciative of any help that I receive</p>