title: Python BeautifulSoup4 WebCrawler .findAll() not parsing
body: <p>All! </p>
  
  <p>I am attempting to make a python web scraper to pull all the product names from a retail website. The code to do this (in PyCharm) is below:</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  def louis_spider(max_pages):
      page = 0
      while page &lt;= max_pages:
              url = 'https://us.testcompany.com/eng-us/women/hanbags/_/N-r4xtxc/to-' + str(page)
              source_code = requests.get(url)
              plain_text = source_code.text
              soup = BeautifulSoup(plain_text, 'html.parser')
              for eachItem in soup.findAll('main', {'class': 'content'}):
                 printable = eachItem.get('id')
                 print(printable)
                 print('Test1')
              page += 1
  
  louis_spider(0)
  </code></pre>
  
  <p>As it currently is (above), the code does not print anything - not even "Test1." I've run this with other inputs in the .findAll()&amp;.get() methods with luck:
  <code>.findAll('a', {'class':'skiplinks'})</code>and<code>.get('href')</code> yielded '#content Test1' and <code>.findAll('div', {'id':'privateModeMessage'})</code>and<code>.get('style')</code> yielded 'display:none Test1'. Here is a section of the 'inspect element' code from the website, for your reference: </p>
  
  <p><a href="https://i.stack.imgur.com/O0vmk.png" rel="nofollow noreferrer">a snippet of the website's code, providing context for my mentioned attempts which worked</a></p>
  
  <p>Unfortunately, my code block above, as is, does not yield anything! The problem seems to arise when I try referencing items in the <code>&lt;main&gt;</code> section - I get results when referencing the lines up until it. Ideally I would be able to pull the name of each item on the webpage (see this other snapshot of the website's code for a particular reference to what the relevant lines from the website). These lines are within the <code>&lt;main&gt;</code> portion of the website's code, so I suspect my for loop is never entered here for the same reason it isn't at any of the other lines within <code>&lt;main&gt;</code>, like those in my block above...
  <a href="https://i.stack.imgur.com/ErXJk.png" rel="nofollow noreferrer">the way I'd write this is .findAll('a', {'class': 'productName'}): and .get('class')</a></p>
  
  <p>This being said, I can't find a reason stuff in <code>&lt;main&gt;</code> would be inaccessible to BeautifulSoup. Does anyone know why this appears to be happening? Thanks in advance!</p>