title: Amazon page links with request_HTML
body: <p>I am scraping some product details. I wrote a program that worked fine on the first day, but on the next day when I ran it two or three times it started to give me an error, when I tried to move on to the next page. It moved to the second page but not to the third.</p>
  <p>My program:</p>
  <pre class="lang-none prettyprint-override"><code>from requests_html import HTMLSession
  
  session = HTMLSession()
  urls = []
  data = []
  url = &quot;https://www.amazon.com/s?k=gpu+graphics+card&amp;i=computers-intl-ship&amp;qid=1651764839&amp;sprefix=gpu%2Ccomputers-intl-ship%2C453&amp;ref=sr_pg_1&quot;
  
  
  def getdata(url):
      r = session.get(url)
      r.html.render(timeout=100)
      # products = r.html.find(&quot;div[data-asin]&quot;)
      #
      # for product in products:
      #     item = {}
      #     try:
      #         title = product.find('span.a-size-medium.a-color-base.a-text-normal')[0].text
      #     except:
      #         title = ''
      #     try:
      #         rating = product.find('span.a-icon-alt')[0].text
      #     except:
      #         rating = ''
      #     try:
      #         price = product.find('span.a-price-whole')[0].text
      #     except:
      #         price = ''
      #     item['Title'] = title
      #     item['rating'] = rating
      #     item['Price'] = price
      #     data.append(item)
      return r
  
  def nextpage(link):
      nextpage_menu = link.html.find(&quot;.s-pagination-strip&quot;, first=True)
  
      if not nextpage_menu.find(&quot;span.s-pagination-item.s-pagination-next.s-pagination-disabled&quot;):
          all_aTags = nextpage_menu.find(&quot;a&quot;)
          next_url = &quot;https://www.amazon.com/&quot; + str(all_aTags[len(all_aTags) - 1].attrs['href'])
          urls.append(next_url)
          return next_url
      else:
          urls.append('')
          return
  
  while True:
      link = getdata(url)
      url = nextpage(link)
      if not url:
          break
      print(url)
  print(&quot;++++++++++++++++++++++++++++&quot;)
  print(urls)
  print(data)
  </code></pre>
  <p>Error:</p>
  <blockquote>
  <p>https://www.amazon.com//s?k=gpu+graphics+card&amp;i=computers-intl-ship&amp;page=2&amp;qid=1651923364&amp;sprefix=gpu%2Ccomputers-intl-ship%2C453&amp;ref=sr_pg_1
  Traceback (most recent call last):
  File &quot;C:/Users/Muhammad/Documents/HTML_CSS_JAVASCRIPT/Scraping_modules/amazon_links.py&quot;, line 50, in 
  url = nextpage(link)
  File &quot;C:/Users/Muhammad/Documents/HTML_CSS_JAVASCRIPT/Scraping_modules/amazon_links.py&quot;, line 38, in nextpage
  if not nextpage_menu.find(&quot;span.s-pagination-item.s-pagination-next.s-pagination-disabled&quot;):
  AttributeError: 'NoneType' object has no attribute 'find'</p>
  </blockquote>
  <p>You can see that it scrapes the next page link, but it doesn't scrape any further. I checked the element section on Chrome Developer Tools. The <code>.s-pagination-strip</code> class is present there, but it's returning nothing.</p>