title: Cannot web scrape a page which includes pagination and products in grid layout using python
body: <p>I want to web scrape the following webpage </p>
  
  <p><a href="https://www.websupplies.gr/laptop#/pageSize=48&amp;viewMode=list&amp;orderBy=10&amp;pageNumber=1" rel="nofollow noreferrer">https://www.websupplies.gr/laptop#/pageSize=48&amp;viewMode=list&amp;orderBy=10&amp;pageNumber=1</a></p>
  
  <p>But I keep getting a part of the url links the first 12 of the first 2 pages, not the 3rd, and not the total links. I used the following:</p>
  
  <pre><code>initial_url = 'https://www.websupplies.gr/laptop#/pageSize=48&amp;viewMode=list&amp;orderBy=10'
  
  caturl = 'https://www.websupplies.gr/laptop#/pageSize=48&amp;viewMode=list&amp;orderBy=10&amp;pageNumber={}'
  
      r = requests.get(initial_url)        
  
      if r.status_code == 200:
  
          Myhtml = r.text        
          soup = BeautifulSoup(Myhtml, 'html.parser')        
          #GETTING THE LAST PAGE
          last_page = soup.find('div', class_='pager').find('li', class_='next-page').a['href'].split('=')[1]
      #GETTING THE PAGE URL LINKS
      dept_page_url = [caturl.format(i) for i in range(1, int(last_page)+1)]
  
      time.sleep(2)      
  
      for pageurl in dept_page_url:        
  
          r = requests.get(pageurl)
  
          if r.status_code == 200:
  
              Myhtml = r.text        
              soup = BeautifulSoup(Myhtml, 'html.parser')
              #GETTING THE PRODUCT LINKS
              productlist = soup.find('div',attrs={'class':'item-grid'})    
  
              atagslist = productlist.findAll('a', href=True)
  
              links_with_text = []
              final_links = []
              for a in atagslist:
                  if a.text: 
                      mlink = a['href']
                      if mlink !='#':
                          links_with_text.append(infodomain+mlink)
                  #DELETE DUPLICATES
                  links_with_text = list(dict.fromkeys(links_with_text))
  
          links_with_text.extend(links_with_text)
  </code></pre>
  
  <p>How can I get all URL links</p>