title: Pagination with BeautifulSoup
body: <p>I am trying to get some data from the following website. <a href="https://www.drugbank.ca/drugs" rel="nofollow noreferrer">https://www.drugbank.ca/drugs</a></p>
  <p>For every drug in the table, I will need to go deeply and have the name and some other specific features like categories, structured indication (please click on drug name to see the features I will use).</p>
  <p>I wrote the following code but the issue that I can't make my code handle pagination (as you see there more than 2000 pages!).</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  
  def drug_data():
  url = 'https://www.drugbank.ca/drugs/'
  r = requests.get(url)
  soup = BeautifulSoup(r.text ,&quot;lxml&quot;)
  for link in soup.select('name-head a'):
      href = 'https://www.drugbank.ca/drugs/' + link.get('href')
      pages_data(href)
  
  
  def pages_data(item_url):
  r = requests.get(item_url)
  soup = BeautifulSoup(r.text, &quot;lxml&quot;)
  g_data = soup.select('div.content-container')
  
  for item in g_data:
      print item.contents[1].text
      print item.contents[3].findAll('td')[1].text
      try:
          print item.contents[5].findAll('td',{'class':'col-md-2 col-sm-4'})
      [0].text
      except:
          pass
      print item_url
      drug_data()
  </code></pre>
  <p>How can I scrape all of the data and handle pagination properly?</p>