title: Python web scraping page loop
body: <p>Appreciate this is been asked many time on here but I cant seem to get it to work for me.</p>
  
  <p>I've written a scraper which successfully scrapes everything I need from the first page of the site. But, I cant figure out how to get it to loop through the various pages.</p>
  
  <p>The url simply increments like this BLAH/3 + 'page=x'</p>
  
  <p>I haven't been learning to code for very long, so any advice would be appreciated! </p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  
  url = 'http://www.URL.org/BLAH1/BLAH2/BLAH3'
  
  soup = BeautifulSoup(r.content, "html.parser")
  
  # String substitution for HTML
  for link in soup.find_all("a"):
  "&lt;a href='&gt;%s'&gt;%s&lt;/a&gt;" %(link.get("href"), link.text)
  
  # Fetch and print general data from title class
  general_data = soup.find_all('div', {'class' : 'title'})
  
  
  for item in general_data:
      name = print(item.contents[0].text)
      address = print(item.contents[1].text.replace('.',''))
      care_type = print(item.contents[2].text)
  </code></pre>
  
  <p>Update:</p>
  
  <pre><code>r = requests.get('http://www.URL.org/BLAH1/BLAH2/BLAH3')
  
  for page in range(10):
  
      r = requests.get('http://www.URL.org/BLAH1/BLAH2/BLAH3' + 'page=' + page)
  
  soup = BeautifulSoup(r.content, "html.parser")
  #print(soup.prettify())
  
  
  # String substitution for HTML
  for link in soup.find_all("a"):
      "&lt;a href='&gt;%s'&gt;%s&lt;/a&gt;" %(link.get("href"), link.text)
  
  # Fetch and print general data from title class
  general_data = soup.find_all('div', {'class' : 'title'})
  
  
  for item in general_data:
      name = print(item.contents[0].text)
      address = print(item.contents[1].text.replace('.',''))
      care_type = print(item.contents[2].text)
  </code></pre>
  
  <p>Update 2!: </p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  url = 'http://www.URL.org/BLAH1/BLAH2/BLAH3&amp;page='
  
  for page in range(10):
  
  r = requests.get(url + str(page))
  
  soup = BeautifulSoup(r.content, "html.parser")
  
  # String substitution for HTML
  for link in soup.find_all("a"):
      print("&lt;a href='&gt;%s'&gt;%s&lt;/a&gt;" % (link.get("href"), link.text))
  
  # Fetch and print general data from title class
  general_data = soup.find_all('div', {'class' : 'title'})
  
  for item in general_data:
      print(item.contents[0].text)
      print(item.contents[1].text.replace('.',''))
      print(item.contents[2].text)
  </code></pre>