title: Crawler : go to the next page, best pratice
body: <p>I currently write a python crawler, I want to switch to the next page but what is the best pratice ?</p>
  
  <p>Actually it's simple, the end of url is .html?page=1, so I can increment page number but is there a best pratice to do this thing as clean as possible ?</p>
  
  <p>I use urllib, url parse and beautifulSoup</p>
  
  <pre><code>#!/usr/bin/env python2
  import urllib
  import urlparse
  from bs4 import BeautifulSoup
  
  
  def getURL():
      try:
          fo = open("WebsiteToCrawl", "rw")
          print ok() + "Data to crawl a store in : ", fo.name
      except:
          print fail() + "File doesn't exist, please create WebSiteTOCrawl file for store website     listing"
      line = fo.readlines()
      print ok() + "Return website : %s" % (line)
      fo.close()
      i= 0
      while i&lt;len(line):
          try:
              returnDATA = urllib.urlopen(line[i]).read()
              print ok() + "Handle :" + line[i]            
              handleDATA(returnDATA)
          except:
              print fail() + "Can't open url"
          i += 1
  
  def handleDATA(returnDATA):
  try:
          soup = BeautifulSoup(returnDATA)
          for link in soup.find_all('a'):
              urls = link.get('href')
          try:
             print urls
          except:
              print end() + "EOF: All site crawled"
  
  def main():
      useDATA = getURL()
      handleDATA(useDATA)
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>NB: I've simpfly the code than the original</p>