title: Avoiding 503 errors with urllib2
body: <p>I'm new to web scraping with python, so I don't know if I'm doing this right.</p>
  
  <p>I'm using a script that calls BeautifulSoup to parse the URLs from the first 10 pages of a google search. Tested with stackoverflow.com, worked just fine out-of-the-box. I tested with another site a few times, trying to see if the script was really working with higher google page requests, then it 503'd on me. I switched to another URL to test and worked for a couple, low-page requests, then also 503'd. Now every URL I pass to it is 503'ing. Any suggestions?</p>
  
  <pre><code>import sys # Used to add the BeautifulSoup folder the import path
  import urllib2 # Used to read the html document
  
  if __name__ == "__main__":
  ### Import Beautiful Soup
  ### Here, I have the BeautifulSoup folder in the level of this Python script
  ### So I need to tell Python where to look.
  sys.path.append("./BeautifulSoup")
  from BeautifulSoup import BeautifulSoup
  
  ### Create opener with Google-friendly user agent
  opener = urllib2.build_opener()
  opener.addheaders = [('User-agent', 'Mozilla/5.0')]
  
  ### Open page &amp; generate soup
  ### the "start" variable will be used to iterate through 10 pages.
  for start in range(0,10):
      url = "http://www.google.com/search?q=site:stackoverflow.com&amp;start=" + str(start*10)
      page = opener.open(url)
      soup = BeautifulSoup(page)
  
      ### Parse and find
      ### Looks like google contains URLs in &lt;cite&gt; tags.
      ### So for each cite tag on each page (10), print its contents (url)
      for cite in soup.findAll('cite'):
          print cite.text
  </code></pre>