title: Why is urllib.urlopen() only working once? - Python
body: <p>I'm writing a crawler to download the static html pages using <code>urllib</code>. </p>
  
  <p>The get_page function works for 1 cycle but when i try to loop it, it doesn't open the content to the next url i've fed in. </p>
  
  <ul>
  <li><strong>How do i make <code>urllib.urlopen</code> continuously download HTML pages?</strong> </li>
  <li><em>If it is not possible, is there any other suggestion to download
  webpages within my python code?</em></li>
  </ul>
  
  <p>my code below only returns the html for the 1st website in the <code>seed</code> list:</p>
  
  <pre><code>import urllib
  def get_page(url):
      return urllib.urlopen(url).read().decode('utf8')
  
  seed = ['http://www.pmo.gov.sg/content/pmosite/home.html', 
              'http://www.pmo.gov.sg/content/pmosite/aboutpmo.html']    
  
  for j in seed:
        print "here"
        print get_page(j)
  </code></pre>
  
  <p>The same crawl "once-only" problem also occurs with <code>urllib2</code>:</p>
  
  <pre><code>import urllib2
  def get_page(url):
      req = urllib2.Request(url)
      response = urllib2.urlopen(req)
      return response.read().decode('utf8')
  
  seed = ['http://www.pmo.gov.sg/content/pmosite/home.html', 
              'http://www.pmo.gov.sg/content/pmosite/aboutpmo.html']    
  
  for j in seed:
        print "here"
        print get_page(j)
  </code></pre>
  
  <p>Without the exception, i'm getting an IOError with urllib:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/alvas/workspace/SingCorp/sgcrawl.py", line 91, in &lt;module&gt;
      print get_page(j)
    File "/home/alvas/workspace/SingCorp/sgcrawl.py", line 4, in get_page
      return urllib.urlopen(url).read().decode('utf8')
    File "/usr/lib/python2.7/urllib.py", line 86, in urlopen
      return opener.open(url)
    File "/usr/lib/python2.7/urllib.py", line 207, in open
      return getattr(self, name)(url)
    File "/usr/lib/python2.7/urllib.py", line 462, in open_file
      return self.open_local_file(url)
    File "/usr/lib/python2.7/urllib.py", line 476, in open_local_file
      raise IOError(e.errno, e.strerror, e.filename)
  IOError: [Errno 2] No such file or directory: 'http://www.pmo.gov.sg/content/pmosite/aboutpmo.html'
  </code></pre>
  
  <p>Without the exception, i'm getting a ValueError with urllib2:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/alvas/workspace/SingCorp/sgcrawl.py", line 95, in &lt;module&gt;
      print get_page(j)
    File "/home/alvas/workspace/SingCorp/sgcrawl.py", line 7, in get_page
      response = urllib2.urlopen(req)
    File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
      return _opener.open(url, data, timeout)
    File "/usr/lib/python2.7/urllib2.py", line 392, in open
      protocol = req.get_type()
    File "/usr/lib/python2.7/urllib2.py", line 254, in get_type
      raise ValueError, "unknown url type: %s" % self.__original
  ValueError: unknown url type: http://www.pmo.gov.sg/content/pmosite/aboutpmo.html
  </code></pre>
  
  <p><strong>ANSWERED:</strong></p>
  
  <p>The IOError and ValueError occurred because there was some sort of Unicode byte order mark (BOM). A non-break space was found in the second URL. Thanks for all your help and suggestion in solving the problem!!</p>