title: BeautifulSoup </div> scraping suddenly stopped working
body: <p>I am trying to scrape some <code>div</code>s from a NASA website and put all the contents in a list. <strong>THIS CODE WAS WORKING EARLIER</strong> and has suddenly decided not to. I didn't change anything knowingly except to add some print statements, all of which return nothing or <code>[]</code> with no error of any kind.  </p>
  
  <pre><code>import re
  import urllib2 
  from BeautifulSoup import BeautifulSoup as soup
  import ssl
  
  
  url = "https://climate.nasa.gov"
  context = ssl._create_unverified_context()
  gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
  web_soup = soup(urllib2.urlopen(url,  context=context))
  
  l = []
  
  # get main-content div
  main_div = web_soup.findAll(name="div", attrs={'class': 'change_number'})
  print main_div
  for element in main_div:
      print element
      l.append(float(str(element)[27:-7]))
  
  print l
  </code></pre>
  
  <p>Any help with pinpointing this sudden error would be greatly appreciated!</p>
  
  <p><strong>UPDATE1</strong>: Just tried it in the interpreter, fails in the same way. <code>main_div</code> seems to be returning <code>[]</code>.</p>
  
  <p><strong>UPDATE2</strong>: Just checked the website to ensure that the div <code>change_number</code> exists. It does. </p>
  
  <p><strong>UPDATE3</strong>: Now I am really confused. I have this code, which I am pretty sure is basically identical to the one above: </p>
  
  <pre><code>import re
  import urllib2 
  from BeautifulSoup import BeautifulSoup as soup
  import ssl
  
  
  url = "https://climate.nasa.gov"
  context = ssl._create_unverified_context()
  gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
  web_soup = soup(urllib2.urlopen(url,  context=context))
  
  l = []
  
  # get main-content div
  main_div = web_soup.findAll(name="div", attrs={'class': 'change_number'})
  for element in main_div:
      print element
      l.append(float(str(element)[27:-7]))
  
  print l
  </code></pre>
  
  <p>But it is throwing a <code>UnicodeEncodeError</code> on the websoup definition line:</p>
  
  <pre><code>Traceback (most recent call last):
    File "climate_nasa_change.py", line 10, in &lt;module&gt;
      web_soup = soup(urllib2.urlopen(url,  context=context))
    File "C:\Python27\lib\site-packages\BeautifulSoup.py", line 1522, in __init__
      BeautifulStoneSoup.__init__(self, *args, **kwargs)
    File "C:\Python27\lib\site-packages\BeautifulSoup.py", line 1147, in __init__
      self._feed(isHTML=isHTML)
    File "C:\Python27\lib\site-packages\BeautifulSoup.py", line 1189, in _feed
      SGMLParser.feed(self, markup)
    File "C:\Python27\lib\sgmllib.py", line 104, in feed
      self.goahead(0)
    File "C:\Python27\lib\sgmllib.py", line 143, in goahead
      k = self.parse_endtag(i)
    File "C:\Python27\lib\sgmllib.py", line 320, in parse_endtag
      self.finish_endtag(tag)
    File "C:\Python27\lib\sgmllib.py", line 358, in finish_endtag
      method = getattr(self, 'end_' + tag)
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 11-12: ordinal not in range(128)
  </code></pre>
  
  <p><strong>UPDATE4</strong>: aaaaaaaaaaaaaand now its magically working. I really have no idea what the hell is going on right now. </p>
  
  <p><strong>UPDATE5</strong>: and now its broken. I swear I am not changing. Seriously considering exorcising my computer. Please help. </p>
  
  <p><strong>UPDATE6</strong>: Just tried pinging climate.nasa.gov. It doesn't always go through, despite the page consistently loading in my browser. Could this be causing BeautifulSoup to fail?</p>