title: Python - Problems creating a list of URLs using BeautifulSoup
body: <p>I am trying to make a Python Crawler using BeautifulSoup, but I receive an error that I am trying to write a non-String or other character buffer type to a file.  From examining the program output, I found that my list contains many items that are None.  In addition to having None, I also have a lot of images and things that are not links but are image links inside of my list.  How can I only add the URLs to my list?</p>
  
  <pre><code>    import urllib
      from BeautifulSoup import *
  
      try:
          with open('url_file', 'r') as f:
              url_list = [line.rstrip('\n') for line in f]
              f.close()
          with open('old_file', 'r') as x:
              old_list = [line.rstrip('\n') for line in f]
              f.close()
      except:
          url_list = list()
          old_list = list()
          #for Testing
          url_list.append("http://www.dinamalar.com/")
  
  
      count = 0
  
  
      for item in url_list:
          try:
              count = count + 1
              if count &gt; 5:
                  break
  
              html = urllib.urlopen(item).read()
              soup = BeautifulSoup(html)
              tags = soup('a')
  
              for tag in tags:
  
                  if tag in old_list:
                      continue
                  else:
                      url_list.append(tag.get('href', None))
  
  
              old_list.append(item)
              #for testing
              print url_list
          except:
              continue
  
      with open('url_file', 'w') as f:
          for s in url_list:
              f.write(s)
              f.write('\n')
  
  
      with open('old_file', 'w') as f:
          for s in old_list:
              f.write(s)
  </code></pre>