title: Scrapy - Non-ascii-character declared, but no encoding declared
body: <p>I'm attempting to scrape some basic data off this site as an exercise to learn more about scrapy, and as proof of concept for a university project:
  <a href="http://steamdb.info/sales/" rel="nofollow">http://steamdb.info/sales/</a></p>
  
  <p>When I was using the scrapy shell I was able to get the information I wanted using the following XPath:</p>
  
  <pre><code>sel.xpath(‘//tbody/tr[1]/td[2]/a/text()’).extract()
  </code></pre>
  
  <p>which should return the title of the game of the first row of the table, in the structure:</p>
  
  <pre><code>&lt;tbody&gt;
       &lt;tr&gt;
            &lt;td&gt;&lt;/td&gt;
            &lt;td&gt;&lt;a&gt;stuff I want here&lt;/a&gt;&lt;/td&gt;
  ...
  </code></pre>
  
  <p>And it does, in the shell. </p>
  
  <p>However, when I attempt to put this into a spider (steam.py):</p>
  
  <pre><code>1 from scrapy.spider import BaseSpider
  2 from scrapy.selector import HtmlXPathSelector
  3 from steam_crawler.items import SteamItem
  4 from scrapy.selector import Selector
  5 
  6 class SteamSpider(BaseSpider):
  7     name = "steam"
  8     allowed_domains = ["http://steamdb.info/"]
  9     start_urls = ['http://steamdb.info/sales/?displayOnly=all&amp;category=0&amp;cc=uk']
  10     def parse(self, response):
  11         sel = Selector(response)
  12         sites = sel.xpath("//tbody")
  13         items = []
  14         count = 1
  15         for site in sites:
  16             item = SteamItem()
  17             item ['title'] = sel.xpath('//tr['+ str(count) +']/td[2]/a/text()').extract().encode('utf-8')
  18             item ['price'] = sel.xpath('//tr['+ str(count) +']/td[@class=“price-final”]/text()').extract().encode('utf-8')
  19             items.append(item)
  20             count = count + 1
  21         return items
  </code></pre>
  
  <p>I get the following error:</p>
  
  <pre><code>    ricks-mbp:steam_crawler someuser$ scrapy crawl steam -o items.csv -t csv
  Traceback (most recent call last):
    File "/usr/local/bin/scrapy", line 5, in &lt;module&gt;
      pkg_resources.run_script('Scrapy==0.20.0', 'scrapy')
    File "build/bdist.macosx-10.9-intel/egg/pkg_resources.py", line 492, in run_script
  
    File "build/bdist.macosx-10.9-intel/egg/pkg_resources.py", line 1350, in run_script
      for name in eagers:
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/EGG-INFO/scripts/scrapy", line 4, in &lt;module&gt;
      execute()
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/cmdline.py", line 143, in execute
      _run_print_help(parser, _run_command, cmd, args, opts)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/cmdline.py", line 89, in _run_print_help
      func(*a, **kw)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/cmdline.py", line 150, in _run_command
      cmd.run(args, opts)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/commands/crawl.py", line 47, in run
      crawler = self.crawler_process.create_crawler()
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/crawler.py", line 87, in create_crawler
      self.crawlers[name] = Crawler(self.settings)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/crawler.py", line 25, in __init__
      self.spiders = spman_cls.from_crawler(self)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/spidermanager.py", line 35, in from_crawler
      sm = cls.from_settings(crawler.settings)
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/spidermanager.py", line 31, in from_settings
      return cls(settings.getlist('SPIDER_MODULES'))
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/spidermanager.py", line 22, in __init__
      for module in walk_modules(name):
    File "/Library/Python/2.7/site-packages/Scrapy-0.20.0-py2.7.egg/scrapy/utils/misc.py", line 68, in walk_modules
      submod = import_module(fullpath)
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
      __import__(name)
    File "/xxx/scrape/steam/steam_crawler/spiders/steam.py", line 18
  SyntaxError: Non-ASCII character '\xe2' in file /xxx/scrape/steam/steam_crawler/spiders/steam.py on line 18, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
  </code></pre>
  
  <p>I have a feeling that all I need to do is somehow tell scrapy that the characters will follow utf-8 not ascii - as there are £'s etc. But from what I can gather, its supposed to gather this information from the head of the page its scraping which in the case of this site is:</p>
  
  <pre><code>&lt;meta charset="utf-8"&gt;
  </code></pre>
  
  <p>Which leaves me baffled! Any insight/reading that isn't the scrapy docs themselves I would be interested in too!</p>