title: Problems with character-encoding when webscraping with scrapy
body: <p>I have problem with the encoding of the text, I am scraping from a website. Specifically the Danish letters æ, ø, and å are coming out wrong. I feel confident that the encoding of the webpage is UTF-8, since the browser is showing it correctly with this encoding.</p>
  
  <p>I have tried using BeautifulSoup as many of the other posts have suggested, but it wasn't for the better. However, I probably did it wrong.</p>
  
  <p>I am using python 2.7 on a windows 7 32 bit OS.</p>
  
  <p>The code I have is this:</p>
  
  <pre><code># -*- coding: UTF-8 -*-
  
  from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  from scrapy.item import Item, Field
  
  class Sale(Item):
      Adresse = Field()
      Pris = Field()
      Salgsdato = Field()
      SalgsType = Field()
      KvmPris = Field()
      Rum = Field()
      Postnummer = Field()
      Boligtype = Field()
      Kvm = Field()
      Bygget = Field()
  
  class HouseSpider(BaseSpider):
      name = 'House'
      allowed_domains = ["http://boliga.dk/"]
      start_urls = ['http://www.boliga.dk/salg/resultater?so=1&amp;type=Villa&amp;type=Ejerlejlighed&amp;type=R%%C3%%A6kkehus&amp;kom=&amp;amt=&amp;fraPostnr=&amp;tilPostnr=&amp;iPostnr=&amp;gade=&amp;min=&amp;max=&amp;byggetMin=&amp;byggetMax=&amp;minRooms=&amp;maxRooms=&amp;minSize=&amp;maxSize=&amp;minsaledate=1992&amp;maxsaledate=today&amp;kode=&amp;p=%d' %n for n in xrange(1, 3, 1)]
  
      def parse(self, response):
          hxs = HtmlXPathSelector(response)
          sites = hxs.select("id('searchresult')/tr")
          items = []      
          for site in sites:
              item = Sale()
              item['Adresse'] = site.select("td[1]/a[1]/text()").extract()
              item['Pris'] = site.select("td[2]/text()").extract()
              item['Salgsdato'] = site.select("td[3]/text()").extract()
              item['SalgsType'] = site.select("td[4]/text()").extract()
              item['KvmPris'] = site.select("td[5]/text()").extract()
              item['Rum'] = site.select("td[6]/text()").extract()
              item['Postnummer'] = site.select("td[7]/text()").extract()
              item['Boligtype'] = site.select("td[8]/text()").extract()
              item['Kvm'] = site.select("td[9]/text()").extract()
              item['Bygget'] = site.select("td[10]/text()").extract()
              items.append(item)
          return items
  </code></pre>
  
  <p>It is the items 'Adresse' and 'Salgstype' that contain æ, ø, and å. Any help is greatly appreciated!</p>
  
  <p>Cheers,</p>