Problems with character-encoding when webscraping with scrapy <p>I have problem with the encoding of the text, I am scraping from a website. Specifically the Danish letters æ, ø, and å are coming out wrong. I feel confident that the encoding of the webpage is UTF-8, since the browser is showing it correctly with this encoding.</p> <p>I have tried using BeautifulSoup as many of the other posts have suggested, but it wasn't for the better. However, I probably did it wrong.</p> <p>I am using python 2.7 on a windows 7 32 bit OS.</p> <p>The code I have is this:</p> <pre><code># -*- coding: UTF-8 -*- from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from scrapy.item import Item, Field class Sale(Item): Adresse = Field() Pris = Field() Salgsdato = Field() SalgsType = Field() KvmPris = Field() Rum = Field() Postnummer = Field() Boligtype = Field() Kvm = Field() Bygget = Field() class HouseSpider(BaseSpider): name = 'House' allowed_domains = ["http://boliga.dk/"] start_urls = ['http://www.boliga.dk/salg/resultater?so=1&amp;type=Villa&amp;type=Ejerlejlighed&amp;type=R%%C3%%A6kkehus&amp;kom=&amp;amt=&amp;fraPostnr=&amp;tilPostnr=&amp;iPostnr=&amp;gade=&amp;min=&amp;max=&amp;byggetMin=&amp;byggetMax=&amp;minRooms=&amp;maxRooms=&amp;minSize=&amp;maxSize=&amp;minsaledate=1992&amp;maxsaledate=today&amp;kode=&amp;p=%d' %n for n in xrange(1, 3, 1)] def parse(self, response): hxs = HtmlXPathSelector(response) sites = hxs.select("id('searchresult')/tr") items = [] for site in sites: item = Sale() item['Adresse'] = site.select("td[1]/a[1]/text()").extract() item['Pris'] = site.select("td[2]/text()").extract() item['Salgsdato'] = site.select("td[3]/text()").extract() item['SalgsType'] = site.select("td[4]/text()").extract() item['KvmPris'] = site.select("td[5]/text()").extract() item['Rum'] = site.select("td[6]/text()").extract() item['Postnummer'] = site.select("td[7]/text()").extract() item['Boligtype'] = site.select("td[8]/text()").extract() item['Kvm'] = site.select("td[9]/text()").extract() item['Bygget'] = site.select("td[10]/text()").extract() items.append(item) return items </code></pre> <p>It is the items 'Adresse' and 'Salgstype' that contain æ, ø, and å. Any help is greatly appreciated!</p> <p>Cheers,</p>