Scrapy pipeline html parsing <p>I have a spider with 3 items: url, title and category. </p> <p>They are loading fine in raw html but now I would like to convert title and category into plain text using html2test in a pipeline..</p> <p>Here is my incorrect pipeline code, could someone help debugging it.</p> <p>Thank you</p> <pre><code>import html2text import csv from tutorial import settings def write_to_csv(item): writer = csv.writer(open(settings.csv_file_path, 'a'), lineterminator='\n') writer.writerow([item[key] for key in item.keys()]) class TutorialPipeline(object): def process_item(self, item, spider): h = html2text.HTML2Text() h.ignore_images = True h.handle(item['title']).strip() h.handle(item['category']).strip() write_to_csv(item) return item </code></pre> <p>Spider code</p> <pre><code>import scrapy from scrapy.http import Request from scrapy.contrib.spiders import CrawlSpider,Rule from scrapy.contrib.linkextractors import LinkExtractor from tutorial.items import TutorialItem class tuto(CrawlSpider): name = "tuto" allowed_domains = ['emedicine.medscape.com'] start_urls=["http://emedicine.medscape.com"] rules=( Rule( LinkExtractor(restrict_xpaths ='//div[@id="browsespecialties"]'),callback='follow_pages', follow=True), ) def follow_pages(self, response): for sel in response.xpath('//div[@class="maincolbox"]//a/@href').extract(): yield Request("http://emedicine.medscape.com/" + sel, callback = self.parse_item) def parse_item(self, response): item = TutorialItem() item['url'] = response.url item['background'] = response.xpath('//div[@class="refsection_content"]').extract() item['title'] = response.xpath('//h1').extract() yield item </code></pre>