I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I've tried to handle the list index out of error issue using a customized function within scrapy. There might be better ways to do so but What I noticed with the one I've tried that when there are several fields to scrape then this customized function play a vital role to deal with those fields whose values are none. I'll be vary glad to have any better ideas to do the same. Thanks in advance.

Here is what I've tried:

import scrapy
from scrapy.crawler import CrawlerProcess

class CraigsspSpider(scrapy.Spider):
    name = 'craigssp'

    start_urls = ['http://bangalore.craigslist.co.in/search/rea?s=120']
    #the below function is able to handle IndexError
    get_result = lambda item,path:item.css(path).extract_first() if item.css(path) else ""

    def parse(self, response):
        for items in response.css(".result-info"):
            date = CraigsspSpider.get_result(items,".result-date::text")
            name = CraigsspSpider.get_result(items,".hdrlnk::text")
            price = CraigsspSpider.get_result(items,".result-price::text")

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',   
  • \$\begingroup\$ Does it work as intended? \$\endgroup\$
    – Mast
    Mar 30 '18 at 17:27
  • 1
    \$\begingroup\$ Yes, it works flawlessly. \$\endgroup\$
    – SIM
    Mar 30 '18 at 17:44


Browse other questions tagged .