Take the 2-minute tour ×

This same code crawls yellowbook with no issues and as expected. Change the rule over to CL and it hits the first url and then teeters out with no relevant output.

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigs.items import CraigsItem

class MySpider(CrawlSpider):
        name = "craigs"
        allowed_domains = ["craiglist.org"]

        start_urls = ["http://newyork.craigslist.org/cpg/"]

        rules = [Rule(SgmlLinkExtractor(restrict_xpaths=('/html/body/blockquote[3]/p/a',)), follow=True, callback='parse_profile')]

        def parse_profile(self, response):
                found = []
                img = CraigsItem()
                hxs = HtmlXPathSelector(response)
                img['title'] = hxs.select('//h2[contains(@class, "postingtitle")]/text()').extract()
                img['text'] = hxs.select('//section[contains(@id, "postingbody")]/text()').extract()
                img['tags'] =  hxs.select('//html/body/article/section/section[2]/section[2]/ul/li[1]').extract()

                print found[0]
                return found[0]

Here is the output http://pastie.org/6087878 As you can see, it has no issue getting the first url to crawl http://newyork.craigslist.org/mnh/cpg/3600242403.html> but then dies.

I can use the CLI and dump all the links like this SgmlLinkExtractor(restrict_xpaths=('/html/body/blockquote[3]/p/a',)).extract_links(response) with xpaths or keyword SgmlLinkExtractor(allow=r'/cpg/.+').extract_links(response)
output -> http://pastie.org/6085322

but in the crawl, the same query fails. WTF??

share|improve this question
A tcpdump packet capture, when comparing the CLI action with 'does' recover the data, to the crawl action which does 'not' recover the data. You can clearly see that craigslist 'is' giving up the data. Following the GET, I recieve plenty of http data from the URL. But scrapy doesn't do anything with it, in the crawl. This 100% looks like a scrapy/(user) side error. I am stuck. tinypic.com/r/289lw9/6 tinypic.com/r/2dugcj/6 –  user1544207 Feb 7 '13 at 17:16
The script is run and performs the first GET of the start URL and the data is received from CL, as expected. But, there is never a second GET request that even leaves the NIC despite what scrapy SgmlLinkExtractor is saying in the debug. [craigs] DEBUG: Filtered offsite request to 'newyork.craigslist.org': <GET newyork.craigslist.org/lgi/cpg/3576532314.html>; Scrapy says it happens, but tcpdump tells a different story. –  user1544207 Feb 7 '13 at 18:34
My parse 'does' work. But data collection is never executed?? –  user1544207 Feb 7 '13 at 19:24
pastie.org/6089422 –  user1544207 Feb 7 '13 at 19:24
add comment

1 Answer

if you look in documentation you will see


your allowed domain is

 allowed_domains = ["craiglist.org"]

but you are trying to fetch a subdomain

02-07 15:39:03+0000 [craigs] DEBUG: Filtered offsite request to 'newyork.craigslist.org': <GET http://newyork.craigslist.org/mnh/cpg/3600242403.html>

that is why it is filtered

either remove allowed_domains from your crawler of add proper domains in it for avoiding filtered offsite requests

share|improve this answer
Thank you! Removing the allowed domains worked –  user1544207 Feb 8 '13 at 23:12
@user1544207 dont forget to accept the answer so that other can get benefit from this post (can accept this answer by clicking on tick mark bellow upvote sign) –  akhter wahab Feb 10 '13 at 18:32
add comment

Your Answer


