title: poor performance of my crawlspider following links conditional on page-text
body: <p>I am using a crawlspider to follow links only if a page contains some key words. In general this works but the performance is poor. Main parts of the code are:</p>
  
  <p>class CrawlSpider_PlainVanilla(CrawlSpider):</p>
  
  <pre><code>name = "my_crawlspider"
  
  rules = (Rule(link_extractor=None,callback='parse_link', follow=True, process_request='filter_response'),)
  </code></pre>
  
  <p>and </p>
  
  <pre><code>def filter_response(self, request, response):
      my_body = response.xpath(self.my_str).re(self.my_reg)
      if my_body != []:
          return request
  </code></pre>
  
  <p>where self.my_str is a xpath selector (excluding elements like meta or script) and self.my_reg is a regular expression. The rest of the code is standard. The strange thing is, that I have implemented the same stuff with a standard scrapy.Spider and here the performance is quite good. I would appreciate any hint concerning drawbacks of my code.</p>
  
  <p>Best regards</p>
  
  <p>Gerd</p>