title: Scrapy CrawlSpider retry scrape
body: <p>For a page that I'm trying to scrape, I sometimes get a "placeholder" page back in my response that contains some javascript that autoreloads until it gets the real page. I can detect when this happens and I want to retry downloading and scraping the page. The logic that I use in my CrawlSpider is something like:</p>
  
  <pre><code>def parse_page(self, response):
      url = response.url
  
      # Check to make sure the page is loaded
      if 'var PageIsLoaded = false;' in response.body:
          self.logger.warning('parse_page encountered an incomplete rendering of {}'.format(url))
          yield Request(url, self.parse, dont_filter=True)
          return
  
      ...
      # Normal parsing logic
  </code></pre>
  
  <p>However, it seems like when the retry logic gets called and a new Request is issued, the pages and the links they contain don't get crawled or scraped. My thought was that by using <code>self.parse</code> which the CrawlSpider uses to apply the crawl rules and <code>dont_filter=True</code>, I could avoid the duplicate filter. However with <code>DUPEFILTER_DEBUG = True</code>, I can see that the retry requests get filtered away. </p>
  
  <p>Am I missing something, or is there a better way to handle this? I'd like to avoid the complication of doing dynamic js rendering using something like splash if possible, and this only happens intermittently. </p>