title: Re-feed some already crawled URL to spider/scheduler
body: <p>There is some URL (domain.com/list) which lists 10 links I need to crawl periodically. These links change pretty often each 30 seconds roughly, so I need to constantly re-crawl domain.com/list to check for new links. Crawling all this links sometimes takes more than 30 seconds because of size so I cannot cron a script to do it each 30 seconds since I could end up with several concurrent spiders. Missing some links because the spider takes too long during the first run is an acceptable situation though.</p>
  
  <p>I wrote some spider middleware to remove already visited links (for cases in which the links change only partially). I tried to include in that process_spider_output a new Request for domain.com/list with dont_filter=True so the list is feeded again to the scheduler, but I end up with tons of those requests. My code is:</p>
  
  <pre><code>def process_spider_output(self, response, result, spider):
  
      for i in result:
          if isinstance(i, Request):
              state = spider.state.get('crawled_links', deque([]))
              if unquote(i.url) in state or i.url in state:
                  print "removed %s" % i
                  continue
          yield i
  
      yield spider.make_requests_from_url('http://domain.com/list')
  </code></pre>
  
  <p>This seems pretty ugly and I am not sure whether it works as intended. I also tried to hook spider idle and closed signals to try to re-crawl the site without success. </p>
  
  <p>What's the best way to re-crawl specific urls to monitor changes that occur very often, and without closing the spider in use?</p>
  
  <p>thanks in advance</p>