title: Why does running multiple scrapy spiders through CrawlerProcess cause the spider_idle signal to fail?
body: <p>I need to make thousands of requests that require a session token for authorization.</p>
  
  <p>Queuing all the requests at once results in thousands of requests failing because the session token expires before the later requests are issued.</p>
  
  <p>So, I am issuing a reasonable number of requests that will reliably complete before the session token expires.</p>
  
  <p>When a batch of requests completes, the spider_idle signal is triggered.</p>
  
  <p>If further requests are needed, the signal handler requests a new session token be used with the next batch of requests.</p>
  
  <p>This works when running one spider normally, or one spider through CrawlerProcess.</p>
  
  <p>However, the spider_idle signal fails with multiple spiders run through CrawlerProcess.</p>
  
  <p>One spider will execute the spider_idle signal as expected, but the others fail with this exception:</p>
  
  <pre><code>2019-06-14 10:41:22 [scrapy.utils.signal] ERROR: Error caught on signal handler: &lt;bound method ?.spider_idle of &lt;SpideIdleTest None at 0x7f514b33c550&gt;&gt;
  Traceback (most recent call last):
    File "/home/loren/.virtualenv/spider_idle_test/local/lib/python2.7/site-packages/scrapy/utils/signal.py", line 30, in send_catch_log
      *arguments, **named)
    File "/home/loren/.virtualenv/spider_idle_test/local/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
      return receiver(*arguments, **named)
    File "fails_with_multiple_spiders.py", line 25, in spider_idle
      spider)
    File "/home/loren/.virtualenv/spider_idle_test/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 209, in crawl
      "Spider %r not opened when crawling: %s" % (spider.name, request)
  </code></pre>
  
  <p>I created a repo that shows the spider_idle behaving as expected with a single spider, and failing with multiple spiders.</p>
  
  <p><a href="https://github.com/loren-magnuson/scrapy_spider_idle_test" rel="nofollow noreferrer">https://github.com/loren-magnuson/scrapy_spider_idle_test</a></p>
  
  <p>Here is the version that shows the failures:</p>
  
  <pre><code>import scrapy
  from scrapy.crawler import CrawlerProcess
  from scrapy import Request, signals
  from scrapy.exceptions import DontCloseSpider
  from scrapy.xlib.pydispatch import dispatcher
  
  
  class SpiderIdleTest(scrapy.Spider):
      custom_settings = {
          'CONCURRENT_REQUESTS': 1,
          'DOWNLOAD_DELAY': 2,
      }
  
      def __init__(self):
          dispatcher.connect(self.spider_idle, signals.spider_idle)
          self.idle_retries = 0
  
      def spider_idle(self, spider):
          self.idle_retries += 1
          if self.idle_retries &lt; 3:
              self.crawler.engine.crawl(
                  Request('https://www.google.com',
                          self.parse,
                          dont_filter=True),
                  spider)
              raise DontCloseSpider("Stayin' alive")
  
      def start_requests(self):
          yield Request('https://www.google.com', self.parse)
  
      def parse(self, response):
          print(response.css('title::text').extract_first())
  
  
  process = CrawlerProcess()
  process.crawl(SpiderIdleTest)
  process.crawl(SpiderIdleTest)
  process.crawl(SpiderIdleTest)
  process.start()
  </code></pre>