title: Queueing and monitoring concurrent spiders with CrawlProcess
body: <p>I have an actively developing Scrapy project of 70+ spiders that I want to run regularly (daily/weekly), with about 4 spiders running in concurrency, with ability to stop, start, and monitor the spiders individually.</p>
  
  <p>I've ruled out using ScrapyD because it adds layers of complexity to what I already have and requires deploying a new egg for even a small change to any part of the project. </p>
  
  <p>There's a more general version of this question <a href="https://stackoverflow.com/questions/48088582/running-dozens-of-scrapy-spiders-in-a-controlled-manner">here</a> with an accepted answer using the command line:</p>
  
  <pre><code>$ scrapy list | xargs -P 4 -n 1 scrapy crawl
  </code></pre>
  
  <p>This a pretty good solution but requires working through the bash shell, which seems clumsy, and there is no way to stop or interact with spiders programmatically once running. I want to keep things as simple as they can be, preferably not leaving my Python IDE. I also need to be able to stop, add, or remove a spider from this queue without doing so in separate bash windows or starting over entirely. </p>
  
  <p>I'm puzzled why there is no clear way to go about this in the docs. Looking at similar but not identical questions <a href="https://stackoverflow.com/questions/21345092/running-multiple-scrapy-spiders-the-easy-way-python">here</a>, <a href="https://stackoverflow.com/questions/48537040/run-multiple-spiders-from-script-in-scrapy-in-loop">here</a> and <a href="https://stackoverflow.com/questions/15564844/locally-run-all-of-the-spiders-in-scrapy">here</a>, answers mostly point either to ScrapyD or to starting separate Twisted reactors separately, which seems complicated and potentially error-prone.</p>
  
  <p>Using CrawlSpider, I know I can do something like:</p>
  
  <pre><code>from scrapy.spiderloader import SpiderLoader
  from scrapy.utils.project import get_project_settings
  from scrapy.crawler import CrawlerProcess
  
  settings = get_project_settings()
  crawler = SpiderLoader.from_settings(settings)
  spiders= SpiderLoader.list(crawler)
  process = CrawlerProcess(settings)
  
  def run_all_spiders():
     for spider in spiders:
       process.crawl(spider)
  
     process.start()
  
  run_all_spiders()
  </code></pre>
  
  <p>But this seems to try running all the spiders at once the minute process.start() is called and stops processing further code until completed. <strong>Using only python, is there a way to enqueue many spiders so a certain number run at once, still having control over them?</strong></p>