title: Running Scrapy spiders in a Celery task
body: <p>I have a Django site where a scrape happens when a user requests it, and my code kicks off a Scrapy spider standalone script in a new process. Naturally, this isn't working  with an increase of users.</p>
  
  <p>Something like this:</p>
  
  <pre><code>class StandAloneSpider(Spider):
      #a regular spider
  
  settings.overrides['LOG_ENABLED'] = True
  #more settings can be changed...
  
  crawler = CrawlerProcess( settings )
  crawler.install()
  crawler.configure()
  
  spider = StandAloneSpider()
  
  crawler.crawl( spider )
  crawler.start()
  </code></pre>
  
  <p>I've decided to use Celery and use workers to queue up the crawl requests.</p>
  
  <p>However, I'm running into issues with Tornado reactors not being able to restart. The first and second spider runs successfully, but subsequent spiders will throw the ReactorNotRestartable error.</p>
  
  <p>Anyone can share any tips with running Spiders within the Celery framework?</p>